* do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH. @ 2017-04-26 15:03 Kristian Amlie 2017-04-26 15:03 ` [PATCH] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature Kristian Amlie 2017-05-22 7:08 ` do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH Kristian Amlie 0 siblings, 2 replies; 41+ messages in thread From: Kristian Amlie @ 2017-04-26 15:03 UTC (permalink / raw) To: openembedded-core This is a direct followup from the earlier f6a064d969f414 commit in wic. It works more or less the same way: The variable specifies a list of directories relative to the root of the rootfs, and these directories will be excluded from the resulting rootfs image. If an entry ends with a slash, only the contents are omitted, not the directory itself. This is early feedback call, and several things are missing here (see below). However, I would like to know if at least the basic approach is acceptable. Reviewers pressed on time really only need to look at the topmost hunk as well as the last couple of lines of the two functions. Those show what the basic idea is. What's missing: * Documentation * Test coverage * Respecting IMAGE_ROOTFS_EXCLUDE_PATH is a behavior change, and isn't appropriate for all image creators. In fact, for the the main use case [1] to work it must not always be respected. My idea for the last point is that the variable will not be respected by default, and each of the pure filesystem image creators (btrfs, ext2/3/4, etc) will set a flag that mark their image creator as a "pure filesystem", which would respect this variable. Image creators that make partitions, however, should not respect it, because they will use wic for that purpose instead, and then they need to have access to the entire rootfs. Hence they should not set the flag. I'm imagining something like this: d.setVarFlag('do_image_%s' % t, 'respect_exclude_path', '1') although I'm not sure if that's the best approach (tips are welcome). [1] The use case for this feature is the following: We are creating two types of images in Mender: One complete partitioned image, and one image containing only the rootfs. In order to be able to populate non-rootfs partitions, we want to use the regular bitbake recipes for building these components, but then exclude certain directories from the rootfs image. Using wic, we are adding those directories back into the partitioned image, but under different partitions. Unlike the rootfs image, the other partitions are not expected to be rebuilt after the first rollout. -- Kristian ^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-04-26 15:03 do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH Kristian Amlie @ 2017-04-26 15:03 ` Kristian Amlie 2017-05-22 7:08 ` do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH Kristian Amlie 1 sibling, 0 replies; 41+ messages in thread From: Kristian Amlie @ 2017-04-26 15:03 UTC (permalink / raw) To: openembedded-core This is a direct followup from the earlier f6a064d969f414 commit in wic. It works more or less the same way: The variable specifies a list of directories relative to the root of the rootfs, and these directories will be excluded from the resulting rootfs image. If an entry ends with a slash, only the contents are omitted, not the directory itself. Signed-off-by: Kristian Amlie <kristian.amlie@mender.io> --- meta/classes/image.bbclass | 73 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 71 insertions(+), 2 deletions(-) diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 405fd73..a309435 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -496,8 +496,8 @@ python () { d.setVar('do_image_%s' % t, '\n'.join(cmds)) d.setVarFlag('do_image_%s' % t, 'func', '1') d.setVarFlag('do_image_%s' % t, 'fakeroot', '1') - d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size') - d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks') + d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size prepare_excluded_directories') + d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks cleanup_excluded_directories') d.setVarFlag('do_image_%s' % t, 'subimages', ' '.join(subimages)) d.appendVarFlag('do_image_%s' % t, 'vardeps', ' '.join(vardeps)) d.appendVarFlag('do_image_%s' % t, 'vardepsexclude', 'DATETIME') @@ -506,6 +506,75 @@ python () { bb.build.addtask('do_image_%s' % t, 'do_image_complete', after, d) } +python prepare_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + import shutil + from oe.path import copyhardlinktree + + exclude_list = exclude_var.split() + + taskname = d.getVar("BB_CURRENTTASK") + + rootfs_orig = d.getVar('IMAGE_ROOTFS') + # We need a new rootfs directory we can delete files from. Copy to + # workdir. + new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname)) + + if os.path.lexists(new_rootfs): + shutil.rmtree(os.path.join(new_rootfs)) + + copyhardlinktree(rootfs_orig, new_rootfs) + + for orig_path in exclude_list: + path = orig_path + if os.path.isabs(path): + msger.error("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path) + + full_path = os.path.realpath(os.path.join(new_rootfs, path)) + + # Disallow climbing outside of parent directory using '..', + # because doing so could be quite disastrous (we will delete the + # directory). + if not full_path.startswith(new_rootfs): + msger.error("'%s' points to a path outside the rootfs" % orig_path) + + if path.endswith(os.sep): + # Delete content only. + for entry in os.listdir(full_path): + full_entry = os.path.join(full_path, entry) + if os.path.isdir(full_entry) and not os.path.islink(full_entry): + shutil.rmtree(full_entry) + else: + os.remove(full_entry) + else: + # Delete whole directory. + shutil.rmtree(full_path) + + # Save old value for cleanup later. + d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig) + d.setVar('IMAGE_ROOTFS', new_rootfs) +} + +python cleanup_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + import shutil + + rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS') + rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG') + # This should never happen, since we should have set it to a different + # directory in the prepare function. + assert rootfs_dirs_excluded != rootfs_orig + + shutil.rmtree(rootfs_dirs_excluded) + d.setVar('IMAGE_ROOTFS', rootfs_orig) +} + # # Compute the rootfs size # -- 2.7.4 ^ permalink raw reply related [flat|nested] 41+ messages in thread
* Re: do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH. 2017-04-26 15:03 do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH Kristian Amlie 2017-04-26 15:03 ` [PATCH] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature Kristian Amlie @ 2017-05-22 7:08 ` Kristian Amlie 2017-05-22 8:46 ` Ed Bartosh 1 sibling, 1 reply; 41+ messages in thread From: Kristian Amlie @ 2017-05-22 7:08 UTC (permalink / raw) To: openembedded-core Ping. Would be great to get some feedback so I can continue working on this. -- Kristian On 26/04/17 17:03, Kristian Amlie wrote: > This is a direct followup from the earlier f6a064d969f414 commit in wic. It > works more or less the same way: The variable specifies a list of directories > relative to the root of the rootfs, and these directories will be excluded from > the resulting rootfs image. If an entry ends with a slash, only the contents are > omitted, not the directory itself. > > This is early feedback call, and several things are missing here (see > below). However, I would like to know if at least the basic approach is > acceptable. Reviewers pressed on time really only need to look at the topmost > hunk as well as the last couple of lines of the two functions. Those show what > the basic idea is. > > What's missing: > > * Documentation > > * Test coverage > > * Respecting IMAGE_ROOTFS_EXCLUDE_PATH is a behavior change, and isn't > appropriate for all image creators. In fact, for the the main use case [1] to > work it must not always be respected. > > My idea for the last point is that the variable will not be respected by > default, and each of the pure filesystem image creators (btrfs, ext2/3/4, etc) > will set a flag that mark their image creator as a "pure filesystem", which > would respect this variable. Image creators that make partitions, however, > should not respect it, because they will use wic for that purpose instead, and > then they need to have access to the entire rootfs. Hence they should not set > the flag. > > I'm imagining something like this: > > d.setVarFlag('do_image_%s' % t, 'respect_exclude_path', '1') > > although I'm not sure if that's the best approach (tips are welcome). > > [1] The use case for this feature is the following: We are creating two types of > images in Mender: One complete partitioned image, and one image containing only > the rootfs. In order to be able to populate non-rootfs partitions, we want to > use the regular bitbake recipes for building these components, but then exclude > certain directories from the rootfs image. Using wic, we are adding those > directories back into the partitioned image, but under different > partitions. Unlike the rootfs image, the other partitions are not expected to be > rebuilt after the first rollout. > ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH. 2017-05-22 7:08 ` do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH Kristian Amlie @ 2017-05-22 8:46 ` Ed Bartosh 2017-05-22 9:38 ` Kristian Amlie 0 siblings, 1 reply; 41+ messages in thread From: Ed Bartosh @ 2017-05-22 8:46 UTC (permalink / raw) To: Kristian Amlie; +Cc: openembedded-core On Mon, May 22, 2017 at 09:08:41AM +0200, Kristian Amlie wrote: > Ping. Would be great to get some feedback so I can continue working on this. > I like the idea. Lately I was working on generic EFI implementation and thought that it would be nice to be able to package EFI artifacts and get them installed the same way as other packages. With IMAGE_ROOTFS_EXCLUDE_PATH = '<directory with efi artifacts>' it would be possible to not include that directory into the rootfs if needed. > -- > Kristian > > On 26/04/17 17:03, Kristian Amlie wrote: > > This is a direct followup from the earlier f6a064d969f414 commit in wic. It > > works more or less the same way: The variable specifies a list of directories > > relative to the root of the rootfs, and these directories will be excluded from > > the resulting rootfs image. If an entry ends with a slash, only the contents are > > omitted, not the directory itself. > > > > This is early feedback call, and several things are missing here (see > > below). However, I would like to know if at least the basic approach is > > acceptable. Reviewers pressed on time really only need to look at the topmost > > hunk as well as the last couple of lines of the two functions. Those show what > > the basic idea is. > > > > What's missing: > > > > * Documentation > > > > * Test coverage > > > > * Respecting IMAGE_ROOTFS_EXCLUDE_PATH is a behavior change, and isn't > > appropriate for all image creators. In fact, for the the main use case [1] to > > work it must not always be respected. > > > > My idea for the last point is that the variable will not be respected by > > default, and each of the pure filesystem image creators (btrfs, ext2/3/4, etc) > > will set a flag that mark their image creator as a "pure filesystem", which > > would respect this variable. Image creators that make partitions, however, > > should not respect it, because they will use wic for that purpose instead, and > > then they need to have access to the entire rootfs. Hence they should not set > > the flag. > > > > I'm imagining something like this: > > > > d.setVarFlag('do_image_%s' % t, 'respect_exclude_path', '1') > > > > although I'm not sure if that's the best approach (tips are welcome). > > > > [1] The use case for this feature is the following: We are creating two types of > > images in Mender: One complete partitioned image, and one image containing only > > the rootfs. In order to be able to populate non-rootfs partitions, we want to > > use the regular bitbake recipes for building these components, but then exclude > > certain directories from the rootfs image. Using wic, we are adding those > > directories back into the partitioned image, but under different > > partitions. Unlike the rootfs image, the other partitions are not expected to be > > rebuilt after the first rollout. > > > -- -- Regards, Ed ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH. 2017-05-22 8:46 ` Ed Bartosh @ 2017-05-22 9:38 ` Kristian Amlie 2017-05-30 7:46 ` Kristian Amlie ` (5 more replies) 0 siblings, 6 replies; 41+ messages in thread From: Kristian Amlie @ 2017-05-22 9:38 UTC (permalink / raw) To: ed.bartosh; +Cc: openembedded-core On 22/05/17 10:46, Ed Bartosh wrote: > On Mon, May 22, 2017 at 09:08:41AM +0200, Kristian Amlie wrote: >> Ping. Would be great to get some feedback so I can continue working on this. >> > I like the idea. > > Lately I was working on generic EFI implementation > and thought that it would be nice to be able to package EFI artifacts > and get them installed the same way as other packages. With > IMAGE_ROOTFS_EXCLUDE_PATH = '<directory with efi artifacts>' it would be > possible to not include that directory into the rootfs if needed. Yep, great example, that's exactly what this feature is about! I'll continue with my current suggestion then. The thing I'm most unsure about is the "d.setVarFlag('do_image_%s' % t, 'respect_exclude_path', '1')" approach, but I will attempt that one if I don't hear any other suggestions or hit a road block. -- Kristian >> On 26/04/17 17:03, Kristian Amlie wrote: >>> This is a direct followup from the earlier f6a064d969f414 commit in wic. It >>> works more or less the same way: The variable specifies a list of directories >>> relative to the root of the rootfs, and these directories will be excluded from >>> the resulting rootfs image. If an entry ends with a slash, only the contents are >>> omitted, not the directory itself. >>> >>> This is early feedback call, and several things are missing here (see >>> below). However, I would like to know if at least the basic approach is >>> acceptable. Reviewers pressed on time really only need to look at the topmost >>> hunk as well as the last couple of lines of the two functions. Those show what >>> the basic idea is. >>> >>> What's missing: >>> >>> * Documentation >>> >>> * Test coverage >>> >>> * Respecting IMAGE_ROOTFS_EXCLUDE_PATH is a behavior change, and isn't >>> appropriate for all image creators. In fact, for the the main use case [1] to >>> work it must not always be respected. >>> >>> My idea for the last point is that the variable will not be respected by >>> default, and each of the pure filesystem image creators (btrfs, ext2/3/4, etc) >>> will set a flag that mark their image creator as a "pure filesystem", which >>> would respect this variable. Image creators that make partitions, however, >>> should not respect it, because they will use wic for that purpose instead, and >>> then they need to have access to the entire rootfs. Hence they should not set >>> the flag. >>> >>> I'm imagining something like this: >>> >>> d.setVarFlag('do_image_%s' % t, 'respect_exclude_path', '1') >>> >>> although I'm not sure if that's the best approach (tips are welcome). >>> >>> [1] The use case for this feature is the following: We are creating two types of >>> images in Mender: One complete partitioned image, and one image containing only >>> the rootfs. In order to be able to populate non-rootfs partitions, we want to >>> use the regular bitbake recipes for building these components, but then exclude >>> certain directories from the rootfs image. Using wic, we are adding those >>> directories back into the partitioned image, but under different >>> partitions. Unlike the rootfs image, the other partitions are not expected to be >>> rebuilt after the first rollout. >>> >> > ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH. 2017-05-22 9:38 ` Kristian Amlie @ 2017-05-30 7:46 ` Kristian Amlie 2017-05-30 7:46 ` [PATCH v2] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature Kristian Amlie 2017-05-30 8:01 ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev2) Patchwork ` (4 subsequent siblings) 5 siblings, 1 reply; 41+ messages in thread From: Kristian Amlie @ 2017-05-30 7:46 UTC (permalink / raw) To: openembedded-core Updated the patch and included documentation. It wasn't clear to me how to write a test for this. I couldn't spot any tests for the surrounding variables, but I might be missing something. If this is important, please let me know where I should add them. -- Kristian ^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH v2] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-05-30 7:46 ` Kristian Amlie @ 2017-05-30 7:46 ` Kristian Amlie 2017-06-19 9:01 ` Kristian Amlie 0 siblings, 1 reply; 41+ messages in thread From: Kristian Amlie @ 2017-05-30 7:46 UTC (permalink / raw) To: openembedded-core This is a direct followup from the earlier f6a064d969f414 commit in wic. It works more or less the same way: The variable specifies a list of directories relative to the root of the rootfs, and these directories will be excluded from the resulting rootfs image. If an entry ends with a slash, only the contents are omitted, not the directory itself. Since the intended use of the variable is to exclude certain directories from the rootfs, and then include said directories in other partitions, it is not natural for this variable to be respected for image creators that create multi partition images. Therefore the default is to ignore the variable, and image creators that create single root filesystems need to set do_image_myfs[respect_exclude_path] = "1" in order to honor it. Specifically, "wic" and "multiubi" have not received this variable flag, while others have. Signed-off-by: Kristian Amlie <kristian.amlie@mender.io> --- documentation/ref-manual/ref-variables.xml | 35 +++++++++++++ meta/classes/image.bbclass | 83 ++++++++++++++++++++++++++++-- meta/classes/image_types.bbclass | 13 +++++ 3 files changed, 128 insertions(+), 3 deletions(-) diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml index 94558df..27f8b8e 100644 --- a/documentation/ref-manual/ref-variables.xml +++ b/documentation/ref-manual/ref-variables.xml @@ -6041,6 +6041,41 @@ </glossdef> </glossentry> + <glossentry id='var-IMAGE_ROOTFS_EXCLUDE_PATH'><glossterm>IMAGE_ROOTFS_EXCLUDE_PATH</glossterm> + <info> + IMAGE_ROOTFS_EXCLUDE_PATH[doc] = "Specifies paths to omit from the rootfs." + </info> + <glossdef> + <para role="glossdeffirst"> +<!-- <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> --> + Specifies paths that should be omitted from the root filesystem, separated by + spaces. The paths must be relative to the root of the filesystem (for example + "usr"). Single root filesystem images will not include the path in the + filesystem, except if the path ends with a slash, then the directory will be + included, but without any content. + <literallayout class='monospaced'> + # Omit /usr completely from rootfs. + IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr</replaceable> ..." + # Include the directory /usr in rootfs, but not its content. + IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr/</replaceable> ..." + </literallayout> + </para> + <para> + Image creators that create multi partition images typically do not honor this + variable, and must provide their own method to split the directory + structure. For example, <filename>wic</filename> has + <filename>--exclude-path</filename> and <filename>--rootfs-dir</filename>. + </para> + <para> + The default for custom image creators is to ignore this variable. Those that + need to honor it must set: + <literallayout class='monospaced'> + do_image_<replaceable>custom</replaceable>[respect_exclude_path] = "1" + </literallayout> + </para> + </glossdef> + </glossentry> + <glossentry id='var-IMAGE_ROOTFS_EXTRA_SPACE'><glossterm>IMAGE_ROOTFS_EXTRA_SPACE</glossterm> <info> IMAGE_ROOTFS_EXTRA_SPACE[doc] = "Defines additional free disk space created in the image in Kbytes. By default, this variable is set to '0'." diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 85f6b9a..c53e383 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -118,7 +118,7 @@ def rootfs_variables(d): 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS', 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS', - 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY'] + 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'IMAGE_ROOTFS_EXCLUDE_PATH'] variables.extend(rootfs_command_variables(d)) variables.extend(variable_depends(d)) return " ".join(variables) @@ -496,8 +496,8 @@ python () { d.setVar('do_image_%s' % t, '\n'.join(cmds)) d.setVarFlag('do_image_%s' % t, 'func', '1') d.setVarFlag('do_image_%s' % t, 'fakeroot', '1') - d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size') - d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks') + d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size prepare_excluded_directories') + d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks cleanup_excluded_directories') d.setVarFlag('do_image_%s' % t, 'subimages', ' '.join(subimages)) d.appendVarFlag('do_image_%s' % t, 'vardeps', ' '.join(vardeps)) d.appendVarFlag('do_image_%s' % t, 'vardepsexclude', 'DATETIME') @@ -506,6 +506,83 @@ python () { bb.build.addtask('do_image_%s' % t, 'do_image_complete', after, d) } +python prepare_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + taskname = d.getVar("BB_CURRENTTASK") + + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1': + return + + import shutil + from oe.path import copyhardlinktree + + exclude_list = exclude_var.split() + + rootfs_orig = d.getVar('IMAGE_ROOTFS') + # We need a new rootfs directory we can delete files from. Copy to + # workdir. + new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname)) + + if os.path.lexists(new_rootfs): + shutil.rmtree(os.path.join(new_rootfs)) + + copyhardlinktree(rootfs_orig, new_rootfs) + + for orig_path in exclude_list: + path = orig_path + if os.path.isabs(path): + bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path) + + full_path = os.path.realpath(os.path.join(new_rootfs, path)) + + # Disallow climbing outside of parent directory using '..', + # because doing so could be quite disastrous (we will delete the + # directory). + if not full_path.startswith(new_rootfs): + bb.fatal("'%s' points to a path outside the rootfs" % orig_path) + + if path.endswith(os.sep): + # Delete content only. + for entry in os.listdir(full_path): + full_entry = os.path.join(full_path, entry) + if os.path.isdir(full_entry) and not os.path.islink(full_entry): + shutil.rmtree(full_entry) + else: + os.remove(full_entry) + else: + # Delete whole directory. + shutil.rmtree(full_path) + + # Save old value for cleanup later. + d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig) + d.setVar('IMAGE_ROOTFS', new_rootfs) +} + +python cleanup_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + taskname = d.getVar("BB_CURRENTTASK") + + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1': + return + + import shutil + + rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS') + rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG') + # This should never happen, since we should have set it to a different + # directory in the prepare function. + assert rootfs_dirs_excluded != rootfs_orig + + shutil.rmtree(rootfs_dirs_excluded) + d.setVar('IMAGE_ROOTFS', rootfs_orig) +} + # # Compute the rootfs size # diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass index 7749b00..b27dfe3 100644 --- a/meta/classes/image_types.bbclass +++ b/meta/classes/image_types.bbclass @@ -51,8 +51,10 @@ ZIP_COMPRESSION_LEVEL ?= "-9" JFFS2_SUM_EXTRA_ARGS ?= "" IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 ${EXTRA_IMAGECMD}" +do_image_jffs2[respect_exclude_path] = "1" IMAGE_CMD_cramfs = "mkfs.cramfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cramfs ${EXTRA_IMAGECMD}" +do_image_cramfs[respect_exclude_path] = "1" oe_mkext234fs () { fstype=$1 @@ -81,6 +83,9 @@ oe_mkext234fs () { IMAGE_CMD_ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}" IMAGE_CMD_ext3 = "oe_mkext234fs ext3 ${EXTRA_IMAGECMD}" IMAGE_CMD_ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}" +do_image_ext2[respect_exclude_path] = "1" +do_image_ext3[respect_exclude_path] = "1" +do_image_ext4[respect_exclude_path] = "1" MIN_BTRFS_SIZE ?= "16384" IMAGE_CMD_btrfs () { @@ -92,10 +97,14 @@ IMAGE_CMD_btrfs () { dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs count=${size} bs=1024 mkfs.btrfs ${EXTRA_IMAGECMD} -r ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs } +do_image_btrfs[respect_exclude_path] = "1" IMAGE_CMD_squashfs = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs ${EXTRA_IMAGECMD} -noappend" IMAGE_CMD_squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz" IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo" +do_image_squashfs[respect_exclude_path] = "1" +do_image_squashfs-xz[respect_exclude_path] = "1" +do_image_squashfs-lzo[respect_exclude_path] = "1" # By default, tar from the host is used, which can be quite old. If # you need special parameters (like --xattrs) which are only supported @@ -109,6 +118,7 @@ IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAM # required when extracting, but it seems prudent to use it in both cases. IMAGE_CMD_TAR ?= "tar" IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cvf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} ." +do_image_tar[respect_exclude_path] = "1" do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append" IMAGE_CMD_cpio () { @@ -129,6 +139,7 @@ IMAGE_CMD_cpio () { fi fi } +do_image_cpio[respect_exclude_path] = "1" ELF_KERNEL ?= "${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE}" ELF_APPEND ?= "ramdisk_size=32768 root=/dev/ram0 rw console=" @@ -137,6 +148,7 @@ IMAGE_CMD_elf () { test -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf && rm -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf mkelfImage --kernel=${ELF_KERNEL} --initrd=${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.cpio.gz --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf --append='${ELF_APPEND}' ${EXTRA_IMAGECMD} } +do_image_elf[respect_exclude_path] = "1" IMAGE_TYPEDEP_elf = "cpio.gz" @@ -194,6 +206,7 @@ IMAGE_CMD_ubi () { } IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ubifs ${MKUBIFS_ARGS}" +do_image_ubifs[respect_exclude_path] = "1" EXTRA_IMAGECMD = "" -- 2.7.4 ^ permalink raw reply related [flat|nested] 41+ messages in thread
* Re: [PATCH v2] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-05-30 7:46 ` [PATCH v2] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature Kristian Amlie @ 2017-06-19 9:01 ` Kristian Amlie 2017-06-19 14:01 ` Kristian Amlie 0 siblings, 1 reply; 41+ messages in thread From: Kristian Amlie @ 2017-06-19 9:01 UTC (permalink / raw) To: Kristian Amlie, openembedded-core Ping. I created https://bugzilla.yoctoproject.org/show_bug.cgi?id=11688 to help track this. -- Kristian On 30/05/17 09:46, Kristian Amlie wrote: > This is a direct followup from the earlier f6a064d969f414 commit in > wic. It works more or less the same way: The variable specifies a list > of directories relative to the root of the rootfs, and these > directories will be excluded from the resulting rootfs image. If an > entry ends with a slash, only the contents are omitted, not the > directory itself. > > Since the intended use of the variable is to exclude certain > directories from the rootfs, and then include said directories in > other partitions, it is not natural for this variable to be respected > for image creators that create multi partition images. Therefore the > default is to ignore the variable, and image creators that create > single root filesystems need to set > > do_image_myfs[respect_exclude_path] = "1" > > in order to honor it. Specifically, "wic" and "multiubi" have not > received this variable flag, while others have. > > Signed-off-by: Kristian Amlie <kristian.amlie@mender.io> > --- > documentation/ref-manual/ref-variables.xml | 35 +++++++++++++ > meta/classes/image.bbclass | 83 ++++++++++++++++++++++++++++-- > meta/classes/image_types.bbclass | 13 +++++ > 3 files changed, 128 insertions(+), 3 deletions(-) > > diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml > index 94558df..27f8b8e 100644 > --- a/documentation/ref-manual/ref-variables.xml > +++ b/documentation/ref-manual/ref-variables.xml > @@ -6041,6 +6041,41 @@ > </glossdef> > </glossentry> > > + <glossentry id='var-IMAGE_ROOTFS_EXCLUDE_PATH'><glossterm>IMAGE_ROOTFS_EXCLUDE_PATH</glossterm> > + <info> > + IMAGE_ROOTFS_EXCLUDE_PATH[doc] = "Specifies paths to omit from the rootfs." > + </info> > + <glossdef> > + <para role="glossdeffirst"> > +<!-- <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> --> > + Specifies paths that should be omitted from the root filesystem, separated by > + spaces. The paths must be relative to the root of the filesystem (for example > + "usr"). Single root filesystem images will not include the path in the > + filesystem, except if the path ends with a slash, then the directory will be > + included, but without any content. > + <literallayout class='monospaced'> > + # Omit /usr completely from rootfs. > + IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr</replaceable> ..." > + # Include the directory /usr in rootfs, but not its content. > + IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr/</replaceable> ..." > + </literallayout> > + </para> > + <para> > + Image creators that create multi partition images typically do not honor this > + variable, and must provide their own method to split the directory > + structure. For example, <filename>wic</filename> has > + <filename>--exclude-path</filename> and <filename>--rootfs-dir</filename>. > + </para> > + <para> > + The default for custom image creators is to ignore this variable. Those that > + need to honor it must set: > + <literallayout class='monospaced'> > + do_image_<replaceable>custom</replaceable>[respect_exclude_path] = "1" > + </literallayout> > + </para> > + </glossdef> > + </glossentry> > + > <glossentry id='var-IMAGE_ROOTFS_EXTRA_SPACE'><glossterm>IMAGE_ROOTFS_EXTRA_SPACE</glossterm> > <info> > IMAGE_ROOTFS_EXTRA_SPACE[doc] = "Defines additional free disk space created in the image in Kbytes. By default, this variable is set to '0'." > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index 85f6b9a..c53e383 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -118,7 +118,7 @@ def rootfs_variables(d): > 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', > 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS', > 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS', > - 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY'] > + 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'IMAGE_ROOTFS_EXCLUDE_PATH'] > variables.extend(rootfs_command_variables(d)) > variables.extend(variable_depends(d)) > return " ".join(variables) > @@ -496,8 +496,8 @@ python () { > d.setVar('do_image_%s' % t, '\n'.join(cmds)) > d.setVarFlag('do_image_%s' % t, 'func', '1') > d.setVarFlag('do_image_%s' % t, 'fakeroot', '1') > - d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size') > - d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks') > + d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size prepare_excluded_directories') > + d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks cleanup_excluded_directories') > d.setVarFlag('do_image_%s' % t, 'subimages', ' '.join(subimages)) > d.appendVarFlag('do_image_%s' % t, 'vardeps', ' '.join(vardeps)) > d.appendVarFlag('do_image_%s' % t, 'vardepsexclude', 'DATETIME') > @@ -506,6 +506,83 @@ python () { > bb.build.addtask('do_image_%s' % t, 'do_image_complete', after, d) > } > > +python prepare_excluded_directories() { > + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') > + if not exclude_var: > + return > + > + taskname = d.getVar("BB_CURRENTTASK") > + > + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1': > + return > + > + import shutil > + from oe.path import copyhardlinktree > + > + exclude_list = exclude_var.split() > + > + rootfs_orig = d.getVar('IMAGE_ROOTFS') > + # We need a new rootfs directory we can delete files from. Copy to > + # workdir. > + new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname)) > + > + if os.path.lexists(new_rootfs): > + shutil.rmtree(os.path.join(new_rootfs)) > + > + copyhardlinktree(rootfs_orig, new_rootfs) > + > + for orig_path in exclude_list: > + path = orig_path > + if os.path.isabs(path): > + bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path) > + > + full_path = os.path.realpath(os.path.join(new_rootfs, path)) > + > + # Disallow climbing outside of parent directory using '..', > + # because doing so could be quite disastrous (we will delete the > + # directory). > + if not full_path.startswith(new_rootfs): > + bb.fatal("'%s' points to a path outside the rootfs" % orig_path) > + > + if path.endswith(os.sep): > + # Delete content only. > + for entry in os.listdir(full_path): > + full_entry = os.path.join(full_path, entry) > + if os.path.isdir(full_entry) and not os.path.islink(full_entry): > + shutil.rmtree(full_entry) > + else: > + os.remove(full_entry) > + else: > + # Delete whole directory. > + shutil.rmtree(full_path) > + > + # Save old value for cleanup later. > + d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig) > + d.setVar('IMAGE_ROOTFS', new_rootfs) > +} > + > +python cleanup_excluded_directories() { > + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') > + if not exclude_var: > + return > + > + taskname = d.getVar("BB_CURRENTTASK") > + > + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1': > + return > + > + import shutil > + > + rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS') > + rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG') > + # This should never happen, since we should have set it to a different > + # directory in the prepare function. > + assert rootfs_dirs_excluded != rootfs_orig > + > + shutil.rmtree(rootfs_dirs_excluded) > + d.setVar('IMAGE_ROOTFS', rootfs_orig) > +} > + > # > # Compute the rootfs size > # > diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass > index 7749b00..b27dfe3 100644 > --- a/meta/classes/image_types.bbclass > +++ b/meta/classes/image_types.bbclass > @@ -51,8 +51,10 @@ ZIP_COMPRESSION_LEVEL ?= "-9" > > JFFS2_SUM_EXTRA_ARGS ?= "" > IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 ${EXTRA_IMAGECMD}" > +do_image_jffs2[respect_exclude_path] = "1" > > IMAGE_CMD_cramfs = "mkfs.cramfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cramfs ${EXTRA_IMAGECMD}" > +do_image_cramfs[respect_exclude_path] = "1" > > oe_mkext234fs () { > fstype=$1 > @@ -81,6 +83,9 @@ oe_mkext234fs () { > IMAGE_CMD_ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}" > IMAGE_CMD_ext3 = "oe_mkext234fs ext3 ${EXTRA_IMAGECMD}" > IMAGE_CMD_ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}" > +do_image_ext2[respect_exclude_path] = "1" > +do_image_ext3[respect_exclude_path] = "1" > +do_image_ext4[respect_exclude_path] = "1" > > MIN_BTRFS_SIZE ?= "16384" > IMAGE_CMD_btrfs () { > @@ -92,10 +97,14 @@ IMAGE_CMD_btrfs () { > dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs count=${size} bs=1024 > mkfs.btrfs ${EXTRA_IMAGECMD} -r ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs > } > +do_image_btrfs[respect_exclude_path] = "1" > > IMAGE_CMD_squashfs = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs ${EXTRA_IMAGECMD} -noappend" > IMAGE_CMD_squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz" > IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo" > +do_image_squashfs[respect_exclude_path] = "1" > +do_image_squashfs-xz[respect_exclude_path] = "1" > +do_image_squashfs-lzo[respect_exclude_path] = "1" > > # By default, tar from the host is used, which can be quite old. If > # you need special parameters (like --xattrs) which are only supported > @@ -109,6 +118,7 @@ IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAM > # required when extracting, but it seems prudent to use it in both cases. > IMAGE_CMD_TAR ?= "tar" > IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cvf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} ." > +do_image_tar[respect_exclude_path] = "1" > > do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append" > IMAGE_CMD_cpio () { > @@ -129,6 +139,7 @@ IMAGE_CMD_cpio () { > fi > fi > } > +do_image_cpio[respect_exclude_path] = "1" > > ELF_KERNEL ?= "${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE}" > ELF_APPEND ?= "ramdisk_size=32768 root=/dev/ram0 rw console=" > @@ -137,6 +148,7 @@ IMAGE_CMD_elf () { > test -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf && rm -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf > mkelfImage --kernel=${ELF_KERNEL} --initrd=${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.cpio.gz --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf --append='${ELF_APPEND}' ${EXTRA_IMAGECMD} > } > +do_image_elf[respect_exclude_path] = "1" > > IMAGE_TYPEDEP_elf = "cpio.gz" > > @@ -194,6 +206,7 @@ IMAGE_CMD_ubi () { > } > > IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ubifs ${MKUBIFS_ARGS}" > +do_image_ubifs[respect_exclude_path] = "1" > > EXTRA_IMAGECMD = "" > > ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v2] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-06-19 9:01 ` Kristian Amlie @ 2017-06-19 14:01 ` Kristian Amlie 2017-06-19 14:01 ` [PATCH v3] " Kristian Amlie 0 siblings, 1 reply; 41+ messages in thread From: Kristian Amlie @ 2017-06-19 14:01 UTC (permalink / raw) To: openembedded-core Turns out this needed a rebase. Here you go. -- Kristian ^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH v3] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-06-19 14:01 ` Kristian Amlie @ 2017-06-19 14:01 ` Kristian Amlie 2017-08-23 11:44 ` [PATCH v4] " Kristian Amlie 2017-08-23 11:46 ` Kristian Amlie 0 siblings, 2 replies; 41+ messages in thread From: Kristian Amlie @ 2017-06-19 14:01 UTC (permalink / raw) To: openembedded-core This is a direct followup from the earlier f6a064d969f414 commit in wic. It works more or less the same way: The variable specifies a list of directories relative to the root of the rootfs, and these directories will be excluded from the resulting rootfs image. If an entry ends with a slash, only the contents are omitted, not the directory itself. Since the intended use of the variable is to exclude certain directories from the rootfs, and then include said directories in other partitions, it is not natural for this variable to be respected for image creators that create multi partition images. Therefore the default is to ignore the variable, and image creators that create single root filesystems need to set do_image_myfs[respect_exclude_path] = "1" in order to honor it. Specifically, "wic" and "multiubi" have not received this variable flag, while others have. Signed-off-by: Kristian Amlie <kristian.amlie@mender.io> --- documentation/ref-manual/ref-variables.xml | 35 +++++++++++++ meta/classes/image.bbclass | 83 ++++++++++++++++++++++++++++-- meta/classes/image_types.bbclass | 13 +++++ 3 files changed, 128 insertions(+), 3 deletions(-) diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml index bd9e517..8456445 100644 --- a/documentation/ref-manual/ref-variables.xml +++ b/documentation/ref-manual/ref-variables.xml @@ -6041,6 +6041,41 @@ </glossdef> </glossentry> + <glossentry id='var-IMAGE_ROOTFS_EXCLUDE_PATH'><glossterm>IMAGE_ROOTFS_EXCLUDE_PATH</glossterm> + <info> + IMAGE_ROOTFS_EXCLUDE_PATH[doc] = "Specifies paths to omit from the rootfs." + </info> + <glossdef> + <para role="glossdeffirst"> +<!-- <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> --> + Specifies paths that should be omitted from the root filesystem, separated by + spaces. The paths must be relative to the root of the filesystem (for example + "usr"). Single root filesystem images will not include the path in the + filesystem, except if the path ends with a slash, then the directory will be + included, but without any content. + <literallayout class='monospaced'> + # Omit /usr completely from rootfs. + IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr</replaceable> ..." + # Include the directory /usr in rootfs, but not its content. + IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr/</replaceable> ..." + </literallayout> + </para> + <para> + Image creators that create multi partition images typically do not honor this + variable, and must provide their own method to split the directory + structure. For example, <filename>wic</filename> has + <filename>--exclude-path</filename> and <filename>--rootfs-dir</filename>. + </para> + <para> + The default for custom image creators is to ignore this variable. Those that + need to honor it must set: + <literallayout class='monospaced'> + do_image_<replaceable>custom</replaceable>[respect_exclude_path] = "1" + </literallayout> + </para> + </glossdef> + </glossentry> + <glossentry id='var-IMAGE_ROOTFS_EXTRA_SPACE'><glossterm>IMAGE_ROOTFS_EXTRA_SPACE</glossterm> <info> IMAGE_ROOTFS_EXTRA_SPACE[doc] = "Defines additional free disk space created in the image in Kbytes. By default, this variable is set to '0'." diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 9a04b43..e4c97b1 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -118,7 +118,7 @@ def rootfs_variables(d): 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS', 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS', - 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY'] + 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'IMAGE_ROOTFS_EXCLUDE_PATH'] variables.extend(rootfs_command_variables(d)) variables.extend(variable_depends(d)) return " ".join(variables) @@ -497,8 +497,8 @@ python () { d.setVarFlag(task, 'func', '1') d.setVarFlag(task, 'fakeroot', '1') - d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size') - d.appendVarFlag(task, 'postfuncs', ' create_symlinks') + d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories') + d.appendVarFlag(task, 'postfuncs', ' create_symlinks cleanup_excluded_directories') d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages)) d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps)) d.appendVarFlag(task, 'vardepsexclude', 'DATETIME') @@ -507,6 +507,83 @@ python () { bb.build.addtask(task, 'do_image_complete', after, d) } +python prepare_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + taskname = d.getVar("BB_CURRENTTASK") + + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1': + return + + import shutil + from oe.path import copyhardlinktree + + exclude_list = exclude_var.split() + + rootfs_orig = d.getVar('IMAGE_ROOTFS') + # We need a new rootfs directory we can delete files from. Copy to + # workdir. + new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname)) + + if os.path.lexists(new_rootfs): + shutil.rmtree(os.path.join(new_rootfs)) + + copyhardlinktree(rootfs_orig, new_rootfs) + + for orig_path in exclude_list: + path = orig_path + if os.path.isabs(path): + bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path) + + full_path = os.path.realpath(os.path.join(new_rootfs, path)) + + # Disallow climbing outside of parent directory using '..', + # because doing so could be quite disastrous (we will delete the + # directory). + if not full_path.startswith(new_rootfs): + bb.fatal("'%s' points to a path outside the rootfs" % orig_path) + + if path.endswith(os.sep): + # Delete content only. + for entry in os.listdir(full_path): + full_entry = os.path.join(full_path, entry) + if os.path.isdir(full_entry) and not os.path.islink(full_entry): + shutil.rmtree(full_entry) + else: + os.remove(full_entry) + else: + # Delete whole directory. + shutil.rmtree(full_path) + + # Save old value for cleanup later. + d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig) + d.setVar('IMAGE_ROOTFS', new_rootfs) +} + +python cleanup_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + taskname = d.getVar("BB_CURRENTTASK") + + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1': + return + + import shutil + + rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS') + rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG') + # This should never happen, since we should have set it to a different + # directory in the prepare function. + assert rootfs_dirs_excluded != rootfs_orig + + shutil.rmtree(rootfs_dirs_excluded) + d.setVar('IMAGE_ROOTFS', rootfs_orig) +} + # # Compute the rootfs size # diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass index 7749b00..b27dfe3 100644 --- a/meta/classes/image_types.bbclass +++ b/meta/classes/image_types.bbclass @@ -51,8 +51,10 @@ ZIP_COMPRESSION_LEVEL ?= "-9" JFFS2_SUM_EXTRA_ARGS ?= "" IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 ${EXTRA_IMAGECMD}" +do_image_jffs2[respect_exclude_path] = "1" IMAGE_CMD_cramfs = "mkfs.cramfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cramfs ${EXTRA_IMAGECMD}" +do_image_cramfs[respect_exclude_path] = "1" oe_mkext234fs () { fstype=$1 @@ -81,6 +83,9 @@ oe_mkext234fs () { IMAGE_CMD_ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}" IMAGE_CMD_ext3 = "oe_mkext234fs ext3 ${EXTRA_IMAGECMD}" IMAGE_CMD_ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}" +do_image_ext2[respect_exclude_path] = "1" +do_image_ext3[respect_exclude_path] = "1" +do_image_ext4[respect_exclude_path] = "1" MIN_BTRFS_SIZE ?= "16384" IMAGE_CMD_btrfs () { @@ -92,10 +97,14 @@ IMAGE_CMD_btrfs () { dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs count=${size} bs=1024 mkfs.btrfs ${EXTRA_IMAGECMD} -r ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs } +do_image_btrfs[respect_exclude_path] = "1" IMAGE_CMD_squashfs = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs ${EXTRA_IMAGECMD} -noappend" IMAGE_CMD_squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz" IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo" +do_image_squashfs[respect_exclude_path] = "1" +do_image_squashfs-xz[respect_exclude_path] = "1" +do_image_squashfs-lzo[respect_exclude_path] = "1" # By default, tar from the host is used, which can be quite old. If # you need special parameters (like --xattrs) which are only supported @@ -109,6 +118,7 @@ IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAM # required when extracting, but it seems prudent to use it in both cases. IMAGE_CMD_TAR ?= "tar" IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cvf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} ." +do_image_tar[respect_exclude_path] = "1" do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append" IMAGE_CMD_cpio () { @@ -129,6 +139,7 @@ IMAGE_CMD_cpio () { fi fi } +do_image_cpio[respect_exclude_path] = "1" ELF_KERNEL ?= "${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE}" ELF_APPEND ?= "ramdisk_size=32768 root=/dev/ram0 rw console=" @@ -137,6 +148,7 @@ IMAGE_CMD_elf () { test -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf && rm -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf mkelfImage --kernel=${ELF_KERNEL} --initrd=${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.cpio.gz --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf --append='${ELF_APPEND}' ${EXTRA_IMAGECMD} } +do_image_elf[respect_exclude_path] = "1" IMAGE_TYPEDEP_elf = "cpio.gz" @@ -194,6 +206,7 @@ IMAGE_CMD_ubi () { } IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ubifs ${MKUBIFS_ARGS}" +do_image_ubifs[respect_exclude_path] = "1" EXTRA_IMAGECMD = "" -- 2.7.4 ^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v4] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-06-19 14:01 ` [PATCH v3] " Kristian Amlie @ 2017-08-23 11:44 ` Kristian Amlie 2017-08-23 11:46 ` Kristian Amlie 1 sibling, 0 replies; 41+ messages in thread From: Kristian Amlie @ 2017-08-23 11:44 UTC (permalink / raw) To: openembedded-core From: Kristian Amlie <kristian.amlie@mender.io> This is a direct followup from the earlier f6a064d969f414 commit in wic. It works more or less the same way: The variable specifies a list of directories relative to the root of the rootfs, and these directories will be excluded from the resulting rootfs image. If an entry ends with a slash, only the contents are omitted, not the directory itself. Since the intended use of the variable is to exclude certain directories from the rootfs, and then include said directories in other partitions, it is not natural for this variable to be respected for image creators that create multi partition images. Therefore the default is to ignore the variable, and image creators that create single root filesystems need to set do_image_myfs[respect_exclude_path] = "1" in order to honor it. Specifically, "wic" and "multiubi" have not received this variable flag, while others have. Signed-off-by: Kristian Amlie <kristian.amlie@mender.io> --- documentation/ref-manual/ref-variables.xml | 35 +++++++++++++ meta/classes/image.bbclass | 83 +++++++++++++++++++++++++++++- meta/classes/image_types.bbclass | 13 +++++ 3 files changed, 129 insertions(+), 2 deletions(-) diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml index ff8f4e7..a4f2de5 100644 --- a/documentation/ref-manual/ref-variables.xml +++ b/documentation/ref-manual/ref-variables.xml @@ -6043,6 +6043,41 @@ </glossdef> </glossentry> + <glossentry id='var-IMAGE_ROOTFS_EXCLUDE_PATH'><glossterm>IMAGE_ROOTFS_EXCLUDE_PATH</glossterm> + <info> + IMAGE_ROOTFS_EXCLUDE_PATH[doc] = "Specifies paths to omit from the rootfs." + </info> + <glossdef> + <para role="glossdeffirst"> +<!-- <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> --> + Specifies paths that should be omitted from the root filesystem, separated by + spaces. The paths must be relative to the root of the filesystem (for example + "usr"). Single root filesystem images will not include the path in the + filesystem, except if the path ends with a slash, then the directory will be + included, but without any content. + <literallayout class='monospaced'> + # Omit /usr completely from rootfs. + IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr</replaceable> ..." + # Include the directory /usr in rootfs, but not its content. + IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr/</replaceable> ..." + </literallayout> + </para> + <para> + Image creators that create multi partition images typically do not honor this + variable, and must provide their own method to split the directory + structure. For example, <filename>wic</filename> has + <filename>--exclude-path</filename> and <filename>--rootfs-dir</filename>. + </para> + <para> + The default for custom image creators is to ignore this variable. Those that + need to honor it must set: + <literallayout class='monospaced'> + do_image_<replaceable>custom</replaceable>[respect_exclude_path] = "1" + </literallayout> + </para> + </glossdef> + </glossentry> + <glossentry id='var-IMAGE_ROOTFS_EXTRA_SPACE'><glossterm>IMAGE_ROOTFS_EXTRA_SPACE</glossterm> <info> IMAGE_ROOTFS_EXTRA_SPACE[doc] = "Defines additional free disk space created in the image in Kbytes. By default, this variable is set to '0'." diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 3639aa4..e215541 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -117,7 +117,8 @@ def rootfs_variables(d): 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS', 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS', - 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS'] + 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS', + 'IMAGE_ROOTFS_EXCLUDE_PATH'] variables.extend(rootfs_command_variables(d)) variables.extend(variable_depends(d)) return " ".join(variables) @@ -496,8 +497,9 @@ python () { d.setVarFlag(task, 'func', '1') d.setVarFlag(task, 'fakeroot', '1') - d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size') + d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories') d.prependVarFlag(task, 'postfuncs', ' create_symlinks') + d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories') d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages)) d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps)) d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE') @@ -506,6 +508,83 @@ python () { bb.build.addtask(task, 'do_image_complete', after, d) } +python prepare_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + taskname = d.getVar("BB_CURRENTTASK") + + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1': + return + + import shutil + from oe.path import copyhardlinktree + + exclude_list = exclude_var.split() + + rootfs_orig = d.getVar('IMAGE_ROOTFS') + # We need a new rootfs directory we can delete files from. Copy to + # workdir. + new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname)) + + if os.path.lexists(new_rootfs): + shutil.rmtree(os.path.join(new_rootfs)) + + copyhardlinktree(rootfs_orig, new_rootfs) + + for orig_path in exclude_list: + path = orig_path + if os.path.isabs(path): + bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path) + + full_path = os.path.realpath(os.path.join(new_rootfs, path)) + + # Disallow climbing outside of parent directory using '..', + # because doing so could be quite disastrous (we will delete the + # directory). + if not full_path.startswith(new_rootfs): + bb.fatal("'%s' points to a path outside the rootfs" % orig_path) + + if path.endswith(os.sep): + # Delete content only. + for entry in os.listdir(full_path): + full_entry = os.path.join(full_path, entry) + if os.path.isdir(full_entry) and not os.path.islink(full_entry): + shutil.rmtree(full_entry) + else: + os.remove(full_entry) + else: + # Delete whole directory. + shutil.rmtree(full_path) + + # Save old value for cleanup later. + d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig) + d.setVar('IMAGE_ROOTFS', new_rootfs) +} + +python cleanup_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + taskname = d.getVar("BB_CURRENTTASK") + + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1': + return + + import shutil + + rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS') + rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG') + # This should never happen, since we should have set it to a different + # directory in the prepare function. + assert rootfs_dirs_excluded != rootfs_orig + + shutil.rmtree(rootfs_dirs_excluded) + d.setVar('IMAGE_ROOTFS', rootfs_orig) +} + # # Compute the rootfs size # diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass index e0368c7..a063df9 100644 --- a/meta/classes/image_types.bbclass +++ b/meta/classes/image_types.bbclass @@ -62,8 +62,10 @@ ZIP_COMPRESSION_LEVEL ?= "-9" JFFS2_SUM_EXTRA_ARGS ?= "" IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 ${EXTRA_IMAGECMD}" +do_image_jffs2[respect_exclude_path] = "1" IMAGE_CMD_cramfs = "mkfs.cramfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cramfs ${EXTRA_IMAGECMD}" +do_image_cramfs[respect_exclude_path] = "1" oe_mkext234fs () { fstype=$1 @@ -92,6 +94,9 @@ oe_mkext234fs () { IMAGE_CMD_ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}" IMAGE_CMD_ext3 = "oe_mkext234fs ext3 ${EXTRA_IMAGECMD}" IMAGE_CMD_ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}" +do_image_ext2[respect_exclude_path] = "1" +do_image_ext3[respect_exclude_path] = "1" +do_image_ext4[respect_exclude_path] = "1" MIN_BTRFS_SIZE ?= "16384" IMAGE_CMD_btrfs () { @@ -103,10 +108,14 @@ IMAGE_CMD_btrfs () { dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs count=${size} bs=1024 mkfs.btrfs ${EXTRA_IMAGECMD} -r ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs } +do_image_btrfs[respect_exclude_path] = "1" IMAGE_CMD_squashfs = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs ${EXTRA_IMAGECMD} -noappend" IMAGE_CMD_squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz" IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo" +do_image_squashfs[respect_exclude_path] = "1" +do_image_squashfs-xz[respect_exclude_path] = "1" +do_image_squashfs-lzo[respect_exclude_path] = "1" # By default, tar from the host is used, which can be quite old. If # you need special parameters (like --xattrs) which are only supported @@ -121,6 +130,7 @@ IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAM IMAGE_CMD_TAR ?= "tar" # ignore return code 1 "file changed as we read it" as other tasks(e.g. do_image_wic) may be hardlinking rootfs IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} . || [ $? -eq 1 ]" +do_image_tar[respect_exclude_path] = "1" do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append" IMAGE_CMD_cpio () { @@ -141,6 +151,7 @@ IMAGE_CMD_cpio () { fi fi } +do_image_cpio[respect_exclude_path] = "1" ELF_KERNEL ?= "${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE}" ELF_APPEND ?= "ramdisk_size=32768 root=/dev/ram0 rw console=" @@ -149,6 +160,7 @@ IMAGE_CMD_elf () { test -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf && rm -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf mkelfImage --kernel=${ELF_KERNEL} --initrd=${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.cpio.gz --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf --append='${ELF_APPEND}' ${EXTRA_IMAGECMD} } +do_image_elf[respect_exclude_path] = "1" IMAGE_TYPEDEP_elf = "cpio.gz" @@ -212,6 +224,7 @@ IMAGE_CMD_ubi () { } IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ubifs ${MKUBIFS_ARGS}" +do_image_ubifs[respect_exclude_path] = "1" EXTRA_IMAGECMD = "" -- 2.7.4 ^ permalink raw reply related [flat|nested] 41+ messages in thread
* Re: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-06-19 14:01 ` [PATCH v3] " Kristian Amlie 2017-08-23 11:44 ` [PATCH v4] " Kristian Amlie @ 2017-08-23 11:46 ` Kristian Amlie 2017-08-23 11:46 ` [PATCH v4] " Kristian Amlie 1 sibling, 1 reply; 41+ messages in thread From: Kristian Amlie @ 2017-08-23 11:46 UTC (permalink / raw) To: openembedded-core Ping, third rebase now! This is related to https://bugzilla.yoctoproject.org/show_bug.cgi?id=11688. -- Kristian ^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH v4] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-08-23 11:46 ` Kristian Amlie @ 2017-08-23 11:46 ` Kristian Amlie 2017-08-23 12:39 ` Kristian Amlie 0 siblings, 1 reply; 41+ messages in thread From: Kristian Amlie @ 2017-08-23 11:46 UTC (permalink / raw) To: openembedded-core From: Kristian Amlie <kristian.amlie@mender.io> This is a direct followup from the earlier f6a064d969f414 commit in wic. It works more or less the same way: The variable specifies a list of directories relative to the root of the rootfs, and these directories will be excluded from the resulting rootfs image. If an entry ends with a slash, only the contents are omitted, not the directory itself. Since the intended use of the variable is to exclude certain directories from the rootfs, and then include said directories in other partitions, it is not natural for this variable to be respected for image creators that create multi partition images. Therefore the default is to ignore the variable, and image creators that create single root filesystems need to set do_image_myfs[respect_exclude_path] = "1" in order to honor it. Specifically, "wic" and "multiubi" have not received this variable flag, while others have. Signed-off-by: Kristian Amlie <kristian.amlie@mender.io> --- documentation/ref-manual/ref-variables.xml | 35 +++++++++++++ meta/classes/image.bbclass | 83 +++++++++++++++++++++++++++++- meta/classes/image_types.bbclass | 13 +++++ 3 files changed, 129 insertions(+), 2 deletions(-) diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml index ff8f4e7..a4f2de5 100644 --- a/documentation/ref-manual/ref-variables.xml +++ b/documentation/ref-manual/ref-variables.xml @@ -6043,6 +6043,41 @@ </glossdef> </glossentry> + <glossentry id='var-IMAGE_ROOTFS_EXCLUDE_PATH'><glossterm>IMAGE_ROOTFS_EXCLUDE_PATH</glossterm> + <info> + IMAGE_ROOTFS_EXCLUDE_PATH[doc] = "Specifies paths to omit from the rootfs." + </info> + <glossdef> + <para role="glossdeffirst"> +<!-- <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> --> + Specifies paths that should be omitted from the root filesystem, separated by + spaces. The paths must be relative to the root of the filesystem (for example + "usr"). Single root filesystem images will not include the path in the + filesystem, except if the path ends with a slash, then the directory will be + included, but without any content. + <literallayout class='monospaced'> + # Omit /usr completely from rootfs. + IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr</replaceable> ..." + # Include the directory /usr in rootfs, but not its content. + IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr/</replaceable> ..." + </literallayout> + </para> + <para> + Image creators that create multi partition images typically do not honor this + variable, and must provide their own method to split the directory + structure. For example, <filename>wic</filename> has + <filename>--exclude-path</filename> and <filename>--rootfs-dir</filename>. + </para> + <para> + The default for custom image creators is to ignore this variable. Those that + need to honor it must set: + <literallayout class='monospaced'> + do_image_<replaceable>custom</replaceable>[respect_exclude_path] = "1" + </literallayout> + </para> + </glossdef> + </glossentry> + <glossentry id='var-IMAGE_ROOTFS_EXTRA_SPACE'><glossterm>IMAGE_ROOTFS_EXTRA_SPACE</glossterm> <info> IMAGE_ROOTFS_EXTRA_SPACE[doc] = "Defines additional free disk space created in the image in Kbytes. By default, this variable is set to '0'." diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 3639aa4..e215541 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -117,7 +117,8 @@ def rootfs_variables(d): 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS', 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS', - 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS'] + 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS', + 'IMAGE_ROOTFS_EXCLUDE_PATH'] variables.extend(rootfs_command_variables(d)) variables.extend(variable_depends(d)) return " ".join(variables) @@ -496,8 +497,9 @@ python () { d.setVarFlag(task, 'func', '1') d.setVarFlag(task, 'fakeroot', '1') - d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size') + d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories') d.prependVarFlag(task, 'postfuncs', ' create_symlinks') + d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories') d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages)) d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps)) d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE') @@ -506,6 +508,83 @@ python () { bb.build.addtask(task, 'do_image_complete', after, d) } +python prepare_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + taskname = d.getVar("BB_CURRENTTASK") + + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1': + return + + import shutil + from oe.path import copyhardlinktree + + exclude_list = exclude_var.split() + + rootfs_orig = d.getVar('IMAGE_ROOTFS') + # We need a new rootfs directory we can delete files from. Copy to + # workdir. + new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname)) + + if os.path.lexists(new_rootfs): + shutil.rmtree(os.path.join(new_rootfs)) + + copyhardlinktree(rootfs_orig, new_rootfs) + + for orig_path in exclude_list: + path = orig_path + if os.path.isabs(path): + bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path) + + full_path = os.path.realpath(os.path.join(new_rootfs, path)) + + # Disallow climbing outside of parent directory using '..', + # because doing so could be quite disastrous (we will delete the + # directory). + if not full_path.startswith(new_rootfs): + bb.fatal("'%s' points to a path outside the rootfs" % orig_path) + + if path.endswith(os.sep): + # Delete content only. + for entry in os.listdir(full_path): + full_entry = os.path.join(full_path, entry) + if os.path.isdir(full_entry) and not os.path.islink(full_entry): + shutil.rmtree(full_entry) + else: + os.remove(full_entry) + else: + # Delete whole directory. + shutil.rmtree(full_path) + + # Save old value for cleanup later. + d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig) + d.setVar('IMAGE_ROOTFS', new_rootfs) +} + +python cleanup_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + taskname = d.getVar("BB_CURRENTTASK") + + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1': + return + + import shutil + + rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS') + rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG') + # This should never happen, since we should have set it to a different + # directory in the prepare function. + assert rootfs_dirs_excluded != rootfs_orig + + shutil.rmtree(rootfs_dirs_excluded) + d.setVar('IMAGE_ROOTFS', rootfs_orig) +} + # # Compute the rootfs size # diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass index e0368c7..a063df9 100644 --- a/meta/classes/image_types.bbclass +++ b/meta/classes/image_types.bbclass @@ -62,8 +62,10 @@ ZIP_COMPRESSION_LEVEL ?= "-9" JFFS2_SUM_EXTRA_ARGS ?= "" IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 ${EXTRA_IMAGECMD}" +do_image_jffs2[respect_exclude_path] = "1" IMAGE_CMD_cramfs = "mkfs.cramfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cramfs ${EXTRA_IMAGECMD}" +do_image_cramfs[respect_exclude_path] = "1" oe_mkext234fs () { fstype=$1 @@ -92,6 +94,9 @@ oe_mkext234fs () { IMAGE_CMD_ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}" IMAGE_CMD_ext3 = "oe_mkext234fs ext3 ${EXTRA_IMAGECMD}" IMAGE_CMD_ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}" +do_image_ext2[respect_exclude_path] = "1" +do_image_ext3[respect_exclude_path] = "1" +do_image_ext4[respect_exclude_path] = "1" MIN_BTRFS_SIZE ?= "16384" IMAGE_CMD_btrfs () { @@ -103,10 +108,14 @@ IMAGE_CMD_btrfs () { dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs count=${size} bs=1024 mkfs.btrfs ${EXTRA_IMAGECMD} -r ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs } +do_image_btrfs[respect_exclude_path] = "1" IMAGE_CMD_squashfs = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs ${EXTRA_IMAGECMD} -noappend" IMAGE_CMD_squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz" IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo" +do_image_squashfs[respect_exclude_path] = "1" +do_image_squashfs-xz[respect_exclude_path] = "1" +do_image_squashfs-lzo[respect_exclude_path] = "1" # By default, tar from the host is used, which can be quite old. If # you need special parameters (like --xattrs) which are only supported @@ -121,6 +130,7 @@ IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAM IMAGE_CMD_TAR ?= "tar" # ignore return code 1 "file changed as we read it" as other tasks(e.g. do_image_wic) may be hardlinking rootfs IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} . || [ $? -eq 1 ]" +do_image_tar[respect_exclude_path] = "1" do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append" IMAGE_CMD_cpio () { @@ -141,6 +151,7 @@ IMAGE_CMD_cpio () { fi fi } +do_image_cpio[respect_exclude_path] = "1" ELF_KERNEL ?= "${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE}" ELF_APPEND ?= "ramdisk_size=32768 root=/dev/ram0 rw console=" @@ -149,6 +160,7 @@ IMAGE_CMD_elf () { test -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf && rm -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf mkelfImage --kernel=${ELF_KERNEL} --initrd=${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.cpio.gz --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf --append='${ELF_APPEND}' ${EXTRA_IMAGECMD} } +do_image_elf[respect_exclude_path] = "1" IMAGE_TYPEDEP_elf = "cpio.gz" @@ -212,6 +224,7 @@ IMAGE_CMD_ubi () { } IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ubifs ${MKUBIFS_ARGS}" +do_image_ubifs[respect_exclude_path] = "1" EXTRA_IMAGECMD = "" -- 2.7.4 ^ permalink raw reply related [flat|nested] 41+ messages in thread
* Re: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-08-23 11:46 ` [PATCH v4] " Kristian Amlie @ 2017-08-23 12:39 ` Kristian Amlie 2017-08-23 12:39 ` [PATCH v5] " Kristian Amlie 2017-08-23 12:46 ` Richard Purdie 0 siblings, 2 replies; 41+ messages in thread From: Kristian Amlie @ 2017-08-23 12:39 UTC (permalink / raw) To: openembedded-core I don't understand why I'm getting patch failures. This is rebased to cutting edge master. Trying one more time... -- Kristian ^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH v5] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-08-23 12:39 ` Kristian Amlie @ 2017-08-23 12:39 ` Kristian Amlie 2017-08-23 12:47 ` Richard Purdie 2017-08-23 12:46 ` Richard Purdie 1 sibling, 1 reply; 41+ messages in thread From: Kristian Amlie @ 2017-08-23 12:39 UTC (permalink / raw) To: openembedded-core From: Kristian Amlie <kristian.amlie@mender.io> This is a direct followup from the earlier f6a064d969f414 commit in wic. It works more or less the same way: The variable specifies a list of directories relative to the root of the rootfs, and these directories will be excluded from the resulting rootfs image. If an entry ends with a slash, only the contents are omitted, not the directory itself. Since the intended use of the variable is to exclude certain directories from the rootfs, and then include said directories in other partitions, it is not natural for this variable to be respected for image creators that create multi partition images. Therefore the default is to ignore the variable, and image creators that create single root filesystems need to set do_image_myfs[respect_exclude_path] = "1" in order to honor it. Specifically, "wic" and "multiubi" have not received this variable flag, while others have. Signed-off-by: Kristian Amlie <kristian.amlie@mender.io> --- documentation/ref-manual/ref-variables.xml | 35 +++++++++++++ meta/classes/image.bbclass | 83 +++++++++++++++++++++++++++++- meta/classes/image_types.bbclass | 13 +++++ 3 files changed, 129 insertions(+), 2 deletions(-) diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml index ff8f4e7..a4f2de5 100644 --- a/documentation/ref-manual/ref-variables.xml +++ b/documentation/ref-manual/ref-variables.xml @@ -6043,6 +6043,41 @@ </glossdef> </glossentry> + <glossentry id='var-IMAGE_ROOTFS_EXCLUDE_PATH'><glossterm>IMAGE_ROOTFS_EXCLUDE_PATH</glossterm> + <info> + IMAGE_ROOTFS_EXCLUDE_PATH[doc] = "Specifies paths to omit from the rootfs." + </info> + <glossdef> + <para role="glossdeffirst"> +<!-- <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> --> + Specifies paths that should be omitted from the root filesystem, separated by + spaces. The paths must be relative to the root of the filesystem (for example + "usr"). Single root filesystem images will not include the path in the + filesystem, except if the path ends with a slash, then the directory will be + included, but without any content. + <literallayout class='monospaced'> + # Omit /usr completely from rootfs. + IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr</replaceable> ..." + # Include the directory /usr in rootfs, but not its content. + IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr/</replaceable> ..." + </literallayout> + </para> + <para> + Image creators that create multi partition images typically do not honor this + variable, and must provide their own method to split the directory + structure. For example, <filename>wic</filename> has + <filename>--exclude-path</filename> and <filename>--rootfs-dir</filename>. + </para> + <para> + The default for custom image creators is to ignore this variable. Those that + need to honor it must set: + <literallayout class='monospaced'> + do_image_<replaceable>custom</replaceable>[respect_exclude_path] = "1" + </literallayout> + </para> + </glossdef> + </glossentry> + <glossentry id='var-IMAGE_ROOTFS_EXTRA_SPACE'><glossterm>IMAGE_ROOTFS_EXTRA_SPACE</glossterm> <info> IMAGE_ROOTFS_EXTRA_SPACE[doc] = "Defines additional free disk space created in the image in Kbytes. By default, this variable is set to '0'." diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 3639aa4..e215541 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -117,7 +117,8 @@ def rootfs_variables(d): 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS', 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS', - 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS'] + 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS', + 'IMAGE_ROOTFS_EXCLUDE_PATH'] variables.extend(rootfs_command_variables(d)) variables.extend(variable_depends(d)) return " ".join(variables) @@ -496,8 +497,9 @@ python () { d.setVarFlag(task, 'func', '1') d.setVarFlag(task, 'fakeroot', '1') - d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size') + d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories') d.prependVarFlag(task, 'postfuncs', ' create_symlinks') + d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories') d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages)) d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps)) d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE') @@ -506,6 +508,83 @@ python () { bb.build.addtask(task, 'do_image_complete', after, d) } +python prepare_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + taskname = d.getVar("BB_CURRENTTASK") + + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1': + return + + import shutil + from oe.path import copyhardlinktree + + exclude_list = exclude_var.split() + + rootfs_orig = d.getVar('IMAGE_ROOTFS') + # We need a new rootfs directory we can delete files from. Copy to + # workdir. + new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname)) + + if os.path.lexists(new_rootfs): + shutil.rmtree(os.path.join(new_rootfs)) + + copyhardlinktree(rootfs_orig, new_rootfs) + + for orig_path in exclude_list: + path = orig_path + if os.path.isabs(path): + bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path) + + full_path = os.path.realpath(os.path.join(new_rootfs, path)) + + # Disallow climbing outside of parent directory using '..', + # because doing so could be quite disastrous (we will delete the + # directory). + if not full_path.startswith(new_rootfs): + bb.fatal("'%s' points to a path outside the rootfs" % orig_path) + + if path.endswith(os.sep): + # Delete content only. + for entry in os.listdir(full_path): + full_entry = os.path.join(full_path, entry) + if os.path.isdir(full_entry) and not os.path.islink(full_entry): + shutil.rmtree(full_entry) + else: + os.remove(full_entry) + else: + # Delete whole directory. + shutil.rmtree(full_path) + + # Save old value for cleanup later. + d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig) + d.setVar('IMAGE_ROOTFS', new_rootfs) +} + +python cleanup_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + taskname = d.getVar("BB_CURRENTTASK") + + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1': + return + + import shutil + + rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS') + rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG') + # This should never happen, since we should have set it to a different + # directory in the prepare function. + assert rootfs_dirs_excluded != rootfs_orig + + shutil.rmtree(rootfs_dirs_excluded) + d.setVar('IMAGE_ROOTFS', rootfs_orig) +} + # # Compute the rootfs size # diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass index e0368c7..a063df9 100644 --- a/meta/classes/image_types.bbclass +++ b/meta/classes/image_types.bbclass @@ -62,8 +62,10 @@ ZIP_COMPRESSION_LEVEL ?= "-9" JFFS2_SUM_EXTRA_ARGS ?= "" IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 ${EXTRA_IMAGECMD}" +do_image_jffs2[respect_exclude_path] = "1" IMAGE_CMD_cramfs = "mkfs.cramfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cramfs ${EXTRA_IMAGECMD}" +do_image_cramfs[respect_exclude_path] = "1" oe_mkext234fs () { fstype=$1 @@ -92,6 +94,9 @@ oe_mkext234fs () { IMAGE_CMD_ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}" IMAGE_CMD_ext3 = "oe_mkext234fs ext3 ${EXTRA_IMAGECMD}" IMAGE_CMD_ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}" +do_image_ext2[respect_exclude_path] = "1" +do_image_ext3[respect_exclude_path] = "1" +do_image_ext4[respect_exclude_path] = "1" MIN_BTRFS_SIZE ?= "16384" IMAGE_CMD_btrfs () { @@ -103,10 +108,14 @@ IMAGE_CMD_btrfs () { dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs count=${size} bs=1024 mkfs.btrfs ${EXTRA_IMAGECMD} -r ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs } +do_image_btrfs[respect_exclude_path] = "1" IMAGE_CMD_squashfs = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs ${EXTRA_IMAGECMD} -noappend" IMAGE_CMD_squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz" IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo" +do_image_squashfs[respect_exclude_path] = "1" +do_image_squashfs-xz[respect_exclude_path] = "1" +do_image_squashfs-lzo[respect_exclude_path] = "1" # By default, tar from the host is used, which can be quite old. If # you need special parameters (like --xattrs) which are only supported @@ -121,6 +130,7 @@ IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAM IMAGE_CMD_TAR ?= "tar" # ignore return code 1 "file changed as we read it" as other tasks(e.g. do_image_wic) may be hardlinking rootfs IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} . || [ $? -eq 1 ]" +do_image_tar[respect_exclude_path] = "1" do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append" IMAGE_CMD_cpio () { @@ -141,6 +151,7 @@ IMAGE_CMD_cpio () { fi fi } +do_image_cpio[respect_exclude_path] = "1" ELF_KERNEL ?= "${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE}" ELF_APPEND ?= "ramdisk_size=32768 root=/dev/ram0 rw console=" @@ -149,6 +160,7 @@ IMAGE_CMD_elf () { test -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf && rm -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf mkelfImage --kernel=${ELF_KERNEL} --initrd=${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.cpio.gz --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf --append='${ELF_APPEND}' ${EXTRA_IMAGECMD} } +do_image_elf[respect_exclude_path] = "1" IMAGE_TYPEDEP_elf = "cpio.gz" @@ -212,6 +224,7 @@ IMAGE_CMD_ubi () { } IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ubifs ${MKUBIFS_ARGS}" +do_image_ubifs[respect_exclude_path] = "1" EXTRA_IMAGECMD = "" -- 2.7.4 ^ permalink raw reply related [flat|nested] 41+ messages in thread
* Re: [PATCH v5] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-08-23 12:39 ` [PATCH v5] " Kristian Amlie @ 2017-08-23 12:47 ` Richard Purdie 2017-08-23 13:19 ` Kristian Amlie 0 siblings, 1 reply; 41+ messages in thread From: Richard Purdie @ 2017-08-23 12:47 UTC (permalink / raw) To: Kristian Amlie, openembedded-core On Wed, 2017-08-23 at 14:39 +0200, Kristian Amlie wrote: > From: Kristian Amlie <kristian.amlie@mender.io> > > This is a direct followup from the earlier f6a064d969f414 commit in > wic. It works more or less the same way: The variable specifies a > list > of directories relative to the root of the rootfs, and these > directories will be excluded from the resulting rootfs image. If an > entry ends with a slash, only the contents are omitted, not the > directory itself. > > Since the intended use of the variable is to exclude certain > directories from the rootfs, and then include said directories in > other partitions, it is not natural for this variable to be respected > for image creators that create multi partition images. Therefore the > default is to ignore the variable, and image creators that create > single root filesystems need to set > > do_image_myfs[respect_exclude_path] = "1" > > in order to honor it. Specifically, "wic" and "multiubi" have not > received this variable flag, while others have. I have to wonder if this is implemented the correct way around. Would it not be better to mark multiubi and wic as not wanting this feature, rather than requiring every other image creation method (which may even be in other layers) to set this? Cheers, Richard ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v5] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-08-23 12:47 ` Richard Purdie @ 2017-08-23 13:19 ` Kristian Amlie 2017-08-28 15:47 ` Kristian Amlie 0 siblings, 1 reply; 41+ messages in thread From: Kristian Amlie @ 2017-08-23 13:19 UTC (permalink / raw) To: Richard Purdie, openembedded-core On 23/08/17 14:47, Richard Purdie wrote: > On Wed, 2017-08-23 at 14:39 +0200, Kristian Amlie wrote: >> From: Kristian Amlie <kristian.amlie@mender.io> >> >> This is a direct followup from the earlier f6a064d969f414 commit in >> wic. It works more or less the same way: The variable specifies a >> list >> of directories relative to the root of the rootfs, and these >> directories will be excluded from the resulting rootfs image. If an >> entry ends with a slash, only the contents are omitted, not the >> directory itself. >> >> Since the intended use of the variable is to exclude certain >> directories from the rootfs, and then include said directories in >> other partitions, it is not natural for this variable to be respected >> for image creators that create multi partition images. Therefore the >> default is to ignore the variable, and image creators that create >> single root filesystems need to set >> >> do_image_myfs[respect_exclude_path] = "1" >> >> in order to honor it. Specifically, "wic" and "multiubi" have not >> received this variable flag, while others have. > > I have to wonder if this is implemented the correct way around. Would > it not be better to mark multiubi and wic as not wanting this feature, > rather than requiring every other image creation method (which may even > be in other layers) to set this? The idea was that people in downstream repositories (other layers) are more likely to have their own complete image creators than to have filesystem creators. So rather than requiring all of the downstream repositories to opt out of the feature, we opt in in the filesystem creators which are mostly contained in OE. Or so was the thinking. Not sure if it's a good reason though. I can invert the behavior if you think it's better! -- Kristian ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v5] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-08-23 13:19 ` Kristian Amlie @ 2017-08-28 15:47 ` Kristian Amlie 2017-08-28 15:47 ` [PATCH v6] " Kristian Amlie 0 siblings, 1 reply; 41+ messages in thread From: Kristian Amlie @ 2017-08-28 15:47 UTC (permalink / raw) To: Richard Purdie, openembedded-core On 23/08/17 15:19, Kristian Amlie wrote: > On 23/08/17 14:47, Richard Purdie wrote: >> On Wed, 2017-08-23 at 14:39 +0200, Kristian Amlie wrote: >>> From: Kristian Amlie <kristian.amlie@mender.io> >>> >>> This is a direct followup from the earlier f6a064d969f414 commit in >>> wic. It works more or less the same way: The variable specifies a >>> list >>> of directories relative to the root of the rootfs, and these >>> directories will be excluded from the resulting rootfs image. If an >>> entry ends with a slash, only the contents are omitted, not the >>> directory itself. >>> >>> Since the intended use of the variable is to exclude certain >>> directories from the rootfs, and then include said directories in >>> other partitions, it is not natural for this variable to be respected >>> for image creators that create multi partition images. Therefore the >>> default is to ignore the variable, and image creators that create >>> single root filesystems need to set >>> >>> do_image_myfs[respect_exclude_path] = "1" >>> >>> in order to honor it. Specifically, "wic" and "multiubi" have not >>> received this variable flag, while others have. >> >> I have to wonder if this is implemented the correct way around. Would >> it not be better to mark multiubi and wic as not wanting this feature, >> rather than requiring every other image creation method (which may even >> be in other layers) to set this? > > The idea was that people in downstream repositories (other layers) are > more likely to have their own complete image creators than to have > filesystem creators. So rather than requiring all of the downstream > repositories to opt out of the feature, we opt in in the filesystem > creators which are mostly contained in OE. > > Or so was the thinking. Not sure if it's a good reason though. I can > invert the behavior if you think it's better! Since I heard nothing more I assume that it's better inverted, and I kind of agree as well, the patch is much smaller. So here follows the next patch, the changes from the last are: * Inverted default of the respect_exclude_path flag, now on by default * Added a debug message when it's turned off, to lessen confusion. * Documentation is now stripped out as it should be, I will send it as a separate patch as soon as we've concluded on this patch. -- Kristian ^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH v6] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-08-28 15:47 ` Kristian Amlie @ 2017-08-28 15:47 ` Kristian Amlie 2017-09-18 6:45 ` Kristian Amlie 0 siblings, 1 reply; 41+ messages in thread From: Kristian Amlie @ 2017-08-28 15:47 UTC (permalink / raw) To: Richard Purdie, openembedded-core This is a direct followup from the earlier 6602392db3d39 commit in wic. It works more or less the same way: The variable specifies a list of directories relative to the root of the rootfs, and these directories will be excluded from the resulting rootfs image. If an entry ends with a slash, only the contents are omitted, not the directory itself. Since the intended use of the variable is to exclude certain directories from the rootfs, and then include said directories in other partitions, it is not natural for this variable to be respected for image creators that create multi partition images. These can turn the feature off locally by defining: do_image_myfs[respect_exclude_path] = "0" Specifically, "wic" and "multiubi" come with the feature disabled. Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech> --- meta/classes/image.bbclass | 84 +++++++++++++++++++++++++++++++++++- meta/classes/image_types.bbclass | 1 + meta/classes/image_types_wic.bbclass | 1 + 3 files changed, 84 insertions(+), 2 deletions(-) diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 3639aa4..9cefff1 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -117,7 +117,8 @@ def rootfs_variables(d): 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS', 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS', - 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS'] + 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS', + 'IMAGE_ROOTFS_EXCLUDE_PATH'] variables.extend(rootfs_command_variables(d)) variables.extend(variable_depends(d)) return " ".join(variables) @@ -496,8 +497,9 @@ python () { d.setVarFlag(task, 'func', '1') d.setVarFlag(task, 'fakeroot', '1') - d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size') + d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories') d.prependVarFlag(task, 'postfuncs', ' create_symlinks') + d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories') d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages)) d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps)) d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE') @@ -506,6 +508,84 @@ python () { bb.build.addtask(task, 'do_image_complete', after, d) } +python prepare_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + taskname = d.getVar("BB_CURRENTTASK") + + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0': + bb.debug(1, "'IMAGE_ROOTFS_EXCLUDE_PATH' is set but 'respect_exclude_path' variable flag is 0 for this image type, so ignoring it") + return + + import shutil + from oe.path import copyhardlinktree + + exclude_list = exclude_var.split() + + rootfs_orig = d.getVar('IMAGE_ROOTFS') + # We need a new rootfs directory we can delete files from. Copy to + # workdir. + new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname)) + + if os.path.lexists(new_rootfs): + shutil.rmtree(os.path.join(new_rootfs)) + + copyhardlinktree(rootfs_orig, new_rootfs) + + for orig_path in exclude_list: + path = orig_path + if os.path.isabs(path): + bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path) + + full_path = os.path.realpath(os.path.join(new_rootfs, path)) + + # Disallow climbing outside of parent directory using '..', + # because doing so could be quite disastrous (we will delete the + # directory). + if not full_path.startswith(new_rootfs): + bb.fatal("'%s' points to a path outside the rootfs" % orig_path) + + if path.endswith(os.sep): + # Delete content only. + for entry in os.listdir(full_path): + full_entry = os.path.join(full_path, entry) + if os.path.isdir(full_entry) and not os.path.islink(full_entry): + shutil.rmtree(full_entry) + else: + os.remove(full_entry) + else: + # Delete whole directory. + shutil.rmtree(full_path) + + # Save old value for cleanup later. + d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig) + d.setVar('IMAGE_ROOTFS', new_rootfs) +} + +python cleanup_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + taskname = d.getVar("BB_CURRENTTASK") + + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0': + return + + import shutil + + rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS') + rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG') + # This should never happen, since we should have set it to a different + # directory in the prepare function. + assert rootfs_dirs_excluded != rootfs_orig + + shutil.rmtree(rootfs_dirs_excluded) + d.setVar('IMAGE_ROOTFS', rootfs_orig) +} + # # Compute the rootfs size # diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass index e0368c7..3b9250b 100644 --- a/meta/classes/image_types.bbclass +++ b/meta/classes/image_types.bbclass @@ -206,6 +206,7 @@ IMAGE_CMD_multiubi () { multiubi_mkfs "${mkubifs_args}" "${ubinize_args}" "${name}" done } +do_image_multiubi[respect_exclude_path] = "0" IMAGE_CMD_ubi () { multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}" diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass index b825b47..a34a7c6 100644 --- a/meta/classes/image_types_wic.bbclass +++ b/meta/classes/image_types_wic.bbclass @@ -34,6 +34,7 @@ IMAGE_CMD_wic () { rm -rf "$out/" } IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES" +do_image_wic[respect_exclude_path] = "0" # Rebuild when the wks file or vars in WICVARS change USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}" -- 2.7.4 ^ permalink raw reply related [flat|nested] 41+ messages in thread
* Re: [PATCH v6] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-08-28 15:47 ` [PATCH v6] " Kristian Amlie @ 2017-09-18 6:45 ` Kristian Amlie 2017-10-13 9:08 ` Kristian Amlie 0 siblings, 1 reply; 41+ messages in thread From: Kristian Amlie @ 2017-09-18 6:45 UTC (permalink / raw) To: openembedded-core Ping. Anything missing here? -- Kristian On 28/08/17 17:47, Kristian Amlie wrote: > This is a direct followup from the earlier 6602392db3d39 commit in > wic. It works more or less the same way: The variable specifies a list > of directories relative to the root of the rootfs, and these > directories will be excluded from the resulting rootfs image. If an > entry ends with a slash, only the contents are omitted, not the > directory itself. > > Since the intended use of the variable is to exclude certain > directories from the rootfs, and then include said directories in > other partitions, it is not natural for this variable to be respected > for image creators that create multi partition images. These can turn > the feature off locally by defining: > > do_image_myfs[respect_exclude_path] = "0" > > Specifically, "wic" and "multiubi" come with the feature disabled. > > Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech> > --- > meta/classes/image.bbclass | 84 +++++++++++++++++++++++++++++++++++- > meta/classes/image_types.bbclass | 1 + > meta/classes/image_types_wic.bbclass | 1 + > 3 files changed, 84 insertions(+), 2 deletions(-) > > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index 3639aa4..9cefff1 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -117,7 +117,8 @@ def rootfs_variables(d): > 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', > 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS', > 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS', > - 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS'] > + 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS', > + 'IMAGE_ROOTFS_EXCLUDE_PATH'] > variables.extend(rootfs_command_variables(d)) > variables.extend(variable_depends(d)) > return " ".join(variables) > @@ -496,8 +497,9 @@ python () { > d.setVarFlag(task, 'func', '1') > d.setVarFlag(task, 'fakeroot', '1') > > - d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size') > + d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories') > d.prependVarFlag(task, 'postfuncs', ' create_symlinks') > + d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories') > d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages)) > d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps)) > d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE') > @@ -506,6 +508,84 @@ python () { > bb.build.addtask(task, 'do_image_complete', after, d) > } > > +python prepare_excluded_directories() { > + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') > + if not exclude_var: > + return > + > + taskname = d.getVar("BB_CURRENTTASK") > + > + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0': > + bb.debug(1, "'IMAGE_ROOTFS_EXCLUDE_PATH' is set but 'respect_exclude_path' variable flag is 0 for this image type, so ignoring it") > + return > + > + import shutil > + from oe.path import copyhardlinktree > + > + exclude_list = exclude_var.split() > + > + rootfs_orig = d.getVar('IMAGE_ROOTFS') > + # We need a new rootfs directory we can delete files from. Copy to > + # workdir. > + new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname)) > + > + if os.path.lexists(new_rootfs): > + shutil.rmtree(os.path.join(new_rootfs)) > + > + copyhardlinktree(rootfs_orig, new_rootfs) > + > + for orig_path in exclude_list: > + path = orig_path > + if os.path.isabs(path): > + bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path) > + > + full_path = os.path.realpath(os.path.join(new_rootfs, path)) > + > + # Disallow climbing outside of parent directory using '..', > + # because doing so could be quite disastrous (we will delete the > + # directory). > + if not full_path.startswith(new_rootfs): > + bb.fatal("'%s' points to a path outside the rootfs" % orig_path) > + > + if path.endswith(os.sep): > + # Delete content only. > + for entry in os.listdir(full_path): > + full_entry = os.path.join(full_path, entry) > + if os.path.isdir(full_entry) and not os.path.islink(full_entry): > + shutil.rmtree(full_entry) > + else: > + os.remove(full_entry) > + else: > + # Delete whole directory. > + shutil.rmtree(full_path) > + > + # Save old value for cleanup later. > + d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig) > + d.setVar('IMAGE_ROOTFS', new_rootfs) > +} > + > +python cleanup_excluded_directories() { > + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') > + if not exclude_var: > + return > + > + taskname = d.getVar("BB_CURRENTTASK") > + > + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0': > + return > + > + import shutil > + > + rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS') > + rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG') > + # This should never happen, since we should have set it to a different > + # directory in the prepare function. > + assert rootfs_dirs_excluded != rootfs_orig > + > + shutil.rmtree(rootfs_dirs_excluded) > + d.setVar('IMAGE_ROOTFS', rootfs_orig) > +} > + > # > # Compute the rootfs size > # > diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass > index e0368c7..3b9250b 100644 > --- a/meta/classes/image_types.bbclass > +++ b/meta/classes/image_types.bbclass > @@ -206,6 +206,7 @@ IMAGE_CMD_multiubi () { > multiubi_mkfs "${mkubifs_args}" "${ubinize_args}" "${name}" > done > } > +do_image_multiubi[respect_exclude_path] = "0" > > IMAGE_CMD_ubi () { > multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}" > diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass > index b825b47..a34a7c6 100644 > --- a/meta/classes/image_types_wic.bbclass > +++ b/meta/classes/image_types_wic.bbclass > @@ -34,6 +34,7 @@ IMAGE_CMD_wic () { > rm -rf "$out/" > } > IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES" > +do_image_wic[respect_exclude_path] = "0" > > # Rebuild when the wks file or vars in WICVARS change > USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}" > ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v6] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-09-18 6:45 ` Kristian Amlie @ 2017-10-13 9:08 ` Kristian Amlie 2017-10-13 9:08 ` [PATCH v7] " Kristian Amlie 2017-10-13 10:37 ` [PATCH v6] " Alexander Kanavin 0 siblings, 2 replies; 41+ messages in thread From: Kristian Amlie @ 2017-10-13 9:08 UTC (permalink / raw) To: openembedded-core Rebased again. Can we get this in please? It has been reviewed and rebased several times already, and has lingered on the list for a long time. Regards Kristian Amlie ^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH v7] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-10-13 9:08 ` Kristian Amlie @ 2017-10-13 9:08 ` Kristian Amlie 2017-10-13 10:37 ` [PATCH v6] " Alexander Kanavin 1 sibling, 0 replies; 41+ messages in thread From: Kristian Amlie @ 2017-10-13 9:08 UTC (permalink / raw) To: openembedded-core This is a direct followup from the earlier 6602392db3d39 commit in wic. It works more or less the same way: The variable specifies a list of directories relative to the root of the rootfs, and these directories will be excluded from the resulting rootfs image. If an entry ends with a slash, only the contents are omitted, not the directory itself. Since the intended use of the variable is to exclude certain directories from the rootfs, and then include said directories in other partitions, it is not natural for this variable to be respected for image creators that create multi partition images. These can turn the feature off locally by defining: do_image_myfs[respect_exclude_path] = "0" Specifically, "wic" and "multiubi" come with the feature disabled. Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech> --- meta/classes/image.bbclass | 84 +++++++++++++++++++++++++++++++++++- meta/classes/image_types.bbclass | 1 + meta/classes/image_types_wic.bbclass | 1 + 3 files changed, 84 insertions(+), 2 deletions(-) diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 0d14250..2adfbe4 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -117,7 +117,8 @@ def rootfs_variables(d): 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS', 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS', - 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS'] + 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS', + 'IMAGE_ROOTFS_EXCLUDE_PATH'] variables.extend(rootfs_command_variables(d)) variables.extend(variable_depends(d)) return " ".join(variables) @@ -499,8 +500,9 @@ python () { d.setVarFlag(task, 'func', '1') d.setVarFlag(task, 'fakeroot', '1') - d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size') + d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories') d.prependVarFlag(task, 'postfuncs', ' create_symlinks') + d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories') d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages)) d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps)) d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE') @@ -509,6 +511,84 @@ python () { bb.build.addtask(task, 'do_image_complete', after, d) } +python prepare_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + taskname = d.getVar("BB_CURRENTTASK") + + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0': + bb.debug(1, "'IMAGE_ROOTFS_EXCLUDE_PATH' is set but 'respect_exclude_path' variable flag is 0 for this image type, so ignoring it") + return + + import shutil + from oe.path import copyhardlinktree + + exclude_list = exclude_var.split() + + rootfs_orig = d.getVar('IMAGE_ROOTFS') + # We need a new rootfs directory we can delete files from. Copy to + # workdir. + new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname)) + + if os.path.lexists(new_rootfs): + shutil.rmtree(os.path.join(new_rootfs)) + + copyhardlinktree(rootfs_orig, new_rootfs) + + for orig_path in exclude_list: + path = orig_path + if os.path.isabs(path): + bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path) + + full_path = os.path.realpath(os.path.join(new_rootfs, path)) + + # Disallow climbing outside of parent directory using '..', + # because doing so could be quite disastrous (we will delete the + # directory). + if not full_path.startswith(new_rootfs): + bb.fatal("'%s' points to a path outside the rootfs" % orig_path) + + if path.endswith(os.sep): + # Delete content only. + for entry in os.listdir(full_path): + full_entry = os.path.join(full_path, entry) + if os.path.isdir(full_entry) and not os.path.islink(full_entry): + shutil.rmtree(full_entry) + else: + os.remove(full_entry) + else: + # Delete whole directory. + shutil.rmtree(full_path) + + # Save old value for cleanup later. + d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig) + d.setVar('IMAGE_ROOTFS', new_rootfs) +} + +python cleanup_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + taskname = d.getVar("BB_CURRENTTASK") + + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0': + return + + import shutil + + rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS') + rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG') + # This should never happen, since we should have set it to a different + # directory in the prepare function. + assert rootfs_dirs_excluded != rootfs_orig + + shutil.rmtree(rootfs_dirs_excluded) + d.setVar('IMAGE_ROOTFS', rootfs_orig) +} + # # Compute the rootfs size # diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass index b373b21..d7d0b42 100644 --- a/meta/classes/image_types.bbclass +++ b/meta/classes/image_types.bbclass @@ -207,6 +207,7 @@ IMAGE_CMD_multiubi () { multiubi_mkfs "${mkubifs_args}" "${ubinize_args}" "${name}" done } +do_image_multiubi[respect_exclude_path] = "0" IMAGE_CMD_ubi () { multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}" diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass index 222ae00..9d74e51 100644 --- a/meta/classes/image_types_wic.bbclass +++ b/meta/classes/image_types_wic.bbclass @@ -34,6 +34,7 @@ IMAGE_CMD_wic () { rm -rf "$out/" } IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR" +do_image_wic[respect_exclude_path] = "0" # Rebuild when the wks file or vars in WICVARS change USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}" -- 2.7.4 ^ permalink raw reply related [flat|nested] 41+ messages in thread
* Re: [PATCH v6] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-10-13 9:08 ` Kristian Amlie 2017-10-13 9:08 ` [PATCH v7] " Kristian Amlie @ 2017-10-13 10:37 ` Alexander Kanavin 2017-10-13 11:22 ` Kristian Amlie 2017-11-22 13:13 ` Kristian Amlie 1 sibling, 2 replies; 41+ messages in thread From: Alexander Kanavin @ 2017-10-13 10:37 UTC (permalink / raw) To: Kristian Amlie, openembedded-core On 10/13/2017 12:08 PM, Kristian Amlie wrote: > Rebased again. > > Can we get this in please? It has been reviewed and rebased several times > already, and has lingered on the list for a long time. Bear with the pre-release freeze period please; once rocko is out patches will start flowing to master again. Alex ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v6] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-10-13 10:37 ` [PATCH v6] " Alexander Kanavin @ 2017-10-13 11:22 ` Kristian Amlie 2017-11-22 13:13 ` Kristian Amlie 1 sibling, 0 replies; 41+ messages in thread From: Kristian Amlie @ 2017-10-13 11:22 UTC (permalink / raw) To: Alexander Kanavin, openembedded-core On 13/10/17 12:37, Alexander Kanavin wrote: > On 10/13/2017 12:08 PM, Kristian Amlie wrote: >> Rebased again. >> >> Can we get this in please? It has been reviewed and rebased several times >> already, and has lingered on the list for a long time. > > Bear with the pre-release freeze period please; once rocko is out > patches will start flowing to master again. Oh ok, I'm not following the schedule. I will try again in a while then. -- Kristian ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v6] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-10-13 10:37 ` [PATCH v6] " Alexander Kanavin 2017-10-13 11:22 ` Kristian Amlie @ 2017-11-22 13:13 ` Kristian Amlie 2017-11-22 13:13 ` [PATCH v7] " Kristian Amlie 2017-11-22 13:31 ` [PATCH v6] " Alexander Kanavin 1 sibling, 2 replies; 41+ messages in thread From: Kristian Amlie @ 2017-11-22 13:13 UTC (permalink / raw) To: openembedded-core On 13/10/17 12:37, Alexander Kanavin wrote: >> ... > > Bear with the pre-release freeze period please; once rocko is out > patches will start flowing to master again. So now that rocko has been out for a while, can we get this in please? Already reviewed, and rebased patch is included. -- Kristian ^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH v7] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-11-22 13:13 ` Kristian Amlie @ 2017-11-22 13:13 ` Kristian Amlie 2017-11-22 13:31 ` [PATCH v6] " Alexander Kanavin 1 sibling, 0 replies; 41+ messages in thread From: Kristian Amlie @ 2017-11-22 13:13 UTC (permalink / raw) To: openembedded-core This is a direct followup from the earlier 6602392db3d39 commit in wic. It works more or less the same way: The variable specifies a list of directories relative to the root of the rootfs, and these directories will be excluded from the resulting rootfs image. If an entry ends with a slash, only the contents are omitted, not the directory itself. Since the intended use of the variable is to exclude certain directories from the rootfs, and then include said directories in other partitions, it is not natural for this variable to be respected for image creators that create multi partition images. These can turn the feature off locally by defining: do_image_myfs[respect_exclude_path] = "0" Specifically, "wic" and "multiubi" come with the feature disabled. Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech> --- meta/classes/image.bbclass | 84 +++++++++++++++++++++++++++++++++++- meta/classes/image_types.bbclass | 1 + meta/classes/image_types_wic.bbclass | 1 + 3 files changed, 84 insertions(+), 2 deletions(-) diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index d93de02..d378663 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -117,7 +117,8 @@ def rootfs_variables(d): 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS', 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS', - 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS'] + 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS', + 'IMAGE_ROOTFS_EXCLUDE_PATH'] variables.extend(rootfs_command_variables(d)) variables.extend(variable_depends(d)) return " ".join(variables) @@ -506,8 +507,9 @@ python () { d.setVarFlag(task, 'func', '1') d.setVarFlag(task, 'fakeroot', '1') - d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size') + d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories') d.prependVarFlag(task, 'postfuncs', ' create_symlinks') + d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories') d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages)) d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps)) d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE ' + ' '.join(vardepsexclude)) @@ -516,6 +518,84 @@ python () { bb.build.addtask(task, 'do_image_complete', after, d) } +python prepare_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + taskname = d.getVar("BB_CURRENTTASK") + + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0': + bb.debug(1, "'IMAGE_ROOTFS_EXCLUDE_PATH' is set but 'respect_exclude_path' variable flag is 0 for this image type, so ignoring it") + return + + import shutil + from oe.path import copyhardlinktree + + exclude_list = exclude_var.split() + + rootfs_orig = d.getVar('IMAGE_ROOTFS') + # We need a new rootfs directory we can delete files from. Copy to + # workdir. + new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname)) + + if os.path.lexists(new_rootfs): + shutil.rmtree(os.path.join(new_rootfs)) + + copyhardlinktree(rootfs_orig, new_rootfs) + + for orig_path in exclude_list: + path = orig_path + if os.path.isabs(path): + bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path) + + full_path = os.path.realpath(os.path.join(new_rootfs, path)) + + # Disallow climbing outside of parent directory using '..', + # because doing so could be quite disastrous (we will delete the + # directory). + if not full_path.startswith(new_rootfs): + bb.fatal("'%s' points to a path outside the rootfs" % orig_path) + + if path.endswith(os.sep): + # Delete content only. + for entry in os.listdir(full_path): + full_entry = os.path.join(full_path, entry) + if os.path.isdir(full_entry) and not os.path.islink(full_entry): + shutil.rmtree(full_entry) + else: + os.remove(full_entry) + else: + # Delete whole directory. + shutil.rmtree(full_path) + + # Save old value for cleanup later. + d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig) + d.setVar('IMAGE_ROOTFS', new_rootfs) +} + +python cleanup_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + taskname = d.getVar("BB_CURRENTTASK") + + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0': + return + + import shutil + + rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS') + rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG') + # This should never happen, since we should have set it to a different + # directory in the prepare function. + assert rootfs_dirs_excluded != rootfs_orig + + shutil.rmtree(rootfs_dirs_excluded) + d.setVar('IMAGE_ROOTFS', rootfs_orig) +} + # # Compute the rootfs size # diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass index d09d127..2d8a945 100644 --- a/meta/classes/image_types.bbclass +++ b/meta/classes/image_types.bbclass @@ -211,6 +211,7 @@ IMAGE_CMD_multiubi () { multiubi_mkfs "${mkubifs_args}" "${ubinize_args}" "${name}" done } +do_image_multiubi[respect_exclude_path] = "0" IMAGE_CMD_ubi () { multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}" diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass index 222ae00..9d74e51 100644 --- a/meta/classes/image_types_wic.bbclass +++ b/meta/classes/image_types_wic.bbclass @@ -34,6 +34,7 @@ IMAGE_CMD_wic () { rm -rf "$out/" } IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR" +do_image_wic[respect_exclude_path] = "0" # Rebuild when the wks file or vars in WICVARS change USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}" -- 2.7.4 ^ permalink raw reply related [flat|nested] 41+ messages in thread
* Re: [PATCH v6] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-11-22 13:13 ` Kristian Amlie 2017-11-22 13:13 ` [PATCH v7] " Kristian Amlie @ 2017-11-22 13:31 ` Alexander Kanavin 2017-11-22 13:35 ` Kristian Amlie 2018-01-25 10:33 ` Kristian Amlie 1 sibling, 2 replies; 41+ messages in thread From: Alexander Kanavin @ 2017-11-22 13:31 UTC (permalink / raw) To: Kristian Amlie, openembedded-core On 11/22/2017 03:13 PM, Kristian Amlie wrote: >> Bear with the pre-release freeze period please; once rocko is out >> patches will start flowing to master again. > > So now that rocko has been out for a while, can we get this in please? > Already reviewed, and rebased patch is included. Please do pay attention to what the recent Yocto Project Status emails say. Specifically: " Until we resolve these issues patches will continue to be slow to merge, if at all. This also blocks several core developers from doing any feature work at this point in time (e.g. layer setup tool is on hold, again). · We can only continue to stress that unless others step up and help to try and root cause these issues, things will stall with the project." The way things are with the project now, contributors are many, maintainers are few. If you can balance things out a little bit, that would be much appreciated. Alex ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v6] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-11-22 13:31 ` [PATCH v6] " Alexander Kanavin @ 2017-11-22 13:35 ` Kristian Amlie 2018-01-25 10:33 ` Kristian Amlie 1 sibling, 0 replies; 41+ messages in thread From: Kristian Amlie @ 2017-11-22 13:35 UTC (permalink / raw) To: Alexander Kanavin, openembedded-core On 22/11/17 14:31, Alexander Kanavin wrote: > On 11/22/2017 03:13 PM, Kristian Amlie wrote: > >>> Bear with the pre-release freeze period please; once rocko is out >>> patches will start flowing to master again. >> >> So now that rocko has been out for a while, can we get this in please? >> Already reviewed, and rebased patch is included. > > Please do pay attention to what the recent Yocto Project Status emails > say. Specifically: > > " Until we resolve these issues patches will continue to be slow to > merge, if at all. This also blocks several core developers from doing > any feature work at this point in time (e.g. layer setup tool is on > hold, again). > > · We can only continue to stress that unless others step up and > help to try and root cause these issues, things will stall with the > project." > > The way things are with the project now, contributors are many, > maintainers are few. If you can balance things out a little bit, that > would be much appreciated. Oh sorry, I missed that, I will keep an eye on the status updates! -- Kristian ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v6] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-11-22 13:31 ` [PATCH v6] " Alexander Kanavin 2017-11-22 13:35 ` Kristian Amlie @ 2018-01-25 10:33 ` Kristian Amlie 2018-01-25 10:33 ` [PATCH v7] " Kristian Amlie 1 sibling, 1 reply; 41+ messages in thread From: Kristian Amlie @ 2018-01-25 10:33 UTC (permalink / raw) To: openembedded-core >>> Bear with the pre-release freeze period please; once rocko is out >>> patches will start flowing to master again. >> >> So now that rocko has been out for a while, can we get this in please? >> Already reviewed, and rebased patch is included. > > Please do pay attention to what the recent Yocto Project Status emails > say. Specifically: > > " Until we resolve these issues patches will continue to be slow to merge, if at > all. This also blocks several core developers from doing any feature work at > this point in time (e.g. layer setup tool is on hold, again). Things seem to have calmed down a bit, so I'm submitting a rebased patch. Let me know if is not the appropriate time. -- Kristian ^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH v7] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2018-01-25 10:33 ` Kristian Amlie @ 2018-01-25 10:33 ` Kristian Amlie 2018-01-25 10:58 ` Martin Hundebøll 2018-02-13 1:29 ` Cal Sullivan 0 siblings, 2 replies; 41+ messages in thread From: Kristian Amlie @ 2018-01-25 10:33 UTC (permalink / raw) To: openembedded-core This is a direct followup from the earlier 6602392db3d39 commit in wic. It works more or less the same way: The variable specifies a list of directories relative to the root of the rootfs, and these directories will be excluded from the resulting rootfs image. If an entry ends with a slash, only the contents are omitted, not the directory itself. Since the intended use of the variable is to exclude certain directories from the rootfs, and then include said directories in other partitions, it is not natural for this variable to be respected for image creators that create multi partition images. These can turn the feature off locally by defining: do_image_myfs[respect_exclude_path] = "0" Specifically, "wic" and "multiubi" come with the feature disabled. Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech> --- meta/classes/image.bbclass | 84 +++++++++++++++++++++++++++++++++++- meta/classes/image_types.bbclass | 1 + meta/classes/image_types_wic.bbclass | 1 + 3 files changed, 84 insertions(+), 2 deletions(-) diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 4531aa2..849a19c 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -117,7 +117,8 @@ def rootfs_variables(d): 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS', 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS', - 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS'] + 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS', + 'IMAGE_ROOTFS_EXCLUDE_PATH'] variables.extend(rootfs_command_variables(d)) variables.extend(variable_depends(d)) return " ".join(variables) @@ -508,8 +509,9 @@ python () { d.setVarFlag(task, 'func', '1') d.setVarFlag(task, 'fakeroot', '1') - d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size') + d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories') d.prependVarFlag(task, 'postfuncs', ' create_symlinks') + d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories') d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages)) d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps)) d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE ' + ' '.join(vardepsexclude)) @@ -518,6 +520,84 @@ python () { bb.build.addtask(task, 'do_image_complete', after, d) } +python prepare_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + taskname = d.getVar("BB_CURRENTTASK") + + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0': + bb.debug(1, "'IMAGE_ROOTFS_EXCLUDE_PATH' is set but 'respect_exclude_path' variable flag is 0 for this image type, so ignoring it") + return + + import shutil + from oe.path import copyhardlinktree + + exclude_list = exclude_var.split() + + rootfs_orig = d.getVar('IMAGE_ROOTFS') + # We need a new rootfs directory we can delete files from. Copy to + # workdir. + new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname)) + + if os.path.lexists(new_rootfs): + shutil.rmtree(os.path.join(new_rootfs)) + + copyhardlinktree(rootfs_orig, new_rootfs) + + for orig_path in exclude_list: + path = orig_path + if os.path.isabs(path): + bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path) + + full_path = os.path.realpath(os.path.join(new_rootfs, path)) + + # Disallow climbing outside of parent directory using '..', + # because doing so could be quite disastrous (we will delete the + # directory). + if not full_path.startswith(new_rootfs): + bb.fatal("'%s' points to a path outside the rootfs" % orig_path) + + if path.endswith(os.sep): + # Delete content only. + for entry in os.listdir(full_path): + full_entry = os.path.join(full_path, entry) + if os.path.isdir(full_entry) and not os.path.islink(full_entry): + shutil.rmtree(full_entry) + else: + os.remove(full_entry) + else: + # Delete whole directory. + shutil.rmtree(full_path) + + # Save old value for cleanup later. + d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig) + d.setVar('IMAGE_ROOTFS', new_rootfs) +} + +python cleanup_excluded_directories() { + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') + if not exclude_var: + return + + taskname = d.getVar("BB_CURRENTTASK") + + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0': + return + + import shutil + + rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS') + rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG') + # This should never happen, since we should have set it to a different + # directory in the prepare function. + assert rootfs_dirs_excluded != rootfs_orig + + shutil.rmtree(rootfs_dirs_excluded) + d.setVar('IMAGE_ROOTFS', rootfs_orig) +} + # # Compute the rootfs size # diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass index c736f92..8cd8fd5 100644 --- a/meta/classes/image_types.bbclass +++ b/meta/classes/image_types.bbclass @@ -201,6 +201,7 @@ IMAGE_CMD_multiubi () { multiubi_mkfs "${mkubifs_args}" "${ubinize_args}" "${name}" done } +do_image_multiubi[respect_exclude_path] = "0" IMAGE_CMD_ubi () { multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}" diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass index dcf620c..452d7f3 100644 --- a/meta/classes/image_types_wic.bbclass +++ b/meta/classes/image_types_wic.bbclass @@ -34,6 +34,7 @@ IMAGE_CMD_wic () { rm -rf "$out/" } IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR" +do_image_wic[respect_exclude_path] = "0" # Rebuild when the wks file or vars in WICVARS change USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}" -- 2.7.4 ^ permalink raw reply related [flat|nested] 41+ messages in thread
* Re: [PATCH v7] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2018-01-25 10:33 ` [PATCH v7] " Kristian Amlie @ 2018-01-25 10:58 ` Martin Hundebøll 2018-02-13 1:29 ` Cal Sullivan 1 sibling, 0 replies; 41+ messages in thread From: Martin Hundebøll @ 2018-01-25 10:58 UTC (permalink / raw) To: openembedded-core Hi Kristian, Thanks for your work on this! On 2018-01-25 11:33, Kristian Amlie wrote: > This is a direct followup from the earlier 6602392db3d39 commit in > wic. It works more or less the same way: The variable specifies a list > of directories relative to the root of the rootfs, and these > directories will be excluded from the resulting rootfs image. If an > entry ends with a slash, only the contents are omitted, not the > directory itself. > > Since the intended use of the variable is to exclude certain > directories from the rootfs, and then include said directories in > other partitions, it is not natural for this variable to be respected > for image creators that create multi partition images. These can turn > the feature off locally by defining: > > do_image_myfs[respect_exclude_path] = "0" > > Specifically, "wic" and "multiubi" come with the feature disabled. > > Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech> > --- > meta/classes/image.bbclass | 84 +++++++++++++++++++++++++++++++++++- > meta/classes/image_types.bbclass | 1 + > meta/classes/image_types_wic.bbclass | 1 + > 3 files changed, 84 insertions(+), 2 deletions(-) > > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index 4531aa2..849a19c 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -117,7 +117,8 @@ def rootfs_variables(d): > 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', > 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS', > 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS', > - 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS'] > + 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS', > + 'IMAGE_ROOTFS_EXCLUDE_PATH'] > variables.extend(rootfs_command_variables(d)) > variables.extend(variable_depends(d)) > return " ".join(variables) > @@ -508,8 +509,9 @@ python () { > d.setVarFlag(task, 'func', '1') > d.setVarFlag(task, 'fakeroot', '1') > > - d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size') > + d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories') > d.prependVarFlag(task, 'postfuncs', ' create_symlinks') > + d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories') > d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages)) > d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps)) > d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE ' + ' '.join(vardepsexclude)) > @@ -518,6 +520,84 @@ python () { > bb.build.addtask(task, 'do_image_complete', after, d) > } > > +python prepare_excluded_directories() { > + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') > + if not exclude_var: > + return > + > + taskname = d.getVar("BB_CURRENTTASK") > + > + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0': > + bb.debug(1, "'IMAGE_ROOTFS_EXCLUDE_PATH' is set but 'respect_exclude_path' variable flag is 0 for this image type, so ignoring it") > + return If I understand this correctly, paths like "/data" aren't allowed in "IMAGE_ROOTFS_EXCLUDE_PATH". That seems counter-intuitive to me, as I would expect this feature to take absolute paths inside the target image to be excluded. If the check is there to avoid "os.path.join()" returning the RHS only instead of the joined path, then please use "oe.path.join()" instead. > + import shutil > + from oe.path import copyhardlinktree > + > + exclude_list = exclude_var.split() > + > + rootfs_orig = d.getVar('IMAGE_ROOTFS') > + # We need a new rootfs directory we can delete files from. Copy to > + # workdir. > + new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname)) > + > + if os.path.lexists(new_rootfs): > + shutil.rmtree(os.path.join(new_rootfs)) Wouldn't it be sufficient to add "new_rootfs" to "do_image[cleandirs]" ? > + > + copyhardlinktree(rootfs_orig, new_rootfs) > + > + for orig_path in exclude_list: > + path = orig_path > + if os.path.isabs(path): > + bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path) > + > + full_path = os.path.realpath(os.path.join(new_rootfs, path)) > + > + # Disallow climbing outside of parent directory using '..', > + # because doing so could be quite disastrous (we will delete the > + # directory). > + if not full_path.startswith(new_rootfs): > + bb.fatal("'%s' points to a path outside the rootfs" % orig_path) > + > + if path.endswith(os.sep): > + # Delete content only. > + for entry in os.listdir(full_path): > + full_entry = os.path.join(full_path, entry) > + if os.path.isdir(full_entry) and not os.path.islink(full_entry): > + shutil.rmtree(full_entry) > + else: > + os.remove(full_entry) > + else: > + # Delete whole directory. > + shutil.rmtree(full_path) > + > + # Save old value for cleanup later. > + d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig) > + d.setVar('IMAGE_ROOTFS', new_rootfs) > +} > + > +python cleanup_excluded_directories() { > + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') > + if not exclude_var: > + return > + > + taskname = d.getVar("BB_CURRENTTASK") > + > + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0': > + return > + > + import shutil > + > + rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS') > + rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG') > + # This should never happen, since we should have set it to a different > + # directory in the prepare function. > + assert rootfs_dirs_excluded != rootfs_orig > + > + shutil.rmtree(rootfs_dirs_excluded) > + d.setVar('IMAGE_ROOTFS', rootfs_orig) > +} Are we sure the cleanup is needed? I can image cases where I would like to have a look in "rootfs_dirs_excluded" to see what how it differs from "rootfs_orig". > + > # > # Compute the rootfs size > # > diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass > index c736f92..8cd8fd5 100644 > --- a/meta/classes/image_types.bbclass > +++ b/meta/classes/image_types.bbclass > @@ -201,6 +201,7 @@ IMAGE_CMD_multiubi () { > multiubi_mkfs "${mkubifs_args}" "${ubinize_args}" "${name}" > done > } > +do_image_multiubi[respect_exclude_path] = "0" > > IMAGE_CMD_ubi () { > multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}" > diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass > index dcf620c..452d7f3 100644 > --- a/meta/classes/image_types_wic.bbclass > +++ b/meta/classes/image_types_wic.bbclass > @@ -34,6 +34,7 @@ IMAGE_CMD_wic () { > rm -rf "$out/" > } > IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR" > +do_image_wic[respect_exclude_path] = "0" > > # Rebuild when the wks file or vars in WICVARS change > USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}" > -- MARTIN HUNDEBØLL, Prevas A/S Software Developer Hedeager 3, DK-8200 Aarhus N Phone +45 87438070 Mobile +45 25562438 Martin.Hundeboll@prevas.dk www.prevas.com ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v7] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2018-01-25 10:33 ` [PATCH v7] " Kristian Amlie 2018-01-25 10:58 ` Martin Hundebøll @ 2018-02-13 1:29 ` Cal Sullivan 2018-03-15 9:29 ` Kristian Amlie 1 sibling, 1 reply; 41+ messages in thread From: Cal Sullivan @ 2018-02-13 1:29 UTC (permalink / raw) To: Kristian Amlie, openembedded-core On 01/25/2018 02:33 AM, Kristian Amlie wrote: > This is a direct followup from the earlier 6602392db3d39 commit in > wic. It works more or less the same way: The variable specifies a list > of directories relative to the root of the rootfs, and these > directories will be excluded from the resulting rootfs image. If an > entry ends with a slash, only the contents are omitted, not the > directory itself. > > Since the intended use of the variable is to exclude certain > directories from the rootfs, and then include said directories in > other partitions, it is not natural for this variable to be respected > for image creators that create multi partition images. These can turn > the feature off locally by defining: > > do_image_myfs[respect_exclude_path] = "0" > > Specifically, "wic" and "multiubi" come with the feature disabled. > > Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech> > --- > meta/classes/image.bbclass | 84 +++++++++++++++++++++++++++++++++++- > meta/classes/image_types.bbclass | 1 + > meta/classes/image_types_wic.bbclass | 1 + > 3 files changed, 84 insertions(+), 2 deletions(-) > > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index 4531aa2..849a19c 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -117,7 +117,8 @@ def rootfs_variables(d): > 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', > 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS', > 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS', > - 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS'] > + 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS', > + 'IMAGE_ROOTFS_EXCLUDE_PATH'] > variables.extend(rootfs_command_variables(d)) > variables.extend(variable_depends(d)) > return " ".join(variables) > @@ -508,8 +509,9 @@ python () { > d.setVarFlag(task, 'func', '1') > d.setVarFlag(task, 'fakeroot', '1') > > - d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size') > + d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories') > d.prependVarFlag(task, 'postfuncs', ' create_symlinks') > + d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories') > d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages)) > d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps)) > d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE ' + ' '.join(vardepsexclude)) > @@ -518,6 +520,84 @@ python () { > bb.build.addtask(task, 'do_image_complete', after, d) > } > > +python prepare_excluded_directories() { > + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') > + if not exclude_var: > + return > + > + taskname = d.getVar("BB_CURRENTTASK") > + > + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0': > + bb.debug(1, "'IMAGE_ROOTFS_EXCLUDE_PATH' is set but 'respect_exclude_path' variable flag is 0 for this image type, so ignoring it") > + return > + > + import shutil > + from oe.path import copyhardlinktree > + > + exclude_list = exclude_var.split() > + > + rootfs_orig = d.getVar('IMAGE_ROOTFS') > + # We need a new rootfs directory we can delete files from. Copy to > + # workdir. > + new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname)) > + > + if os.path.lexists(new_rootfs): > + shutil.rmtree(os.path.join(new_rootfs)) > + > + copyhardlinktree(rootfs_orig, new_rootfs) > + > + for orig_path in exclude_list: > + path = orig_path > + if os.path.isabs(path): > + bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path) > + > + full_path = os.path.realpath(os.path.join(new_rootfs, path)) > + > + # Disallow climbing outside of parent directory using '..', > + # because doing so could be quite disastrous (we will delete the > + # directory). > + if not full_path.startswith(new_rootfs): > + bb.fatal("'%s' points to a path outside the rootfs" % orig_path) > + > + if path.endswith(os.sep): We need to check if full_path exists here, else it may try modifying things that don't exist. E.g., if I have IMAGE_ROOTFS_EXCLUDE_PATH = "boot" in local.conf and do 'bitbake core-image-minimal-initramfs' I get the following: ERROR: core-image-minimal-initramfs-1.0-r0 do_image_cpio: Error executing a python function in exec_python_func() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:prepare_excluded_directories(d) 0003: File: '/home/clsulliv/yocto/poky/meta/classes/image.bbclass', lineno: 572, function: prepare_excluded_directories 0568: else: 0569: os.remove(full_entry) 0570: else: 0571: # Delete whole directory. *** 0572: shutil.rmtree(full_path) 0573: 0574: # Save old value for cleanup later. 0575: d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig) 0576: d.setVar('IMAGE_ROOTFS', new_rootfs) File: '/usr/lib64/python3.6/shutil.py', lineno: 471, function: rmtree 0467: # lstat()/open()/fstat() trick. 0468: try: 0469: orig_st = os.lstat(path) 0470: except Exception: *** 0471: onerror(os.lstat, path, sys.exc_info()) 0472: return 0473: try: 0474: fd = os.open(path, os.O_RDONLY) 0475: except Exception: File: '/usr/lib64/python3.6/shutil.py', lineno: 469, function: rmtree 0465: path = os.fsdecode(path) 0466: # Note: To guard against symlink races, we use the standard 0467: # lstat()/open()/fstat() trick. 0468: try: *** 0469: orig_st = os.lstat(path) 0470: except Exception: 0471: onerror(os.lstat, path, sys.exc_info()) 0472: return 0473: try: Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/clsulliv/yocto/poky/build/tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/rootfs.image_cpio/boot' ERROR: core-image-minimal-initramfs-1.0-r0 do_image_cpio: Function failed: prepare_excluded_directories ERROR: Logfile of failure stored in: /home/clsulliv/yocto/poky/build/tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/temp/log.do_image_cpio.23514 ERROR: Task (/home/clsulliv/yocto/poky/meta/recipes-core/images/core-image-minimal-initramfs.bb:do_image_cpio) failed with exit code '1' Besides this, it works well. I'm using it here[1] as part of my attempt to improve EFI images. [1]. https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=clsulliv/image-creation-exclude-path Thanks, Cal > + # Delete content only. > + for entry in os.listdir(full_path): > + full_entry = os.path.join(full_path, entry) > + if os.path.isdir(full_entry) and not os.path.islink(full_entry): > + shutil.rmtree(full_entry) > + else: > + os.remove(full_entry) > + else: > + # Delete whole directory. > + shutil.rmtree(full_path) > + > + # Save old value for cleanup later. > + d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig) > + d.setVar('IMAGE_ROOTFS', new_rootfs) > +} > + > +python cleanup_excluded_directories() { > + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') > + if not exclude_var: > + return > + > + taskname = d.getVar("BB_CURRENTTASK") > + > + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0': > + return > + > + import shutil > + > + rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS') > + rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG') > + # This should never happen, since we should have set it to a different > + # directory in the prepare function. > + assert rootfs_dirs_excluded != rootfs_orig > + > + shutil.rmtree(rootfs_dirs_excluded) > + d.setVar('IMAGE_ROOTFS', rootfs_orig) > +} > + > # > # Compute the rootfs size > # > diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass > index c736f92..8cd8fd5 100644 > --- a/meta/classes/image_types.bbclass > +++ b/meta/classes/image_types.bbclass > @@ -201,6 +201,7 @@ IMAGE_CMD_multiubi () { > multiubi_mkfs "${mkubifs_args}" "${ubinize_args}" "${name}" > done > } > +do_image_multiubi[respect_exclude_path] = "0" > > IMAGE_CMD_ubi () { > multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}" > diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass > index dcf620c..452d7f3 100644 > --- a/meta/classes/image_types_wic.bbclass > +++ b/meta/classes/image_types_wic.bbclass > @@ -34,6 +34,7 @@ IMAGE_CMD_wic () { > rm -rf "$out/" > } > IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR" > +do_image_wic[respect_exclude_path] = "0" > > # Rebuild when the wks file or vars in WICVARS change > USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}" ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v7] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2018-02-13 1:29 ` Cal Sullivan @ 2018-03-15 9:29 ` Kristian Amlie 0 siblings, 0 replies; 41+ messages in thread From: Kristian Amlie @ 2018-03-15 9:29 UTC (permalink / raw) To: Cal Sullivan, openembedded-core On 13/02/18 02:29, Cal Sullivan wrote: >> ... >> + >> + if path.endswith(os.sep): > We need to check if full_path exists here, else it may try modifying > things that don't exist. E.g., if I have IMAGE_ROOTFS_EXCLUDE_PATH = > "boot" in local.conf and do 'bitbake core-image-minimal-initramfs' I get > the following: > > ERROR: core-image-minimal-initramfs-1.0-r0 do_image_cpio: Error > executing a python function in exec_python_func() autogenerated: > > The stack trace of python calls that resulted in this exception/failure > was: > File: 'exec_python_func() autogenerated', lineno: 2, function: <module> > 0001: > *** 0002:prepare_excluded_directories(d) > 0003: > File: '/home/clsulliv/yocto/poky/meta/classes/image.bbclass', lineno: > 572, function: prepare_excluded_directories > 0568: else: > 0569: os.remove(full_entry) > 0570: else: > 0571: # Delete whole directory. > *** 0572: shutil.rmtree(full_path) > 0573: > 0574: # Save old value for cleanup later. > 0575: d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig) > 0576: d.setVar('IMAGE_ROOTFS', new_rootfs) > File: '/usr/lib64/python3.6/shutil.py', lineno: 471, function: rmtree > 0467: # lstat()/open()/fstat() trick. > 0468: try: > 0469: orig_st = os.lstat(path) > 0470: except Exception: > *** 0471: onerror(os.lstat, path, sys.exc_info()) > 0472: return > 0473: try: > 0474: fd = os.open(path, os.O_RDONLY) > 0475: except Exception: > File: '/usr/lib64/python3.6/shutil.py', lineno: 469, function: rmtree > 0465: path = os.fsdecode(path) > 0466: # Note: To guard against symlink races, we use the > standard > 0467: # lstat()/open()/fstat() trick. > 0468: try: > *** 0469: orig_st = os.lstat(path) > 0470: except Exception: > 0471: onerror(os.lstat, path, sys.exc_info()) > 0472: return > 0473: try: > Exception: FileNotFoundError: [Errno 2] No such file or directory: > '/home/clsulliv/yocto/poky/build/tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/rootfs.image_cpio/boot' > > > ERROR: core-image-minimal-initramfs-1.0-r0 do_image_cpio: Function > failed: prepare_excluded_directories > ERROR: Logfile of failure stored in: > /home/clsulliv/yocto/poky/build/tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/temp/log.do_image_cpio.23514 > > ERROR: Task > (/home/clsulliv/yocto/poky/meta/recipes-core/images/core-image-minimal-initramfs.bb:do_image_cpio) > failed with exit code '1' > > Besides this, it works well. I'm using it here[1] as part of my attempt > to improve EFI images. > > [1]. > https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=clsulliv/image-creation-exclude-path Sorry for coming late to the party here. Nice find though, and the fix looks good too. Will you be pushing to get this merged in master? The process of getting it merged has been quite slow until now, so I've been considering implementing this downstream in the meta-mender layer instead, even though I think it could benefit many OE/poky users. -- Kristian ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-08-23 12:39 ` Kristian Amlie 2017-08-23 12:39 ` [PATCH v5] " Kristian Amlie @ 2017-08-23 12:46 ` Richard Purdie 2017-08-23 13:19 ` Kristian Amlie 1 sibling, 1 reply; 41+ messages in thread From: Richard Purdie @ 2017-08-23 12:46 UTC (permalink / raw) To: Kristian Amlie, openembedded-core On Wed, 2017-08-23 at 14:39 +0200, Kristian Amlie wrote: > I don't understand why I'm getting patch failures. This is rebased to > cutting edge master. Trying one more time... Its because you're patching documentation which is maintained in a separate repository. Separating that from the patch will avoid that error. Cheers, Richard ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. 2017-08-23 12:46 ` Richard Purdie @ 2017-08-23 13:19 ` Kristian Amlie 0 siblings, 0 replies; 41+ messages in thread From: Kristian Amlie @ 2017-08-23 13:19 UTC (permalink / raw) To: Richard Purdie, openembedded-core On 23/08/17 14:46, Richard Purdie wrote: > On Wed, 2017-08-23 at 14:39 +0200, Kristian Amlie wrote: >> I don't understand why I'm getting patch failures. This is rebased to >> cutting edge master. Trying one more time... > > Its because you're patching documentation which is maintained in a > separate repository. Separating that from the patch will avoid that > error. Aaah, right, this is from poky. Thanks for explaining, I'll fix that! -- Kristian ^ permalink raw reply [flat|nested] 41+ messages in thread
* ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev2) 2017-05-22 9:38 ` Kristian Amlie 2017-05-30 7:46 ` Kristian Amlie @ 2017-05-30 8:01 ` Patchwork 2017-06-19 14:31 ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev3) Patchwork ` (3 subsequent siblings) 5 siblings, 0 replies; 41+ messages in thread From: Patchwork @ 2017-05-30 8:01 UTC (permalink / raw) To: Kristian Amlie; +Cc: openembedded-core == Series Details == Series: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev2) Revision: 2 URL : https://patchwork.openembedded.org/series/6525/ State : failure == Summary == Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures: * Issue Series does not apply on top of target branch [test_series_merge_on_head] Suggested fix Rebase your series on top of targeted branch Targeted branch master (currently at 598e5da5a2) If you believe any of these test results are incorrect, please reply to the mailing list (openembedded-core@lists.openembedded.org) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...). --- Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe ^ permalink raw reply [flat|nested] 41+ messages in thread
* ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev3) 2017-05-22 9:38 ` Kristian Amlie 2017-05-30 7:46 ` Kristian Amlie 2017-05-30 8:01 ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev2) Patchwork @ 2017-06-19 14:31 ` Patchwork 2017-08-23 12:04 ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev4) Patchwork ` (2 subsequent siblings) 5 siblings, 0 replies; 41+ messages in thread From: Patchwork @ 2017-06-19 14:31 UTC (permalink / raw) To: Kristian Amlie; +Cc: openembedded-core == Series Details == Series: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev3) Revision: 3 URL : https://patchwork.openembedded.org/series/6525/ State : failure == Summary == Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures: * Issue Series does not apply on top of target branch [test_series_merge_on_head] Suggested fix Rebase your series on top of targeted branch Targeted branch master (currently at 1d9a88f635) If you believe any of these test results are incorrect, please reply to the mailing list (openembedded-core@lists.openembedded.org) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...). --- Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe ^ permalink raw reply [flat|nested] 41+ messages in thread
* ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev4) 2017-05-22 9:38 ` Kristian Amlie ` (2 preceding siblings ...) 2017-06-19 14:31 ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev3) Patchwork @ 2017-08-23 12:04 ` Patchwork 2017-08-23 12:04 ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev5) Patchwork 2017-08-23 13:04 ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev6) Patchwork 5 siblings, 0 replies; 41+ messages in thread From: Patchwork @ 2017-08-23 12:04 UTC (permalink / raw) To: Kristian Amlie; +Cc: openembedded-core == Series Details == Series: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev4) Revision: 4 URL : https://patchwork.openembedded.org/series/6525/ State : failure == Summary == Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures: * Issue Series does not apply on top of target branch [test_series_merge_on_head] Suggested fix Rebase your series on top of targeted branch Targeted branch master (currently at f59e729f98) If you believe any of these test results are incorrect, please reply to the mailing list (openembedded-core@lists.openembedded.org) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...). --- Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe ^ permalink raw reply [flat|nested] 41+ messages in thread
* ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev5) 2017-05-22 9:38 ` Kristian Amlie ` (3 preceding siblings ...) 2017-08-23 12:04 ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev4) Patchwork @ 2017-08-23 12:04 ` Patchwork 2017-08-23 13:04 ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev6) Patchwork 5 siblings, 0 replies; 41+ messages in thread From: Patchwork @ 2017-08-23 12:04 UTC (permalink / raw) To: Kristian Amlie; +Cc: openembedded-core == Series Details == Series: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev5) Revision: 5 URL : https://patchwork.openembedded.org/series/6525/ State : failure == Summary == Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures: * Issue Series does not apply on top of target branch [test_series_merge_on_head] Suggested fix Rebase your series on top of targeted branch Targeted branch master (currently at f59e729f98) If you believe any of these test results are incorrect, please reply to the mailing list (openembedded-core@lists.openembedded.org) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...). --- Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe ^ permalink raw reply [flat|nested] 41+ messages in thread
* ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev6) 2017-05-22 9:38 ` Kristian Amlie ` (4 preceding siblings ...) 2017-08-23 12:04 ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev5) Patchwork @ 2017-08-23 13:04 ` Patchwork 2017-08-23 15:12 ` Leonardo Sandoval 5 siblings, 1 reply; 41+ messages in thread From: Patchwork @ 2017-08-23 13:04 UTC (permalink / raw) To: Kristian Amlie; +Cc: openembedded-core == Series Details == Series: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev6) Revision: 6 URL : https://patchwork.openembedded.org/series/6525/ State : failure == Summary == Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures: * Issue Series does not apply on top of target branch [test_series_merge_on_head] Suggested fix Rebase your series on top of targeted branch Targeted branch master (currently at b1e482ae20) If you believe any of these test results are incorrect, please reply to the mailing list (openembedded-core@lists.openembedded.org) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...). --- Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev6) 2017-08-23 13:04 ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev6) Patchwork @ 2017-08-23 15:12 ` Leonardo Sandoval 0 siblings, 0 replies; 41+ messages in thread From: Leonardo Sandoval @ 2017-08-23 15:12 UTC (permalink / raw) To: openembedded-core On Wed, 2017-08-23 at 13:04 +0000, Patchwork wrote: > == Series Details == > > Series: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev6) > Revision: 6 > URL : https://patchwork.openembedded.org/series/6525/ > State : failure > > == Summary == > > > Thank you for submitting this patch series to OpenEmbedded Core. This is > an automated response. Several tests have been executed on the proposed > series by patchtest resulting in the following failures: > > > > * Issue Series does not apply on top of target branch [test_series_merge_on_head] > Suggested fix Rebase your series on top of targeted branch > Targeted branch master (currently at b1e482ae20) > it is not pretty clear what the problem is with your patch with the message above but the problem is that your are changing documentation on your series and the latter goes into another mailing list. Check poky's README for more info. > > > If you believe any of these test results are incorrect, please reply to the > mailing list (openembedded-core@lists.openembedded.org) raising your concerns. > Otherwise we would appreciate you correcting the issues and submitting a new > version of the patchset if applicable. Please ensure you add/increment the > version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> > [PATCH v3] -> ...). > > --- > Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest > Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe > ^ permalink raw reply [flat|nested] 41+ messages in thread
end of thread, other threads:[~2018-03-15 9:29 UTC | newest] Thread overview: 41+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-04-26 15:03 do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH Kristian Amlie 2017-04-26 15:03 ` [PATCH] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature Kristian Amlie 2017-05-22 7:08 ` do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH Kristian Amlie 2017-05-22 8:46 ` Ed Bartosh 2017-05-22 9:38 ` Kristian Amlie 2017-05-30 7:46 ` Kristian Amlie 2017-05-30 7:46 ` [PATCH v2] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature Kristian Amlie 2017-06-19 9:01 ` Kristian Amlie 2017-06-19 14:01 ` Kristian Amlie 2017-06-19 14:01 ` [PATCH v3] " Kristian Amlie 2017-08-23 11:44 ` [PATCH v4] " Kristian Amlie 2017-08-23 11:46 ` Kristian Amlie 2017-08-23 11:46 ` [PATCH v4] " Kristian Amlie 2017-08-23 12:39 ` Kristian Amlie 2017-08-23 12:39 ` [PATCH v5] " Kristian Amlie 2017-08-23 12:47 ` Richard Purdie 2017-08-23 13:19 ` Kristian Amlie 2017-08-28 15:47 ` Kristian Amlie 2017-08-28 15:47 ` [PATCH v6] " Kristian Amlie 2017-09-18 6:45 ` Kristian Amlie 2017-10-13 9:08 ` Kristian Amlie 2017-10-13 9:08 ` [PATCH v7] " Kristian Amlie 2017-10-13 10:37 ` [PATCH v6] " Alexander Kanavin 2017-10-13 11:22 ` Kristian Amlie 2017-11-22 13:13 ` Kristian Amlie 2017-11-22 13:13 ` [PATCH v7] " Kristian Amlie 2017-11-22 13:31 ` [PATCH v6] " Alexander Kanavin 2017-11-22 13:35 ` Kristian Amlie 2018-01-25 10:33 ` Kristian Amlie 2018-01-25 10:33 ` [PATCH v7] " Kristian Amlie 2018-01-25 10:58 ` Martin Hundebøll 2018-02-13 1:29 ` Cal Sullivan 2018-03-15 9:29 ` Kristian Amlie 2017-08-23 12:46 ` Richard Purdie 2017-08-23 13:19 ` Kristian Amlie 2017-05-30 8:01 ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev2) Patchwork 2017-06-19 14:31 ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev3) Patchwork 2017-08-23 12:04 ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev4) Patchwork 2017-08-23 12:04 ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev5) Patchwork 2017-08-23 13:04 ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev6) Patchwork 2017-08-23 15:12 ` Leonardo Sandoval
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.