From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf0-f48.google.com (mail-lf0-f48.google.com [209.85.215.48]) by mail.openembedded.org (Postfix) with ESMTP id 47DAA78274 for ; Mon, 19 Jun 2017 09:01:42 +0000 (UTC) Received: by mail-lf0-f48.google.com with SMTP id v20so51472509lfa.1 for ; Mon, 19 Jun 2017 02:01:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mender-io.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=caFmtgL6/OrNx7uoa8nfStdc5jbBYwPx2usCRAu/98g=; b=stupuRHVdYxLcmxwvfMhGKP17Yuz0PQETQno9OxOQ/oUNDXcPm7XFwK80eOvpD+fAu /NsAcpAZ9OaaSQn+WhptEp9gSmhkd69P42YoP7wJ7bBLJIT0sZwBg1GgPM5DSDJhjUUK sabJhSvO8wK+s+X4Qyr0e1Nf3/aOlovcicgOlH9O2Al5wG+v1B5+/Yr3Q/4HKCWmueZd 8N3p4p0JsgNsmQ8mSHZlhFuCR9JTo+iEVDj8L8fVgd5IXCmymlgepViLQZvy5TYgMLNg bUv/LKmo8BWSlAr9+KkfM6tYc3sJC8UnlRyb313UoyM5x2VCdK7P1kjx/FRY2UqTy9ND Zm8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=caFmtgL6/OrNx7uoa8nfStdc5jbBYwPx2usCRAu/98g=; b=Bv+CHWWPnd3WGtEVvN4HrHJHFs1/Wy5e9Ge9W1aaxkTeAgk0zdD72qrCRhkiPTPyzp UjsiXcIor/7S6NwlO+mVeQk05uUXqhfiNqUH6pet8Yow7jbBjoJALRFGYWxkY1/7H4B0 PY5O6Zl39TRcQEJPnE9nBDNNfV+6Yk4ecLUVLt9WYPgtQizDYd0bQtb8yBM5c0uyEsQm wnBJr2ilnVR4ERsWg/JcUZK/pKV9db/C9+iEk8uJTvDN/I61gA8uqfoeOr4n8xzvAqTa mISwwW8PzF+wj/v2pstzgv/H/aivkkU7btJcGcmLOxPwCRyNpMV3ZoOV6yrAludyfOLL r2Sw== X-Gm-Message-State: AKS2vOwcSTG9M0CZ5/MzCZ87T6r2nppoajIxVKWXmWFBpyt7swxA6OH5 ieP+YFu61+kWnoc04EVuAA== X-Received: by 10.25.15.157 with SMTP id 29mr6548307lfp.10.1497862902742; Mon, 19 Jun 2017 02:01:42 -0700 (PDT) Received: from [10.20.33.170] ([195.159.234.190]) by smtp.googlemail.com with ESMTPSA id z10sm2193774lja.6.2017.06.19.02.01.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Jun 2017 02:01:41 -0700 (PDT) To: Kristian Amlie , openembedded-core@lists.openembedded.org References: <8e87fb4b-6180-efdc-6f60-4e8ca6d5564c@mender.io> <1496130382-852-1-git-send-email-kristian.amlie@mender.io> <1496130382-852-2-git-send-email-kristian.amlie@mender.io> From: Kristian Amlie Message-ID: Date: Mon, 19 Jun 2017 11:01:41 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: <1496130382-852-2-git-send-email-kristian.amlie@mender.io> Subject: Re: [PATCH v2] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Jun 2017 09:01:42 -0000 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Ping. I created https://bugzilla.yoctoproject.org/show_bug.cgi?id=11688 to help track this. -- Kristian On 30/05/17 09:46, Kristian Amlie wrote: > This is a direct followup from the earlier f6a064d969f414 commit in > wic. It works more or less the same way: The variable specifies a list > of directories relative to the root of the rootfs, and these > directories will be excluded from the resulting rootfs image. If an > entry ends with a slash, only the contents are omitted, not the > directory itself. > > Since the intended use of the variable is to exclude certain > directories from the rootfs, and then include said directories in > other partitions, it is not natural for this variable to be respected > for image creators that create multi partition images. Therefore the > default is to ignore the variable, and image creators that create > single root filesystems need to set > > do_image_myfs[respect_exclude_path] = "1" > > in order to honor it. Specifically, "wic" and "multiubi" have not > received this variable flag, while others have. > > Signed-off-by: Kristian Amlie > --- > documentation/ref-manual/ref-variables.xml | 35 +++++++++++++ > meta/classes/image.bbclass | 83 ++++++++++++++++++++++++++++-- > meta/classes/image_types.bbclass | 13 +++++ > 3 files changed, 128 insertions(+), 3 deletions(-) > > diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml > index 94558df..27f8b8e 100644 > --- a/documentation/ref-manual/ref-variables.xml > +++ b/documentation/ref-manual/ref-variables.xml > @@ -6041,6 +6041,41 @@ > > > > + IMAGE_ROOTFS_EXCLUDE_PATH > + > + IMAGE_ROOTFS_EXCLUDE_PATH[doc] = "Specifies paths to omit from the rootfs." > + > + > + > + > + Specifies paths that should be omitted from the root filesystem, separated by > + spaces. The paths must be relative to the root of the filesystem (for example > + "usr"). Single root filesystem images will not include the path in the > + filesystem, except if the path ends with a slash, then the directory will be > + included, but without any content. > + > + # Omit /usr completely from rootfs. > + IMAGE_ROOTFS_EXCLUDE_PATH += "usr ..." > + # Include the directory /usr in rootfs, but not its content. > + IMAGE_ROOTFS_EXCLUDE_PATH += "usr/ ..." > + > + > + > + Image creators that create multi partition images typically do not honor this > + variable, and must provide their own method to split the directory > + structure. For example, wic has > + --exclude-path and --rootfs-dir. > + > + > + The default for custom image creators is to ignore this variable. Those that > + need to honor it must set: > + > + do_image_custom[respect_exclude_path] = "1" > + > + > + > + > + > IMAGE_ROOTFS_EXTRA_SPACE > > IMAGE_ROOTFS_EXTRA_SPACE[doc] = "Defines additional free disk space created in the image in Kbytes. By default, this variable is set to '0'." > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index 85f6b9a..c53e383 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -118,7 +118,7 @@ def rootfs_variables(d): > 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', > 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS', > 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS', > - 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY'] > + 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'IMAGE_ROOTFS_EXCLUDE_PATH'] > variables.extend(rootfs_command_variables(d)) > variables.extend(variable_depends(d)) > return " ".join(variables) > @@ -496,8 +496,8 @@ python () { > d.setVar('do_image_%s' % t, '\n'.join(cmds)) > d.setVarFlag('do_image_%s' % t, 'func', '1') > d.setVarFlag('do_image_%s' % t, 'fakeroot', '1') > - d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size') > - d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks') > + d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size prepare_excluded_directories') > + d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks cleanup_excluded_directories') > d.setVarFlag('do_image_%s' % t, 'subimages', ' '.join(subimages)) > d.appendVarFlag('do_image_%s' % t, 'vardeps', ' '.join(vardeps)) > d.appendVarFlag('do_image_%s' % t, 'vardepsexclude', 'DATETIME') > @@ -506,6 +506,83 @@ python () { > bb.build.addtask('do_image_%s' % t, 'do_image_complete', after, d) > } > > +python prepare_excluded_directories() { > + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') > + if not exclude_var: > + return > + > + taskname = d.getVar("BB_CURRENTTASK") > + > + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1': > + return > + > + import shutil > + from oe.path import copyhardlinktree > + > + exclude_list = exclude_var.split() > + > + rootfs_orig = d.getVar('IMAGE_ROOTFS') > + # We need a new rootfs directory we can delete files from. Copy to > + # workdir. > + new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname)) > + > + if os.path.lexists(new_rootfs): > + shutil.rmtree(os.path.join(new_rootfs)) > + > + copyhardlinktree(rootfs_orig, new_rootfs) > + > + for orig_path in exclude_list: > + path = orig_path > + if os.path.isabs(path): > + bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path) > + > + full_path = os.path.realpath(os.path.join(new_rootfs, path)) > + > + # Disallow climbing outside of parent directory using '..', > + # because doing so could be quite disastrous (we will delete the > + # directory). > + if not full_path.startswith(new_rootfs): > + bb.fatal("'%s' points to a path outside the rootfs" % orig_path) > + > + if path.endswith(os.sep): > + # Delete content only. > + for entry in os.listdir(full_path): > + full_entry = os.path.join(full_path, entry) > + if os.path.isdir(full_entry) and not os.path.islink(full_entry): > + shutil.rmtree(full_entry) > + else: > + os.remove(full_entry) > + else: > + # Delete whole directory. > + shutil.rmtree(full_path) > + > + # Save old value for cleanup later. > + d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig) > + d.setVar('IMAGE_ROOTFS', new_rootfs) > +} > + > +python cleanup_excluded_directories() { > + exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH') > + if not exclude_var: > + return > + > + taskname = d.getVar("BB_CURRENTTASK") > + > + if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1': > + return > + > + import shutil > + > + rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS') > + rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG') > + # This should never happen, since we should have set it to a different > + # directory in the prepare function. > + assert rootfs_dirs_excluded != rootfs_orig > + > + shutil.rmtree(rootfs_dirs_excluded) > + d.setVar('IMAGE_ROOTFS', rootfs_orig) > +} > + > # > # Compute the rootfs size > # > diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass > index 7749b00..b27dfe3 100644 > --- a/meta/classes/image_types.bbclass > +++ b/meta/classes/image_types.bbclass > @@ -51,8 +51,10 @@ ZIP_COMPRESSION_LEVEL ?= "-9" > > JFFS2_SUM_EXTRA_ARGS ?= "" > IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 ${EXTRA_IMAGECMD}" > +do_image_jffs2[respect_exclude_path] = "1" > > IMAGE_CMD_cramfs = "mkfs.cramfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cramfs ${EXTRA_IMAGECMD}" > +do_image_cramfs[respect_exclude_path] = "1" > > oe_mkext234fs () { > fstype=$1 > @@ -81,6 +83,9 @@ oe_mkext234fs () { > IMAGE_CMD_ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}" > IMAGE_CMD_ext3 = "oe_mkext234fs ext3 ${EXTRA_IMAGECMD}" > IMAGE_CMD_ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}" > +do_image_ext2[respect_exclude_path] = "1" > +do_image_ext3[respect_exclude_path] = "1" > +do_image_ext4[respect_exclude_path] = "1" > > MIN_BTRFS_SIZE ?= "16384" > IMAGE_CMD_btrfs () { > @@ -92,10 +97,14 @@ IMAGE_CMD_btrfs () { > dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs count=${size} bs=1024 > mkfs.btrfs ${EXTRA_IMAGECMD} -r ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs > } > +do_image_btrfs[respect_exclude_path] = "1" > > IMAGE_CMD_squashfs = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs ${EXTRA_IMAGECMD} -noappend" > IMAGE_CMD_squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz" > IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo" > +do_image_squashfs[respect_exclude_path] = "1" > +do_image_squashfs-xz[respect_exclude_path] = "1" > +do_image_squashfs-lzo[respect_exclude_path] = "1" > > # By default, tar from the host is used, which can be quite old. If > # you need special parameters (like --xattrs) which are only supported > @@ -109,6 +118,7 @@ IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAM > # required when extracting, but it seems prudent to use it in both cases. > IMAGE_CMD_TAR ?= "tar" > IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cvf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} ." > +do_image_tar[respect_exclude_path] = "1" > > do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append" > IMAGE_CMD_cpio () { > @@ -129,6 +139,7 @@ IMAGE_CMD_cpio () { > fi > fi > } > +do_image_cpio[respect_exclude_path] = "1" > > ELF_KERNEL ?= "${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE}" > ELF_APPEND ?= "ramdisk_size=32768 root=/dev/ram0 rw console=" > @@ -137,6 +148,7 @@ IMAGE_CMD_elf () { > test -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf && rm -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf > mkelfImage --kernel=${ELF_KERNEL} --initrd=${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.cpio.gz --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf --append='${ELF_APPEND}' ${EXTRA_IMAGECMD} > } > +do_image_elf[respect_exclude_path] = "1" > > IMAGE_TYPEDEP_elf = "cpio.gz" > > @@ -194,6 +206,7 @@ IMAGE_CMD_ubi () { > } > > IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ubifs ${MKUBIFS_ARGS}" > +do_image_ubifs[respect_exclude_path] = "1" > > EXTRA_IMAGECMD = "" > >