All of lore.kernel.org
 help / color / mirror / Atom feed
* do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH.
@ 2017-04-26 15:03 Kristian Amlie
  2017-04-26 15:03 ` [PATCH] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature Kristian Amlie
  2017-05-22  7:08 ` do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH Kristian Amlie
  0 siblings, 2 replies; 41+ messages in thread
From: Kristian Amlie @ 2017-04-26 15:03 UTC (permalink / raw)
  To: openembedded-core

This is a direct followup from the earlier f6a064d969f414 commit in wic. It
works more or less the same way: The variable specifies a list of directories
relative to the root of the rootfs, and these directories will be excluded from
the resulting rootfs image. If an entry ends with a slash, only the contents are
omitted, not the directory itself.

This is early feedback call, and several things are missing here (see
below). However, I would like to know if at least the basic approach is
acceptable. Reviewers pressed on time really only need to look at the topmost
hunk as well as the last couple of lines of the two functions. Those show what
the basic idea is.

What's missing:

* Documentation

* Test coverage

* Respecting IMAGE_ROOTFS_EXCLUDE_PATH is a behavior change, and isn't
  appropriate for all image creators. In fact, for the the main use case [1] to
  work it must not always be respected.

My idea for the last point is that the variable will not be respected by
default, and each of the pure filesystem image creators (btrfs, ext2/3/4, etc)
will set a flag that mark their image creator as a "pure filesystem", which
would respect this variable. Image creators that make partitions, however,
should not respect it, because they will use wic for that purpose instead, and
then they need to have access to the entire rootfs. Hence they should not set
the flag.

I'm imagining something like this:

  d.setVarFlag('do_image_%s' % t, 'respect_exclude_path', '1')

although I'm not sure if that's the best approach (tips are welcome).

[1] The use case for this feature is the following: We are creating two types of
images in Mender: One complete partitioned image, and one image containing only
the rootfs. In order to be able to populate non-rootfs partitions, we want to
use the regular bitbake recipes for building these components, but then exclude
certain directories from the rootfs image. Using wic, we are adding those
directories back into the partitioned image, but under different
partitions. Unlike the rootfs image, the other partitions are not expected to be
rebuilt after the first rollout.

-- 
Kristian



^ permalink raw reply	[flat|nested] 41+ messages in thread

* [PATCH] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-04-26 15:03 do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH Kristian Amlie
@ 2017-04-26 15:03 ` Kristian Amlie
  2017-05-22  7:08 ` do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH Kristian Amlie
  1 sibling, 0 replies; 41+ messages in thread
From: Kristian Amlie @ 2017-04-26 15:03 UTC (permalink / raw)
  To: openembedded-core

This is a direct followup from the earlier f6a064d969f414 commit in
wic. It works more or less the same way: The variable specifies a list
of directories relative to the root of the rootfs, and these
directories will be excluded from the resulting rootfs image. If an
entry ends with a slash, only the contents are omitted, not the
directory itself.

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
---
 meta/classes/image.bbclass | 73 ++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 71 insertions(+), 2 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 405fd73..a309435 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -496,8 +496,8 @@ python () {
         d.setVar('do_image_%s' % t, '\n'.join(cmds))
         d.setVarFlag('do_image_%s' % t, 'func', '1')
         d.setVarFlag('do_image_%s' % t, 'fakeroot', '1')
-        d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size')
-        d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks')
+        d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size prepare_excluded_directories')
+        d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks cleanup_excluded_directories')
         d.setVarFlag('do_image_%s' % t, 'subimages', ' '.join(subimages))
         d.appendVarFlag('do_image_%s' % t, 'vardeps', ' '.join(vardeps))
         d.appendVarFlag('do_image_%s' % t, 'vardepsexclude', 'DATETIME')
@@ -506,6 +506,75 @@ python () {
         bb.build.addtask('do_image_%s' % t, 'do_image_complete', after, d)
 }
 
+python prepare_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    import shutil
+    from oe.path import copyhardlinktree
+
+    exclude_list = exclude_var.split()
+
+    taskname = d.getVar("BB_CURRENTTASK")
+
+    rootfs_orig = d.getVar('IMAGE_ROOTFS')
+    # We need a new rootfs directory we can delete files from. Copy to
+    # workdir.
+    new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname))
+
+    if os.path.lexists(new_rootfs):
+        shutil.rmtree(os.path.join(new_rootfs))
+
+    copyhardlinktree(rootfs_orig, new_rootfs)
+
+    for orig_path in exclude_list:
+        path = orig_path
+        if os.path.isabs(path):
+            msger.error("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path)
+
+        full_path = os.path.realpath(os.path.join(new_rootfs, path))
+
+        # Disallow climbing outside of parent directory using '..',
+        # because doing so could be quite disastrous (we will delete the
+        # directory).
+        if not full_path.startswith(new_rootfs):
+            msger.error("'%s' points to a path outside the rootfs" % orig_path)
+
+        if path.endswith(os.sep):
+            # Delete content only.
+            for entry in os.listdir(full_path):
+                full_entry = os.path.join(full_path, entry)
+                if os.path.isdir(full_entry) and not os.path.islink(full_entry):
+                    shutil.rmtree(full_entry)
+                else:
+                    os.remove(full_entry)
+        else:
+            # Delete whole directory.
+            shutil.rmtree(full_path)
+
+    # Save old value for cleanup later.
+    d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig)
+    d.setVar('IMAGE_ROOTFS', new_rootfs)
+}
+
+python cleanup_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    import shutil
+
+    rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS')
+    rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG')
+    # This should never happen, since we should have set it to a different
+    # directory in the prepare function.
+    assert rootfs_dirs_excluded != rootfs_orig
+
+    shutil.rmtree(rootfs_dirs_excluded)
+    d.setVar('IMAGE_ROOTFS', rootfs_orig)
+}
+
 #
 # Compute the rootfs size
 #
-- 
2.7.4



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* Re: do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH.
  2017-04-26 15:03 do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH Kristian Amlie
  2017-04-26 15:03 ` [PATCH] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature Kristian Amlie
@ 2017-05-22  7:08 ` Kristian Amlie
  2017-05-22  8:46   ` Ed Bartosh
  1 sibling, 1 reply; 41+ messages in thread
From: Kristian Amlie @ 2017-05-22  7:08 UTC (permalink / raw)
  To: openembedded-core

Ping. Would be great to get some feedback so I can continue working on this.

--
Kristian

On 26/04/17 17:03, Kristian Amlie wrote:
> This is a direct followup from the earlier f6a064d969f414 commit in wic. It
> works more or less the same way: The variable specifies a list of directories
> relative to the root of the rootfs, and these directories will be excluded from
> the resulting rootfs image. If an entry ends with a slash, only the contents are
> omitted, not the directory itself.
> 
> This is early feedback call, and several things are missing here (see
> below). However, I would like to know if at least the basic approach is
> acceptable. Reviewers pressed on time really only need to look at the topmost
> hunk as well as the last couple of lines of the two functions. Those show what
> the basic idea is.
> 
> What's missing:
> 
> * Documentation
> 
> * Test coverage
> 
> * Respecting IMAGE_ROOTFS_EXCLUDE_PATH is a behavior change, and isn't
>   appropriate for all image creators. In fact, for the the main use case [1] to
>   work it must not always be respected.
> 
> My idea for the last point is that the variable will not be respected by
> default, and each of the pure filesystem image creators (btrfs, ext2/3/4, etc)
> will set a flag that mark their image creator as a "pure filesystem", which
> would respect this variable. Image creators that make partitions, however,
> should not respect it, because they will use wic for that purpose instead, and
> then they need to have access to the entire rootfs. Hence they should not set
> the flag.
> 
> I'm imagining something like this:
> 
>   d.setVarFlag('do_image_%s' % t, 'respect_exclude_path', '1')
> 
> although I'm not sure if that's the best approach (tips are welcome).
> 
> [1] The use case for this feature is the following: We are creating two types of
> images in Mender: One complete partitioned image, and one image containing only
> the rootfs. In order to be able to populate non-rootfs partitions, we want to
> use the regular bitbake recipes for building these components, but then exclude
> certain directories from the rootfs image. Using wic, we are adding those
> directories back into the partitioned image, but under different
> partitions. Unlike the rootfs image, the other partitions are not expected to be
> rebuilt after the first rollout.
> 



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH.
  2017-05-22  7:08 ` do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH Kristian Amlie
@ 2017-05-22  8:46   ` Ed Bartosh
  2017-05-22  9:38     ` Kristian Amlie
  0 siblings, 1 reply; 41+ messages in thread
From: Ed Bartosh @ 2017-05-22  8:46 UTC (permalink / raw)
  To: Kristian Amlie; +Cc: openembedded-core

On Mon, May 22, 2017 at 09:08:41AM +0200, Kristian Amlie wrote:
> Ping. Would be great to get some feedback so I can continue working on this.
> 
I like the idea.

Lately I was working on generic EFI implementation
and thought that it would be nice to be able to package EFI artifacts
and get them installed the same way as other packages. With
IMAGE_ROOTFS_EXCLUDE_PATH = '<directory with efi artifacts>' it would be
possible to not include that directory into the rootfs if needed.

> --
> Kristian
> 
> On 26/04/17 17:03, Kristian Amlie wrote:
> > This is a direct followup from the earlier f6a064d969f414 commit in wic. It
> > works more or less the same way: The variable specifies a list of directories
> > relative to the root of the rootfs, and these directories will be excluded from
> > the resulting rootfs image. If an entry ends with a slash, only the contents are
> > omitted, not the directory itself.
> > 
> > This is early feedback call, and several things are missing here (see
> > below). However, I would like to know if at least the basic approach is
> > acceptable. Reviewers pressed on time really only need to look at the topmost
> > hunk as well as the last couple of lines of the two functions. Those show what
> > the basic idea is.
> > 
> > What's missing:
> > 
> > * Documentation
> > 
> > * Test coverage
> > 
> > * Respecting IMAGE_ROOTFS_EXCLUDE_PATH is a behavior change, and isn't
> >   appropriate for all image creators. In fact, for the the main use case [1] to
> >   work it must not always be respected.
> > 
> > My idea for the last point is that the variable will not be respected by
> > default, and each of the pure filesystem image creators (btrfs, ext2/3/4, etc)
> > will set a flag that mark their image creator as a "pure filesystem", which
> > would respect this variable. Image creators that make partitions, however,
> > should not respect it, because they will use wic for that purpose instead, and
> > then they need to have access to the entire rootfs. Hence they should not set
> > the flag.
> > 
> > I'm imagining something like this:
> > 
> >   d.setVarFlag('do_image_%s' % t, 'respect_exclude_path', '1')
> > 
> > although I'm not sure if that's the best approach (tips are welcome).
> > 
> > [1] The use case for this feature is the following: We are creating two types of
> > images in Mender: One complete partitioned image, and one image containing only
> > the rootfs. In order to be able to populate non-rootfs partitions, we want to
> > use the regular bitbake recipes for building these components, but then exclude
> > certain directories from the rootfs image. Using wic, we are adding those
> > directories back into the partitioned image, but under different
> > partitions. Unlike the rootfs image, the other partitions are not expected to be
> > rebuilt after the first rollout.
> > 
> 

-- 
--
Regards,
Ed


^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH.
  2017-05-22  8:46   ` Ed Bartosh
@ 2017-05-22  9:38     ` Kristian Amlie
  2017-05-30  7:46       ` Kristian Amlie
                         ` (5 more replies)
  0 siblings, 6 replies; 41+ messages in thread
From: Kristian Amlie @ 2017-05-22  9:38 UTC (permalink / raw)
  To: ed.bartosh; +Cc: openembedded-core

On 22/05/17 10:46, Ed Bartosh wrote:
> On Mon, May 22, 2017 at 09:08:41AM +0200, Kristian Amlie wrote:
>> Ping. Would be great to get some feedback so I can continue working on this.
>>
> I like the idea.
> 
> Lately I was working on generic EFI implementation
> and thought that it would be nice to be able to package EFI artifacts
> and get them installed the same way as other packages. With
> IMAGE_ROOTFS_EXCLUDE_PATH = '<directory with efi artifacts>' it would be
> possible to not include that directory into the rootfs if needed.

Yep, great example, that's exactly what this feature is about!

I'll continue with my current suggestion then. The thing I'm most unsure
about is the "d.setVarFlag('do_image_%s' % t, 'respect_exclude_path',
'1')" approach, but I will attempt that one if I don't hear any other
suggestions or hit a road block.

-- 
Kristian


>> On 26/04/17 17:03, Kristian Amlie wrote:
>>> This is a direct followup from the earlier f6a064d969f414 commit in wic. It
>>> works more or less the same way: The variable specifies a list of directories
>>> relative to the root of the rootfs, and these directories will be excluded from
>>> the resulting rootfs image. If an entry ends with a slash, only the contents are
>>> omitted, not the directory itself.
>>>
>>> This is early feedback call, and several things are missing here (see
>>> below). However, I would like to know if at least the basic approach is
>>> acceptable. Reviewers pressed on time really only need to look at the topmost
>>> hunk as well as the last couple of lines of the two functions. Those show what
>>> the basic idea is.
>>>
>>> What's missing:
>>>
>>> * Documentation
>>>
>>> * Test coverage
>>>
>>> * Respecting IMAGE_ROOTFS_EXCLUDE_PATH is a behavior change, and isn't
>>>   appropriate for all image creators. In fact, for the the main use case [1] to
>>>   work it must not always be respected.
>>>
>>> My idea for the last point is that the variable will not be respected by
>>> default, and each of the pure filesystem image creators (btrfs, ext2/3/4, etc)
>>> will set a flag that mark their image creator as a "pure filesystem", which
>>> would respect this variable. Image creators that make partitions, however,
>>> should not respect it, because they will use wic for that purpose instead, and
>>> then they need to have access to the entire rootfs. Hence they should not set
>>> the flag.
>>>
>>> I'm imagining something like this:
>>>
>>>   d.setVarFlag('do_image_%s' % t, 'respect_exclude_path', '1')
>>>
>>> although I'm not sure if that's the best approach (tips are welcome).
>>>
>>> [1] The use case for this feature is the following: We are creating two types of
>>> images in Mender: One complete partitioned image, and one image containing only
>>> the rootfs. In order to be able to populate non-rootfs partitions, we want to
>>> use the regular bitbake recipes for building these components, but then exclude
>>> certain directories from the rootfs image. Using wic, we are adding those
>>> directories back into the partitioned image, but under different
>>> partitions. Unlike the rootfs image, the other partitions are not expected to be
>>> rebuilt after the first rollout.
>>>
>>
> 



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH.
  2017-05-22  9:38     ` Kristian Amlie
@ 2017-05-30  7:46       ` Kristian Amlie
  2017-05-30  7:46         ` [PATCH v2] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature Kristian Amlie
  2017-05-30  8:01       ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev2) Patchwork
                         ` (4 subsequent siblings)
  5 siblings, 1 reply; 41+ messages in thread
From: Kristian Amlie @ 2017-05-30  7:46 UTC (permalink / raw)
  To: openembedded-core


Updated the patch and included documentation.

It wasn't clear to me how to write a test for this. I couldn't spot any tests
for the surrounding variables, but I might be missing something. If this is
important, please let me know where I should add them.

-- 
Kristian


^ permalink raw reply	[flat|nested] 41+ messages in thread

* [PATCH v2] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-05-30  7:46       ` Kristian Amlie
@ 2017-05-30  7:46         ` Kristian Amlie
  2017-06-19  9:01           ` Kristian Amlie
  0 siblings, 1 reply; 41+ messages in thread
From: Kristian Amlie @ 2017-05-30  7:46 UTC (permalink / raw)
  To: openembedded-core

This is a direct followup from the earlier f6a064d969f414 commit in
wic. It works more or less the same way: The variable specifies a list
of directories relative to the root of the rootfs, and these
directories will be excluded from the resulting rootfs image. If an
entry ends with a slash, only the contents are omitted, not the
directory itself.

Since the intended use of the variable is to exclude certain
directories from the rootfs, and then include said directories in
other partitions, it is not natural for this variable to be respected
for image creators that create multi partition images. Therefore the
default is to ignore the variable, and image creators that create
single root filesystems need to set

  do_image_myfs[respect_exclude_path] = "1"

in order to honor it. Specifically, "wic" and "multiubi" have not
received this variable flag, while others have.

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
---
 documentation/ref-manual/ref-variables.xml | 35 +++++++++++++
 meta/classes/image.bbclass                 | 83 ++++++++++++++++++++++++++++--
 meta/classes/image_types.bbclass           | 13 +++++
 3 files changed, 128 insertions(+), 3 deletions(-)

diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml
index 94558df..27f8b8e 100644
--- a/documentation/ref-manual/ref-variables.xml
+++ b/documentation/ref-manual/ref-variables.xml
@@ -6041,6 +6041,41 @@
             </glossdef>
         </glossentry>
 
+        <glossentry id='var-IMAGE_ROOTFS_EXCLUDE_PATH'><glossterm>IMAGE_ROOTFS_EXCLUDE_PATH</glossterm>
+            <info>
+                IMAGE_ROOTFS_EXCLUDE_PATH[doc] = "Specifies paths to omit from the rootfs."
+            </info>
+            <glossdef>
+                <para role="glossdeffirst">
+<!--                <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> -->
+                    Specifies paths that should be omitted from the root filesystem, separated by
+                    spaces. The paths must be relative to the root of the filesystem (for example
+                    "usr"). Single root filesystem images will not include the path in the
+                    filesystem, except if the path ends with a slash, then the directory will be
+                    included, but without any content.
+                    <literallayout class='monospaced'>
+     # Omit /usr completely from rootfs.
+     IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr</replaceable> ..."
+     # Include the directory /usr in rootfs, but not its content.
+     IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr/</replaceable> ..."
+                    </literallayout>
+                </para>
+                <para>
+                    Image creators that create multi partition images typically do not honor this
+                    variable, and must provide their own method to split the directory
+                    structure. For example, <filename>wic</filename> has
+                    <filename>--exclude-path</filename> and <filename>--rootfs-dir</filename>.
+                </para>
+                <para>
+                    The default for custom image creators is to ignore this variable. Those that
+                    need to honor it must set:
+                    <literallayout class='monospaced'>
+     do_image_<replaceable>custom</replaceable>[respect_exclude_path] = "1"
+                    </literallayout>
+                </para>
+            </glossdef>
+        </glossentry>
+
         <glossentry id='var-IMAGE_ROOTFS_EXTRA_SPACE'><glossterm>IMAGE_ROOTFS_EXTRA_SPACE</glossterm>
             <info>
                 IMAGE_ROOTFS_EXTRA_SPACE[doc] = "Defines additional free disk space created in the image in Kbytes. By default, this variable is set to '0'."
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 85f6b9a..c53e383 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -118,7 +118,7 @@ def rootfs_variables(d):
                  'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
                  'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
                  'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
-                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY']
+                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'IMAGE_ROOTFS_EXCLUDE_PATH']
     variables.extend(rootfs_command_variables(d))
     variables.extend(variable_depends(d))
     return " ".join(variables)
@@ -496,8 +496,8 @@ python () {
         d.setVar('do_image_%s' % t, '\n'.join(cmds))
         d.setVarFlag('do_image_%s' % t, 'func', '1')
         d.setVarFlag('do_image_%s' % t, 'fakeroot', '1')
-        d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size')
-        d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks')
+        d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size prepare_excluded_directories')
+        d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks cleanup_excluded_directories')
         d.setVarFlag('do_image_%s' % t, 'subimages', ' '.join(subimages))
         d.appendVarFlag('do_image_%s' % t, 'vardeps', ' '.join(vardeps))
         d.appendVarFlag('do_image_%s' % t, 'vardepsexclude', 'DATETIME')
@@ -506,6 +506,83 @@ python () {
         bb.build.addtask('do_image_%s' % t, 'do_image_complete', after, d)
 }
 
+python prepare_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    taskname = d.getVar("BB_CURRENTTASK")
+
+    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1':
+        return
+
+    import shutil
+    from oe.path import copyhardlinktree
+
+    exclude_list = exclude_var.split()
+
+    rootfs_orig = d.getVar('IMAGE_ROOTFS')
+    # We need a new rootfs directory we can delete files from. Copy to
+    # workdir.
+    new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname))
+
+    if os.path.lexists(new_rootfs):
+        shutil.rmtree(os.path.join(new_rootfs))
+
+    copyhardlinktree(rootfs_orig, new_rootfs)
+
+    for orig_path in exclude_list:
+        path = orig_path
+        if os.path.isabs(path):
+            bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path)
+
+        full_path = os.path.realpath(os.path.join(new_rootfs, path))
+
+        # Disallow climbing outside of parent directory using '..',
+        # because doing so could be quite disastrous (we will delete the
+        # directory).
+        if not full_path.startswith(new_rootfs):
+            bb.fatal("'%s' points to a path outside the rootfs" % orig_path)
+
+        if path.endswith(os.sep):
+            # Delete content only.
+            for entry in os.listdir(full_path):
+                full_entry = os.path.join(full_path, entry)
+                if os.path.isdir(full_entry) and not os.path.islink(full_entry):
+                    shutil.rmtree(full_entry)
+                else:
+                    os.remove(full_entry)
+        else:
+            # Delete whole directory.
+            shutil.rmtree(full_path)
+
+    # Save old value for cleanup later.
+    d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig)
+    d.setVar('IMAGE_ROOTFS', new_rootfs)
+}
+
+python cleanup_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    taskname = d.getVar("BB_CURRENTTASK")
+
+    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1':
+        return
+
+    import shutil
+
+    rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS')
+    rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG')
+    # This should never happen, since we should have set it to a different
+    # directory in the prepare function.
+    assert rootfs_dirs_excluded != rootfs_orig
+
+    shutil.rmtree(rootfs_dirs_excluded)
+    d.setVar('IMAGE_ROOTFS', rootfs_orig)
+}
+
 #
 # Compute the rootfs size
 #
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 7749b00..b27dfe3 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -51,8 +51,10 @@ ZIP_COMPRESSION_LEVEL ?= "-9"
 
 JFFS2_SUM_EXTRA_ARGS ?= ""
 IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 ${EXTRA_IMAGECMD}"
+do_image_jffs2[respect_exclude_path] = "1"
 
 IMAGE_CMD_cramfs = "mkfs.cramfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cramfs ${EXTRA_IMAGECMD}"
+do_image_cramfs[respect_exclude_path] = "1"
 
 oe_mkext234fs () {
 	fstype=$1
@@ -81,6 +83,9 @@ oe_mkext234fs () {
 IMAGE_CMD_ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}"
 IMAGE_CMD_ext3 = "oe_mkext234fs ext3 ${EXTRA_IMAGECMD}"
 IMAGE_CMD_ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}"
+do_image_ext2[respect_exclude_path] = "1"
+do_image_ext3[respect_exclude_path] = "1"
+do_image_ext4[respect_exclude_path] = "1"
 
 MIN_BTRFS_SIZE ?= "16384"
 IMAGE_CMD_btrfs () {
@@ -92,10 +97,14 @@ IMAGE_CMD_btrfs () {
 	dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs count=${size} bs=1024
 	mkfs.btrfs ${EXTRA_IMAGECMD} -r ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs
 }
+do_image_btrfs[respect_exclude_path] = "1"
 
 IMAGE_CMD_squashfs = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs ${EXTRA_IMAGECMD} -noappend"
 IMAGE_CMD_squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz"
 IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo"
+do_image_squashfs[respect_exclude_path] = "1"
+do_image_squashfs-xz[respect_exclude_path] = "1"
+do_image_squashfs-lzo[respect_exclude_path] = "1"
 
 # By default, tar from the host is used, which can be quite old. If
 # you need special parameters (like --xattrs) which are only supported
@@ -109,6 +118,7 @@ IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAM
 # required when extracting, but it seems prudent to use it in both cases.
 IMAGE_CMD_TAR ?= "tar"
 IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cvf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} ."
+do_image_tar[respect_exclude_path] = "1"
 
 do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
 IMAGE_CMD_cpio () {
@@ -129,6 +139,7 @@ IMAGE_CMD_cpio () {
 		fi
 	fi
 }
+do_image_cpio[respect_exclude_path] = "1"
 
 ELF_KERNEL ?= "${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE}"
 ELF_APPEND ?= "ramdisk_size=32768 root=/dev/ram0 rw console="
@@ -137,6 +148,7 @@ IMAGE_CMD_elf () {
 	test -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf && rm -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf
 	mkelfImage --kernel=${ELF_KERNEL} --initrd=${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.cpio.gz --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf --append='${ELF_APPEND}' ${EXTRA_IMAGECMD}
 }
+do_image_elf[respect_exclude_path] = "1"
 
 IMAGE_TYPEDEP_elf = "cpio.gz"
 
@@ -194,6 +206,7 @@ IMAGE_CMD_ubi () {
 }
 
 IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ubifs ${MKUBIFS_ARGS}"
+do_image_ubifs[respect_exclude_path] = "1"
 
 EXTRA_IMAGECMD = ""
 
-- 
2.7.4



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev2)
  2017-05-22  9:38     ` Kristian Amlie
  2017-05-30  7:46       ` Kristian Amlie
@ 2017-05-30  8:01       ` Patchwork
  2017-06-19 14:31       ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev3) Patchwork
                         ` (3 subsequent siblings)
  5 siblings, 0 replies; 41+ messages in thread
From: Patchwork @ 2017-05-30  8:01 UTC (permalink / raw)
  To: Kristian Amlie; +Cc: openembedded-core

== Series Details ==

Series: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev2)
Revision: 2
URL   : https://patchwork.openembedded.org/series/6525/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at 598e5da5a2)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [PATCH v2] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-05-30  7:46         ` [PATCH v2] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature Kristian Amlie
@ 2017-06-19  9:01           ` Kristian Amlie
  2017-06-19 14:01             ` Kristian Amlie
  0 siblings, 1 reply; 41+ messages in thread
From: Kristian Amlie @ 2017-06-19  9:01 UTC (permalink / raw)
  To: Kristian Amlie, openembedded-core

Ping. I created https://bugzilla.yoctoproject.org/show_bug.cgi?id=11688
to help track this.

-- 
Kristian

On 30/05/17 09:46, Kristian Amlie wrote:
> This is a direct followup from the earlier f6a064d969f414 commit in
> wic. It works more or less the same way: The variable specifies a list
> of directories relative to the root of the rootfs, and these
> directories will be excluded from the resulting rootfs image. If an
> entry ends with a slash, only the contents are omitted, not the
> directory itself.
> 
> Since the intended use of the variable is to exclude certain
> directories from the rootfs, and then include said directories in
> other partitions, it is not natural for this variable to be respected
> for image creators that create multi partition images. Therefore the
> default is to ignore the variable, and image creators that create
> single root filesystems need to set
> 
>   do_image_myfs[respect_exclude_path] = "1"
> 
> in order to honor it. Specifically, "wic" and "multiubi" have not
> received this variable flag, while others have.
> 
> Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
> ---
>  documentation/ref-manual/ref-variables.xml | 35 +++++++++++++
>  meta/classes/image.bbclass                 | 83 ++++++++++++++++++++++++++++--
>  meta/classes/image_types.bbclass           | 13 +++++
>  3 files changed, 128 insertions(+), 3 deletions(-)
> 
> diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml
> index 94558df..27f8b8e 100644
> --- a/documentation/ref-manual/ref-variables.xml
> +++ b/documentation/ref-manual/ref-variables.xml
> @@ -6041,6 +6041,41 @@
>              </glossdef>
>          </glossentry>
>  
> +        <glossentry id='var-IMAGE_ROOTFS_EXCLUDE_PATH'><glossterm>IMAGE_ROOTFS_EXCLUDE_PATH</glossterm>
> +            <info>
> +                IMAGE_ROOTFS_EXCLUDE_PATH[doc] = "Specifies paths to omit from the rootfs."
> +            </info>
> +            <glossdef>
> +                <para role="glossdeffirst">
> +<!--                <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> -->
> +                    Specifies paths that should be omitted from the root filesystem, separated by
> +                    spaces. The paths must be relative to the root of the filesystem (for example
> +                    "usr"). Single root filesystem images will not include the path in the
> +                    filesystem, except if the path ends with a slash, then the directory will be
> +                    included, but without any content.
> +                    <literallayout class='monospaced'>
> +     # Omit /usr completely from rootfs.
> +     IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr</replaceable> ..."
> +     # Include the directory /usr in rootfs, but not its content.
> +     IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr/</replaceable> ..."
> +                    </literallayout>
> +                </para>
> +                <para>
> +                    Image creators that create multi partition images typically do not honor this
> +                    variable, and must provide their own method to split the directory
> +                    structure. For example, <filename>wic</filename> has
> +                    <filename>--exclude-path</filename> and <filename>--rootfs-dir</filename>.
> +                </para>
> +                <para>
> +                    The default for custom image creators is to ignore this variable. Those that
> +                    need to honor it must set:
> +                    <literallayout class='monospaced'>
> +     do_image_<replaceable>custom</replaceable>[respect_exclude_path] = "1"
> +                    </literallayout>
> +                </para>
> +            </glossdef>
> +        </glossentry>
> +
>          <glossentry id='var-IMAGE_ROOTFS_EXTRA_SPACE'><glossterm>IMAGE_ROOTFS_EXTRA_SPACE</glossterm>
>              <info>
>                  IMAGE_ROOTFS_EXTRA_SPACE[doc] = "Defines additional free disk space created in the image in Kbytes. By default, this variable is set to '0'."
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 85f6b9a..c53e383 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -118,7 +118,7 @@ def rootfs_variables(d):
>                   'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
>                   'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
>                   'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
> -                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY']
> +                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'IMAGE_ROOTFS_EXCLUDE_PATH']
>      variables.extend(rootfs_command_variables(d))
>      variables.extend(variable_depends(d))
>      return " ".join(variables)
> @@ -496,8 +496,8 @@ python () {
>          d.setVar('do_image_%s' % t, '\n'.join(cmds))
>          d.setVarFlag('do_image_%s' % t, 'func', '1')
>          d.setVarFlag('do_image_%s' % t, 'fakeroot', '1')
> -        d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size')
> -        d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks')
> +        d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size prepare_excluded_directories')
> +        d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks cleanup_excluded_directories')
>          d.setVarFlag('do_image_%s' % t, 'subimages', ' '.join(subimages))
>          d.appendVarFlag('do_image_%s' % t, 'vardeps', ' '.join(vardeps))
>          d.appendVarFlag('do_image_%s' % t, 'vardepsexclude', 'DATETIME')
> @@ -506,6 +506,83 @@ python () {
>          bb.build.addtask('do_image_%s' % t, 'do_image_complete', after, d)
>  }
>  
> +python prepare_excluded_directories() {
> +    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
> +    if not exclude_var:
> +        return
> +
> +    taskname = d.getVar("BB_CURRENTTASK")
> +
> +    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1':
> +        return
> +
> +    import shutil
> +    from oe.path import copyhardlinktree
> +
> +    exclude_list = exclude_var.split()
> +
> +    rootfs_orig = d.getVar('IMAGE_ROOTFS')
> +    # We need a new rootfs directory we can delete files from. Copy to
> +    # workdir.
> +    new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname))
> +
> +    if os.path.lexists(new_rootfs):
> +        shutil.rmtree(os.path.join(new_rootfs))
> +
> +    copyhardlinktree(rootfs_orig, new_rootfs)
> +
> +    for orig_path in exclude_list:
> +        path = orig_path
> +        if os.path.isabs(path):
> +            bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path)
> +
> +        full_path = os.path.realpath(os.path.join(new_rootfs, path))
> +
> +        # Disallow climbing outside of parent directory using '..',
> +        # because doing so could be quite disastrous (we will delete the
> +        # directory).
> +        if not full_path.startswith(new_rootfs):
> +            bb.fatal("'%s' points to a path outside the rootfs" % orig_path)
> +
> +        if path.endswith(os.sep):
> +            # Delete content only.
> +            for entry in os.listdir(full_path):
> +                full_entry = os.path.join(full_path, entry)
> +                if os.path.isdir(full_entry) and not os.path.islink(full_entry):
> +                    shutil.rmtree(full_entry)
> +                else:
> +                    os.remove(full_entry)
> +        else:
> +            # Delete whole directory.
> +            shutil.rmtree(full_path)
> +
> +    # Save old value for cleanup later.
> +    d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig)
> +    d.setVar('IMAGE_ROOTFS', new_rootfs)
> +}
> +
> +python cleanup_excluded_directories() {
> +    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
> +    if not exclude_var:
> +        return
> +
> +    taskname = d.getVar("BB_CURRENTTASK")
> +
> +    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1':
> +        return
> +
> +    import shutil
> +
> +    rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS')
> +    rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG')
> +    # This should never happen, since we should have set it to a different
> +    # directory in the prepare function.
> +    assert rootfs_dirs_excluded != rootfs_orig
> +
> +    shutil.rmtree(rootfs_dirs_excluded)
> +    d.setVar('IMAGE_ROOTFS', rootfs_orig)
> +}
> +
>  #
>  # Compute the rootfs size
>  #
> diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
> index 7749b00..b27dfe3 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -51,8 +51,10 @@ ZIP_COMPRESSION_LEVEL ?= "-9"
>  
>  JFFS2_SUM_EXTRA_ARGS ?= ""
>  IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 ${EXTRA_IMAGECMD}"
> +do_image_jffs2[respect_exclude_path] = "1"
>  
>  IMAGE_CMD_cramfs = "mkfs.cramfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cramfs ${EXTRA_IMAGECMD}"
> +do_image_cramfs[respect_exclude_path] = "1"
>  
>  oe_mkext234fs () {
>  	fstype=$1
> @@ -81,6 +83,9 @@ oe_mkext234fs () {
>  IMAGE_CMD_ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}"
>  IMAGE_CMD_ext3 = "oe_mkext234fs ext3 ${EXTRA_IMAGECMD}"
>  IMAGE_CMD_ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}"
> +do_image_ext2[respect_exclude_path] = "1"
> +do_image_ext3[respect_exclude_path] = "1"
> +do_image_ext4[respect_exclude_path] = "1"
>  
>  MIN_BTRFS_SIZE ?= "16384"
>  IMAGE_CMD_btrfs () {
> @@ -92,10 +97,14 @@ IMAGE_CMD_btrfs () {
>  	dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs count=${size} bs=1024
>  	mkfs.btrfs ${EXTRA_IMAGECMD} -r ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs
>  }
> +do_image_btrfs[respect_exclude_path] = "1"
>  
>  IMAGE_CMD_squashfs = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs ${EXTRA_IMAGECMD} -noappend"
>  IMAGE_CMD_squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz"
>  IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo"
> +do_image_squashfs[respect_exclude_path] = "1"
> +do_image_squashfs-xz[respect_exclude_path] = "1"
> +do_image_squashfs-lzo[respect_exclude_path] = "1"
>  
>  # By default, tar from the host is used, which can be quite old. If
>  # you need special parameters (like --xattrs) which are only supported
> @@ -109,6 +118,7 @@ IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAM
>  # required when extracting, but it seems prudent to use it in both cases.
>  IMAGE_CMD_TAR ?= "tar"
>  IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cvf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} ."
> +do_image_tar[respect_exclude_path] = "1"
>  
>  do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
>  IMAGE_CMD_cpio () {
> @@ -129,6 +139,7 @@ IMAGE_CMD_cpio () {
>  		fi
>  	fi
>  }
> +do_image_cpio[respect_exclude_path] = "1"
>  
>  ELF_KERNEL ?= "${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE}"
>  ELF_APPEND ?= "ramdisk_size=32768 root=/dev/ram0 rw console="
> @@ -137,6 +148,7 @@ IMAGE_CMD_elf () {
>  	test -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf && rm -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf
>  	mkelfImage --kernel=${ELF_KERNEL} --initrd=${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.cpio.gz --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf --append='${ELF_APPEND}' ${EXTRA_IMAGECMD}
>  }
> +do_image_elf[respect_exclude_path] = "1"
>  
>  IMAGE_TYPEDEP_elf = "cpio.gz"
>  
> @@ -194,6 +206,7 @@ IMAGE_CMD_ubi () {
>  }
>  
>  IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ubifs ${MKUBIFS_ARGS}"
> +do_image_ubifs[respect_exclude_path] = "1"
>  
>  EXTRA_IMAGECMD = ""
>  
> 



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [PATCH v2] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-06-19  9:01           ` Kristian Amlie
@ 2017-06-19 14:01             ` Kristian Amlie
  2017-06-19 14:01               ` [PATCH v3] " Kristian Amlie
  0 siblings, 1 reply; 41+ messages in thread
From: Kristian Amlie @ 2017-06-19 14:01 UTC (permalink / raw)
  To: openembedded-core

Turns out this needed a rebase. Here you go.

-- 
Kristian


^ permalink raw reply	[flat|nested] 41+ messages in thread

* [PATCH v3] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-06-19 14:01             ` Kristian Amlie
@ 2017-06-19 14:01               ` Kristian Amlie
  2017-08-23 11:44                 ` [PATCH v4] " Kristian Amlie
  2017-08-23 11:46                 ` Kristian Amlie
  0 siblings, 2 replies; 41+ messages in thread
From: Kristian Amlie @ 2017-06-19 14:01 UTC (permalink / raw)
  To: openembedded-core

This is a direct followup from the earlier f6a064d969f414 commit in
wic. It works more or less the same way: The variable specifies a list
of directories relative to the root of the rootfs, and these
directories will be excluded from the resulting rootfs image. If an
entry ends with a slash, only the contents are omitted, not the
directory itself.

Since the intended use of the variable is to exclude certain
directories from the rootfs, and then include said directories in
other partitions, it is not natural for this variable to be respected
for image creators that create multi partition images. Therefore the
default is to ignore the variable, and image creators that create
single root filesystems need to set

  do_image_myfs[respect_exclude_path] = "1"

in order to honor it. Specifically, "wic" and "multiubi" have not
received this variable flag, while others have.

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
---
 documentation/ref-manual/ref-variables.xml | 35 +++++++++++++
 meta/classes/image.bbclass                 | 83 ++++++++++++++++++++++++++++--
 meta/classes/image_types.bbclass           | 13 +++++
 3 files changed, 128 insertions(+), 3 deletions(-)

diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml
index bd9e517..8456445 100644
--- a/documentation/ref-manual/ref-variables.xml
+++ b/documentation/ref-manual/ref-variables.xml
@@ -6041,6 +6041,41 @@
             </glossdef>
         </glossentry>
 
+        <glossentry id='var-IMAGE_ROOTFS_EXCLUDE_PATH'><glossterm>IMAGE_ROOTFS_EXCLUDE_PATH</glossterm>
+            <info>
+                IMAGE_ROOTFS_EXCLUDE_PATH[doc] = "Specifies paths to omit from the rootfs."
+            </info>
+            <glossdef>
+                <para role="glossdeffirst">
+<!--                <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> -->
+                    Specifies paths that should be omitted from the root filesystem, separated by
+                    spaces. The paths must be relative to the root of the filesystem (for example
+                    "usr"). Single root filesystem images will not include the path in the
+                    filesystem, except if the path ends with a slash, then the directory will be
+                    included, but without any content.
+                    <literallayout class='monospaced'>
+     # Omit /usr completely from rootfs.
+     IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr</replaceable> ..."
+     # Include the directory /usr in rootfs, but not its content.
+     IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr/</replaceable> ..."
+                    </literallayout>
+                </para>
+                <para>
+                    Image creators that create multi partition images typically do not honor this
+                    variable, and must provide their own method to split the directory
+                    structure. For example, <filename>wic</filename> has
+                    <filename>--exclude-path</filename> and <filename>--rootfs-dir</filename>.
+                </para>
+                <para>
+                    The default for custom image creators is to ignore this variable. Those that
+                    need to honor it must set:
+                    <literallayout class='monospaced'>
+     do_image_<replaceable>custom</replaceable>[respect_exclude_path] = "1"
+                    </literallayout>
+                </para>
+            </glossdef>
+        </glossentry>
+
         <glossentry id='var-IMAGE_ROOTFS_EXTRA_SPACE'><glossterm>IMAGE_ROOTFS_EXTRA_SPACE</glossterm>
             <info>
                 IMAGE_ROOTFS_EXTRA_SPACE[doc] = "Defines additional free disk space created in the image in Kbytes. By default, this variable is set to '0'."
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 9a04b43..e4c97b1 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -118,7 +118,7 @@ def rootfs_variables(d):
                  'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
                  'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
                  'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
-                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY']
+                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'IMAGE_ROOTFS_EXCLUDE_PATH']
     variables.extend(rootfs_command_variables(d))
     variables.extend(variable_depends(d))
     return " ".join(variables)
@@ -497,8 +497,8 @@ python () {
         d.setVarFlag(task, 'func', '1')
         d.setVarFlag(task, 'fakeroot', '1')
 
-        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size')
-        d.appendVarFlag(task, 'postfuncs', ' create_symlinks')
+        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories')
+        d.appendVarFlag(task, 'postfuncs', ' create_symlinks cleanup_excluded_directories')
         d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages))
         d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps))
         d.appendVarFlag(task, 'vardepsexclude', 'DATETIME')
@@ -507,6 +507,83 @@ python () {
         bb.build.addtask(task, 'do_image_complete', after, d)
 }
 
+python prepare_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    taskname = d.getVar("BB_CURRENTTASK")
+
+    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1':
+        return
+
+    import shutil
+    from oe.path import copyhardlinktree
+
+    exclude_list = exclude_var.split()
+
+    rootfs_orig = d.getVar('IMAGE_ROOTFS')
+    # We need a new rootfs directory we can delete files from. Copy to
+    # workdir.
+    new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname))
+
+    if os.path.lexists(new_rootfs):
+        shutil.rmtree(os.path.join(new_rootfs))
+
+    copyhardlinktree(rootfs_orig, new_rootfs)
+
+    for orig_path in exclude_list:
+        path = orig_path
+        if os.path.isabs(path):
+            bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path)
+
+        full_path = os.path.realpath(os.path.join(new_rootfs, path))
+
+        # Disallow climbing outside of parent directory using '..',
+        # because doing so could be quite disastrous (we will delete the
+        # directory).
+        if not full_path.startswith(new_rootfs):
+            bb.fatal("'%s' points to a path outside the rootfs" % orig_path)
+
+        if path.endswith(os.sep):
+            # Delete content only.
+            for entry in os.listdir(full_path):
+                full_entry = os.path.join(full_path, entry)
+                if os.path.isdir(full_entry) and not os.path.islink(full_entry):
+                    shutil.rmtree(full_entry)
+                else:
+                    os.remove(full_entry)
+        else:
+            # Delete whole directory.
+            shutil.rmtree(full_path)
+
+    # Save old value for cleanup later.
+    d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig)
+    d.setVar('IMAGE_ROOTFS', new_rootfs)
+}
+
+python cleanup_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    taskname = d.getVar("BB_CURRENTTASK")
+
+    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1':
+        return
+
+    import shutil
+
+    rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS')
+    rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG')
+    # This should never happen, since we should have set it to a different
+    # directory in the prepare function.
+    assert rootfs_dirs_excluded != rootfs_orig
+
+    shutil.rmtree(rootfs_dirs_excluded)
+    d.setVar('IMAGE_ROOTFS', rootfs_orig)
+}
+
 #
 # Compute the rootfs size
 #
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 7749b00..b27dfe3 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -51,8 +51,10 @@ ZIP_COMPRESSION_LEVEL ?= "-9"
 
 JFFS2_SUM_EXTRA_ARGS ?= ""
 IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 ${EXTRA_IMAGECMD}"
+do_image_jffs2[respect_exclude_path] = "1"
 
 IMAGE_CMD_cramfs = "mkfs.cramfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cramfs ${EXTRA_IMAGECMD}"
+do_image_cramfs[respect_exclude_path] = "1"
 
 oe_mkext234fs () {
 	fstype=$1
@@ -81,6 +83,9 @@ oe_mkext234fs () {
 IMAGE_CMD_ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}"
 IMAGE_CMD_ext3 = "oe_mkext234fs ext3 ${EXTRA_IMAGECMD}"
 IMAGE_CMD_ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}"
+do_image_ext2[respect_exclude_path] = "1"
+do_image_ext3[respect_exclude_path] = "1"
+do_image_ext4[respect_exclude_path] = "1"
 
 MIN_BTRFS_SIZE ?= "16384"
 IMAGE_CMD_btrfs () {
@@ -92,10 +97,14 @@ IMAGE_CMD_btrfs () {
 	dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs count=${size} bs=1024
 	mkfs.btrfs ${EXTRA_IMAGECMD} -r ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs
 }
+do_image_btrfs[respect_exclude_path] = "1"
 
 IMAGE_CMD_squashfs = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs ${EXTRA_IMAGECMD} -noappend"
 IMAGE_CMD_squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz"
 IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo"
+do_image_squashfs[respect_exclude_path] = "1"
+do_image_squashfs-xz[respect_exclude_path] = "1"
+do_image_squashfs-lzo[respect_exclude_path] = "1"
 
 # By default, tar from the host is used, which can be quite old. If
 # you need special parameters (like --xattrs) which are only supported
@@ -109,6 +118,7 @@ IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAM
 # required when extracting, but it seems prudent to use it in both cases.
 IMAGE_CMD_TAR ?= "tar"
 IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cvf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} ."
+do_image_tar[respect_exclude_path] = "1"
 
 do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
 IMAGE_CMD_cpio () {
@@ -129,6 +139,7 @@ IMAGE_CMD_cpio () {
 		fi
 	fi
 }
+do_image_cpio[respect_exclude_path] = "1"
 
 ELF_KERNEL ?= "${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE}"
 ELF_APPEND ?= "ramdisk_size=32768 root=/dev/ram0 rw console="
@@ -137,6 +148,7 @@ IMAGE_CMD_elf () {
 	test -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf && rm -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf
 	mkelfImage --kernel=${ELF_KERNEL} --initrd=${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.cpio.gz --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf --append='${ELF_APPEND}' ${EXTRA_IMAGECMD}
 }
+do_image_elf[respect_exclude_path] = "1"
 
 IMAGE_TYPEDEP_elf = "cpio.gz"
 
@@ -194,6 +206,7 @@ IMAGE_CMD_ubi () {
 }
 
 IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ubifs ${MKUBIFS_ARGS}"
+do_image_ubifs[respect_exclude_path] = "1"
 
 EXTRA_IMAGECMD = ""
 
-- 
2.7.4



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev3)
  2017-05-22  9:38     ` Kristian Amlie
  2017-05-30  7:46       ` Kristian Amlie
  2017-05-30  8:01       ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev2) Patchwork
@ 2017-06-19 14:31       ` Patchwork
  2017-08-23 12:04       ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev4) Patchwork
                         ` (2 subsequent siblings)
  5 siblings, 0 replies; 41+ messages in thread
From: Patchwork @ 2017-06-19 14:31 UTC (permalink / raw)
  To: Kristian Amlie; +Cc: openembedded-core

== Series Details ==

Series: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev3)
Revision: 3
URL   : https://patchwork.openembedded.org/series/6525/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at 1d9a88f635)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe



^ permalink raw reply	[flat|nested] 41+ messages in thread

* [PATCH v4] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-06-19 14:01               ` [PATCH v3] " Kristian Amlie
@ 2017-08-23 11:44                 ` Kristian Amlie
  2017-08-23 11:46                 ` Kristian Amlie
  1 sibling, 0 replies; 41+ messages in thread
From: Kristian Amlie @ 2017-08-23 11:44 UTC (permalink / raw)
  To: openembedded-core

From: Kristian Amlie <kristian.amlie@mender.io>

This is a direct followup from the earlier f6a064d969f414 commit in
wic. It works more or less the same way: The variable specifies a list
of directories relative to the root of the rootfs, and these
directories will be excluded from the resulting rootfs image. If an
entry ends with a slash, only the contents are omitted, not the
directory itself.

Since the intended use of the variable is to exclude certain
directories from the rootfs, and then include said directories in
other partitions, it is not natural for this variable to be respected
for image creators that create multi partition images. Therefore the
default is to ignore the variable, and image creators that create
single root filesystems need to set

  do_image_myfs[respect_exclude_path] = "1"

in order to honor it. Specifically, "wic" and "multiubi" have not
received this variable flag, while others have.

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
---
 documentation/ref-manual/ref-variables.xml | 35 +++++++++++++
 meta/classes/image.bbclass                 | 83 +++++++++++++++++++++++++++++-
 meta/classes/image_types.bbclass           | 13 +++++
 3 files changed, 129 insertions(+), 2 deletions(-)

diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml
index ff8f4e7..a4f2de5 100644
--- a/documentation/ref-manual/ref-variables.xml
+++ b/documentation/ref-manual/ref-variables.xml
@@ -6043,6 +6043,41 @@
             </glossdef>
         </glossentry>
 
+        <glossentry id='var-IMAGE_ROOTFS_EXCLUDE_PATH'><glossterm>IMAGE_ROOTFS_EXCLUDE_PATH</glossterm>
+            <info>
+                IMAGE_ROOTFS_EXCLUDE_PATH[doc] = "Specifies paths to omit from the rootfs."
+            </info>
+            <glossdef>
+                <para role="glossdeffirst">
+<!--                <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> -->
+                    Specifies paths that should be omitted from the root filesystem, separated by
+                    spaces. The paths must be relative to the root of the filesystem (for example
+                    "usr"). Single root filesystem images will not include the path in the
+                    filesystem, except if the path ends with a slash, then the directory will be
+                    included, but without any content.
+                    <literallayout class='monospaced'>
+     # Omit /usr completely from rootfs.
+     IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr</replaceable> ..."
+     # Include the directory /usr in rootfs, but not its content.
+     IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr/</replaceable> ..."
+                    </literallayout>
+                </para>
+                <para>
+                    Image creators that create multi partition images typically do not honor this
+                    variable, and must provide their own method to split the directory
+                    structure. For example, <filename>wic</filename> has
+                    <filename>--exclude-path</filename> and <filename>--rootfs-dir</filename>.
+                </para>
+                <para>
+                    The default for custom image creators is to ignore this variable. Those that
+                    need to honor it must set:
+                    <literallayout class='monospaced'>
+     do_image_<replaceable>custom</replaceable>[respect_exclude_path] = "1"
+                    </literallayout>
+                </para>
+            </glossdef>
+        </glossentry>
+
         <glossentry id='var-IMAGE_ROOTFS_EXTRA_SPACE'><glossterm>IMAGE_ROOTFS_EXTRA_SPACE</glossterm>
             <info>
                 IMAGE_ROOTFS_EXTRA_SPACE[doc] = "Defines additional free disk space created in the image in Kbytes. By default, this variable is set to '0'."
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 3639aa4..e215541 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -117,7 +117,8 @@ def rootfs_variables(d):
                  'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
                  'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
                  'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
-                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS']
+                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS',
+                 'IMAGE_ROOTFS_EXCLUDE_PATH']
     variables.extend(rootfs_command_variables(d))
     variables.extend(variable_depends(d))
     return " ".join(variables)
@@ -496,8 +497,9 @@ python () {
         d.setVarFlag(task, 'func', '1')
         d.setVarFlag(task, 'fakeroot', '1')
 
-        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size')
+        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories')
         d.prependVarFlag(task, 'postfuncs', ' create_symlinks')
+        d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories')
         d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages))
         d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps))
         d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE')
@@ -506,6 +508,83 @@ python () {
         bb.build.addtask(task, 'do_image_complete', after, d)
 }
 
+python prepare_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    taskname = d.getVar("BB_CURRENTTASK")
+
+    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1':
+        return
+
+    import shutil
+    from oe.path import copyhardlinktree
+
+    exclude_list = exclude_var.split()
+
+    rootfs_orig = d.getVar('IMAGE_ROOTFS')
+    # We need a new rootfs directory we can delete files from. Copy to
+    # workdir.
+    new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname))
+
+    if os.path.lexists(new_rootfs):
+        shutil.rmtree(os.path.join(new_rootfs))
+
+    copyhardlinktree(rootfs_orig, new_rootfs)
+
+    for orig_path in exclude_list:
+        path = orig_path
+        if os.path.isabs(path):
+            bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path)
+
+        full_path = os.path.realpath(os.path.join(new_rootfs, path))
+
+        # Disallow climbing outside of parent directory using '..',
+        # because doing so could be quite disastrous (we will delete the
+        # directory).
+        if not full_path.startswith(new_rootfs):
+            bb.fatal("'%s' points to a path outside the rootfs" % orig_path)
+
+        if path.endswith(os.sep):
+            # Delete content only.
+            for entry in os.listdir(full_path):
+                full_entry = os.path.join(full_path, entry)
+                if os.path.isdir(full_entry) and not os.path.islink(full_entry):
+                    shutil.rmtree(full_entry)
+                else:
+                    os.remove(full_entry)
+        else:
+            # Delete whole directory.
+            shutil.rmtree(full_path)
+
+    # Save old value for cleanup later.
+    d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig)
+    d.setVar('IMAGE_ROOTFS', new_rootfs)
+}
+
+python cleanup_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    taskname = d.getVar("BB_CURRENTTASK")
+
+    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1':
+        return
+
+    import shutil
+
+    rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS')
+    rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG')
+    # This should never happen, since we should have set it to a different
+    # directory in the prepare function.
+    assert rootfs_dirs_excluded != rootfs_orig
+
+    shutil.rmtree(rootfs_dirs_excluded)
+    d.setVar('IMAGE_ROOTFS', rootfs_orig)
+}
+
 #
 # Compute the rootfs size
 #
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index e0368c7..a063df9 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -62,8 +62,10 @@ ZIP_COMPRESSION_LEVEL ?= "-9"
 
 JFFS2_SUM_EXTRA_ARGS ?= ""
 IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 ${EXTRA_IMAGECMD}"
+do_image_jffs2[respect_exclude_path] = "1"
 
 IMAGE_CMD_cramfs = "mkfs.cramfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cramfs ${EXTRA_IMAGECMD}"
+do_image_cramfs[respect_exclude_path] = "1"
 
 oe_mkext234fs () {
 	fstype=$1
@@ -92,6 +94,9 @@ oe_mkext234fs () {
 IMAGE_CMD_ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}"
 IMAGE_CMD_ext3 = "oe_mkext234fs ext3 ${EXTRA_IMAGECMD}"
 IMAGE_CMD_ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}"
+do_image_ext2[respect_exclude_path] = "1"
+do_image_ext3[respect_exclude_path] = "1"
+do_image_ext4[respect_exclude_path] = "1"
 
 MIN_BTRFS_SIZE ?= "16384"
 IMAGE_CMD_btrfs () {
@@ -103,10 +108,14 @@ IMAGE_CMD_btrfs () {
 	dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs count=${size} bs=1024
 	mkfs.btrfs ${EXTRA_IMAGECMD} -r ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs
 }
+do_image_btrfs[respect_exclude_path] = "1"
 
 IMAGE_CMD_squashfs = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs ${EXTRA_IMAGECMD} -noappend"
 IMAGE_CMD_squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz"
 IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo"
+do_image_squashfs[respect_exclude_path] = "1"
+do_image_squashfs-xz[respect_exclude_path] = "1"
+do_image_squashfs-lzo[respect_exclude_path] = "1"
 
 # By default, tar from the host is used, which can be quite old. If
 # you need special parameters (like --xattrs) which are only supported
@@ -121,6 +130,7 @@ IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAM
 IMAGE_CMD_TAR ?= "tar"
 # ignore return code 1 "file changed as we read it" as other tasks(e.g. do_image_wic) may be hardlinking rootfs
 IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} . || [ $? -eq 1 ]"
+do_image_tar[respect_exclude_path] = "1"
 
 do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
 IMAGE_CMD_cpio () {
@@ -141,6 +151,7 @@ IMAGE_CMD_cpio () {
 		fi
 	fi
 }
+do_image_cpio[respect_exclude_path] = "1"
 
 ELF_KERNEL ?= "${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE}"
 ELF_APPEND ?= "ramdisk_size=32768 root=/dev/ram0 rw console="
@@ -149,6 +160,7 @@ IMAGE_CMD_elf () {
 	test -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf && rm -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf
 	mkelfImage --kernel=${ELF_KERNEL} --initrd=${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.cpio.gz --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf --append='${ELF_APPEND}' ${EXTRA_IMAGECMD}
 }
+do_image_elf[respect_exclude_path] = "1"
 
 IMAGE_TYPEDEP_elf = "cpio.gz"
 
@@ -212,6 +224,7 @@ IMAGE_CMD_ubi () {
 }
 
 IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ubifs ${MKUBIFS_ARGS}"
+do_image_ubifs[respect_exclude_path] = "1"
 
 EXTRA_IMAGECMD = ""
 
-- 
2.7.4



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* Re: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-06-19 14:01               ` [PATCH v3] " Kristian Amlie
  2017-08-23 11:44                 ` [PATCH v4] " Kristian Amlie
@ 2017-08-23 11:46                 ` Kristian Amlie
  2017-08-23 11:46                   ` [PATCH v4] " Kristian Amlie
  1 sibling, 1 reply; 41+ messages in thread
From: Kristian Amlie @ 2017-08-23 11:46 UTC (permalink / raw)
  To: openembedded-core


Ping, third rebase now! This is related to
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11688.

-- 
Kristian


^ permalink raw reply	[flat|nested] 41+ messages in thread

* [PATCH v4] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-08-23 11:46                 ` Kristian Amlie
@ 2017-08-23 11:46                   ` Kristian Amlie
  2017-08-23 12:39                     ` Kristian Amlie
  0 siblings, 1 reply; 41+ messages in thread
From: Kristian Amlie @ 2017-08-23 11:46 UTC (permalink / raw)
  To: openembedded-core

From: Kristian Amlie <kristian.amlie@mender.io>

This is a direct followup from the earlier f6a064d969f414 commit in
wic. It works more or less the same way: The variable specifies a list
of directories relative to the root of the rootfs, and these
directories will be excluded from the resulting rootfs image. If an
entry ends with a slash, only the contents are omitted, not the
directory itself.

Since the intended use of the variable is to exclude certain
directories from the rootfs, and then include said directories in
other partitions, it is not natural for this variable to be respected
for image creators that create multi partition images. Therefore the
default is to ignore the variable, and image creators that create
single root filesystems need to set

  do_image_myfs[respect_exclude_path] = "1"

in order to honor it. Specifically, "wic" and "multiubi" have not
received this variable flag, while others have.

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
---
 documentation/ref-manual/ref-variables.xml | 35 +++++++++++++
 meta/classes/image.bbclass                 | 83 +++++++++++++++++++++++++++++-
 meta/classes/image_types.bbclass           | 13 +++++
 3 files changed, 129 insertions(+), 2 deletions(-)

diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml
index ff8f4e7..a4f2de5 100644
--- a/documentation/ref-manual/ref-variables.xml
+++ b/documentation/ref-manual/ref-variables.xml
@@ -6043,6 +6043,41 @@
             </glossdef>
         </glossentry>
 
+        <glossentry id='var-IMAGE_ROOTFS_EXCLUDE_PATH'><glossterm>IMAGE_ROOTFS_EXCLUDE_PATH</glossterm>
+            <info>
+                IMAGE_ROOTFS_EXCLUDE_PATH[doc] = "Specifies paths to omit from the rootfs."
+            </info>
+            <glossdef>
+                <para role="glossdeffirst">
+<!--                <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> -->
+                    Specifies paths that should be omitted from the root filesystem, separated by
+                    spaces. The paths must be relative to the root of the filesystem (for example
+                    "usr"). Single root filesystem images will not include the path in the
+                    filesystem, except if the path ends with a slash, then the directory will be
+                    included, but without any content.
+                    <literallayout class='monospaced'>
+     # Omit /usr completely from rootfs.
+     IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr</replaceable> ..."
+     # Include the directory /usr in rootfs, but not its content.
+     IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr/</replaceable> ..."
+                    </literallayout>
+                </para>
+                <para>
+                    Image creators that create multi partition images typically do not honor this
+                    variable, and must provide their own method to split the directory
+                    structure. For example, <filename>wic</filename> has
+                    <filename>--exclude-path</filename> and <filename>--rootfs-dir</filename>.
+                </para>
+                <para>
+                    The default for custom image creators is to ignore this variable. Those that
+                    need to honor it must set:
+                    <literallayout class='monospaced'>
+     do_image_<replaceable>custom</replaceable>[respect_exclude_path] = "1"
+                    </literallayout>
+                </para>
+            </glossdef>
+        </glossentry>
+
         <glossentry id='var-IMAGE_ROOTFS_EXTRA_SPACE'><glossterm>IMAGE_ROOTFS_EXTRA_SPACE</glossterm>
             <info>
                 IMAGE_ROOTFS_EXTRA_SPACE[doc] = "Defines additional free disk space created in the image in Kbytes. By default, this variable is set to '0'."
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 3639aa4..e215541 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -117,7 +117,8 @@ def rootfs_variables(d):
                  'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
                  'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
                  'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
-                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS']
+                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS',
+                 'IMAGE_ROOTFS_EXCLUDE_PATH']
     variables.extend(rootfs_command_variables(d))
     variables.extend(variable_depends(d))
     return " ".join(variables)
@@ -496,8 +497,9 @@ python () {
         d.setVarFlag(task, 'func', '1')
         d.setVarFlag(task, 'fakeroot', '1')
 
-        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size')
+        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories')
         d.prependVarFlag(task, 'postfuncs', ' create_symlinks')
+        d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories')
         d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages))
         d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps))
         d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE')
@@ -506,6 +508,83 @@ python () {
         bb.build.addtask(task, 'do_image_complete', after, d)
 }
 
+python prepare_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    taskname = d.getVar("BB_CURRENTTASK")
+
+    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1':
+        return
+
+    import shutil
+    from oe.path import copyhardlinktree
+
+    exclude_list = exclude_var.split()
+
+    rootfs_orig = d.getVar('IMAGE_ROOTFS')
+    # We need a new rootfs directory we can delete files from. Copy to
+    # workdir.
+    new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname))
+
+    if os.path.lexists(new_rootfs):
+        shutil.rmtree(os.path.join(new_rootfs))
+
+    copyhardlinktree(rootfs_orig, new_rootfs)
+
+    for orig_path in exclude_list:
+        path = orig_path
+        if os.path.isabs(path):
+            bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path)
+
+        full_path = os.path.realpath(os.path.join(new_rootfs, path))
+
+        # Disallow climbing outside of parent directory using '..',
+        # because doing so could be quite disastrous (we will delete the
+        # directory).
+        if not full_path.startswith(new_rootfs):
+            bb.fatal("'%s' points to a path outside the rootfs" % orig_path)
+
+        if path.endswith(os.sep):
+            # Delete content only.
+            for entry in os.listdir(full_path):
+                full_entry = os.path.join(full_path, entry)
+                if os.path.isdir(full_entry) and not os.path.islink(full_entry):
+                    shutil.rmtree(full_entry)
+                else:
+                    os.remove(full_entry)
+        else:
+            # Delete whole directory.
+            shutil.rmtree(full_path)
+
+    # Save old value for cleanup later.
+    d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig)
+    d.setVar('IMAGE_ROOTFS', new_rootfs)
+}
+
+python cleanup_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    taskname = d.getVar("BB_CURRENTTASK")
+
+    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1':
+        return
+
+    import shutil
+
+    rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS')
+    rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG')
+    # This should never happen, since we should have set it to a different
+    # directory in the prepare function.
+    assert rootfs_dirs_excluded != rootfs_orig
+
+    shutil.rmtree(rootfs_dirs_excluded)
+    d.setVar('IMAGE_ROOTFS', rootfs_orig)
+}
+
 #
 # Compute the rootfs size
 #
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index e0368c7..a063df9 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -62,8 +62,10 @@ ZIP_COMPRESSION_LEVEL ?= "-9"
 
 JFFS2_SUM_EXTRA_ARGS ?= ""
 IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 ${EXTRA_IMAGECMD}"
+do_image_jffs2[respect_exclude_path] = "1"
 
 IMAGE_CMD_cramfs = "mkfs.cramfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cramfs ${EXTRA_IMAGECMD}"
+do_image_cramfs[respect_exclude_path] = "1"
 
 oe_mkext234fs () {
 	fstype=$1
@@ -92,6 +94,9 @@ oe_mkext234fs () {
 IMAGE_CMD_ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}"
 IMAGE_CMD_ext3 = "oe_mkext234fs ext3 ${EXTRA_IMAGECMD}"
 IMAGE_CMD_ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}"
+do_image_ext2[respect_exclude_path] = "1"
+do_image_ext3[respect_exclude_path] = "1"
+do_image_ext4[respect_exclude_path] = "1"
 
 MIN_BTRFS_SIZE ?= "16384"
 IMAGE_CMD_btrfs () {
@@ -103,10 +108,14 @@ IMAGE_CMD_btrfs () {
 	dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs count=${size} bs=1024
 	mkfs.btrfs ${EXTRA_IMAGECMD} -r ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs
 }
+do_image_btrfs[respect_exclude_path] = "1"
 
 IMAGE_CMD_squashfs = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs ${EXTRA_IMAGECMD} -noappend"
 IMAGE_CMD_squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz"
 IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo"
+do_image_squashfs[respect_exclude_path] = "1"
+do_image_squashfs-xz[respect_exclude_path] = "1"
+do_image_squashfs-lzo[respect_exclude_path] = "1"
 
 # By default, tar from the host is used, which can be quite old. If
 # you need special parameters (like --xattrs) which are only supported
@@ -121,6 +130,7 @@ IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAM
 IMAGE_CMD_TAR ?= "tar"
 # ignore return code 1 "file changed as we read it" as other tasks(e.g. do_image_wic) may be hardlinking rootfs
 IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} . || [ $? -eq 1 ]"
+do_image_tar[respect_exclude_path] = "1"
 
 do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
 IMAGE_CMD_cpio () {
@@ -141,6 +151,7 @@ IMAGE_CMD_cpio () {
 		fi
 	fi
 }
+do_image_cpio[respect_exclude_path] = "1"
 
 ELF_KERNEL ?= "${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE}"
 ELF_APPEND ?= "ramdisk_size=32768 root=/dev/ram0 rw console="
@@ -149,6 +160,7 @@ IMAGE_CMD_elf () {
 	test -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf && rm -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf
 	mkelfImage --kernel=${ELF_KERNEL} --initrd=${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.cpio.gz --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf --append='${ELF_APPEND}' ${EXTRA_IMAGECMD}
 }
+do_image_elf[respect_exclude_path] = "1"
 
 IMAGE_TYPEDEP_elf = "cpio.gz"
 
@@ -212,6 +224,7 @@ IMAGE_CMD_ubi () {
 }
 
 IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ubifs ${MKUBIFS_ARGS}"
+do_image_ubifs[respect_exclude_path] = "1"
 
 EXTRA_IMAGECMD = ""
 
-- 
2.7.4



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev4)
  2017-05-22  9:38     ` Kristian Amlie
                         ` (2 preceding siblings ...)
  2017-06-19 14:31       ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev3) Patchwork
@ 2017-08-23 12:04       ` Patchwork
  2017-08-23 12:04       ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev5) Patchwork
  2017-08-23 13:04       ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev6) Patchwork
  5 siblings, 0 replies; 41+ messages in thread
From: Patchwork @ 2017-08-23 12:04 UTC (permalink / raw)
  To: Kristian Amlie; +Cc: openembedded-core

== Series Details ==

Series: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev4)
Revision: 4
URL   : https://patchwork.openembedded.org/series/6525/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at f59e729f98)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe



^ permalink raw reply	[flat|nested] 41+ messages in thread

* ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev5)
  2017-05-22  9:38     ` Kristian Amlie
                         ` (3 preceding siblings ...)
  2017-08-23 12:04       ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev4) Patchwork
@ 2017-08-23 12:04       ` Patchwork
  2017-08-23 13:04       ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev6) Patchwork
  5 siblings, 0 replies; 41+ messages in thread
From: Patchwork @ 2017-08-23 12:04 UTC (permalink / raw)
  To: Kristian Amlie; +Cc: openembedded-core

== Series Details ==

Series: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev5)
Revision: 5
URL   : https://patchwork.openembedded.org/series/6525/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at f59e729f98)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-08-23 11:46                   ` [PATCH v4] " Kristian Amlie
@ 2017-08-23 12:39                     ` Kristian Amlie
  2017-08-23 12:39                       ` [PATCH v5] " Kristian Amlie
  2017-08-23 12:46                       ` Richard Purdie
  0 siblings, 2 replies; 41+ messages in thread
From: Kristian Amlie @ 2017-08-23 12:39 UTC (permalink / raw)
  To: openembedded-core


I don't understand why I'm getting patch failures. This is rebased to
cutting edge master. Trying one more time...

-- 
Kristian


^ permalink raw reply	[flat|nested] 41+ messages in thread

* [PATCH v5] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-08-23 12:39                     ` Kristian Amlie
@ 2017-08-23 12:39                       ` Kristian Amlie
  2017-08-23 12:47                         ` Richard Purdie
  2017-08-23 12:46                       ` Richard Purdie
  1 sibling, 1 reply; 41+ messages in thread
From: Kristian Amlie @ 2017-08-23 12:39 UTC (permalink / raw)
  To: openembedded-core

From: Kristian Amlie <kristian.amlie@mender.io>

This is a direct followup from the earlier f6a064d969f414 commit in
wic. It works more or less the same way: The variable specifies a list
of directories relative to the root of the rootfs, and these
directories will be excluded from the resulting rootfs image. If an
entry ends with a slash, only the contents are omitted, not the
directory itself.

Since the intended use of the variable is to exclude certain
directories from the rootfs, and then include said directories in
other partitions, it is not natural for this variable to be respected
for image creators that create multi partition images. Therefore the
default is to ignore the variable, and image creators that create
single root filesystems need to set

  do_image_myfs[respect_exclude_path] = "1"

in order to honor it. Specifically, "wic" and "multiubi" have not
received this variable flag, while others have.

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
---
 documentation/ref-manual/ref-variables.xml | 35 +++++++++++++
 meta/classes/image.bbclass                 | 83 +++++++++++++++++++++++++++++-
 meta/classes/image_types.bbclass           | 13 +++++
 3 files changed, 129 insertions(+), 2 deletions(-)

diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml
index ff8f4e7..a4f2de5 100644
--- a/documentation/ref-manual/ref-variables.xml
+++ b/documentation/ref-manual/ref-variables.xml
@@ -6043,6 +6043,41 @@
             </glossdef>
         </glossentry>
 
+        <glossentry id='var-IMAGE_ROOTFS_EXCLUDE_PATH'><glossterm>IMAGE_ROOTFS_EXCLUDE_PATH</glossterm>
+            <info>
+                IMAGE_ROOTFS_EXCLUDE_PATH[doc] = "Specifies paths to omit from the rootfs."
+            </info>
+            <glossdef>
+                <para role="glossdeffirst">
+<!--                <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> -->
+                    Specifies paths that should be omitted from the root filesystem, separated by
+                    spaces. The paths must be relative to the root of the filesystem (for example
+                    "usr"). Single root filesystem images will not include the path in the
+                    filesystem, except if the path ends with a slash, then the directory will be
+                    included, but without any content.
+                    <literallayout class='monospaced'>
+     # Omit /usr completely from rootfs.
+     IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr</replaceable> ..."
+     # Include the directory /usr in rootfs, but not its content.
+     IMAGE_ROOTFS_EXCLUDE_PATH += "<replaceable>usr/</replaceable> ..."
+                    </literallayout>
+                </para>
+                <para>
+                    Image creators that create multi partition images typically do not honor this
+                    variable, and must provide their own method to split the directory
+                    structure. For example, <filename>wic</filename> has
+                    <filename>--exclude-path</filename> and <filename>--rootfs-dir</filename>.
+                </para>
+                <para>
+                    The default for custom image creators is to ignore this variable. Those that
+                    need to honor it must set:
+                    <literallayout class='monospaced'>
+     do_image_<replaceable>custom</replaceable>[respect_exclude_path] = "1"
+                    </literallayout>
+                </para>
+            </glossdef>
+        </glossentry>
+
         <glossentry id='var-IMAGE_ROOTFS_EXTRA_SPACE'><glossterm>IMAGE_ROOTFS_EXTRA_SPACE</glossterm>
             <info>
                 IMAGE_ROOTFS_EXTRA_SPACE[doc] = "Defines additional free disk space created in the image in Kbytes. By default, this variable is set to '0'."
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 3639aa4..e215541 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -117,7 +117,8 @@ def rootfs_variables(d):
                  'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
                  'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
                  'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
-                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS']
+                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS',
+                 'IMAGE_ROOTFS_EXCLUDE_PATH']
     variables.extend(rootfs_command_variables(d))
     variables.extend(variable_depends(d))
     return " ".join(variables)
@@ -496,8 +497,9 @@ python () {
         d.setVarFlag(task, 'func', '1')
         d.setVarFlag(task, 'fakeroot', '1')
 
-        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size')
+        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories')
         d.prependVarFlag(task, 'postfuncs', ' create_symlinks')
+        d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories')
         d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages))
         d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps))
         d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE')
@@ -506,6 +508,83 @@ python () {
         bb.build.addtask(task, 'do_image_complete', after, d)
 }
 
+python prepare_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    taskname = d.getVar("BB_CURRENTTASK")
+
+    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1':
+        return
+
+    import shutil
+    from oe.path import copyhardlinktree
+
+    exclude_list = exclude_var.split()
+
+    rootfs_orig = d.getVar('IMAGE_ROOTFS')
+    # We need a new rootfs directory we can delete files from. Copy to
+    # workdir.
+    new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname))
+
+    if os.path.lexists(new_rootfs):
+        shutil.rmtree(os.path.join(new_rootfs))
+
+    copyhardlinktree(rootfs_orig, new_rootfs)
+
+    for orig_path in exclude_list:
+        path = orig_path
+        if os.path.isabs(path):
+            bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path)
+
+        full_path = os.path.realpath(os.path.join(new_rootfs, path))
+
+        # Disallow climbing outside of parent directory using '..',
+        # because doing so could be quite disastrous (we will delete the
+        # directory).
+        if not full_path.startswith(new_rootfs):
+            bb.fatal("'%s' points to a path outside the rootfs" % orig_path)
+
+        if path.endswith(os.sep):
+            # Delete content only.
+            for entry in os.listdir(full_path):
+                full_entry = os.path.join(full_path, entry)
+                if os.path.isdir(full_entry) and not os.path.islink(full_entry):
+                    shutil.rmtree(full_entry)
+                else:
+                    os.remove(full_entry)
+        else:
+            # Delete whole directory.
+            shutil.rmtree(full_path)
+
+    # Save old value for cleanup later.
+    d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig)
+    d.setVar('IMAGE_ROOTFS', new_rootfs)
+}
+
+python cleanup_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    taskname = d.getVar("BB_CURRENTTASK")
+
+    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') != '1':
+        return
+
+    import shutil
+
+    rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS')
+    rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG')
+    # This should never happen, since we should have set it to a different
+    # directory in the prepare function.
+    assert rootfs_dirs_excluded != rootfs_orig
+
+    shutil.rmtree(rootfs_dirs_excluded)
+    d.setVar('IMAGE_ROOTFS', rootfs_orig)
+}
+
 #
 # Compute the rootfs size
 #
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index e0368c7..a063df9 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -62,8 +62,10 @@ ZIP_COMPRESSION_LEVEL ?= "-9"
 
 JFFS2_SUM_EXTRA_ARGS ?= ""
 IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 ${EXTRA_IMAGECMD}"
+do_image_jffs2[respect_exclude_path] = "1"
 
 IMAGE_CMD_cramfs = "mkfs.cramfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cramfs ${EXTRA_IMAGECMD}"
+do_image_cramfs[respect_exclude_path] = "1"
 
 oe_mkext234fs () {
 	fstype=$1
@@ -92,6 +94,9 @@ oe_mkext234fs () {
 IMAGE_CMD_ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}"
 IMAGE_CMD_ext3 = "oe_mkext234fs ext3 ${EXTRA_IMAGECMD}"
 IMAGE_CMD_ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}"
+do_image_ext2[respect_exclude_path] = "1"
+do_image_ext3[respect_exclude_path] = "1"
+do_image_ext4[respect_exclude_path] = "1"
 
 MIN_BTRFS_SIZE ?= "16384"
 IMAGE_CMD_btrfs () {
@@ -103,10 +108,14 @@ IMAGE_CMD_btrfs () {
 	dd if=/dev/zero of=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs count=${size} bs=1024
 	mkfs.btrfs ${EXTRA_IMAGECMD} -r ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs
 }
+do_image_btrfs[respect_exclude_path] = "1"
 
 IMAGE_CMD_squashfs = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs ${EXTRA_IMAGECMD} -noappend"
 IMAGE_CMD_squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz"
 IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo"
+do_image_squashfs[respect_exclude_path] = "1"
+do_image_squashfs-xz[respect_exclude_path] = "1"
+do_image_squashfs-lzo[respect_exclude_path] = "1"
 
 # By default, tar from the host is used, which can be quite old. If
 # you need special parameters (like --xattrs) which are only supported
@@ -121,6 +130,7 @@ IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAM
 IMAGE_CMD_TAR ?= "tar"
 # ignore return code 1 "file changed as we read it" as other tasks(e.g. do_image_wic) may be hardlinking rootfs
 IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} . || [ $? -eq 1 ]"
+do_image_tar[respect_exclude_path] = "1"
 
 do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
 IMAGE_CMD_cpio () {
@@ -141,6 +151,7 @@ IMAGE_CMD_cpio () {
 		fi
 	fi
 }
+do_image_cpio[respect_exclude_path] = "1"
 
 ELF_KERNEL ?= "${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE}"
 ELF_APPEND ?= "ramdisk_size=32768 root=/dev/ram0 rw console="
@@ -149,6 +160,7 @@ IMAGE_CMD_elf () {
 	test -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf && rm -f ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf
 	mkelfImage --kernel=${ELF_KERNEL} --initrd=${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.cpio.gz --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf --append='${ELF_APPEND}' ${EXTRA_IMAGECMD}
 }
+do_image_elf[respect_exclude_path] = "1"
 
 IMAGE_TYPEDEP_elf = "cpio.gz"
 
@@ -212,6 +224,7 @@ IMAGE_CMD_ubi () {
 }
 
 IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ubifs ${MKUBIFS_ARGS}"
+do_image_ubifs[respect_exclude_path] = "1"
 
 EXTRA_IMAGECMD = ""
 
-- 
2.7.4



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* Re: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-08-23 12:39                     ` Kristian Amlie
  2017-08-23 12:39                       ` [PATCH v5] " Kristian Amlie
@ 2017-08-23 12:46                       ` Richard Purdie
  2017-08-23 13:19                         ` Kristian Amlie
  1 sibling, 1 reply; 41+ messages in thread
From: Richard Purdie @ 2017-08-23 12:46 UTC (permalink / raw)
  To: Kristian Amlie, openembedded-core

On Wed, 2017-08-23 at 14:39 +0200, Kristian Amlie wrote:
> I don't understand why I'm getting patch failures. This is rebased to
> cutting edge master. Trying one more time...

Its because you're patching documentation which is maintained in a
separate repository. Separating that from the patch will avoid that
error.

Cheers,

Richard


^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [PATCH v5] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-08-23 12:39                       ` [PATCH v5] " Kristian Amlie
@ 2017-08-23 12:47                         ` Richard Purdie
  2017-08-23 13:19                           ` Kristian Amlie
  0 siblings, 1 reply; 41+ messages in thread
From: Richard Purdie @ 2017-08-23 12:47 UTC (permalink / raw)
  To: Kristian Amlie, openembedded-core

On Wed, 2017-08-23 at 14:39 +0200, Kristian Amlie wrote:
> From: Kristian Amlie <kristian.amlie@mender.io>
> 
> This is a direct followup from the earlier f6a064d969f414 commit in
> wic. It works more or less the same way: The variable specifies a
> list
> of directories relative to the root of the rootfs, and these
> directories will be excluded from the resulting rootfs image. If an
> entry ends with a slash, only the contents are omitted, not the
> directory itself.
> 
> Since the intended use of the variable is to exclude certain
> directories from the rootfs, and then include said directories in
> other partitions, it is not natural for this variable to be respected
> for image creators that create multi partition images. Therefore the
> default is to ignore the variable, and image creators that create
> single root filesystems need to set
> 
>   do_image_myfs[respect_exclude_path] = "1"
> 
> in order to honor it. Specifically, "wic" and "multiubi" have not
> received this variable flag, while others have.

I have to wonder if this is implemented the correct way around. Would
it not be better to mark multiubi and wic as not wanting this feature,
rather than requiring every other image creation method (which may even
be in other layers) to set this?

Cheers,

Richard



^ permalink raw reply	[flat|nested] 41+ messages in thread

* ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev6)
  2017-05-22  9:38     ` Kristian Amlie
                         ` (4 preceding siblings ...)
  2017-08-23 12:04       ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev5) Patchwork
@ 2017-08-23 13:04       ` Patchwork
  2017-08-23 15:12         ` Leonardo Sandoval
  5 siblings, 1 reply; 41+ messages in thread
From: Patchwork @ 2017-08-23 13:04 UTC (permalink / raw)
  To: Kristian Amlie; +Cc: openembedded-core

== Series Details ==

Series: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev6)
Revision: 6
URL   : https://patchwork.openembedded.org/series/6525/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at b1e482ae20)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [PATCH v5] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-08-23 12:47                         ` Richard Purdie
@ 2017-08-23 13:19                           ` Kristian Amlie
  2017-08-28 15:47                             ` Kristian Amlie
  0 siblings, 1 reply; 41+ messages in thread
From: Kristian Amlie @ 2017-08-23 13:19 UTC (permalink / raw)
  To: Richard Purdie, openembedded-core

On 23/08/17 14:47, Richard Purdie wrote:
> On Wed, 2017-08-23 at 14:39 +0200, Kristian Amlie wrote:
>> From: Kristian Amlie <kristian.amlie@mender.io>
>>
>> This is a direct followup from the earlier f6a064d969f414 commit in
>> wic. It works more or less the same way: The variable specifies a
>> list
>> of directories relative to the root of the rootfs, and these
>> directories will be excluded from the resulting rootfs image. If an
>> entry ends with a slash, only the contents are omitted, not the
>> directory itself.
>>
>> Since the intended use of the variable is to exclude certain
>> directories from the rootfs, and then include said directories in
>> other partitions, it is not natural for this variable to be respected
>> for image creators that create multi partition images. Therefore the
>> default is to ignore the variable, and image creators that create
>> single root filesystems need to set
>>
>>   do_image_myfs[respect_exclude_path] = "1"
>>
>> in order to honor it. Specifically, "wic" and "multiubi" have not
>> received this variable flag, while others have.
> 
> I have to wonder if this is implemented the correct way around. Would
> it not be better to mark multiubi and wic as not wanting this feature,
> rather than requiring every other image creation method (which may even
> be in other layers) to set this?

The idea was that people in downstream repositories (other layers) are
more likely to have their own complete image creators than to have
filesystem creators. So rather than requiring all of the downstream
repositories to opt out of the feature, we opt in in the filesystem
creators which are mostly contained in OE.

Or so was the thinking. Not sure if it's a good reason though. I can
invert the behavior if you think it's better!

-- 
Kristian


^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-08-23 12:46                       ` Richard Purdie
@ 2017-08-23 13:19                         ` Kristian Amlie
  0 siblings, 0 replies; 41+ messages in thread
From: Kristian Amlie @ 2017-08-23 13:19 UTC (permalink / raw)
  To: Richard Purdie, openembedded-core

On 23/08/17 14:46, Richard Purdie wrote:
> On Wed, 2017-08-23 at 14:39 +0200, Kristian Amlie wrote:
>> I don't understand why I'm getting patch failures. This is rebased to
>> cutting edge master. Trying one more time...
> 
> Its because you're patching documentation which is maintained in a
> separate repository. Separating that from the patch will avoid that
> error.

Aaah, right, this is from poky. Thanks for explaining, I'll fix that!

-- 
Kristian


^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev6)
  2017-08-23 13:04       ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev6) Patchwork
@ 2017-08-23 15:12         ` Leonardo Sandoval
  0 siblings, 0 replies; 41+ messages in thread
From: Leonardo Sandoval @ 2017-08-23 15:12 UTC (permalink / raw)
  To: openembedded-core

On Wed, 2017-08-23 at 13:04 +0000, Patchwork wrote:
> == Series Details ==
> 
> Series: do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev6)
> Revision: 6
> URL   : https://patchwork.openembedded.org/series/6525/
> State : failure
> 
> == Summary ==
> 
> 
> Thank you for submitting this patch series to OpenEmbedded Core. This is
> an automated response. Several tests have been executed on the proposed
> series by patchtest resulting in the following failures:
> 
> 
> 
> * Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
>   Suggested fix    Rebase your series on top of targeted branch
>   Targeted branch  master (currently at b1e482ae20)
> 

it is not pretty clear what the problem is with your patch with the
message above but the problem is that your are changing documentation on
your series and the latter goes into another mailing list.

Check poky's README for more info.


> 
> 
> If you believe any of these test results are incorrect, please reply to the
> mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
> Otherwise we would appreciate you correcting the issues and submitting a new
> version of the patchset if applicable. Please ensure you add/increment the
> version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
> [PATCH v3] -> ...).
> 
> ---
> Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
> Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
> 




^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [PATCH v5] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-08-23 13:19                           ` Kristian Amlie
@ 2017-08-28 15:47                             ` Kristian Amlie
  2017-08-28 15:47                               ` [PATCH v6] " Kristian Amlie
  0 siblings, 1 reply; 41+ messages in thread
From: Kristian Amlie @ 2017-08-28 15:47 UTC (permalink / raw)
  To: Richard Purdie, openembedded-core

On 23/08/17 15:19, Kristian Amlie wrote:
> On 23/08/17 14:47, Richard Purdie wrote:
>> On Wed, 2017-08-23 at 14:39 +0200, Kristian Amlie wrote:
>>> From: Kristian Amlie <kristian.amlie@mender.io>
>>>
>>> This is a direct followup from the earlier f6a064d969f414 commit in
>>> wic. It works more or less the same way: The variable specifies a
>>> list
>>> of directories relative to the root of the rootfs, and these
>>> directories will be excluded from the resulting rootfs image. If an
>>> entry ends with a slash, only the contents are omitted, not the
>>> directory itself.
>>>
>>> Since the intended use of the variable is to exclude certain
>>> directories from the rootfs, and then include said directories in
>>> other partitions, it is not natural for this variable to be respected
>>> for image creators that create multi partition images. Therefore the
>>> default is to ignore the variable, and image creators that create
>>> single root filesystems need to set
>>>
>>>   do_image_myfs[respect_exclude_path] = "1"
>>>
>>> in order to honor it. Specifically, "wic" and "multiubi" have not
>>> received this variable flag, while others have.
>>
>> I have to wonder if this is implemented the correct way around. Would
>> it not be better to mark multiubi and wic as not wanting this feature,
>> rather than requiring every other image creation method (which may even
>> be in other layers) to set this?
> 
> The idea was that people in downstream repositories (other layers) are
> more likely to have their own complete image creators than to have
> filesystem creators. So rather than requiring all of the downstream
> repositories to opt out of the feature, we opt in in the filesystem
> creators which are mostly contained in OE.
> 
> Or so was the thinking. Not sure if it's a good reason though. I can
> invert the behavior if you think it's better!

Since I heard nothing more I assume that it's better inverted, and I
kind of agree as well, the patch is much smaller. So here follows the
next patch, the changes from the last are:

* Inverted default of the respect_exclude_path flag, now on by default

* Added a debug message when it's turned off, to lessen confusion.

* Documentation is now stripped out as it should be, I will send it as a
separate patch as soon as we've concluded on this patch.

-- 
Kristian


^ permalink raw reply	[flat|nested] 41+ messages in thread

* [PATCH v6] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-08-28 15:47                             ` Kristian Amlie
@ 2017-08-28 15:47                               ` Kristian Amlie
  2017-09-18  6:45                                 ` Kristian Amlie
  0 siblings, 1 reply; 41+ messages in thread
From: Kristian Amlie @ 2017-08-28 15:47 UTC (permalink / raw)
  To: Richard Purdie, openembedded-core

This is a direct followup from the earlier 6602392db3d39 commit in
wic. It works more or less the same way: The variable specifies a list
of directories relative to the root of the rootfs, and these
directories will be excluded from the resulting rootfs image. If an
entry ends with a slash, only the contents are omitted, not the
directory itself.

Since the intended use of the variable is to exclude certain
directories from the rootfs, and then include said directories in
other partitions, it is not natural for this variable to be respected
for image creators that create multi partition images. These can turn
the feature off locally by defining:

  do_image_myfs[respect_exclude_path] = "0"

Specifically, "wic" and "multiubi" come with the feature disabled.

Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>
---
 meta/classes/image.bbclass           | 84 +++++++++++++++++++++++++++++++++++-
 meta/classes/image_types.bbclass     |  1 +
 meta/classes/image_types_wic.bbclass |  1 +
 3 files changed, 84 insertions(+), 2 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 3639aa4..9cefff1 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -117,7 +117,8 @@ def rootfs_variables(d):
                  'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
                  'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
                  'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
-                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS']
+                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS',
+                 'IMAGE_ROOTFS_EXCLUDE_PATH']
     variables.extend(rootfs_command_variables(d))
     variables.extend(variable_depends(d))
     return " ".join(variables)
@@ -496,8 +497,9 @@ python () {
         d.setVarFlag(task, 'func', '1')
         d.setVarFlag(task, 'fakeroot', '1')
 
-        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size')
+        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories')
         d.prependVarFlag(task, 'postfuncs', ' create_symlinks')
+        d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories')
         d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages))
         d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps))
         d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE')
@@ -506,6 +508,84 @@ python () {
         bb.build.addtask(task, 'do_image_complete', after, d)
 }
 
+python prepare_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    taskname = d.getVar("BB_CURRENTTASK")
+
+    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0':
+        bb.debug(1, "'IMAGE_ROOTFS_EXCLUDE_PATH' is set but 'respect_exclude_path' variable flag is 0 for this image type, so ignoring it")
+        return
+
+    import shutil
+    from oe.path import copyhardlinktree
+
+    exclude_list = exclude_var.split()
+
+    rootfs_orig = d.getVar('IMAGE_ROOTFS')
+    # We need a new rootfs directory we can delete files from. Copy to
+    # workdir.
+    new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname))
+
+    if os.path.lexists(new_rootfs):
+        shutil.rmtree(os.path.join(new_rootfs))
+
+    copyhardlinktree(rootfs_orig, new_rootfs)
+
+    for orig_path in exclude_list:
+        path = orig_path
+        if os.path.isabs(path):
+            bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path)
+
+        full_path = os.path.realpath(os.path.join(new_rootfs, path))
+
+        # Disallow climbing outside of parent directory using '..',
+        # because doing so could be quite disastrous (we will delete the
+        # directory).
+        if not full_path.startswith(new_rootfs):
+            bb.fatal("'%s' points to a path outside the rootfs" % orig_path)
+
+        if path.endswith(os.sep):
+            # Delete content only.
+            for entry in os.listdir(full_path):
+                full_entry = os.path.join(full_path, entry)
+                if os.path.isdir(full_entry) and not os.path.islink(full_entry):
+                    shutil.rmtree(full_entry)
+                else:
+                    os.remove(full_entry)
+        else:
+            # Delete whole directory.
+            shutil.rmtree(full_path)
+
+    # Save old value for cleanup later.
+    d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig)
+    d.setVar('IMAGE_ROOTFS', new_rootfs)
+}
+
+python cleanup_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    taskname = d.getVar("BB_CURRENTTASK")
+
+    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0':
+        return
+
+    import shutil
+
+    rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS')
+    rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG')
+    # This should never happen, since we should have set it to a different
+    # directory in the prepare function.
+    assert rootfs_dirs_excluded != rootfs_orig
+
+    shutil.rmtree(rootfs_dirs_excluded)
+    d.setVar('IMAGE_ROOTFS', rootfs_orig)
+}
+
 #
 # Compute the rootfs size
 #
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index e0368c7..3b9250b 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -206,6 +206,7 @@ IMAGE_CMD_multiubi () {
 		multiubi_mkfs "${mkubifs_args}" "${ubinize_args}" "${name}"
 	done
 }
+do_image_multiubi[respect_exclude_path] = "0"
 
 IMAGE_CMD_ubi () {
 	multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}"
diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
index b825b47..a34a7c6 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -34,6 +34,7 @@ IMAGE_CMD_wic () {
 	rm -rf "$out/"
 }
 IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES"
+do_image_wic[respect_exclude_path] = "0"
 
 # Rebuild when the wks file or vars in WICVARS change
 USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}"
-- 
2.7.4



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* Re: [PATCH v6] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-08-28 15:47                               ` [PATCH v6] " Kristian Amlie
@ 2017-09-18  6:45                                 ` Kristian Amlie
  2017-10-13  9:08                                   ` Kristian Amlie
  0 siblings, 1 reply; 41+ messages in thread
From: Kristian Amlie @ 2017-09-18  6:45 UTC (permalink / raw)
  To: openembedded-core

Ping. Anything missing here?

-- 
Kristian

On 28/08/17 17:47, Kristian Amlie wrote:
> This is a direct followup from the earlier 6602392db3d39 commit in
> wic. It works more or less the same way: The variable specifies a list
> of directories relative to the root of the rootfs, and these
> directories will be excluded from the resulting rootfs image. If an
> entry ends with a slash, only the contents are omitted, not the
> directory itself.
> 
> Since the intended use of the variable is to exclude certain
> directories from the rootfs, and then include said directories in
> other partitions, it is not natural for this variable to be respected
> for image creators that create multi partition images. These can turn
> the feature off locally by defining:
> 
>   do_image_myfs[respect_exclude_path] = "0"
> 
> Specifically, "wic" and "multiubi" come with the feature disabled.
> 
> Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>
> ---
>  meta/classes/image.bbclass           | 84 +++++++++++++++++++++++++++++++++++-
>  meta/classes/image_types.bbclass     |  1 +
>  meta/classes/image_types_wic.bbclass |  1 +
>  3 files changed, 84 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 3639aa4..9cefff1 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -117,7 +117,8 @@ def rootfs_variables(d):
>                   'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
>                   'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
>                   'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
> -                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS']
> +                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS',
> +                 'IMAGE_ROOTFS_EXCLUDE_PATH']
>      variables.extend(rootfs_command_variables(d))
>      variables.extend(variable_depends(d))
>      return " ".join(variables)
> @@ -496,8 +497,9 @@ python () {
>          d.setVarFlag(task, 'func', '1')
>          d.setVarFlag(task, 'fakeroot', '1')
>  
> -        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size')
> +        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories')
>          d.prependVarFlag(task, 'postfuncs', ' create_symlinks')
> +        d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories')
>          d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages))
>          d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps))
>          d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE')
> @@ -506,6 +508,84 @@ python () {
>          bb.build.addtask(task, 'do_image_complete', after, d)
>  }
>  
> +python prepare_excluded_directories() {
> +    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
> +    if not exclude_var:
> +        return
> +
> +    taskname = d.getVar("BB_CURRENTTASK")
> +
> +    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0':
> +        bb.debug(1, "'IMAGE_ROOTFS_EXCLUDE_PATH' is set but 'respect_exclude_path' variable flag is 0 for this image type, so ignoring it")
> +        return
> +
> +    import shutil
> +    from oe.path import copyhardlinktree
> +
> +    exclude_list = exclude_var.split()
> +
> +    rootfs_orig = d.getVar('IMAGE_ROOTFS')
> +    # We need a new rootfs directory we can delete files from. Copy to
> +    # workdir.
> +    new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname))
> +
> +    if os.path.lexists(new_rootfs):
> +        shutil.rmtree(os.path.join(new_rootfs))
> +
> +    copyhardlinktree(rootfs_orig, new_rootfs)
> +
> +    for orig_path in exclude_list:
> +        path = orig_path
> +        if os.path.isabs(path):
> +            bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path)
> +
> +        full_path = os.path.realpath(os.path.join(new_rootfs, path))
> +
> +        # Disallow climbing outside of parent directory using '..',
> +        # because doing so could be quite disastrous (we will delete the
> +        # directory).
> +        if not full_path.startswith(new_rootfs):
> +            bb.fatal("'%s' points to a path outside the rootfs" % orig_path)
> +
> +        if path.endswith(os.sep):
> +            # Delete content only.
> +            for entry in os.listdir(full_path):
> +                full_entry = os.path.join(full_path, entry)
> +                if os.path.isdir(full_entry) and not os.path.islink(full_entry):
> +                    shutil.rmtree(full_entry)
> +                else:
> +                    os.remove(full_entry)
> +        else:
> +            # Delete whole directory.
> +            shutil.rmtree(full_path)
> +
> +    # Save old value for cleanup later.
> +    d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig)
> +    d.setVar('IMAGE_ROOTFS', new_rootfs)
> +}
> +
> +python cleanup_excluded_directories() {
> +    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
> +    if not exclude_var:
> +        return
> +
> +    taskname = d.getVar("BB_CURRENTTASK")
> +
> +    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0':
> +        return
> +
> +    import shutil
> +
> +    rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS')
> +    rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG')
> +    # This should never happen, since we should have set it to a different
> +    # directory in the prepare function.
> +    assert rootfs_dirs_excluded != rootfs_orig
> +
> +    shutil.rmtree(rootfs_dirs_excluded)
> +    d.setVar('IMAGE_ROOTFS', rootfs_orig)
> +}
> +
>  #
>  # Compute the rootfs size
>  #
> diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
> index e0368c7..3b9250b 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -206,6 +206,7 @@ IMAGE_CMD_multiubi () {
>  		multiubi_mkfs "${mkubifs_args}" "${ubinize_args}" "${name}"
>  	done
>  }
> +do_image_multiubi[respect_exclude_path] = "0"
>  
>  IMAGE_CMD_ubi () {
>  	multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}"
> diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
> index b825b47..a34a7c6 100644
> --- a/meta/classes/image_types_wic.bbclass
> +++ b/meta/classes/image_types_wic.bbclass
> @@ -34,6 +34,7 @@ IMAGE_CMD_wic () {
>  	rm -rf "$out/"
>  }
>  IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES"
> +do_image_wic[respect_exclude_path] = "0"
>  
>  # Rebuild when the wks file or vars in WICVARS change
>  USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}"
> 



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [PATCH v6] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-09-18  6:45                                 ` Kristian Amlie
@ 2017-10-13  9:08                                   ` Kristian Amlie
  2017-10-13  9:08                                     ` [PATCH v7] " Kristian Amlie
  2017-10-13 10:37                                     ` [PATCH v6] " Alexander Kanavin
  0 siblings, 2 replies; 41+ messages in thread
From: Kristian Amlie @ 2017-10-13  9:08 UTC (permalink / raw)
  To: openembedded-core

Rebased again.

Can we get this in please? It has been reviewed and rebased several times
already, and has lingered on the list for a long time.

Regards
Kristian Amlie


^ permalink raw reply	[flat|nested] 41+ messages in thread

* [PATCH v7] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-10-13  9:08                                   ` Kristian Amlie
@ 2017-10-13  9:08                                     ` Kristian Amlie
  2017-10-13 10:37                                     ` [PATCH v6] " Alexander Kanavin
  1 sibling, 0 replies; 41+ messages in thread
From: Kristian Amlie @ 2017-10-13  9:08 UTC (permalink / raw)
  To: openembedded-core

This is a direct followup from the earlier 6602392db3d39 commit in
wic. It works more or less the same way: The variable specifies a list
of directories relative to the root of the rootfs, and these
directories will be excluded from the resulting rootfs image. If an
entry ends with a slash, only the contents are omitted, not the
directory itself.

Since the intended use of the variable is to exclude certain
directories from the rootfs, and then include said directories in
other partitions, it is not natural for this variable to be respected
for image creators that create multi partition images. These can turn
the feature off locally by defining:

  do_image_myfs[respect_exclude_path] = "0"

Specifically, "wic" and "multiubi" come with the feature disabled.

Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>
---
 meta/classes/image.bbclass           | 84 +++++++++++++++++++++++++++++++++++-
 meta/classes/image_types.bbclass     |  1 +
 meta/classes/image_types_wic.bbclass |  1 +
 3 files changed, 84 insertions(+), 2 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 0d14250..2adfbe4 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -117,7 +117,8 @@ def rootfs_variables(d):
                  'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
                  'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
                  'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
-                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS']
+                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS',
+                 'IMAGE_ROOTFS_EXCLUDE_PATH']
     variables.extend(rootfs_command_variables(d))
     variables.extend(variable_depends(d))
     return " ".join(variables)
@@ -499,8 +500,9 @@ python () {
         d.setVarFlag(task, 'func', '1')
         d.setVarFlag(task, 'fakeroot', '1')
 
-        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size')
+        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories')
         d.prependVarFlag(task, 'postfuncs', ' create_symlinks')
+        d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories')
         d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages))
         d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps))
         d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE')
@@ -509,6 +511,84 @@ python () {
         bb.build.addtask(task, 'do_image_complete', after, d)
 }
 
+python prepare_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    taskname = d.getVar("BB_CURRENTTASK")
+
+    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0':
+        bb.debug(1, "'IMAGE_ROOTFS_EXCLUDE_PATH' is set but 'respect_exclude_path' variable flag is 0 for this image type, so ignoring it")
+        return
+
+    import shutil
+    from oe.path import copyhardlinktree
+
+    exclude_list = exclude_var.split()
+
+    rootfs_orig = d.getVar('IMAGE_ROOTFS')
+    # We need a new rootfs directory we can delete files from. Copy to
+    # workdir.
+    new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname))
+
+    if os.path.lexists(new_rootfs):
+        shutil.rmtree(os.path.join(new_rootfs))
+
+    copyhardlinktree(rootfs_orig, new_rootfs)
+
+    for orig_path in exclude_list:
+        path = orig_path
+        if os.path.isabs(path):
+            bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path)
+
+        full_path = os.path.realpath(os.path.join(new_rootfs, path))
+
+        # Disallow climbing outside of parent directory using '..',
+        # because doing so could be quite disastrous (we will delete the
+        # directory).
+        if not full_path.startswith(new_rootfs):
+            bb.fatal("'%s' points to a path outside the rootfs" % orig_path)
+
+        if path.endswith(os.sep):
+            # Delete content only.
+            for entry in os.listdir(full_path):
+                full_entry = os.path.join(full_path, entry)
+                if os.path.isdir(full_entry) and not os.path.islink(full_entry):
+                    shutil.rmtree(full_entry)
+                else:
+                    os.remove(full_entry)
+        else:
+            # Delete whole directory.
+            shutil.rmtree(full_path)
+
+    # Save old value for cleanup later.
+    d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig)
+    d.setVar('IMAGE_ROOTFS', new_rootfs)
+}
+
+python cleanup_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    taskname = d.getVar("BB_CURRENTTASK")
+
+    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0':
+        return
+
+    import shutil
+
+    rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS')
+    rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG')
+    # This should never happen, since we should have set it to a different
+    # directory in the prepare function.
+    assert rootfs_dirs_excluded != rootfs_orig
+
+    shutil.rmtree(rootfs_dirs_excluded)
+    d.setVar('IMAGE_ROOTFS', rootfs_orig)
+}
+
 #
 # Compute the rootfs size
 #
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index b373b21..d7d0b42 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -207,6 +207,7 @@ IMAGE_CMD_multiubi () {
 		multiubi_mkfs "${mkubifs_args}" "${ubinize_args}" "${name}"
 	done
 }
+do_image_multiubi[respect_exclude_path] = "0"
 
 IMAGE_CMD_ubi () {
 	multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}"
diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
index 222ae00..9d74e51 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -34,6 +34,7 @@ IMAGE_CMD_wic () {
 	rm -rf "$out/"
 }
 IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
+do_image_wic[respect_exclude_path] = "0"
 
 # Rebuild when the wks file or vars in WICVARS change
 USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}"
-- 
2.7.4



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* Re: [PATCH v6] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-10-13  9:08                                   ` Kristian Amlie
  2017-10-13  9:08                                     ` [PATCH v7] " Kristian Amlie
@ 2017-10-13 10:37                                     ` Alexander Kanavin
  2017-10-13 11:22                                       ` Kristian Amlie
  2017-11-22 13:13                                       ` Kristian Amlie
  1 sibling, 2 replies; 41+ messages in thread
From: Alexander Kanavin @ 2017-10-13 10:37 UTC (permalink / raw)
  To: Kristian Amlie, openembedded-core

On 10/13/2017 12:08 PM, Kristian Amlie wrote:
> Rebased again.
> 
> Can we get this in please? It has been reviewed and rebased several times
> already, and has lingered on the list for a long time.

Bear with the pre-release freeze period please; once rocko is out 
patches will start flowing to master again.

Alex



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [PATCH v6] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-10-13 10:37                                     ` [PATCH v6] " Alexander Kanavin
@ 2017-10-13 11:22                                       ` Kristian Amlie
  2017-11-22 13:13                                       ` Kristian Amlie
  1 sibling, 0 replies; 41+ messages in thread
From: Kristian Amlie @ 2017-10-13 11:22 UTC (permalink / raw)
  To: Alexander Kanavin, openembedded-core

On 13/10/17 12:37, Alexander Kanavin wrote:
> On 10/13/2017 12:08 PM, Kristian Amlie wrote:
>> Rebased again.
>>
>> Can we get this in please? It has been reviewed and rebased several times
>> already, and has lingered on the list for a long time.
> 
> Bear with the pre-release freeze period please; once rocko is out
> patches will start flowing to master again.

Oh ok, I'm not following the schedule. I will try again in a while then.

-- 
Kristian


^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [PATCH v6] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-10-13 10:37                                     ` [PATCH v6] " Alexander Kanavin
  2017-10-13 11:22                                       ` Kristian Amlie
@ 2017-11-22 13:13                                       ` Kristian Amlie
  2017-11-22 13:13                                         ` [PATCH v7] " Kristian Amlie
  2017-11-22 13:31                                         ` [PATCH v6] " Alexander Kanavin
  1 sibling, 2 replies; 41+ messages in thread
From: Kristian Amlie @ 2017-11-22 13:13 UTC (permalink / raw)
  To: openembedded-core


On 13/10/17 12:37, Alexander Kanavin wrote:
>> ...
> 
> Bear with the pre-release freeze period please; once rocko is out
> patches will start flowing to master again.

So now that rocko has been out for a while, can we get this in please?
Already reviewed, and rebased patch is included.

-- 
Kristian


^ permalink raw reply	[flat|nested] 41+ messages in thread

* [PATCH v7] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-11-22 13:13                                       ` Kristian Amlie
@ 2017-11-22 13:13                                         ` Kristian Amlie
  2017-11-22 13:31                                         ` [PATCH v6] " Alexander Kanavin
  1 sibling, 0 replies; 41+ messages in thread
From: Kristian Amlie @ 2017-11-22 13:13 UTC (permalink / raw)
  To: openembedded-core

This is a direct followup from the earlier 6602392db3d39 commit in
wic. It works more or less the same way: The variable specifies a list
of directories relative to the root of the rootfs, and these
directories will be excluded from the resulting rootfs image. If an
entry ends with a slash, only the contents are omitted, not the
directory itself.

Since the intended use of the variable is to exclude certain
directories from the rootfs, and then include said directories in
other partitions, it is not natural for this variable to be respected
for image creators that create multi partition images. These can turn
the feature off locally by defining:

  do_image_myfs[respect_exclude_path] = "0"

Specifically, "wic" and "multiubi" come with the feature disabled.

Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>
---
 meta/classes/image.bbclass           | 84 +++++++++++++++++++++++++++++++++++-
 meta/classes/image_types.bbclass     |  1 +
 meta/classes/image_types_wic.bbclass |  1 +
 3 files changed, 84 insertions(+), 2 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index d93de02..d378663 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -117,7 +117,8 @@ def rootfs_variables(d):
                  'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
                  'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
                  'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
-                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS']
+                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS',
+                 'IMAGE_ROOTFS_EXCLUDE_PATH']
     variables.extend(rootfs_command_variables(d))
     variables.extend(variable_depends(d))
     return " ".join(variables)
@@ -506,8 +507,9 @@ python () {
         d.setVarFlag(task, 'func', '1')
         d.setVarFlag(task, 'fakeroot', '1')
 
-        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size')
+        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories')
         d.prependVarFlag(task, 'postfuncs', ' create_symlinks')
+        d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories')
         d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages))
         d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps))
         d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE ' + ' '.join(vardepsexclude))
@@ -516,6 +518,84 @@ python () {
         bb.build.addtask(task, 'do_image_complete', after, d)
 }
 
+python prepare_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    taskname = d.getVar("BB_CURRENTTASK")
+
+    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0':
+        bb.debug(1, "'IMAGE_ROOTFS_EXCLUDE_PATH' is set but 'respect_exclude_path' variable flag is 0 for this image type, so ignoring it")
+        return
+
+    import shutil
+    from oe.path import copyhardlinktree
+
+    exclude_list = exclude_var.split()
+
+    rootfs_orig = d.getVar('IMAGE_ROOTFS')
+    # We need a new rootfs directory we can delete files from. Copy to
+    # workdir.
+    new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname))
+
+    if os.path.lexists(new_rootfs):
+        shutil.rmtree(os.path.join(new_rootfs))
+
+    copyhardlinktree(rootfs_orig, new_rootfs)
+
+    for orig_path in exclude_list:
+        path = orig_path
+        if os.path.isabs(path):
+            bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path)
+
+        full_path = os.path.realpath(os.path.join(new_rootfs, path))
+
+        # Disallow climbing outside of parent directory using '..',
+        # because doing so could be quite disastrous (we will delete the
+        # directory).
+        if not full_path.startswith(new_rootfs):
+            bb.fatal("'%s' points to a path outside the rootfs" % orig_path)
+
+        if path.endswith(os.sep):
+            # Delete content only.
+            for entry in os.listdir(full_path):
+                full_entry = os.path.join(full_path, entry)
+                if os.path.isdir(full_entry) and not os.path.islink(full_entry):
+                    shutil.rmtree(full_entry)
+                else:
+                    os.remove(full_entry)
+        else:
+            # Delete whole directory.
+            shutil.rmtree(full_path)
+
+    # Save old value for cleanup later.
+    d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig)
+    d.setVar('IMAGE_ROOTFS', new_rootfs)
+}
+
+python cleanup_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    taskname = d.getVar("BB_CURRENTTASK")
+
+    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0':
+        return
+
+    import shutil
+
+    rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS')
+    rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG')
+    # This should never happen, since we should have set it to a different
+    # directory in the prepare function.
+    assert rootfs_dirs_excluded != rootfs_orig
+
+    shutil.rmtree(rootfs_dirs_excluded)
+    d.setVar('IMAGE_ROOTFS', rootfs_orig)
+}
+
 #
 # Compute the rootfs size
 #
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index d09d127..2d8a945 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -211,6 +211,7 @@ IMAGE_CMD_multiubi () {
 		multiubi_mkfs "${mkubifs_args}" "${ubinize_args}" "${name}"
 	done
 }
+do_image_multiubi[respect_exclude_path] = "0"
 
 IMAGE_CMD_ubi () {
 	multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}"
diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
index 222ae00..9d74e51 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -34,6 +34,7 @@ IMAGE_CMD_wic () {
 	rm -rf "$out/"
 }
 IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
+do_image_wic[respect_exclude_path] = "0"
 
 # Rebuild when the wks file or vars in WICVARS change
 USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}"
-- 
2.7.4



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* Re: [PATCH v6] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-11-22 13:13                                       ` Kristian Amlie
  2017-11-22 13:13                                         ` [PATCH v7] " Kristian Amlie
@ 2017-11-22 13:31                                         ` Alexander Kanavin
  2017-11-22 13:35                                           ` Kristian Amlie
  2018-01-25 10:33                                           ` Kristian Amlie
  1 sibling, 2 replies; 41+ messages in thread
From: Alexander Kanavin @ 2017-11-22 13:31 UTC (permalink / raw)
  To: Kristian Amlie, openembedded-core

On 11/22/2017 03:13 PM, Kristian Amlie wrote:

>> Bear with the pre-release freeze period please; once rocko is out
>> patches will start flowing to master again.
> 
> So now that rocko has been out for a while, can we get this in please?
> Already reviewed, and rebased patch is included.

Please do pay attention to what the recent Yocto Project Status emails 
say. Specifically:

" Until we resolve these issues patches will continue to be slow to 
merge, if at all. This also blocks several core developers from doing 
any feature work at this point in time (e.g. layer setup tool is on 
hold, again).

·        We can only continue to stress that unless others step up and 
help to try and root cause these issues, things will stall with the 
project."

The way things are with the project now, contributors are many, 
maintainers are few. If you can balance things out a little bit, that 
would be much appreciated.

Alex


^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [PATCH v6] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-11-22 13:31                                         ` [PATCH v6] " Alexander Kanavin
@ 2017-11-22 13:35                                           ` Kristian Amlie
  2018-01-25 10:33                                           ` Kristian Amlie
  1 sibling, 0 replies; 41+ messages in thread
From: Kristian Amlie @ 2017-11-22 13:35 UTC (permalink / raw)
  To: Alexander Kanavin, openembedded-core

On 22/11/17 14:31, Alexander Kanavin wrote:
> On 11/22/2017 03:13 PM, Kristian Amlie wrote:
> 
>>> Bear with the pre-release freeze period please; once rocko is out
>>> patches will start flowing to master again.
>>
>> So now that rocko has been out for a while, can we get this in please?
>> Already reviewed, and rebased patch is included.
> 
> Please do pay attention to what the recent Yocto Project Status emails
> say. Specifically:
> 
> " Until we resolve these issues patches will continue to be slow to
> merge, if at all. This also blocks several core developers from doing
> any feature work at this point in time (e.g. layer setup tool is on
> hold, again).
> 
> ·        We can only continue to stress that unless others step up and
> help to try and root cause these issues, things will stall with the
> project."
> 
> The way things are with the project now, contributors are many,
> maintainers are few. If you can balance things out a little bit, that
> would be much appreciated.

Oh sorry, I missed that, I will keep an eye on the status updates!

-- 
Kristian


^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [PATCH v6] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2017-11-22 13:31                                         ` [PATCH v6] " Alexander Kanavin
  2017-11-22 13:35                                           ` Kristian Amlie
@ 2018-01-25 10:33                                           ` Kristian Amlie
  2018-01-25 10:33                                             ` [PATCH v7] " Kristian Amlie
  1 sibling, 1 reply; 41+ messages in thread
From: Kristian Amlie @ 2018-01-25 10:33 UTC (permalink / raw)
  To: openembedded-core


>>> Bear with the pre-release freeze period please; once rocko is out
>>> patches will start flowing to master again.
>> 
>> So now that rocko has been out for a while, can we get this in please?
>> Already reviewed, and rebased patch is included.
> 
> Please do pay attention to what the recent Yocto Project Status emails
> say. Specifically:
> 
> " Until we resolve these issues patches will continue to be slow to merge, if at
> all. This also blocks several core developers from doing any feature work at
> this point in time (e.g. layer setup tool is on hold, again).

Things seem to have calmed down a bit, so I'm submitting a rebased
patch. Let me know if is not the appropriate time.

-- 
Kristian


^ permalink raw reply	[flat|nested] 41+ messages in thread

* [PATCH v7] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2018-01-25 10:33                                           ` Kristian Amlie
@ 2018-01-25 10:33                                             ` Kristian Amlie
  2018-01-25 10:58                                               ` Martin Hundebøll
  2018-02-13  1:29                                               ` Cal Sullivan
  0 siblings, 2 replies; 41+ messages in thread
From: Kristian Amlie @ 2018-01-25 10:33 UTC (permalink / raw)
  To: openembedded-core

This is a direct followup from the earlier 6602392db3d39 commit in
wic. It works more or less the same way: The variable specifies a list
of directories relative to the root of the rootfs, and these
directories will be excluded from the resulting rootfs image. If an
entry ends with a slash, only the contents are omitted, not the
directory itself.

Since the intended use of the variable is to exclude certain
directories from the rootfs, and then include said directories in
other partitions, it is not natural for this variable to be respected
for image creators that create multi partition images. These can turn
the feature off locally by defining:

  do_image_myfs[respect_exclude_path] = "0"

Specifically, "wic" and "multiubi" come with the feature disabled.

Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>
---
 meta/classes/image.bbclass           | 84 +++++++++++++++++++++++++++++++++++-
 meta/classes/image_types.bbclass     |  1 +
 meta/classes/image_types_wic.bbclass |  1 +
 3 files changed, 84 insertions(+), 2 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 4531aa2..849a19c 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -117,7 +117,8 @@ def rootfs_variables(d):
                  'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
                  'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
                  'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
-                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS']
+                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS',
+                 'IMAGE_ROOTFS_EXCLUDE_PATH']
     variables.extend(rootfs_command_variables(d))
     variables.extend(variable_depends(d))
     return " ".join(variables)
@@ -508,8 +509,9 @@ python () {
         d.setVarFlag(task, 'func', '1')
         d.setVarFlag(task, 'fakeroot', '1')
 
-        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size')
+        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories')
         d.prependVarFlag(task, 'postfuncs', ' create_symlinks')
+        d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories')
         d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages))
         d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps))
         d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE ' + ' '.join(vardepsexclude))
@@ -518,6 +520,84 @@ python () {
         bb.build.addtask(task, 'do_image_complete', after, d)
 }
 
+python prepare_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    taskname = d.getVar("BB_CURRENTTASK")
+
+    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0':
+        bb.debug(1, "'IMAGE_ROOTFS_EXCLUDE_PATH' is set but 'respect_exclude_path' variable flag is 0 for this image type, so ignoring it")
+        return
+
+    import shutil
+    from oe.path import copyhardlinktree
+
+    exclude_list = exclude_var.split()
+
+    rootfs_orig = d.getVar('IMAGE_ROOTFS')
+    # We need a new rootfs directory we can delete files from. Copy to
+    # workdir.
+    new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname))
+
+    if os.path.lexists(new_rootfs):
+        shutil.rmtree(os.path.join(new_rootfs))
+
+    copyhardlinktree(rootfs_orig, new_rootfs)
+
+    for orig_path in exclude_list:
+        path = orig_path
+        if os.path.isabs(path):
+            bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path)
+
+        full_path = os.path.realpath(os.path.join(new_rootfs, path))
+
+        # Disallow climbing outside of parent directory using '..',
+        # because doing so could be quite disastrous (we will delete the
+        # directory).
+        if not full_path.startswith(new_rootfs):
+            bb.fatal("'%s' points to a path outside the rootfs" % orig_path)
+
+        if path.endswith(os.sep):
+            # Delete content only.
+            for entry in os.listdir(full_path):
+                full_entry = os.path.join(full_path, entry)
+                if os.path.isdir(full_entry) and not os.path.islink(full_entry):
+                    shutil.rmtree(full_entry)
+                else:
+                    os.remove(full_entry)
+        else:
+            # Delete whole directory.
+            shutil.rmtree(full_path)
+
+    # Save old value for cleanup later.
+    d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig)
+    d.setVar('IMAGE_ROOTFS', new_rootfs)
+}
+
+python cleanup_excluded_directories() {
+    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
+    if not exclude_var:
+        return
+
+    taskname = d.getVar("BB_CURRENTTASK")
+
+    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0':
+        return
+
+    import shutil
+
+    rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS')
+    rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG')
+    # This should never happen, since we should have set it to a different
+    # directory in the prepare function.
+    assert rootfs_dirs_excluded != rootfs_orig
+
+    shutil.rmtree(rootfs_dirs_excluded)
+    d.setVar('IMAGE_ROOTFS', rootfs_orig)
+}
+
 #
 # Compute the rootfs size
 #
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index c736f92..8cd8fd5 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -201,6 +201,7 @@ IMAGE_CMD_multiubi () {
 		multiubi_mkfs "${mkubifs_args}" "${ubinize_args}" "${name}"
 	done
 }
+do_image_multiubi[respect_exclude_path] = "0"
 
 IMAGE_CMD_ubi () {
 	multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}"
diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
index dcf620c..452d7f3 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -34,6 +34,7 @@ IMAGE_CMD_wic () {
 	rm -rf "$out/"
 }
 IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
+do_image_wic[respect_exclude_path] = "0"
 
 # Rebuild when the wks file or vars in WICVARS change
 USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}"
-- 
2.7.4



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* Re: [PATCH v7] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2018-01-25 10:33                                             ` [PATCH v7] " Kristian Amlie
@ 2018-01-25 10:58                                               ` Martin Hundebøll
  2018-02-13  1:29                                               ` Cal Sullivan
  1 sibling, 0 replies; 41+ messages in thread
From: Martin Hundebøll @ 2018-01-25 10:58 UTC (permalink / raw)
  To: openembedded-core

Hi Kristian,

Thanks for your work on this!

On 2018-01-25 11:33, Kristian Amlie wrote:
> This is a direct followup from the earlier 6602392db3d39 commit in
> wic. It works more or less the same way: The variable specifies a list
> of directories relative to the root of the rootfs, and these
> directories will be excluded from the resulting rootfs image. If an
> entry ends with a slash, only the contents are omitted, not the
> directory itself.
> 
> Since the intended use of the variable is to exclude certain
> directories from the rootfs, and then include said directories in
> other partitions, it is not natural for this variable to be respected
> for image creators that create multi partition images. These can turn
> the feature off locally by defining:
> 
>    do_image_myfs[respect_exclude_path] = "0"
> 
> Specifically, "wic" and "multiubi" come with the feature disabled.
> 
> Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>
> ---
>   meta/classes/image.bbclass           | 84 +++++++++++++++++++++++++++++++++++-
>   meta/classes/image_types.bbclass     |  1 +
>   meta/classes/image_types_wic.bbclass |  1 +
>   3 files changed, 84 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 4531aa2..849a19c 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -117,7 +117,8 @@ def rootfs_variables(d):
>                    'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
>                    'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
>                    'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
> -                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS']
> +                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS',
> +                 'IMAGE_ROOTFS_EXCLUDE_PATH']
>       variables.extend(rootfs_command_variables(d))
>       variables.extend(variable_depends(d))
>       return " ".join(variables)
> @@ -508,8 +509,9 @@ python () {
>           d.setVarFlag(task, 'func', '1')
>           d.setVarFlag(task, 'fakeroot', '1')
>   
> -        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size')
> +        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories')
>           d.prependVarFlag(task, 'postfuncs', ' create_symlinks')
> +        d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories')
>           d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages))
>           d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps))
>           d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE ' + ' '.join(vardepsexclude))
> @@ -518,6 +520,84 @@ python () {
>           bb.build.addtask(task, 'do_image_complete', after, d)
>   }
>   
> +python prepare_excluded_directories() {
> +    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
> +    if not exclude_var:
> +        return
> +
> +    taskname = d.getVar("BB_CURRENTTASK")
> +
> +    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0':
> +        bb.debug(1, "'IMAGE_ROOTFS_EXCLUDE_PATH' is set but 'respect_exclude_path' variable flag is 0 for this image type, so ignoring it")
> +        return

If I understand this correctly, paths like "/data" aren't allowed in 
"IMAGE_ROOTFS_EXCLUDE_PATH". That seems counter-intuitive to me, as I 
would expect this feature to take absolute paths inside the target image 
to be excluded.

If the check is there to avoid "os.path.join()" returning the RHS only 
instead of the joined path, then please use "oe.path.join()" instead.

> +    import shutil
> +    from oe.path import copyhardlinktree
> +
> +    exclude_list = exclude_var.split()
> +
> +    rootfs_orig = d.getVar('IMAGE_ROOTFS')
> +    # We need a new rootfs directory we can delete files from. Copy to
> +    # workdir.
> +    new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname))
> +
> +    if os.path.lexists(new_rootfs):
> +        shutil.rmtree(os.path.join(new_rootfs))

Wouldn't it be sufficient to add "new_rootfs" to "do_image[cleandirs]" ?

> +
> +    copyhardlinktree(rootfs_orig, new_rootfs)
> +
> +    for orig_path in exclude_list:
> +        path = orig_path
> +        if os.path.isabs(path):
> +            bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path)
> +
> +        full_path = os.path.realpath(os.path.join(new_rootfs, path))
> +
> +        # Disallow climbing outside of parent directory using '..',
> +        # because doing so could be quite disastrous (we will delete the
> +        # directory).
> +        if not full_path.startswith(new_rootfs):
> +            bb.fatal("'%s' points to a path outside the rootfs" % orig_path)
> +
> +        if path.endswith(os.sep):
> +            # Delete content only.
> +            for entry in os.listdir(full_path):
> +                full_entry = os.path.join(full_path, entry)
> +                if os.path.isdir(full_entry) and not os.path.islink(full_entry):
> +                    shutil.rmtree(full_entry)
> +                else:
> +                    os.remove(full_entry)
> +        else:
> +            # Delete whole directory.
> +            shutil.rmtree(full_path)
> +
> +    # Save old value for cleanup later.
> +    d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig)
> +    d.setVar('IMAGE_ROOTFS', new_rootfs)
> +}
> +
> +python cleanup_excluded_directories() {
> +    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
> +    if not exclude_var:
> +        return
> +
> +    taskname = d.getVar("BB_CURRENTTASK")
> +
> +    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0':
> +        return
> +
> +    import shutil
> +
> +    rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS')
> +    rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG')
> +    # This should never happen, since we should have set it to a different
> +    # directory in the prepare function.
> +    assert rootfs_dirs_excluded != rootfs_orig
> +
> +    shutil.rmtree(rootfs_dirs_excluded)
> +    d.setVar('IMAGE_ROOTFS', rootfs_orig)
> +}

Are we sure the cleanup is needed? I can image cases where I would like 
to have a look in "rootfs_dirs_excluded" to see what how it differs from 
"rootfs_orig".

> +
>   #
>   # Compute the rootfs size
>   #
> diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
> index c736f92..8cd8fd5 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -201,6 +201,7 @@ IMAGE_CMD_multiubi () {
>   		multiubi_mkfs "${mkubifs_args}" "${ubinize_args}" "${name}"
>   	done
>   }
> +do_image_multiubi[respect_exclude_path] = "0"
>   
>   IMAGE_CMD_ubi () {
>   	multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}"
> diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
> index dcf620c..452d7f3 100644
> --- a/meta/classes/image_types_wic.bbclass
> +++ b/meta/classes/image_types_wic.bbclass
> @@ -34,6 +34,7 @@ IMAGE_CMD_wic () {
>   	rm -rf "$out/"
>   }
>   IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
> +do_image_wic[respect_exclude_path] = "0"
>   
>   # Rebuild when the wks file or vars in WICVARS change
>   USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}"
> 

-- 
MARTIN HUNDEBØLL, Prevas A/S
Software Developer

Hedeager 3, DK-8200 Aarhus N
Phone +45 87438070
Mobile +45 25562438
Martin.Hundeboll@prevas.dk
www.prevas.com


^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [PATCH v7] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2018-01-25 10:33                                             ` [PATCH v7] " Kristian Amlie
  2018-01-25 10:58                                               ` Martin Hundebøll
@ 2018-02-13  1:29                                               ` Cal Sullivan
  2018-03-15  9:29                                                 ` Kristian Amlie
  1 sibling, 1 reply; 41+ messages in thread
From: Cal Sullivan @ 2018-02-13  1:29 UTC (permalink / raw)
  To: Kristian Amlie, openembedded-core



On 01/25/2018 02:33 AM, Kristian Amlie wrote:
> This is a direct followup from the earlier 6602392db3d39 commit in
> wic. It works more or less the same way: The variable specifies a list
> of directories relative to the root of the rootfs, and these
> directories will be excluded from the resulting rootfs image. If an
> entry ends with a slash, only the contents are omitted, not the
> directory itself.
>
> Since the intended use of the variable is to exclude certain
> directories from the rootfs, and then include said directories in
> other partitions, it is not natural for this variable to be respected
> for image creators that create multi partition images. These can turn
> the feature off locally by defining:
>
>    do_image_myfs[respect_exclude_path] = "0"
>
> Specifically, "wic" and "multiubi" come with the feature disabled.
>
> Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>
> ---
>   meta/classes/image.bbclass           | 84 +++++++++++++++++++++++++++++++++++-
>   meta/classes/image_types.bbclass     |  1 +
>   meta/classes/image_types_wic.bbclass |  1 +
>   3 files changed, 84 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 4531aa2..849a19c 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -117,7 +117,8 @@ def rootfs_variables(d):
>                    'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
>                    'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
>                    'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
> -                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS']
> +                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS',
> +                 'IMAGE_ROOTFS_EXCLUDE_PATH']
>       variables.extend(rootfs_command_variables(d))
>       variables.extend(variable_depends(d))
>       return " ".join(variables)
> @@ -508,8 +509,9 @@ python () {
>           d.setVarFlag(task, 'func', '1')
>           d.setVarFlag(task, 'fakeroot', '1')
>   
> -        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size')
> +        d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size prepare_excluded_directories')
>           d.prependVarFlag(task, 'postfuncs', ' create_symlinks')
> +        d.appendVarFlag(task, 'postfuncs', ' cleanup_excluded_directories')
>           d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages))
>           d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps))
>           d.appendVarFlag(task, 'vardepsexclude', 'DATETIME DATE ' + ' '.join(vardepsexclude))
> @@ -518,6 +520,84 @@ python () {
>           bb.build.addtask(task, 'do_image_complete', after, d)
>   }
>   
> +python prepare_excluded_directories() {
> +    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
> +    if not exclude_var:
> +        return
> +
> +    taskname = d.getVar("BB_CURRENTTASK")
> +
> +    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0':
> +        bb.debug(1, "'IMAGE_ROOTFS_EXCLUDE_PATH' is set but 'respect_exclude_path' variable flag is 0 for this image type, so ignoring it")
> +        return
> +
> +    import shutil
> +    from oe.path import copyhardlinktree
> +
> +    exclude_list = exclude_var.split()
> +
> +    rootfs_orig = d.getVar('IMAGE_ROOTFS')
> +    # We need a new rootfs directory we can delete files from. Copy to
> +    # workdir.
> +    new_rootfs = os.path.realpath(os.path.join(d.getVar("WORKDIR"), "rootfs.%s" % taskname))
> +
> +    if os.path.lexists(new_rootfs):
> +        shutil.rmtree(os.path.join(new_rootfs))
> +
> +    copyhardlinktree(rootfs_orig, new_rootfs)
> +
> +    for orig_path in exclude_list:
> +        path = orig_path
> +        if os.path.isabs(path):
> +            bb.fatal("IMAGE_ROOTFS_EXCLUDE_PATH: Must be relative: %s" % orig_path)
> +
> +        full_path = os.path.realpath(os.path.join(new_rootfs, path))
> +
> +        # Disallow climbing outside of parent directory using '..',
> +        # because doing so could be quite disastrous (we will delete the
> +        # directory).
> +        if not full_path.startswith(new_rootfs):
> +            bb.fatal("'%s' points to a path outside the rootfs" % orig_path)
> +
> +        if path.endswith(os.sep):
We need to check if full_path exists here, else it may try modifying 
things that don't exist. E.g., if I have IMAGE_ROOTFS_EXCLUDE_PATH = 
"boot" in local.conf and do 'bitbake core-image-minimal-initramfs' I get 
the following:

ERROR: core-image-minimal-initramfs-1.0-r0 do_image_cpio: Error 
executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
      0001:
  *** 0002:prepare_excluded_directories(d)
      0003:
File: '/home/clsulliv/yocto/poky/meta/classes/image.bbclass', lineno: 
572, function: prepare_excluded_directories
      0568:                else:
      0569:                    os.remove(full_entry)
      0570:        else:
      0571:            # Delete whole directory.
  *** 0572:            shutil.rmtree(full_path)
      0573:
      0574:    # Save old value for cleanup later.
      0575:    d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig)
      0576:    d.setVar('IMAGE_ROOTFS', new_rootfs)
File: '/usr/lib64/python3.6/shutil.py', lineno: 471, function: rmtree
      0467:        # lstat()/open()/fstat() trick.
      0468:        try:
      0469:            orig_st = os.lstat(path)
      0470:        except Exception:
  *** 0471:            onerror(os.lstat, path, sys.exc_info())
      0472:            return
      0473:        try:
      0474:            fd = os.open(path, os.O_RDONLY)
      0475:        except Exception:
File: '/usr/lib64/python3.6/shutil.py', lineno: 469, function: rmtree
      0465:            path = os.fsdecode(path)
      0466:        # Note: To guard against symlink races, we use the 
standard
      0467:        # lstat()/open()/fstat() trick.
      0468:        try:
  *** 0469:            orig_st = os.lstat(path)
      0470:        except Exception:
      0471:            onerror(os.lstat, path, sys.exc_info())
      0472:            return
      0473:        try:
Exception: FileNotFoundError: [Errno 2] No such file or directory: 
'/home/clsulliv/yocto/poky/build/tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/rootfs.image_cpio/boot'

ERROR: core-image-minimal-initramfs-1.0-r0 do_image_cpio: Function 
failed: prepare_excluded_directories
ERROR: Logfile of failure stored in: 
/home/clsulliv/yocto/poky/build/tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/temp/log.do_image_cpio.23514
ERROR: Task 
(/home/clsulliv/yocto/poky/meta/recipes-core/images/core-image-minimal-initramfs.bb:do_image_cpio) 
failed with exit code '1'

Besides this, it works well. I'm using it here[1] as part of my attempt 
to improve EFI images.

[1]. 
https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=clsulliv/image-creation-exclude-path

Thanks,
Cal

> +            # Delete content only.
> +            for entry in os.listdir(full_path):
> +                full_entry = os.path.join(full_path, entry)
> +                if os.path.isdir(full_entry) and not os.path.islink(full_entry):
> +                    shutil.rmtree(full_entry)
> +                else:
> +                    os.remove(full_entry)
> +        else:
> +            # Delete whole directory.
> +            shutil.rmtree(full_path)
> +
> +    # Save old value for cleanup later.
> +    d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig)
> +    d.setVar('IMAGE_ROOTFS', new_rootfs)
> +}
> +
> +python cleanup_excluded_directories() {
> +    exclude_var = d.getVar('IMAGE_ROOTFS_EXCLUDE_PATH')
> +    if not exclude_var:
> +        return
> +
> +    taskname = d.getVar("BB_CURRENTTASK")
> +
> +    if d.getVarFlag('do_%s' % taskname, 'respect_exclude_path') == '0':
> +        return
> +
> +    import shutil
> +
> +    rootfs_dirs_excluded = d.getVar('IMAGE_ROOTFS')
> +    rootfs_orig = d.getVar('IMAGE_ROOTFS_ORIG')
> +    # This should never happen, since we should have set it to a different
> +    # directory in the prepare function.
> +    assert rootfs_dirs_excluded != rootfs_orig
> +
> +    shutil.rmtree(rootfs_dirs_excluded)
> +    d.setVar('IMAGE_ROOTFS', rootfs_orig)
> +}
> +
>   #
>   # Compute the rootfs size
>   #
> diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
> index c736f92..8cd8fd5 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -201,6 +201,7 @@ IMAGE_CMD_multiubi () {
>   		multiubi_mkfs "${mkubifs_args}" "${ubinize_args}" "${name}"
>   	done
>   }
> +do_image_multiubi[respect_exclude_path] = "0"
>   
>   IMAGE_CMD_ubi () {
>   	multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}"
> diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
> index dcf620c..452d7f3 100644
> --- a/meta/classes/image_types_wic.bbclass
> +++ b/meta/classes/image_types_wic.bbclass
> @@ -34,6 +34,7 @@ IMAGE_CMD_wic () {
>   	rm -rf "$out/"
>   }
>   IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
> +do_image_wic[respect_exclude_path] = "0"
>   
>   # Rebuild when the wks file or vars in WICVARS change
>   USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}"



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [PATCH v7] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature.
  2018-02-13  1:29                                               ` Cal Sullivan
@ 2018-03-15  9:29                                                 ` Kristian Amlie
  0 siblings, 0 replies; 41+ messages in thread
From: Kristian Amlie @ 2018-03-15  9:29 UTC (permalink / raw)
  To: Cal Sullivan, openembedded-core

On 13/02/18 02:29, Cal Sullivan wrote:
>> ...
>> +
>> +        if path.endswith(os.sep):
> We need to check if full_path exists here, else it may try modifying
> things that don't exist. E.g., if I have IMAGE_ROOTFS_EXCLUDE_PATH =
> "boot" in local.conf and do 'bitbake core-image-minimal-initramfs' I get
> the following:
> 
> ERROR: core-image-minimal-initramfs-1.0-r0 do_image_cpio: Error
> executing a python function in exec_python_func() autogenerated:
> 
> The stack trace of python calls that resulted in this exception/failure
> was:
> File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
>      0001:
>  *** 0002:prepare_excluded_directories(d)
>      0003:
> File: '/home/clsulliv/yocto/poky/meta/classes/image.bbclass', lineno:
> 572, function: prepare_excluded_directories
>      0568:                else:
>      0569:                    os.remove(full_entry)
>      0570:        else:
>      0571:            # Delete whole directory.
>  *** 0572:            shutil.rmtree(full_path)
>      0573:
>      0574:    # Save old value for cleanup later.
>      0575:    d.setVar('IMAGE_ROOTFS_ORIG', rootfs_orig)
>      0576:    d.setVar('IMAGE_ROOTFS', new_rootfs)
> File: '/usr/lib64/python3.6/shutil.py', lineno: 471, function: rmtree
>      0467:        # lstat()/open()/fstat() trick.
>      0468:        try:
>      0469:            orig_st = os.lstat(path)
>      0470:        except Exception:
>  *** 0471:            onerror(os.lstat, path, sys.exc_info())
>      0472:            return
>      0473:        try:
>      0474:            fd = os.open(path, os.O_RDONLY)
>      0475:        except Exception:
> File: '/usr/lib64/python3.6/shutil.py', lineno: 469, function: rmtree
>      0465:            path = os.fsdecode(path)
>      0466:        # Note: To guard against symlink races, we use the
> standard
>      0467:        # lstat()/open()/fstat() trick.
>      0468:        try:
>  *** 0469:            orig_st = os.lstat(path)
>      0470:        except Exception:
>      0471:            onerror(os.lstat, path, sys.exc_info())
>      0472:            return
>      0473:        try:
> Exception: FileNotFoundError: [Errno 2] No such file or directory:
> '/home/clsulliv/yocto/poky/build/tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/rootfs.image_cpio/boot'
> 
> 
> ERROR: core-image-minimal-initramfs-1.0-r0 do_image_cpio: Function
> failed: prepare_excluded_directories
> ERROR: Logfile of failure stored in:
> /home/clsulliv/yocto/poky/build/tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/temp/log.do_image_cpio.23514
> 
> ERROR: Task
> (/home/clsulliv/yocto/poky/meta/recipes-core/images/core-image-minimal-initramfs.bb:do_image_cpio)
> failed with exit code '1'
> 
> Besides this, it works well. I'm using it here[1] as part of my attempt
> to improve EFI images.
> 
> [1].
> https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=clsulliv/image-creation-exclude-path

Sorry for coming late to the party here. Nice find though, and the fix
looks good too.

Will you be pushing to get this merged in master? The process of getting
it merged has been quite slow until now, so I've been considering
implementing this downstream in the meta-mender layer instead, even
though I think it could benefit many OE/poky users.

-- 
Kristian


^ permalink raw reply	[flat|nested] 41+ messages in thread

end of thread, other threads:[~2018-03-15  9:29 UTC | newest]

Thread overview: 41+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-04-26 15:03 do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH Kristian Amlie
2017-04-26 15:03 ` [PATCH] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature Kristian Amlie
2017-05-22  7:08 ` do_image: Adding support for IMAGE_ROOTFS_EXCLUDE_PATH Kristian Amlie
2017-05-22  8:46   ` Ed Bartosh
2017-05-22  9:38     ` Kristian Amlie
2017-05-30  7:46       ` Kristian Amlie
2017-05-30  7:46         ` [PATCH v2] do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature Kristian Amlie
2017-06-19  9:01           ` Kristian Amlie
2017-06-19 14:01             ` Kristian Amlie
2017-06-19 14:01               ` [PATCH v3] " Kristian Amlie
2017-08-23 11:44                 ` [PATCH v4] " Kristian Amlie
2017-08-23 11:46                 ` Kristian Amlie
2017-08-23 11:46                   ` [PATCH v4] " Kristian Amlie
2017-08-23 12:39                     ` Kristian Amlie
2017-08-23 12:39                       ` [PATCH v5] " Kristian Amlie
2017-08-23 12:47                         ` Richard Purdie
2017-08-23 13:19                           ` Kristian Amlie
2017-08-28 15:47                             ` Kristian Amlie
2017-08-28 15:47                               ` [PATCH v6] " Kristian Amlie
2017-09-18  6:45                                 ` Kristian Amlie
2017-10-13  9:08                                   ` Kristian Amlie
2017-10-13  9:08                                     ` [PATCH v7] " Kristian Amlie
2017-10-13 10:37                                     ` [PATCH v6] " Alexander Kanavin
2017-10-13 11:22                                       ` Kristian Amlie
2017-11-22 13:13                                       ` Kristian Amlie
2017-11-22 13:13                                         ` [PATCH v7] " Kristian Amlie
2017-11-22 13:31                                         ` [PATCH v6] " Alexander Kanavin
2017-11-22 13:35                                           ` Kristian Amlie
2018-01-25 10:33                                           ` Kristian Amlie
2018-01-25 10:33                                             ` [PATCH v7] " Kristian Amlie
2018-01-25 10:58                                               ` Martin Hundebøll
2018-02-13  1:29                                               ` Cal Sullivan
2018-03-15  9:29                                                 ` Kristian Amlie
2017-08-23 12:46                       ` Richard Purdie
2017-08-23 13:19                         ` Kristian Amlie
2017-05-30  8:01       ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev2) Patchwork
2017-06-19 14:31       ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev3) Patchwork
2017-08-23 12:04       ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev4) Patchwork
2017-08-23 12:04       ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev5) Patchwork
2017-08-23 13:04       ` ✗ patchtest: failure for do_image: Implement IMAGE_ROOTFS_EXCLUDE_PATH feature. (rev6) Patchwork
2017-08-23 15:12         ` Leonardo Sandoval

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.