[PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Kristian Amlie]

It will omit the given path from the resulting partition, and if the
given path ends in a slash, it will only delete the content, and keep
the directory.

Since mkfs only accepts whole directories as input, we need to copy
the rootfs directory to the workdir so that we can selectively delete
files from it.

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
---
 scripts/lib/wic/help.py                  |  6 ++++
 scripts/lib/wic/ksparser.py              |  1 +
 scripts/lib/wic/partition.py             |  1 +
 scripts/lib/wic/plugins/source/rootfs.py | 51 +++++++++++++++++++++++++++++++-
 4 files changed, 58 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index e5347ec..9dab670 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -715,6 +715,12 @@ DESCRIPTION
                      partition table. It may be useful for
                      bootloaders.
 
+         --exclude-path: This option is specific to wic. It excludes the given
+                         absolute path from the resulting image. If the path
+                         ends with a slash, only the content of the directory
+                         is omitted, not the directory itself. This option only
+                         has an effect with the rootfs source plugin.
+
          --extra-space: This option is specific to wic. It adds extra
                         space after the space filled by the content
                         of the partition. The final size can go
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 0894e2b..17b97fd0 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -127,6 +127,7 @@ class KickStart():
         part.add_argument('mountpoint', nargs='?')
         part.add_argument('--active', action='store_true')
         part.add_argument('--align', type=int)
+        part.add_argument('--exclude-path', nargs='+')
         part.add_argument("--extra-space", type=sizetype, default=10*1024)
         part.add_argument('--fsoptions', dest='fsopts')
         part.add_argument('--fstype')
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index ac4c836..cba78a5 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -45,6 +45,7 @@ class Partition():
         self.align = args.align
         self.disk = args.disk
         self.extra_space = args.extra_space
+        self.exclude_path = args.exclude_path
         self.fsopts = args.fsopts
         self.fstype = args.fstype
         self.label = args.label
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 425da8b..d97d99c 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -26,10 +26,11 @@
 #
 
 import os
+import shutil
 
 from wic import msger
 from wic.pluginbase import SourcePlugin
-from wic.utils.oe.misc import get_bitbake_var
+from wic.utils.oe.misc import get_bitbake_var, exec_cmd
 
 class RootfsPlugin(SourcePlugin):
     """
@@ -78,6 +79,54 @@ class RootfsPlugin(SourcePlugin):
 
         real_rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
 
+        # Handle excluded paths.
+        if part.exclude_path is not None:
+            # We need a new rootfs directory we can delete files from. Copy to
+            # workdir.
+            new_rootfs = os.path.join(cr_workdir, "rootfs")
+
+            if os.path.lexists(new_rootfs):
+                shutil.rmtree(os.path.join(new_rootfs))
+
+            if os.stat(real_rootfs_dir).st_dev == os.stat(cr_workdir).st_dev:
+                # Optimization if both directories are on the same file system:
+                # copy using hardlinks.
+                cp_args = "-al"
+            else:
+                cp_args = "-a"
+            exec_cmd("cp %s %s %s" % (cp_args, real_rootfs_dir, new_rootfs))
+
+            real_rootfs_dir = new_rootfs
+
+            for orig_path in part.exclude_path:
+                path = orig_path
+                if not os.path.isabs(path):
+                    msger.error("Must be absolute: --exclude-path=%s" % orig_path)
+
+                while os.path.isabs(path):
+                    path = path[1:]
+
+                # Disallow '..', because doing so could be quite disastrous
+                # (we will delete the directory).
+                remaining = path
+                while True:
+                    (head, tail) = os.path.split(remaining)
+                    if tail == '..':
+                        msger.error("'..' not allowed: --exclude-path=%s" % orig_path)
+                    elif head == "":
+                        break
+                    remaining = head
+
+                full_path = os.path.join(new_rootfs, path)
+
+                if path.endswith(os.sep):
+                    # Delete content only.
+                    for entry in os.listdir(full_path):
+                        shutil.rmtree(os.path.join(full_path, entry))
+                else:
+                    # Delete whole directory.
+                    shutil.rmtree(full_path)
+
         part.rootfs_dir = real_rootfs_dir
         part.prepare_rootfs(cr_workdir, oe_builddir, real_rootfs_dir, native_sysroot)
 
-- 
2.7.4

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Patrick Ohly]

Hi!

Wow, that was fast :-)

On Fri, 2016-11-25 at 11:15 +0100, Kristian Amlie wrote:
> +            if os.stat(real_rootfs_dir).st_dev ==
> os.stat(cr_workdir).st_dev:
> +                # Optimization if both directories are on the same
> file system:
> +                # copy using hardlinks.
> +                cp_args = "-al"
> +            else:
> +                cp_args = "-a"
> +            exec_cmd("cp %s %s %s" % (cp_args, real_rootfs_dir,
> new_rootfs))

Not a full review (I'll leave that to Ed), just one thing which caught
my eye: when the rootfs contains xattrs, they get lost here.

Use oe.path.copyhardlinktree() instead, it also does the hardlinking
trick.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Kristian Amlie]

On 25/11/16 11:33, Patrick Ohly wrote:
> On Fri, 2016-11-25 at 11:15 +0100, Kristian Amlie wrote:
>> +            if os.stat(real_rootfs_dir).st_dev ==
>> os.stat(cr_workdir).st_dev:
>> +                # Optimization if both directories are on the same
>> file system:
>> +                # copy using hardlinks.
>> +                cp_args = "-al"
>> +            else:
>> +                cp_args = "-a"
>> +            exec_cmd("cp %s %s %s" % (cp_args, real_rootfs_dir,
>> new_rootfs))
> 
> Not a full review (I'll leave that to Ed), just one thing which caught
> my eye: when the rootfs contains xattrs, they get lost here.
> 
> Use oe.path.copyhardlinktree() instead, it also does the hardlinking
> trick.

Thanks, that's a good tip! I'll include that in the next patchset.

-- 
Kristian

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Ed Bartosh]

On Fri, Nov 25, 2016 at 01:07:34PM +0100, Kristian Amlie wrote:
> On 25/11/16 11:33, Patrick Ohly wrote:
> > On Fri, 2016-11-25 at 11:15 +0100, Kristian Amlie wrote:
> >> +            if os.stat(real_rootfs_dir).st_dev ==
> >> os.stat(cr_workdir).st_dev:
> >> +                # Optimization if both directories are on the same
> >> file system:
> >> +                # copy using hardlinks.
> >> +                cp_args = "-al"
> >> +            else:
> >> +                cp_args = "-a"
> >> +            exec_cmd("cp %s %s %s" % (cp_args, real_rootfs_dir,
> >> new_rootfs))
> > 
> > Not a full review (I'll leave that to Ed), just one thing which caught
> > my eye: when the rootfs contains xattrs, they get lost here.
> > 
> > Use oe.path.copyhardlinktree() instead, it also does the hardlinking
> > trick.
> 
> Thanks, that's a good tip! I'll include that in the next patchset.
> 

Thank you for so fast implementation!

Sorry for not answering on original e-mail. I've lost it somehow.

My comments so far:

What's the reason of insisting that path must be absolute?
May be it's just me, but I find it a bit scaring to use absolute path in .wks
The patch is relative to the rootfs directory from my point of view.

It also looks quite strange in the code to insist on absolute path
+                if not os.path.isabs(path):
+                    msger.error("Must be absolute: --exclude-path=%s" %

and then immediately making it relative:
+
+                while os.path.isabs(path):
+                    path = path[1:]


I know, this is just a matter of taste, but I'd not use brackets around
head, tail here. They're redundant and the code looks better without
them from my point of view.
+                    (head, tail) = os.path.split(remaining)

This causes rmtree to throw NotADirectoryError on files:
+                    for entry in os.listdir(full_path):
+                        shutil.rmtree(os.path.join(full_path, entry))

--
Regards,
Ed

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Kristian Amlie]

On 25/11/16 17:31, Ed Bartosh wrote:
> On Fri, Nov 25, 2016 at 01:07:34PM +0100, Kristian Amlie wrote:
>> On 25/11/16 11:33, Patrick Ohly wrote:
>>> On Fri, 2016-11-25 at 11:15 +0100, Kristian Amlie wrote:
>>>> +            if os.stat(real_rootfs_dir).st_dev ==
>>>> os.stat(cr_workdir).st_dev:
>>>> +                # Optimization if both directories are on the same
>>>> file system:
>>>> +                # copy using hardlinks.
>>>> +                cp_args = "-al"
>>>> +            else:
>>>> +                cp_args = "-a"
>>>> +            exec_cmd("cp %s %s %s" % (cp_args, real_rootfs_dir,
>>>> new_rootfs))
>>>
>>> Not a full review (I'll leave that to Ed), just one thing which caught
>>> my eye: when the rootfs contains xattrs, they get lost here.
>>>
>>> Use oe.path.copyhardlinktree() instead, it also does the hardlinking
>>> trick.
>>
>> Thanks, that's a good tip! I'll include that in the next patchset.
>>
> 
> Thank you for so fast implementation!
> 
> Sorry for not answering on original e-mail. I've lost it somehow.
> 
> My comments so far:
> 
> What's the reason of insisting that path must be absolute?
> May be it's just me, but I find it a bit scaring to use absolute path in .wks
> The patch is relative to the rootfs directory from my point of view.
> 
> It also looks quite strange in the code to insist on absolute path
> +                if not os.path.isabs(path):
> +                    msger.error("Must be absolute: --exclude-path=%s" %
> 
> and then immediately making it relative:
> +
> +                while os.path.isabs(path):
> +                    path = path[1:]

Not really any strong reason. I just thought it was a logical thing to
do from a user perspective: When you're making an image you're thinking
about paths in the final image, and the path after "part" is absolute,
so I thought this one should be too.

The fact that it's made relative in the code is just an implementation
detail to make join() work correctly.

I'm fine either way, so just let me know which you prefer.

> I know, this is just a matter of taste, but I'd not use brackets around
> head, tail here. They're redundant and the code looks better without
> them from my point of view.
> +                    (head, tail) = os.path.split(remaining)

Noted, I'll remove the brackets.

> This causes rmtree to throw NotADirectoryError on files:
> +                    for entry in os.listdir(full_path):
> +                        shutil.rmtree(os.path.join(full_path, entry))

Oh, nice catch. I will fix that one as well!

Thanks for the review!

-- 
Kristian

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Ed Bartosh]

On Mon, Nov 28, 2016 at 08:15:26AM +0100, Kristian Amlie wrote:
> > 
> > What's the reason of insisting that path must be absolute?
> > May be it's just me, but I find it a bit scaring to use absolute path in .wks
> > The patch is relative to the rootfs directory from my point of view.
> > 
> > It also looks quite strange in the code to insist on absolute path
> > +                if not os.path.isabs(path):
> > +                    msger.error("Must be absolute: --exclude-path=%s" %
> > 
> > and then immediately making it relative:
> > +
> > +                while os.path.isabs(path):
> > +                    path = path[1:]
> 
> Not really any strong reason. I just thought it was a logical thing to
> do from a user perspective: When you're making an image you're thinking
> about paths in the final image, and the path after "part" is absolute,
> so I thought this one should be too.
> 
> The fact that it's made relative in the code is just an implementation
> detail to make join() work correctly.
> 
> I'm fine either way, so just let me know which you prefer.
> 
I'd prefer relative path as I think about --exclude-path as a path in
the source rootfs directory that we want to exclude from copying to
the target partition. However, I agree that that it can be also treated
as a path to the target directory.

--
Regards,
Ed

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Kristian Amlie]

On 28/11/16 11:52, Ed Bartosh wrote:
> On Mon, Nov 28, 2016 at 08:15:26AM +0100, Kristian Amlie wrote:
>>>
>>> What's the reason of insisting that path must be absolute?
>>> May be it's just me, but I find it a bit scaring to use absolute path in .wks
>>> The patch is relative to the rootfs directory from my point of view.
>>>
>>> It also looks quite strange in the code to insist on absolute path
>>> +                if not os.path.isabs(path):
>>> +                    msger.error("Must be absolute: --exclude-path=%s" %
>>>
>>> and then immediately making it relative:
>>> +
>>> +                while os.path.isabs(path):
>>> +                    path = path[1:]
>>
>> Not really any strong reason. I just thought it was a logical thing to
>> do from a user perspective: When you're making an image you're thinking
>> about paths in the final image, and the path after "part" is absolute,
>> so I thought this one should be too.
>>
>> The fact that it's made relative in the code is just an implementation
>> detail to make join() work correctly.
>>
>> I'm fine either way, so just let me know which you prefer.
>>
> I'd prefer relative path as I think about --exclude-path as a path in
> the source rootfs directory that we want to exclude from copying to
> the target partition. However, I agree that that it can be also treated
> as a path to the target directory.

Ok!

I will get a new patchset up soon.

-- 
Kristian

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Ed Bartosh]

On Mon, Nov 28, 2016 at 12:01:09PM +0100, Kristian Amlie wrote:
> On 28/11/16 11:52, Ed Bartosh wrote:
> > On Mon, Nov 28, 2016 at 08:15:26AM +0100, Kristian Amlie wrote:
> >>>
> >>> What's the reason of insisting that path must be absolute?
> >>> May be it's just me, but I find it a bit scaring to use absolute path in .wks
> >>> The patch is relative to the rootfs directory from my point of view.
> >>>
> >>> It also looks quite strange in the code to insist on absolute path
> >>> +                if not os.path.isabs(path):
> >>> +                    msger.error("Must be absolute: --exclude-path=%s" %
> >>>
> >>> and then immediately making it relative:
> >>> +
> >>> +                while os.path.isabs(path):
> >>> +                    path = path[1:]
> >>
> >> Not really any strong reason. I just thought it was a logical thing to
> >> do from a user perspective: When you're making an image you're thinking
> >> about paths in the final image, and the path after "part" is absolute,
> >> so I thought this one should be too.
> >>
> >> The fact that it's made relative in the code is just an implementation
> >> detail to make join() work correctly.
> >>
> >> I'm fine either way, so just let me know which you prefer.
> >>
> > I'd prefer relative path as I think about --exclude-path as a path in
> > the source rootfs directory that we want to exclude from copying to
> > the target partition. However, I agree that that it can be also treated
> > as a path to the target directory.
> 
> Ok!
> 
> I will get a new patchset up soon.
> 

Great!

I'd appreciate if you also do the following before sending v2:
 - check that your code doesn't regress pylint checks
 - add test case for your code to meta/lib/oeqa/selftest/wic.py
 - check that your code doesn't break oe-selftest --coverage -r wic and
   doesn't regress test coverage

--
Regards,
Ed

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Kristian Amlie]

On 28/11/16 12:18, Ed Bartosh wrote:
> I'd appreciate if you also do the following before sending v2:
>  - check that your code doesn't regress pylint checks

Is there an official process here? Sorry, it's not very well documented.

Just running pylint3 manually on the scripts/lib/wic folder gives
identical results, however, running on rootfs.py alone, the rating
increases:

  Your code has been rated at 5.67/10 (previous run: 2.26/10, +3.41)

>  - add test case for your code to meta/lib/oeqa/selftest/wic.py
>  - check that your code doesn't break oe-selftest --coverage -r wic and
>    doesn't regress test coverage

In the interest of saving me some build time, am I correct in assuming
that I need the "qemux86-64" MACHINE target for this to work? It appears
to be dependent on syslinux which has COMPATIBLE_HOSTS set to non-arm
only. Sorry, I'm a bit new to this framework.

-- 
Kristian

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Maciej Borzęcki]

On Wed, Nov 30, 2016 at 2:30 PM, Kristian Amlie
<kristian.amlie@mender.io> wrote:
> On 28/11/16 12:18, Ed Bartosh wrote:
>> I'd appreciate if you also do the following before sending v2:
>>  - check that your code doesn't regress pylint checks
>
> Is there an official process here? Sorry, it's not very well documented.
>
> Just running pylint3 manually on the scripts/lib/wic folder gives
> identical results, however, running on rootfs.py alone, the rating
> increases:
>
>   Your code has been rated at 5.67/10 (previous run: 2.26/10, +3.41)
>
>>  - add test case for your code to meta/lib/oeqa/selftest/wic.py
>>  - check that your code doesn't break oe-selftest --coverage -r wic and
>>    doesn't regress test coverage
>
> In the interest of saving me some build time, am I correct in assuming
> that I need the "qemux86-64" MACHINE target for this to work? It appears
> to be dependent on syslinux which has COMPATIBLE_HOSTS set to non-arm
> only. Sorry, I'm a bit new to this framework.

You can either build for qemux86-64 or you can try patches from this series:
http://lists.openembedded.org/pipermail/openembedded-core/2016-November/129187.html

Basically, you should be interested in these:

  oe-selftest: enforce en_US.UTF-8 locale
  oeqa/utils/commands.py: allow use of binaries from native sysroot
  wic: selftest: avoid COMPATIBLE_HOST issues
  wic: selftest: do not assume bzImage kernel image

This is how you add a test:
http://lists.openembedded.org/pipermail/openembedded-core/2016-November/129192.html

Running tests is as simple as:

 oe-selftest -r wic.Wic.test_your_test  # runs single test
 oe-selftest -r wic.Wic                 # runs all wic tests

Regards,
-- 
Maciej Borzecki
RnDity

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Ed Bartosh]

On Wed, Nov 30, 2016 at 02:30:12PM +0100, Kristian Amlie wrote:
> On 28/11/16 12:18, Ed Bartosh wrote:
> > I'd appreciate if you also do the following before sending v2:
> >  - check that your code doesn't regress pylint checks
> 
> Is there an official process here? Sorry, it's not very well documented.
No official process yet, I'm sorry. I can put it on wiki if it helps.

> Just running pylint3 manually on the scripts/lib/wic folder gives
> identical results, however, running on rootfs.py alone, the rating
> increases:
> 
>   Your code has been rated at 5.67/10 (previous run: 2.26/10, +3.41)
Looks good to me. Thanks.
Running pylint on the files you've changed before and after the changes should be enough.

> >  - add test case for your code to meta/lib/oeqa/selftest/wic.py
> >  - check that your code doesn't break oe-selftest --coverage -r wic and
> >    doesn't regress test coverage
> 
> In the interest of saving me some build time, am I correct in assuming
> that I need the "qemux86-64" MACHINE target for this to work? It appears
> to be dependent on syslinux which has COMPATIBLE_HOSTS set to non-arm
> only. Sorry, I'm a bit new to this framework.
> 

You're right. oe-selftest requires qemux86-64 target.

Even if you usually work in different environment it makes sense to run
oe-selftest to ensure that your changes don't breake it.
This is because our maintainers use oe-selftest results as one of the
acceptance criterias. It means that your changes would be be rejected anyway
if they break oe-selftest, so it's better to check it in advance.

--
Regards,
Ed

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Paul Eggleton]

On Wed, 30 Nov 2016 17:29:23 Ed Bartosh wrote:
> On Wed, Nov 30, 2016 at 02:30:12PM +0100, Kristian Amlie wrote:
> > In the interest of saving me some build time, am I correct in assuming
> > that I need the "qemux86-64" MACHINE target for this to work? It appears
> > to be dependent on syslinux which has COMPATIBLE_HOSTS set to non-arm
> > only. Sorry, I'm a bit new to this framework.
> 
> You're right. oe-selftest requires qemux86-64 target.

That's definitely not a design decision of oe-selftest. If we have certain 
tests that can only work with a particular configuration, those tests need to 
be explicitly skipped if that configuration is not active.

Cheers,
Paul


-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Kristian Amlie]

On 30/11/16 16:29, Ed Bartosh wrote:
>> ...
> 
> You're right. oe-selftest requires qemux86-64 target.
> 
> Even if you usually work in different environment it makes sense to run
> oe-selftest to ensure that your changes don't breake it.
> This is because our maintainers use oe-selftest results as one of the
> acceptance criterias. It means that your changes would be be rejected anyway
> if they break oe-selftest, so it's better to check it in advance.

This is taking some time due to unrelated issues, but in the meantime:

What are the thoughts on having a similar --exclude-path mechanism for
the standard rootfs building? IOW, the do_image_ext4 and friends?

Our usecase is that we use wic to build a complete image, using
--exclude-path to group things into partitions. And then we want to
build a regular ext4 image and use this as an update to the device, and
in this case we want the same content as wic generates, but in a raw
rootfs file instead.

As an alternative to do_image_ext4, it might be possible to use wic to
build a rootfs raw filesystem instead of a partitioned file, but I
haven't looked into that yet.

Thoughts?

-- 
Kristian

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Ed Bartosh]

On Fri, Dec 02, 2016 at 03:36:47PM +0100, Kristian Amlie wrote:
> On 30/11/16 16:29, Ed Bartosh wrote:
> >> ...
> > 
> > You're right. oe-selftest requires qemux86-64 target.
> > 
> > Even if you usually work in different environment it makes sense to run
> > oe-selftest to ensure that your changes don't breake it.
> > This is because our maintainers use oe-selftest results as one of the
> > acceptance criterias. It means that your changes would be be rejected anyway
> > if they break oe-selftest, so it's better to check it in advance.
> 
> This is taking some time due to unrelated issues, but in the meantime:
> 
> What are the thoughts on having a similar --exclude-path mechanism for
> the standard rootfs building? IOW, the do_image_ext4 and friends?

That's a great idea! If source rootfs direcotry contains only requred
content then we don't even need to change wic code. It's much better
then implementing the same functionality in wic and in oe code.

> Our usecase is that we use wic to build a complete image, using
> --exclude-path to group things into partitions. And then we want to
> build a regular ext4 image and use this as an update to the device, and
> in this case we want the same content as wic generates, but in a raw
> rootfs file instead.

--
Regards,
Ed

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Kristian Amlie]

On 02/12/16 16:05, Ed Bartosh wrote:
> On Fri, Dec 02, 2016 at 03:36:47PM +0100, Kristian Amlie wrote:
>> What are the thoughts on having a similar --exclude-path mechanism for
>> the standard rootfs building? IOW, the do_image_ext4 and friends?
> 
> That's a great idea! If source rootfs direcotry contains only requred
> content then we don't even need to change wic code. It's much better
> then implementing the same functionality in wic and in oe code.

Right, I didn't even consider it like that. But yeah, theoretically we
could drop the wic variant if you only use fsimage in your wks file.

But I think they sort of complement each other though. Since the rootfs
plugin does exist in wic, it makes sense that it would also support
--exclude-path. If you have a partition setup like this:

part1: Everything except /usr
part2: Everything inside /usr except /usr/share
part3: /usr/share

This would not be possible without --exclude-path support in wic, since
part2 would have to be constructed using the wic rootfs plugin, but
would need to exclude /usr/share.

In any case, I will definitely look at implementing the do_image_ext4
variant as well. I'll post more info about that when I have it.

-- 
Kristian

[PATCH v2] wic: Add --exclude-path option to rootfs source plugin.

[Kristian Amlie]

Alright, this took a bit longer than I expected, I had to learn a bit about the
test framework in OpenEmbedded. But here is the updated patch with additional
patches enabling tests.

Two things I'm a bit unsure about: I had to add a path to the wic python script
in order to use the copyhardlinktree() function that was suggested earlier. And
I also had to modify the path in the test itself in order to use the e2ls tool
which I provided in a recipe in order to inspect the image. I hope those things
are ok.

-- 
Kristian

[PATCH v2 1/3] wic: Add --exclude-path option to rootfs source plugin.

[Kristian Amlie]

It will omit the given path from the resulting partition, and if the
given path ends in a slash, it will only delete the content, and keep
the directory.

Since mkfs only accepts whole directories as input, we need to copy
the rootfs directory to the workdir so that we can selectively delete
files from it.

Since we want to use the copyhardlinktree() function, we need to put
the generic oe lib in the module search path.

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
---
 scripts/lib/wic/help.py                  |  6 +++++
 scripts/lib/wic/ksparser.py              |  1 +
 scripts/lib/wic/partition.py             |  1 +
 scripts/lib/wic/plugins/source/rootfs.py | 43 +++++++++++++++++++++++++++++++-
 scripts/wic                              |  2 ++
 5 files changed, 52 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index e5347ec..9dab670 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -715,6 +715,12 @@ DESCRIPTION
                      partition table. It may be useful for
                      bootloaders.
 
+         --exclude-path: This option is specific to wic. It excludes the given
+                         absolute path from the resulting image. If the path
+                         ends with a slash, only the content of the directory
+                         is omitted, not the directory itself. This option only
+                         has an effect with the rootfs source plugin.
+
          --extra-space: This option is specific to wic. It adds extra
                         space after the space filled by the content
                         of the partition. The final size can go
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 0894e2b..17b97fd 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -127,6 +127,7 @@ class KickStart():
         part.add_argument('mountpoint', nargs='?')
         part.add_argument('--active', action='store_true')
         part.add_argument('--align', type=int)
+        part.add_argument('--exclude-path', nargs='+')
         part.add_argument("--extra-space", type=sizetype, default=10*1024)
         part.add_argument('--fsoptions', dest='fsopts')
         part.add_argument('--fstype')
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index ac4c836..cba78a5 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -45,6 +45,7 @@ class Partition():
         self.align = args.align
         self.disk = args.disk
         self.extra_space = args.extra_space
+        self.exclude_path = args.exclude_path
         self.fsopts = args.fsopts
         self.fstype = args.fstype
         self.label = args.label
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 425da8b..794ad69 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -26,10 +26,13 @@
 #
 
 import os
+import shutil
+
+from oe.path import copyhardlinktree
 
 from wic import msger
 from wic.pluginbase import SourcePlugin
-from wic.utils.oe.misc import get_bitbake_var
+from wic.utils.oe.misc import get_bitbake_var, exec_cmd
 
 class RootfsPlugin(SourcePlugin):
     """
@@ -78,6 +81,44 @@ class RootfsPlugin(SourcePlugin):
 
         real_rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
 
+        # Handle excluded paths.
+        if part.exclude_path is not None:
+            # We need a new rootfs directory we can delete files from. Copy to
+            # workdir.
+            new_rootfs = os.path.join(cr_workdir, "rootfs")
+
+            if os.path.lexists(new_rootfs):
+                shutil.rmtree(os.path.join(new_rootfs))
+
+            copyhardlinktree(real_rootfs_dir, new_rootfs)
+
+            real_rootfs_dir = new_rootfs
+
+            for orig_path in part.exclude_path:
+                path = orig_path
+                if os.path.isabs(path):
+                    msger.error("Must be relative: --exclude-path=%s" % orig_path)
+
+                full_path = os.path.realpath(os.path.join(new_rootfs, path))
+
+                # Disallow climbing outside of parent directory using '..',
+                # because doing so could be quite disastrous (we will delete the
+                # directory).
+                if not full_path.startswith(new_rootfs):
+                    msger.error("'%s' points to a path outside the rootfs" % orig_path)
+
+                if path.endswith(os.sep):
+                    # Delete content only.
+                    for entry in os.listdir(full_path):
+                        full_entry = os.path.join(full_path, entry)
+                        if os.path.isdir(full_entry) and not os.path.islink(full_entry):
+                            shutil.rmtree(full_entry)
+                        else:
+                            os.remove(full_entry)
+                else:
+                    # Delete whole directory.
+                    shutil.rmtree(full_path)
+
         part.rootfs_dir = real_rootfs_dir
         part.prepare_rootfs(cr_workdir, oe_builddir, real_rootfs_dir, native_sysroot)
 
diff --git a/scripts/wic b/scripts/wic
index fe2c33f..f58843e 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -41,6 +41,8 @@ from distutils import spawn
 scripts_path = os.path.abspath(os.path.dirname(__file__))
 lib_path = scripts_path + '/lib'
 sys.path.insert(0, lib_path)
+oe_lib_path = os.path.join(os.path.dirname(scripts_path), 'meta', 'lib')
+sys.path.insert(0, oe_lib_path)
 
 bitbake_exe = spawn.find_executable('bitbake')
 if bitbake_exe:
-- 
2.7.4

[PATCH v2 2/3] Add e2tools recipe, in order to test contents of images.

[Kristian Amlie]

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
---
 meta/recipes-devtools/e2tools/e2tools-native_0.0.16.bb | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
 create mode 100644 meta/recipes-devtools/e2tools/e2tools-native_0.0.16.bb

diff --git a/meta/recipes-devtools/e2tools/e2tools-native_0.0.16.bb b/meta/recipes-devtools/e2tools/e2tools-native_0.0.16.bb
new file mode 100644
index 0000000..5483287
--- /dev/null
+++ b/meta/recipes-devtools/e2tools/e2tools-native_0.0.16.bb
@@ -0,0 +1,17 @@
+SRC_URI = "http://home.earthlink.net/~k_sheff/sw/e2tools/e2tools-0.0.16.tar.gz;md5sum=1829b2b261e0e0d07566066769b5b28b"
+LICENSE = "GPL-2.0"
+LIC_FILES_CHKSUM = "file://COPYING;md5=fa8321a71778d26ff40690a4d371ea85"
+
+inherit native
+
+do_configure() {
+    ./configure --prefix=${prefix}
+}
+
+do_compile() {
+    oe_runmake
+}
+
+do_install() {
+    oe_runmake DESTDIR=${D} install
+}
-- 
2.7.4

[PATCH v2 3/3] selftest/wic: Add tests for --exclude-dir option.

[Kristian Amlie]

Based partially on an earlier patch by Maciej Borzecki.

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
---
 meta/lib/oeqa/selftest/wic.py | 106 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 105 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index faac11e..b2b0fe8 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -49,7 +49,8 @@ class Wic(oeSelfTest):
         # setUpClass being unavailable.
         if not Wic.image_is_ready:
             bitbake('syslinux syslinux-native parted-native gptfdisk-native '
-                    'dosfstools-native mtools-native bmap-tools-native')
+                    'dosfstools-native mtools-native bmap-tools-native '
+                    'e2tools-native')
             bitbake('core-image-minimal')
             Wic.image_is_ready = True
 
@@ -299,3 +300,106 @@ class Wic(oeSelfTest):
         self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
                                    % image).status)
         self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
+
+    def test_exclude_path(self):
+        """Test --exclude-path wks option."""
+
+        # For using 'e2ls'.
+        old_path = os.environ['PATH']
+        os.environ['PATH'] = get_bb_var('PATH', 'core-image-minimal')
+
+        wks_file = 'temp.wks'
+        ks = open(wks_file, 'w')
+        rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
+        ks.write("""part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path usr
+part /usr --source rootfs --ondisk mmcblk0 --fstype=ext4 --rootfs-dir %s/usr
+part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --rootfs-dir %s/usr"""
+                 % (rootfs_dir, rootfs_dir))
+        ks.close()
+        self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
+                                   % wks_file).status)
+
+        os.remove(wks_file)
+        wicout = glob(self.resultdir + "%s-*direct" % 'temp')
+        self.assertEqual(1, len(wicout))
+
+        wicimg = wicout[0]
+
+        # verify partition size with wic
+        res = runCmd("parted -m %s unit b p 2>/dev/null" % wicimg)
+        self.assertEqual(0, res.status)
+
+        # parse parted output which looks like this:
+        # BYT;\n
+        # /var/tmp/wic/build/tmpfwvjjkf_-201611101222-hda.direct:200MiB:file:512:512:msdos::;\n
+        # 1:0.00MiB:200MiB:200MiB:ext4::;\n
+        partlns = res.output.splitlines()[2:]
+
+        self.assertEqual(3, len(partlns))
+
+        for part in [1, 2, 3]:
+            part_file = os.path.join(self.resultdir, "selftest_img.part%d" % part)
+            partln = partlns[part-1].split(":")
+            self.assertEqual(7, len(partln))
+            start = int(partln[1].rstrip("B")) / 512
+            length = int(partln[3].rstrip("B")) / 512
+            self.assertEqual(0, runCmd("dd if=%s of=%s skip=%d count=%d" %
+                                       (wicimg, part_file, start, length)).status)
+
+        # Test partition 1, should contain the normal root directories, except
+        # /usr.
+        res = runCmd("e2ls %s" % os.path.join(self.resultdir, "selftest_img.part1"))
+        self.assertEqual(0, res.status)
+        files = res.output.split()
+        self.assertIn("etc", files)
+        self.assertNotIn("usr", files)
+
+        # Partition 2, should contain common directories for /usr, not root
+        # directories.
+        res = runCmd("e2ls %s" % os.path.join(self.resultdir, "selftest_img.part2"))
+        self.assertEqual(0, res.status)
+        files = res.output.split()
+        self.assertNotIn("etc", files)
+        self.assertNotIn("usr", files)
+        self.assertIn("share", files)
+
+        # Partition 3, should contain the same as partition 2, including the bin
+        # directory, but not the files inside it.
+        res = runCmd("e2ls %s" % os.path.join(self.resultdir, "selftest_img.part3"))
+        self.assertEqual(0, res.status)
+        files = res.output.split()
+        self.assertNotIn("etc", files)
+        self.assertNotIn("usr", files)
+        self.assertIn("share", files)
+        self.assertIn("bin", files)
+        res = runCmd("e2ls %s:bin" % os.path.join(self.resultdir, "selftest_img.part3"))
+        self.assertEqual(0, res.status)
+        self.assertEqual("No files found!", res.output.strip())
+
+        for part in [1, 2, 3]:
+            part_file = os.path.join(self.resultdir, "selftest_img.part%d" % part)
+            os.remove(part_file)
+
+        os.environ['PATH'] = old_path
+
+    def test_exclude_path_errors(self):
+        """Test --exclude-path wks option error handling."""
+        wks_file = 'temp.wks'
+
+        rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
+
+        # Absolute argument.
+        ks = open(wks_file, 'w')
+        ks.write("part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path /usr")
+        ks.close()
+        self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal" \
+                                      % wks_file, ignore_status=True).status)
+        os.remove(wks_file)
+
+        # Argument pointing to parent directory.
+        ks = open(wks_file, 'w')
+        ks.write("part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path ././..")
+        ks.close()
+        self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal" \
+                                      % wks_file, ignore_status=True).status)
+        os.remove(wks_file)
-- 
2.7.4

Re: [PATCH v2 3/3] selftest/wic: Add tests for --exclude-dir option.

[Ed Bartosh]

On Wed, Dec 14, 2016 at 05:28:54PM +0100, Kristian Amlie wrote:
> Based partially on an earlier patch by Maciej Borzecki.
> 
> Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
> ---
>  meta/lib/oeqa/selftest/wic.py | 106 +++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 105 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
> index faac11e..b2b0fe8 100644
> --- a/meta/lib/oeqa/selftest/wic.py
> +++ b/meta/lib/oeqa/selftest/wic.py
> @@ -49,7 +49,8 @@ class Wic(oeSelfTest):
>          # setUpClass being unavailable.
>          if not Wic.image_is_ready:
>              bitbake('syslinux syslinux-native parted-native gptfdisk-native '
> -                    'dosfstools-native mtools-native bmap-tools-native')
> +                    'dosfstools-native mtools-native bmap-tools-native '
> +                    'e2tools-native')
>              bitbake('core-image-minimal')
>              Wic.image_is_ready = True
>  
> @@ -299,3 +300,106 @@ class Wic(oeSelfTest):
>          self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
>                                     % image).status)
>          self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
> +
> +    def test_exclude_path(self):
> +        """Test --exclude-path wks option."""
> +
> +        # For using 'e2ls'.
> +        old_path = os.environ['PATH']
> +        os.environ['PATH'] = get_bb_var('PATH', 'core-image-minimal')
> +
> +        wks_file = 'temp.wks'
> +        ks = open(wks_file, 'w')
I'd use more pythonic 'with open(wks_file, 'w') as wks:' here.

> +        rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
> +        ks.write("""part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path usr
> +part /usr --source rootfs --ondisk mmcblk0 --fstype=ext4 --rootfs-dir %s/usr
> +part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --rootfs-dir %s/usr"""
> +                 % (rootfs_dir, rootfs_dir))
> +        ks.close()
> +        self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
> +                                   % wks_file).status)
> +
> +        os.remove(wks_file)
> +        wicout = glob(self.resultdir + "%s-*direct" % 'temp')
> +        self.assertEqual(1, len(wicout))
> +
> +        wicimg = wicout[0]
> +
> +        # verify partition size with wic
> +        res = runCmd("parted -m %s unit b p 2>/dev/null" % wicimg)
> +        self.assertEqual(0, res.status)
> +
> +        # parse parted output which looks like this:
> +        # BYT;\n
> +        # /var/tmp/wic/build/tmpfwvjjkf_-201611101222-hda.direct:200MiB:file:512:512:msdos::;\n
> +        # 1:0.00MiB:200MiB:200MiB:ext4::;\n
> +        partlns = res.output.splitlines()[2:]
> +
> +        self.assertEqual(3, len(partlns))
> +
> +        for part in [1, 2, 3]:
> +            part_file = os.path.join(self.resultdir, "selftest_img.part%d" % part)
> +            partln = partlns[part-1].split(":")
> +            self.assertEqual(7, len(partln))
> +            start = int(partln[1].rstrip("B")) / 512
> +            length = int(partln[3].rstrip("B")) / 512
> +            self.assertEqual(0, runCmd("dd if=%s of=%s skip=%d count=%d" %
> +                                       (wicimg, part_file, start, length)).status)
> +
> +        # Test partition 1, should contain the normal root directories, except
> +        # /usr.
> +        res = runCmd("e2ls %s" % os.path.join(self.resultdir, "selftest_img.part1"))
> +        self.assertEqual(0, res.status)
> +        files = res.output.split()
> +        self.assertIn("etc", files)
> +        self.assertNotIn("usr", files)
> +
> +        # Partition 2, should contain common directories for /usr, not root
> +        # directories.
> +        res = runCmd("e2ls %s" % os.path.join(self.resultdir, "selftest_img.part2"))
> +        self.assertEqual(0, res.status)
> +        files = res.output.split()
> +        self.assertNotIn("etc", files)
> +        self.assertNotIn("usr", files)
> +        self.assertIn("share", files)
> +
> +        # Partition 3, should contain the same as partition 2, including the bin
> +        # directory, but not the files inside it.
> +        res = runCmd("e2ls %s" % os.path.join(self.resultdir, "selftest_img.part3"))
> +        self.assertEqual(0, res.status)
> +        files = res.output.split()
> +        self.assertNotIn("etc", files)
> +        self.assertNotIn("usr", files)
> +        self.assertIn("share", files)
> +        self.assertIn("bin", files)
> +        res = runCmd("e2ls %s:bin" % os.path.join(self.resultdir, "selftest_img.part3"))
> +        self.assertEqual(0, res.status)
> +        self.assertEqual("No files found!", res.output.strip())
> +
> +        for part in [1, 2, 3]:
> +            part_file = os.path.join(self.resultdir, "selftest_img.part%d" % part)
> +            os.remove(part_file)
> +
> +        os.environ['PATH'] = old_path
> +
> +    def test_exclude_path_errors(self):
> +        """Test --exclude-path wks option error handling."""
> +        wks_file = 'temp.wks'
> +
> +        rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
> +
> +        # Absolute argument.
> +        ks = open(wks_file, 'w')
and here
> +        ks.write("part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path /usr")
> +        ks.close()
> +        self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal" \
> +                                      % wks_file, ignore_status=True).status)
> +        os.remove(wks_file)
> +
> +        # Argument pointing to parent directory.
> +        ks = open(wks_file, 'w')
and here
> +        ks.write("part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path ././..")
> +        ks.close()
> +        self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal" \
> +                                      % wks_file, ignore_status=True).status)
> +        os.remove(wks_file)

--
Regards,
Ed

[PATCH v2 3/3] selftest/wic: Add tests for --exclude-dir option.

[Kristian Amlie]

>> +    def test_exclude_path(self):
>> +        """Test --exclude-path wks option."""
>> +
>> +        # For using 'e2ls'.
>> +        old_path = os.environ['PATH']
>> +        os.environ['PATH'] = get_bb_var('PATH', 'core-image-minimal')
>> +
>> +        wks_file = 'temp.wks'
>> +        ks = open(wks_file, 'w')
> 
> I'd use more pythonic 'with open(wks_file, 'w') as wks:' here.

Done for all three blocks!

-- 
Kristian

[PATCH v3 1/3] wic: Add --exclude-path option to rootfs source plugin.

[Kristian Amlie]

It will omit the given path from the resulting partition, and if the
given path ends in a slash, it will only delete the content, and keep
the directory.

Since mkfs only accepts whole directories as input, we need to copy
the rootfs directory to the workdir so that we can selectively delete
files from it.

Since we want to use the copyhardlinktree() function, we need to put
the generic oe lib in the module search path.

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
---
 scripts/lib/wic/help.py                  |  6 +++++
 scripts/lib/wic/ksparser.py              |  1 +
 scripts/lib/wic/partition.py             |  1 +
 scripts/lib/wic/plugins/source/rootfs.py | 43 +++++++++++++++++++++++++++++++-
 scripts/wic                              |  2 ++
 5 files changed, 52 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index e5347ec..9dab670 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -715,6 +715,12 @@ DESCRIPTION
                      partition table. It may be useful for
                      bootloaders.
 
+         --exclude-path: This option is specific to wic. It excludes the given
+                         absolute path from the resulting image. If the path
+                         ends with a slash, only the content of the directory
+                         is omitted, not the directory itself. This option only
+                         has an effect with the rootfs source plugin.
+
          --extra-space: This option is specific to wic. It adds extra
                         space after the space filled by the content
                         of the partition. The final size can go
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 0894e2b..17b97fd 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -127,6 +127,7 @@ class KickStart():
         part.add_argument('mountpoint', nargs='?')
         part.add_argument('--active', action='store_true')
         part.add_argument('--align', type=int)
+        part.add_argument('--exclude-path', nargs='+')
         part.add_argument("--extra-space", type=sizetype, default=10*1024)
         part.add_argument('--fsoptions', dest='fsopts')
         part.add_argument('--fstype')
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index ac4c836..cba78a5 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -45,6 +45,7 @@ class Partition():
         self.align = args.align
         self.disk = args.disk
         self.extra_space = args.extra_space
+        self.exclude_path = args.exclude_path
         self.fsopts = args.fsopts
         self.fstype = args.fstype
         self.label = args.label
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 425da8b..794ad69 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -26,10 +26,13 @@
 #
 
 import os
+import shutil
+
+from oe.path import copyhardlinktree
 
 from wic import msger
 from wic.pluginbase import SourcePlugin
-from wic.utils.oe.misc import get_bitbake_var
+from wic.utils.oe.misc import get_bitbake_var, exec_cmd
 
 class RootfsPlugin(SourcePlugin):
     """
@@ -78,6 +81,44 @@ class RootfsPlugin(SourcePlugin):
 
         real_rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
 
+        # Handle excluded paths.
+        if part.exclude_path is not None:
+            # We need a new rootfs directory we can delete files from. Copy to
+            # workdir.
+            new_rootfs = os.path.join(cr_workdir, "rootfs")
+
+            if os.path.lexists(new_rootfs):
+                shutil.rmtree(os.path.join(new_rootfs))
+
+            copyhardlinktree(real_rootfs_dir, new_rootfs)
+
+            real_rootfs_dir = new_rootfs
+
+            for orig_path in part.exclude_path:
+                path = orig_path
+                if os.path.isabs(path):
+                    msger.error("Must be relative: --exclude-path=%s" % orig_path)
+
+                full_path = os.path.realpath(os.path.join(new_rootfs, path))
+
+                # Disallow climbing outside of parent directory using '..',
+                # because doing so could be quite disastrous (we will delete the
+                # directory).
+                if not full_path.startswith(new_rootfs):
+                    msger.error("'%s' points to a path outside the rootfs" % orig_path)
+
+                if path.endswith(os.sep):
+                    # Delete content only.
+                    for entry in os.listdir(full_path):
+                        full_entry = os.path.join(full_path, entry)
+                        if os.path.isdir(full_entry) and not os.path.islink(full_entry):
+                            shutil.rmtree(full_entry)
+                        else:
+                            os.remove(full_entry)
+                else:
+                    # Delete whole directory.
+                    shutil.rmtree(full_path)
+
         part.rootfs_dir = real_rootfs_dir
         part.prepare_rootfs(cr_workdir, oe_builddir, real_rootfs_dir, native_sysroot)
 
diff --git a/scripts/wic b/scripts/wic
index fe2c33f..f58843e 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -41,6 +41,8 @@ from distutils import spawn
 scripts_path = os.path.abspath(os.path.dirname(__file__))
 lib_path = scripts_path + '/lib'
 sys.path.insert(0, lib_path)
+oe_lib_path = os.path.join(os.path.dirname(scripts_path), 'meta', 'lib')
+sys.path.insert(0, oe_lib_path)
 
 bitbake_exe = spawn.find_executable('bitbake')
 if bitbake_exe:
-- 
2.7.4

[PATCH v3 2/3] Add e2tools recipe, in order to test contents of images.

[Kristian Amlie]

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
---
 meta/recipes-devtools/e2tools/e2tools-native_0.0.16.bb | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
 create mode 100644 meta/recipes-devtools/e2tools/e2tools-native_0.0.16.bb

diff --git a/meta/recipes-devtools/e2tools/e2tools-native_0.0.16.bb b/meta/recipes-devtools/e2tools/e2tools-native_0.0.16.bb
new file mode 100644
index 0000000..5483287
--- /dev/null
+++ b/meta/recipes-devtools/e2tools/e2tools-native_0.0.16.bb
@@ -0,0 +1,17 @@
+SRC_URI = "http://home.earthlink.net/~k_sheff/sw/e2tools/e2tools-0.0.16.tar.gz;md5sum=1829b2b261e0e0d07566066769b5b28b"
+LICENSE = "GPL-2.0"
+LIC_FILES_CHKSUM = "file://COPYING;md5=fa8321a71778d26ff40690a4d371ea85"
+
+inherit native
+
+do_configure() {
+    ./configure --prefix=${prefix}
+}
+
+do_compile() {
+    oe_runmake
+}
+
+do_install() {
+    oe_runmake DESTDIR=${D} install
+}
-- 
2.7.4

[PATCH v3 3/3] selftest/wic: Add tests for --exclude-dir option.

[Kristian Amlie]

Based partially on an earlier patch by Maciej Borzecki.

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
---
 meta/lib/oeqa/selftest/wic.py | 103 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 102 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index faac11e..09f475c 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -49,7 +49,8 @@ class Wic(oeSelfTest):
         # setUpClass being unavailable.
         if not Wic.image_is_ready:
             bitbake('syslinux syslinux-native parted-native gptfdisk-native '
-                    'dosfstools-native mtools-native bmap-tools-native')
+                    'dosfstools-native mtools-native bmap-tools-native '
+                    'e2tools-native')
             bitbake('core-image-minimal')
             Wic.image_is_ready = True
 
@@ -299,3 +300,103 @@ class Wic(oeSelfTest):
         self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
                                    % image).status)
         self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
+
+    def test_exclude_path(self):
+        """Test --exclude-path wks option."""
+
+        # For using 'e2ls'.
+        old_path = os.environ['PATH']
+        os.environ['PATH'] = get_bb_var('PATH', 'core-image-minimal')
+
+        wks_file = 'temp.wks'
+        with open(wks_file, 'w') as wks:
+            rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
+            wks.write("""part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path usr
+part /usr --source rootfs --ondisk mmcblk0 --fstype=ext4 --rootfs-dir %s/usr
+part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --rootfs-dir %s/usr"""
+                      % (rootfs_dir, rootfs_dir))
+        self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
+                                   % wks_file).status)
+
+        os.remove(wks_file)
+        wicout = glob(self.resultdir + "%s-*direct" % 'temp')
+        self.assertEqual(1, len(wicout))
+
+        wicimg = wicout[0]
+
+        # verify partition size with wic
+        res = runCmd("parted -m %s unit b p 2>/dev/null" % wicimg)
+        self.assertEqual(0, res.status)
+
+        # parse parted output which looks like this:
+        # BYT;\n
+        # /var/tmp/wic/build/tmpfwvjjkf_-201611101222-hda.direct:200MiB:file:512:512:msdos::;\n
+        # 1:0.00MiB:200MiB:200MiB:ext4::;\n
+        partlns = res.output.splitlines()[2:]
+
+        self.assertEqual(3, len(partlns))
+
+        for part in [1, 2, 3]:
+            part_file = os.path.join(self.resultdir, "selftest_img.part%d" % part)
+            partln = partlns[part-1].split(":")
+            self.assertEqual(7, len(partln))
+            start = int(partln[1].rstrip("B")) / 512
+            length = int(partln[3].rstrip("B")) / 512
+            self.assertEqual(0, runCmd("dd if=%s of=%s skip=%d count=%d" %
+                                       (wicimg, part_file, start, length)).status)
+
+        # Test partition 1, should contain the normal root directories, except
+        # /usr.
+        res = runCmd("e2ls %s" % os.path.join(self.resultdir, "selftest_img.part1"))
+        self.assertEqual(0, res.status)
+        files = res.output.split()
+        self.assertIn("etc", files)
+        self.assertNotIn("usr", files)
+
+        # Partition 2, should contain common directories for /usr, not root
+        # directories.
+        res = runCmd("e2ls %s" % os.path.join(self.resultdir, "selftest_img.part2"))
+        self.assertEqual(0, res.status)
+        files = res.output.split()
+        self.assertNotIn("etc", files)
+        self.assertNotIn("usr", files)
+        self.assertIn("share", files)
+
+        # Partition 3, should contain the same as partition 2, including the bin
+        # directory, but not the files inside it.
+        res = runCmd("e2ls %s" % os.path.join(self.resultdir, "selftest_img.part3"))
+        self.assertEqual(0, res.status)
+        files = res.output.split()
+        self.assertNotIn("etc", files)
+        self.assertNotIn("usr", files)
+        self.assertIn("share", files)
+        self.assertIn("bin", files)
+        res = runCmd("e2ls %s:bin" % os.path.join(self.resultdir, "selftest_img.part3"))
+        self.assertEqual(0, res.status)
+        self.assertEqual("No files found!", res.output.strip())
+
+        for part in [1, 2, 3]:
+            part_file = os.path.join(self.resultdir, "selftest_img.part%d" % part)
+            os.remove(part_file)
+
+        os.environ['PATH'] = old_path
+
+    def test_exclude_path_errors(self):
+        """Test --exclude-path wks option error handling."""
+        wks_file = 'temp.wks'
+
+        rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
+
+        # Absolute argument.
+        with open(wks_file, 'w') as wks:
+            wks.write("part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path /usr")
+        self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal" \
+                                      % wks_file, ignore_status=True).status)
+        os.remove(wks_file)
+
+        # Argument pointing to parent directory.
+        with open(wks_file, 'w') as wks:
+            wks.write("part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path ././..")
+        self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal" \
+                                      % wks_file, ignore_status=True).status)
+        os.remove(wks_file)
-- 
2.7.4

Re: [PATCH v3 3/3] selftest/wic: Add tests for --exclude-dir option.

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 320 bytes --]

On 19 December 2016 at 09:09, Kristian Amlie <kristian.amlie@mender.io>
wrote:

> +        res = runCmd("e2ls %s" % os.path.join(self.resultdir,
> "selftest_img.part1"))
>

To avoid adding recipes for software that hasn't been touched for about a
decade, can this be done using debugfs from util-linux?

Ross

[-- Attachment #2: Type: text/html, Size: 717 bytes --]

Re: [PATCH v3 3/3] selftest/wic: Add tests for --exclude-dir option.

[Kristian Amlie]

On 19/12/16 21:49, Burton, Ross wrote:
> 
> On 19 December 2016 at 09:09, Kristian Amlie <kristian.amlie@mender.io
> <mailto:kristian.amlie@mender.io>> wrote:
> 
>     +        res = runCmd("e2ls %s" % os.path.join(self.resultdir,
>     "selftest_img.part1"))
> 
> 
> To avoid adding recipes for software that hasn't been touched for about
> a decade, can this be done using debugfs from util-linux?

Probably, I'll check. It would be nice to use something that's up to date.

-- 
Kristian

Re: [PATCH v3 3/3] selftest/wic: Add tests for --exclude-dir option.

[Kristian Amlie]

Ok, new patchset:

* Removed e2tools recipe.
* Switched to using 'debugfs' instead of 'e2ls' in selftest.

-- 
Kristian

[PATCH v4 1/2] wic: Add --exclude-path option to rootfs source plugin.

[Kristian Amlie]

It will omit the given path from the resulting partition, and if the
given path ends in a slash, it will only delete the content, and keep
the directory.

Since mkfs only accepts whole directories as input, we need to copy
the rootfs directory to the workdir so that we can selectively delete
files from it.

Since we want to use the copyhardlinktree() function, we need to put
the generic oe lib in the module search path.

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
---
 scripts/lib/wic/help.py                  |  6 +++++
 scripts/lib/wic/ksparser.py              |  1 +
 scripts/lib/wic/partition.py             |  1 +
 scripts/lib/wic/plugins/source/rootfs.py | 43 +++++++++++++++++++++++++++++++-
 scripts/wic                              |  2 ++
 5 files changed, 52 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index e5347ec..9dab670 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -715,6 +715,12 @@ DESCRIPTION
                      partition table. It may be useful for
                      bootloaders.
 
+         --exclude-path: This option is specific to wic. It excludes the given
+                         absolute path from the resulting image. If the path
+                         ends with a slash, only the content of the directory
+                         is omitted, not the directory itself. This option only
+                         has an effect with the rootfs source plugin.
+
          --extra-space: This option is specific to wic. It adds extra
                         space after the space filled by the content
                         of the partition. The final size can go
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 0894e2b..17b97fd 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -127,6 +127,7 @@ class KickStart():
         part.add_argument('mountpoint', nargs='?')
         part.add_argument('--active', action='store_true')
         part.add_argument('--align', type=int)
+        part.add_argument('--exclude-path', nargs='+')
         part.add_argument("--extra-space", type=sizetype, default=10*1024)
         part.add_argument('--fsoptions', dest='fsopts')
         part.add_argument('--fstype')
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index ac4c836..cba78a5 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -45,6 +45,7 @@ class Partition():
         self.align = args.align
         self.disk = args.disk
         self.extra_space = args.extra_space
+        self.exclude_path = args.exclude_path
         self.fsopts = args.fsopts
         self.fstype = args.fstype
         self.label = args.label
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 425da8b..794ad69 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -26,10 +26,13 @@
 #
 
 import os
+import shutil
+
+from oe.path import copyhardlinktree
 
 from wic import msger
 from wic.pluginbase import SourcePlugin
-from wic.utils.oe.misc import get_bitbake_var
+from wic.utils.oe.misc import get_bitbake_var, exec_cmd
 
 class RootfsPlugin(SourcePlugin):
     """
@@ -78,6 +81,44 @@ class RootfsPlugin(SourcePlugin):
 
         real_rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
 
+        # Handle excluded paths.
+        if part.exclude_path is not None:
+            # We need a new rootfs directory we can delete files from. Copy to
+            # workdir.
+            new_rootfs = os.path.join(cr_workdir, "rootfs")
+
+            if os.path.lexists(new_rootfs):
+                shutil.rmtree(os.path.join(new_rootfs))
+
+            copyhardlinktree(real_rootfs_dir, new_rootfs)
+
+            real_rootfs_dir = new_rootfs
+
+            for orig_path in part.exclude_path:
+                path = orig_path
+                if os.path.isabs(path):
+                    msger.error("Must be relative: --exclude-path=%s" % orig_path)
+
+                full_path = os.path.realpath(os.path.join(new_rootfs, path))
+
+                # Disallow climbing outside of parent directory using '..',
+                # because doing so could be quite disastrous (we will delete the
+                # directory).
+                if not full_path.startswith(new_rootfs):
+                    msger.error("'%s' points to a path outside the rootfs" % orig_path)
+
+                if path.endswith(os.sep):
+                    # Delete content only.
+                    for entry in os.listdir(full_path):
+                        full_entry = os.path.join(full_path, entry)
+                        if os.path.isdir(full_entry) and not os.path.islink(full_entry):
+                            shutil.rmtree(full_entry)
+                        else:
+                            os.remove(full_entry)
+                else:
+                    # Delete whole directory.
+                    shutil.rmtree(full_path)
+
         part.rootfs_dir = real_rootfs_dir
         part.prepare_rootfs(cr_workdir, oe_builddir, real_rootfs_dir, native_sysroot)
 
diff --git a/scripts/wic b/scripts/wic
index fe2c33f..f58843e 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -41,6 +41,8 @@ from distutils import spawn
 scripts_path = os.path.abspath(os.path.dirname(__file__))
 lib_path = scripts_path + '/lib'
 sys.path.insert(0, lib_path)
+oe_lib_path = os.path.join(os.path.dirname(scripts_path), 'meta', 'lib')
+sys.path.insert(0, oe_lib_path)
 
 bitbake_exe = spawn.find_executable('bitbake')
 if bitbake_exe:
-- 
2.7.4

[PATCH v4 2/2] selftest/wic: Add tests for --exclude-dir option.

[Kristian Amlie]

Based partially on an earlier patch by Maciej Borzecki.

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
---
 meta/lib/oeqa/selftest/wic.py | 97 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 97 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index faac11e..9afa696 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -299,3 +299,100 @@ class Wic(oeSelfTest):
         self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
                                    % image).status)
         self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
+
+    def test_exclude_path(self):
+        """Test --exclude-path wks option."""
+
+        wks_file = 'temp.wks'
+        with open(wks_file, 'w') as wks:
+            rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
+            wks.write("""part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path usr
+part /usr --source rootfs --ondisk mmcblk0 --fstype=ext4 --rootfs-dir %s/usr
+part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --rootfs-dir %s/usr"""
+                      % (rootfs_dir, rootfs_dir))
+        self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
+                                   % wks_file).status)
+
+        os.remove(wks_file)
+        wicout = glob(self.resultdir + "%s-*direct" % 'temp')
+        self.assertEqual(1, len(wicout))
+
+        wicimg = wicout[0]
+
+        # verify partition size with wic
+        res = runCmd("parted -m %s unit b p 2>/dev/null" % wicimg)
+        self.assertEqual(0, res.status)
+
+        # parse parted output which looks like this:
+        # BYT;\n
+        # /var/tmp/wic/build/tmpfwvjjkf_-201611101222-hda.direct:200MiB:file:512:512:msdos::;\n
+        # 1:0.00MiB:200MiB:200MiB:ext4::;\n
+        partlns = res.output.splitlines()[2:]
+
+        self.assertEqual(3, len(partlns))
+
+        for part in [1, 2, 3]:
+            part_file = os.path.join(self.resultdir, "selftest_img.part%d" % part)
+            partln = partlns[part-1].split(":")
+            self.assertEqual(7, len(partln))
+            start = int(partln[1].rstrip("B")) / 512
+            length = int(partln[3].rstrip("B")) / 512
+            self.assertEqual(0, runCmd("dd if=%s of=%s skip=%d count=%d" %
+                                       (wicimg, part_file, start, length)).status)
+
+        # Test partition 1, should contain the normal root directories, except
+        # /usr.
+        res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % os.path.join(self.resultdir, "selftest_img.part1"))
+        self.assertEqual(0, res.status)
+        files = [line.split('/')[5] for line in res.output.split('\n')]
+        self.assertIn("etc", files)
+        self.assertNotIn("usr", files)
+
+        # Partition 2, should contain common directories for /usr, not root
+        # directories.
+        res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % os.path.join(self.resultdir, "selftest_img.part2"))
+        self.assertEqual(0, res.status)
+        files = [line.split('/')[5] for line in res.output.split('\n')]
+        self.assertNotIn("etc", files)
+        self.assertNotIn("usr", files)
+        self.assertIn("share", files)
+
+        # Partition 3, should contain the same as partition 2, including the bin
+        # directory, but not the files inside it.
+        res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % os.path.join(self.resultdir, "selftest_img.part3"))
+        self.assertEqual(0, res.status)
+        files = [line.split('/')[5] for line in res.output.split('\n')]
+        self.assertNotIn("etc", files)
+        self.assertNotIn("usr", files)
+        self.assertIn("share", files)
+        self.assertIn("bin", files)
+        res = runCmd("debugfs -R 'ls -p bin' %s 2>/dev/null" % os.path.join(self.resultdir, "selftest_img.part3"))
+        self.assertEqual(0, res.status)
+        files = [line.split('/')[5] for line in res.output.split('\n')]
+        self.assertIn(".", files)
+        self.assertIn("..", files)
+        self.assertEqual(2, len(files))
+
+        for part in [1, 2, 3]:
+            part_file = os.path.join(self.resultdir, "selftest_img.part%d" % part)
+            os.remove(part_file)
+
+    def test_exclude_path_errors(self):
+        """Test --exclude-path wks option error handling."""
+        wks_file = 'temp.wks'
+
+        rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
+
+        # Absolute argument.
+        with open(wks_file, 'w') as wks:
+            wks.write("part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path /usr")
+        self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal" \
+                                      % wks_file, ignore_status=True).status)
+        os.remove(wks_file)
+
+        # Argument pointing to parent directory.
+        with open(wks_file, 'w') as wks:
+            wks.write("part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path ././..")
+        self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal" \
+                                      % wks_file, ignore_status=True).status)
+        os.remove(wks_file)
-- 
2.7.4

Re: [PATCH v4 2/2] selftest/wic: Add tests for --exclude-dir option.

[Kristian Amlie]

Sorry, I forgot to rebase, I will repost.

-- 
Kristian


On 29/12/16 11:05, Kristian Amlie wrote:
> Based partially on an earlier patch by Maciej Borzecki.
> 
> Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
> ---
>  meta/lib/oeqa/selftest/wic.py | 97 +++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 97 insertions(+)
> 
> diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
> index faac11e..9afa696 100644
> --- a/meta/lib/oeqa/selftest/wic.py
> +++ b/meta/lib/oeqa/selftest/wic.py
> @@ -299,3 +299,100 @@ class Wic(oeSelfTest):
>          self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
>                                     % image).status)
>          self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
> +
> +    def test_exclude_path(self):
> +        """Test --exclude-path wks option."""
> +
> +        wks_file = 'temp.wks'
> +        with open(wks_file, 'w') as wks:
> +            rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
> +            wks.write("""part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path usr
> +part /usr --source rootfs --ondisk mmcblk0 --fstype=ext4 --rootfs-dir %s/usr
> +part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --rootfs-dir %s/usr"""
> +                      % (rootfs_dir, rootfs_dir))
> +        self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
> +                                   % wks_file).status)
> +
> +        os.remove(wks_file)
> +        wicout = glob(self.resultdir + "%s-*direct" % 'temp')
> +        self.assertEqual(1, len(wicout))
> +
> +        wicimg = wicout[0]
> +
> +        # verify partition size with wic
> +        res = runCmd("parted -m %s unit b p 2>/dev/null" % wicimg)
> +        self.assertEqual(0, res.status)
> +
> +        # parse parted output which looks like this:
> +        # BYT;\n
> +        # /var/tmp/wic/build/tmpfwvjjkf_-201611101222-hda.direct:200MiB:file:512:512:msdos::;\n
> +        # 1:0.00MiB:200MiB:200MiB:ext4::;\n
> +        partlns = res.output.splitlines()[2:]
> +
> +        self.assertEqual(3, len(partlns))
> +
> +        for part in [1, 2, 3]:
> +            part_file = os.path.join(self.resultdir, "selftest_img.part%d" % part)
> +            partln = partlns[part-1].split(":")
> +            self.assertEqual(7, len(partln))
> +            start = int(partln[1].rstrip("B")) / 512
> +            length = int(partln[3].rstrip("B")) / 512
> +            self.assertEqual(0, runCmd("dd if=%s of=%s skip=%d count=%d" %
> +                                       (wicimg, part_file, start, length)).status)
> +
> +        # Test partition 1, should contain the normal root directories, except
> +        # /usr.
> +        res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % os.path.join(self.resultdir, "selftest_img.part1"))
> +        self.assertEqual(0, res.status)
> +        files = [line.split('/')[5] for line in res.output.split('\n')]
> +        self.assertIn("etc", files)
> +        self.assertNotIn("usr", files)
> +
> +        # Partition 2, should contain common directories for /usr, not root
> +        # directories.
> +        res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % os.path.join(self.resultdir, "selftest_img.part2"))
> +        self.assertEqual(0, res.status)
> +        files = [line.split('/')[5] for line in res.output.split('\n')]
> +        self.assertNotIn("etc", files)
> +        self.assertNotIn("usr", files)
> +        self.assertIn("share", files)
> +
> +        # Partition 3, should contain the same as partition 2, including the bin
> +        # directory, but not the files inside it.
> +        res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % os.path.join(self.resultdir, "selftest_img.part3"))
> +        self.assertEqual(0, res.status)
> +        files = [line.split('/')[5] for line in res.output.split('\n')]
> +        self.assertNotIn("etc", files)
> +        self.assertNotIn("usr", files)
> +        self.assertIn("share", files)
> +        self.assertIn("bin", files)
> +        res = runCmd("debugfs -R 'ls -p bin' %s 2>/dev/null" % os.path.join(self.resultdir, "selftest_img.part3"))
> +        self.assertEqual(0, res.status)
> +        files = [line.split('/')[5] for line in res.output.split('\n')]
> +        self.assertIn(".", files)
> +        self.assertIn("..", files)
> +        self.assertEqual(2, len(files))
> +
> +        for part in [1, 2, 3]:
> +            part_file = os.path.join(self.resultdir, "selftest_img.part%d" % part)
> +            os.remove(part_file)
> +
> +    def test_exclude_path_errors(self):
> +        """Test --exclude-path wks option error handling."""
> +        wks_file = 'temp.wks'
> +
> +        rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
> +
> +        # Absolute argument.
> +        with open(wks_file, 'w') as wks:
> +            wks.write("part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path /usr")
> +        self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal" \
> +                                      % wks_file, ignore_status=True).status)
> +        os.remove(wks_file)
> +
> +        # Argument pointing to parent directory.
> +        with open(wks_file, 'w') as wks:
> +            wks.write("part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path ././..")
> +        self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal" \
> +                                      % wks_file, ignore_status=True).status)
> +        os.remove(wks_file)
> 

Re: [PATCH v4 2/2] selftest/wic: Add tests for --exclude-dir option.

[Kristian Amlie]

Rebased patchset, otherwise identical to the last one:

* Removed e2tools recipe.
* Switched to using 'debugfs' instead of 'e2ls' in selftest.

-- 
Kristian

[PATCH v5 1/2] wic: Add --exclude-path option to rootfs source plugin.

[Kristian Amlie]

It will omit the given path from the resulting partition, and if the
given path ends in a slash, it will only delete the content, and keep
the directory.

Since mkfs only accepts whole directories as input, we need to copy
the rootfs directory to the workdir so that we can selectively delete
files from it.

Since we want to use the copyhardlinktree() function, we need to put
the generic oe lib in the module search path.

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
---
 scripts/lib/wic/help.py                  |  6 +++++
 scripts/lib/wic/ksparser.py              |  1 +
 scripts/lib/wic/partition.py             |  1 +
 scripts/lib/wic/plugins/source/rootfs.py | 43 +++++++++++++++++++++++++++++++-
 scripts/wic                              |  2 ++
 5 files changed, 52 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index daa11bf..4d48fb7 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -721,6 +721,12 @@ DESCRIPTION
                      partition table. It may be useful for
                      bootloaders.
 
+         --exclude-path: This option is specific to wic. It excludes the given
+                         absolute path from the resulting image. If the path
+                         ends with a slash, only the content of the directory
+                         is omitted, not the directory itself. This option only
+                         has an effect with the rootfs source plugin.
+
          --extra-space: This option is specific to wic. It adds extra
                         space after the space filled by the content
                         of the partition. The final size can go
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 62c4902..56a12b0 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -130,6 +130,7 @@ class KickStart():
         part.add_argument('mountpoint', nargs='?')
         part.add_argument('--active', action='store_true')
         part.add_argument('--align', type=int)
+        part.add_argument('--exclude-path', nargs='+')
         part.add_argument("--extra-space", type=sizetype)
         part.add_argument('--fsoptions', dest='fsopts')
         part.add_argument('--fstype')
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index aa8f8a7..9c130c4 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -45,6 +45,7 @@ class Partition():
         self.align = args.align
         self.disk = args.disk
         self.extra_space = args.extra_space
+        self.exclude_path = args.exclude_path
         self.fsopts = args.fsopts
         self.fstype = args.fstype
         self.label = args.label
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 425da8b..794ad69 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -26,10 +26,13 @@
 #
 
 import os
+import shutil
+
+from oe.path import copyhardlinktree
 
 from wic import msger
 from wic.pluginbase import SourcePlugin
-from wic.utils.oe.misc import get_bitbake_var
+from wic.utils.oe.misc import get_bitbake_var, exec_cmd
 
 class RootfsPlugin(SourcePlugin):
     """
@@ -78,6 +81,44 @@ class RootfsPlugin(SourcePlugin):
 
         real_rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
 
+        # Handle excluded paths.
+        if part.exclude_path is not None:
+            # We need a new rootfs directory we can delete files from. Copy to
+            # workdir.
+            new_rootfs = os.path.join(cr_workdir, "rootfs")
+
+            if os.path.lexists(new_rootfs):
+                shutil.rmtree(os.path.join(new_rootfs))
+
+            copyhardlinktree(real_rootfs_dir, new_rootfs)
+
+            real_rootfs_dir = new_rootfs
+
+            for orig_path in part.exclude_path:
+                path = orig_path
+                if os.path.isabs(path):
+                    msger.error("Must be relative: --exclude-path=%s" % orig_path)
+
+                full_path = os.path.realpath(os.path.join(new_rootfs, path))
+
+                # Disallow climbing outside of parent directory using '..',
+                # because doing so could be quite disastrous (we will delete the
+                # directory).
+                if not full_path.startswith(new_rootfs):
+                    msger.error("'%s' points to a path outside the rootfs" % orig_path)
+
+                if path.endswith(os.sep):
+                    # Delete content only.
+                    for entry in os.listdir(full_path):
+                        full_entry = os.path.join(full_path, entry)
+                        if os.path.isdir(full_entry) and not os.path.islink(full_entry):
+                            shutil.rmtree(full_entry)
+                        else:
+                            os.remove(full_entry)
+                else:
+                    # Delete whole directory.
+                    shutil.rmtree(full_path)
+
         part.rootfs_dir = real_rootfs_dir
         part.prepare_rootfs(cr_workdir, oe_builddir, real_rootfs_dir, native_sysroot)
 
diff --git a/scripts/wic b/scripts/wic
index 1ad1666..4328ba5 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -41,6 +41,8 @@ from distutils import spawn
 scripts_path = os.path.abspath(os.path.dirname(__file__))
 lib_path = scripts_path + '/lib'
 sys.path.insert(0, lib_path)
+oe_lib_path = os.path.join(os.path.dirname(scripts_path), 'meta', 'lib')
+sys.path.insert(0, oe_lib_path)
 
 bitbake_exe = spawn.find_executable('bitbake')
 if bitbake_exe:
-- 
2.7.4

[PATCH v5 2/2] selftest/wic: Add tests for --exclude-dir option.

[Kristian Amlie]

Based partially on an earlier patch by Maciej Borzecki.

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
---
 meta/lib/oeqa/selftest/wic.py | 97 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 97 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index 3db00a6..cfe7d30 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -322,6 +322,103 @@ class Wic(oeSelfTest):
         self.assertEqual(0, status)
         self.assertEqual(1, len(glob(self.resultdir + "%(wks)s-*.direct" % bbvars)))
 
+    def test_exclude_path(self):
+        """Test --exclude-path wks option."""
+
+        wks_file = 'temp.wks'
+        with open(wks_file, 'w') as wks:
+            rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
+            wks.write("""part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path usr
+part /usr --source rootfs --ondisk mmcblk0 --fstype=ext4 --rootfs-dir %s/usr
+part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --rootfs-dir %s/usr"""
+                      % (rootfs_dir, rootfs_dir))
+        self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
+                                   % wks_file).status)
+
+        os.remove(wks_file)
+        wicout = glob(self.resultdir + "%s-*direct" % 'temp')
+        self.assertEqual(1, len(wicout))
+
+        wicimg = wicout[0]
+
+        # verify partition size with wic
+        res = runCmd("parted -m %s unit b p 2>/dev/null" % wicimg)
+        self.assertEqual(0, res.status)
+
+        # parse parted output which looks like this:
+        # BYT;\n
+        # /var/tmp/wic/build/tmpfwvjjkf_-201611101222-hda.direct:200MiB:file:512:512:msdos::;\n
+        # 1:0.00MiB:200MiB:200MiB:ext4::;\n
+        partlns = res.output.splitlines()[2:]
+
+        self.assertEqual(3, len(partlns))
+
+        for part in [1, 2, 3]:
+            part_file = os.path.join(self.resultdir, "selftest_img.part%d" % part)
+            partln = partlns[part-1].split(":")
+            self.assertEqual(7, len(partln))
+            start = int(partln[1].rstrip("B")) / 512
+            length = int(partln[3].rstrip("B")) / 512
+            self.assertEqual(0, runCmd("dd if=%s of=%s skip=%d count=%d" %
+                                       (wicimg, part_file, start, length)).status)
+
+        # Test partition 1, should contain the normal root directories, except
+        # /usr.
+        res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % os.path.join(self.resultdir, "selftest_img.part1"))
+        self.assertEqual(0, res.status)
+        files = [line.split('/')[5] for line in res.output.split('\n')]
+        self.assertIn("etc", files)
+        self.assertNotIn("usr", files)
+
+        # Partition 2, should contain common directories for /usr, not root
+        # directories.
+        res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % os.path.join(self.resultdir, "selftest_img.part2"))
+        self.assertEqual(0, res.status)
+        files = [line.split('/')[5] for line in res.output.split('\n')]
+        self.assertNotIn("etc", files)
+        self.assertNotIn("usr", files)
+        self.assertIn("share", files)
+
+        # Partition 3, should contain the same as partition 2, including the bin
+        # directory, but not the files inside it.
+        res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % os.path.join(self.resultdir, "selftest_img.part3"))
+        self.assertEqual(0, res.status)
+        files = [line.split('/')[5] for line in res.output.split('\n')]
+        self.assertNotIn("etc", files)
+        self.assertNotIn("usr", files)
+        self.assertIn("share", files)
+        self.assertIn("bin", files)
+        res = runCmd("debugfs -R 'ls -p bin' %s 2>/dev/null" % os.path.join(self.resultdir, "selftest_img.part3"))
+        self.assertEqual(0, res.status)
+        files = [line.split('/')[5] for line in res.output.split('\n')]
+        self.assertIn(".", files)
+        self.assertIn("..", files)
+        self.assertEqual(2, len(files))
+
+        for part in [1, 2, 3]:
+            part_file = os.path.join(self.resultdir, "selftest_img.part%d" % part)
+            os.remove(part_file)
+
+    def test_exclude_path_errors(self):
+        """Test --exclude-path wks option error handling."""
+        wks_file = 'temp.wks'
+
+        rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
+
+        # Absolute argument.
+        with open(wks_file, 'w') as wks:
+            wks.write("part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path /usr")
+        self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal" \
+                                      % wks_file, ignore_status=True).status)
+        os.remove(wks_file)
+
+        # Argument pointing to parent directory.
+        with open(wks_file, 'w') as wks:
+            wks.write("part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path ././..")
+        self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal" \
+                                      % wks_file, ignore_status=True).status)
+        os.remove(wks_file)
+
     @testcase(1496)
     def test_bmap(self):
         """Test generation of .bmap file"""
-- 
2.7.4

Re: [PATCH v4 2/2] selftest/wic: Add tests for --exclude-dir option.

[Kristian Amlie]

On 29/12/16 15:03, Kristian Amlie wrote:
> Rebased patchset, otherwise identical to the last one:
> 
> * Removed e2tools recipe.
> * Switched to using 'debugfs' instead of 'e2ls' in selftest.

Is anything missing here?

Would be nice to get this done so I can start on the do_image part of this.

-- 
Kristian

Re: [PATCH v4 2/2] selftest/wic: Add tests for --exclude-dir option.

[Kristian Amlie]

Rebased and reran tests. Changes since last patchset:

- Because of recipe specific sysroots we now need to import the PATH
  used by wic-tools into the test in order to use its tools.

- Add specific '-o' parameter while testing wic so that we write into
  resultdir.

- Several unrelated changes to poky now means that we need to use an
  absolute path for new_rootfs in the rootfs.py implementation. This
  would have been a good idea from the start anyway.

I hope there are no more outstanding issues with this patchset.

-- 
Kristian

[PATCH 1/2] wic: Add --exclude-path option to rootfs source plugin.

[Kristian Amlie]

It will omit the given path from the resulting partition, and if the
given path ends in a slash, it will only delete the content, and keep
the directory.

Since mkfs only accepts whole directories as input, we need to copy
the rootfs directory to the workdir so that we can selectively delete
files from it.

Since we want to use the copyhardlinktree() function, we need to put
the generic oe lib in the module search path.

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
---
 scripts/lib/wic/help.py                  |  6 +++++
 scripts/lib/wic/ksparser.py              |  1 +
 scripts/lib/wic/partition.py             |  1 +
 scripts/lib/wic/plugins/source/rootfs.py | 43 +++++++++++++++++++++++++++++++-
 scripts/wic                              |  2 ++
 5 files changed, 52 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 1bd411d..63bbc23 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -721,6 +721,12 @@ DESCRIPTION
                      partition table. It may be useful for
                      bootloaders.
 
+         --exclude-path: This option is specific to wic. It excludes the given
+                         absolute path from the resulting image. If the path
+                         ends with a slash, only the content of the directory
+                         is omitted, not the directory itself. This option only
+                         has an effect with the rootfs source plugin.
+
          --extra-space: This option is specific to wic. It adds extra
                         space after the space filled by the content
                         of the partition. The final size can go
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 41d3cc6..f0aa5d0 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -130,6 +130,7 @@ class KickStart():
         part.add_argument('mountpoint', nargs='?')
         part.add_argument('--active', action='store_true')
         part.add_argument('--align', type=int)
+        part.add_argument('--exclude-path', nargs='+')
         part.add_argument("--extra-space", type=sizetype)
         part.add_argument('--fsoptions', dest='fsopts')
         part.add_argument('--fstype')
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 69b369c..5f6cf76 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -39,6 +39,7 @@ class Partition():
         self.align = args.align
         self.disk = args.disk
         self.extra_space = args.extra_space
+        self.exclude_path = args.exclude_path
         self.fsopts = args.fsopts
         self.fstype = args.fstype
         self.label = args.label
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 9d959fa..c57a434 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -26,10 +26,13 @@
 #
 
 import os
+import shutil
+
+from oe.path import copyhardlinktree
 
 from wic import msger
 from wic.pluginbase import SourcePlugin
-from wic.utils.misc import get_bitbake_var
+from wic.utils.misc import get_bitbake_var, exec_cmd
 
 class RootfsPlugin(SourcePlugin):
     """
@@ -78,6 +81,44 @@ class RootfsPlugin(SourcePlugin):
 
         real_rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
 
+        # Handle excluded paths.
+        if part.exclude_path is not None:
+            # We need a new rootfs directory we can delete files from. Copy to
+            # workdir.
+            new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs"))
+
+            if os.path.lexists(new_rootfs):
+                shutil.rmtree(os.path.join(new_rootfs))
+
+            copyhardlinktree(real_rootfs_dir, new_rootfs)
+
+            real_rootfs_dir = new_rootfs
+
+            for orig_path in part.exclude_path:
+                path = orig_path
+                if os.path.isabs(path):
+                    msger.error("Must be relative: --exclude-path=%s" % orig_path)
+
+                full_path = os.path.realpath(os.path.join(new_rootfs, path))
+
+                # Disallow climbing outside of parent directory using '..',
+                # because doing so could be quite disastrous (we will delete the
+                # directory).
+                if not full_path.startswith(new_rootfs):
+                    msger.error("'%s' points to a path outside the rootfs" % orig_path)
+
+                if path.endswith(os.sep):
+                    # Delete content only.
+                    for entry in os.listdir(full_path):
+                        full_entry = os.path.join(full_path, entry)
+                        if os.path.isdir(full_entry) and not os.path.islink(full_entry):
+                            shutil.rmtree(full_entry)
+                        else:
+                            os.remove(full_entry)
+                else:
+                    # Delete whole directory.
+                    shutil.rmtree(full_path)
+
         part.rootfs_dir = real_rootfs_dir
         part.prepare_rootfs(cr_workdir, oe_builddir,
                             real_rootfs_dir, native_sysroot)
diff --git a/scripts/wic b/scripts/wic
index 8a959a0..e872096 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -41,6 +41,8 @@ from distutils import spawn
 scripts_path = os.path.abspath(os.path.dirname(__file__))
 lib_path = scripts_path + '/lib'
 sys.path.insert(0, lib_path)
+oe_lib_path = os.path.join(os.path.dirname(scripts_path), 'meta', 'lib')
+sys.path.insert(0, oe_lib_path)
 
 bitbake_exe = spawn.find_executable('bitbake')
 if bitbake_exe:
-- 
2.7.4

[PATCH 2/2] selftest/wic: Add tests for --exclude-dir option.

[Kristian Amlie]

Based partially on an earlier patch by Maciej Borzecki.

Note that because tools now reside in recipe specific sysroots, we
need to import the path from bitbake and apply it during the test.

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
---
 meta/lib/oeqa/selftest/wic.py | 104 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 104 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index 417ba3d..0144d77 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -339,6 +339,110 @@ class Wic(oeSelfTest):
         self.assertEqual(0, status)
         self.assertEqual(1, len(glob(self.resultdir + "%(wks)s-*.direct" % bbvars)))
 
+    def test_exclude_path(self):
+        """Test --exclude-path wks option."""
+
+        oldpath = os.environ['PATH']
+        os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
+
+        try:
+            wks_file = 'temp.wks'
+            with open(wks_file, 'w') as wks:
+                rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
+                wks.write("""part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path usr
+part /usr --source rootfs --ondisk mmcblk0 --fstype=ext4 --rootfs-dir %s/usr
+part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --rootfs-dir %s/usr"""
+                          % (rootfs_dir, rootfs_dir))
+            self.assertEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
+                                       % (wks_file, self.resultdir)).status)
+
+            os.remove(wks_file)
+            wicout = glob(self.resultdir + "%s-*direct" % 'temp')
+            self.assertEqual(1, len(wicout))
+
+            wicimg = wicout[0]
+
+            # verify partition size with wic
+            res = runCmd("parted -m %s unit b p 2>/dev/null" % wicimg)
+            self.assertEqual(0, res.status)
+
+            # parse parted output which looks like this:
+            # BYT;\n
+            # /var/tmp/wic/build/tmpfwvjjkf_-201611101222-hda.direct:200MiB:file:512:512:msdos::;\n
+            # 1:0.00MiB:200MiB:200MiB:ext4::;\n
+            partlns = res.output.splitlines()[2:]
+
+            self.assertEqual(3, len(partlns))
+
+            for part in [1, 2, 3]:
+                part_file = os.path.join(self.resultdir, "selftest_img.part%d" % part)
+                partln = partlns[part-1].split(":")
+                self.assertEqual(7, len(partln))
+                start = int(partln[1].rstrip("B")) / 512
+                length = int(partln[3].rstrip("B")) / 512
+                self.assertEqual(0, runCmd("dd if=%s of=%s skip=%d count=%d" %
+                                           (wicimg, part_file, start, length)).status)
+
+            # Test partition 1, should contain the normal root directories, except
+            # /usr.
+            res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % os.path.join(self.resultdir, "selftest_img.part1"))
+            self.assertEqual(0, res.status)
+            files = [line.split('/')[5] for line in res.output.split('\n')]
+            self.assertIn("etc", files)
+            self.assertNotIn("usr", files)
+
+            # Partition 2, should contain common directories for /usr, not root
+            # directories.
+            res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % os.path.join(self.resultdir, "selftest_img.part2"))
+            self.assertEqual(0, res.status)
+            files = [line.split('/')[5] for line in res.output.split('\n')]
+            self.assertNotIn("etc", files)
+            self.assertNotIn("usr", files)
+            self.assertIn("share", files)
+
+            # Partition 3, should contain the same as partition 2, including the bin
+            # directory, but not the files inside it.
+            res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % os.path.join(self.resultdir, "selftest_img.part3"))
+            self.assertEqual(0, res.status)
+            files = [line.split('/')[5] for line in res.output.split('\n')]
+            self.assertNotIn("etc", files)
+            self.assertNotIn("usr", files)
+            self.assertIn("share", files)
+            self.assertIn("bin", files)
+            res = runCmd("debugfs -R 'ls -p bin' %s 2>/dev/null" % os.path.join(self.resultdir, "selftest_img.part3"))
+            self.assertEqual(0, res.status)
+            files = [line.split('/')[5] for line in res.output.split('\n')]
+            self.assertIn(".", files)
+            self.assertIn("..", files)
+            self.assertEqual(2, len(files))
+
+            for part in [1, 2, 3]:
+                part_file = os.path.join(self.resultdir, "selftest_img.part%d" % part)
+                os.remove(part_file)
+
+        finally:
+            os.environ['PATH'] = oldpath
+
+    def test_exclude_path_errors(self):
+        """Test --exclude-path wks option error handling."""
+        wks_file = 'temp.wks'
+
+        rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
+
+        # Absolute argument.
+        with open(wks_file, 'w') as wks:
+            wks.write("part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path /usr")
+        self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
+                                      % (wks_file, self.resultdir), ignore_status=True).status)
+        os.remove(wks_file)
+
+        # Argument pointing to parent directory.
+        with open(wks_file, 'w') as wks:
+            wks.write("part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path ././..")
+        self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
+                                      % (wks_file, self.resultdir), ignore_status=True).status)
+        os.remove(wks_file)
+
     @testcase(1496)
     def test_bmap_short(self):
         """Test generation of .bmap file -m option"""
-- 
2.7.4

Re: [PATCH v4 2/2] selftest/wic: Add tests for --exclude-dir option.

[Ed Bartosh]

On Mon, Feb 06, 2017 at 05:16:45PM +0100, Kristian Amlie wrote:
> 
> Rebased and reran tests. Changes since last patchset:
> 
> - Because of recipe specific sysroots we now need to import the PATH
>   used by wic-tools into the test in order to use its tools.
> 
> - Add specific '-o' parameter while testing wic so that we write into
>   resultdir.
> 
> - Several unrelated changes to poky now means that we need to use an
>   absolute path for new_rootfs in the rootfs.py implementation. This
>   would have been a good idea from the start anyway.
> 
> I hope there are no more outstanding issues with this patchset.
> 

The patchset looks good to me.
All wic test cases have been passed.

--
Regards,
Ed

Re: [PATCH v2 3/3] selftest/wic: Add tests for --exclude-dir option.

[Ed Bartosh]

On Mon, Dec 19, 2016 at 10:09:22AM +0100, Kristian Amlie wrote:
> 
> >> +    def test_exclude_path(self):
> >> +        """Test --exclude-path wks option."""
> >> +
> >> +        # For using 'e2ls'.
> >> +        old_path = os.environ['PATH']
> >> +        os.environ['PATH'] = get_bb_var('PATH', 'core-image-minimal')
> >> +
> >> +        wks_file = 'temp.wks'
> >> +        ks = open(wks_file, 'w')
> > 
> > I'd use more pythonic 'with open(wks_file, 'w') as wks:' here.
> 
> Done for all three blocks!
> 
Thank you!

The patchset looks good to me.

--
Regards,
Ed

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Maciej Borzęcki]

On Fri, Nov 25, 2016 at 11:15 AM, Kristian Amlie
<kristian.amlie@mender.io> wrote:
> It will omit the given path from the resulting partition, and if the
> given path ends in a slash, it will only delete the content, and keep
> the directory.
>
> Since mkfs only accepts whole directories as input, we need to copy
> the rootfs directory to the workdir so that we can selectively delete
> files from it.
>
> Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
> ---
>  scripts/lib/wic/help.py                  |  6 ++++
>  scripts/lib/wic/ksparser.py              |  1 +
>  scripts/lib/wic/partition.py             |  1 +
>  scripts/lib/wic/plugins/source/rootfs.py | 51 +++++++++++++++++++++++++++++++-
>  4 files changed, 58 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
> index e5347ec..9dab670 100644
> --- a/scripts/lib/wic/help.py
> +++ b/scripts/lib/wic/help.py
> @@ -715,6 +715,12 @@ DESCRIPTION
>                       partition table. It may be useful for
>                       bootloaders.
>
> +         --exclude-path: This option is specific to wic. It excludes the given
> +                         absolute path from the resulting image. If the path
> +                         ends with a slash, only the content of the directory
> +                         is omitted, not the directory itself. This option only
> +                         has an effect with the rootfs source plugin.
> +
>           --extra-space: This option is specific to wic. It adds extra
>                          space after the space filled by the content
>                          of the partition. The final size can go
> diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> index 0894e2b..17b97fd0 100644
> --- a/scripts/lib/wic/ksparser.py
> +++ b/scripts/lib/wic/ksparser.py
> @@ -127,6 +127,7 @@ class KickStart():
>          part.add_argument('mountpoint', nargs='?')
>          part.add_argument('--active', action='store_true')
>          part.add_argument('--align', type=int)
> +        part.add_argument('--exclude-path', nargs='+')
>          part.add_argument("--extra-space", type=sizetype, default=10*1024)
>          part.add_argument('--fsoptions', dest='fsopts')
>          part.add_argument('--fstype')
> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> index ac4c836..cba78a5 100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -45,6 +45,7 @@ class Partition():
>          self.align = args.align
>          self.disk = args.disk
>          self.extra_space = args.extra_space
> +        self.exclude_path = args.exclude_path
>          self.fsopts = args.fsopts
>          self.fstype = args.fstype
>          self.label = args.label
> diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
> index 425da8b..d97d99c 100644
> --- a/scripts/lib/wic/plugins/source/rootfs.py
> +++ b/scripts/lib/wic/plugins/source/rootfs.py
> @@ -26,10 +26,11 @@
>  #
>
>  import os
> +import shutil
>
>  from wic import msger
>  from wic.pluginbase import SourcePlugin
> -from wic.utils.oe.misc import get_bitbake_var
> +from wic.utils.oe.misc import get_bitbake_var, exec_cmd
>
>  class RootfsPlugin(SourcePlugin):
>      """
> @@ -78,6 +79,54 @@ class RootfsPlugin(SourcePlugin):
>
>          real_rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
>
> +        # Handle excluded paths.
> +        if part.exclude_path is not None:
> +            # We need a new rootfs directory we can delete files from. Copy to
> +            # workdir.
> +            new_rootfs = os.path.join(cr_workdir, "rootfs")
> +
> +            if os.path.lexists(new_rootfs):
> +                shutil.rmtree(os.path.join(new_rootfs))
> +
> +            if os.stat(real_rootfs_dir).st_dev == os.stat(cr_workdir).st_dev:
> +                # Optimization if both directories are on the same file system:
> +                # copy using hardlinks.
> +                cp_args = "-al"
> +            else:
> +                cp_args = "-a"
> +            exec_cmd("cp %s %s %s" % (cp_args, real_rootfs_dir, new_rootfs))
> +
> +            real_rootfs_dir = new_rootfs
> +
> +            for orig_path in part.exclude_path:
> +                path = orig_path
> +                if not os.path.isabs(path):
> +                    msger.error("Must be absolute: --exclude-path=%s" % orig_path)
> +
> +                while os.path.isabs(path):
> +                    path = path[1:]
> +
> +                # Disallow '..', because doing so could be quite disastrous
> +                # (we will delete the directory).
> +                remaining = path
> +                while True:
> +                    (head, tail) = os.path.split(remaining)
> +                    if tail == '..':
> +                        msger.error("'..' not allowed: --exclude-path=%s" % orig_path)
> +                    elif head == "":
> +                        break
> +                    remaining = head

Why not do this instead?

    if '..' in path:
        msger.error("'..' not allowed: --exclude-path=%s" % orig_path)

> +
> +                full_path = os.path.join(new_rootfs, path)
> +
> +                if path.endswith(os.sep):
> +                    # Delete content only.
> +                    for entry in os.listdir(full_path):
> +                        shutil.rmtree(os.path.join(full_path, entry))
> +                else:
> +                    # Delete whole directory.
> +                    shutil.rmtree(full_path)
> +
>          part.rootfs_dir = real_rootfs_dir
>          part.prepare_rootfs(cr_workdir, oe_builddir, real_rootfs_dir, native_sysroot)
>
> --
> 2.7.4
>



-- 
Maciej Borzecki
RnDity

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Kristian Amlie]

On 25/11/16 13:28, Maciej Borzęcki wrote:
> On Fri, Nov 25, 2016 at 11:15 AM, Kristian Amlie
>> +                # Disallow '..', because doing so could be quite disastrous
>> +                # (we will delete the directory).
>> +                remaining = path
>> +                while True:
>> +                    (head, tail) = os.path.split(remaining)
>> +                    if tail == '..':
>> +                        msger.error("'..' not allowed: --exclude-path=%s" % orig_path)
>> +                    elif head == "":
>> +                        break
>> +                    remaining = head
> 
> Why not do this instead?
> 
>     if '..' in path:
>         msger.error("'..' not allowed: --exclude-path=%s" % orig_path)
> 

Because '..' can be part of a longer file name. Not overly likely, but
not impossible either.

-- 
Kristian

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Ed Bartosh]

On Fri, Nov 25, 2016 at 01:35:53PM +0100, Kristian Amlie wrote:
> On 25/11/16 13:28, Maciej Borzęcki wrote:
> > On Fri, Nov 25, 2016 at 11:15 AM, Kristian Amlie
> >> +                # Disallow '..', because doing so could be quite disastrous
> >> +                # (we will delete the directory).
> >> +                remaining = path
> >> +                while True:
> >> +                    (head, tail) = os.path.split(remaining)
> >> +                    if tail == '..':
> >> +                        msger.error("'..' not allowed: --exclude-path=%s" % orig_path)
> >> +                    elif head == "":
> >> +                        break
> >> +                    remaining = head
> > 
> > Why not do this instead?
> > 
> >     if '..' in path:
> >         msger.error("'..' not allowed: --exclude-path=%s" % orig_path)
> > 
would "'/..' in path" or something similar work?

--
Regards,
Ed

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Kristian Amlie]

On 25/11/16 17:33, Ed Bartosh wrote:
> On Fri, Nov 25, 2016 at 01:35:53PM +0100, Kristian Amlie wrote:
>> On 25/11/16 13:28, Maciej Borzęcki wrote:
>>> On Fri, Nov 25, 2016 at 11:15 AM, Kristian Amlie
>>>> +                # Disallow '..', because doing so could be quite disastrous
>>>> +                # (we will delete the directory).
>>>> +                remaining = path
>>>> +                while True:
>>>> +                    (head, tail) = os.path.split(remaining)
>>>> +                    if tail == '..':
>>>> +                        msger.error("'..' not allowed: --exclude-path=%s" % orig_path)
>>>> +                    elif head == "":
>>>> +                        break
>>>> +                    remaining = head
>>>
>>> Why not do this instead?
>>>
>>>     if '..' in path:
>>>         msger.error("'..' not allowed: --exclude-path=%s" % orig_path)
>>>
> would "'/..' in path" or something similar work?

'/..my-file' is a valid file name, so no, it wouldn't work. I realize
that this would be uncommon, and it makes the check more complex, so I'm
ok with making it simpler if you want me to, but we have to trade
simplicity for correctness in that case.

-- 
Kristian

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Ed Bartosh]

On Mon, Nov 28, 2016 at 08:07:05AM +0100, Kristian Amlie wrote:
> On 25/11/16 17:33, Ed Bartosh wrote:
> > On Fri, Nov 25, 2016 at 01:35:53PM +0100, Kristian Amlie wrote:
> >> On 25/11/16 13:28, Maciej Borzęcki wrote:
> >>> On Fri, Nov 25, 2016 at 11:15 AM, Kristian Amlie
> >>>> +                # Disallow '..', because doing so could be quite disastrous
> >>>> +                # (we will delete the directory).
> >>>> +                remaining = path
> >>>> +                while True:
> >>>> +                    (head, tail) = os.path.split(remaining)
> >>>> +                    if tail == '..':
> >>>> +                        msger.error("'..' not allowed: --exclude-path=%s" % orig_path)
> >>>> +                    elif head == "":
> >>>> +                        break
> >>>> +                    remaining = head
> >>>
> >>> Why not do this instead?
> >>>
> >>>     if '..' in path:
> >>>         msger.error("'..' not allowed: --exclude-path=%s" % orig_path)
> >>>
> > would "'/..' in path" or something similar work?
> 
> '/..my-file' is a valid file name, so no, it wouldn't work. I realize
> that this would be uncommon, and it makes the check more complex, so I'm
> ok with making it simpler if you want me to, but we have to trade
> simplicity for correctness in that case.
>
OK, makes sense.

Would something like this work for you?

> mkdir -p /tmp/rootfs/bla/bla/bla/bla
> python -c "import os;print os.path.realpath('/tmp/rootfs/bla/../bla/../bla/../bla/../').startswith('/tmp/rootfs/')"
False
> python -c "import os;print os.path.realpath('/tmp/rootfs/bla/../bla/../bla/bla/../').startswith('/tmp/rootfs/')"
True

--
Regards,
Ed

Re: [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Kristian Amlie]

On 28/11/16 11:46, Ed Bartosh wrote:
> On Mon, Nov 28, 2016 at 08:07:05AM +0100, Kristian Amlie wrote:
>> On 25/11/16 17:33, Ed Bartosh wrote:
>>> On Fri, Nov 25, 2016 at 01:35:53PM +0100, Kristian Amlie wrote:
>>>> On 25/11/16 13:28, Maciej Borzęcki wrote:
>>>>> On Fri, Nov 25, 2016 at 11:15 AM, Kristian Amlie
>>>>>> +                # Disallow '..', because doing so could be quite disastrous
>>>>>> +                # (we will delete the directory).
>>>>>> +                remaining = path
>>>>>> +                while True:
>>>>>> +                    (head, tail) = os.path.split(remaining)
>>>>>> +                    if tail == '..':
>>>>>> +                        msger.error("'..' not allowed: --exclude-path=%s" % orig_path)
>>>>>> +                    elif head == "":
>>>>>> +                        break
>>>>>> +                    remaining = head
>>>>>
>>>>> Why not do this instead?
>>>>>
>>>>>     if '..' in path:
>>>>>         msger.error("'..' not allowed: --exclude-path=%s" % orig_path)
>>>>>
>>> would "'/..' in path" or something similar work?
>>
>> '/..my-file' is a valid file name, so no, it wouldn't work. I realize
>> that this would be uncommon, and it makes the check more complex, so I'm
>> ok with making it simpler if you want me to, but we have to trade
>> simplicity for correctness in that case.
>>
> OK, makes sense.
> 
> Would something like this work for you?
> 
>> mkdir -p /tmp/rootfs/bla/bla/bla/bla
>> python -c "import os;print os.path.realpath('/tmp/rootfs/bla/../bla/../bla/../bla/../').startswith('/tmp/rootfs/')"
> False
>> python -c "import os;print os.path.realpath('/tmp/rootfs/bla/../bla/../bla/bla/../').startswith('/tmp/rootfs/')"
> True

Ah, that's a good idea, makes it much simpler, and is still correct. I
will change to that.

-- 
Kristian

✗ patchtest: failure for wic: Add --exclude-path option to rootfs source plugin. (rev10)

[Patchwork]

== Series Details ==

Series: wic: Add --exclude-path option to rootfs source plugin. (rev10)
Revision: 10
URL   : https://patchwork.openembedded.org/series/4087/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Patch            [v2,2/3] Add e2tools recipe, in order to test contents of images.
 Issue             Shortlog does not follow expected format [test_shortlog_format] 
  Suggested fix    Commit shortlog (first line of commit message) should follow the format "<target>: <summary>"

* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at b3a74335e1)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

✗ patchtest: failure for wic: Add --exclude-path option to rootfs source plugin. (rev11)

[Patchwork]

== Series Details ==

Series: wic: Add --exclude-path option to rootfs source plugin. (rev11)
Revision: 11
URL   : https://patchwork.openembedded.org/series/4087/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Patch            [v2,2/3] Add e2tools recipe, in order to test contents of images.
 Issue             Shortlog does not follow expected format [test_shortlog_format] 
  Suggested fix    Commit shortlog (first line of commit message) should follow the format "<target>: <summary>"

* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at b3a74335e1)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

Re: ✗ patchtest: failure for wic: Add --exclude-path option to rootfs source plugin. (rev11)

[Kristian Amlie]

Is this old? The patch it mentions is not part of the patchset anymore.

I noticed I forgot to add the 'v5' tag to the patches I posted
yesterday. Maybe that's why?

-- 
Kristian

On 06/02/17 17:29, Patchwork wrote:
> == Series Details ==
> 
> Series: wic: Add --exclude-path option to rootfs source plugin. (rev11)
> Revision: 11
> URL   : https://patchwork.openembedded.org/series/4087/
> State : failure
> 
> == Summary ==
> 
> 
> Thank you for submitting this patch series to OpenEmbedded Core. This is
> an automated response. Several tests have been executed on the proposed
> series by patchtest resulting in the following failures:
> 
> 
> 
> * Patch            [v2,2/3] Add e2tools recipe, in order to test contents of images.
>  Issue             Shortlog does not follow expected format [test_shortlog_format] 
>   Suggested fix    Commit shortlog (first line of commit message) should follow the format "<target>: <summary>"
> 
> * Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
>   Suggested fix    Rebase your series on top of targeted branch
>   Targeted branch  master (currently at b3a74335e1)
> 
> 
> 
> If you believe any of these test results are incorrect, please reply to the
> mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
> Otherwise we would appreciate you correcting the issues and submitting a new
> version of the patchset if applicable. Please ensure you add/increment the
> version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
> [PATCH v3] -> ...).
> 
> ---
> Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
> Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
>