[PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda Delgado]

When parameters include_path or exclude_path are passed to the rootfs
plugin, it will copy the partition content into a folder and make all
the modifications there.

This is done using copyhardlinktree(), which does not take into
consideration the content of the pseudo folder, which contains the
information about the right permissions and ownership of the folders.

This results in a rootfs owned by the user that is running the wic
command (usually UID 1000), which makes some rootfs unbootable.

To fix this we copy the content of the pseudo folders to the new folder
and modify the pseudo database using the "pseudo -B" command.

Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
 scripts/lib/wic/plugins/source/rootfs.py | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 705aeb5563..40419a64b3 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -16,11 +16,11 @@ import os
 import shutil
 import sys
 
-from oe.path import copyhardlinktree
+from oe.path import copyhardlinktree, copytree
 
 from wic import WicError
 from wic.pluginbase import SourcePlugin
-from wic.misc import get_bitbake_var
+from wic.misc import get_bitbake_var, exec_native_cmd
 
 logger = logging.getLogger('wic')
 
@@ -44,6 +44,15 @@ class RootfsPlugin(SourcePlugin):
 
         return os.path.realpath(image_rootfs_dir)
 
+    @staticmethod
+    def __get_pseudo(native_sysroot, rootfs):
+        pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
+        pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % os.path.join(rootfs, "../pseudo")
+        pseudo += "export PSEUDO_PASSWD=%s;" % rootfs
+        pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
+        pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+        return pseudo
+
     @classmethod
     def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
                              oe_builddir, bootimg_dir, kernel_dir,
@@ -78,9 +87,16 @@ class RootfsPlugin(SourcePlugin):
 
             if os.path.lexists(new_rootfs):
                 shutil.rmtree(os.path.join(new_rootfs))
-
             copyhardlinktree(part.rootfs_dir, new_rootfs)
 
+            if os.path.lexists(os.path.join(new_rootfs, "../pseudo")):
+                shutil.rmtree(os.path.join(new_rootfs, "../pseudo"))
+            copytree(os.path.join(part.rootfs_dir, "../pseudo"),
+                     os.path.join(new_rootfs, "../pseudo"))
+            pseudo_cmd = "%s -B -m %s -M %s" % (cls.__get_pseudo(native_sysroot,new_rootfs),
+                                                part.rootfs_dir, new_rootfs)
+            exec_native_cmd(pseudo_cmd, native_sysroot)
+
             for path in part.include_path or []:
                 copyhardlinktree(path, new_rootfs)
 
-- 
2.25.1

[PATCH v2 2/2] wic: Add --embed-rootfs argument

[Ricardo Ribalda Delgado]

This option adds the content of a rootfs on a specific location on the
rootfs.

It is very useful for making a partition that contains the rootfs for a
host and a target Eg:

/ -> Roofs for the host
/export/ -> Rootfs for the target (which will netboot)

Although today we support making a partition for "/export" this might
not be compatible with some upgrade systems, or we might be limited by
the number of partitions.

With this patch we can use something like:

part / --source rootfs --embed-rootfs target-image /export --embed-rootfs target-image2 /export2

on the .wks file.

Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
 scripts/lib/wic/help.py                  |  8 ++++++++
 scripts/lib/wic/ksparser.py              |  1 +
 scripts/lib/wic/partition.py             |  1 +
 scripts/lib/wic/plugins/source/rootfs.py | 22 +++++++++++++++++++++-
 4 files changed, 31 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 4d342fcf05..140dc504cd 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -979,6 +979,14 @@ DESCRIPTION
                          copies. This option only has an effect with the rootfs
                          source plugin.
 
+         --embed-rootfs: This option is specific to wic. It embeds a rootfs into
+                         the given path to the resulting image. The option
+                         contains two fields, the roofs and the path, separated
+                         by a space. The rootfs follows the same logic as the
+                         rootfs-dir argument. Multiple options can be provided
+                         in order to embed multiple rootfs. This option only has
+                         an effect with the rootfs source plugin.
+
          --extra-space: This option is specific to wic. It adds extra
                         space after the space filled by the content
                         of the partition. The final size can go
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 650b976223..64c8c1175e 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -138,6 +138,7 @@ class KickStart():
         part.add_argument('--align', type=int)
         part.add_argument('--exclude-path', nargs='+')
         part.add_argument('--include-path', nargs='+')
+        part.add_argument('--embed-rootfs', nargs=2, action='append')
         part.add_argument("--extra-space", type=sizetype)
         part.add_argument('--fsoptions', dest='fsopts')
         part.add_argument('--fstype', default='vfat',
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 2d95f78439..13857df82f 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -31,6 +31,7 @@ class Partition():
         self.extra_space = args.extra_space
         self.exclude_path = args.exclude_path
         self.include_path = args.include_path
+        self.embed_rootfs = args.embed_rootfs
         self.fsopts = args.fsopts
         self.fstype = args.fstype
         self.label = args.label
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 40419a64b3..089aaea477 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -17,6 +17,7 @@ import shutil
 import sys
 
 from oe.path import copyhardlinktree, copytree
+from pathlib import Path
 
 from wic import WicError
 from wic.pluginbase import SourcePlugin
@@ -80,7 +81,7 @@ class RootfsPlugin(SourcePlugin):
 
         new_rootfs = None
         # Handle excluded paths.
-        if part.exclude_path or part.include_path:
+        if part.exclude_path or part.include_path or part.embed_rootfs:
             # We need a new rootfs directory we can delete files from. Copy to
             # workdir.
             new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno))
@@ -100,6 +101,25 @@ class RootfsPlugin(SourcePlugin):
             for path in part.include_path or []:
                 copyhardlinktree(path, new_rootfs)
 
+            for embed in part.embed_rootfs or []:
+                [embed_rootfs, path] = embed
+                #we need to remove the initial / for os.path.join to work
+                if os.path.isabs(path):
+                    path = path[1:]
+                if embed_rootfs in krootfs_dir:
+                    embed_rootfs = krootfs_dir[embed_rootfs]
+                embed_rootfs = cls.__get_rootfs_dir(embed_rootfs)
+                tar_file = os.path.realpath(os.path.join(cr_workdir, "aux.tar"))
+                tar_cmd = "%s tar cpf %s -C %s ." % (cls.__get_pseudo(native_sysroot,
+                                                     embed_rootfs), tar_file, embed_rootfs)
+                exec_native_cmd(tar_cmd, native_sysroot)
+                untar_cmd = "%s tar xf %s -C %s ." % (cls.__get_pseudo(native_sysroot, new_rootfs),
+                                                      tar_file, os.path.join(new_rootfs, path))
+                Path(os.path.join(new_rootfs, path)).mkdir(parents=True, exist_ok=True)
+                exec_native_cmd(untar_cmd, native_sysroot,
+                                cls.__get_pseudo(native_sysroot, new_rootfs))
+                os.remove(tar_file)
+
             for orig_path in part.exclude_path or []:
                 path = orig_path
                 if os.path.isabs(path):
-- 
2.25.1

Re: [PATCH v2 2/2] wic: Add --embed-rootfs argument

[Paul Barker]

On Wed,  4 Mar 2020 15:49:36 +0100
Ricardo Ribalda Delgado <ricardo@ribalda.com> wrote:

> This option adds the content of a rootfs on a specific location on the
> rootfs.
> 
> It is very useful for making a partition that contains the rootfs for a
> host and a target Eg:
> 
> / -> Roofs for the host
> /export/ -> Rootfs for the target (which will netboot)
> 
> Although today we support making a partition for "/export" this might
> not be compatible with some upgrade systems, or we might be limited by
> the number of partitions.
> 
> With this patch we can use something like:
> 
> part / --source rootfs --embed-rootfs target-image /export --embed-rootfs target-image2 /export2

I like this but it still leaves confusion between `--include-path` and
--embed-rootfs` as they're similar but slightly different. Can we just modify
`--include-path` to have this syntax?

> 
> on the .wks file.
> 
> Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
> ---
>  scripts/lib/wic/help.py                  |  8 ++++++++
>  scripts/lib/wic/ksparser.py              |  1 +
>  scripts/lib/wic/partition.py             |  1 +
>  scripts/lib/wic/plugins/source/rootfs.py | 22 +++++++++++++++++++++-
>  4 files changed, 31 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
> index 4d342fcf05..140dc504cd 100644
> --- a/scripts/lib/wic/help.py
> +++ b/scripts/lib/wic/help.py
> @@ -979,6 +979,14 @@ DESCRIPTION
>                           copies. This option only has an effect with the rootfs
>                           source plugin.
>  
> +         --embed-rootfs: This option is specific to wic. It embeds a rootfs into
> +                         the given path to the resulting image. The option
> +                         contains two fields, the roofs and the path, separated
> +                         by a space. The rootfs follows the same logic as the
> +                         rootfs-dir argument. Multiple options can be provided
> +                         in order to embed multiple rootfs. This option only has
> +                         an effect with the rootfs source plugin.
> +
>           --extra-space: This option is specific to wic. It adds extra
>                          space after the space filled by the content
>                          of the partition. The final size can go
> diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> index 650b976223..64c8c1175e 100644
> --- a/scripts/lib/wic/ksparser.py
> +++ b/scripts/lib/wic/ksparser.py
> @@ -138,6 +138,7 @@ class KickStart():
>          part.add_argument('--align', type=int)
>          part.add_argument('--exclude-path', nargs='+')
>          part.add_argument('--include-path', nargs='+')
> +        part.add_argument('--embed-rootfs', nargs=2, action='append')
>          part.add_argument("--extra-space", type=sizetype)
>          part.add_argument('--fsoptions', dest='fsopts')
>          part.add_argument('--fstype', default='vfat',
> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> index 2d95f78439..13857df82f 100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -31,6 +31,7 @@ class Partition():
>          self.extra_space = args.extra_space
>          self.exclude_path = args.exclude_path
>          self.include_path = args.include_path
> +        self.embed_rootfs = args.embed_rootfs
>          self.fsopts = args.fsopts
>          self.fstype = args.fstype
>          self.label = args.label
> diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
> index 40419a64b3..089aaea477 100644
> --- a/scripts/lib/wic/plugins/source/rootfs.py
> +++ b/scripts/lib/wic/plugins/source/rootfs.py
> @@ -17,6 +17,7 @@ import shutil
>  import sys
>  
>  from oe.path import copyhardlinktree, copytree
> +from pathlib import Path
>  
>  from wic import WicError
>  from wic.pluginbase import SourcePlugin
> @@ -80,7 +81,7 @@ class RootfsPlugin(SourcePlugin):
>  
>          new_rootfs = None
>          # Handle excluded paths.
> -        if part.exclude_path or part.include_path:
> +        if part.exclude_path or part.include_path or part.embed_rootfs:
>              # We need a new rootfs directory we can delete files from. Copy to
>              # workdir.
>              new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno))
> @@ -100,6 +101,25 @@ class RootfsPlugin(SourcePlugin):
>              for path in part.include_path or []:
>                  copyhardlinktree(path, new_rootfs)
>  
> +            for embed in part.embed_rootfs or []:
> +                [embed_rootfs, path] = embed
> +                #we need to remove the initial / for os.path.join to work
> +                if os.path.isabs(path):
> +                    path = path[1:]
> +                if embed_rootfs in krootfs_dir:
> +                    embed_rootfs = krootfs_dir[embed_rootfs]
> +                embed_rootfs = cls.__get_rootfs_dir(embed_rootfs)
> +                tar_file = os.path.realpath(os.path.join(cr_workdir, "aux.tar"))
> +                tar_cmd = "%s tar cpf %s -C %s ." % (cls.__get_pseudo(native_sysroot,
> +                                                     embed_rootfs), tar_file, embed_rootfs)
> +                exec_native_cmd(tar_cmd, native_sysroot)
> +                untar_cmd = "%s tar xf %s -C %s ." % (cls.__get_pseudo(native_sysroot, new_rootfs),
> +                                                      tar_file, os.path.join(new_rootfs, path))
> +                Path(os.path.join(new_rootfs, path)).mkdir(parents=True, exist_ok=True)
> +                exec_native_cmd(untar_cmd, native_sysroot,
> +                                cls.__get_pseudo(native_sysroot, new_rootfs))
> +                os.remove(tar_file)
> +
>              for orig_path in part.exclude_path or []:
>                  path = orig_path
>                  if os.path.isabs(path):

As said in my other email, if you're running wic outside bitbake I'm not sure
you can guarantee pseudo is available. And now I think about it, if you're
running wic inside bitbake (as part of do_image_wic), you'd be running pseudo
under pseudo and I have no idea how that would work out.

-- 
Paul Barker
Konsulko Group

Re: [PATCH v2 2/2] wic: Add --embed-rootfs argument

[Ricardo Ribalda Delgado]

Hi Paul,

On Thu, Mar 5, 2020 at 10:37 AM Paul Barker <pbarker@konsulko.com> wrote:
>
> On Wed,  4 Mar 2020 15:49:36 +0100
> Ricardo Ribalda Delgado <ricardo@ribalda.com> wrote:
>
> > This option adds the content of a rootfs on a specific location on the
> > rootfs.
> >
> > It is very useful for making a partition that contains the rootfs for a
> > host and a target Eg:
> >
> > / -> Roofs for the host
> > /export/ -> Rootfs for the target (which will netboot)
> >
> > Although today we support making a partition for "/export" this might
> > not be compatible with some upgrade systems, or we might be limited by
> > the number of partitions.
> >
> > With this patch we can use something like:
> >
> > part / --source rootfs --embed-rootfs target-image /export --embed-rootfs target-image2 /export2
>
> I like this but it still leaves confusion between `--include-path` and
> --embed-rootfs` as they're similar but slightly different. Can we just modify
> `--include-path` to have this syntax?

I think they are different enough.

- include-path ads a file/folder
- embed-rootfs adds a rootfs, which is either a folder or a image.bb file.


>
> >
> > on the .wks file.
> >
> > Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
> > ---
> >  scripts/lib/wic/help.py                  |  8 ++++++++
> >  scripts/lib/wic/ksparser.py              |  1 +
> >  scripts/lib/wic/partition.py             |  1 +
> >  scripts/lib/wic/plugins/source/rootfs.py | 22 +++++++++++++++++++++-
> >  4 files changed, 31 insertions(+), 1 deletion(-)
> >
> > diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
> > index 4d342fcf05..140dc504cd 100644
> > --- a/scripts/lib/wic/help.py
> > +++ b/scripts/lib/wic/help.py
> > @@ -979,6 +979,14 @@ DESCRIPTION
> >                           copies. This option only has an effect with the rootfs
> >                           source plugin.
> >
> > +         --embed-rootfs: This option is specific to wic. It embeds a rootfs into
> > +                         the given path to the resulting image. The option
> > +                         contains two fields, the roofs and the path, separated
> > +                         by a space. The rootfs follows the same logic as the
> > +                         rootfs-dir argument. Multiple options can be provided
> > +                         in order to embed multiple rootfs. This option only has
> > +                         an effect with the rootfs source plugin.
> > +
> >           --extra-space: This option is specific to wic. It adds extra
> >                          space after the space filled by the content
> >                          of the partition. The final size can go
> > diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> > index 650b976223..64c8c1175e 100644
> > --- a/scripts/lib/wic/ksparser.py
> > +++ b/scripts/lib/wic/ksparser.py
> > @@ -138,6 +138,7 @@ class KickStart():
> >          part.add_argument('--align', type=int)
> >          part.add_argument('--exclude-path', nargs='+')
> >          part.add_argument('--include-path', nargs='+')
> > +        part.add_argument('--embed-rootfs', nargs=2, action='append')
> >          part.add_argument("--extra-space", type=sizetype)
> >          part.add_argument('--fsoptions', dest='fsopts')
> >          part.add_argument('--fstype', default='vfat',
> > diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> > index 2d95f78439..13857df82f 100644
> > --- a/scripts/lib/wic/partition.py
> > +++ b/scripts/lib/wic/partition.py
> > @@ -31,6 +31,7 @@ class Partition():
> >          self.extra_space = args.extra_space
> >          self.exclude_path = args.exclude_path
> >          self.include_path = args.include_path
> > +        self.embed_rootfs = args.embed_rootfs
> >          self.fsopts = args.fsopts
> >          self.fstype = args.fstype
> >          self.label = args.label
> > diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
> > index 40419a64b3..089aaea477 100644
> > --- a/scripts/lib/wic/plugins/source/rootfs.py
> > +++ b/scripts/lib/wic/plugins/source/rootfs.py
> > @@ -17,6 +17,7 @@ import shutil
> >  import sys
> >
> >  from oe.path import copyhardlinktree, copytree
> > +from pathlib import Path
> >
> >  from wic import WicError
> >  from wic.pluginbase import SourcePlugin
> > @@ -80,7 +81,7 @@ class RootfsPlugin(SourcePlugin):
> >
> >          new_rootfs = None
> >          # Handle excluded paths.
> > -        if part.exclude_path or part.include_path:
> > +        if part.exclude_path or part.include_path or part.embed_rootfs:
> >              # We need a new rootfs directory we can delete files from. Copy to
> >              # workdir.
> >              new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno))
> > @@ -100,6 +101,25 @@ class RootfsPlugin(SourcePlugin):
> >              for path in part.include_path or []:
> >                  copyhardlinktree(path, new_rootfs)
> >
> > +            for embed in part.embed_rootfs or []:
> > +                [embed_rootfs, path] = embed
> > +                #we need to remove the initial / for os.path.join to work
> > +                if os.path.isabs(path):
> > +                    path = path[1:]
> > +                if embed_rootfs in krootfs_dir:
> > +                    embed_rootfs = krootfs_dir[embed_rootfs]
> > +                embed_rootfs = cls.__get_rootfs_dir(embed_rootfs)
> > +                tar_file = os.path.realpath(os.path.join(cr_workdir, "aux.tar"))
> > +                tar_cmd = "%s tar cpf %s -C %s ." % (cls.__get_pseudo(native_sysroot,
> > +                                                     embed_rootfs), tar_file, embed_rootfs)
> > +                exec_native_cmd(tar_cmd, native_sysroot)
> > +                untar_cmd = "%s tar xf %s -C %s ." % (cls.__get_pseudo(native_sysroot, new_rootfs),
> > +                                                      tar_file, os.path.join(new_rootfs, path))
> > +                Path(os.path.join(new_rootfs, path)).mkdir(parents=True, exist_ok=True)
> > +                exec_native_cmd(untar_cmd, native_sysroot,
> > +                                cls.__get_pseudo(native_sysroot, new_rootfs))
> > +                os.remove(tar_file)
> > +
> >              for orig_path in part.exclude_path or []:
> >                  path = orig_path
> >                  if os.path.isabs(path):
>
> As said in my other email, if you're running wic outside bitbake I'm not sure
> you can guarantee pseudo is available. And now I think about it, if you're
> running wic inside bitbake (as part of do_image_wic), you'd be running pseudo
> under pseudo and I have no idea how that would work out.
>

pseudo is warranted because wic launches "bitbake wic-tools", which
has as DEPENDS pseudo-native.

scripts/wic:        subprocess.check_call(["bitbake", "wic-tools"])


I have tried to run it inside bitbake with --exclude-path and
everything seems to work fine.



> --
> Paul Barker
> Konsulko Group



-- 
Ricardo Ribalda

Re: [PATCH v2 2/2] wic: Add --embed-rootfs argument

[Ricardo Ribalda]

ping?

On Thu, Mar 5, 2020 at 1:26 PM Ricardo Ribalda Delgado
<ricardo@ribalda.com> wrote:
>
> Hi Paul,
>
> On Thu, Mar 5, 2020 at 10:37 AM Paul Barker <pbarker@konsulko.com> wrote:
> >
> > On Wed,  4 Mar 2020 15:49:36 +0100
> > Ricardo Ribalda Delgado <ricardo@ribalda.com> wrote:
> >
> > > This option adds the content of a rootfs on a specific location on the
> > > rootfs.
> > >
> > > It is very useful for making a partition that contains the rootfs for a
> > > host and a target Eg:
> > >
> > > / -> Roofs for the host
> > > /export/ -> Rootfs for the target (which will netboot)
> > >
> > > Although today we support making a partition for "/export" this might
> > > not be compatible with some upgrade systems, or we might be limited by
> > > the number of partitions.
> > >
> > > With this patch we can use something like:
> > >
> > > part / --source rootfs --embed-rootfs target-image /export --embed-rootfs target-image2 /export2
> >
> > I like this but it still leaves confusion between `--include-path` and
> > --embed-rootfs` as they're similar but slightly different. Can we just modify
> > `--include-path` to have this syntax?
>
> I think they are different enough.
>
> - include-path ads a file/folder
> - embed-rootfs adds a rootfs, which is either a folder or a image.bb file.
>
>
> >
> > >
> > > on the .wks file.
> > >
> > > Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
> > > ---
> > >  scripts/lib/wic/help.py                  |  8 ++++++++
> > >  scripts/lib/wic/ksparser.py              |  1 +
> > >  scripts/lib/wic/partition.py             |  1 +
> > >  scripts/lib/wic/plugins/source/rootfs.py | 22 +++++++++++++++++++++-
> > >  4 files changed, 31 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
> > > index 4d342fcf05..140dc504cd 100644
> > > --- a/scripts/lib/wic/help.py
> > > +++ b/scripts/lib/wic/help.py
> > > @@ -979,6 +979,14 @@ DESCRIPTION
> > >                           copies. This option only has an effect with the rootfs
> > >                           source plugin.
> > >
> > > +         --embed-rootfs: This option is specific to wic. It embeds a rootfs into
> > > +                         the given path to the resulting image. The option
> > > +                         contains two fields, the roofs and the path, separated
> > > +                         by a space. The rootfs follows the same logic as the
> > > +                         rootfs-dir argument. Multiple options can be provided
> > > +                         in order to embed multiple rootfs. This option only has
> > > +                         an effect with the rootfs source plugin.
> > > +
> > >           --extra-space: This option is specific to wic. It adds extra
> > >                          space after the space filled by the content
> > >                          of the partition. The final size can go
> > > diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> > > index 650b976223..64c8c1175e 100644
> > > --- a/scripts/lib/wic/ksparser.py
> > > +++ b/scripts/lib/wic/ksparser.py
> > > @@ -138,6 +138,7 @@ class KickStart():
> > >          part.add_argument('--align', type=int)
> > >          part.add_argument('--exclude-path', nargs='+')
> > >          part.add_argument('--include-path', nargs='+')
> > > +        part.add_argument('--embed-rootfs', nargs=2, action='append')
> > >          part.add_argument("--extra-space", type=sizetype)
> > >          part.add_argument('--fsoptions', dest='fsopts')
> > >          part.add_argument('--fstype', default='vfat',
> > > diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> > > index 2d95f78439..13857df82f 100644
> > > --- a/scripts/lib/wic/partition.py
> > > +++ b/scripts/lib/wic/partition.py
> > > @@ -31,6 +31,7 @@ class Partition():
> > >          self.extra_space = args.extra_space
> > >          self.exclude_path = args.exclude_path
> > >          self.include_path = args.include_path
> > > +        self.embed_rootfs = args.embed_rootfs
> > >          self.fsopts = args.fsopts
> > >          self.fstype = args.fstype
> > >          self.label = args.label
> > > diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
> > > index 40419a64b3..089aaea477 100644
> > > --- a/scripts/lib/wic/plugins/source/rootfs.py
> > > +++ b/scripts/lib/wic/plugins/source/rootfs.py
> > > @@ -17,6 +17,7 @@ import shutil
> > >  import sys
> > >
> > >  from oe.path import copyhardlinktree, copytree
> > > +from pathlib import Path
> > >
> > >  from wic import WicError
> > >  from wic.pluginbase import SourcePlugin
> > > @@ -80,7 +81,7 @@ class RootfsPlugin(SourcePlugin):
> > >
> > >          new_rootfs = None
> > >          # Handle excluded paths.
> > > -        if part.exclude_path or part.include_path:
> > > +        if part.exclude_path or part.include_path or part.embed_rootfs:
> > >              # We need a new rootfs directory we can delete files from. Copy to
> > >              # workdir.
> > >              new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno))
> > > @@ -100,6 +101,25 @@ class RootfsPlugin(SourcePlugin):
> > >              for path in part.include_path or []:
> > >                  copyhardlinktree(path, new_rootfs)
> > >
> > > +            for embed in part.embed_rootfs or []:
> > > +                [embed_rootfs, path] = embed
> > > +                #we need to remove the initial / for os.path.join to work
> > > +                if os.path.isabs(path):
> > > +                    path = path[1:]
> > > +                if embed_rootfs in krootfs_dir:
> > > +                    embed_rootfs = krootfs_dir[embed_rootfs]
> > > +                embed_rootfs = cls.__get_rootfs_dir(embed_rootfs)
> > > +                tar_file = os.path.realpath(os.path.join(cr_workdir, "aux.tar"))
> > > +                tar_cmd = "%s tar cpf %s -C %s ." % (cls.__get_pseudo(native_sysroot,
> > > +                                                     embed_rootfs), tar_file, embed_rootfs)
> > > +                exec_native_cmd(tar_cmd, native_sysroot)
> > > +                untar_cmd = "%s tar xf %s -C %s ." % (cls.__get_pseudo(native_sysroot, new_rootfs),
> > > +                                                      tar_file, os.path.join(new_rootfs, path))
> > > +                Path(os.path.join(new_rootfs, path)).mkdir(parents=True, exist_ok=True)
> > > +                exec_native_cmd(untar_cmd, native_sysroot,
> > > +                                cls.__get_pseudo(native_sysroot, new_rootfs))
> > > +                os.remove(tar_file)
> > > +
> > >              for orig_path in part.exclude_path or []:
> > >                  path = orig_path
> > >                  if os.path.isabs(path):
> >
> > As said in my other email, if you're running wic outside bitbake I'm not sure
> > you can guarantee pseudo is available. And now I think about it, if you're
> > running wic inside bitbake (as part of do_image_wic), you'd be running pseudo
> > under pseudo and I have no idea how that would work out.
> >
>
> pseudo is warranted because wic launches "bitbake wic-tools", which
> has as DEPENDS pseudo-native.
>
> scripts/wic:        subprocess.check_call(["bitbake", "wic-tools"])
>
>
> I have tried to run it inside bitbake with --exclude-path and
> everything seems to work fine.
>
>
>
> > --
> > Paul Barker
> > Konsulko Group
>
>
>
> --
> Ricardo Ribalda



-- 
Ricardo Ribalda

Re: [OE-core] [PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Richard Purdie]

On Wed, 2020-03-04 at 15:49 +0100, Ricardo Ribalda Delgado wrote:
> When parameters include_path or exclude_path are passed to the rootfs
> plugin, it will copy the partition content into a folder and make all
> the modifications there.
> 
> This is done using copyhardlinktree(), which does not take into
> consideration the content of the pseudo folder, which contains the
> information about the right permissions and ownership of the folders.
> 
> This results in a rootfs owned by the user that is running the wic
> command (usually UID 1000), which makes some rootfs unbootable.
> 
> To fix this we copy the content of the pseudo folders to the new
> folder
> and modify the pseudo database using the "pseudo -B" command.
> 
> Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
> ---
>  scripts/lib/wic/plugins/source/rootfs.py | 22 +++++++++++++++++++---
>  1 file changed, 19 insertions(+), 3 deletions(-)

I added these two changes to -next but we're seeing a test failure on
Fedora, twice now:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/834
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/831

I suspect its related to these patches but haven't confirmed that as
yet...

Cheers,

Richard

Re: [OE-core] [PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

[-- Attachment #1: Type: text/plain, Size: 1554 bytes --]

Hi Richard

It looks pretty much related to my patch. Will try to replicate locally and
fix it.

Thanks!

On Sun, 5 Apr 2020, 09:01 Richard Purdie, <
richard.purdie@linuxfoundation.org> wrote:

> On Wed, 2020-03-04 at 15:49 +0100, Ricardo Ribalda Delgado wrote:
> > When parameters include_path or exclude_path are passed to the rootfs
> > plugin, it will copy the partition content into a folder and make all
> > the modifications there.
> >
> > This is done using copyhardlinktree(), which does not take into
> > consideration the content of the pseudo folder, which contains the
> > information about the right permissions and ownership of the folders.
> >
> > This results in a rootfs owned by the user that is running the wic
> > command (usually UID 1000), which makes some rootfs unbootable.
> >
> > To fix this we copy the content of the pseudo folders to the new
> > folder
> > and modify the pseudo database using the "pseudo -B" command.
> >
> > Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
> > ---
> >  scripts/lib/wic/plugins/source/rootfs.py | 22 +++++++++++++++++++---
> >  1 file changed, 19 insertions(+), 3 deletions(-)
>
> I added these two changes to -next but we're seeing a test failure on
> Fedora, twice now:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/834
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/831
>
> I suspect its related to these patches but haven't confirmed that as
> yet...
>
> Cheers,
>
> Richard
>
> 
>

[-- Attachment #2: Type: text/html, Size: 2424 bytes --]

Re: [OE-core] [PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

Hi Richard

I have tried with master  (0c91fcba446418ad1f71d3df9aa3b186bbd353c7) +
my two patches.
Then I have used crops/poky:fedora-30

and: oe-selftest  --run-tests wic.Wic.test_exclude_path oe-selftest
--run-tests wic.Wic.test_include_path wic.Wic.test_exclude_path

With an OK result :(
[pokyuser@fac53302a692 build]$ oe-selftest  --run-tests
wic.Wic.test_exclude_path oe-selftest  --run-tests
wic.Wic.test_include_path
2020-04-05 07:40:50,186 - oe-selftest - INFO - Adding layer libraries:
2020-04-05 07:40:50,186 - oe-selftest - INFO - /workdir/meta/lib
2020-04-05 07:40:50,186 - oe-selftest - INFO - /workdir/meta-selftest/lib
2020-04-05 07:40:50,187 - oe-selftest - INFO - Running bitbake -e to
test the configuration is valid/parsable
2020-04-05 07:40:52,149 - oe-selftest - INFO - Adding: "include
selftest.inc" in /workdir/build/conf/local.conf
2020-04-05 07:40:52,150 - oe-selftest - INFO - Adding: "include
bblayers.inc" in bblayers.conf
2020-04-05 07:40:52,150 - oe-selftest - INFO - test_include_path (wic.Wic)
2020-04-05 09:04:12,656 - oe-selftest - INFO - Keepalive message
2020-04-05 09:29:51,214 - oe-selftest - INFO -  ... ok
2020-04-05 09:29:51,230 - oe-selftest - INFO -
----------------------------------------------------------------------
2020-04-05 09:29:51,230 - oe-selftest - INFO - Ran 1 test in 6539.080s
2020-04-05 09:29:51,231 - oe-selftest - INFO - OK
2020-04-05 09:29:52,869 - oe-selftest - INFO - RESULTS:
2020-04-05 09:29:52,870 - oe-selftest - INFO - RESULTS -
wic.Wic.test_include_path: PASSED (6539.07s)
2020-04-05 09:29:52,874 - oe-selftest - INFO - SUMMARY:
2020-04-05 09:29:52,875 - oe-selftest - INFO - oe-selftest () - Ran 1
test in 6540.086s
2020-04-05 09:29:52,875 - oe-selftest - INFO - oe-selftest - OK - All
required tests passed (successes=1, skipped=0, failures=0, errors=0)

I will start a test with mater-next + my two patches. Hopefuly I can
replicate the bug.

It is a bit weird that only crashes on fedora and not on the other distros :S


Thanks again

On Sun, Apr 5, 2020 at 9:21 AM Ricardo Ribalda via
lists.openembedded.org
<ricardo.ribalda=gmail.com@lists.openembedded.org> wrote:
>
> Hi Richard
>
> It looks pretty much related to my patch. Will try to replicate locally and fix it.
>
> Thanks!
>
> On Sun, 5 Apr 2020, 09:01 Richard Purdie, <richard.purdie@linuxfoundation.org> wrote:
>>
>> On Wed, 2020-03-04 at 15:49 +0100, Ricardo Ribalda Delgado wrote:
>> > When parameters include_path or exclude_path are passed to the rootfs
>> > plugin, it will copy the partition content into a folder and make all
>> > the modifications there.
>> >
>> > This is done using copyhardlinktree(), which does not take into
>> > consideration the content of the pseudo folder, which contains the
>> > information about the right permissions and ownership of the folders.
>> >
>> > This results in a rootfs owned by the user that is running the wic
>> > command (usually UID 1000), which makes some rootfs unbootable.
>> >
>> > To fix this we copy the content of the pseudo folders to the new
>> > folder
>> > and modify the pseudo database using the "pseudo -B" command.
>> >
>> > Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
>> > ---
>> >  scripts/lib/wic/plugins/source/rootfs.py | 22 +++++++++++++++++++---
>> >  1 file changed, 19 insertions(+), 3 deletions(-)
>>
>> I added these two changes to -next but we're seeing a test failure on
>> Fedora, twice now:
>>
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/834
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/831
>>
>> I suspect its related to these patches but haven't confirmed that as
>> yet...
>>
>> Cheers,
>>
>> Richard
>>
>>
> 



-- 
Ricardo Ribalda

Re: [OE-core] [PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Richard Purdie]

On Sun, 2020-04-05 at 11:56 +0200, Ricardo Ribalda Delgado wrote:
> Hi Richard
> 
> I have tried with master  (0c91fcba446418ad1f71d3df9aa3b186bbd353c7)
> +
> my two patches.
> Then I have used crops/poky:fedora-30
> 
> and: oe-selftest  --run-tests wic.Wic.test_exclude_path oe-selftest
> --run-tests wic.Wic.test_include_path wic.Wic.test_exclude_path
> 
> With an OK result :(

I worried this may be the case. There definitely is something wrong and
I could replicate it a third time too.

My guess is that there are perhaps files in /tmp/ which are logged in
the pseudo database but don't exist when the second rootfs is created.
Since the files no longer exist, the database integrity check fails.

Why that only occurs on Fedora case on the autobuilder, I don't know,
I'm just hypothesising right now...

Cheers,

Richard

Re: [OE-core] [PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

[-- Attachment #1: Type: text/plain, Size: 1411 bytes --]

Hi Richard


On Sun, Apr 5, 2020 at 1:04 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Sun, 2020-04-05 at 11:56 +0200, Ricardo Ribalda Delgado wrote:
> > Hi Richard
> >
> > I have tried with master  (0c91fcba446418ad1f71d3df9aa3b186bbd353c7)
> > +
> > my two patches.
> > Then I have used crops/poky:fedora-30
> >
> > and: oe-selftest  --run-tests wic.Wic.test_exclude_path oe-selftest
> > --run-tests wic.Wic.test_include_path wic.Wic.test_exclude_path
> >
> > With an OK result :(
>
> I worried this may be the case. There definitely is something wrong and
> I could replicate it a third time too.
>
> My guess is that there are perhaps files in /tmp/ which are logged in
> the pseudo database but don't exist when the second rootfs is created.
> Since the files no longer exist, the database integrity check fails.

That is definitely a good hypothesis. Since you can replicate the bug,
can you try to add also the attached patch to your tree and re-run the
test.

What it does, is to remove a file from the database if it has
vanished, during the bitbake move.

If it works I can add it to the beginning of my series, or even trying
to upstream it to pseudo.

Best regards

>
> Why that only occurs on Fedora case on the autobuilder, I don't know,
> I'm just hypothesising right now...
>
> Cheers,
>
> Richard
>


-- 
Ricardo Ribalda

[-- Attachment #2: 0001-pseudo-During-DB-fix-remove-files-that-do-not-exist.patch --]
[-- Type: text/x-patch, Size: 3340 bytes --]

From bbddec527d9056e1feda0d21edd8ddb9d64623b2 Mon Sep 17 00:00:00 2001
From: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Date: Sun, 5 Apr 2020 13:47:12 +0200
Subject: [PATCH] pseudo: During DB fix, remove files that do not exist

If a file does not exist, either because it has been removed outside
bitbake, or because only some of the files have been moved to a
different location, delete it from the pseudo-db is the user decides to
fix the database.

Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
 ...ixup-remove-files-that-do-not-exist-.patch | 48 +++++++++++++++++++
 meta/recipes-devtools/pseudo/pseudo_git.bb    |  1 +
 2 files changed, 49 insertions(+)
 create mode 100644 meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch

diff --git a/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch b/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch
new file mode 100644
index 0000000000..8aec1c9551
--- /dev/null
+++ b/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch
@@ -0,0 +1,48 @@
+From b0902e36108b49e6bc88d6b251cc2f8cffcd5a13 Mon Sep 17 00:00:00 2001
+From: Ricardo Ribalda <ricardo@ribalda.com>
+Date: Sun, 5 Apr 2020 11:40:30 +0000
+Subject: [PATCH] pseudo: On a DB fixup remove files that do not exist anymore
+
+If the user decides to fix a database, remove the files that do not
+exist anymore.
+If only DB test is selected do not change the behaviour (return error).
+
+Signed-off-by: Ricardo Ribalda <ricardo@ribalda.com>
+---
+ pseudo.c | 13 ++++++++++---
+ 1 file changed, 10 insertions(+), 3 deletions(-)
+
+diff --git a/pseudo.c b/pseudo.c
+index 0f5850e..98e5b0c 100644
+--- a/pseudo.c
++++ b/pseudo.c
+@@ -1087,9 +1087,15 @@ pseudo_db_check(int fix) {
+ 			int fixup_needed = 0;
+ 			pseudo_debug(PDBGF_DB, "Checking <%s>\n", m->path);
+ 			if (lstat(m->path, &buf)) {
+-				errors = EXIT_FAILURE;
+-				pseudo_diag("can't stat <%s>\n", m->path);
+-				continue;
++				if (!fix) {
++					pseudo_diag("can't stat <%s>\n", m->path);
++					errors = EXIT_FAILURE;
++					continue;
++				} else {
++					pseudo_debug("can't stat <%s>\n", m->path);
++					fixup_needed = 2;
++					goto do_fixup;
++				}
+ 			}
+ 			/* can't check for device type mismatches, uid/gid, or
+ 			 * permissions, because those are the very things we
+@@ -1125,6 +1131,7 @@ pseudo_db_check(int fix) {
+ 					S_ISDIR(m->mode));
+ 				fixup_needed = 2;
+ 			}
++			do_fixup:
+ 			if (fixup_needed) {
+ 				/* in fixup mode, either delete (mismatches) or
+ 				 * correct (dev/ino).
+-- 
+2.21.1
+
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index 89e43c5996..a3049c5e6c 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -11,6 +11,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo \
            file://0001-realpath.c-Remove-trailing-slashes.patch \
            file://0006-xattr-adjust-for-attr-2.4.48-release.patch \
            file://seccomp.patch \
+           file://0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch \
            "
 
 SRCREV = "060058bb29f70b244e685b3c704eb0641b736f73"
-- 
2.25.1

Re: [OE-core] [PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

[-- Attachment #1: Type: text/plain, Size: 1673 bytes --]

Ups, wrong version

This is the one, sorry.

On Sun, Apr 5, 2020 at 1:53 PM Ricardo Ribalda Delgado
<ricardo.ribalda@gmail.com> wrote:
>
> Hi Richard
>
>
> On Sun, Apr 5, 2020 at 1:04 PM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> >
> > On Sun, 2020-04-05 at 11:56 +0200, Ricardo Ribalda Delgado wrote:
> > > Hi Richard
> > >
> > > I have tried with master  (0c91fcba446418ad1f71d3df9aa3b186bbd353c7)
> > > +
> > > my two patches.
> > > Then I have used crops/poky:fedora-30
> > >
> > > and: oe-selftest  --run-tests wic.Wic.test_exclude_path oe-selftest
> > > --run-tests wic.Wic.test_include_path wic.Wic.test_exclude_path
> > >
> > > With an OK result :(
> >
> > I worried this may be the case. There definitely is something wrong and
> > I could replicate it a third time too.
> >
> > My guess is that there are perhaps files in /tmp/ which are logged in
> > the pseudo database but don't exist when the second rootfs is created.
> > Since the files no longer exist, the database integrity check fails.
>
> That is definitely a good hypothesis. Since you can replicate the bug,
> can you try to add also the attached patch to your tree and re-run the
> test.
>
> What it does, is to remove a file from the database if it has
> vanished, during the bitbake move.
>
> If it works I can add it to the beginning of my series, or even trying
> to upstream it to pseudo.
>
> Best regards
>
> >
> > Why that only occurs on Fedora case on the autobuilder, I don't know,
> > I'm just hypothesising right now...
> >
> > Cheers,
> >
> > Richard
> >
>
>
> --
> Ricardo Ribalda



-- 
Ricardo Ribalda

[-- Attachment #2: 0001-pseudo-During-DB-fix-remove-files-that-do-not-exist.patch --]
[-- Type: text/x-patch, Size: 3340 bytes --]

From bbddec527d9056e1feda0d21edd8ddb9d64623b2 Mon Sep 17 00:00:00 2001
From: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Date: Sun, 5 Apr 2020 13:47:12 +0200
Subject: [PATCH] pseudo: During DB fix, remove files that do not exist

If a file does not exist, either because it has been removed outside
bitbake, or because only some of the files have been moved to a
different location, delete it from the pseudo-db is the user decides to
fix the database.

Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
 ...ixup-remove-files-that-do-not-exist-.patch | 48 +++++++++++++++++++
 meta/recipes-devtools/pseudo/pseudo_git.bb    |  1 +
 2 files changed, 49 insertions(+)
 create mode 100644 meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch

diff --git a/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch b/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch
new file mode 100644
index 0000000000..8aec1c9551
--- /dev/null
+++ b/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch
@@ -0,0 +1,48 @@
+From b0902e36108b49e6bc88d6b251cc2f8cffcd5a13 Mon Sep 17 00:00:00 2001
+From: Ricardo Ribalda <ricardo@ribalda.com>
+Date: Sun, 5 Apr 2020 11:40:30 +0000
+Subject: [PATCH] pseudo: On a DB fixup remove files that do not exist anymore
+
+If the user decides to fix a database, remove the files that do not
+exist anymore.
+If only DB test is selected do not change the behaviour (return error).
+
+Signed-off-by: Ricardo Ribalda <ricardo@ribalda.com>
+---
+ pseudo.c | 13 ++++++++++---
+ 1 file changed, 10 insertions(+), 3 deletions(-)
+
+diff --git a/pseudo.c b/pseudo.c
+index 0f5850e..98e5b0c 100644
+--- a/pseudo.c
++++ b/pseudo.c
+@@ -1087,9 +1087,15 @@ pseudo_db_check(int fix) {
+ 			int fixup_needed = 0;
+ 			pseudo_debug(PDBGF_DB, "Checking <%s>\n", m->path);
+ 			if (lstat(m->path, &buf)) {
+-				errors = EXIT_FAILURE;
+-				pseudo_diag("can't stat <%s>\n", m->path);
+-				continue;
++				if (!fix) {
++					pseudo_diag("can't stat <%s>\n", m->path);
++					errors = EXIT_FAILURE;
++					continue;
++				} else {
++					pseudo_debug("can't stat <%s>\n", m->path);
++					fixup_needed = 2;
++					goto do_fixup;
++				}
+ 			}
+ 			/* can't check for device type mismatches, uid/gid, or
+ 			 * permissions, because those are the very things we
+@@ -1125,6 +1131,7 @@ pseudo_db_check(int fix) {
+ 					S_ISDIR(m->mode));
+ 				fixup_needed = 2;
+ 			}
++			do_fixup:
+ 			if (fixup_needed) {
+ 				/* in fixup mode, either delete (mismatches) or
+ 				 * correct (dev/ino).
+-- 
+2.21.1
+
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index 89e43c5996..a3049c5e6c 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -11,6 +11,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo \
            file://0001-realpath.c-Remove-trailing-slashes.patch \
            file://0006-xattr-adjust-for-attr-2.4.48-release.patch \
            file://seccomp.patch \
+           file://0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch \
            "
 
 SRCREV = "060058bb29f70b244e685b3c704eb0641b736f73"
-- 
2.25.1

Re: [OE-core] [PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Richard Purdie]

On Sun, 2020-04-05 at 14:01 +0200, Ricardo Ribalda Delgado wrote:
> Ups, wrong version
> 
> This is the one, sorry.

It didn't build unfortunately:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/837

Cheers,

Richard

Re: [OE-core] [PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

[-- Attachment #1: Type: text/plain, Size: 832 bytes --]

Hi Richard

Thanks for testing it out. Seems that I am either not very clever or I
should not work on Sundays (or both), v1, and v2 where exactly the
same. I am terribly sorry to spam you.

I know it is abusing your patience, but can you try again. I would
love to try it on my setup , but I cannot replicate the bug.

As we speak I am running oe-selftest    --run-tests wic.Wic on poky/fedora-30...

Thanks, and sorry again.

On Sun, Apr 5, 2020 at 5:29 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Sun, 2020-04-05 at 14:01 +0200, Ricardo Ribalda Delgado wrote:
> > Ups, wrong version
> >
> > This is the one, sorry.
>
> It didn't build unfortunately:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/837
>
> Cheers,
>
> Richard
>


-- 
Ricardo Ribalda

[-- Attachment #2: v2-0001-pseudo-During-DB-fix-remove-files-that-do-not-exi.patch --]
[-- Type: text/x-patch, Size: 3353 bytes --]

From 9c2bd9553e81fd16a9e3aad2f20b7a86918de4be Mon Sep 17 00:00:00 2001
From: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Date: Sun, 5 Apr 2020 13:47:12 +0200
Subject: [PATCH v2] pseudo: During DB fix, remove files that do not exist

If a file does not exist, either because it has been removed outside
bitbake, or because only some of the files have been moved to a
different location, delete it from the pseudo-db is the user decides to
fix the database.

Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
 ...ixup-remove-files-that-do-not-exist-.patch | 48 +++++++++++++++++++
 meta/recipes-devtools/pseudo/pseudo_git.bb    |  1 +
 2 files changed, 49 insertions(+)
 create mode 100644 meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch

diff --git a/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch b/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch
new file mode 100644
index 0000000000..26efb0a113
--- /dev/null
+++ b/meta/recipes-devtools/pseudo/files/0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch
@@ -0,0 +1,48 @@
+From b0902e36108b49e6bc88d6b251cc2f8cffcd5a13 Mon Sep 17 00:00:00 2001
+From: Ricardo Ribalda <ricardo@ribalda.com>
+Date: Sun, 5 Apr 2020 11:40:30 +0000
+Subject: [PATCH] pseudo: On a DB fixup remove files that do not exist anymore
+
+If the user decides to fix a database, remove the files that do not
+exist anymore.
+If only DB test is selected do not change the behaviour (return error).
+
+Signed-off-by: Ricardo Ribalda <ricardo@ribalda.com>
+---
+ pseudo.c | 13 ++++++++++---
+ 1 file changed, 10 insertions(+), 3 deletions(-)
+
+diff --git a/pseudo.c b/pseudo.c
+index 0f5850e..98e5b0c 100644
+--- a/pseudo.c
++++ b/pseudo.c
+@@ -1087,9 +1087,15 @@ pseudo_db_check(int fix) {
+ 			int fixup_needed = 0;
+ 			pseudo_debug(PDBGF_DB, "Checking <%s>\n", m->path);
+ 			if (lstat(m->path, &buf)) {
+-				errors = EXIT_FAILURE;
+-				pseudo_diag("can't stat <%s>\n", m->path);
+-				continue;
++				if (!fix) {
++					pseudo_diag("can't stat <%s>\n", m->path);
++					errors = EXIT_FAILURE;
++					continue;
++				} else {
++					pseudo_debug(PDBGF_DB, "can't stat <%s>\n", m->path);
++					fixup_needed = 2;
++					goto do_fixup;
++				}
+ 			}
+ 			/* can't check for device type mismatches, uid/gid, or
+ 			 * permissions, because those are the very things we
+@@ -1125,6 +1131,7 @@ pseudo_db_check(int fix) {
+ 					S_ISDIR(m->mode));
+ 				fixup_needed = 2;
+ 			}
++			do_fixup:
+ 			if (fixup_needed) {
+ 				/* in fixup mode, either delete (mismatches) or
+ 				 * correct (dev/ino).
+-- 
+2.21.1
+
diff --git a/meta/recipes-devtools/pseudo/pseudo_git.bb b/meta/recipes-devtools/pseudo/pseudo_git.bb
index 89e43c5996..a3049c5e6c 100644
--- a/meta/recipes-devtools/pseudo/pseudo_git.bb
+++ b/meta/recipes-devtools/pseudo/pseudo_git.bb
@@ -11,6 +11,7 @@ SRC_URI = "git://git.yoctoproject.org/pseudo \
            file://0001-realpath.c-Remove-trailing-slashes.patch \
            file://0006-xattr-adjust-for-attr-2.4.48-release.patch \
            file://seccomp.patch \
+           file://0001-pseudo-On-a-DB-fixup-remove-files-that-do-not-exist-.patch \
            "
 
 SRCREV = "060058bb29f70b244e685b3c704eb0641b736f73"
-- 
2.25.1

Re: [OE-core] [PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Richard Purdie]

On Sun, 2020-04-05 at 17:44 +0200, Ricardo Ribalda Delgado wrote:
> Hi Richard
> 
> Thanks for testing it out. Seems that I am either not very clever or
> I
> should not work on Sundays (or both), v1, and v2 where exactly the
> same. I am terribly sorry to spam you.
> 
> I know it is abusing your patience, but can you try again. I would
> love to try it on my setup , but I cannot replicate the bug.
> 
> As we speak I am running oe-selftest    --run-tests wic.Wic on
> poky/fedora-30...

I did re-run it, it took a while since a pseudo change invalidates
sstate. It passed:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/838

Cheers,

Richard

Re: [OE-core] [PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

Hi

On Mon, Apr 6, 2020 at 11:00 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Sun, 2020-04-05 at 17:44 +0200, Ricardo Ribalda Delgado wrote:
> > Hi Richard
> >
> > Thanks for testing it out. Seems that I am either not very clever or
> > I
> > should not work on Sundays (or both), v1, and v2 where exactly the
> > same. I am terribly sorry to spam you.
> >
> > I know it is abusing your patience, but can you try again. I would
> > love to try it on my setup , but I cannot replicate the bug.
> >
> > As we speak I am running oe-selftest    --run-tests wic.Wic on
> > poky/fedora-30...
>
> I did re-run it, it took a while since a pseudo change invalidates
> sstate. It passed:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/838
>

Thanks a lot for testing it. I will prepare a new series including that patch.

> Cheers,
>
> Richard
>


-- 
Ricardo Ribalda

Re: [OE-core] [PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

Hi

On Mon, Apr 6, 2020 at 11:04 AM Ricardo Ribalda via
lists.openembedded.org
<ricardo.ribalda=gmail.com@lists.openembedded.org> wrote:
>
> Hi
>
> On Mon, Apr 6, 2020 at 11:00 AM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> >
> > On Sun, 2020-04-05 at 17:44 +0200, Ricardo Ribalda Delgado wrote:
> > > Hi Richard
> > >
> > > Thanks for testing it out. Seems that I am either not very clever or
> > > I
> > > should not work on Sundays (or both), v1, and v2 where exactly the
> > > same. I am terribly sorry to spam you.
> > >
> > > I know it is abusing your patience, but can you try again. I would
> > > love to try it on my setup , but I cannot replicate the bug.
> > >
> > > As we speak I am running oe-selftest    --run-tests wic.Wic on
> > > poky/fedora-30...
> >
> > I did re-run it, it took a while since a pseudo change invalidates
> > sstate. It passed:
> >
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/838
> >
>
> Thanks a lot for testing it. I will prepare a new series including that patch.

I saw that you already queue it for master-next:

https://git.openembedded.org/openembedded-core/log/?h=master-next

So I will not resend the series ;)



I have send the Individual patch to pseudo, lets see if it is merged.

Thanks!


>
> > Cheers,
> >
> > Richard
> >
>
>
> --
> Ricardo Ribalda
> 



--
Ricardo Ribalda

Re: [OE-core] [PATCH v2 1/2] wic: Fix permissions when using exclude or include path

[Richard Purdie]

On Mon, 2020-04-06 at 11:21 +0200, Ricardo Ribalda Delgado wrote:
> Hi
> 
> On Mon, Apr 6, 2020 at 11:04 AM Ricardo Ribalda via
> lists.openembedded.org
> <ricardo.ribalda=gmail.com@lists.openembedded.org> wrote:
> > Hi
> > 
> > On Mon, Apr 6, 2020 at 11:00 AM Richard Purdie
> > <richard.purdie@linuxfoundation.org> wrote:
> > > On Sun, 2020-04-05 at 17:44 +0200, Ricardo Ribalda Delgado wrote:
> > > > Hi Richard
> > > > 
> > > > Thanks for testing it out. Seems that I am either not very
> > > > clever or
> > > > I
> > > > should not work on Sundays (or both), v1, and v2 where exactly
> > > > the
> > > > same. I am terribly sorry to spam you.
> > > > 
> > > > I know it is abusing your patience, but can you try again. I
> > > > would
> > > > love to try it on my setup , but I cannot replicate the bug.
> > > > 
> > > > As we speak I am running oe-selftest    --run-tests wic.Wic on
> > > > poky/fedora-30...
> > > 
> > > I did re-run it, it took a while since a pseudo change
> > > invalidates
> > > sstate. It passed:
> > > 
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/838
> > > 
> > 
> > Thanks a lot for testing it. I will prepare a new series including
> > that patch.
> 
> I saw that you already queue it for master-next:
> 
> https://git.openembedded.org/openembedded-core/log/?h=master-next
> 
> So I will not resend the series ;)
> 
> 
> 
> I have send the Individual patch to pseudo, lets see if it is merged.
> 
> Thanks!

The pseudo patch is missing an Upstream-Status but I've added one to
the version in master-next.

We're struggling with an active maintainer for pseudo right now but
I'll make sure itsin the queue. I've mentioned it to Peter just in case
he can spot anything.

Cheers,

Richard

Re: [PATCH v2 2/2] wic: Add --embed-rootfs argument

[Paul Barker]

On Fri, 3 Apr 2020 21:52:52 +0200
Ricardo Ribalda Delgado <ricardo@ribalda.com> wrote:

> ping?

I'm still concerned that `--embed-rootfs` and `--include-path` are too
similar and we should instead have one option that can handle both use cases.

Has this already been merged?

> 
> On Thu, Mar 5, 2020 at 1:26 PM Ricardo Ribalda Delgado
> <ricardo@ribalda.com> wrote:
> >
> > Hi Paul,
> >
> > On Thu, Mar 5, 2020 at 10:37 AM Paul Barker <pbarker@konsulko.com> wrote:  
> > >
> > > On Wed,  4 Mar 2020 15:49:36 +0100
> > > Ricardo Ribalda Delgado <ricardo@ribalda.com> wrote:
> > >  
> > > > This option adds the content of a rootfs on a specific location on the
> > > > rootfs.
> > > >
> > > > It is very useful for making a partition that contains the rootfs for a
> > > > host and a target Eg:
> > > >
> > > > / -> Roofs for the host
> > > > /export/ -> Rootfs for the target (which will netboot)
> > > >
> > > > Although today we support making a partition for "/export" this might
> > > > not be compatible with some upgrade systems, or we might be limited by
> > > > the number of partitions.
> > > >
> > > > With this patch we can use something like:
> > > >
> > > > part / --source rootfs --embed-rootfs target-image /export --embed-rootfs target-image2 /export2  
> > >
> > > I like this but it still leaves confusion between `--include-path` and
> > > --embed-rootfs` as they're similar but slightly different. Can we just modify
> > > `--include-path` to have this syntax?  
> >
> > I think they are different enough.
> >
> > - include-path ads a file/folder
> > - embed-rootfs adds a rootfs, which is either a folder or a image.bb file.
> >
> >  
> > >  
> > > >
> > > > on the .wks file.
> > > >
> > > > Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
> > > > ---
> > > >  scripts/lib/wic/help.py                  |  8 ++++++++
> > > >  scripts/lib/wic/ksparser.py              |  1 +
> > > >  scripts/lib/wic/partition.py             |  1 +
> > > >  scripts/lib/wic/plugins/source/rootfs.py | 22 +++++++++++++++++++++-
> > > >  4 files changed, 31 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
> > > > index 4d342fcf05..140dc504cd 100644
> > > > --- a/scripts/lib/wic/help.py
> > > > +++ b/scripts/lib/wic/help.py
> > > > @@ -979,6 +979,14 @@ DESCRIPTION
> > > >                           copies. This option only has an effect with the rootfs
> > > >                           source plugin.
> > > >
> > > > +         --embed-rootfs: This option is specific to wic. It embeds a rootfs into
> > > > +                         the given path to the resulting image. The option
> > > > +                         contains two fields, the roofs and the path, separated
> > > > +                         by a space. The rootfs follows the same logic as the
> > > > +                         rootfs-dir argument. Multiple options can be provided
> > > > +                         in order to embed multiple rootfs. This option only has
> > > > +                         an effect with the rootfs source plugin.
> > > > +
> > > >           --extra-space: This option is specific to wic. It adds extra
> > > >                          space after the space filled by the content
> > > >                          of the partition. The final size can go
> > > > diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> > > > index 650b976223..64c8c1175e 100644
> > > > --- a/scripts/lib/wic/ksparser.py
> > > > +++ b/scripts/lib/wic/ksparser.py
> > > > @@ -138,6 +138,7 @@ class KickStart():
> > > >          part.add_argument('--align', type=int)
> > > >          part.add_argument('--exclude-path', nargs='+')
> > > >          part.add_argument('--include-path', nargs='+')
> > > > +        part.add_argument('--embed-rootfs', nargs=2, action='append')
> > > >          part.add_argument("--extra-space", type=sizetype)
> > > >          part.add_argument('--fsoptions', dest='fsopts')
> > > >          part.add_argument('--fstype', default='vfat',
> > > > diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> > > > index 2d95f78439..13857df82f 100644
> > > > --- a/scripts/lib/wic/partition.py
> > > > +++ b/scripts/lib/wic/partition.py
> > > > diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
> > > > index 40419a64b3..089aaea477 100644
> > > > --- a/scripts/lib/wic/plugins/source/rootfs.py
> > > > +++ b/scripts/lib/wic/plugins/source/rootfs.py
> > > > @@ -17,6 +17,7 @@ import shutil
> > > >  import sys
> > > >
> > > >  from oe.path import copyhardlinktree, copytree
> > > > +from pathlib import Path
> > > >
> > > >  from wic import WicError
> > > >  from wic.pluginbase import SourcePlugin
> > > > @@ -80,7 +81,7 @@ class RootfsPlugin(SourcePlugin):
> > > >
> > > >          new_rootfs = None
> > > >          # Handle excluded paths.
> > > > -        if part.exclude_path or part.include_path:
> > > > +        if part.exclude_path or part.include_path or part.embed_rootfs:
> > > >              # We need a new rootfs directory we can delete files from. Copy to
> > > >              # workdir.
> > > >              new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno))
> > > > @@ -100,6 +101,25 @@ class RootfsPlugin(SourcePlugin):
> > > >              for path in part.include_path or []:
> > > >                  copyhardlinktree(path, new_rootfs)
> > > >
> > > > +            for embed in part.embed_rootfs or []:
> > > > +                [embed_rootfs, path] = embed
> > > > +                #we need to remove the initial / for os.path.join to work
> > > > +                if os.path.isabs(path):
> > > > +                    path = path[1:]
> > > > +                if embed_rootfs in krootfs_dir:
> > > > +                    embed_rootfs = krootfs_dir[embed_rootfs]
> > > > +                embed_rootfs = cls.__get_rootfs_dir(embed_rootfs)
> > > > +                tar_file = os.path.realpath(os.path.join(cr_workdir, "aux.tar"))
> > > > +                tar_cmd = "%s tar cpf %s -C %s ." % (cls.__get_pseudo(native_sysroot,
> > > > +                                                     embed_rootfs), tar_file, embed_rootfs)
> > > > +                exec_native_cmd(tar_cmd, native_sysroot)
> > > > +                untar_cmd = "%s tar xf %s -C %s ." % (cls.__get_pseudo(native_sysroot, new_rootfs),
> > > > +                                                      tar_file, os.path.join(new_rootfs, path))
> > > > +                Path(os.path.join(new_rootfs, path)).mkdir(parents=True, exist_ok=True)
> > > > +                exec_native_cmd(untar_cmd, native_sysroot,
> > > > +                                cls.__get_pseudo(native_sysroot, new_rootfs))
> > > > +                os.remove(tar_file)
> > > > +
> > > >              for orig_path in part.exclude_path or []:
> > > >                  path = orig_path
> > > >                  if os.path.isabs(path):  
> > >
> > > As said in my other email, if you're running wic outside bitbake I'm not sure
> > > you can guarantee pseudo is available. And now I think about it, if you're
> > > running wic inside bitbake (as part of do_image_wic), you'd be running pseudo
> > > under pseudo and I have no idea how that would work out.
> > >  
> >
> > pseudo is warranted because wic launches "bitbake wic-tools", which
> > has as DEPENDS pseudo-native.
> >
> > scripts/wic:        subprocess.check_call(["bitbake", "wic-tools"])
> >
> >
> > I have tried to run it inside bitbake with --exclude-path and
> > everything seems to work fine.
> >
> >
> >  
> > > --
> > > Paul Barker
> > > Konsulko Group  
> >
> >
> >
> > --
> > Ricardo Ribalda  
> 
> 
> 



-- 
Paul Barker
Konsulko Group

Re: [PATCH v2 2/2] wic: Add --embed-rootfs argument

[Ricardo Ribalda]

Hi Paul

Yes it got merged
https://git.openembedded.org/openembedded-core/commit/?id=efdcf94801f6abe8e4099e324d9a3deccd8d4384

Even though the functionality looks similar, the devil is on the
details ;). One thing is adding a file, this is  just  file/directory
that is referenced as a path, and the other is a complete rootfs,
which takes care of permissions and can be invoked with a path, or a
bitbake target...

But if you come up with a interface that cover both usecases I can
help implementing it.

Best regards.

On Tue, Apr 7, 2020 at 7:47 PM Paul Barker <pbarker@konsulko.com> wrote:
>
> On Fri, 3 Apr 2020 21:52:52 +0200
> Ricardo Ribalda Delgado <ricardo@ribalda.com> wrote:
>
> > ping?
>
> I'm still concerned that `--embed-rootfs` and `--include-path` are too
> similar and we should instead have one option that can handle both use cases.
>
> Has this already been merged?
>
> >
> > On Thu, Mar 5, 2020 at 1:26 PM Ricardo Ribalda Delgado
> > <ricardo@ribalda.com> wrote:
> > >
> > > Hi Paul,
> > >
> > > On Thu, Mar 5, 2020 at 10:37 AM Paul Barker <pbarker@konsulko.com> wrote:
> > > >
> > > > On Wed,  4 Mar 2020 15:49:36 +0100
> > > > Ricardo Ribalda Delgado <ricardo@ribalda.com> wrote:
> > > >
> > > > > This option adds the content of a rootfs on a specific location on the
> > > > > rootfs.
> > > > >
> > > > > It is very useful for making a partition that contains the rootfs for a
> > > > > host and a target Eg:
> > > > >
> > > > > / -> Roofs for the host
> > > > > /export/ -> Rootfs for the target (which will netboot)
> > > > >
> > > > > Although today we support making a partition for "/export" this might
> > > > > not be compatible with some upgrade systems, or we might be limited by
> > > > > the number of partitions.
> > > > >
> > > > > With this patch we can use something like:
> > > > >
> > > > > part / --source rootfs --embed-rootfs target-image /export --embed-rootfs target-image2 /export2
> > > >
> > > > I like this but it still leaves confusion between `--include-path` and
> > > > --embed-rootfs` as they're similar but slightly different. Can we just modify
> > > > `--include-path` to have this syntax?
> > >
> > > I think they are different enough.
> > >
> > > - include-path ads a file/folder
> > > - embed-rootfs adds a rootfs, which is either a folder or a image.bb file.
> > >
> > >
> > > >
> > > > >
> > > > > on the .wks file.
> > > > >
> > > > > Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
> > > > > ---
> > > > >  scripts/lib/wic/help.py                  |  8 ++++++++
> > > > >  scripts/lib/wic/ksparser.py              |  1 +
> > > > >  scripts/lib/wic/partition.py             |  1 +
> > > > >  scripts/lib/wic/plugins/source/rootfs.py | 22 +++++++++++++++++++++-
> > > > >  4 files changed, 31 insertions(+), 1 deletion(-)
> > > > >
> > > > > diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
> > > > > index 4d342fcf05..140dc504cd 100644
> > > > > --- a/scripts/lib/wic/help.py
> > > > > +++ b/scripts/lib/wic/help.py
> > > > > @@ -979,6 +979,14 @@ DESCRIPTION
> > > > >                           copies. This option only has an effect with the rootfs
> > > > >                           source plugin.
> > > > >
> > > > > +         --embed-rootfs: This option is specific to wic. It embeds a rootfs into
> > > > > +                         the given path to the resulting image. The option
> > > > > +                         contains two fields, the roofs and the path, separated
> > > > > +                         by a space. The rootfs follows the same logic as the
> > > > > +                         rootfs-dir argument. Multiple options can be provided
> > > > > +                         in order to embed multiple rootfs. This option only has
> > > > > +                         an effect with the rootfs source plugin.
> > > > > +
> > > > >           --extra-space: This option is specific to wic. It adds extra
> > > > >                          space after the space filled by the content
> > > > >                          of the partition. The final size can go
> > > > > diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> > > > > index 650b976223..64c8c1175e 100644
> > > > > --- a/scripts/lib/wic/ksparser.py
> > > > > +++ b/scripts/lib/wic/ksparser.py
> > > > > @@ -138,6 +138,7 @@ class KickStart():
> > > > >          part.add_argument('--align', type=int)
> > > > >          part.add_argument('--exclude-path', nargs='+')
> > > > >          part.add_argument('--include-path', nargs='+')
> > > > > +        part.add_argument('--embed-rootfs', nargs=2, action='append')
> > > > >          part.add_argument("--extra-space", type=sizetype)
> > > > >          part.add_argument('--fsoptions', dest='fsopts')
> > > > >          part.add_argument('--fstype', default='vfat',
> > > > > diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> > > > > index 2d95f78439..13857df82f 100644
> > > > > --- a/scripts/lib/wic/partition.py
> > > > > +++ b/scripts/lib/wic/partition.py
> > > > > diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
> > > > > index 40419a64b3..089aaea477 100644
> > > > > --- a/scripts/lib/wic/plugins/source/rootfs.py
> > > > > +++ b/scripts/lib/wic/plugins/source/rootfs.py
> > > > > @@ -17,6 +17,7 @@ import shutil
> > > > >  import sys
> > > > >
> > > > >  from oe.path import copyhardlinktree, copytree
> > > > > +from pathlib import Path
> > > > >
> > > > >  from wic import WicError
> > > > >  from wic.pluginbase import SourcePlugin
> > > > > @@ -80,7 +81,7 @@ class RootfsPlugin(SourcePlugin):
> > > > >
> > > > >          new_rootfs = None
> > > > >          # Handle excluded paths.
> > > > > -        if part.exclude_path or part.include_path:
> > > > > +        if part.exclude_path or part.include_path or part.embed_rootfs:
> > > > >              # We need a new rootfs directory we can delete files from. Copy to
> > > > >              # workdir.
> > > > >              new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno))
> > > > > @@ -100,6 +101,25 @@ class RootfsPlugin(SourcePlugin):
> > > > >              for path in part.include_path or []:
> > > > >                  copyhardlinktree(path, new_rootfs)
> > > > >
> > > > > +            for embed in part.embed_rootfs or []:
> > > > > +                [embed_rootfs, path] = embed
> > > > > +                #we need to remove the initial / for os.path.join to work
> > > > > +                if os.path.isabs(path):
> > > > > +                    path = path[1:]
> > > > > +                if embed_rootfs in krootfs_dir:
> > > > > +                    embed_rootfs = krootfs_dir[embed_rootfs]
> > > > > +                embed_rootfs = cls.__get_rootfs_dir(embed_rootfs)
> > > > > +                tar_file = os.path.realpath(os.path.join(cr_workdir, "aux.tar"))
> > > > > +                tar_cmd = "%s tar cpf %s -C %s ." % (cls.__get_pseudo(native_sysroot,
> > > > > +                                                     embed_rootfs), tar_file, embed_rootfs)
> > > > > +                exec_native_cmd(tar_cmd, native_sysroot)
> > > > > +                untar_cmd = "%s tar xf %s -C %s ." % (cls.__get_pseudo(native_sysroot, new_rootfs),
> > > > > +                                                      tar_file, os.path.join(new_rootfs, path))
> > > > > +                Path(os.path.join(new_rootfs, path)).mkdir(parents=True, exist_ok=True)
> > > > > +                exec_native_cmd(untar_cmd, native_sysroot,
> > > > > +                                cls.__get_pseudo(native_sysroot, new_rootfs))
> > > > > +                os.remove(tar_file)
> > > > > +
> > > > >              for orig_path in part.exclude_path or []:
> > > > >                  path = orig_path
> > > > >                  if os.path.isabs(path):
> > > >
> > > > As said in my other email, if you're running wic outside bitbake I'm not sure
> > > > you can guarantee pseudo is available. And now I think about it, if you're
> > > > running wic inside bitbake (as part of do_image_wic), you'd be running pseudo
> > > > under pseudo and I have no idea how that would work out.
> > > >
> > >
> > > pseudo is warranted because wic launches "bitbake wic-tools", which
> > > has as DEPENDS pseudo-native.
> > >
> > > scripts/wic:        subprocess.check_call(["bitbake", "wic-tools"])
> > >
> > >
> > > I have tried to run it inside bitbake with --exclude-path and
> > > everything seems to work fine.
> > >
> > >
> > >
> > > > --
> > > > Paul Barker
> > > > Konsulko Group
> > >
> > >
> > >
> > > --
> > > Ricardo Ribalda
> >
> >
> >
>
>
>
> --
> Paul Barker
> Konsulko Group



-- 
Ricardo Ribalda

Re: [OE-core] [PATCH v2 2/2] wic: Add --embed-rootfs argument

[Ricardo Ribalda]

Hi Again



On Tue, Apr 7, 2020 at 8:19 PM Ricardo Ribalda via
lists.openembedded.org
<ricardo.ribalda=gmail.com@lists.openembedded.org> wrote:
>
> Hi Paul
>
> Yes it got merged
> https://git.openembedded.org/openembedded-core/commit/?id=efdcf94801f6abe8e4099e324d9a3deccd8d4384
>
> Even though the functionality looks similar, the devil is on the
> details ;). One thing is adding a file, this is  just  file/directory
> that is referenced as a path, and the other is a complete rootfs,
> which takes care of permissions and can be invoked with a path, or a
> bitbake target...
>
> But if you come up with a interface that cover both usecases I can
> help implementing it.

Looking into merging the functionality of embed-rootfs into include-path

I guess the first step is to modify include-path so it can accept 1 or
2 arguments.

If 1 argument:
the functionality is the same as now

If 2 arguments:
The second argument is considered the destination:
eg: --include-path home  /home/


The second step is to accept not only file/folders but also images.
The challenge here is that we can name an image in multiple ways:
- path: tmp/work/machine/core-image-minimal/1.0-r0/rootfs/
- name: core-image-minimal
- keyword that it is provided by wic: -rootfs-dir target-image=/workir....

Which can lead to a name collision. What do we do in this case? We
prefer the directory among the image?


The third step is, as you mentioned on the irc,  to fix also the
permissions of the files/directories, not only for images.
- Maybe leave files owned by root if no pseudo database exists and
clear setuid/setgid bits for safety.

Am I in the right direction ? something I am missing?



>
> Best regards.
>
> On Tue, Apr 7, 2020 at 7:47 PM Paul Barker <pbarker@konsulko.com> wrote:
> >
> > On Fri, 3 Apr 2020 21:52:52 +0200
> > Ricardo Ribalda Delgado <ricardo@ribalda.com> wrote:
> >
> > > ping?
> >
> > I'm still concerned that `--embed-rootfs` and `--include-path` are too
> > similar and we should instead have one option that can handle both use cases.
> >
> > Has this already been merged?
> >
> > >
> > > On Thu, Mar 5, 2020 at 1:26 PM Ricardo Ribalda Delgado
> > > <ricardo@ribalda.com> wrote:
> > > >
> > > > Hi Paul,
> > > >
> > > > On Thu, Mar 5, 2020 at 10:37 AM Paul Barker <pbarker@konsulko.com> wrote:
> > > > >
> > > > > On Wed,  4 Mar 2020 15:49:36 +0100
> > > > > Ricardo Ribalda Delgado <ricardo@ribalda.com> wrote:
> > > > >
> > > > > > This option adds the content of a rootfs on a specific location on the
> > > > > > rootfs.
> > > > > >
> > > > > > It is very useful for making a partition that contains the rootfs for a
> > > > > > host and a target Eg:
> > > > > >
> > > > > > / -> Roofs for the host
> > > > > > /export/ -> Rootfs for the target (which will netboot)
> > > > > >
> > > > > > Although today we support making a partition for "/export" this might
> > > > > > not be compatible with some upgrade systems, or we might be limited by
> > > > > > the number of partitions.
> > > > > >
> > > > > > With this patch we can use something like:
> > > > > >
> > > > > > part / --source rootfs --embed-rootfs target-image /export --embed-rootfs target-image2 /export2
> > > > >
> > > > > I like this but it still leaves confusion between `--include-path` and
> > > > > --embed-rootfs` as they're similar but slightly different. Can we just modify
> > > > > `--include-path` to have this syntax?
> > > >
> > > > I think they are different enough.
> > > >
> > > > - include-path ads a file/folder
> > > > - embed-rootfs adds a rootfs, which is either a folder or a image.bb file.
> > > >
> > > >
> > > > >
> > > > > >
> > > > > > on the .wks file.
> > > > > >
> > > > > > Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
> > > > > > ---
> > > > > >  scripts/lib/wic/help.py                  |  8 ++++++++
> > > > > >  scripts/lib/wic/ksparser.py              |  1 +
> > > > > >  scripts/lib/wic/partition.py             |  1 +
> > > > > >  scripts/lib/wic/plugins/source/rootfs.py | 22 +++++++++++++++++++++-
> > > > > >  4 files changed, 31 insertions(+), 1 deletion(-)
> > > > > >
> > > > > > diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
> > > > > > index 4d342fcf05..140dc504cd 100644
> > > > > > --- a/scripts/lib/wic/help.py
> > > > > > +++ b/scripts/lib/wic/help.py
> > > > > > @@ -979,6 +979,14 @@ DESCRIPTION
> > > > > >                           copies. This option only has an effect with the rootfs
> > > > > >                           source plugin.
> > > > > >
> > > > > > +         --embed-rootfs: This option is specific to wic. It embeds a rootfs into
> > > > > > +                         the given path to the resulting image. The option
> > > > > > +                         contains two fields, the roofs and the path, separated
> > > > > > +                         by a space. The rootfs follows the same logic as the
> > > > > > +                         rootfs-dir argument. Multiple options can be provided
> > > > > > +                         in order to embed multiple rootfs. This option only has
> > > > > > +                         an effect with the rootfs source plugin.
> > > > > > +
> > > > > >           --extra-space: This option is specific to wic. It adds extra
> > > > > >                          space after the space filled by the content
> > > > > >                          of the partition. The final size can go
> > > > > > diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> > > > > > index 650b976223..64c8c1175e 100644
> > > > > > --- a/scripts/lib/wic/ksparser.py
> > > > > > +++ b/scripts/lib/wic/ksparser.py
> > > > > > @@ -138,6 +138,7 @@ class KickStart():
> > > > > >          part.add_argument('--align', type=int)
> > > > > >          part.add_argument('--exclude-path', nargs='+')
> > > > > >          part.add_argument('--include-path', nargs='+')
> > > > > > +        part.add_argument('--embed-rootfs', nargs=2, action='append')
> > > > > >          part.add_argument("--extra-space", type=sizetype)
> > > > > >          part.add_argument('--fsoptions', dest='fsopts')
> > > > > >          part.add_argument('--fstype', default='vfat',
> > > > > > diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> > > > > > index 2d95f78439..13857df82f 100644
> > > > > > --- a/scripts/lib/wic/partition.py
> > > > > > +++ b/scripts/lib/wic/partition.py
> > > > > > diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
> > > > > > index 40419a64b3..089aaea477 100644
> > > > > > --- a/scripts/lib/wic/plugins/source/rootfs.py
> > > > > > +++ b/scripts/lib/wic/plugins/source/rootfs.py
> > > > > > @@ -17,6 +17,7 @@ import shutil
> > > > > >  import sys
> > > > > >
> > > > > >  from oe.path import copyhardlinktree, copytree
> > > > > > +from pathlib import Path
> > > > > >
> > > > > >  from wic import WicError
> > > > > >  from wic.pluginbase import SourcePlugin
> > > > > > @@ -80,7 +81,7 @@ class RootfsPlugin(SourcePlugin):
> > > > > >
> > > > > >          new_rootfs = None
> > > > > >          # Handle excluded paths.
> > > > > > -        if part.exclude_path or part.include_path:
> > > > > > +        if part.exclude_path or part.include_path or part.embed_rootfs:
> > > > > >              # We need a new rootfs directory we can delete files from. Copy to
> > > > > >              # workdir.
> > > > > >              new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno))
> > > > > > @@ -100,6 +101,25 @@ class RootfsPlugin(SourcePlugin):
> > > > > >              for path in part.include_path or []:
> > > > > >                  copyhardlinktree(path, new_rootfs)
> > > > > >
> > > > > > +            for embed in part.embed_rootfs or []:
> > > > > > +                [embed_rootfs, path] = embed
> > > > > > +                #we need to remove the initial / for os.path.join to work
> > > > > > +                if os.path.isabs(path):
> > > > > > +                    path = path[1:]
> > > > > > +                if embed_rootfs in krootfs_dir:
> > > > > > +                    embed_rootfs = krootfs_dir[embed_rootfs]
> > > > > > +                embed_rootfs = cls.__get_rootfs_dir(embed_rootfs)
> > > > > > +                tar_file = os.path.realpath(os.path.join(cr_workdir, "aux.tar"))
> > > > > > +                tar_cmd = "%s tar cpf %s -C %s ." % (cls.__get_pseudo(native_sysroot,
> > > > > > +                                                     embed_rootfs), tar_file, embed_rootfs)
> > > > > > +                exec_native_cmd(tar_cmd, native_sysroot)
> > > > > > +                untar_cmd = "%s tar xf %s -C %s ." % (cls.__get_pseudo(native_sysroot, new_rootfs),
> > > > > > +                                                      tar_file, os.path.join(new_rootfs, path))
> > > > > > +                Path(os.path.join(new_rootfs, path)).mkdir(parents=True, exist_ok=True)
> > > > > > +                exec_native_cmd(untar_cmd, native_sysroot,
> > > > > > +                                cls.__get_pseudo(native_sysroot, new_rootfs))
> > > > > > +                os.remove(tar_file)
> > > > > > +
> > > > > >              for orig_path in part.exclude_path or []:
> > > > > >                  path = orig_path
> > > > > >                  if os.path.isabs(path):
> > > > >
> > > > > As said in my other email, if you're running wic outside bitbake I'm not sure
> > > > > you can guarantee pseudo is available. And now I think about it, if you're
> > > > > running wic inside bitbake (as part of do_image_wic), you'd be running pseudo
> > > > > under pseudo and I have no idea how that would work out.
> > > > >
> > > >
> > > > pseudo is warranted because wic launches "bitbake wic-tools", which
> > > > has as DEPENDS pseudo-native.
> > > >
> > > > scripts/wic:        subprocess.check_call(["bitbake", "wic-tools"])
> > > >
> > > >
> > > > I have tried to run it inside bitbake with --exclude-path and
> > > > everything seems to work fine.
> > > >
> > > >
> > > >
> > > > > --
> > > > > Paul Barker
> > > > > Konsulko Group
> > > >
> > > >
> > > >
> > > > --
> > > > Ricardo Ribalda
> > >
> > >
> > >
> >
> >
> >
> > --
> > Paul Barker
> > Konsulko Group
>
>
>
> --
> Ricardo Ribalda
> 



--
Ricardo Ribalda