* [PATCH v6 00/10] Fix permissions and embed-rotofs
@ 2020-04-14 13:36 Ricardo Ribalda
2020-04-14 13:36 ` [PATCH v6 01/10] wic: Fix permissions when using exclude or include path Ricardo Ribalda
` (9 more replies)
0 siblings, 10 replies; 17+ messages in thread
From: Ricardo Ribalda @ 2020-04-14 13:36 UTC (permalink / raw)
To: openembedded-core, Paul Barker; +Cc: Ricardo Ribalda Delgado
Today wic behaves differently if we run it from bitbake of directly from the
commandline.
When it is run from bitbake, the permissions/usersnames are handled by the
pseudo database of the main image.
When it is run from the comandline ,it is run outside the main image database.
This results on permissions/usernames not working ok on some usecases on both
bitbake and wic.
This is an attempt to fix all the permission bugs that I am aware from wic.
Using the following usecases:
#exclude-path
part / --source rootfs --fstype=ext4 --exclude-path=home
#split_partition
part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/
part /etc --source rootfs --rootfs-dir=tmp/work/qt5222-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/ --fstype=ext4
#multi_partition
part / --source rootfs --ondisk sda --fstype=ext4
part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4
With the current master:
-------------------------
#exclude-path
From Bitbake: OK
wic cmdline: FAIL
#split_partition
From Bitbake: OK
wic cmdline: FAIL, permissions invalid on both partitions
#multi_partition
From Bitbake: FAIL second partition
wic cmdline: OK
After: wic: Fix permissions when using exclude or include path
--------------------------------------------------------------------
#exclude-path
From Bitbake: OK
wic cmdline: OK
#split_partition
From Bitbake: OK
wic cmdline: FAIL, permissions invalid on second partition
#multi_partition
From Bitbake: FAIL second partition
wic cmdline: OK
After: wic: Fix multi images .wks with bitbake
----------------------------------------------
#exclude-path
From Bitbake: OK
wic cmdline: OK
#split_partition
From Bitbake: FAIL, permissions invalid on second partition
wic cmdline: FAIL, permissions invalid on second partition
#multi_partition
From Bitbake: OK
wic cmdline: OK
After: wic: Add --change-directory argument
--------------------------------------------
we can have a .wks like:
part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/
part /etc --source rootfs --fstype=ext4 --change-directory=etc
Wich works fine from bitbake and from cmdline, and has the same functionality as
split_partition.
Changelog v5->v6:
- Allow two arguments to include-path
- Fix permission on include-path
- exec_native with ASSUME_PROVIDED
- More unittesting
Changelog v4->v5:
- Add unittesting
- wic: Avoid creating invalid pseudo directory
Ricardo Ribalda Delgado (10):
wic: Fix permissions when using exclude or include path
wic: Fix multi images .wks with bitbake
wic: Add --change-directory argument
wic: Continue if excluded_path does not exist
wic: Avoid creating invalid pseudo directory
oeqa: wic: Add tests for permissions and change-directory
wic: misc: Do not find for executables in ALREADY_PROVIDED
wic: root: Add an opt. destination on include-path
wic: rootfs: Combine path_validation in one function
oeqa: wic: Add more tests for include_path
meta/classes/image_types_wic.bbclass | 8 +-
meta/lib/oeqa/selftest/cases/wic.py | 155 ++++++++++++++++++++++-
meta/recipes-core/meta/wic-tools.bb | 2 +-
scripts/lib/wic/help.py | 26 ++--
scripts/lib/wic/ksparser.py | 3 +-
scripts/lib/wic/misc.py | 14 +-
scripts/lib/wic/partition.py | 20 +--
scripts/lib/wic/plugins/source/rootfs.py | 126 +++++++++++++++---
8 files changed, 313 insertions(+), 41 deletions(-)
--
2.25.1
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH v6 01/10] wic: Fix permissions when using exclude or include path
2020-04-14 13:36 [PATCH v6 00/10] Fix permissions and embed-rotofs Ricardo Ribalda
@ 2020-04-14 13:36 ` Ricardo Ribalda
2020-04-18 19:03 ` Paul Barker
2020-04-14 13:36 ` [PATCH v6 02/10] wic: Fix multi images .wks with bitbake Ricardo Ribalda
` (8 subsequent siblings)
9 siblings, 1 reply; 17+ messages in thread
From: Ricardo Ribalda @ 2020-04-14 13:36 UTC (permalink / raw)
To: openembedded-core, Paul Barker; +Cc: Ricardo Ribalda Delgado
When parameters include_path or exclude_path are passed to the rootfs
plugin, it will copy the partition content into a folder and make all
the modifications there.
This is done using copyhardlinktree(), which does not take into
consideration the content of the pseudo folder, which contains the
information about the right permissions and ownership of the folders.
This results in a rootfs owned by the user that is running the wic
command (usually UID 1000), which makes some rootfs unbootable.
This bug can be easily triggerd with the following .wks
part / --source rootfs --fstype=ext4 --exclude-path=home
And this sequence:
$ wic create test-permissions -e core-image-minimal -o test/
$ sudo mount test/test-permissions-202004080823-sda.direct.p1 /mnt
$ ls -la /mnt/etc/shadow
To fix this we copy the content of the pseudo folders to the new folder
and modify the pseudo database using the "pseudo -B" command.
If the rootfs is not a rootfs generated by bitbake a warning is shown
making the user aware that the permissions on the target might not match
what he expects.
WARNING: /tmp/test/../pseudo folder does not exist. Usernames and permissions will be invalid
Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
scripts/lib/wic/partition.py | 7 +++--
scripts/lib/wic/plugins/source/rootfs.py | 36 ++++++++++++++++++++++--
2 files changed, 37 insertions(+), 6 deletions(-)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 2d95f78439..b02711be37 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -190,7 +190,7 @@ class Partition():
(self.mountpoint, self.size, self.fixed_size))
def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir,
- native_sysroot, real_rootfs = True):
+ native_sysroot, real_rootfs = True, pseudo_dir = None):
"""
Prepare content for a rootfs partition i.e. create a partition
and fill it from a /rootfs dir.
@@ -198,8 +198,9 @@ class Partition():
Currently handles ext2/3/4, btrfs, vfat and squashfs.
"""
p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
- p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
- "%s/../pseudo" % rootfs_dir)
+ if (pseudo_dir == None):
+ pseudo_dir = "%s/../pseudo" % rootfs_dir
+ p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", pseudo_dir)
p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 705aeb5563..caad9efccc 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -20,7 +20,7 @@ from oe.path import copyhardlinktree
from wic import WicError
from wic.pluginbase import SourcePlugin
-from wic.misc import get_bitbake_var
+from wic.misc import get_bitbake_var, exec_native_cmd
logger = logging.getLogger('wic')
@@ -44,6 +44,15 @@ class RootfsPlugin(SourcePlugin):
return os.path.realpath(image_rootfs_dir)
+ @staticmethod
+ def __get_pseudo(native_sysroot, rootfs, pseudo_dir):
+ pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
+ pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+ pseudo += "export PSEUDO_PASSWD=%s;" % rootfs
+ pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
+ pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+ return pseudo
+
@classmethod
def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
oe_builddir, bootimg_dir, kernel_dir,
@@ -68,8 +77,14 @@ class RootfsPlugin(SourcePlugin):
"it is not a valid path, exiting" % part.rootfs_dir)
part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
+ pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo")
+ if not os.path.lexists(pseudo_dir):
+ logger.warn("%s folder does not exist. "
+ "Usernames and permissions will be invalid " % pseudo_dir)
+ pseudo_dir = None
new_rootfs = None
+ new_pseudo = None
# Handle excluded paths.
if part.exclude_path or part.include_path:
# We need a new rootfs directory we can delete files from. Copy to
@@ -78,9 +93,23 @@ class RootfsPlugin(SourcePlugin):
if os.path.lexists(new_rootfs):
shutil.rmtree(os.path.join(new_rootfs))
-
copyhardlinktree(part.rootfs_dir, new_rootfs)
+ # Convert the pseudo directory to its new location
+ if (pseudo_dir):
+ new_pseudo = os.path.join(new_rootfs, "../pseudo%d" % part.lineno)
+ if os.path.lexists(new_pseudo):
+ shutil.rmtree(new_pseudo)
+ os.mkdir(new_pseudo)
+ shutil.copy(os.path.join(pseudo_dir, "files.db"),
+ os.path.join(new_pseudo, "files.db"))
+
+ pseudo_cmd = "%s -B -m %s -M %s" % (cls.__get_pseudo(native_sysroot,
+ new_rootfs,
+ new_pseudo),
+ part.rootfs_dir, new_rootfs)
+ exec_native_cmd(pseudo_cmd, native_sysroot)
+
for path in part.include_path or []:
copyhardlinktree(path, new_rootfs)
@@ -112,4 +141,5 @@ class RootfsPlugin(SourcePlugin):
shutil.rmtree(full_path)
part.prepare_rootfs(cr_workdir, oe_builddir,
- new_rootfs or part.rootfs_dir, native_sysroot)
+ new_rootfs or part.rootfs_dir, native_sysroot,
+ pseudo_dir = new_pseudo or pseudo_dir)
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v6 02/10] wic: Fix multi images .wks with bitbake
2020-04-14 13:36 [PATCH v6 00/10] Fix permissions and embed-rotofs Ricardo Ribalda
2020-04-14 13:36 ` [PATCH v6 01/10] wic: Fix permissions when using exclude or include path Ricardo Ribalda
@ 2020-04-14 13:36 ` Ricardo Ribalda
2020-04-14 13:36 ` [PATCH v6 03/10] wic: Add --change-directory argument Ricardo Ribalda
` (7 subsequent siblings)
9 siblings, 0 replies; 17+ messages in thread
From: Ricardo Ribalda @ 2020-04-14 13:36 UTC (permalink / raw)
To: openembedded-core, Paul Barker; +Cc: Ricardo Ribalda Delgado
In order to support .wks files with multiple images inside bitbake we
need to explicitly set the pseudo database in use.
Eg: If we try this .mks:
part / --source rootfs --ondisk sda --fstype=ext4
part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4
The username for all the files under /export will be set to the runner
of bitbake (usually UID 1000).
Before we run wic, we need to make sure that the pseudo database will be
flushed, and contains all the data needed.
Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
meta/classes/image_types_wic.bbclass | 8 ++++++--
scripts/lib/wic/partition.py | 9 +++------
2 files changed, 9 insertions(+), 8 deletions(-)
diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
index b83308b45c..96ed0473ee 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -32,8 +32,7 @@ IMAGE_CMD_wic () {
if [ -z "$wks" ]; then
bbfatal "No kickstart files from WKS_FILES were found: ${WKS_FILES}. Please set WKS_FILE or WKS_FILES appropriately."
fi
-
- BUILDDIR="${TOPDIR}" wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" ${WIC_CREATE_EXTRA_ARGS}
+ BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" ${WIC_CREATE_EXTRA_ARGS}
mv "$build_wic/$(basename "${wks%.wks}")"*.direct "$out${IMAGE_NAME_SUFFIX}.wic"
}
IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
@@ -86,6 +85,10 @@ python do_write_wks_template () {
bb.utils.copyfile(wks_file, "%s/%s" % (depdir, basename + '-' + os.path.basename(wks_file)))
}
+do_flush_pseudodb() {
+ ${FAKEROOTENV} ${FAKEROOTCMD} -S
+}
+
python () {
if d.getVar('USING_WIC'):
wks_file_u = d.getVar('WKS_FULL_PATH', False)
@@ -139,6 +142,7 @@ python do_rootfs_wicenv () {
depdir = d.getVar('IMGDEPLOYDIR')
bb.utils.copyfile(os.path.join(outdir, basename) + '.env', os.path.join(depdir, basename) + '.env')
}
+addtask do_flush_pseudodb after do_image before do_image_wic
addtask do_rootfs_wicenv after do_image before do_image_wic
do_rootfs_wicenv[vardeps] += "${WICVARS}"
do_rootfs_wicenv[prefuncs] = 'set_image_size'
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index b02711be37..d850fbd1b1 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -200,13 +200,10 @@ class Partition():
p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
if (pseudo_dir == None):
pseudo_dir = "%s/../pseudo" % rootfs_dir
- p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", pseudo_dir)
- p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
- p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
- pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % p_localstatedir
- pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd
- pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp
+ pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+ pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
+ pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v6 03/10] wic: Add --change-directory argument
2020-04-14 13:36 [PATCH v6 00/10] Fix permissions and embed-rotofs Ricardo Ribalda
2020-04-14 13:36 ` [PATCH v6 01/10] wic: Fix permissions when using exclude or include path Ricardo Ribalda
2020-04-14 13:36 ` [PATCH v6 02/10] wic: Fix multi images .wks with bitbake Ricardo Ribalda
@ 2020-04-14 13:36 ` Ricardo Ribalda
2020-04-14 13:36 ` [PATCH v6 04/10] wic: Continue if excluded_path does not exist Ricardo Ribalda
` (6 subsequent siblings)
9 siblings, 0 replies; 17+ messages in thread
From: Ricardo Ribalda @ 2020-04-14 13:36 UTC (permalink / raw)
To: openembedded-core, Paul Barker; +Cc: Ricardo Ribalda Delgado
This option allows to specify which part of a rootfs is going to be
included, the same way the -C argument on tar.
Thanks to this option we can make sure the permissions and usernames
on the target partition are respected, and also simplify the creation of
splitted partitons, not neeting to invoke external vars or using .wks.in
files. Eg:
part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/
part /etc --source rootfs --fstype=ext4 --change-directory=etc
Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
scripts/lib/wic/help.py | 6 ++++++
scripts/lib/wic/ksparser.py | 1 +
scripts/lib/wic/partition.py | 1 +
scripts/lib/wic/plugins/source/rootfs.py | 21 ++++++++++++++++++---
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 1e3d06a87b..62a2a90e79 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -980,6 +980,12 @@ DESCRIPTION
copies. This option only has an effect with the rootfs
source plugin.
+ --change-directory: This option is specific to wic. It changes to the
+ given directory before copying the files. This
+ option is useful when we want to split a rootfs in
+ multiple partitions and we want to keep the right
+ permissions and usernames in all the partitions.
+
--extra-space: This option is specific to wic. It adds extra
space after the space filled by the content
of the partition. The final size can go
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 650b976223..c60869d397 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -138,6 +138,7 @@ class KickStart():
part.add_argument('--align', type=int)
part.add_argument('--exclude-path', nargs='+')
part.add_argument('--include-path', nargs='+')
+ part.add_argument('--change-directory')
part.add_argument("--extra-space", type=sizetype)
part.add_argument('--fsoptions', dest='fsopts')
part.add_argument('--fstype', default='vfat',
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index d850fbd1b1..3240be072a 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -31,6 +31,7 @@ class Partition():
self.extra_space = args.extra_space
self.exclude_path = args.exclude_path
self.include_path = args.include_path
+ self.change_directory = args.change_directory
self.fsopts = args.fsopts
self.fstype = args.fstype
self.label = args.label
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index caad9efccc..c831829398 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -86,14 +86,29 @@ class RootfsPlugin(SourcePlugin):
new_rootfs = None
new_pseudo = None
# Handle excluded paths.
- if part.exclude_path or part.include_path:
+ if part.exclude_path or part.include_path or part.change_directory:
# We need a new rootfs directory we can delete files from. Copy to
# workdir.
new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno))
if os.path.lexists(new_rootfs):
shutil.rmtree(os.path.join(new_rootfs))
- copyhardlinktree(part.rootfs_dir, new_rootfs)
+
+ if part.change_directory:
+ cd = part.change_directory
+ if cd[-1] == '/':
+ cd = cd[:-1]
+ if os.path.isabs(cd):
+ logger.error("Must be relative: --change-directory=%s" % cd)
+ sys.exit(1)
+ orig_dir = os.path.realpath(os.path.join(part.rootfs_dir, cd))
+ if not orig_dir.startswith(part.rootfs_dir):
+ logger.error("'%s' points to a path outside the rootfs" % orig_dir)
+ sys.exit(1)
+
+ else:
+ orig_dir = part.rootfs_dir
+ copyhardlinktree(orig_dir, new_rootfs)
# Convert the pseudo directory to its new location
if (pseudo_dir):
@@ -107,7 +122,7 @@ class RootfsPlugin(SourcePlugin):
pseudo_cmd = "%s -B -m %s -M %s" % (cls.__get_pseudo(native_sysroot,
new_rootfs,
new_pseudo),
- part.rootfs_dir, new_rootfs)
+ orig_dir, new_rootfs)
exec_native_cmd(pseudo_cmd, native_sysroot)
for path in part.include_path or []:
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v6 04/10] wic: Continue if excluded_path does not exist
2020-04-14 13:36 [PATCH v6 00/10] Fix permissions and embed-rotofs Ricardo Ribalda
` (2 preceding siblings ...)
2020-04-14 13:36 ` [PATCH v6 03/10] wic: Add --change-directory argument Ricardo Ribalda
@ 2020-04-14 13:36 ` Ricardo Ribalda
2020-04-14 13:36 ` [PATCH v6 05/10] wic: Avoid creating invalid pseudo directory Ricardo Ribalda
` (5 subsequent siblings)
9 siblings, 0 replies; 17+ messages in thread
From: Ricardo Ribalda @ 2020-04-14 13:36 UTC (permalink / raw)
To: openembedded-core, Paul Barker; +Cc: Ricardo Ribalda Delgado
If an excuded path does not exist, continue without an error.
This allows to seamleasly reuse .wks among different projects.
Eg:
part / --source rootfs --fstype=ext4 --exclude-path=opt/private_keys
Where /opt/private_keys in only populated by some of the image.bb files.
Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
scripts/lib/wic/plugins/source/rootfs.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index c831829398..1d21ec2252 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -135,7 +135,6 @@ class RootfsPlugin(SourcePlugin):
sys.exit(1)
full_path = os.path.realpath(os.path.join(new_rootfs, path))
-
# Disallow climbing outside of parent directory using '..',
# because doing so could be quite disastrous (we will delete the
# directory).
@@ -143,6 +142,9 @@ class RootfsPlugin(SourcePlugin):
logger.error("'%s' points to a path outside the rootfs" % orig_path)
sys.exit(1)
+ if not os.path.lexists(full_path):
+ continue
+
if path.endswith(os.sep):
# Delete content only.
for entry in os.listdir(full_path):
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v6 05/10] wic: Avoid creating invalid pseudo directory
2020-04-14 13:36 [PATCH v6 00/10] Fix permissions and embed-rotofs Ricardo Ribalda
` (3 preceding siblings ...)
2020-04-14 13:36 ` [PATCH v6 04/10] wic: Continue if excluded_path does not exist Ricardo Ribalda
@ 2020-04-14 13:36 ` Ricardo Ribalda
2020-04-14 13:36 ` [PATCH v6 06/10] oeqa: wic: Add tests for permissions and change-directory Ricardo Ribalda
` (4 subsequent siblings)
9 siblings, 0 replies; 17+ messages in thread
From: Ricardo Ribalda @ 2020-04-14 13:36 UTC (permalink / raw)
To: openembedded-core, Paul Barker; +Cc: Ricardo Ribalda Delgado
If the source of the rootfs is not a bitbake cooked image, or it is not
pointing to the root of one, we call pseudo again, which will produce
a new pseudo folder at rootfs/../pseudo
Eg:
part /etc --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/home --fstype=ext4
Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
scripts/lib/wic/partition.py | 15 ++++++++-------
1 file changed, 8 insertions(+), 7 deletions(-)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 3240be072a..7d9dd616a6 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -199,13 +199,14 @@ class Partition():
Currently handles ext2/3/4, btrfs, vfat and squashfs.
"""
p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
- if (pseudo_dir == None):
- pseudo_dir = "%s/../pseudo" % rootfs_dir
- pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
- pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
- pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
- pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
- pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+ if (pseudo_dir):
+ pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
+ pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+ pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
+ pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
+ pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+ else:
+ pseudo = None
rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
self.lineno, self.fstype)
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v6 06/10] oeqa: wic: Add tests for permissions and change-directory
2020-04-14 13:36 [PATCH v6 00/10] Fix permissions and embed-rotofs Ricardo Ribalda
` (4 preceding siblings ...)
2020-04-14 13:36 ` [PATCH v6 05/10] wic: Avoid creating invalid pseudo directory Ricardo Ribalda
@ 2020-04-14 13:36 ` Ricardo Ribalda
2020-04-18 20:07 ` Paul Barker
2020-04-14 13:36 ` [PATCH v6 07/10] wic: misc: Do not find for executables in ALREADY_PROVIDED Ricardo Ribalda
` (3 subsequent siblings)
9 siblings, 1 reply; 17+ messages in thread
From: Ricardo Ribalda @ 2020-04-14 13:36 UTC (permalink / raw)
To: openembedded-core, Paul Barker; +Cc: Ricardo Ribalda Delgado
Make sure that the permissions and username are respected when using all
the rootfs modifiers.
Add tests for change-directory command
Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
meta/lib/oeqa/selftest/cases/wic.py | 91 +++++++++++++++++++++++++++++
1 file changed, 91 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py
index 626a217e69..8ef08c3d77 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -62,6 +62,12 @@ def extract_files(debugfs_output):
return [line.split('/')[5].strip() for line in \
debugfs_output.strip().split('/\n')]
+def files_own_by_root(debugfs_output):
+ for line in debugfs_output.strip().split('/\n'):
+ if line.split('/')[3:5] != ['0', '0']:
+ print(debugfs_output)
+ return False
+ return True
class WicTestCase(OESelftestTestCase):
"""Wic test class."""
@@ -84,6 +90,7 @@ class WicTestCase(OESelftestTestCase):
self.skipTest('wic-tools cannot be built due its (intltool|gettext)-native dependency and NLS disable')
bitbake('core-image-minimal')
+ bitbake('core-image-minimal-mtdutils')
WicTestCase.image_is_ready = True
rmtree(self.resultdir, ignore_errors=True)
@@ -506,6 +513,90 @@ part /part2 --source rootfs --ondisk mmcblk0 --fstype=ext4 --include-path %s"""
% (wks_file, self.resultdir), ignore_status=True).status)
os.remove(wks_file)
+ def test_permissions(self):
+ """Test permissions are respected"""
+
+ oldpath = os.environ['PATH']
+ os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
+
+ t_normal = """
+part / --source rootfs --fstype=ext4
+"""
+ t_exclude = """
+part / --source rootfs --fstype=ext4 --exclude-path=home
+"""
+ t_multi = """
+part / --source rootfs --ondisk sda --fstype=ext4
+part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4
+"""
+ t_change = """
+part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/
+part /etc --source rootfs --fstype=ext4 --change-directory=etc
+"""
+ tests = [t_normal, t_exclude, t_multi, t_change]
+
+ try:
+ for test in tests:
+ include_path = os.path.join(self.resultdir, 'test-include')
+ os.makedirs(include_path)
+ wks_file = os.path.join(include_path, 'temp.wks')
+ with open(wks_file, 'w') as wks:
+ wks.write(test)
+ runCmd("wic create %s -e core-image-minimal -o %s" \
+ % (wks_file, self.resultdir))
+
+ for part in glob(os.path.join(self.resultdir, 'temp-*.direct.p*')):
+ res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part))
+ self.assertEqual(True, files_own_by_root(res.output))
+
+ rmtree(self.resultdir, ignore_errors=True)
+
+ finally:
+ os.environ['PATH'] = oldpath
+
+ def test_change_directory(self):
+ """Test --change-directory wks option."""
+
+ oldpath = os.environ['PATH']
+ os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
+
+ try:
+ include_path = os.path.join(self.resultdir, 'test-include')
+ os.makedirs(include_path)
+ wks_file = os.path.join(include_path, 'temp.wks')
+ with open(wks_file, 'w') as wks:
+ rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
+ wks.write("part /etc --source rootfs --fstype=ext4 --change-directory=etc")
+ runCmd("wic create %s -e core-image-minimal -o %s" \
+ % (wks_file, self.resultdir))
+
+ part1 = glob(os.path.join(self.resultdir, 'temp-*.direct.p1'))[0]
+
+ res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
+ files = extract_files(res.output)
+ self.assertIn('shadow', files)
+
+ finally:
+ os.environ['PATH'] = oldpath
+
+ def test_change_directory_errors(self):
+ """Test --change-directory wks option error handling."""
+ wks_file = 'temp.wks'
+
+ # Absolute argument.
+ with open(wks_file, 'w') as wks:
+ wks.write("part / --source rootfs --fstype=ext4 --change-directory /usr")
+ self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
+ % (wks_file, self.resultdir), ignore_status=True).status)
+ os.remove(wks_file)
+
+ # Argument pointing to parent directory.
+ with open(wks_file, 'w') as wks:
+ wks.write("part / --source rootfs --fstype=ext4 --change-directory ././..")
+ self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
+ % (wks_file, self.resultdir), ignore_status=True).status)
+ os.remove(wks_file)
+
class Wic2(WicTestCase):
def test_bmap_short(self):
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v6 07/10] wic: misc: Do not find for executables in ALREADY_PROVIDED
2020-04-14 13:36 [PATCH v6 00/10] Fix permissions and embed-rotofs Ricardo Ribalda
` (5 preceding siblings ...)
2020-04-14 13:36 ` [PATCH v6 06/10] oeqa: wic: Add tests for permissions and change-directory Ricardo Ribalda
@ 2020-04-14 13:36 ` Ricardo Ribalda
2020-04-14 13:36 ` [PATCH v6 08/10] wic: root: Add an opt. destination on include-path Ricardo Ribalda
` (2 subsequent siblings)
9 siblings, 0 replies; 17+ messages in thread
From: Ricardo Ribalda @ 2020-04-14 13:36 UTC (permalink / raw)
To: openembedded-core, Paul Barker; +Cc: Ricardo Ribalda Delgado
Executables like tar won't be available on the native sysroot, as they
are part of the ALREADY_PROVIDED variable.
Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
scripts/lib/wic/misc.py | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py
index 1f199b9f23..91975ba151 100644
--- a/scripts/lib/wic/misc.py
+++ b/scripts/lib/wic/misc.py
@@ -45,7 +45,8 @@ NATIVE_RECIPES = {"bmaptool": "bmap-tools",
"parted": "parted",
"sfdisk": "util-linux",
"sgdisk": "gptfdisk",
- "syslinux": "syslinux"
+ "syslinux": "syslinux",
+ "tar": "tar"
}
def runtool(cmdln_or_args):
@@ -112,6 +113,15 @@ def exec_cmd(cmd_and_args, as_shell=False):
"""
return _exec_cmd(cmd_and_args, as_shell)[1]
+def find_executable(cmd, paths):
+ recipe = cmd
+ if recipe in NATIVE_RECIPES:
+ recipe = NATIVE_RECIPES[recipe]
+ provided = get_bitbake_var("ASSUME_PROVIDED")
+ if provided and "%s-native" % recipe in provided:
+ return True
+
+ return spawn.find_executable(cmd, paths)
def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
"""
@@ -136,7 +146,7 @@ def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
logger.debug("exec_native_cmd: %s", native_cmd_and_args)
# If the command isn't in the native sysroot say we failed.
- if spawn.find_executable(args[0], native_paths):
+ if find_executable(args[0], native_paths):
ret, out = _exec_cmd(native_cmd_and_args, True)
else:
ret = 127
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v6 08/10] wic: root: Add an opt. destination on include-path
2020-04-14 13:36 [PATCH v6 00/10] Fix permissions and embed-rotofs Ricardo Ribalda
` (6 preceding siblings ...)
2020-04-14 13:36 ` [PATCH v6 07/10] wic: misc: Do not find for executables in ALREADY_PROVIDED Ricardo Ribalda
@ 2020-04-14 13:36 ` Ricardo Ribalda
2020-04-18 19:02 ` Paul Barker
2020-04-14 13:36 ` [PATCH v6 09/10] wic: rootfs: Combine path_validation in one function Ricardo Ribalda
2020-04-14 13:36 ` [PATCH v6 10/10] oeqa: wic: Add more tests for include_path Ricardo Ribalda
9 siblings, 1 reply; 17+ messages in thread
From: Ricardo Ribalda @ 2020-04-14 13:36 UTC (permalink / raw)
To: openembedded-core, Paul Barker; +Cc: Ricardo Ribalda Delgado
Allow specifying an optional destination to include-path and make the
option aware of permissions and owners.
It is very useful for making a partition that contains the rootfs for a
host and a target Eg:
/ -> Roofs for the host
/export/ -> Rootfs for the target (which will netboot)
Although today we support making a partition for "/export" this might
not be compatible with some upgrade systems, or we might be limited by
the number of partitions.
With this patch we can use something like:
part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils export/ --include-path hello
on the .wks file.
Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
meta/recipes-core/meta/wic-tools.bb | 2 +-
scripts/lib/wic/help.py | 20 +++++----
scripts/lib/wic/ksparser.py | 2 +-
scripts/lib/wic/plugins/source/rootfs.py | 54 +++++++++++++++++++++++-
4 files changed, 66 insertions(+), 12 deletions(-)
diff --git a/meta/recipes-core/meta/wic-tools.bb b/meta/recipes-core/meta/wic-tools.bb
index 09eb409e87..8aeb942ed2 100644
--- a/meta/recipes-core/meta/wic-tools.bb
+++ b/meta/recipes-core/meta/wic-tools.bb
@@ -6,7 +6,7 @@ DEPENDS = "\
parted-native syslinux-native gptfdisk-native dosfstools-native \
mtools-native bmap-tools-native grub-efi-native cdrtools-native \
btrfs-tools-native squashfs-tools-native pseudo-native \
- e2fsprogs-native util-linux-native \
+ e2fsprogs-native util-linux-native tar-native\
"
DEPENDS_append_x86 = " syslinux grub-efi systemd-boot"
DEPENDS_append_x86-64 = " syslinux grub-efi systemd-boot"
diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 62a2a90e79..bd3a2b97df 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -971,14 +971,18 @@ DESCRIPTION
has an effect with the rootfs source plugin.
--include-path: This option is specific to wic. It adds the contents
- of the given path to the resulting image. The path is
- relative to the directory in which wic is running not
- the rootfs itself so use of an absolute path is
- recommended. This option is most useful when multiple
- copies of the rootfs are added to an image and it is
- required to add extra content to only one of these
- copies. This option only has an effect with the rootfs
- source plugin.
+ of the given path or a rootfs to the resulting image.
+ The option contains two fields, the origin and the
+ destination. When the origin is a rootfs, it follows
+ the same logic as the rootfs-dir argument and the
+ permissions and owners are kept. When the origin is a
+ path, it is relative to the directory in which wic is
+ running not the rootfs itself so use of an absolute
+ path is recommended, and the owner and group is set to
+ root:root. If no destination is given it is
+ automatically set to the root of the rootfs. This
+ option only has an effect with the rootfs source
+ plugin.
--change-directory: This option is specific to wic. It changes to the
given directory before copying the files. This
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index c60869d397..b8befe78e3 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -137,7 +137,7 @@ class KickStart():
part.add_argument('--active', action='store_true')
part.add_argument('--align', type=int)
part.add_argument('--exclude-path', nargs='+')
- part.add_argument('--include-path', nargs='+')
+ part.add_argument('--include-path', nargs='+', action='append')
part.add_argument('--change-directory')
part.add_argument("--extra-space", type=sizetype)
part.add_argument('--fsoptions', dest='fsopts')
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 1d21ec2252..9d0666b638 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -17,6 +17,7 @@ import shutil
import sys
from oe.path import copyhardlinktree
+from pathlib import Path
from wic import WicError
from wic.pluginbase import SourcePlugin
@@ -125,8 +126,57 @@ class RootfsPlugin(SourcePlugin):
orig_dir, new_rootfs)
exec_native_cmd(pseudo_cmd, native_sysroot)
- for path in part.include_path or []:
- copyhardlinktree(path, new_rootfs)
+ for in_path in part.include_path or []:
+ #parse arguments
+ include_path = in_path[0]
+ if len(in_path) > 2:
+ logger.error("'Invalid number of arguments for include-path")
+ sys.exit(1)
+ if len(in_path) == 2:
+ path = in_path[1]
+ else:
+ path = None
+
+ #pack content
+ tar_file = os.path.realpath(os.path.join(cr_workdir, "aux.tar"))
+ if os.path.isfile(include_path):
+ parent = os.path.dirname(include_path)
+ tar_cmd = "tar c --owner=root --group=root -f %s -C %s %s" % (
+ tar_file, parent, os.path.relpath(include_path, parent))
+ exec_native_cmd(tar_cmd, native_sysroot)
+ else:
+ if include_path in krootfs_dir:
+ include_path = krootfs_dir[include_path]
+ include_path = cls.__get_rootfs_dir(include_path)
+ include_pseudo = os.path.join(include_path, "../pseudo")
+ if os.path.lexists(include_pseudo):
+ pseudo = cls.__get_pseudo(native_sysroot, include_path,
+ include_pseudo)
+ tar_cmd = "tar cf %s -C %s ." % (tar_file, include_path)
+ else:
+ pseudo = None
+ tar_cmd = "tar c --owner=root --group=root -f %s -C %s ." % (
+ tar_file, include_path)
+ exec_native_cmd(tar_cmd, native_sysroot, pseudo)
+
+ #create destination
+ if path:
+ destination = os.path.realpath(os.path.join(new_rootfs, path))
+ if not destination.startswith(new_rootfs):
+ logger.error("%s %s" % (destination, new_rootfs))
+ sys.exit(1)
+ Path(destination).mkdir(parents=True, exist_ok=True)
+ else:
+ destination = new_rootfs
+
+ #extract destination
+ untar_cmd = "tar xf %s -C %s" % (tar_file, destination)
+ if new_pseudo:
+ pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo)
+ else:
+ pseudo = None
+ exec_native_cmd(untar_cmd, native_sysroot, pseudo)
+ os.remove(tar_file)
for orig_path in part.exclude_path or []:
path = orig_path
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v6 09/10] wic: rootfs: Combine path_validation in one function
2020-04-14 13:36 [PATCH v6 00/10] Fix permissions and embed-rotofs Ricardo Ribalda
` (7 preceding siblings ...)
2020-04-14 13:36 ` [PATCH v6 08/10] wic: root: Add an opt. destination on include-path Ricardo Ribalda
@ 2020-04-14 13:36 ` Ricardo Ribalda
2020-04-14 13:36 ` [PATCH v6 10/10] oeqa: wic: Add more tests for include_path Ricardo Ribalda
9 siblings, 0 replies; 17+ messages in thread
From: Ricardo Ribalda @ 2020-04-14 13:36 UTC (permalink / raw)
To: openembedded-core, Paul Barker; +Cc: Ricardo Ribalda Delgado
Combine all the common path validation in a function to avoid code
duplication.
Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
scripts/lib/wic/plugins/source/rootfs.py | 41 +++++++++++-------------
1 file changed, 19 insertions(+), 22 deletions(-)
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 9d0666b638..53942cc4b7 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -32,6 +32,22 @@ class RootfsPlugin(SourcePlugin):
name = 'rootfs'
+ @staticmethod
+ def __validate_path(cmd, rootfs_dir, path):
+ if os.path.isabs(path):
+ logger.error("%s: Must be relative: %s" % (cmd, orig_path))
+ sys.exit(1)
+
+ # Disallow climbing outside of parent directory using '..',
+ # because doing so could be quite disastrous (we will delete the
+ # directory, or modify a directory outside OpenEmbedded).
+ full_path = os.path.realpath(os.path.join(rootfs_dir, path))
+ if not full_path.startswith(os.path.realpath(rootfs_dir)):
+ logger.error("%s: Must point inside the rootfs:" % (cmd, path))
+ sys.exit(1)
+
+ return full_path
+
@staticmethod
def __get_rootfs_dir(rootfs_dir):
if os.path.isdir(rootfs_dir):
@@ -99,14 +115,7 @@ class RootfsPlugin(SourcePlugin):
cd = part.change_directory
if cd[-1] == '/':
cd = cd[:-1]
- if os.path.isabs(cd):
- logger.error("Must be relative: --change-directory=%s" % cd)
- sys.exit(1)
- orig_dir = os.path.realpath(os.path.join(part.rootfs_dir, cd))
- if not orig_dir.startswith(part.rootfs_dir):
- logger.error("'%s' points to a path outside the rootfs" % orig_dir)
- sys.exit(1)
-
+ orig_dir = cls.__validate_path("--change-directory", part.rootfs_dir, cd)
else:
orig_dir = part.rootfs_dir
copyhardlinktree(orig_dir, new_rootfs)
@@ -161,10 +170,7 @@ class RootfsPlugin(SourcePlugin):
#create destination
if path:
- destination = os.path.realpath(os.path.join(new_rootfs, path))
- if not destination.startswith(new_rootfs):
- logger.error("%s %s" % (destination, new_rootfs))
- sys.exit(1)
+ destination = cls.__validate_path("--include-path", new_rootfs, path)
Path(destination).mkdir(parents=True, exist_ok=True)
else:
destination = new_rootfs
@@ -180,17 +186,8 @@ class RootfsPlugin(SourcePlugin):
for orig_path in part.exclude_path or []:
path = orig_path
- if os.path.isabs(path):
- logger.error("Must be relative: --exclude-path=%s" % orig_path)
- sys.exit(1)
- full_path = os.path.realpath(os.path.join(new_rootfs, path))
- # Disallow climbing outside of parent directory using '..',
- # because doing so could be quite disastrous (we will delete the
- # directory).
- if not full_path.startswith(new_rootfs):
- logger.error("'%s' points to a path outside the rootfs" % orig_path)
- sys.exit(1)
+ full_path = cls.__validate_path("--exclude-path", new_rootfs, path)
if not os.path.lexists(full_path):
continue
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v6 10/10] oeqa: wic: Add more tests for include_path
2020-04-14 13:36 [PATCH v6 00/10] Fix permissions and embed-rotofs Ricardo Ribalda
` (8 preceding siblings ...)
2020-04-14 13:36 ` [PATCH v6 09/10] wic: rootfs: Combine path_validation in one function Ricardo Ribalda
@ 2020-04-14 13:36 ` Ricardo Ribalda
2020-04-18 20:10 ` Paul Barker
9 siblings, 1 reply; 17+ messages in thread
From: Ricardo Ribalda @ 2020-04-14 13:36 UTC (permalink / raw)
To: openembedded-core, Paul Barker; +Cc: Ricardo Ribalda Delgado
Make sure permissions are respected.
Add new test for orig/destination option.
Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
meta/lib/oeqa/selftest/cases/wic.py | 64 ++++++++++++++++++++++++++++-
1 file changed, 63 insertions(+), 1 deletion(-)
diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py
index 8ef08c3d77..1036269b64 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -486,15 +486,77 @@ part /part2 --source rootfs --ondisk mmcblk0 --fstype=ext4 --include-path %s"""
res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
files = extract_files(res.output)
self.assertNotIn('test-file', files)
+ self.assertEqual(True, files_own_by_root(res.output))
- # Test partition 2, should not contain 'test-file'
+ # Test partition 2, should contain 'test-file'
res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part2))
files = extract_files(res.output)
self.assertIn('test-file', files)
+ self.assertEqual(True, files_own_by_root(res.output))
finally:
os.environ['PATH'] = oldpath
+ def test_include_path_embeded(self):
+ """Test --include-path wks option."""
+
+ oldpath = os.environ['PATH']
+ os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
+
+ try:
+ include_path = os.path.join(self.resultdir, 'test-include')
+ os.makedirs(include_path)
+ with open(os.path.join(include_path, 'test-file'), 'w') as t:
+ t.write("test\n")
+ wks_file = os.path.join(include_path, 'temp.wks')
+ with open(wks_file, 'w') as wks:
+ rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
+ wks.write("""
+part / --source rootfs --fstype=ext4 --include-path %s --include-path core-image-minimal-mtdutils export/"""
+ % (include_path))
+ runCmd("wic create %s -e core-image-minimal -o %s" \
+ % (wks_file, self.resultdir))
+
+ part1 = glob(os.path.join(self.resultdir, 'temp-*.direct.p1'))[0]
+
+ res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
+ files = extract_files(res.output)
+ self.assertIn('test-file', files)
+ self.assertEqual(True, files_own_by_root(res.output))
+
+ res = runCmd("debugfs -R 'ls -p /export/etc/' %s 2>/dev/null" % (part1))
+ files = extract_files(res.output)
+ self.assertIn('shadow', files)
+ self.assertEqual(True, files_own_by_root(res.output))
+
+ finally:
+ os.environ['PATH'] = oldpath
+
+ def test_include_path_errors(self):
+ """Test --include-path wks option error handling."""
+ wks_file = 'temp.wks'
+
+ # Absolute argument.
+ with open(wks_file, 'w') as wks:
+ wks.write("part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils /export")
+ self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
+ % (wks_file, self.resultdir), ignore_status=True).status)
+ os.remove(wks_file)
+
+ # Argument pointing to parent directory.
+ with open(wks_file, 'w') as wks:
+ wks.write("part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils ././..")
+ self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
+ % (wks_file, self.resultdir), ignore_status=True).status)
+ os.remove(wks_file)
+
+ # 3 Argument pointing to parent directory.
+ with open(wks_file, 'w') as wks:
+ wks.write("part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils export/ dummy")
+ self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
+ % (wks_file, self.resultdir), ignore_status=True).status)
+ os.remove(wks_file)
+
def test_exclude_path_errors(self):
"""Test --exclude-path wks option error handling."""
wks_file = 'temp.wks'
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH v6 08/10] wic: root: Add an opt. destination on include-path
2020-04-14 13:36 ` [PATCH v6 08/10] wic: root: Add an opt. destination on include-path Ricardo Ribalda
@ 2020-04-18 19:02 ` Paul Barker
0 siblings, 0 replies; 17+ messages in thread
From: Paul Barker @ 2020-04-18 19:02 UTC (permalink / raw)
To: Ricardo Ribalda Delgado; +Cc: openembedded-core
On Tue, 14 Apr 2020 15:36:12 +0200
Ricardo Ribalda Delgado <ricardo@ribalda.com> wrote:
> Allow specifying an optional destination to include-path and make the
> option aware of permissions and owners.
>
> It is very useful for making a partition that contains the rootfs for a
> host and a target Eg:
>
> / -> Roofs for the host
> /export/ -> Rootfs for the target (which will netboot)
>
> Although today we support making a partition for "/export" this might
> not be compatible with some upgrade systems, or we might be limited by
> the number of partitions.
>
> With this patch we can use something like:
>
> part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils export/ --include-path hello
>
> on the .wks file.
>
> Cc: Paul Barker <pbarker@konsulko.com>
> Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
> ---
> meta/recipes-core/meta/wic-tools.bb | 2 +-
> scripts/lib/wic/help.py | 20 +++++----
> scripts/lib/wic/ksparser.py | 2 +-
> scripts/lib/wic/plugins/source/rootfs.py | 54 +++++++++++++++++++++++-
> 4 files changed, 66 insertions(+), 12 deletions(-)
>
> diff --git a/meta/recipes-core/meta/wic-tools.bb b/meta/recipes-core/meta/wic-tools.bb
> index 09eb409e87..8aeb942ed2 100644
> --- a/meta/recipes-core/meta/wic-tools.bb
> +++ b/meta/recipes-core/meta/wic-tools.bb
> @@ -6,7 +6,7 @@ DEPENDS = "\
> parted-native syslinux-native gptfdisk-native dosfstools-native \
> mtools-native bmap-tools-native grub-efi-native cdrtools-native \
> btrfs-tools-native squashfs-tools-native pseudo-native \
> - e2fsprogs-native util-linux-native \
> + e2fsprogs-native util-linux-native tar-native\
> "
> DEPENDS_append_x86 = " syslinux grub-efi systemd-boot"
> DEPENDS_append_x86-64 = " syslinux grub-efi systemd-boot"
> diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
> index 62a2a90e79..bd3a2b97df 100644
> --- a/scripts/lib/wic/help.py
> +++ b/scripts/lib/wic/help.py
> @@ -971,14 +971,18 @@ DESCRIPTION
> has an effect with the rootfs source plugin.
>
> --include-path: This option is specific to wic. It adds the contents
> - of the given path to the resulting image. The path is
> - relative to the directory in which wic is running not
> - the rootfs itself so use of an absolute path is
> - recommended. This option is most useful when multiple
> - copies of the rootfs are added to an image and it is
> - required to add extra content to only one of these
> - copies. This option only has an effect with the rootfs
> - source plugin.
> + of the given path or a rootfs to the resulting image.
> + The option contains two fields, the origin and the
> + destination. When the origin is a rootfs, it follows
> + the same logic as the rootfs-dir argument and the
> + permissions and owners are kept. When the origin is a
> + path, it is relative to the directory in which wic is
> + running not the rootfs itself so use of an absolute
> + path is recommended, and the owner and group is set to
> + root:root. If no destination is given it is
> + automatically set to the root of the rootfs. This
> + option only has an effect with the rootfs source
> + plugin.
>
> --change-directory: This option is specific to wic. It changes to the
> given directory before copying the files. This
> diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> index c60869d397..b8befe78e3 100644
> --- a/scripts/lib/wic/ksparser.py
> +++ b/scripts/lib/wic/ksparser.py
> @@ -137,7 +137,7 @@ class KickStart():
> part.add_argument('--active', action='store_true')
> part.add_argument('--align', type=int)
> part.add_argument('--exclude-path', nargs='+')
> - part.add_argument('--include-path', nargs='+')
> + part.add_argument('--include-path', nargs='+', action='append')
> part.add_argument('--change-directory')
> part.add_argument("--extra-space", type=sizetype)
> part.add_argument('--fsoptions', dest='fsopts')
> diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
> index 1d21ec2252..9d0666b638 100644
> --- a/scripts/lib/wic/plugins/source/rootfs.py
> +++ b/scripts/lib/wic/plugins/source/rootfs.py
> @@ -17,6 +17,7 @@ import shutil
> import sys
>
> from oe.path import copyhardlinktree
> +from pathlib import Path
>
> from wic import WicError
> from wic.pluginbase import SourcePlugin
> @@ -125,8 +126,57 @@ class RootfsPlugin(SourcePlugin):
> orig_dir, new_rootfs)
> exec_native_cmd(pseudo_cmd, native_sysroot)
>
> - for path in part.include_path or []:
> - copyhardlinktree(path, new_rootfs)
> + for in_path in part.include_path or []:
> + #parse arguments
> + include_path = in_path[0]
> + if len(in_path) > 2:
> + logger.error("'Invalid number of arguments for include-path")
> + sys.exit(1)
> + if len(in_path) == 2:
> + path = in_path[1]
> + else:
> + path = None
> +
> + #pack content
> + tar_file = os.path.realpath(os.path.join(cr_workdir, "aux.tar"))
Could you explain why you're using tar here either in the commit
message or the comment?
And can we use the line number from the wks in the tarfile name to
match what we elsewhere in this script? If there's multiple lines using
`--include-path` that would let me examine each intermediate tarball if
I need to for debugging (assuming I locally commented out
`os.remove(tar_file)` below).
> + if os.path.isfile(include_path):
> + parent = os.path.dirname(include_path)
> + tar_cmd = "tar c --owner=root --group=root -f %s -C %s %s" % (
> + tar_file, parent, os.path.relpath(include_path, parent))
> + exec_native_cmd(tar_cmd, native_sysroot)
> + else:
> + if include_path in krootfs_dir:
> + include_path = krootfs_dir[include_path]
> + include_path = cls.__get_rootfs_dir(include_path)
> + include_pseudo = os.path.join(include_path, "../pseudo")
> + if os.path.lexists(include_pseudo):
> + pseudo = cls.__get_pseudo(native_sysroot, include_path,
> + include_pseudo)
> + tar_cmd = "tar cf %s -C %s ." % (tar_file, include_path)
> + else:
> + pseudo = None
> + tar_cmd = "tar c --owner=root --group=root -f %s -C %s ." % (
> + tar_file, include_path)
> + exec_native_cmd(tar_cmd, native_sysroot, pseudo)
> +
> + #create destination
> + if path:
> + destination = os.path.realpath(os.path.join(new_rootfs, path))
> + if not destination.startswith(new_rootfs):
> + logger.error("%s %s" % (destination, new_rootfs))
> + sys.exit(1)
> + Path(destination).mkdir(parents=True, exist_ok=True)
> + else:
> + destination = new_rootfs
> +
> + #extract destination
> + untar_cmd = "tar xf %s -C %s" % (tar_file, destination)
> + if new_pseudo:
> + pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo)
> + else:
> + pseudo = None
> + exec_native_cmd(untar_cmd, native_sysroot, pseudo)
> + os.remove(tar_file)
>
> for orig_path in part.exclude_path or []:
> path = orig_path
--
Paul Barker
Konsulko Group
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH v6 01/10] wic: Fix permissions when using exclude or include path
2020-04-14 13:36 ` [PATCH v6 01/10] wic: Fix permissions when using exclude or include path Ricardo Ribalda
@ 2020-04-18 19:03 ` Paul Barker
0 siblings, 0 replies; 17+ messages in thread
From: Paul Barker @ 2020-04-18 19:03 UTC (permalink / raw)
To: Ricardo Ribalda Delgado; +Cc: openembedded-core
On Tue, 14 Apr 2020 15:36:05 +0200
Ricardo Ribalda Delgado <ricardo@ribalda.com> wrote:
> When parameters include_path or exclude_path are passed to the rootfs
> plugin, it will copy the partition content into a folder and make all
> the modifications there.
>
> This is done using copyhardlinktree(), which does not take into
> consideration the content of the pseudo folder, which contains the
> information about the right permissions and ownership of the folders.
>
> This results in a rootfs owned by the user that is running the wic
> command (usually UID 1000), which makes some rootfs unbootable.
>
> This bug can be easily triggerd with the following .wks
>
> part / --source rootfs --fstype=ext4 --exclude-path=home
>
> And this sequence:
>
> $ wic create test-permissions -e core-image-minimal -o test/
> $ sudo mount test/test-permissions-202004080823-sda.direct.p1 /mnt
> $ ls -la /mnt/etc/shadow
>
> To fix this we copy the content of the pseudo folders to the new folder
> and modify the pseudo database using the "pseudo -B" command.
>
> If the rootfs is not a rootfs generated by bitbake a warning is shown
> making the user aware that the permissions on the target might not match
> what he expects.
>
> WARNING: /tmp/test/../pseudo folder does not exist. Usernames and permissions will be invalid
>
> Cc: Paul Barker <pbarker@konsulko.com>
> Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
> ---
> scripts/lib/wic/partition.py | 7 +++--
> scripts/lib/wic/plugins/source/rootfs.py | 36 ++++++++++++++++++++++--
> 2 files changed, 37 insertions(+), 6 deletions(-)
>
> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> index 2d95f78439..b02711be37 100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -190,7 +190,7 @@ class Partition():
> (self.mountpoint, self.size, self.fixed_size))
>
> def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir,
> - native_sysroot, real_rootfs = True):
> + native_sysroot, real_rootfs = True, pseudo_dir = None):
> """
> Prepare content for a rootfs partition i.e. create a partition
> and fill it from a /rootfs dir.
> @@ -198,8 +198,9 @@ class Partition():
> Currently handles ext2/3/4, btrfs, vfat and squashfs.
> """
> p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
> - p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
> - "%s/../pseudo" % rootfs_dir)
> + if (pseudo_dir == None):
> + pseudo_dir = "%s/../pseudo" % rootfs_dir
> + p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", pseudo_dir)
> p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
> p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
> pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
> diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
> index 705aeb5563..caad9efccc 100644
> --- a/scripts/lib/wic/plugins/source/rootfs.py
> +++ b/scripts/lib/wic/plugins/source/rootfs.py
> @@ -20,7 +20,7 @@ from oe.path import copyhardlinktree
>
> from wic import WicError
> from wic.pluginbase import SourcePlugin
> -from wic.misc import get_bitbake_var
> +from wic.misc import get_bitbake_var, exec_native_cmd
>
> logger = logging.getLogger('wic')
>
> @@ -44,6 +44,15 @@ class RootfsPlugin(SourcePlugin):
>
> return os.path.realpath(image_rootfs_dir)
>
> + @staticmethod
> + def __get_pseudo(native_sysroot, rootfs, pseudo_dir):
> + pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
> + pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
> + pseudo += "export PSEUDO_PASSWD=%s;" % rootfs
> + pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
> + pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
> + return pseudo
> +
> @classmethod
> def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
> oe_builddir, bootimg_dir, kernel_dir,
> @@ -68,8 +77,14 @@ class RootfsPlugin(SourcePlugin):
> "it is not a valid path, exiting" % part.rootfs_dir)
>
> part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
> + pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo")
> + if not os.path.lexists(pseudo_dir):
> + logger.warn("%s folder does not exist. "
> + "Usernames and permissions will be invalid " % pseudo_dir)
> + pseudo_dir = None
>
> new_rootfs = None
> + new_pseudo = None
> # Handle excluded paths.
> if part.exclude_path or part.include_path:
> # We need a new rootfs directory we can delete files from. Copy to
> @@ -78,9 +93,23 @@ class RootfsPlugin(SourcePlugin):
>
> if os.path.lexists(new_rootfs):
> shutil.rmtree(os.path.join(new_rootfs))
> -
> copyhardlinktree(part.rootfs_dir, new_rootfs)
>
> + # Convert the pseudo directory to its new location
> + if (pseudo_dir):
> + new_pseudo = os.path.join(new_rootfs, "../pseudo%d" % part.lineno)
I still don't like this, it's unclear from looking at the patch alone
that it's safe to step up the directory tree here.
As I've said before, let's use an explicit path instead, such as:
new_pseudo = os.path.realpath(os.path.join(cr_workdir, "pseudo%d" % part.lineno))
Then there's no '..', it's obviously safe and it's less likely someone
will break things by accident in the future.
> + if os.path.lexists(new_pseudo):
> + shutil.rmtree(new_pseudo)
> + os.mkdir(new_pseudo)
> + shutil.copy(os.path.join(pseudo_dir, "files.db"),
> + os.path.join(new_pseudo, "files.db"))
> +
> + pseudo_cmd = "%s -B -m %s -M %s" % (cls.__get_pseudo(native_sysroot,
> + new_rootfs,
> + new_pseudo),
> + part.rootfs_dir, new_rootfs)
> + exec_native_cmd(pseudo_cmd, native_sysroot)
> +
> for path in part.include_path or []:
> copyhardlinktree(path, new_rootfs)
>
> @@ -112,4 +141,5 @@ class RootfsPlugin(SourcePlugin):
> shutil.rmtree(full_path)
>
> part.prepare_rootfs(cr_workdir, oe_builddir,
> - new_rootfs or part.rootfs_dir, native_sysroot)
> + new_rootfs or part.rootfs_dir, native_sysroot,
> + pseudo_dir = new_pseudo or pseudo_dir)
--
Paul Barker
Konsulko Group
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH v6 06/10] oeqa: wic: Add tests for permissions and change-directory
2020-04-14 13:36 ` [PATCH v6 06/10] oeqa: wic: Add tests for permissions and change-directory Ricardo Ribalda
@ 2020-04-18 20:07 ` Paul Barker
0 siblings, 0 replies; 17+ messages in thread
From: Paul Barker @ 2020-04-18 20:07 UTC (permalink / raw)
To: Ricardo Ribalda Delgado; +Cc: openembedded-core
On Tue, 14 Apr 2020 15:36:10 +0200
Ricardo Ribalda Delgado <ricardo@ribalda.com> wrote:
> Make sure that the permissions and username are respected when using all
> the rootfs modifiers.
>
> Add tests for change-directory command
>
> Cc: Paul Barker <pbarker@konsulko.com>
> Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
> ---
> meta/lib/oeqa/selftest/cases/wic.py | 91 +++++++++++++++++++++++++++++
> 1 file changed, 91 insertions(+)
>
> diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py
> index 626a217e69..8ef08c3d77 100644
> --- a/meta/lib/oeqa/selftest/cases/wic.py
> +++ b/meta/lib/oeqa/selftest/cases/wic.py
> @@ -62,6 +62,12 @@ def extract_files(debugfs_output):
> return [line.split('/')[5].strip() for line in \
> debugfs_output.strip().split('/\n')]
>
> +def files_own_by_root(debugfs_output):
> + for line in debugfs_output.strip().split('/\n'):
> + if line.split('/')[3:5] != ['0', '0']:
> + print(debugfs_output)
> + return False
> + return True
>
> class WicTestCase(OESelftestTestCase):
> """Wic test class."""
> @@ -84,6 +90,7 @@ class WicTestCase(OESelftestTestCase):
> self.skipTest('wic-tools cannot be built due its (intltool|gettext)-native dependency and NLS disable')
>
> bitbake('core-image-minimal')
> + bitbake('core-image-minimal-mtdutils')
> WicTestCase.image_is_ready = True
>
> rmtree(self.resultdir, ignore_errors=True)
> @@ -506,6 +513,90 @@ part /part2 --source rootfs --ondisk mmcblk0 --fstype=ext4 --include-path %s"""
> % (wks_file, self.resultdir), ignore_status=True).status)
> os.remove(wks_file)
>
> + def test_permissions(self):
> + """Test permissions are respected"""
> +
> + oldpath = os.environ['PATH']
> + os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
> +
> + t_normal = """
> +part / --source rootfs --fstype=ext4
> +"""
> + t_exclude = """
> +part / --source rootfs --fstype=ext4 --exclude-path=home
> +"""
> + t_multi = """
> +part / --source rootfs --ondisk sda --fstype=ext4
> +part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4
> +"""
> + t_change = """
> +part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/
> +part /etc --source rootfs --fstype=ext4 --change-directory=etc
> +"""
> + tests = [t_normal, t_exclude, t_multi, t_change]
> +
> + try:
> + for test in tests:
> + include_path = os.path.join(self.resultdir, 'test-include')
> + os.makedirs(include_path)
> + wks_file = os.path.join(include_path, 'temp.wks')
> + with open(wks_file, 'w') as wks:
> + wks.write(test)
> + runCmd("wic create %s -e core-image-minimal -o %s" \
> + % (wks_file, self.resultdir))
> +
> + for part in glob(os.path.join(self.resultdir, 'temp-*.direct.p*')):
> + res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part))
> + self.assertEqual(True, files_own_by_root(res.output))
> +
> + rmtree(self.resultdir, ignore_errors=True)
> +
> + finally:
> + os.environ['PATH'] = oldpath
> +
> + def test_change_directory(self):
> + """Test --change-directory wks option."""
> +
> + oldpath = os.environ['PATH']
> + os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
> +
> + try:
> + include_path = os.path.join(self.resultdir, 'test-include')
> + os.makedirs(include_path)
> + wks_file = os.path.join(include_path, 'temp.wks')
> + with open(wks_file, 'w') as wks:
> + rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
> + wks.write("part /etc --source rootfs --fstype=ext4 --change-directory=etc")
> + runCmd("wic create %s -e core-image-minimal -o %s" \
> + % (wks_file, self.resultdir))
> +
> + part1 = glob(os.path.join(self.resultdir, 'temp-*.direct.p1'))[0]
> +
> + res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
> + files = extract_files(res.output)
> + self.assertIn('shadow', files)
This fails if `shadow` isn't part of the image:
2020-04-18 20:57:31,625 - oe-selftest - INFO - test_change_directory (wic.Wic)
2020-04-18 20:57:42,105 - oe-selftest - INFO - ... FAIL
2020-04-18 20:57:42,105 - oe-selftest - INFO - Traceback (most recent call last):
File "/home/pbarker/Projects/Yocto/poky/meta/lib/oeqa/selftest/cases/wic.py", line 639, in test_change_directory
self.assertIn('shadow', files)
AssertionError: 'shadow' not found in [...]
> +
> + finally:
> + os.environ['PATH'] = oldpath
> +
> + def test_change_directory_errors(self):
> + """Test --change-directory wks option error handling."""
> + wks_file = 'temp.wks'
> +
> + # Absolute argument.
> + with open(wks_file, 'w') as wks:
> + wks.write("part / --source rootfs --fstype=ext4 --change-directory /usr")
> + self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
> + % (wks_file, self.resultdir), ignore_status=True).status)
> + os.remove(wks_file)
> +
> + # Argument pointing to parent directory.
> + with open(wks_file, 'w') as wks:
> + wks.write("part / --source rootfs --fstype=ext4 --change-directory ././..")
> + self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
> + % (wks_file, self.resultdir), ignore_status=True).status)
> + os.remove(wks_file)
> +
> class Wic2(WicTestCase):
>
> def test_bmap_short(self):
--
Paul Barker
Konsulko Group
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH v6 10/10] oeqa: wic: Add more tests for include_path
2020-04-14 13:36 ` [PATCH v6 10/10] oeqa: wic: Add more tests for include_path Ricardo Ribalda
@ 2020-04-18 20:10 ` Paul Barker
2020-04-18 20:16 ` [OE-core] " Ricardo Ribalda
[not found] ` <1607034A6B86A480.2683@lists.openembedded.org>
0 siblings, 2 replies; 17+ messages in thread
From: Paul Barker @ 2020-04-18 20:10 UTC (permalink / raw)
To: Ricardo Ribalda Delgado; +Cc: openembedded-core
On Tue, 14 Apr 2020 15:36:14 +0200
Ricardo Ribalda Delgado <ricardo@ribalda.com> wrote:
> Make sure permissions are respected.
>
> Add new test for orig/destination option.
>
> Cc: Paul Barker <pbarker@konsulko.com>
> Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
> ---
> meta/lib/oeqa/selftest/cases/wic.py | 64 ++++++++++++++++++++++++++++-
> 1 file changed, 63 insertions(+), 1 deletion(-)
>
> diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py
> index 8ef08c3d77..1036269b64 100644
> --- a/meta/lib/oeqa/selftest/cases/wic.py
> +++ b/meta/lib/oeqa/selftest/cases/wic.py
> @@ -486,15 +486,77 @@ part /part2 --source rootfs --ondisk mmcblk0 --fstype=ext4 --include-path %s"""
> res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
> files = extract_files(res.output)
> self.assertNotIn('test-file', files)
> + self.assertEqual(True, files_own_by_root(res.output))
>
> - # Test partition 2, should not contain 'test-file'
> + # Test partition 2, should contain 'test-file'
> res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part2))
> files = extract_files(res.output)
> self.assertIn('test-file', files)
> + self.assertEqual(True, files_own_by_root(res.output))
>
> finally:
> os.environ['PATH'] = oldpath
>
> + def test_include_path_embeded(self):
> + """Test --include-path wks option."""
> +
> + oldpath = os.environ['PATH']
> + os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
> +
> + try:
> + include_path = os.path.join(self.resultdir, 'test-include')
> + os.makedirs(include_path)
> + with open(os.path.join(include_path, 'test-file'), 'w') as t:
> + t.write("test\n")
> + wks_file = os.path.join(include_path, 'temp.wks')
> + with open(wks_file, 'w') as wks:
> + rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
> + wks.write("""
> +part / --source rootfs --fstype=ext4 --include-path %s --include-path core-image-minimal-mtdutils export/"""
> + % (include_path))
> + runCmd("wic create %s -e core-image-minimal -o %s" \
> + % (wks_file, self.resultdir))
> +
> + part1 = glob(os.path.join(self.resultdir, 'temp-*.direct.p1'))[0]
> +
> + res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
> + files = extract_files(res.output)
> + self.assertIn('test-file', files)
> + self.assertEqual(True, files_own_by_root(res.output))
> +
> + res = runCmd("debugfs -R 'ls -p /export/etc/' %s 2>/dev/null" % (part1))
> + files = extract_files(res.output)
> + self.assertIn('shadow', files)
This fails if `shadow` isn't part of the image:
2020-04-18 21:00:01,411 - oe-selftest - INFO - test_include_path_embeded (wic.Wic)
2020-04-18 21:00:14,102 - oe-selftest - INFO - ... FAIL
2020-04-18 21:00:14,103 - oe-selftest - INFO - Traceback (most recent call last):
File "/home/pbarker/Projects/Yocto/poky/meta/lib/oeqa/selftest/cases/wic.py", line 529, in test_include_path_embeded
self.assertIn('shadow', files)
AssertionError: 'shadow' not found in [...]
> + self.assertEqual(True, files_own_by_root(res.output))
> +
> + finally:
> + os.environ['PATH'] = oldpath
> +
> + def test_include_path_errors(self):
> + """Test --include-path wks option error handling."""
> + wks_file = 'temp.wks'
> +
> + # Absolute argument.
> + with open(wks_file, 'w') as wks:
> + wks.write("part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils /export")
> + self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
> + % (wks_file, self.resultdir), ignore_status=True).status)
> + os.remove(wks_file)
> +
> + # Argument pointing to parent directory.
> + with open(wks_file, 'w') as wks:
> + wks.write("part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils ././..")
> + self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
> + % (wks_file, self.resultdir), ignore_status=True).status)
> + os.remove(wks_file)
> +
> + # 3 Argument pointing to parent directory.
> + with open(wks_file, 'w') as wks:
> + wks.write("part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils export/ dummy")
> + self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
> + % (wks_file, self.resultdir), ignore_status=True).status)
> + os.remove(wks_file)
> +
> def test_exclude_path_errors(self):
> """Test --exclude-path wks option error handling."""
> wks_file = 'temp.wks'
--
Paul Barker
Konsulko Group
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [OE-core] [PATCH v6 10/10] oeqa: wic: Add more tests for include_path
2020-04-18 20:10 ` Paul Barker
@ 2020-04-18 20:16 ` Ricardo Ribalda
[not found] ` <1607034A6B86A480.2683@lists.openembedded.org>
1 sibling, 0 replies; 17+ messages in thread
From: Ricardo Ribalda @ 2020-04-18 20:16 UTC (permalink / raw)
To: Paul Barker; +Cc: openembedded-core
Hi Paul
On Sat, Apr 18, 2020 at 10:10 PM Paul Barker <pbarker@konsulko.com> wrote:
>
> On Tue, 14 Apr 2020 15:36:14 +0200
> Ricardo Ribalda Delgado <ricardo@ribalda.com> wrote:
>
> > Make sure permissions are respected.
> >
> > Add new test for orig/destination option.
> >
> > Cc: Paul Barker <pbarker@konsulko.com>
> > Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
> > ---
> > meta/lib/oeqa/selftest/cases/wic.py | 64 ++++++++++++++++++++++++++++-
> > 1 file changed, 63 insertions(+), 1 deletion(-)
> >
> > diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py
> > index 8ef08c3d77..1036269b64 100644
> > --- a/meta/lib/oeqa/selftest/cases/wic.py
> > +++ b/meta/lib/oeqa/selftest/cases/wic.py
> > @@ -486,15 +486,77 @@ part /part2 --source rootfs --ondisk mmcblk0 --fstype=ext4 --include-path %s"""
> > res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
> > files = extract_files(res.output)
> > self.assertNotIn('test-file', files)
> > + self.assertEqual(True, files_own_by_root(res.output))
> >
> > - # Test partition 2, should not contain 'test-file'
> > + # Test partition 2, should contain 'test-file'
> > res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part2))
> > files = extract_files(res.output)
> > self.assertIn('test-file', files)
> > + self.assertEqual(True, files_own_by_root(res.output))
> >
> > finally:
> > os.environ['PATH'] = oldpath
> >
> > + def test_include_path_embeded(self):
> > + """Test --include-path wks option."""
> > +
> > + oldpath = os.environ['PATH']
> > + os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
> > +
> > + try:
> > + include_path = os.path.join(self.resultdir, 'test-include')
> > + os.makedirs(include_path)
> > + with open(os.path.join(include_path, 'test-file'), 'w') as t:
> > + t.write("test\n")
> > + wks_file = os.path.join(include_path, 'temp.wks')
> > + with open(wks_file, 'w') as wks:
> > + rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
> > + wks.write("""
> > +part / --source rootfs --fstype=ext4 --include-path %s --include-path core-image-minimal-mtdutils export/"""
> > + % (include_path))
> > + runCmd("wic create %s -e core-image-minimal -o %s" \
> > + % (wks_file, self.resultdir))
> > +
> > + part1 = glob(os.path.join(self.resultdir, 'temp-*.direct.p1'))[0]
> > +
> > + res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
> > + files = extract_files(res.output)
> > + self.assertIn('test-file', files)
> > + self.assertEqual(True, files_own_by_root(res.output))
> > +
> > + res = runCmd("debugfs -R 'ls -p /export/etc/' %s 2>/dev/null" % (part1))
> > + files = extract_files(res.output)
> > + self.assertIn('shadow', files)
>
> This fails if `shadow` isn't part of the image:
>
> 2020-04-18 21:00:01,411 - oe-selftest - INFO - test_include_path_embeded (wic.Wic)
> 2020-04-18 21:00:14,102 - oe-selftest - INFO - ... FAIL
> 2020-04-18 21:00:14,103 - oe-selftest - INFO - Traceback (most recent call last):
> File "/home/pbarker/Projects/Yocto/poky/meta/lib/oeqa/selftest/cases/wic.py", line 529, in test_include_path_embeded
> self.assertIn('shadow', files)
> AssertionError: 'shadow' not found in [...]
I though I could assume that /etc/shadow was always part of
core-image-minimal and core-image-minimal-mtdutil. Shall I simply
change it to /etc/passwd?
How did you managed to trigger the error?
Thanks
>
> > + self.assertEqual(True, files_own_by_root(res.output))
> > +
> > + finally:
> > + os.environ['PATH'] = oldpath
> > +
> > + def test_include_path_errors(self):
> > + """Test --include-path wks option error handling."""
> > + wks_file = 'temp.wks'
> > +
> > + # Absolute argument.
> > + with open(wks_file, 'w') as wks:
> > + wks.write("part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils /export")
> > + self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
> > + % (wks_file, self.resultdir), ignore_status=True).status)
> > + os.remove(wks_file)
> > +
> > + # Argument pointing to parent directory.
> > + with open(wks_file, 'w') as wks:
> > + wks.write("part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils ././..")
> > + self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
> > + % (wks_file, self.resultdir), ignore_status=True).status)
> > + os.remove(wks_file)
> > +
> > + # 3 Argument pointing to parent directory.
> > + with open(wks_file, 'w') as wks:
> > + wks.write("part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils export/ dummy")
> > + self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
> > + % (wks_file, self.resultdir), ignore_status=True).status)
> > + os.remove(wks_file)
> > +
> > def test_exclude_path_errors(self):
> > """Test --exclude-path wks option error handling."""
> > wks_file = 'temp.wks'
>
>
>
> --
> Paul Barker
> Konsulko Group
>
--
Ricardo Ribalda
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [OE-core] [PATCH v6 10/10] oeqa: wic: Add more tests for include_path
[not found] ` <1607034A6B86A480.2683@lists.openembedded.org>
@ 2020-04-18 21:39 ` Ricardo Ribalda
0 siblings, 0 replies; 17+ messages in thread
From: Ricardo Ribalda @ 2020-04-18 21:39 UTC (permalink / raw)
To: Ricardo Ribalda Delgado; +Cc: Paul Barker, openembedded-core
Hi again
On Sat, Apr 18, 2020 at 10:16 PM Ricardo Ribalda via
lists.openembedded.org
<ricardo.ribalda=gmail.com@lists.openembedded.org> wrote:
>
> Hi Paul
>
>
>
> On Sat, Apr 18, 2020 at 10:10 PM Paul Barker <pbarker@konsulko.com> wrote:
> >
> > On Tue, 14 Apr 2020 15:36:14 +0200
> > Ricardo Ribalda Delgado <ricardo@ribalda.com> wrote:
> >
> > > Make sure permissions are respected.
> > >
> > > Add new test for orig/destination option.
> > >
> > > Cc: Paul Barker <pbarker@konsulko.com>
> > > Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
> > > ---
> > > meta/lib/oeqa/selftest/cases/wic.py | 64 ++++++++++++++++++++++++++++-
> > > 1 file changed, 63 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py
> > > index 8ef08c3d77..1036269b64 100644
> > > --- a/meta/lib/oeqa/selftest/cases/wic.py
> > > +++ b/meta/lib/oeqa/selftest/cases/wic.py
> > > @@ -486,15 +486,77 @@ part /part2 --source rootfs --ondisk mmcblk0 --fstype=ext4 --include-path %s"""
> > > res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
> > > files = extract_files(res.output)
> > > self.assertNotIn('test-file', files)
> > > + self.assertEqual(True, files_own_by_root(res.output))
> > >
> > > - # Test partition 2, should not contain 'test-file'
> > > + # Test partition 2, should contain 'test-file'
> > > res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part2))
> > > files = extract_files(res.output)
> > > self.assertIn('test-file', files)
> > > + self.assertEqual(True, files_own_by_root(res.output))
> > >
> > > finally:
> > > os.environ['PATH'] = oldpath
> > >
> > > + def test_include_path_embeded(self):
> > > + """Test --include-path wks option."""
> > > +
> > > + oldpath = os.environ['PATH']
> > > + os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
> > > +
> > > + try:
> > > + include_path = os.path.join(self.resultdir, 'test-include')
> > > + os.makedirs(include_path)
> > > + with open(os.path.join(include_path, 'test-file'), 'w') as t:
> > > + t.write("test\n")
> > > + wks_file = os.path.join(include_path, 'temp.wks')
> > > + with open(wks_file, 'w') as wks:
> > > + rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
> > > + wks.write("""
> > > +part / --source rootfs --fstype=ext4 --include-path %s --include-path core-image-minimal-mtdutils export/"""
> > > + % (include_path))
> > > + runCmd("wic create %s -e core-image-minimal -o %s" \
> > > + % (wks_file, self.resultdir))
> > > +
> > > + part1 = glob(os.path.join(self.resultdir, 'temp-*.direct.p1'))[0]
> > > +
> > > + res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
> > > + files = extract_files(res.output)
> > > + self.assertIn('test-file', files)
> > > + self.assertEqual(True, files_own_by_root(res.output))
> > > +
> > > + res = runCmd("debugfs -R 'ls -p /export/etc/' %s 2>/dev/null" % (part1))
> > > + files = extract_files(res.output)
> > > + self.assertIn('shadow', files)
> >
> > This fails if `shadow` isn't part of the image:
> >
> > 2020-04-18 21:00:01,411 - oe-selftest - INFO - test_include_path_embeded (wic.Wic)
> > 2020-04-18 21:00:14,102 - oe-selftest - INFO - ... FAIL
> > 2020-04-18 21:00:14,103 - oe-selftest - INFO - Traceback (most recent call last):
> > File "/home/pbarker/Projects/Yocto/poky/meta/lib/oeqa/selftest/cases/wic.py", line 529, in test_include_path_embeded
> > self.assertIn('shadow', files)
> > AssertionError: 'shadow' not found in [...]
>
> I though I could assume that /etc/shadow was always part of
> core-image-minimal and core-image-minimal-mtdutil. Shall I simply
> change it to /etc/passwd?
> How did you managed to trigger the error?
>
> Thanks
>
Nevermind, I managed to replicate it wit the current master.
Thanks for catching it up. will fix all and resend.
> >
> > > + self.assertEqual(True, files_own_by_root(res.output))
> > > +
> > > + finally:
> > > + os.environ['PATH'] = oldpath
> > > +
> > > + def test_include_path_errors(self):
> > > + """Test --include-path wks option error handling."""
> > > + wks_file = 'temp.wks'
> > > +
> > > + # Absolute argument.
> > > + with open(wks_file, 'w') as wks:
> > > + wks.write("part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils /export")
> > > + self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
> > > + % (wks_file, self.resultdir), ignore_status=True).status)
> > > + os.remove(wks_file)
> > > +
> > > + # Argument pointing to parent directory.
> > > + with open(wks_file, 'w') as wks:
> > > + wks.write("part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils ././..")
> > > + self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
> > > + % (wks_file, self.resultdir), ignore_status=True).status)
> > > + os.remove(wks_file)
> > > +
> > > + # 3 Argument pointing to parent directory.
> > > + with open(wks_file, 'w') as wks:
> > > + wks.write("part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils export/ dummy")
> > > + self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
> > > + % (wks_file, self.resultdir), ignore_status=True).status)
> > > + os.remove(wks_file)
> > > +
> > > def test_exclude_path_errors(self):
> > > """Test --exclude-path wks option error handling."""
> > > wks_file = 'temp.wks'
> >
> >
> >
> > --
> > Paul Barker
> > Konsulko Group
> >
>
>
>
> --
> Ricardo Ribalda
>
--
Ricardo Ribalda
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2020-04-18 21:39 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-04-14 13:36 [PATCH v6 00/10] Fix permissions and embed-rotofs Ricardo Ribalda
2020-04-14 13:36 ` [PATCH v6 01/10] wic: Fix permissions when using exclude or include path Ricardo Ribalda
2020-04-18 19:03 ` Paul Barker
2020-04-14 13:36 ` [PATCH v6 02/10] wic: Fix multi images .wks with bitbake Ricardo Ribalda
2020-04-14 13:36 ` [PATCH v6 03/10] wic: Add --change-directory argument Ricardo Ribalda
2020-04-14 13:36 ` [PATCH v6 04/10] wic: Continue if excluded_path does not exist Ricardo Ribalda
2020-04-14 13:36 ` [PATCH v6 05/10] wic: Avoid creating invalid pseudo directory Ricardo Ribalda
2020-04-14 13:36 ` [PATCH v6 06/10] oeqa: wic: Add tests for permissions and change-directory Ricardo Ribalda
2020-04-18 20:07 ` Paul Barker
2020-04-14 13:36 ` [PATCH v6 07/10] wic: misc: Do not find for executables in ALREADY_PROVIDED Ricardo Ribalda
2020-04-14 13:36 ` [PATCH v6 08/10] wic: root: Add an opt. destination on include-path Ricardo Ribalda
2020-04-18 19:02 ` Paul Barker
2020-04-14 13:36 ` [PATCH v6 09/10] wic: rootfs: Combine path_validation in one function Ricardo Ribalda
2020-04-14 13:36 ` [PATCH v6 10/10] oeqa: wic: Add more tests for include_path Ricardo Ribalda
2020-04-18 20:10 ` Paul Barker
2020-04-18 20:16 ` [OE-core] " Ricardo Ribalda
[not found] ` <1607034A6B86A480.2683@lists.openembedded.org>
2020-04-18 21:39 ` Ricardo Ribalda
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.