[PATCH 0/9] wic: Fix fstype handling

[PATCH 0/9] wic: Fix fstype handling

[Ed Bartosh]

Hi,

This patchset adds support for msdos(FAT16) fstype (#11137) and fixes number of issues
in handling fstypes in wic code:
   - restricted supported fstypes in wks parser code
   - fixed the code that creates partitions of certain fstypes
   - removed support of empty squashfs partitions

The following changes since commit 6bbf3176fe227712a478f5b4b1fdcf9417a89568:

  oe-selftest: add test_image_bootpart_globbed test for wic (2017-03-30 13:27:19 +0300)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/wic/wip
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wic/wip

Ed Bartosh (9):
  oe-selftest: fix incorrect fstype
  wic: allow only supported fstypes
  wic: don't silently skip unknown fstypes
  wic: support 'msdos' fstype
  wic: set FAT 16 for msdos partitions
  wic: set correct system id for msdos partitions
  wic: fix list of supported fstypes in help content
  oe-selftest: test creation of msdos partition
  wic: remove prepare_empty_partition_squashfs

 meta/lib/oeqa/selftest/wic.py            |  4 +-
 scripts/lib/wic/help.py                  |  2 +
 scripts/lib/wic/ksparser.py              |  4 +-
 scripts/lib/wic/partition.py             | 99 +++++++++++++-------------------
 scripts/lib/wic/plugins/imager/direct.py | 15 +----
 5 files changed, 48 insertions(+), 76 deletions(-)

--
Regards,
Ed

[PATCH 1/9] oe-selftest: fix incorrect fstype

[Ed Bartosh]

Fixed typo in wks content: squash->squashfs

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index 2401aaa..ff13eb8 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -729,7 +729,7 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --r
                             'part emptyvfat   --fstype vfat   --size 1M\n',
                             'part emptyext2   --fstype ext2   --size 1M\n',
                             'part emptybtrfs  --fstype btrfs  --size 100M\n',
-                            'part emptysquash --fstype squash --size 1M\n'])
+                            'part emptysquash --fstype squashfs --size 1M\n'])
             wks.flush()
             cmd = "wic create %s -e %s -o %s" % (wks.name, img, self.resultdir)
             self.assertEqual(0, runCmd(cmd).status)
-- 
2.1.4

[PATCH 2/9] wic: allow only supported fstypes

[Ed Bartosh]

Restricted possible values of --fstype to the list of
supported types. This should catch incorrect values
when .wks file is being parsed.

Removed checks for empty fstype and mentioning of
unsupported fstype 'ontrackdm6aux3'.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/ksparser.py              | 4 +++-
 scripts/lib/wic/partition.py             | 8 ++------
 scripts/lib/wic/plugins/imager/direct.py | 2 --
 3 files changed, 5 insertions(+), 9 deletions(-)

diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index a039300..d026caa 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -136,7 +136,9 @@ class KickStart():
         part.add_argument('--exclude-path', nargs='+')
         part.add_argument("--extra-space", type=sizetype)
         part.add_argument('--fsoptions', dest='fsopts')
-        part.add_argument('--fstype')
+        part.add_argument('--fstype', default='vfat',
+                          choices=('ext2', 'ext3', 'ext4', 'btrfs',
+                                   'squashfs', 'vfat', 'msdos', 'swap'))
         part.add_argument('--label')
         part.add_argument('--no-table', action='store_true')
         part.add_argument('--ondisk', '--ondrive', dest='disk', default='sda')
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 647a6fb..f0e88fb 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -136,11 +136,11 @@ class Partition():
                                "specify a non-zero --size/--fixed-size for that "
                                "partition." % self.mountpoint)
 
-            if self.fstype and self.fstype == "swap":
+            if self.fstype == "swap":
                 self.prepare_swap_partition(cr_workdir, oe_builddir,
                                             native_sysroot)
                 self.source_file = "%s/fs.%s" % (cr_workdir, self.fstype)
-            elif self.fstype:
+            else:
                 rootfs = "%s/fs_%s.%s.%s" % (cr_workdir, self.label,
                                              self.lineno, self.fstype)
                 if os.path.isfile(rootfs):
@@ -217,10 +217,6 @@ class Partition():
         if os.path.isfile(rootfs):
             os.remove(rootfs)
 
-        if not self.fstype:
-            raise WicError("File system for partition %s not specified in "
-                           "kickstart, use --fstype option" % self.mountpoint)
-
         # Get rootfs size from bitbake variable if it's not set in .ks file
         if not self.size:
             # Bitbake variable ROOTFS_SIZE is calculated in
diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
index 67fd183..79b948a 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -487,8 +487,6 @@ class PartitionedImage():
                 parted_fs_type = "fat32"
             elif part.fstype == "msdos":
                 parted_fs_type = "fat16"
-            elif part.fstype == "ontrackdm6aux3":
-                parted_fs_type = "ontrackdm6aux3"
             else:
                 # Type for ext2/ext3/ext4/btrfs
                 parted_fs_type = "ext2"
-- 
2.1.4

[PATCH 3/9] wic: don't silently skip unknown fstypes

[Ed Bartosh]

Fixed wic code that loops through hard-coded list of known fstypes
to find prepare_rootfs_<fstype> or prepare_empty_partition_<fstype>
methods and silently skipping unknown fstypes.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/partition.py | 32 +++++++++++++-------------------
 1 file changed, 13 insertions(+), 19 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index f0e88fb..28ad3e6 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -145,13 +145,11 @@ class Partition():
                                              self.lineno, self.fstype)
                 if os.path.isfile(rootfs):
                     os.remove(rootfs)
-                for prefix in ("ext", "btrfs", "vfat", "squashfs"):
-                    if self.fstype.startswith(prefix):
-                        method = getattr(self,
-                                         "prepare_empty_partition_" + prefix)
-                        method(rootfs, oe_builddir, native_sysroot)
-                        self.source_file = rootfs
-                        break
+
+                prefix = "ext" if self.fstype.startswith("ext") else self.fstype
+                method = getattr(self, "prepare_empty_partition_" + prefix)
+                method(rootfs, oe_builddir, native_sysroot)
+                self.source_file = rootfs
             return
 
         plugins = PluginMgr.get_plugins('source')
@@ -231,19 +229,15 @@ class Partition():
                                '--overhead-factor will be applied')
                 self.size = int(round(float(rsize_bb)))
 
-        for prefix in ("ext", "btrfs", "vfat", "squashfs"):
-            if self.fstype.startswith(prefix):
-                method = getattr(self, "prepare_rootfs_" + prefix)
-                method(rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo)
-
-                self.source_file = rootfs
+        prefix = "ext" if self.fstype.startswith("ext") else self.fstype
+        method = getattr(self, "prepare_rootfs_" + prefix)
+        method(rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo)
+        self.source_file = rootfs
 
-                # get the rootfs size in the right units for kickstart (kB)
-                du_cmd = "du -Lbks %s" % rootfs
-                out = exec_cmd(du_cmd)
-                self.size = int(out.split()[0])
-
-                break
+        # get the rootfs size in the right units for kickstart (kB)
+        du_cmd = "du -Lbks %s" % rootfs
+        out = exec_cmd(du_cmd)
+        self.size = int(out.split()[0])
 
     def prepare_rootfs_ext(self, rootfs, oe_builddir, rootfs_dir,
                            native_sysroot, pseudo):
-- 
2.1.4

[PATCH 4/9] wic: support 'msdos' fstype

[Ed Bartosh]

Added prepare_empty_partition_msdos and prepare_rootfs_msdos
methods to support 'msdos' filesystem type.

Created aliases prepare_empty_partition_vfat and prepare_rootfs_vfat
to continue supporting creation of vfat patitiions.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/partition.py | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 28ad3e6..d59351c 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -290,10 +290,10 @@ class Partition():
             (self.fstype, rootfs_size * 1024, rootfs_dir, label_str, rootfs)
         exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
 
-    def prepare_rootfs_vfat(self, rootfs, oe_builddir, rootfs_dir,
-                            native_sysroot, pseudo):
+    def prepare_rootfs_msdos(self, rootfs, oe_builddir, rootfs_dir,
+                             native_sysroot, pseudo):
         """
-        Prepare content for a vfat rootfs partition.
+        Prepare content for a msdos/vfat rootfs partition.
         """
         du_cmd = "du -bks %s" % rootfs_dir
         out = exec_cmd(du_cmd)
@@ -314,6 +314,8 @@ class Partition():
         chmod_cmd = "chmod 644 %s" % rootfs
         exec_cmd(chmod_cmd)
 
+    prepare_rootfs_vfat = prepare_rootfs_msdos
+
     def prepare_rootfs_squashfs(self, rootfs, oe_builddir, rootfs_dir,
                                 native_sysroot, pseudo):
         """
@@ -359,8 +361,8 @@ class Partition():
             (self.fstype, self.size * 1024, label_str, rootfs)
         exec_native_cmd(mkfs_cmd, native_sysroot)
 
-    def prepare_empty_partition_vfat(self, rootfs, oe_builddir,
-                                     native_sysroot):
+    def prepare_empty_partition_msdos(self, rootfs, oe_builddir,
+                                      native_sysroot):
         """
         Prepare an empty vfat partition.
         """
@@ -376,6 +378,8 @@ class Partition():
         chmod_cmd = "chmod 644 %s" % rootfs
         exec_cmd(chmod_cmd)
 
+    prepare_empty_partition_vfat = prepare_empty_partition_msdos
+
     def prepare_empty_partition_squashfs(self, cr_workdir, oe_builddir,
                                          native_sysroot):
         """
-- 
2.1.4

[PATCH 5/9] wic: set FAT 16 for msdos partitions

[Ed Bartosh]

Used '-F 16' parameter for mkdosfs to create FAT16 partitions for
'msdos' partition type.

[YOCTO #11137]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/partition.py | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index d59351c..6f324ad 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -305,7 +305,12 @@ class Partition():
         if self.label:
             label_str = "-n %s" % self.label
 
-        dosfs_cmd = "mkdosfs %s -S 512 -C %s %d" % (label_str, rootfs, rootfs_size)
+        size_str = ""
+        if self.fstype == 'msdos':
+            size_str = "-F 16" # FAT 16
+
+        dosfs_cmd = "mkdosfs %s -S 512 %s -C %s %d" % (label_str, size_str,
+                                                       rootfs, rootfs_size)
         exec_native_cmd(dosfs_cmd, native_sysroot)
 
         mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir)
@@ -372,7 +377,12 @@ class Partition():
         if self.label:
             label_str = "-n %s" % self.label
 
-        dosfs_cmd = "mkdosfs %s -S 512 -C %s %d" % (label_str, rootfs, blocks)
+        size_str = ""
+        if self.fstype == 'msdos':
+            size_str = "-F 16" # FAT 16
+
+        dosfs_cmd = "mkdosfs %s -S 512 %s -C %s %d" % (label_str, size_str,
+                                                       rootfs, blocks)
         exec_native_cmd(dosfs_cmd, native_sysroot)
 
         chmod_cmd = "chmod 644 %s" % rootfs
-- 
2.1.4

[PATCH 6/9] wic: set correct system id for msdos partitions

[Ed Bartosh]

Explicitly set system id 0x6(FAT16) for msdos partitions.

Removed old code that attempts to achieve the same result
using 'parted ... lba off'.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/plugins/imager/direct.py | 13 ++-----------
 1 file changed, 2 insertions(+), 11 deletions(-)

diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
index 79b948a..f2e6127 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -487,6 +487,8 @@ class PartitionedImage():
                 parted_fs_type = "fat32"
             elif part.fstype == "msdos":
                 parted_fs_type = "fat16"
+                if not part.system_id:
+                    part.system_id = '0x6' # FAT16
             else:
                 # Type for ext2/ext3/ext4/btrfs
                 parted_fs_type = "ext2"
@@ -536,17 +538,6 @@ class PartitionedImage():
                                 (self.path, part.num, part.system_id),
                                 self.native_sysroot)
 
-            # Parted defaults to enabling the lba flag for fat16 partitions,
-            # which causes compatibility issues with some firmware (and really
-            # isn't necessary).
-            if parted_fs_type == "fat16":
-                if self.ptable_format == 'msdos':
-                    logger.debug("Disable 'lba' flag for partition '%s' on disk '%s'",
-                                 part.num, self.path)
-                    exec_native_cmd("parted -s %s set %d lba off" % \
-                                    (self.path, part.num),
-                                    self.native_sysroot)
-
     def cleanup(self):
         # remove partition images
         for image in set(self.partimages):
-- 
2.1.4

[PATCH 7/9] wic: fix list of supported fstypes in help content

[Ed Bartosh]

Added vfat and msdos to the list of supported fstypes in
'wic help kickstart' output.

[YOCTO #11137]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/help.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 148da89..aee2451 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -687,6 +687,8 @@ DESCRIPTION
            apply to partitions created using '--source rootfs' (see
            --source above).  Valid values are:
 
+             vfat
+             msdos
              ext2
              ext3
              ext4
-- 
2.1.4

[PATCH 8/9] oe-selftest: test creation of msdos partition

[Ed Bartosh]

Added msdos partition to the .wks file in test_fs_types
wic test case.

[YOCTO #11137]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index ff13eb8..c160f5f 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -727,6 +727,7 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --r
                             'part squash --fstype squashfs --source rootfs\n',
                             'part swap   --fstype swap --size 1M\n',
                             'part emptyvfat   --fstype vfat   --size 1M\n',
+                            'part emptymsdos  --fstype msdos  --size 1M\n',
                             'part emptyext2   --fstype ext2   --size 1M\n',
                             'part emptybtrfs  --fstype btrfs  --size 100M\n',
                             'part emptysquash --fstype squashfs --size 1M\n'])
-- 
2.1.4

[PATCH 9/9] wic: remove prepare_empty_partition_squashfs

[Ed Bartosh]

There is not much sense in creation of empty squashfs
partition. It's also not possible to create empty squashfs
partition of specified size.

Even more, prepare_empty_partition_squashfs method is
absolutely broken. It raises exception when called and
even its signature differs from the rest of of similar
methods. It means that nobody uses it and it's safe
to remove it.

Removed prepare_empty_partition_squashfs method and
testing of empty squashfs partition.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py |  3 +--
 scripts/lib/wic/partition.py  | 33 ++++-----------------------------
 2 files changed, 5 insertions(+), 31 deletions(-)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index c160f5f..df5e060 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -729,8 +729,7 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --r
                             'part emptyvfat   --fstype vfat   --size 1M\n',
                             'part emptymsdos  --fstype msdos  --size 1M\n',
                             'part emptyext2   --fstype ext2   --size 1M\n',
-                            'part emptybtrfs  --fstype btrfs  --size 100M\n',
-                            'part emptysquash --fstype squashfs --size 1M\n'])
+                            'part emptybtrfs  --fstype btrfs  --size 100M\n'])
             wks.flush()
             cmd = "wic create %s -e %s -o %s" % (wks.name, img, self.resultdir)
             self.assertEqual(0, runCmd(cmd).status)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 6f324ad..5edb340 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -141,6 +141,10 @@ class Partition():
                                             native_sysroot)
                 self.source_file = "%s/fs.%s" % (cr_workdir, self.fstype)
             else:
+                if self.fstype == 'squashfs':
+                    raise WicError("It's not possible to create empty squashfs "
+                                   "partition '%s'" % (self.mountpoint))
+
                 rootfs = "%s/fs_%s.%s.%s" % (cr_workdir, self.label,
                                              self.lineno, self.fstype)
                 if os.path.isfile(rootfs):
@@ -390,35 +394,6 @@ class Partition():
 
     prepare_empty_partition_vfat = prepare_empty_partition_msdos
 
-    def prepare_empty_partition_squashfs(self, cr_workdir, oe_builddir,
-                                         native_sysroot):
-        """
-        Prepare an empty squashfs partition.
-        """
-        logger.warning("Creating of an empty squashfs %s partition was attempted. "
-                       "Proceeding as requested.", self.mountpoint)
-
-        path = "%s/fs_%s.%s" % (cr_workdir, self.label, self.fstype)
-        if os.path.isfile(path):
-            os.remove(path)
-
-        # it is not possible to create a squashfs without source data,
-        # thus prepare an empty temp dir that is used as source
-        tmpdir = tempfile.mkdtemp()
-
-        squashfs_cmd = "mksquashfs %s %s -noappend" % \
-                       (tmpdir, path)
-        exec_native_cmd(squashfs_cmd, native_sysroot)
-
-        os.rmdir(tmpdir)
-
-        # get the rootfs size in the right units for kickstart (kB)
-        du_cmd = "du -Lbks %s" % path
-        out = exec_cmd(du_cmd)
-        fs_size = out.split()[0]
-
-        self.size = int(fs_size)
-
     def prepare_swap_partition(self, cr_workdir, oe_builddir, native_sysroot):
         """
         Prepare a swap partition.
-- 
2.1.4

✗ patchtest: failure for wic: Fix fstype handling

[Patchwork]

== Series Details ==

Series: wic: Fix fstype handling
Revision: 1
URL   : https://patchwork.openembedded.org/series/6091/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at 6d6d0737a2)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe