[wic][PATCH 0/4] use PARTUUID for MSDOS partitions

[wic][PATCH 0/4] use PARTUUID for MSDOS partitions

[Ed Bartosh]

Hi,

This is an implementation of PARTUUID support for MSDOS partitions.
Now wic can use 'root=PARTUUID=<uuid> for both types of supported
partition tables - GPT and MSDOS.

This implementation enables usage of mkefidisk as a default wks for genericx86*
machines as most of reference BSPs support EFI.

The following changes since commit f12825d4bf7091576856650123c7d7614b9089af:

  scripts: Rename 'native' to 'oe-run-native' (2016-10-06 07:53:51 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/wic/msdos-partuuid
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wic/msdos-partuuid

Ed Bartosh (4):
  wic: set PARTUUID only for gpt partition table
  wic: generate PARTUUID for MDOS partitions
  wic: rewrite MBR disk identifier
  mkefidisk.wks: use MSDOS partition table

 scripts/lib/wic/canned-wks/mkefidisk.wks |  2 +-
 scripts/lib/wic/imager/direct.py         | 16 ++++++++++++++--
 scripts/lib/wic/partition.py             |  3 ---
 scripts/lib/wic/utils/partitionedfs.py   | 14 +++++++++++---
 4 files changed, 26 insertions(+), 9 deletions(-)

--
Regards,
Ed

[wic][PATCH 1/4] wic: set PARTUUID only for gpt partition table

[Ed Bartosh]

sgdisk fails to set PARTUUID for msdos partitions as it's only
supported for GPT partitions.

Checked partition table format to run sgdisk --partition-guid
only for GPT partitions.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/utils/partitionedfs.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/wic/utils/partitionedfs.py b/scripts/lib/wic/utils/partitionedfs.py
index cafb933..8d93a68 100644
--- a/scripts/lib/wic/utils/partitionedfs.py
+++ b/scripts/lib/wic/utils/partitionedfs.py
@@ -299,7 +299,7 @@ class Image():
                                          (part['num'], part['part_type'],
                                           disk['disk'].device), self.native_sysroot)
 
-            if part['uuid']:
+            if part['uuid'] and disk['ptable_format'] == "gpt":
                 msger.debug("partition %d: set UUID to %s" % \
                             (part['num'], part['uuid']))
                 exec_native_cmd("sgdisk --partition-guid=%d:%s %s" % \
-- 
2.1.4

[wic][PATCH 2/4] wic: generate PARTUUID for MDOS partitions

[Ed Bartosh]

Added generation of partition UUIDs for MSDOS partitions.
UUID for MSDOS partitions is <disk identifier>-<partition number>,
where disk identifier is a random 4 bytes long number. It's usually
generated when MBR/partition table is initialized.

As UUID is used to point to the root partition in bootloader config
we need to generate it before the MBR is initialized.
After MBR is created we need to rewrite system identifier to match
it with what is used in bootloader config. This will be implemented
in the next commit.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/imager/direct.py | 14 +++++++++++++-
 scripts/lib/wic/partition.py     |  3 ---
 2 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/wic/imager/direct.py b/scripts/lib/wic/imager/direct.py
index ffde232..123a078 100644
--- a/scripts/lib/wic/imager/direct.py
+++ b/scripts/lib/wic/imager/direct.py
@@ -26,6 +26,7 @@
 
 import os
 import shutil
+import uuid
 
 from wic import msger
 from wic.utils.oe.misc import get_bitbake_var
@@ -242,12 +243,23 @@ class DirectImageCreator(BaseImageCreator):
 
         self.__image = Image(self.native_sysroot)
 
-        for part in parts:
+        disk_ids = {}
+        for num, part in enumerate(parts, 1):
             # as a convenience, set source to the boot partition source
             # instead of forcing it to be set via bootloader --source
             if not self.ks.bootloader.source and part.mountpoint == "/boot":
                 self.ks.bootloader.source = part.source
 
+            # generate parition UUIDs
+            if not part.uuid and part.use_uuid:
+                if self.ptable_format == 'gpt':
+                    part.uuid = str(uuid.uuid4())
+                else: # msdos partition table
+                    if part.disk not in disk_ids:
+                        disk_ids[part.disk] = int.from_bytes(os.urandom(4), 'little')
+                    disk_id = disk_ids[part.disk]
+                    part.uuid = '%0x-%02d' % (disk_id, self.__get_part_num(num, parts))
+
         fstab_path = self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
 
         shutil.rmtree(self.workdir)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 30cd320..90f65a1 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -26,7 +26,6 @@
 
 import os
 import tempfile
-import uuid
 
 from wic.utils.oe.misc import msger, parse_sourceparams
 from wic.utils.oe.misc import exec_cmd, exec_native_cmd
@@ -60,8 +59,6 @@ class Partition():
         self.system_id = args.system_id
         self.use_uuid = args.use_uuid
         self.uuid = args.uuid
-        if args.use_uuid and not self.uuid:
-            self.uuid = str(uuid.uuid4())
 
         self.lineno = lineno
         self.source_file = ""
-- 
2.1.4

[wic][PATCH 3/4] wic: rewrite MBR disk identifier

[Ed Bartosh]

Disk identifier created by parted doesn't match the one we generated
and used in bootloader config. We need to rewrite it to make our image
bootable.

Modified involved API and data structures to access previously
generated disk identifiers after MBR is initialized. Written disk
identifiers to MBR.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/imager/direct.py       |  2 +-
 scripts/lib/wic/utils/partitionedfs.py | 12 ++++++++++--
 2 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/wic/imager/direct.py b/scripts/lib/wic/imager/direct.py
index 123a078..edf5e5d 100644
--- a/scripts/lib/wic/imager/direct.py
+++ b/scripts/lib/wic/imager/direct.py
@@ -315,7 +315,7 @@ class DirectImageCreator(BaseImageCreator):
                         % (disk_name, full_path, disk['min_size']))
             disk_obj = DiskImage(full_path, disk['min_size'])
             self.__disks[disk_name] = disk_obj
-            self.__image.add_disk(disk_name, disk_obj)
+            self.__image.add_disk(disk_name, disk_obj, disk_ids.get(disk_name))
 
         self.__image.create()
 
diff --git a/scripts/lib/wic/utils/partitionedfs.py b/scripts/lib/wic/utils/partitionedfs.py
index 8d93a68..cb03009 100644
--- a/scripts/lib/wic/utils/partitionedfs.py
+++ b/scripts/lib/wic/utils/partitionedfs.py
@@ -68,15 +68,17 @@ class Image():
                  'offset': 0,      # Offset of next partition (in sectors)
                  # Minimum required disk size to fit all partitions (in bytes)
                  'min_size': 0,
-                 'ptable_format': "msdos"} # Partition table format
+                 'ptable_format': "msdos", # Partition table format
+                 'identifier': None} # Disk system identifier
 
-    def add_disk(self, disk_name, disk_obj):
+    def add_disk(self, disk_name, disk_obj, identifier):
         """ Add a disk object which have to be partitioned. More than one disk
         can be added. In case of multiple disks, disk partitions have to be
         added for each disk separately with 'add_partition()". """
 
         self.__add_disk(disk_name)
         self.disks[disk_name]['disk'] = disk_obj
+        self.disks[disk_name]['identifier'] = identifier
 
     def __add_partition(self, part):
         """ This is a helper function for 'add_partition()' which adds a
@@ -245,6 +247,12 @@ class Image():
                             (disk['disk'].device, disk['ptable_format']),
                             self.native_sysroot)
 
+            if disk['identifier']:
+                msger.debug("Set disk identifier %x" % disk['identifier'])
+                with open(disk['disk'].device, 'r+b') as img:
+                    img.seek(0x1B8)
+                    img.write(disk['identifier'].to_bytes(4, 'little'))
+
         msger.debug("Creating partitions")
 
         for part in self.partitions:
-- 
2.1.4

[wic][PATCH 4/4] mkefidisk.wks: use MSDOS partition table

[Ed Bartosh]

Stopped using GPT partition table in mkefidisk.wks as it's not
supported by all reference hardware.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/canned-wks/mkefidisk.wks | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/wic/canned-wks/mkefidisk.wks b/scripts/lib/wic/canned-wks/mkefidisk.wks
index 73886ab..668ca92 100644
--- a/scripts/lib/wic/canned-wks/mkefidisk.wks
+++ b/scripts/lib/wic/canned-wks/mkefidisk.wks
@@ -8,4 +8,4 @@ part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024
 
 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
 
-bootloader --ptable gpt --timeout=10 --append="rootwait rootfstype=ext4 console=ttyPCH0,115200 console=tty0 vmalloc=256MB snd-hda-intel.enable_msi=0"
+bootloader --timeout=10 --append="rootwait rootfstype=ext4 console=ttyPCH0,115200 console=tty0 vmalloc=256MB snd-hda-intel.enable_msi=0"
-- 
2.1.4

Re: [wic][PATCH 0/4] use PARTUUID for MSDOS partitions

[Saul Wold]

On Thu, 2016-10-06 at 13:33 +0300, Ed Bartosh wrote:
> Hi,
> 
> This is an implementation of PARTUUID support for MSDOS partitions.
> Now wic can use 'root=PARTUUID=<uuid> for both types of supported
> partition tables - GPT and MSDOS.
> 
> This implementation enables usage of mkefidisk as a default wks for
> genericx86*
> machines as most of reference BSPs support EFI.
> 
> The following changes since commit
> f12825d4bf7091576856650123c7d7614b9089af:
> 
>   scripts: Rename 'native' to 'oe-run-native' (2016-10-06 07:53:51
> +0100)
> 
> are available in the git repository at:
> 
>   git://git.yoctoproject.org/poky-contrib ed/wic/msdos-partuuid
>   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wic/msd
> os-partuuid
> 
> Ed Bartosh (4):
>   wic: set PARTUUID only for gpt partition table
>   wic: generate PARTUUID for MDOS partitions
>   wic: rewrite MBR disk identifier
>   mkefidisk.wks: use MSDOS partition table
> 
>  scripts/lib/wic/canned-wks/mkefidisk.wks |  2 +-
We need to change the console setting to ttyS0 by default instead of
the PCH0, which is from the FRI2 hardware, but this should not prevent
this patch from getting merged.

>  scripts/lib/wic/imager/direct.py         | 16 ++++++++++++++--
>  scripts/lib/wic/partition.py             |  3 ---
>  scripts/lib/wic/utils/partitionedfs.py   | 14 +++++++++++---
>  4 files changed, 26 insertions(+), 9 deletions(-)
> 
Reviewed-by: Saul Wold <sgw@linux.intel.com>
Tested-by: Saul Wold <sgw@linux.intel.com>
> --
> Regards,
> Ed

[wic][PATCH] mkefidisk.wks: update kernel command line

[Ed Bartosh]

Used ttyS0 console.
Removed usage of ttyPCH0 (FRI2 leftover)
Decreased bootloader timeout to 5 seconds
Removed 'vmalloc=256MB snd-hda-intel.enable_msi=0' as it's not needed
for any of reference BSPs.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/canned-wks/mkefidisk.wks | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/wic/canned-wks/mkefidisk.wks b/scripts/lib/wic/canned-wks/mkefidisk.wks
index 668ca92..7c287f7 100644
--- a/scripts/lib/wic/canned-wks/mkefidisk.wks
+++ b/scripts/lib/wic/canned-wks/mkefidisk.wks
@@ -8,4 +8,4 @@ part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024
 
 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
 
-bootloader --timeout=10 --append="rootwait rootfstype=ext4 console=ttyPCH0,115200 console=tty0 vmalloc=256MB snd-hda-intel.enable_msi=0"
+bootloader --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
-- 
2.1.4

Re: [wic][PATCH] mkefidisk.wks: update kernel command line

[Saul Wold]

On Thu, 2016-10-06 at 18:57 +0300, Ed Bartosh wrote:
> Used ttyS0 console.
> Removed usage of ttyPCH0 (FRI2 leftover)
> Decreased bootloader timeout to 5 seconds
> Removed 'vmalloc=256MB snd-hda-intel.enable_msi=0' as it's not needed
> for any of reference BSPs.
> 
These changes are needed for newer hardware.

I further tested with and with out the GPT enabled and found that what
ever testing I did original was invalid and GPT is OK for both NUC and
MinnowboardMax.

So I think we can re-enable that for mkefidisk.wks.

These same changes should be applies to systemd-bootdisk.wks also.

Thanks
   Sau!

> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>  scripts/lib/wic/canned-wks/mkefidisk.wks | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/scripts/lib/wic/canned-wks/mkefidisk.wks
> b/scripts/lib/wic/canned-wks/mkefidisk.wks
> index 668ca92..7c287f7 100644
> --- a/scripts/lib/wic/canned-wks/mkefidisk.wks
> +++ b/scripts/lib/wic/canned-wks/mkefidisk.wks
> @@ -8,4 +8,4 @@ part / --source rootfs --ondisk sda --fstype=ext4 --
> label platform --align 1024
>  
>  part swap --ondisk sda --size 44 --label swap1 --fstype=swap
>  
> -bootloader --timeout=10 --append="rootwait rootfstype=ext4
> console=ttyPCH0,115200 console=tty0 vmalloc=256MB snd-hda-
> intel.enable_msi=0"
> +bootloader --timeout=5 --append="rootfstype=ext4
> console=ttyS0,115200 console=tty0"
> -- 
> 2.1.4
> 

[wic][PATCH] canned-wks: use GPT partition table

[Ed Bartosh]

According to UEFI specification all EFI platforms must support
GUID Partition Table(GPT) disk layout. Here is a list of advantages
of using GPT disk layout over the legacy MBR partitioning:

 - Logical Block Addresses (LBAs) are 64 bits (rather than 32 bits).
 - Supports many partitions (rather than just four primary partitions).
 - Provides both a primary and backup partition table for redundancy.
 - Uses version number and size fields for future expansion.
 - Uses CRC32 fields for improved data integrity.
 - Defines a GUID for uniquely identifying each partition.
 - Uses a GUID and attributes to define partition content type.
 - Each partition contains a 36 character human readable name.

Used GPT partitioning in all EFI kickstart files.
Tested result images on NUC, MinnowBoard MAX and MinnowBoard Turbot.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/canned-wks/mkefidisk.wks        | 2 +-
 scripts/lib/wic/canned-wks/mkgummidisk.wks      | 2 +-
 scripts/lib/wic/canned-wks/systemd-bootdisk.wks | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/wic/canned-wks/mkefidisk.wks b/scripts/lib/wic/canned-wks/mkefidisk.wks
index 7c287f7..9f534fe 100644
--- a/scripts/lib/wic/canned-wks/mkefidisk.wks
+++ b/scripts/lib/wic/canned-wks/mkefidisk.wks
@@ -8,4 +8,4 @@ part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024
 
 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
 
-bootloader --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
+bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
diff --git a/scripts/lib/wic/canned-wks/mkgummidisk.wks b/scripts/lib/wic/canned-wks/mkgummidisk.wks
index 616d3ce..f3ae090 100644
--- a/scripts/lib/wic/canned-wks/mkgummidisk.wks
+++ b/scripts/lib/wic/canned-wks/mkgummidisk.wks
@@ -8,4 +8,4 @@ part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024
 
 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
 
-bootloader  --timeout=5  --append="rootwait rootfstype=ext4 console=ttyS0,115200 console=tty0"
+bootloader --ptable gpt --timeout=5  --append="rootwait rootfstype=ext4 console=ttyS0,115200 console=tty0"
diff --git a/scripts/lib/wic/canned-wks/systemd-bootdisk.wks b/scripts/lib/wic/canned-wks/systemd-bootdisk.wks
index a49d130..b900023 100644
--- a/scripts/lib/wic/canned-wks/systemd-bootdisk.wks
+++ b/scripts/lib/wic/canned-wks/systemd-bootdisk.wks
@@ -8,4 +8,4 @@ part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024
 
 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
 
-bootloader --timeout=5 --append="rootwait rootfstype=ext4 console=ttyS0,115200 console=tty0"
+bootloader --ptable gpt --timeout=5 --append="rootwait rootfstype=ext4 console=ttyS0,115200 console=tty0"
-- 
2.1.4