[PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Tom Rini]

When we have been told to use the UUID we should also update the fstab
to make use of PARTUUID instead of hard-coding the device in question.
This will make the resulting image much more portable.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
 scripts/lib/wic/plugins/imager/direct.py | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
index a6abc3d09ef0..b0da54511b77 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -139,9 +139,12 @@ class DirectPlugin(ImagerPlugin):
                or part.mountpoint == "/":
                 continue
 
-            # mmc device partitions are named mmcblk0p1, mmcblk0p2..
-            prefix = 'p' if  part.disk.startswith('mmcblk') else ''
-            device_name = "/dev/%s%s%d" % (part.disk, prefix, part.realnum)
+            if part.use_uuid:
+                device_name = "PARTUUID=%s" % part.uuid
+            else:
+                # mmc device partitions are named mmcblk0p1, mmcblk0p2..
+                prefix = 'p' if  part.disk.startswith('mmcblk') else ''
+                device_name = "/dev/%s%s%d" % (part.disk, prefix, part.realnum)
 
             opts = part.fsopts if part.fsopts else "defaults"
             line = "\t".join([device_name, part.mountpoint, part.fstype,
-- 
1.9.1

[PATCH 2/2] wic: Update canned-wks for systemd to use UUID everywhere

[Tom Rini]

With systemd, the mounting of the swap partition is handled via systemd
and will mount it, regardless of if PARTUUID is parsed or not.  systemd
has a runtime dependency on util-linux-mount so PARTUUID for regular
mount points will be handled correctly.  Make all partitions that we add
to the image make use of UUIDs for maximum portability.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
Note that while systemd will automatically mount swap for you, busybox
swapon/swapoff do not understand PARTUUID syntax.  util-linux-swaponoff
is required for swapon -a / swapoff -a to work as expected from the
command line.
---
 scripts/lib/wic/canned-wks/systemd-bootdisk.wks | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/wic/canned-wks/systemd-bootdisk.wks b/scripts/lib/wic/canned-wks/systemd-bootdisk.wks
index 4bd9d6a65fd7..95d7b97a6063 100644
--- a/scripts/lib/wic/canned-wks/systemd-bootdisk.wks
+++ b/scripts/lib/wic/canned-wks/systemd-bootdisk.wks
@@ -2,10 +2,10 @@
 # long-description: Creates a partitioned EFI disk image that the user
 # can directly dd to boot media. The selected bootloader is systemd-boot.
 
-part /boot --source bootimg-efi --sourceparams="loader=systemd-boot" --ondisk sda --label msdos --active --align 1024
+part /boot --source bootimg-efi --sourceparams="loader=systemd-boot" --ondisk sda --label msdos --active --align 1024 --use-uuid
 
 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
 
-part swap --ondisk sda --size 44 --label swap1 --fstype=swap
+part swap --ondisk sda --size 44 --label swap1 --fstype=swap --use-uuid
 
 bootloader --ptable gpt --timeout=5 --append="rootwait rootfstype=ext4 console=ttyS0,115200 console=tty0"
-- 
1.9.1

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Otavio Salvador]

On Thu, Sep 21, 2017 at 2:46 PM, Tom Rini <trini@konsulko.com> wrote:
> When we have been told to use the UUID we should also update the fstab
> to make use of PARTUUID instead of hard-coding the device in question.
> This will make the resulting image much more portable.
>
> Signed-off-by: Tom Rini <trini@konsulko.com>

Adding more features which changes the fstab goes to the opposite
direction of reproducible builds; is it desirable?


-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 849 bytes --]

On Thu, Sep 21, 2017 at 06:15:56PM -0300, Otavio Salvador wrote:
> On Thu, Sep 21, 2017 at 2:46 PM, Tom Rini <trini@konsulko.com> wrote:
> > When we have been told to use the UUID we should also update the fstab
> > to make use of PARTUUID instead of hard-coding the device in question.
> > This will make the resulting image much more portable.
> >
> > Signed-off-by: Tom Rini <trini@konsulko.com>
> 
> Adding more features which changes the fstab goes to the opposite
> direction of reproducible builds; is it desirable?

UUIDs are already something that would be different every time.  It's
also (at least with just this patch) opt-in.  The second patch shows
that it's quite useful (and there's real-world examples, ie Minnow SD
card vs USB/SATA, various ARM systems where mmcblkX can change depending
on regulators.

-- 
Tom

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Mark Hatle]

On 9/21/17 4:15 PM, Otavio Salvador wrote:
> On Thu, Sep 21, 2017 at 2:46 PM, Tom Rini <trini@konsulko.com> wrote:
>> When we have been told to use the UUID we should also update the fstab
>> to make use of PARTUUID instead of hard-coding the device in question.
>> This will make the resulting image much more portable.
>>
>> Signed-off-by: Tom Rini <trini@konsulko.com>
> 
> Adding more features which changes the fstab goes to the opposite
> direction of reproducible builds; is it desirable?
> 
> 

There should be a corresponding way to specify the UUID for a generated
partition.  This would allow someone to pre-allocate the UUID and use it for
their generated image, and thus the reproducible build.

--Mark

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 913 bytes --]

On Thu, Sep 21, 2017 at 04:20:30PM -0500, Mark Hatle wrote:
> On 9/21/17 4:15 PM, Otavio Salvador wrote:
> > On Thu, Sep 21, 2017 at 2:46 PM, Tom Rini <trini@konsulko.com> wrote:
> >> When we have been told to use the UUID we should also update the fstab
> >> to make use of PARTUUID instead of hard-coding the device in question.
> >> This will make the resulting image much more portable.
> >>
> >> Signed-off-by: Tom Rini <trini@konsulko.com>
> > 
> > Adding more features which changes the fstab goes to the opposite
> > direction of reproducible builds; is it desirable?
> 
> There should be a corresponding way to specify the UUID for a generated
> partition.  This would allow someone to pre-allocate the UUID and use it for
> their generated image, and thus the reproducible build.

WIC does not today, but the underlying tools do, allow you to specify
the UUID to use, yes.

-- 
Tom

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Ed Bartosh]

On Thu, Sep 21, 2017 at 05:21:44PM -0400, Tom Rini wrote:
> On Thu, Sep 21, 2017 at 04:20:30PM -0500, Mark Hatle wrote:
> > On 9/21/17 4:15 PM, Otavio Salvador wrote:
> > > On Thu, Sep 21, 2017 at 2:46 PM, Tom Rini <trini@konsulko.com> wrote:
> > >> When we have been told to use the UUID we should also update the fstab
> > >> to make use of PARTUUID instead of hard-coding the device in question.
> > >> This will make the resulting image much more portable.
> > >>
> > >> Signed-off-by: Tom Rini <trini@konsulko.com>
> > > 
> > > Adding more features which changes the fstab goes to the opposite
> > > direction of reproducible builds; is it desirable?
> > 
> > There should be a corresponding way to specify the UUID for a generated
> > partition.  This would allow someone to pre-allocate the UUID and use it for
> > their generated image, and thus the reproducible build.
> 
> WIC does not today, but the underlying tools do, allow you to specify
> the UUID to use, yes.
> 

It does at least partly:

$ wic help kickstart |grep -A4 '\-\-uuid:'
         --uuid: This option is specific to wic. It specifies partition UUID.
                 It's useful if preconfigured partition UUID is added to kernel command line
                 in bootloader configuration before running wic. In this case .wks file can
                 be generated or modified to set preconfigured parition UUID using this option.

--
Regards,
Ed

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 809 bytes --]

On Thu, Sep 21, 2017 at 01:46:16PM -0400, Tom Rini wrote:
> When we have been told to use the UUID we should also update the fstab
> to make use of PARTUUID instead of hard-coding the device in question.
> This will make the resulting image much more portable.
> 
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
>  scripts/lib/wic/plugins/imager/direct.py | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)

Where we did we end up with this?  Ed pointed out that you can tell wic
to use a specific UUID, so reproducible images are not a problem.  And
making images that are readily portable is why other distros use
UUID/LABEL and not device names as much as possible.  I personally enjoy
being able to put an image on uSD for minnow and have it work :)
Thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 909 bytes --]

On Tue, Oct 10, 2017 at 05:01:49PM -0400, Tom Rini wrote:
> On Thu, Sep 21, 2017 at 01:46:16PM -0400, Tom Rini wrote:
> > When we have been told to use the UUID we should also update the fstab
> > to make use of PARTUUID instead of hard-coding the device in question.
> > This will make the resulting image much more portable.
> > 
> > Signed-off-by: Tom Rini <trini@konsulko.com>
> > ---
> >  scripts/lib/wic/plugins/imager/direct.py | 9 ++++++---
> >  1 file changed, 6 insertions(+), 3 deletions(-)
> 
> Where we did we end up with this?  Ed pointed out that you can tell wic
> to use a specific UUID, so reproducible images are not a problem.  And
> making images that are readily portable is why other distros use
> UUID/LABEL and not device names as much as possible.  I personally enjoy
> being able to put an image on uSD for minnow and have it work :)
> Thanks!

ping?

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 1132 bytes --]

On Fri, Oct 20, 2017 at 09:15:05AM -0400, Tom Rini wrote:
> On Tue, Oct 10, 2017 at 05:01:49PM -0400, Tom Rini wrote:
> > On Thu, Sep 21, 2017 at 01:46:16PM -0400, Tom Rini wrote:
> > > When we have been told to use the UUID we should also update the fstab
> > > to make use of PARTUUID instead of hard-coding the device in question.
> > > This will make the resulting image much more portable.
> > > 
> > > Signed-off-by: Tom Rini <trini@konsulko.com>
> > > ---
> > >  scripts/lib/wic/plugins/imager/direct.py | 9 ++++++---
> > >  1 file changed, 6 insertions(+), 3 deletions(-)
> > 
> > Where we did we end up with this?  Ed pointed out that you can tell wic
> > to use a specific UUID, so reproducible images are not a problem.  And
> > making images that are readily portable is why other distros use
> > UUID/LABEL and not device names as much as possible.  I personally enjoy
> > being able to put an image on uSD for minnow and have it work :)
> > Thanks!
> 
> ping?

I was just reminded about the real problems this solves (swap isn't
/dev/sda3, boot is being excessively slow), so, ping?

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Ed Bartosh]

On Fri, Nov 03, 2017 at 08:51:50AM -0400, Tom Rini wrote:
> On Fri, Oct 20, 2017 at 09:15:05AM -0400, Tom Rini wrote:
> > On Tue, Oct 10, 2017 at 05:01:49PM -0400, Tom Rini wrote:
> > > On Thu, Sep 21, 2017 at 01:46:16PM -0400, Tom Rini wrote:
> > > > When we have been told to use the UUID we should also update the fstab
> > > > to make use of PARTUUID instead of hard-coding the device in question.
> > > > This will make the resulting image much more portable.
> > > > 
> > > > Signed-off-by: Tom Rini <trini@konsulko.com>
> > > > ---
> > > >  scripts/lib/wic/plugins/imager/direct.py | 9 ++++++---
> > > >  1 file changed, 6 insertions(+), 3 deletions(-)
> > > 
> > > Where we did we end up with this?  Ed pointed out that you can tell wic
> > > to use a specific UUID, so reproducible images are not a problem.  And
> > > making images that are readily portable is why other distros use
> > > UUID/LABEL and not device names as much as possible.  I personally enjoy
> > > being able to put an image on uSD for minnow and have it work :)
> > > Thanks!
> > 
> > ping?
> 
> I was just reminded about the real problems this solves (swap isn't
> /dev/sda3, boot is being excessively slow), so, ping?
> 

I'm generally ok with the patchset. The only thing I'm thinking of is if busybox mount
supports PARTUUID syntax in fstab. Can you check this, please?

I'd like to see Otavio's confirmation that --uuid option solves
his reproducible builds concern.

Can you rebase your patchset on top of current master?
It would be also nice to get this functionality covered by tests.

--
Regards,
Ed

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Otavio Salvador]

On Mon, Nov 6, 2017 at 8:08 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> On Fri, Nov 03, 2017 at 08:51:50AM -0400, Tom Rini wrote:
>> On Fri, Oct 20, 2017 at 09:15:05AM -0400, Tom Rini wrote:
>> > On Tue, Oct 10, 2017 at 05:01:49PM -0400, Tom Rini wrote:
>> > > On Thu, Sep 21, 2017 at 01:46:16PM -0400, Tom Rini wrote:
>> > > > When we have been told to use the UUID we should also update the fstab
>> > > > to make use of PARTUUID instead of hard-coding the device in question.
>> > > > This will make the resulting image much more portable.
>> > > >
>> > > > Signed-off-by: Tom Rini <trini@konsulko.com>
>> > > > ---
>> > > >  scripts/lib/wic/plugins/imager/direct.py | 9 ++++++---
>> > > >  1 file changed, 6 insertions(+), 3 deletions(-)
>> > >
>> > > Where we did we end up with this?  Ed pointed out that you can tell wic
>> > > to use a specific UUID, so reproducible images are not a problem.  And
>> > > making images that are readily portable is why other distros use
>> > > UUID/LABEL and not device names as much as possible.  I personally enjoy
>> > > being able to put an image on uSD for minnow and have it work :)
>> > > Thanks!
>> >
>> > ping?
>>
>> I was just reminded about the real problems this solves (swap isn't
>> /dev/sda3, boot is being excessively slow), so, ping?
>>
>
> I'm generally ok with the patchset. The only thing I'm thinking of is if busybox mount
> supports PARTUUID syntax in fstab. Can you check this, please?

It does.

> I'd like to see Otavio's confirmation that --uuid option solves
> his reproducible builds concern.

I am not against it if the no-fstab-update consider this too. Using
uuid when making a fstab makes more sense but the possibility to not
touch the fstab must be kept.


-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 1862 bytes --]

On Mon, Nov 06, 2017 at 12:08:29PM +0200, Ed Bartosh wrote:
> On Fri, Nov 03, 2017 at 08:51:50AM -0400, Tom Rini wrote:
> > On Fri, Oct 20, 2017 at 09:15:05AM -0400, Tom Rini wrote:
> > > On Tue, Oct 10, 2017 at 05:01:49PM -0400, Tom Rini wrote:
> > > > On Thu, Sep 21, 2017 at 01:46:16PM -0400, Tom Rini wrote:
> > > > > When we have been told to use the UUID we should also update the fstab
> > > > > to make use of PARTUUID instead of hard-coding the device in question.
> > > > > This will make the resulting image much more portable.
> > > > > 
> > > > > Signed-off-by: Tom Rini <trini@konsulko.com>
> > > > > ---
> > > > >  scripts/lib/wic/plugins/imager/direct.py | 9 ++++++---
> > > > >  1 file changed, 6 insertions(+), 3 deletions(-)
> > > > 
> > > > Where we did we end up with this?  Ed pointed out that you can tell wic
> > > > to use a specific UUID, so reproducible images are not a problem.  And
> > > > making images that are readily portable is why other distros use
> > > > UUID/LABEL and not device names as much as possible.  I personally enjoy
> > > > being able to put an image on uSD for minnow and have it work :)
> > > > Thanks!
> > > 
> > > ping?
> > 
> > I was just reminded about the real problems this solves (swap isn't
> > /dev/sda3, boot is being excessively slow), so, ping?
> > 
> 
> I'm generally ok with the patchset. The only thing I'm thinking of is if busybox mount
> supports PARTUUID syntax in fstab. Can you check this, please?
> 
> I'd like to see Otavio's confirmation that --uuid option solves
> his reproducible builds concern.
> 
> Can you rebase your patchset on top of current master?

Yes, OK.

> It would be also nice to get this functionality covered by tests.

OK, but what part of oe-selftest, and how exactly?  Can one easily check
the resulting fstab?

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 2389 bytes --]

On Mon, Nov 06, 2017 at 09:36:20AM -0200, Otavio Salvador wrote:
> On Mon, Nov 6, 2017 at 8:08 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> > On Fri, Nov 03, 2017 at 08:51:50AM -0400, Tom Rini wrote:
> >> On Fri, Oct 20, 2017 at 09:15:05AM -0400, Tom Rini wrote:
> >> > On Tue, Oct 10, 2017 at 05:01:49PM -0400, Tom Rini wrote:
> >> > > On Thu, Sep 21, 2017 at 01:46:16PM -0400, Tom Rini wrote:
> >> > > > When we have been told to use the UUID we should also update the fstab
> >> > > > to make use of PARTUUID instead of hard-coding the device in question.
> >> > > > This will make the resulting image much more portable.
> >> > > >
> >> > > > Signed-off-by: Tom Rini <trini@konsulko.com>
> >> > > > ---
> >> > > >  scripts/lib/wic/plugins/imager/direct.py | 9 ++++++---
> >> > > >  1 file changed, 6 insertions(+), 3 deletions(-)
> >> > >
> >> > > Where we did we end up with this?  Ed pointed out that you can tell wic
> >> > > to use a specific UUID, so reproducible images are not a problem.  And
> >> > > making images that are readily portable is why other distros use
> >> > > UUID/LABEL and not device names as much as possible.  I personally enjoy
> >> > > being able to put an image on uSD for minnow and have it work :)
> >> > > Thanks!
> >> >
> >> > ping?
> >>
> >> I was just reminded about the real problems this solves (swap isn't
> >> /dev/sda3, boot is being excessively slow), so, ping?
> >>
> >
> > I'm generally ok with the patchset. The only thing I'm thinking of is if busybox mount
> > supports PARTUUID syntax in fstab. Can you check this, please?
> 
> It does.

It doesn't, you need util-linux for a mount that figures these out (or,
it did when I wrote the patch set).  That's why 2/2 only updates the
systemd canned wks as that _will_ have full mount.  But please note that
you can already make a wks file that doesn't work with busybox mount by
using the --use-uuid flag for non-root partitions :)

> > I'd like to see Otavio's confirmation that --uuid option solves
> > his reproducible builds concern.
> 
> I am not against it if the no-fstab-update consider this too. Using
> uuid when making a fstab makes more sense but the possibility to not
> touch the fstab must be kept.

Right.  I'm only updating the systemd wks file to use this.  Nothing
forces this option to be globally used.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Otavio Salvador]

On Mon, Nov 6, 2017 at 10:44 AM, Tom Rini <trini@konsulko.com> wrote:
> On Mon, Nov 06, 2017 at 09:36:20AM -0200, Otavio Salvador wrote:
>> On Mon, Nov 6, 2017 at 8:08 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
>> > On Fri, Nov 03, 2017 at 08:51:50AM -0400, Tom Rini wrote:
>> >> On Fri, Oct 20, 2017 at 09:15:05AM -0400, Tom Rini wrote:
>> >> > On Tue, Oct 10, 2017 at 05:01:49PM -0400, Tom Rini wrote:
>> >> > > On Thu, Sep 21, 2017 at 01:46:16PM -0400, Tom Rini wrote:
>> >> > > > When we have been told to use the UUID we should also update the fstab
>> >> > > > to make use of PARTUUID instead of hard-coding the device in question.
>> >> > > > This will make the resulting image much more portable.
>> >> > > >
>> >> > > > Signed-off-by: Tom Rini <trini@konsulko.com>
>> >> > > > ---
>> >> > > >  scripts/lib/wic/plugins/imager/direct.py | 9 ++++++---
>> >> > > >  1 file changed, 6 insertions(+), 3 deletions(-)
>> >> > >
>> >> > > Where we did we end up with this?  Ed pointed out that you can tell wic
>> >> > > to use a specific UUID, so reproducible images are not a problem.  And
>> >> > > making images that are readily portable is why other distros use
>> >> > > UUID/LABEL and not device names as much as possible.  I personally enjoy
>> >> > > being able to put an image on uSD for minnow and have it work :)
>> >> > > Thanks!
>> >> >
>> >> > ping?
>> >>
>> >> I was just reminded about the real problems this solves (swap isn't
>> >> /dev/sda3, boot is being excessively slow), so, ping?
>> >>
>> >
>> > I'm generally ok with the patchset. The only thing I'm thinking of is if busybox mount
>> > supports PARTUUID syntax in fstab. Can you check this, please?
>>
>> It does.
>
> It doesn't, you need util-linux for a mount that figures these out (or,
> it did when I wrote the patch set).  That's why 2/2 only updates the
> systemd canned wks as that _will_ have full mount.  But please note that
> you can already make a wks file that doesn't work with busybox mount by
> using the --use-uuid flag for non-root partitions :)

For the root, why you don't use U-Boot to figure the uuid and pass it
as cmdline, in root=? It is just a question as this is how we fixed
this for meta-freescale.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 2730 bytes --]

On Mon, Nov 06, 2017 at 10:51:56AM -0200, Otavio Salvador wrote:
> On Mon, Nov 6, 2017 at 10:44 AM, Tom Rini <trini@konsulko.com> wrote:
> > On Mon, Nov 06, 2017 at 09:36:20AM -0200, Otavio Salvador wrote:
> >> On Mon, Nov 6, 2017 at 8:08 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> >> > On Fri, Nov 03, 2017 at 08:51:50AM -0400, Tom Rini wrote:
> >> >> On Fri, Oct 20, 2017 at 09:15:05AM -0400, Tom Rini wrote:
> >> >> > On Tue, Oct 10, 2017 at 05:01:49PM -0400, Tom Rini wrote:
> >> >> > > On Thu, Sep 21, 2017 at 01:46:16PM -0400, Tom Rini wrote:
> >> >> > > > When we have been told to use the UUID we should also update the fstab
> >> >> > > > to make use of PARTUUID instead of hard-coding the device in question.
> >> >> > > > This will make the resulting image much more portable.
> >> >> > > >
> >> >> > > > Signed-off-by: Tom Rini <trini@konsulko.com>
> >> >> > > > ---
> >> >> > > >  scripts/lib/wic/plugins/imager/direct.py | 9 ++++++---
> >> >> > > >  1 file changed, 6 insertions(+), 3 deletions(-)
> >> >> > >
> >> >> > > Where we did we end up with this?  Ed pointed out that you can tell wic
> >> >> > > to use a specific UUID, so reproducible images are not a problem.  And
> >> >> > > making images that are readily portable is why other distros use
> >> >> > > UUID/LABEL and not device names as much as possible.  I personally enjoy
> >> >> > > being able to put an image on uSD for minnow and have it work :)
> >> >> > > Thanks!
> >> >> >
> >> >> > ping?
> >> >>
> >> >> I was just reminded about the real problems this solves (swap isn't
> >> >> /dev/sda3, boot is being excessively slow), so, ping?
> >> >>
> >> >
> >> > I'm generally ok with the patchset. The only thing I'm thinking of is if busybox mount
> >> > supports PARTUUID syntax in fstab. Can you check this, please?
> >>
> >> It does.
> >
> > It doesn't, you need util-linux for a mount that figures these out (or,
> > it did when I wrote the patch set).  That's why 2/2 only updates the
> > systemd canned wks as that _will_ have full mount.  But please note that
> > you can already make a wks file that doesn't work with busybox mount by
> > using the --use-uuid flag for non-root partitions :)
> 
> For the root, why you don't use U-Boot to figure the uuid and pass it
> as cmdline, in root=? It is just a question as this is how we fixed
> this for meta-freescale.

We already, globally, use uuid in WIC for root.  And in the case since
the kernel does it not mount, busybox isn't a concern.  We need to use
UUID more widely however for platforms such as minnowboard where we
might put the same image on SD card or USB (or SSD) where swap won't
always be /dev/sda3 :)

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Ed Bartosh]

On Mon, Nov 06, 2017 at 07:44:23AM -0500, Tom Rini wrote:
> On Mon, Nov 06, 2017 at 09:36:20AM -0200, Otavio Salvador wrote:
> > On Mon, Nov 6, 2017 at 8:08 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> > > On Fri, Nov 03, 2017 at 08:51:50AM -0400, Tom Rini wrote:
> > >> On Fri, Oct 20, 2017 at 09:15:05AM -0400, Tom Rini wrote:
> > >> > On Tue, Oct 10, 2017 at 05:01:49PM -0400, Tom Rini wrote:
> > >> > > On Thu, Sep 21, 2017 at 01:46:16PM -0400, Tom Rini wrote:
> > >> > > > When we have been told to use the UUID we should also update the fstab
> > >> > > > to make use of PARTUUID instead of hard-coding the device in question.
> > >> > > > This will make the resulting image much more portable.
> > >> > > >
> > >> > > > Signed-off-by: Tom Rini <trini@konsulko.com>
> > >> > > > ---
> > >> > > >  scripts/lib/wic/plugins/imager/direct.py | 9 ++++++---
> > >> > > >  1 file changed, 6 insertions(+), 3 deletions(-)
> > >> > >
> > >> > > Where we did we end up with this?  Ed pointed out that you can tell wic
> > >> > > to use a specific UUID, so reproducible images are not a problem.  And
> > >> > > making images that are readily portable is why other distros use
> > >> > > UUID/LABEL and not device names as much as possible.  I personally enjoy
> > >> > > being able to put an image on uSD for minnow and have it work :)
> > >> > > Thanks!
> > >> >
> > >> > ping?
> > >>
> > >> I was just reminded about the real problems this solves (swap isn't
> > >> /dev/sda3, boot is being excessively slow), so, ping?
> > >>
> > >
> > > I'm generally ok with the patchset. The only thing I'm thinking of is if busybox mount
> > > supports PARTUUID syntax in fstab. Can you check this, please?
> > 
> > It does.
> 
> It doesn't, you need util-linux for a mount that figures these out (or,
> it did when I wrote the patch set).

This practically means that this patch makes it impossible to use
--use-uuid for busybox-powered images. This is quite a big regression
from my point of view. We need to make this feature optional then.

>  That's why 2/2 only updates the
> systemd canned wks as that _will_ have full mount.  But please note that
> you can already make a wks file that doesn't work with busybox mount by
> using the --use-uuid flag for non-root partitions :)
>
What's the point of using --use-uuid for non-root partitions? I thought
it only makes sense for root as it triggers update of kernel command
line.


> > > I'd like to see Otavio's confirmation that --uuid option solves
> > > his reproducible builds concern.
> > 
> > I am not against it if the no-fstab-update consider this too. Using
> > uuid when making a fstab makes more sense but the possibility to not
> > touch the fstab must be kept.
> 
> Right.  I'm only updating the systemd wks file to use this.  Nothing
> forces this option to be globally used.

--
Regards,
Ed

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Ed Bartosh]

On Mon, Nov 06, 2017 at 07:44:17AM -0500, Tom Rini wrote:
> > Can you rebase your patchset on top of current master?
> 
> Yes, OK.
> 
> > It would be also nice to get this functionality covered by tests.
> 
> OK, but what part of oe-selftest, and how exactly?  Can one easily check
> the resulting fstab?

Sorry for not mentioning it. You can look at wic oe-selftest test cases
in meta/lib/oeqa/selftest/cases/wic.py

To check image fstab you can boot image in qemu (see test cases
starting with 'test_qemu' for the reference). This way you can also
check if partition uids match fstab entries.

P.S. Another idea is to extending 'wic cp' or implement 'wic cat'. All ingredients are
already in the wic code, so it shouldn't be too hard.

--
Regards,
Ed

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 4368 bytes --]

On Tue, Nov 07, 2017 at 10:11:35AM +0200, Ed Bartosh wrote:
> On Mon, Nov 06, 2017 at 07:44:23AM -0500, Tom Rini wrote:
> > On Mon, Nov 06, 2017 at 09:36:20AM -0200, Otavio Salvador wrote:
> > > On Mon, Nov 6, 2017 at 8:08 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> > > > On Fri, Nov 03, 2017 at 08:51:50AM -0400, Tom Rini wrote:
> > > >> On Fri, Oct 20, 2017 at 09:15:05AM -0400, Tom Rini wrote:
> > > >> > On Tue, Oct 10, 2017 at 05:01:49PM -0400, Tom Rini wrote:
> > > >> > > On Thu, Sep 21, 2017 at 01:46:16PM -0400, Tom Rini wrote:
> > > >> > > > When we have been told to use the UUID we should also update the fstab
> > > >> > > > to make use of PARTUUID instead of hard-coding the device in question.
> > > >> > > > This will make the resulting image much more portable.
> > > >> > > >
> > > >> > > > Signed-off-by: Tom Rini <trini@konsulko.com>
> > > >> > > > ---
> > > >> > > >  scripts/lib/wic/plugins/imager/direct.py | 9 ++++++---
> > > >> > > >  1 file changed, 6 insertions(+), 3 deletions(-)
> > > >> > >
> > > >> > > Where we did we end up with this?  Ed pointed out that you can tell wic
> > > >> > > to use a specific UUID, so reproducible images are not a problem.  And
> > > >> > > making images that are readily portable is why other distros use
> > > >> > > UUID/LABEL and not device names as much as possible.  I personally enjoy
> > > >> > > being able to put an image on uSD for minnow and have it work :)
> > > >> > > Thanks!
> > > >> >
> > > >> > ping?
> > > >>
> > > >> I was just reminded about the real problems this solves (swap isn't
> > > >> /dev/sda3, boot is being excessively slow), so, ping?
> > > >>
> > > >
> > > > I'm generally ok with the patchset. The only thing I'm thinking of is if busybox mount
> > > > supports PARTUUID syntax in fstab. Can you check this, please?
> > > 
> > > It does.
> > 
> > It doesn't, you need util-linux for a mount that figures these out (or,
> > it did when I wrote the patch set).

Let me correct myself here.  I think what threw me off for a moment
(since I hadn't picked these patches up in a while) was that busybox
mount might handle PARTUUID, but busybox swapon/off doesn't.  But that's
not quite the point, either.

> This practically means that this patch makes it impossible to use
> --use-uuid for busybox-powered images. This is quite a big regression
> from my point of view. We need to make this feature optional then.

How does it change anything at all for that case?  Today there's cases
where you would want to in some cases use PARTUUID, via --use-uuid, but
you can't.  --use-uuid is only valid for the root device as the kernel
handles parsing PARTUUID and our fstab only ever has /dev/root listed,
and all other entires in the table, even when we say --use-uuid, do not
use the UUID but instead use the device name.  Today, we only pass
--use-uuid in, in that case.

What 1/2 allows for is passing in --use-uuid for any partition and
having it be enforced.  There's no canned wks file changes made at all.

What 2/2 does is update the systemd canned wks (where you will not be
able to have busybox swapon/off, you're already forced into util-linux
for utils) to make use of --use-uuid.  I would strongly encourage this
to be applied as well as it makes systemd-based images much more useful
and does not change anything about busybox-only images.  It also allows
for external wks files to use UUIDs when they want to.

> >  That's why 2/2 only updates the
> > systemd canned wks as that _will_ have full mount.  But please note that
> > you can already make a wks file that doesn't work with busybox mount by
> > using the --use-uuid flag for non-root partitions :)
>
> What's the point of using --use-uuid for non-root partitions? I thought
> it only makes sense for root as it triggers update of kernel command
> line.

For the same reasons you want to use PARTUUID for the kernel.  You do
not know that you're going to be "sda".  You might be "vda" or "mmcblk"
or something else.  As a specific example, having "sda3" for swap in the
systemd wks file, but your actual device being something else, means
that you spend 90s during start-up for systemd trying very hard and
being mad that sda3 cannot be swapon'd (it will have swapon'd mmcblk0p3
already, automatically).

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 919 bytes --]

On Tue, Nov 07, 2017 at 10:27:25AM +0200, Ed Bartosh wrote:
> On Mon, Nov 06, 2017 at 07:44:17AM -0500, Tom Rini wrote:
> > > Can you rebase your patchset on top of current master?
> > 
> > Yes, OK.
> > 
> > > It would be also nice to get this functionality covered by tests.
> > 
> > OK, but what part of oe-selftest, and how exactly?  Can one easily check
> > the resulting fstab?
> 
> Sorry for not mentioning it. You can look at wic oe-selftest test cases
> in meta/lib/oeqa/selftest/cases/wic.py
> 
> To check image fstab you can boot image in qemu (see test cases
> starting with 'test_qemu' for the reference). This way you can also
> check if partition uids match fstab entries.
> 
> P.S. Another idea is to extending 'wic cp' or implement 'wic cat'. All
> ingredients are already in the wic code, so it shouldn't be too hard.

OK, I'll see what I can figure out here, thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Ed Bartosh]

On Tue, Nov 07, 2017 at 09:54:57AM -0500, Tom Rini wrote:
> On Tue, Nov 07, 2017 at 10:11:35AM +0200, Ed Bartosh wrote:
> > On Mon, Nov 06, 2017 at 07:44:23AM -0500, Tom Rini wrote:
> > > On Mon, Nov 06, 2017 at 09:36:20AM -0200, Otavio Salvador wrote:
> > > > On Mon, Nov 6, 2017 at 8:08 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> > > > > On Fri, Nov 03, 2017 at 08:51:50AM -0400, Tom Rini wrote:
> > > > >> On Fri, Oct 20, 2017 at 09:15:05AM -0400, Tom Rini wrote:
> > > > >> > On Tue, Oct 10, 2017 at 05:01:49PM -0400, Tom Rini wrote:
> > > > >> > > On Thu, Sep 21, 2017 at 01:46:16PM -0400, Tom Rini wrote:
> > > > >> > > > When we have been told to use the UUID we should also update the fstab
> > > > >> > > > to make use of PARTUUID instead of hard-coding the device in question.
> > > > >> > > > This will make the resulting image much more portable.
> > > > >> > > >
> > > > >> > > > Signed-off-by: Tom Rini <trini@konsulko.com>
> > > > >> > > > ---
> > > > >> > > >  scripts/lib/wic/plugins/imager/direct.py | 9 ++++++---
> > > > >> > > >  1 file changed, 6 insertions(+), 3 deletions(-)
> > > > >> > >
> > > > >> > > Where we did we end up with this?  Ed pointed out that you can tell wic
> > > > >> > > to use a specific UUID, so reproducible images are not a problem.  And
> > > > >> > > making images that are readily portable is why other distros use
> > > > >> > > UUID/LABEL and not device names as much as possible.  I personally enjoy
> > > > >> > > being able to put an image on uSD for minnow and have it work :)
> > > > >> > > Thanks!
> > > > >> >
> > > > >> > ping?
> > > > >>
> > > > >> I was just reminded about the real problems this solves (swap isn't
> > > > >> /dev/sda3, boot is being excessively slow), so, ping?
> > > > >>
> > > > >
> > > > > I'm generally ok with the patchset. The only thing I'm thinking of is if busybox mount
> > > > > supports PARTUUID syntax in fstab. Can you check this, please?
> > > > 
> > > > It does.
> > > 
> > > It doesn't, you need util-linux for a mount that figures these out (or,
> > > it did when I wrote the patch set).
> 
> Let me correct myself here.  I think what threw me off for a moment
> (since I hadn't picked these patches up in a while) was that busybox
> mount might handle PARTUUID, but busybox swapon/off doesn't.  But that's
> not quite the point, either.
> 

Just checked it out with core-image-minimal:

root@qemux86-64:~# grep UUID /etc/fstab
PARTUUID=452e020d-01	/boot	vfat	defaults	0	0
PARTUUID=452e020d-03	/mnt	ext4	defaults	0	0
root@qemux86-64:~# mount |grep mnt
root@qemux86-64:~# mount /mnt
mount: mounting PARTUUID=452e020d-03 on /mnt failed: No such file or
directory


It could be that busybox can be configured to do this, but at least current
oe-core busybox doesn't understand PARTUUID syntax in fstab.

> > This practically means that this patch makes it impossible to use
> > --use-uuid for busybox-powered images. This is quite a big regression
> > from my point of view. We need to make this feature optional then.
> 
> How does it change anything at all for that case?  Today there's cases
> where you would want to in some cases use PARTUUID, via --use-uuid, but
> you can't.  --use-uuid is only valid for the root device as the kernel
> handles parsing PARTUUID and our fstab only ever has /dev/root listed,
> and all other entires in the table, even when we say --use-uuid, do not
> use the UUID but instead use the device name.  Today, we only pass
> --use-uuid in, in that case.
> 
> What 1/2 allows for is passing in --use-uuid for any partition and
> having it be enforced.  There's no canned wks file changes made at all.
> 
> What 2/2 does is update the systemd canned wks (where you will not be
> able to have busybox swapon/off, you're already forced into util-linux
> for utils) to make use of --use-uuid.  I would strongly encourage this
> to be applied as well as it makes systemd-based images much more useful
> and does not change anything about busybox-only images.  It also allows
> for external wks files to use UUIDs when they want to.
> 

This makes sense to me. The patchset doesn't introduce regressions to
the current functionality. It would be nice to at least try to make it
working with busybox-powered images though.

If we can't do that for one or another reason we should at least mention
it in the documentation and wic help. Please, include this into the
patch.

> > >  That's why 2/2 only updates the
> > > systemd canned wks as that _will_ have full mount.  But please note that
> > > you can already make a wks file that doesn't work with busybox mount by
> > > using the --use-uuid flag for non-root partitions :)
> >
> > What's the point of using --use-uuid for non-root partitions? I thought
> > it only makes sense for root as it triggers update of kernel command
> > line.
> 
> For the same reasons you want to use PARTUUID for the kernel.  You do
> not know that you're going to be "sda".  You might be "vda" or "mmcblk"
> or something else.  As a specific example, having "sda3" for swap in the
> systemd wks file, but your actual device being something else, means
> that you spend 90s during start-up for systemd trying very hard and
> being mad that sda3 cannot be swapon'd (it will have swapon'd mmcblk0p3
> already, automatically).
> 

My question was about the current functionaily, i.e. without your patch.
With your patch using --use-uuid for non-root partitions started to make
sense.

P.S. there is a bugzilla item for this:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11525
please, don't hesitate to comment there.

--
Regards,
Ed

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Ed Bartosh]

On Wed, Nov 08, 2017 at 10:00:54AM -0500, Tom Rini wrote:
> On Wed, Nov 08, 2017 at 01:20:15PM +0200, Ed Bartosh wrote:
> > On Tue, Nov 07, 2017 at 09:54:57AM -0500, Tom Rini wrote:
> > > On Tue, Nov 07, 2017 at 10:11:35AM +0200, Ed Bartosh wrote:
> > > > On Mon, Nov 06, 2017 at 07:44:23AM -0500, Tom Rini wrote:
> > > > > On Mon, Nov 06, 2017 at 09:36:20AM -0200, Otavio Salvador wrote:
> > > > > > On Mon, Nov 6, 2017 at 8:08 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> > > > > > > On Fri, Nov 03, 2017 at 08:51:50AM -0400, Tom Rini wrote:
> > > > > > >> On Fri, Oct 20, 2017 at 09:15:05AM -0400, Tom Rini wrote:
> > > > > > >> > On Tue, Oct 10, 2017 at 05:01:49PM -0400, Tom Rini wrote:
> > > > > > >> > > On Thu, Sep 21, 2017 at 01:46:16PM -0400, Tom Rini wrote:
> > > > > > >> > > > When we have been told to use the UUID we should also update the fstab
> > > > > > >> > > > to make use of PARTUUID instead of hard-coding the device in question.
> > > > > > >> > > > This will make the resulting image much more portable.
> > > > > > >> > > >
> > > > > > >> > > > Signed-off-by: Tom Rini <trini@konsulko.com>
> > > > > > >> > > > ---
> > > > > > >> > > >  scripts/lib/wic/plugins/imager/direct.py | 9 ++++++---
> > > > > > >> > > >  1 file changed, 6 insertions(+), 3 deletions(-)
> > > > > > >> > >
> > > > > > >> > > Where we did we end up with this?  Ed pointed out that you can tell wic
> > > > > > >> > > to use a specific UUID, so reproducible images are not a problem.  And
> > > > > > >> > > making images that are readily portable is why other distros use
> > > > > > >> > > UUID/LABEL and not device names as much as possible.  I personally enjoy
> > > > > > >> > > being able to put an image on uSD for minnow and have it work :)
> > > > > > >> > > Thanks!
> > > > > > >> >
> > > > > > >> > ping?
> > > > > > >>
> > > > > > >> I was just reminded about the real problems this solves (swap isn't
> > > > > > >> /dev/sda3, boot is being excessively slow), so, ping?
> > > > > > >>
> > > > > > >
> > > > > > > I'm generally ok with the patchset. The only thing I'm thinking of is if busybox mount
> > > > > > > supports PARTUUID syntax in fstab. Can you check this, please?
> > > > > > 
> > > > > > It does.
> > > > > 
> > > > > It doesn't, you need util-linux for a mount that figures these out (or,
> > > > > it did when I wrote the patch set).
> > > 
> > > Let me correct myself here.  I think what threw me off for a moment
> > > (since I hadn't picked these patches up in a while) was that busybox
> > > mount might handle PARTUUID, but busybox swapon/off doesn't.  But that's
> > > not quite the point, either.
> > > 
> > 
> > Just checked it out with core-image-minimal:
> > 
> > root@qemux86-64:~# grep UUID /etc/fstab
> > PARTUUID=452e020d-01	/boot	vfat	defaults	0	0
> > PARTUUID=452e020d-03	/mnt	ext4	defaults	0	0
> > root@qemux86-64:~# mount |grep mnt
> > root@qemux86-64:~# mount /mnt
> > mount: mounting PARTUUID=452e020d-03 on /mnt failed: No such file or
> > directory
> > 
> > 
> > It could be that busybox can be configured to do this, but at least current
> > oe-core busybox doesn't understand PARTUUID syntax in fstab.
> 
> More clarity required then. :)  busybox mount does support LABEL and
> UUID (filesystem UUID, that is) but not PARTUUID.  The kernel doesn't
> support LABEL nor UUID (that is to say, root=UUID=xxx works via
> initramfs when it's used in other cases).
> 
> But the above leaves out one use case, and it's the one that got me
> started here, swap.  busybox does not support UUID nor PARTUUID in
> swapon/swapoff.  It's an additional feature to have it create swapfiles
> with a UUID at all.
> 
> > > > This practically means that this patch makes it impossible to use
> > > > --use-uuid for busybox-powered images. This is quite a big regression
> > > > from my point of view. We need to make this feature optional then.
> > > 
> > > How does it change anything at all for that case?  Today there's cases
> > > where you would want to in some cases use PARTUUID, via --use-uuid, but
> > > you can't.  --use-uuid is only valid for the root device as the kernel
> > > handles parsing PARTUUID and our fstab only ever has /dev/root listed,
> > > and all other entires in the table, even when we say --use-uuid, do not
> > > use the UUID but instead use the device name.  Today, we only pass
> > > --use-uuid in, in that case.
> > > 
> > > What 1/2 allows for is passing in --use-uuid for any partition and
> > > having it be enforced.  There's no canned wks file changes made at all.
> > > 
> > > What 2/2 does is update the systemd canned wks (where you will not be
> > > able to have busybox swapon/off, you're already forced into util-linux
> > > for utils) to make use of --use-uuid.  I would strongly encourage this
> > > to be applied as well as it makes systemd-based images much more useful
> > > and does not change anything about busybox-only images.  It also allows
> > > for external wks files to use UUIDs when they want to.
> > 
> > This makes sense to me. The patchset doesn't introduce regressions to
> > the current functionality.
> 
> Agreed.
> 
> > It would be nice to at least try to make it
> > working with busybox-powered images though.
> 
> I'm not sure this would be worth the effort.  Given what I said above,
> I don't think it's possible.   We could introduce enough logic to say to
> use PARTUUID for root, call out to blkid to get the filesystem UUID and
> use that for most filesystems, and then still have the busybox issue for
> swap.  PARTUUID is always going to work, with a full util-linux.
>

Even considering what you've said I think it worth the effort. There are
a lot of busybox images around and they'd benefit from at least partial
support of this.

> > If we can't do that for one or another reason we should at least mention
> > it in the documentation and wic help. Please, include this into the
> > patch.
> 
> OK, I can add a little verbage to help.py.  Is there work that needs
> doing to the yocto docs repository as well?
> 

That would be great if you do that as well. You can also mention this in
the bug and ask help from people who maintain Yocto documentation.

> > 
> > > > >  That's why 2/2 only updates the
> > > > > systemd canned wks as that _will_ have full mount.  But please note that
> > > > > you can already make a wks file that doesn't work with busybox mount by
> > > > > using the --use-uuid flag for non-root partitions :)
> > > >
> > > > What's the point of using --use-uuid for non-root partitions? I thought
> > > > it only makes sense for root as it triggers update of kernel command
> > > > line.
> > > 
> > > For the same reasons you want to use PARTUUID for the kernel.  You do
> > > not know that you're going to be "sda".  You might be "vda" or "mmcblk"
> > > or something else.  As a specific example, having "sda3" for swap in the
> > > systemd wks file, but your actual device being something else, means
> > > that you spend 90s during start-up for systemd trying very hard and
> > > being mad that sda3 cannot be swapon'd (it will have swapon'd mmcblk0p3
> > > already, automatically).
> > 
> > My question was about the current functionaily, i.e. without your patch.
> > With your patch using --use-uuid for non-root partitions started to make
> > sense.
> 
> OK.
> 
> > P.S. there is a bugzilla item for this:
> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=11525
> > please, don't hesitate to comment there.
> 
> I've grabbed the bugzilla entry since I'm going to push this until it's
> done.  Thanks!

Thank you!

--
Regards,
Ed

Re: [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 7064 bytes --]

On Wed, Nov 08, 2017 at 01:20:15PM +0200, Ed Bartosh wrote:
> On Tue, Nov 07, 2017 at 09:54:57AM -0500, Tom Rini wrote:
> > On Tue, Nov 07, 2017 at 10:11:35AM +0200, Ed Bartosh wrote:
> > > On Mon, Nov 06, 2017 at 07:44:23AM -0500, Tom Rini wrote:
> > > > On Mon, Nov 06, 2017 at 09:36:20AM -0200, Otavio Salvador wrote:
> > > > > On Mon, Nov 6, 2017 at 8:08 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> > > > > > On Fri, Nov 03, 2017 at 08:51:50AM -0400, Tom Rini wrote:
> > > > > >> On Fri, Oct 20, 2017 at 09:15:05AM -0400, Tom Rini wrote:
> > > > > >> > On Tue, Oct 10, 2017 at 05:01:49PM -0400, Tom Rini wrote:
> > > > > >> > > On Thu, Sep 21, 2017 at 01:46:16PM -0400, Tom Rini wrote:
> > > > > >> > > > When we have been told to use the UUID we should also update the fstab
> > > > > >> > > > to make use of PARTUUID instead of hard-coding the device in question.
> > > > > >> > > > This will make the resulting image much more portable.
> > > > > >> > > >
> > > > > >> > > > Signed-off-by: Tom Rini <trini@konsulko.com>
> > > > > >> > > > ---
> > > > > >> > > >  scripts/lib/wic/plugins/imager/direct.py | 9 ++++++---
> > > > > >> > > >  1 file changed, 6 insertions(+), 3 deletions(-)
> > > > > >> > >
> > > > > >> > > Where we did we end up with this?  Ed pointed out that you can tell wic
> > > > > >> > > to use a specific UUID, so reproducible images are not a problem.  And
> > > > > >> > > making images that are readily portable is why other distros use
> > > > > >> > > UUID/LABEL and not device names as much as possible.  I personally enjoy
> > > > > >> > > being able to put an image on uSD for minnow and have it work :)
> > > > > >> > > Thanks!
> > > > > >> >
> > > > > >> > ping?
> > > > > >>
> > > > > >> I was just reminded about the real problems this solves (swap isn't
> > > > > >> /dev/sda3, boot is being excessively slow), so, ping?
> > > > > >>
> > > > > >
> > > > > > I'm generally ok with the patchset. The only thing I'm thinking of is if busybox mount
> > > > > > supports PARTUUID syntax in fstab. Can you check this, please?
> > > > > 
> > > > > It does.
> > > > 
> > > > It doesn't, you need util-linux for a mount that figures these out (or,
> > > > it did when I wrote the patch set).
> > 
> > Let me correct myself here.  I think what threw me off for a moment
> > (since I hadn't picked these patches up in a while) was that busybox
> > mount might handle PARTUUID, but busybox swapon/off doesn't.  But that's
> > not quite the point, either.
> > 
> 
> Just checked it out with core-image-minimal:
> 
> root@qemux86-64:~# grep UUID /etc/fstab
> PARTUUID=452e020d-01	/boot	vfat	defaults	0	0
> PARTUUID=452e020d-03	/mnt	ext4	defaults	0	0
> root@qemux86-64:~# mount |grep mnt
> root@qemux86-64:~# mount /mnt
> mount: mounting PARTUUID=452e020d-03 on /mnt failed: No such file or
> directory
> 
> 
> It could be that busybox can be configured to do this, but at least current
> oe-core busybox doesn't understand PARTUUID syntax in fstab.

More clarity required then. :)  busybox mount does support LABEL and
UUID (filesystem UUID, that is) but not PARTUUID.  The kernel doesn't
support LABEL nor UUID (that is to say, root=UUID=xxx works via
initramfs when it's used in other cases).

But the above leaves out one use case, and it's the one that got me
started here, swap.  busybox does not support UUID nor PARTUUID in
swapon/swapoff.  It's an additional feature to have it create swapfiles
with a UUID at all.

> > > This practically means that this patch makes it impossible to use
> > > --use-uuid for busybox-powered images. This is quite a big regression
> > > from my point of view. We need to make this feature optional then.
> > 
> > How does it change anything at all for that case?  Today there's cases
> > where you would want to in some cases use PARTUUID, via --use-uuid, but
> > you can't.  --use-uuid is only valid for the root device as the kernel
> > handles parsing PARTUUID and our fstab only ever has /dev/root listed,
> > and all other entires in the table, even when we say --use-uuid, do not
> > use the UUID but instead use the device name.  Today, we only pass
> > --use-uuid in, in that case.
> > 
> > What 1/2 allows for is passing in --use-uuid for any partition and
> > having it be enforced.  There's no canned wks file changes made at all.
> > 
> > What 2/2 does is update the systemd canned wks (where you will not be
> > able to have busybox swapon/off, you're already forced into util-linux
> > for utils) to make use of --use-uuid.  I would strongly encourage this
> > to be applied as well as it makes systemd-based images much more useful
> > and does not change anything about busybox-only images.  It also allows
> > for external wks files to use UUIDs when they want to.
> 
> This makes sense to me. The patchset doesn't introduce regressions to
> the current functionality.

Agreed.

> It would be nice to at least try to make it
> working with busybox-powered images though.

I'm not sure this would be worth the effort.  Given what I said above,
I don't think it's possible.   We could introduce enough logic to say to
use PARTUUID for root, call out to blkid to get the filesystem UUID and
use that for most filesystems, and then still have the busybox issue for
swap.  PARTUUID is always going to work, with a full util-linux.

> If we can't do that for one or another reason we should at least mention
> it in the documentation and wic help. Please, include this into the
> patch.

OK, I can add a little verbage to help.py.  Is there work that needs
doing to the yocto docs repository as well?

> 
> > > >  That's why 2/2 only updates the
> > > > systemd canned wks as that _will_ have full mount.  But please note that
> > > > you can already make a wks file that doesn't work with busybox mount by
> > > > using the --use-uuid flag for non-root partitions :)
> > >
> > > What's the point of using --use-uuid for non-root partitions? I thought
> > > it only makes sense for root as it triggers update of kernel command
> > > line.
> > 
> > For the same reasons you want to use PARTUUID for the kernel.  You do
> > not know that you're going to be "sda".  You might be "vda" or "mmcblk"
> > or something else.  As a specific example, having "sda3" for swap in the
> > systemd wks file, but your actual device being something else, means
> > that you spend 90s during start-up for systemd trying very hard and
> > being mad that sda3 cannot be swapon'd (it will have swapon'd mmcblk0p3
> > already, automatically).
> 
> My question was about the current functionaily, i.e. without your patch.
> With your patch using --use-uuid for non-root partitions started to make
> sense.

OK.

> P.S. there is a bugzilla item for this:
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=11525
> please, don't hesitate to comment there.

I've grabbed the bugzilla entry since I'm going to push this until it's
done.  Thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]