Question about multi rootfs UIDs when using wic

Question about multi rootfs UIDs when using wic

[Volker Vogelhuber]

I'm currently trying to create a multi RootFS WIC image as mentioned in
directdisk-multi-rootfs.wks. For that I have two image recipes. One that 
is creating only an ext4 image (image-recovery), and the second that is 
also creating a WIC image (image-main). I used the IMAGE_FSTYPES 
variable for that. The WKS file for the second image is integrating the 
recovery image by specifying --rootfs-dir=image-recovery in it's part 
description.

# primary / recovery image
part / --source rootfs --rootfs-dir=image-main --exclude-path mnt/data/ 
mnt/data2/ --fstype=ext4 --label primary_rootfs --align 1024 --size 700 
--overhead-factor=1.0
part /recovery --source rootfs --rootfs-dir=image-recovery --fstype=ext4 
--label recovery_rootfs --align 1024 --size 640 --overhead-factor=1.0

The UIDs of the second rootfs (image-main) are correctly set to 0 within 
the file system when calling mkfs.ext4 during prepare_rootfs_ext. For 
the recovery rootfs the UID is always set to my own (host) one which is 
of course not valid for the image where that UID does not exist.
I tried calling the mkfs.ext4 command myself from a terminal and for 
whatever reason an image created out of the rootfs folder of the second 
image (image-main) recipe is deployed with the correct UID 0, while the 
rootfs folder of the first image (image-recovery) recipe always uses the 
UID of the source folder/files.

I search the code of e2fsprogs for the line that sets the UID and added 
a printf in set_inode_extra. There I can see clearly that the source UID 
for the file is 0 for the rootfs of the image-main rootfs folder while 
it is 10000 (my own UID) for the image-recovery. I wonder how the UID of 
the image-main rootfs folder can be zero when I don't call any command 
with root permissions. I searched for a preparation step where the UIDs 
are managed in the scripts folder of Yocto, but didn't found any hint 
for the whole behavior. So while it is good that the rootfs partition of 
the main rootfs has the UID set correctly to zero, I can't understand 
why it happens. On the other side I can understand why the UID of the 
recovery rootfs is set to my own one, but it stops me from booting that 
rootfs because the UIDs of the files and folders are set to a user that 
does not exist on the target system.

Can someone please explain to me, how that UID handling is meant to be done?

Re: Question about multi rootfs UIDs when using wic

[Volker Vogelhuber]

I finally found out, what's the reason why the included recovery rootfs 
has the wrong UIDs, while the main image has the correct one. The reason 
seems to be the PSEUDO_LOCALSTATEDIR variable that is set to the state 
dir of the main image in both cases.
I debugged all the calls up to the environment setup in partition.py's 
prepare_rootfs method where a check for an existing PSEUDO_LOCALSTATEDIR 
environment variable is done. That seems to be the problem.
Instead of using the correct value passed to the prepare_rootfs method, 
an existing ENV value is used that points to the state dir of the main 
image instead of the recovery one's. I guess the reason is that in 
bitbake.conf the PSEUDO_LOCALSTATEDIR is set to the image currently 
build (which is the main image and not the recovery image only 
referenced by the main image). So because that environment variable is 
already set, the call to mkfs.ext4 for the recovery image uses the wrong 
PSEUDO_LOCALSTATEDIR and does not apply the correct UID.

Any ideas how to fix that? I tend to just remove the patch introduced by 
Ed Bartosh three years ago 
(https://patchwork.openembedded.org/patch/90419/) because I don't need a 
custom PSEUDO_PREFIX. But maybe not everyone agrees.

On 19.01.2018 17:00, Volker Vogelhuber wrote:
> I'm currently trying to create a multi RootFS WIC image as mentioned in
> directdisk-multi-rootfs.wks. For that I have two image recipes. One that 
> is creating only an ext4 image (image-recovery), and the second that is 
> also creating a WIC image (image-main). I used the IMAGE_FSTYPES 
> variable for that. The WKS file for the second image is integrating the 
> recovery image by specifying --rootfs-dir=image-recovery in it's part 
> description.
> 
> # primary / recovery image
> part / --source rootfs --rootfs-dir=image-main --exclude-path mnt/data/ 
> mnt/data2/ --fstype=ext4 --label primary_rootfs --align 1024 --size 700 
> --overhead-factor=1.0
> part /recovery --source rootfs --rootfs-dir=image-recovery --fstype=ext4 
> --label recovery_rootfs --align 1024 --size 640 --overhead-factor=1.0
> 
> The UIDs of the second rootfs (image-main) are correctly set to 0 within 
> the file system when calling mkfs.ext4 during prepare_rootfs_ext. For 
> the recovery rootfs the UID is always set to my own (host) one which is 
> of course not valid for the image where that UID does not exist.
> I tried calling the mkfs.ext4 command myself from a terminal and for 
> whatever reason an image created out of the rootfs folder of the second 
> image (image-main) recipe is deployed with the correct UID 0, while the 
> rootfs folder of the first image (image-recovery) recipe always uses the 
> UID of the source folder/files.
> 
> I search the code of e2fsprogs for the line that sets the UID and added 
> a printf in set_inode_extra. There I can see clearly that the source UID 
> for the file is 0 for the rootfs of the image-main rootfs folder while 
> it is 10000 (my own UID) for the image-recovery. I wonder how the UID of 
> the image-main rootfs folder can be zero when I don't call any command 
> with root permissions. I searched for a preparation step where the UIDs 
> are managed in the scripts folder of Yocto, but didn't found any hint 
> for the whole behavior. So while it is good that the rootfs partition of 
> the main rootfs has the UID set correctly to zero, I can't understand 
> why it happens. On the other side I can understand why the UID of the 
> recovery rootfs is set to my own one, but it stops me from booting that 
> rootfs because the UIDs of the files and folders are set to a user that 
> does not exist on the target system.
> 
> Can someone please explain to me, how that UID handling is meant to be 
> done?

-- 




* *

*Volker Vogelhuber* | Head of Software Design



*PENTAX**Medical
*/Excellence in Focus/



*T:* // +49 (0)821 650566-18  | *F:* +49 (0)821 650566-20



Hoya Corporation – Pentax Medical Division

*Digital Endoscopy GmbH*

Paul-Lenz-Str. 5

86316 Friedberg – Germany

pentaxmedical.com



*Curious*about new *PENTAX*Medical products and activities?

*Follow us*on our Social Media channels!



   <https://www.linkedin.com/company/pentax-medical-company>
<https://www.youtube.com/channel/UCn2Ge1Wr7YsPZQiZ25nCkuA>



Handelsregister HRB 27226

Amtsgericht Augsburg

Geschäftsführer: Michael Drexel, Gerald Bottero

_____________________________________



This e-mail may contain confidential and/or privileged information. If
you are not the intended recipient (or have received this e-mail in
error) please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.

Re: Question about multi rootfs UIDs when using wic

[Ricardo Ribalda Delgado]

Hi

I just got hit by this one. It is specially nasty because nfsroot
fails to boot if the uids are wrong.

What is the status on this?

On Mon, Jan 22, 2018 at 11:39 AM, Volker Vogelhuber
<v.vogelhuber@digitalendoscopy.de> wrote:
> I finally found out, what's the reason why the included recovery rootfs has
> the wrong UIDs, while the main image has the correct one. The reason seems
> to be the PSEUDO_LOCALSTATEDIR variable that is set to the state dir of the
> main image in both cases.
> I debugged all the calls up to the environment setup in partition.py's
> prepare_rootfs method where a check for an existing PSEUDO_LOCALSTATEDIR
> environment variable is done. That seems to be the problem.
> Instead of using the correct value passed to the prepare_rootfs method, an
> existing ENV value is used that points to the state dir of the main image
> instead of the recovery one's. I guess the reason is that in bitbake.conf
> the PSEUDO_LOCALSTATEDIR is set to the image currently build (which is the
> main image and not the recovery image only referenced by the main image). So
> because that environment variable is already set, the call to mkfs.ext4 for
> the recovery image uses the wrong PSEUDO_LOCALSTATEDIR and does not apply
> the correct UID.
>
> Any ideas how to fix that? I tend to just remove the patch introduced by Ed
> Bartosh three years ago (https://patchwork.openembedded.org/patch/90419/)
> because I don't need a custom PSEUDO_PREFIX. But maybe not everyone agrees.
>
> On 19.01.2018 17:00, Volker Vogelhuber wrote:
>>
>> I'm currently trying to create a multi RootFS WIC image as mentioned in
>> directdisk-multi-rootfs.wks. For that I have two image recipes. One that
>> is creating only an ext4 image (image-recovery), and the second that is also
>> creating a WIC image (image-main). I used the IMAGE_FSTYPES variable for
>> that. The WKS file for the second image is integrating the recovery image by
>> specifying --rootfs-dir=image-recovery in it's part description.
>>
>> # primary / recovery image
>> part / --source rootfs --rootfs-dir=image-main --exclude-path mnt/data/
>> mnt/data2/ --fstype=ext4 --label primary_rootfs --align 1024 --size 700
>> --overhead-factor=1.0
>> part /recovery --source rootfs --rootfs-dir=image-recovery --fstype=ext4
>> --label recovery_rootfs --align 1024 --size 640 --overhead-factor=1.0
>>
>> The UIDs of the second rootfs (image-main) are correctly set to 0 within
>> the file system when calling mkfs.ext4 during prepare_rootfs_ext. For the
>> recovery rootfs the UID is always set to my own (host) one which is of
>> course not valid for the image where that UID does not exist.
>> I tried calling the mkfs.ext4 command myself from a terminal and for
>> whatever reason an image created out of the rootfs folder of the second
>> image (image-main) recipe is deployed with the correct UID 0, while the
>> rootfs folder of the first image (image-recovery) recipe always uses the UID
>> of the source folder/files.
>>
>> I search the code of e2fsprogs for the line that sets the UID and added a
>> printf in set_inode_extra. There I can see clearly that the source UID for
>> the file is 0 for the rootfs of the image-main rootfs folder while it is
>> 10000 (my own UID) for the image-recovery. I wonder how the UID of the
>> image-main rootfs folder can be zero when I don't call any command with root
>> permissions. I searched for a preparation step where the UIDs are managed in
>> the scripts folder of Yocto, but didn't found any hint for the whole
>> behavior. So while it is good that the rootfs partition of the main rootfs
>> has the UID set correctly to zero, I can't understand why it happens. On the
>> other side I can understand why the UID of the recovery rootfs is set to my
>> own one, but it stops me from booting that rootfs because the UIDs of the
>> files and folders are set to a user that does not exist on the target
>> system.
>>
>> Can someone please explain to me, how that UID handling is meant to be
>> done?
>
>
> --
>
>
>
>
> * *
>
> *Volker Vogelhuber* | Head of Software Design
>
>
>
> *PENTAX**Medical
> */Excellence in Focus/
>
>
>
> *T:* // +49 (0)821 650566-18  | *F:* +49 (0)821 650566-20
>
>
>
> Hoya Corporation – Pentax Medical Division
>
> *Digital Endoscopy GmbH*
>
> Paul-Lenz-Str. 5
>
> 86316 Friedberg – Germany
>
> pentaxmedical.com
>
>
>
> *Curious*about new *PENTAX*Medical products and activities?
>
> *Follow us*on our Social Media channels!
>
>
>
>   <https://www.linkedin.com/company/pentax-medical-company>
> <https://www.youtube.com/channel/UCn2Ge1Wr7YsPZQiZ25nCkuA>
>
>
>
> Handelsregister HRB 27226
>
> Amtsgericht Augsburg
>
> Geschäftsführer: Michael Drexel, Gerald Bottero
>
> _____________________________________
>
>
>
> This e-mail may contain confidential and/or privileged information. If
> you are not the intended recipient (or have received this e-mail in
> error) please notify the sender immediately and destroy this e-mail. Any
> unauthorized copying, disclosure or distribution of the material in this
> e-mail is strictly forbidden.
>
>
>
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core



-- 
Ricardo Ribalda

Re: Question about multi rootfs UIDs when using wic

[Volker Vogelhuber]

Hi,

I never got any response to my message (until now). So AFAIK I solved 
the problem with the following patch:

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 84fe85d62b..66ab757aec 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -204,15 +204,10 @@ class Partition():

          Currently handles ext2/3/4, btrfs and vfat.
          """
-        p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % 
native_sysroot)
-        p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
-                                         "%s/../pseudo" % 
get_bitbake_var("IMAGE_ROOTFS"))
-        p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
+        pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
+        pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % 
self.rootfs_dir
+        pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
          p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
-        pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
-        pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % p_localstatedir
-        pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd
-        pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp
          pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")

          rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,


On 02.05.2018 17:51, Ricardo Ribalda Delgado wrote:
> Hi
> 
> I just got hit by this one. It is specially nasty because nfsroot
> fails to boot if the uids are wrong.
> 
> What is the status on this?
> 
> On Mon, Jan 22, 2018 at 11:39 AM, Volker Vogelhuber
> <v.vogelhuber@digitalendoscopy.de> wrote:
>> I finally found out, what's the reason why the included recovery rootfs has
>> the wrong UIDs, while the main image has the correct one. The reason seems
>> to be the PSEUDO_LOCALSTATEDIR variable that is set to the state dir of the
>> main image in both cases.
>> I debugged all the calls up to the environment setup in partition.py's
>> prepare_rootfs method where a check for an existing PSEUDO_LOCALSTATEDIR
>> environment variable is done. That seems to be the problem.
>> Instead of using the correct value passed to the prepare_rootfs method, an
>> existing ENV value is used that points to the state dir of the main image
>> instead of the recovery one's. I guess the reason is that in bitbake.conf
>> the PSEUDO_LOCALSTATEDIR is set to the image currently build (which is the
>> main image and not the recovery image only referenced by the main image). So
>> because that environment variable is already set, the call to mkfs.ext4 for
>> the recovery image uses the wrong PSEUDO_LOCALSTATEDIR and does not apply
>> the correct UID.
>>
>> Any ideas how to fix that? I tend to just remove the patch introduced by Ed
>> Bartosh three years ago (https://patchwork.openembedded.org/patch/90419/)
>> because I don't need a custom PSEUDO_PREFIX. But maybe not everyone agrees.
>>
>> On 19.01.2018 17:00, Volker Vogelhuber wrote:
>>>
>>> I'm currently trying to create a multi RootFS WIC image as mentioned in
>>> directdisk-multi-rootfs.wks. For that I have two image recipes. One that
>>> is creating only an ext4 image (image-recovery), and the second that is also
>>> creating a WIC image (image-main). I used the IMAGE_FSTYPES variable for
>>> that. The WKS file for the second image is integrating the recovery image by
>>> specifying --rootfs-dir=image-recovery in it's part description.
>>>
>>> # primary / recovery image
>>> part / --source rootfs --rootfs-dir=image-main --exclude-path mnt/data/
>>> mnt/data2/ --fstype=ext4 --label primary_rootfs --align 1024 --size 700
>>> --overhead-factor=1.0
>>> part /recovery --source rootfs --rootfs-dir=image-recovery --fstype=ext4
>>> --label recovery_rootfs --align 1024 --size 640 --overhead-factor=1.0
>>>
>>> The UIDs of the second rootfs (image-main) are correctly set to 0 within
>>> the file system when calling mkfs.ext4 during prepare_rootfs_ext. For the
>>> recovery rootfs the UID is always set to my own (host) one which is of
>>> course not valid for the image where that UID does not exist.
>>> I tried calling the mkfs.ext4 command myself from a terminal and for
>>> whatever reason an image created out of the rootfs folder of the second
>>> image (image-main) recipe is deployed with the correct UID 0, while the
>>> rootfs folder of the first image (image-recovery) recipe always uses the UID
>>> of the source folder/files.
>>>
>>> I search the code of e2fsprogs for the line that sets the UID and added a
>>> printf in set_inode_extra. There I can see clearly that the source UID for
>>> the file is 0 for the rootfs of the image-main rootfs folder while it is
>>> 10000 (my own UID) for the image-recovery. I wonder how the UID of the
>>> image-main rootfs folder can be zero when I don't call any command with root
>>> permissions. I searched for a preparation step where the UIDs are managed in
>>> the scripts folder of Yocto, but didn't found any hint for the whole
>>> behavior. So while it is good that the rootfs partition of the main rootfs
>>> has the UID set correctly to zero, I can't understand why it happens. On the
>>> other side I can understand why the UID of the recovery rootfs is set to my
>>> own one, but it stops me from booting that rootfs because the UIDs of the
>>> files and folders are set to a user that does not exist on the target
>>> system.
>>>
>>> Can someone please explain to me, how that UID handling is meant to be
>>> done?
>>
>>
>> --
>>

Re: Question about multi rootfs UIDs when using wic

[Scott Rifenbark]

[-- Attachment #1: Type: text/plain, Size: 6812 bytes --]

Hi,

I have a couple Wic-related areas in the Yocto documentation.  Wondering if
this affects documentation.  I don't particularly have sections dedicated
to using specific canned wic files such as the "directdisk-multi-rootfs"
file.  However, if there is some mentioning or changing of the docs to help
avoid this I could see what I could do.

The sections related to Wic are:

   *
https://www.yoctoproject.org/docs/2.5/dev-manual/dev-manual.html#creating-partitioned-images-using-wic
   *
https://www.yoctoproject.org/docs/2.5/ref-manual/ref-manual.html#ref-kickstart

Thanks,
Scott

On Wed, May 2, 2018 at 9:31 AM, Volker Vogelhuber <
v.vogelhuber@digitalendoscopy.de> wrote:

> Hi,
>
> I never got any response to my message (until now). So AFAIK I solved the
> problem with the following patch:
>
> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> index 84fe85d62b..66ab757aec 100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -204,15 +204,10 @@ class Partition():
>
>          Currently handles ext2/3/4, btrfs and vfat.
>          """
> -        p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" %
> native_sysroot)
> -        p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
> -                                         "%s/../pseudo" %
> get_bitbake_var("IMAGE_ROOTFS"))
> -        p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
> +        pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
> +        pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" %
> self.rootfs_dir
> +        pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
>          p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
> -        pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
> -        pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % p_localstatedir
> -        pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd
> -        pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp
>          pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
>
>          rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
>
>
>
> On 02.05.2018 17:51, Ricardo Ribalda Delgado wrote:
>
>> Hi
>>
>> I just got hit by this one. It is specially nasty because nfsroot
>> fails to boot if the uids are wrong.
>>
>> What is the status on this?
>>
>> On Mon, Jan 22, 2018 at 11:39 AM, Volker Vogelhuber
>> <v.vogelhuber@digitalendoscopy.de> wrote:
>>
>>> I finally found out, what's the reason why the included recovery rootfs
>>> has
>>> the wrong UIDs, while the main image has the correct one. The reason
>>> seems
>>> to be the PSEUDO_LOCALSTATEDIR variable that is set to the state dir of
>>> the
>>> main image in both cases.
>>> I debugged all the calls up to the environment setup in partition.py's
>>> prepare_rootfs method where a check for an existing PSEUDO_LOCALSTATEDIR
>>> environment variable is done. That seems to be the problem.
>>> Instead of using the correct value passed to the prepare_rootfs method,
>>> an
>>> existing ENV value is used that points to the state dir of the main image
>>> instead of the recovery one's. I guess the reason is that in bitbake.conf
>>> the PSEUDO_LOCALSTATEDIR is set to the image currently build (which is
>>> the
>>> main image and not the recovery image only referenced by the main
>>> image). So
>>> because that environment variable is already set, the call to mkfs.ext4
>>> for
>>> the recovery image uses the wrong PSEUDO_LOCALSTATEDIR and does not apply
>>> the correct UID.
>>>
>>> Any ideas how to fix that? I tend to just remove the patch introduced by
>>> Ed
>>> Bartosh three years ago (https://patchwork.openembedded.org/patch/90419/
>>> )
>>> because I don't need a custom PSEUDO_PREFIX. But maybe not everyone
>>> agrees.
>>>
>>> On 19.01.2018 17:00, Volker Vogelhuber wrote:
>>>
>>>>
>>>> I'm currently trying to create a multi RootFS WIC image as mentioned in
>>>> directdisk-multi-rootfs.wks. For that I have two image recipes. One that
>>>> is creating only an ext4 image (image-recovery), and the second that is
>>>> also
>>>> creating a WIC image (image-main). I used the IMAGE_FSTYPES variable for
>>>> that. The WKS file for the second image is integrating the recovery
>>>> image by
>>>> specifying --rootfs-dir=image-recovery in it's part description.
>>>>
>>>> # primary / recovery image
>>>> part / --source rootfs --rootfs-dir=image-main --exclude-path mnt/data/
>>>> mnt/data2/ --fstype=ext4 --label primary_rootfs --align 1024 --size 700
>>>> --overhead-factor=1.0
>>>> part /recovery --source rootfs --rootfs-dir=image-recovery --fstype=ext4
>>>> --label recovery_rootfs --align 1024 --size 640 --overhead-factor=1.0
>>>>
>>>> The UIDs of the second rootfs (image-main) are correctly set to 0 within
>>>> the file system when calling mkfs.ext4 during prepare_rootfs_ext. For
>>>> the
>>>> recovery rootfs the UID is always set to my own (host) one which is of
>>>> course not valid for the image where that UID does not exist.
>>>> I tried calling the mkfs.ext4 command myself from a terminal and for
>>>> whatever reason an image created out of the rootfs folder of the second
>>>> image (image-main) recipe is deployed with the correct UID 0, while the
>>>> rootfs folder of the first image (image-recovery) recipe always uses
>>>> the UID
>>>> of the source folder/files.
>>>>
>>>> I search the code of e2fsprogs for the line that sets the UID and added
>>>> a
>>>> printf in set_inode_extra. There I can see clearly that the source UID
>>>> for
>>>> the file is 0 for the rootfs of the image-main rootfs folder while it is
>>>> 10000 (my own UID) for the image-recovery. I wonder how the UID of the
>>>> image-main rootfs folder can be zero when I don't call any command with
>>>> root
>>>> permissions. I searched for a preparation step where the UIDs are
>>>> managed in
>>>> the scripts folder of Yocto, but didn't found any hint for the whole
>>>> behavior. So while it is good that the rootfs partition of the main
>>>> rootfs
>>>> has the UID set correctly to zero, I can't understand why it happens.
>>>> On the
>>>> other side I can understand why the UID of the recovery rootfs is set
>>>> to my
>>>> own one, but it stops me from booting that rootfs because the UIDs of
>>>> the
>>>> files and folders are set to a user that does not exist on the target
>>>> system.
>>>>
>>>> Can someone please explain to me, how that UID handling is meant to be
>>>> done?
>>>>
>>>
>>>
>>> --
>>>
>>> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 8493 bytes --]

Re: Question about multi rootfs UIDs when using wic

[Ricardo Ribalda Delgado]

Hi Scott

In this case I believe that it is a bug in wic, for no reason the user
id of bitbake should be leaked into the rootfs.

Regards

On Wed, May 2, 2018 at 6:44 PM, Scott Rifenbark <srifenbark@gmail.com> wrote:
> Hi,
>
> I have a couple Wic-related areas in the Yocto documentation.  Wondering if
> this affects documentation.  I don't particularly have sections dedicated to
> using specific canned wic files such as the "directdisk-multi-rootfs" file.
> However, if there is some mentioning or changing of the docs to help avoid
> this I could see what I could do.
>
> The sections related to Wic are:
>
>    *
> https://www.yoctoproject.org/docs/2.5/dev-manual/dev-manual.html#creating-partitioned-images-using-wic
>    *
> https://www.yoctoproject.org/docs/2.5/ref-manual/ref-manual.html#ref-kickstart
>
> Thanks,
> Scott
>
> On Wed, May 2, 2018 at 9:31 AM, Volker Vogelhuber
> <v.vogelhuber@digitalendoscopy.de> wrote:
>>
>> Hi,
>>
>> I never got any response to my message (until now). So AFAIK I solved the
>> problem with the following patch:
>>
>> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
>> index 84fe85d62b..66ab757aec 100644
>> --- a/scripts/lib/wic/partition.py
>> +++ b/scripts/lib/wic/partition.py
>> @@ -204,15 +204,10 @@ class Partition():
>>
>>          Currently handles ext2/3/4, btrfs and vfat.
>>          """
>> -        p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" %
>> native_sysroot)
>> -        p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
>> -                                         "%s/../pseudo" %
>> get_bitbake_var("IMAGE_ROOTFS"))
>> -        p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
>> +        pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
>> +        pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" %
>> self.rootfs_dir
>> +        pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
>>          p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
>> -        pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
>> -        pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % p_localstatedir
>> -        pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd
>> -        pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp
>>          pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
>>
>>          rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
>>
>>
>>
>> On 02.05.2018 17:51, Ricardo Ribalda Delgado wrote:
>>>
>>> Hi
>>>
>>> I just got hit by this one. It is specially nasty because nfsroot
>>> fails to boot if the uids are wrong.
>>>
>>> What is the status on this?
>>>
>>> On Mon, Jan 22, 2018 at 11:39 AM, Volker Vogelhuber
>>> <v.vogelhuber@digitalendoscopy.de> wrote:
>>>>
>>>> I finally found out, what's the reason why the included recovery rootfs
>>>> has
>>>> the wrong UIDs, while the main image has the correct one. The reason
>>>> seems
>>>> to be the PSEUDO_LOCALSTATEDIR variable that is set to the state dir of
>>>> the
>>>> main image in both cases.
>>>> I debugged all the calls up to the environment setup in partition.py's
>>>> prepare_rootfs method where a check for an existing PSEUDO_LOCALSTATEDIR
>>>> environment variable is done. That seems to be the problem.
>>>> Instead of using the correct value passed to the prepare_rootfs method,
>>>> an
>>>> existing ENV value is used that points to the state dir of the main
>>>> image
>>>> instead of the recovery one's. I guess the reason is that in
>>>> bitbake.conf
>>>> the PSEUDO_LOCALSTATEDIR is set to the image currently build (which is
>>>> the
>>>> main image and not the recovery image only referenced by the main
>>>> image). So
>>>> because that environment variable is already set, the call to mkfs.ext4
>>>> for
>>>> the recovery image uses the wrong PSEUDO_LOCALSTATEDIR and does not
>>>> apply
>>>> the correct UID.
>>>>
>>>> Any ideas how to fix that? I tend to just remove the patch introduced by
>>>> Ed
>>>> Bartosh three years ago
>>>> (https://patchwork.openembedded.org/patch/90419/)
>>>> because I don't need a custom PSEUDO_PREFIX. But maybe not everyone
>>>> agrees.
>>>>
>>>> On 19.01.2018 17:00, Volker Vogelhuber wrote:
>>>>>
>>>>>
>>>>> I'm currently trying to create a multi RootFS WIC image as mentioned in
>>>>> directdisk-multi-rootfs.wks. For that I have two image recipes. One
>>>>> that
>>>>> is creating only an ext4 image (image-recovery), and the second that is
>>>>> also
>>>>> creating a WIC image (image-main). I used the IMAGE_FSTYPES variable
>>>>> for
>>>>> that. The WKS file for the second image is integrating the recovery
>>>>> image by
>>>>> specifying --rootfs-dir=image-recovery in it's part description.
>>>>>
>>>>> # primary / recovery image
>>>>> part / --source rootfs --rootfs-dir=image-main --exclude-path mnt/data/
>>>>> mnt/data2/ --fstype=ext4 --label primary_rootfs --align 1024 --size 700
>>>>> --overhead-factor=1.0
>>>>> part /recovery --source rootfs --rootfs-dir=image-recovery
>>>>> --fstype=ext4
>>>>> --label recovery_rootfs --align 1024 --size 640 --overhead-factor=1.0
>>>>>
>>>>> The UIDs of the second rootfs (image-main) are correctly set to 0
>>>>> within
>>>>> the file system when calling mkfs.ext4 during prepare_rootfs_ext. For
>>>>> the
>>>>> recovery rootfs the UID is always set to my own (host) one which is of
>>>>> course not valid for the image where that UID does not exist.
>>>>> I tried calling the mkfs.ext4 command myself from a terminal and for
>>>>> whatever reason an image created out of the rootfs folder of the second
>>>>> image (image-main) recipe is deployed with the correct UID 0, while the
>>>>> rootfs folder of the first image (image-recovery) recipe always uses
>>>>> the UID
>>>>> of the source folder/files.
>>>>>
>>>>> I search the code of e2fsprogs for the line that sets the UID and added
>>>>> a
>>>>> printf in set_inode_extra. There I can see clearly that the source UID
>>>>> for
>>>>> the file is 0 for the rootfs of the image-main rootfs folder while it
>>>>> is
>>>>> 10000 (my own UID) for the image-recovery. I wonder how the UID of the
>>>>> image-main rootfs folder can be zero when I don't call any command with
>>>>> root
>>>>> permissions. I searched for a preparation step where the UIDs are
>>>>> managed in
>>>>> the scripts folder of Yocto, but didn't found any hint for the whole
>>>>> behavior. So while it is good that the rootfs partition of the main
>>>>> rootfs
>>>>> has the UID set correctly to zero, I can't understand why it happens.
>>>>> On the
>>>>> other side I can understand why the UID of the recovery rootfs is set
>>>>> to my
>>>>> own one, but it stops me from booting that rootfs because the UIDs of
>>>>> the
>>>>> files and folders are set to a user that does not exist on the target
>>>>> system.
>>>>>
>>>>> Can someone please explain to me, how that UID handling is meant to be
>>>>> done?
>>>>
>>>>
>>>>
>>>> --
>>>>
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>



-- 
Ricardo Ribalda

Re: Question about multi rootfs UIDs when using wic

[Scott Rifenbark]

[-- Attachment #1: Type: text/plain, Size: 7884 bytes --]

Thanks Ricardo... that helps me.

Scott

On Wed, May 2, 2018 at 11:18 AM, Ricardo Ribalda Delgado <
ricardo.ribalda@gmail.com> wrote:

> Hi Scott
>
> In this case I believe that it is a bug in wic, for no reason the user
> id of bitbake should be leaked into the rootfs.
>
> Regards
>
> On Wed, May 2, 2018 at 6:44 PM, Scott Rifenbark <srifenbark@gmail.com>
> wrote:
> > Hi,
> >
> > I have a couple Wic-related areas in the Yocto documentation.  Wondering
> if
> > this affects documentation.  I don't particularly have sections
> dedicated to
> > using specific canned wic files such as the "directdisk-multi-rootfs"
> file.
> > However, if there is some mentioning or changing of the docs to help
> avoid
> > this I could see what I could do.
> >
> > The sections related to Wic are:
> >
> >    *
> > https://www.yoctoproject.org/docs/2.5/dev-manual/dev-
> manual.html#creating-partitioned-images-using-wic
> >    *
> > https://www.yoctoproject.org/docs/2.5/ref-manual/ref-
> manual.html#ref-kickstart
> >
> > Thanks,
> > Scott
> >
> > On Wed, May 2, 2018 at 9:31 AM, Volker Vogelhuber
> > <v.vogelhuber@digitalendoscopy.de> wrote:
> >>
> >> Hi,
> >>
> >> I never got any response to my message (until now). So AFAIK I solved
> the
> >> problem with the following patch:
> >>
> >> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> >> index 84fe85d62b..66ab757aec 100644
> >> --- a/scripts/lib/wic/partition.py
> >> +++ b/scripts/lib/wic/partition.py
> >> @@ -204,15 +204,10 @@ class Partition():
> >>
> >>          Currently handles ext2/3/4, btrfs and vfat.
> >>          """
> >> -        p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" %
> >> native_sysroot)
> >> -        p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
> >> -                                         "%s/../pseudo" %
> >> get_bitbake_var("IMAGE_ROOTFS"))
> >> -        p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
> >> +        pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
> >> +        pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" %
> >> self.rootfs_dir
> >> +        pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
> >>          p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
> >> -        pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
> >> -        pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % p_localstatedir
> >> -        pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd
> >> -        pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp
> >>          pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
> >>
> >>          rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
> >>
> >>
> >>
> >> On 02.05.2018 17:51, Ricardo Ribalda Delgado wrote:
> >>>
> >>> Hi
> >>>
> >>> I just got hit by this one. It is specially nasty because nfsroot
> >>> fails to boot if the uids are wrong.
> >>>
> >>> What is the status on this?
> >>>
> >>> On Mon, Jan 22, 2018 at 11:39 AM, Volker Vogelhuber
> >>> <v.vogelhuber@digitalendoscopy.de> wrote:
> >>>>
> >>>> I finally found out, what's the reason why the included recovery
> rootfs
> >>>> has
> >>>> the wrong UIDs, while the main image has the correct one. The reason
> >>>> seems
> >>>> to be the PSEUDO_LOCALSTATEDIR variable that is set to the state dir
> of
> >>>> the
> >>>> main image in both cases.
> >>>> I debugged all the calls up to the environment setup in partition.py's
> >>>> prepare_rootfs method where a check for an existing
> PSEUDO_LOCALSTATEDIR
> >>>> environment variable is done. That seems to be the problem.
> >>>> Instead of using the correct value passed to the prepare_rootfs
> method,
> >>>> an
> >>>> existing ENV value is used that points to the state dir of the main
> >>>> image
> >>>> instead of the recovery one's. I guess the reason is that in
> >>>> bitbake.conf
> >>>> the PSEUDO_LOCALSTATEDIR is set to the image currently build (which is
> >>>> the
> >>>> main image and not the recovery image only referenced by the main
> >>>> image). So
> >>>> because that environment variable is already set, the call to
> mkfs.ext4
> >>>> for
> >>>> the recovery image uses the wrong PSEUDO_LOCALSTATEDIR and does not
> >>>> apply
> >>>> the correct UID.
> >>>>
> >>>> Any ideas how to fix that? I tend to just remove the patch introduced
> by
> >>>> Ed
> >>>> Bartosh three years ago
> >>>> (https://patchwork.openembedded.org/patch/90419/)
> >>>> because I don't need a custom PSEUDO_PREFIX. But maybe not everyone
> >>>> agrees.
> >>>>
> >>>> On 19.01.2018 17:00, Volker Vogelhuber wrote:
> >>>>>
> >>>>>
> >>>>> I'm currently trying to create a multi RootFS WIC image as mentioned
> in
> >>>>> directdisk-multi-rootfs.wks. For that I have two image recipes. One
> >>>>> that
> >>>>> is creating only an ext4 image (image-recovery), and the second that
> is
> >>>>> also
> >>>>> creating a WIC image (image-main). I used the IMAGE_FSTYPES variable
> >>>>> for
> >>>>> that. The WKS file for the second image is integrating the recovery
> >>>>> image by
> >>>>> specifying --rootfs-dir=image-recovery in it's part description.
> >>>>>
> >>>>> # primary / recovery image
> >>>>> part / --source rootfs --rootfs-dir=image-main --exclude-path
> mnt/data/
> >>>>> mnt/data2/ --fstype=ext4 --label primary_rootfs --align 1024 --size
> 700
> >>>>> --overhead-factor=1.0
> >>>>> part /recovery --source rootfs --rootfs-dir=image-recovery
> >>>>> --fstype=ext4
> >>>>> --label recovery_rootfs --align 1024 --size 640 --overhead-factor=1.0
> >>>>>
> >>>>> The UIDs of the second rootfs (image-main) are correctly set to 0
> >>>>> within
> >>>>> the file system when calling mkfs.ext4 during prepare_rootfs_ext. For
> >>>>> the
> >>>>> recovery rootfs the UID is always set to my own (host) one which is
> of
> >>>>> course not valid for the image where that UID does not exist.
> >>>>> I tried calling the mkfs.ext4 command myself from a terminal and for
> >>>>> whatever reason an image created out of the rootfs folder of the
> second
> >>>>> image (image-main) recipe is deployed with the correct UID 0, while
> the
> >>>>> rootfs folder of the first image (image-recovery) recipe always uses
> >>>>> the UID
> >>>>> of the source folder/files.
> >>>>>
> >>>>> I search the code of e2fsprogs for the line that sets the UID and
> added
> >>>>> a
> >>>>> printf in set_inode_extra. There I can see clearly that the source
> UID
> >>>>> for
> >>>>> the file is 0 for the rootfs of the image-main rootfs folder while it
> >>>>> is
> >>>>> 10000 (my own UID) for the image-recovery. I wonder how the UID of
> the
> >>>>> image-main rootfs folder can be zero when I don't call any command
> with
> >>>>> root
> >>>>> permissions. I searched for a preparation step where the UIDs are
> >>>>> managed in
> >>>>> the scripts folder of Yocto, but didn't found any hint for the whole
> >>>>> behavior. So while it is good that the rootfs partition of the main
> >>>>> rootfs
> >>>>> has the UID set correctly to zero, I can't understand why it happens.
> >>>>> On the
> >>>>> other side I can understand why the UID of the recovery rootfs is set
> >>>>> to my
> >>>>> own one, but it stops me from booting that rootfs because the UIDs of
> >>>>> the
> >>>>> files and folders are set to a user that does not exist on the target
> >>>>> system.
> >>>>>
> >>>>> Can someone please explain to me, how that UID handling is meant to
> be
> >>>>> done?
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>>
> >> --
> >> _______________________________________________
> >> Openembedded-core mailing list
> >> Openembedded-core@lists.openembedded.org
> >> http://lists.openembedded.org/mailman/listinfo/openembedded-core
> >
> >
>
>
>
> --
> Ricardo Ribalda
>

[-- Attachment #2: Type: text/html, Size: 11503 bytes --]

Re: Question about multi rootfs UIDs when using wic

[Ricardo Ribalda Delgado]

Hi

I do not know how many people is actively using wic. We have started
using it from last week, it is hard to replace old scripts that work
:)

Try sending it as a proper patch, I can help you if you need it.

Regards

On Wed, May 2, 2018 at 6:31 PM, Volker Vogelhuber
<v.vogelhuber@digitalendoscopy.de> wrote:
> Hi,
>
> I never got any response to my message (until now). So AFAIK I solved the
> problem with the following patch:
>
> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> index 84fe85d62b..66ab757aec 100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -204,15 +204,10 @@ class Partition():
>
>          Currently handles ext2/3/4, btrfs and vfat.
>          """
> -        p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" %
> native_sysroot)
> -        p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
> -                                         "%s/../pseudo" %
> get_bitbake_var("IMAGE_ROOTFS"))
> -        p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
> +        pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
> +        pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" %
> self.rootfs_dir
> +        pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
>          p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
> -        pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
> -        pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % p_localstatedir
> -        pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd
> -        pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp
>          pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
>
>          rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
>
>
>
> On 02.05.2018 17:51, Ricardo Ribalda Delgado wrote:
>>
>> Hi
>>
>> I just got hit by this one. It is specially nasty because nfsroot
>> fails to boot if the uids are wrong.
>>
>> What is the status on this?
>>
>> On Mon, Jan 22, 2018 at 11:39 AM, Volker Vogelhuber
>> <v.vogelhuber@digitalendoscopy.de> wrote:
>>>
>>> I finally found out, what's the reason why the included recovery rootfs
>>> has
>>> the wrong UIDs, while the main image has the correct one. The reason
>>> seems
>>> to be the PSEUDO_LOCALSTATEDIR variable that is set to the state dir of
>>> the
>>> main image in both cases.
>>> I debugged all the calls up to the environment setup in partition.py's
>>> prepare_rootfs method where a check for an existing PSEUDO_LOCALSTATEDIR
>>> environment variable is done. That seems to be the problem.
>>> Instead of using the correct value passed to the prepare_rootfs method,
>>> an
>>> existing ENV value is used that points to the state dir of the main image
>>> instead of the recovery one's. I guess the reason is that in bitbake.conf
>>> the PSEUDO_LOCALSTATEDIR is set to the image currently build (which is
>>> the
>>> main image and not the recovery image only referenced by the main image).
>>> So
>>> because that environment variable is already set, the call to mkfs.ext4
>>> for
>>> the recovery image uses the wrong PSEUDO_LOCALSTATEDIR and does not apply
>>> the correct UID.
>>>
>>> Any ideas how to fix that? I tend to just remove the patch introduced by
>>> Ed
>>> Bartosh three years ago (https://patchwork.openembedded.org/patch/90419/)
>>> because I don't need a custom PSEUDO_PREFIX. But maybe not everyone
>>> agrees.
>>>
>>> On 19.01.2018 17:00, Volker Vogelhuber wrote:
>>>>
>>>>
>>>> I'm currently trying to create a multi RootFS WIC image as mentioned in
>>>> directdisk-multi-rootfs.wks. For that I have two image recipes. One that
>>>> is creating only an ext4 image (image-recovery), and the second that is
>>>> also
>>>> creating a WIC image (image-main). I used the IMAGE_FSTYPES variable for
>>>> that. The WKS file for the second image is integrating the recovery
>>>> image by
>>>> specifying --rootfs-dir=image-recovery in it's part description.
>>>>
>>>> # primary / recovery image
>>>> part / --source rootfs --rootfs-dir=image-main --exclude-path mnt/data/
>>>> mnt/data2/ --fstype=ext4 --label primary_rootfs --align 1024 --size 700
>>>> --overhead-factor=1.0
>>>> part /recovery --source rootfs --rootfs-dir=image-recovery --fstype=ext4
>>>> --label recovery_rootfs --align 1024 --size 640 --overhead-factor=1.0
>>>>
>>>> The UIDs of the second rootfs (image-main) are correctly set to 0 within
>>>> the file system when calling mkfs.ext4 during prepare_rootfs_ext. For
>>>> the
>>>> recovery rootfs the UID is always set to my own (host) one which is of
>>>> course not valid for the image where that UID does not exist.
>>>> I tried calling the mkfs.ext4 command myself from a terminal and for
>>>> whatever reason an image created out of the rootfs folder of the second
>>>> image (image-main) recipe is deployed with the correct UID 0, while the
>>>> rootfs folder of the first image (image-recovery) recipe always uses the
>>>> UID
>>>> of the source folder/files.
>>>>
>>>> I search the code of e2fsprogs for the line that sets the UID and added
>>>> a
>>>> printf in set_inode_extra. There I can see clearly that the source UID
>>>> for
>>>> the file is 0 for the rootfs of the image-main rootfs folder while it is
>>>> 10000 (my own UID) for the image-recovery. I wonder how the UID of the
>>>> image-main rootfs folder can be zero when I don't call any command with
>>>> root
>>>> permissions. I searched for a preparation step where the UIDs are
>>>> managed in
>>>> the scripts folder of Yocto, but didn't found any hint for the whole
>>>> behavior. So while it is good that the rootfs partition of the main
>>>> rootfs
>>>> has the UID set correctly to zero, I can't understand why it happens. On
>>>> the
>>>> other side I can understand why the UID of the recovery rootfs is set to
>>>> my
>>>> own one, but it stops me from booting that rootfs because the UIDs of
>>>> the
>>>> files and folders are set to a user that does not exist on the target
>>>> system.
>>>>
>>>> Can someone please explain to me, how that UID handling is meant to be
>>>> done?
>>>
>>>
>>>
>>> --
>>>
>



-- 
Ricardo Ribalda

Re: Question about multi rootfs UIDs when using wic

[Ricardo Ribalda Delgado]

cc: ed

Is it a very bad idea to revert your patch?

Thanks!

On Wed, May 2, 2018 at 8:17 PM, Ricardo Ribalda Delgado
<ricardo.ribalda@gmail.com> wrote:
> Hi
>
> I do not know how many people is actively using wic. We have started
> using it from last week, it is hard to replace old scripts that work
> :)
>
> Try sending it as a proper patch, I can help you if you need it.
>
> Regards
>
> On Wed, May 2, 2018 at 6:31 PM, Volker Vogelhuber
> <v.vogelhuber@digitalendoscopy.de> wrote:
>> Hi,
>>
>> I never got any response to my message (until now). So AFAIK I solved the
>> problem with the following patch:
>>
>> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
>> index 84fe85d62b..66ab757aec 100644
>> --- a/scripts/lib/wic/partition.py
>> +++ b/scripts/lib/wic/partition.py
>> @@ -204,15 +204,10 @@ class Partition():
>>
>>          Currently handles ext2/3/4, btrfs and vfat.
>>          """
>> -        p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" %
>> native_sysroot)
>> -        p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
>> -                                         "%s/../pseudo" %
>> get_bitbake_var("IMAGE_ROOTFS"))
>> -        p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
>> +        pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
>> +        pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" %
>> self.rootfs_dir
>> +        pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
>>          p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
>> -        pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
>> -        pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % p_localstatedir
>> -        pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd
>> -        pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp
>>          pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
>>
>>          rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
>>
>>
>>
>> On 02.05.2018 17:51, Ricardo Ribalda Delgado wrote:
>>>
>>> Hi
>>>
>>> I just got hit by this one. It is specially nasty because nfsroot
>>> fails to boot if the uids are wrong.
>>>
>>> What is the status on this?
>>>
>>> On Mon, Jan 22, 2018 at 11:39 AM, Volker Vogelhuber
>>> <v.vogelhuber@digitalendoscopy.de> wrote:
>>>>
>>>> I finally found out, what's the reason why the included recovery rootfs
>>>> has
>>>> the wrong UIDs, while the main image has the correct one. The reason
>>>> seems
>>>> to be the PSEUDO_LOCALSTATEDIR variable that is set to the state dir of
>>>> the
>>>> main image in both cases.
>>>> I debugged all the calls up to the environment setup in partition.py's
>>>> prepare_rootfs method where a check for an existing PSEUDO_LOCALSTATEDIR
>>>> environment variable is done. That seems to be the problem.
>>>> Instead of using the correct value passed to the prepare_rootfs method,
>>>> an
>>>> existing ENV value is used that points to the state dir of the main image
>>>> instead of the recovery one's. I guess the reason is that in bitbake.conf
>>>> the PSEUDO_LOCALSTATEDIR is set to the image currently build (which is
>>>> the
>>>> main image and not the recovery image only referenced by the main image).
>>>> So
>>>> because that environment variable is already set, the call to mkfs.ext4
>>>> for
>>>> the recovery image uses the wrong PSEUDO_LOCALSTATEDIR and does not apply
>>>> the correct UID.
>>>>
>>>> Any ideas how to fix that? I tend to just remove the patch introduced by
>>>> Ed
>>>> Bartosh three years ago (https://patchwork.openembedded.org/patch/90419/)
>>>> because I don't need a custom PSEUDO_PREFIX. But maybe not everyone
>>>> agrees.
>>>>
>>>> On 19.01.2018 17:00, Volker Vogelhuber wrote:
>>>>>
>>>>>
>>>>> I'm currently trying to create a multi RootFS WIC image as mentioned in
>>>>> directdisk-multi-rootfs.wks. For that I have two image recipes. One that
>>>>> is creating only an ext4 image (image-recovery), and the second that is
>>>>> also
>>>>> creating a WIC image (image-main). I used the IMAGE_FSTYPES variable for
>>>>> that. The WKS file for the second image is integrating the recovery
>>>>> image by
>>>>> specifying --rootfs-dir=image-recovery in it's part description.
>>>>>
>>>>> # primary / recovery image
>>>>> part / --source rootfs --rootfs-dir=image-main --exclude-path mnt/data/
>>>>> mnt/data2/ --fstype=ext4 --label primary_rootfs --align 1024 --size 700
>>>>> --overhead-factor=1.0
>>>>> part /recovery --source rootfs --rootfs-dir=image-recovery --fstype=ext4
>>>>> --label recovery_rootfs --align 1024 --size 640 --overhead-factor=1.0
>>>>>
>>>>> The UIDs of the second rootfs (image-main) are correctly set to 0 within
>>>>> the file system when calling mkfs.ext4 during prepare_rootfs_ext. For
>>>>> the
>>>>> recovery rootfs the UID is always set to my own (host) one which is of
>>>>> course not valid for the image where that UID does not exist.
>>>>> I tried calling the mkfs.ext4 command myself from a terminal and for
>>>>> whatever reason an image created out of the rootfs folder of the second
>>>>> image (image-main) recipe is deployed with the correct UID 0, while the
>>>>> rootfs folder of the first image (image-recovery) recipe always uses the
>>>>> UID
>>>>> of the source folder/files.
>>>>>
>>>>> I search the code of e2fsprogs for the line that sets the UID and added
>>>>> a
>>>>> printf in set_inode_extra. There I can see clearly that the source UID
>>>>> for
>>>>> the file is 0 for the rootfs of the image-main rootfs folder while it is
>>>>> 10000 (my own UID) for the image-recovery. I wonder how the UID of the
>>>>> image-main rootfs folder can be zero when I don't call any command with
>>>>> root
>>>>> permissions. I searched for a preparation step where the UIDs are
>>>>> managed in
>>>>> the scripts folder of Yocto, but didn't found any hint for the whole
>>>>> behavior. So while it is good that the rootfs partition of the main
>>>>> rootfs
>>>>> has the UID set correctly to zero, I can't understand why it happens. On
>>>>> the
>>>>> other side I can understand why the UID of the recovery rootfs is set to
>>>>> my
>>>>> own one, but it stops me from booting that rootfs because the UIDs of
>>>>> the
>>>>> files and folders are set to a user that does not exist on the target
>>>>> system.
>>>>>
>>>>> Can someone please explain to me, how that UID handling is meant to be
>>>>> done?
>>>>
>>>>
>>>>
>>>> --
>>>>
>>
>
>
>
> --
> Ricardo Ribalda



-- 
Ricardo Ribalda

Re: Question about multi rootfs UIDs when using wic

[Ricardo Ribalda Delgado]

Hi Volker

I have tried your patch and does not work for me :(. My configuration
is a little bit more complex, I have two different machines.

The images are generated with:

host-kiosk.wks:

part /boot/bootloader --source qbootimg-efi --ondisk sda --active
--align 1024 --use-uuid --fsoptions defaults,ro
part /export --source rootfs --rootfs-dir=kiosk-image --ondisk sda
--fstype=ext4 --label export --align 1024 --use-uuid --fsoptions
defaults,ro
part / --source rootfs --ondisk sda --fstype=ext4 --label rootfs
--align 1024 --use-uuid --fsoptions defaults,ro

bootloader --ptable gpt --timeout=1 --append="ro rootfstype=ext4
qtec_mem.size=64


MACHINE=qt5022 bitbake host-image ; MACHINE=qt5122 bitbake kiosk-image
wic create host-kiosk -e host-image --rootfs-dir
kiosk-image=/workdir/build/tmp/work/qt5122-poky-linux/kiosk-image/1.0-r0/rootfs/
--rootfs-dir /workdir/build/tmp/work/qt5022-poky-linux/host-image/1.0-r0/rootfs/
-m


In my case this patch does the trick

commit 269334f2400c6ee0c7b4fec7d9bf0f701e50c329 (HEAD -> europa, origin/europa)
Author: Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>
Date:   Mon May 7 13:54:15 2018 +0200

    wic: Fix partition files UIDs on multi rootfs images

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 84fe85d62b..49921e7494 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -206,7 +206,7 @@ class Partition():
         """
         p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
         p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
-                                         "%s/../pseudo" %
get_bitbake_var("IMAGE_ROOTFS"))
+                                         "%s/../pseudo" %  rootfs_dir)
         p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
         p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
         pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
diff --git a/scripts/lib/wic/plugins/imager/direct.py
b/scripts/lib/wic/plugins/imager/direct.py
index da1c061063..a90a47c926 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -121,6 +121,10 @@ class DirectPlugin(ImagerPlugin):
         if self._update_fstab(fstab_lines, self.parts):
             # copy rootfs dir to workdir to update fstab
             # as rootfs can be used by other tasks and can't be modified
+            new_pseudo = os.path.realpath(os.path.join(self.workdir, "pseudo"))
+            from_dir = os.path.join(os.path.join(image_rootfs, ".."), "pseudo")
+            from_dir = os.path.realpath(from_dir)
+            copyhardlinktree(from_dir, new_pseudo)
             new_rootfs = os.path.realpath(os.path.join(self.workdir,
"rootfs_copy"))
             copyhardlinktree(image_rootfs, new_rootfs)
             fstab_path = os.path.join(new_rootfs, 'etc/fstab')



Can you try it on your setup? If it also works for you I will work on
getting it merged.


Cheers!

On Wed, May 2, 2018 at 8:25 PM, Ricardo Ribalda Delgado
<ricardo.ribalda@gmail.com> wrote:
> cc: ed
>
> Is it a very bad idea to revert your patch?
>
> Thanks!
>
> On Wed, May 2, 2018 at 8:17 PM, Ricardo Ribalda Delgado
> <ricardo.ribalda@gmail.com> wrote:
>> Hi
>>
>> I do not know how many people is actively using wic. We have started
>> using it from last week, it is hard to replace old scripts that work
>> :)
>>
>> Try sending it as a proper patch, I can help you if you need it.
>>
>> Regards
>>
>> On Wed, May 2, 2018 at 6:31 PM, Volker Vogelhuber
>> <v.vogelhuber@digitalendoscopy.de> wrote:
>>> Hi,
>>>
>>> I never got any response to my message (until now). So AFAIK I solved the
>>> problem with the following patch:
>>>
>>> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
>>> index 84fe85d62b..66ab757aec 100644
>>> --- a/scripts/lib/wic/partition.py
>>> +++ b/scripts/lib/wic/partition.py
>>> @@ -204,15 +204,10 @@ class Partition():
>>>
>>>          Currently handles ext2/3/4, btrfs and vfat.
>>>          """
>>> -        p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" %
>>> native_sysroot)
>>> -        p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
>>> -                                         "%s/../pseudo" %
>>> get_bitbake_var("IMAGE_ROOTFS"))
>>> -        p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
>>> +        pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
>>> +        pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" %
>>> self.rootfs_dir
>>> +        pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
>>>          p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
>>> -        pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
>>> -        pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % p_localstatedir
>>> -        pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd
>>> -        pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp
>>>          pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
>>>
>>>          rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
>>>
>>>
>>>
>>> On 02.05.2018 17:51, Ricardo Ribalda Delgado wrote:
>>>>
>>>> Hi
>>>>
>>>> I just got hit by this one. It is specially nasty because nfsroot
>>>> fails to boot if the uids are wrong.
>>>>
>>>> What is the status on this?
>>>>
>>>> On Mon, Jan 22, 2018 at 11:39 AM, Volker Vogelhuber
>>>> <v.vogelhuber@digitalendoscopy.de> wrote:
>>>>>
>>>>> I finally found out, what's the reason why the included recovery rootfs
>>>>> has
>>>>> the wrong UIDs, while the main image has the correct one. The reason
>>>>> seems
>>>>> to be the PSEUDO_LOCALSTATEDIR variable that is set to the state dir of
>>>>> the
>>>>> main image in both cases.
>>>>> I debugged all the calls up to the environment setup in partition.py's
>>>>> prepare_rootfs method where a check for an existing PSEUDO_LOCALSTATEDIR
>>>>> environment variable is done. That seems to be the problem.
>>>>> Instead of using the correct value passed to the prepare_rootfs method,
>>>>> an
>>>>> existing ENV value is used that points to the state dir of the main image
>>>>> instead of the recovery one's. I guess the reason is that in bitbake.conf
>>>>> the PSEUDO_LOCALSTATEDIR is set to the image currently build (which is
>>>>> the
>>>>> main image and not the recovery image only referenced by the main image).
>>>>> So
>>>>> because that environment variable is already set, the call to mkfs.ext4
>>>>> for
>>>>> the recovery image uses the wrong PSEUDO_LOCALSTATEDIR and does not apply
>>>>> the correct UID.
>>>>>
>>>>> Any ideas how to fix that? I tend to just remove the patch introduced by
>>>>> Ed
>>>>> Bartosh three years ago (https://patchwork.openembedded.org/patch/90419/)
>>>>> because I don't need a custom PSEUDO_PREFIX. But maybe not everyone
>>>>> agrees.
>>>>>
>>>>> On 19.01.2018 17:00, Volker Vogelhuber wrote:
>>>>>>
>>>>>>
>>>>>> I'm currently trying to create a multi RootFS WIC image as mentioned in
>>>>>> directdisk-multi-rootfs.wks. For that I have two image recipes. One that
>>>>>> is creating only an ext4 image (image-recovery), and the second that is
>>>>>> also
>>>>>> creating a WIC image (image-main). I used the IMAGE_FSTYPES variable for
>>>>>> that. The WKS file for the second image is integrating the recovery
>>>>>> image by
>>>>>> specifying --rootfs-dir=image-recovery in it's part description.
>>>>>>
>>>>>> # primary / recovery image
>>>>>> part / --source rootfs --rootfs-dir=image-main --exclude-path mnt/data/
>>>>>> mnt/data2/ --fstype=ext4 --label primary_rootfs --align 1024 --size 700
>>>>>> --overhead-factor=1.0
>>>>>> part /recovery --source rootfs --rootfs-dir=image-recovery --fstype=ext4
>>>>>> --label recovery_rootfs --align 1024 --size 640 --overhead-factor=1.0
>>>>>>
>>>>>> The UIDs of the second rootfs (image-main) are correctly set to 0 within
>>>>>> the file system when calling mkfs.ext4 during prepare_rootfs_ext. For
>>>>>> the
>>>>>> recovery rootfs the UID is always set to my own (host) one which is of
>>>>>> course not valid for the image where that UID does not exist.
>>>>>> I tried calling the mkfs.ext4 command myself from a terminal and for
>>>>>> whatever reason an image created out of the rootfs folder of the second
>>>>>> image (image-main) recipe is deployed with the correct UID 0, while the
>>>>>> rootfs folder of the first image (image-recovery) recipe always uses the
>>>>>> UID
>>>>>> of the source folder/files.
>>>>>>
>>>>>> I search the code of e2fsprogs for the line that sets the UID and added
>>>>>> a
>>>>>> printf in set_inode_extra. There I can see clearly that the source UID
>>>>>> for
>>>>>> the file is 0 for the rootfs of the image-main rootfs folder while it is
>>>>>> 10000 (my own UID) for the image-recovery. I wonder how the UID of the
>>>>>> image-main rootfs folder can be zero when I don't call any command with
>>>>>> root
>>>>>> permissions. I searched for a preparation step where the UIDs are
>>>>>> managed in
>>>>>> the scripts folder of Yocto, but didn't found any hint for the whole
>>>>>> behavior. So while it is good that the rootfs partition of the main
>>>>>> rootfs
>>>>>> has the UID set correctly to zero, I can't understand why it happens. On
>>>>>> the
>>>>>> other side I can understand why the UID of the recovery rootfs is set to
>>>>>> my
>>>>>> own one, but it stops me from booting that rootfs because the UIDs of
>>>>>> the
>>>>>> files and folders are set to a user that does not exist on the target
>>>>>> system.
>>>>>>
>>>>>> Can someone please explain to me, how that UID handling is meant to be
>>>>>> done?
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>
>>
>>
>>
>> --
>> Ricardo Ribalda
>
>
>
> --
> Ricardo Ribalda



-- 
Ricardo Ribalda

Re: Question about multi rootfs UIDs when using wic

[Volker Vogelhuber]

Hi Ricardo,

On 07.05.2018 15:22, Ricardo Ribalda Delgado wrote:
> In my case this patch does the trick
Unfortunately your patch results in having the recovery image's UID and 
GID settings to be set to my current user instead of root. So no, your 
patch does not seem to work for me.
I didn't have the time to check in detail what's wrong with your version.

> 
> commit 269334f2400c6ee0c7b4fec7d9bf0f701e50c329 (HEAD -> europa, origin/europa)
> Author: Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>
> Date:   Mon May 7 13:54:15 2018 +0200
> 
>      wic: Fix partition files UIDs on multi rootfs images
> 
> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> index 84fe85d62b..49921e7494 100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -206,7 +206,7 @@ class Partition():
>           """
>           p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
>           p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
> -                                         "%s/../pseudo" %
> get_bitbake_var("IMAGE_ROOTFS"))
> +                                         "%s/../pseudo" %  rootfs_dir)
>           p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
>           p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
>           pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
> diff --git a/scripts/lib/wic/plugins/imager/direct.py
> b/scripts/lib/wic/plugins/imager/direct.py
> index da1c061063..a90a47c926 100644
> --- a/scripts/lib/wic/plugins/imager/direct.py
> +++ b/scripts/lib/wic/plugins/imager/direct.py
> @@ -121,6 +121,10 @@ class DirectPlugin(ImagerPlugin):
>           if self._update_fstab(fstab_lines, self.parts):
>               # copy rootfs dir to workdir to update fstab
>               # as rootfs can be used by other tasks and can't be modified
> +            new_pseudo = os.path.realpath(os.path.join(self.workdir, "pseudo"))
> +            from_dir = os.path.join(os.path.join(image_rootfs, ".."), "pseudo")
> +            from_dir = os.path.realpath(from_dir)
> +            copyhardlinktree(from_dir, new_pseudo)
>               new_rootfs = os.path.realpath(os.path.join(self.workdir,
> "rootfs_copy"))
>               copyhardlinktree(image_rootfs, new_rootfs)
>               fstab_path = os.path.join(new_rootfs, 'etc/fstab')
> 
> 
> 
> Can you try it on your setup? If it also works for you I will work on
> getting it merged.
> 
> 
> Cheers!
> 
> On Wed, May 2, 2018 at 8:25 PM, Ricardo Ribalda Delgado
> <ricardo.ribalda@gmail.com> wrote:
>> cc: ed
>>
>> Is it a very bad idea to revert your patch?
>>
>> Thanks!
>>
>> On Wed, May 2, 2018 at 8:17 PM, Ricardo Ribalda Delgado
>> <ricardo.ribalda@gmail.com> wrote:
>>> Hi
>>>
>>> I do not know how many people is actively using wic. We have started
>>> using it from last week, it is hard to replace old scripts that work
>>> :)
>>>
>>> Try sending it as a proper patch, I can help you if you need it.
>>>
>>> Regards
>>>
>>> On Wed, May 2, 2018 at 6:31 PM, Volker Vogelhuber
>>> <v.vogelhuber@digitalendoscopy.de> wrote:
>>>> Hi,
>>>>
>>>> I never got any response to my message (until now). So AFAIK I solved the
>>>> problem with the following patch:
>>>>
>>>> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
>>>> index 84fe85d62b..66ab757aec 100644
>>>> --- a/scripts/lib/wic/partition.py
>>>> +++ b/scripts/lib/wic/partition.py
>>>> @@ -204,15 +204,10 @@ class Partition():
>>>>
>>>>           Currently handles ext2/3/4, btrfs and vfat.
>>>>           """
>>>> -        p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" %
>>>> native_sysroot)
>>>> -        p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
>>>> -                                         "%s/../pseudo" %
>>>> get_bitbake_var("IMAGE_ROOTFS"))
>>>> -        p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
>>>> +        pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
>>>> +        pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" %
>>>> self.rootfs_dir
>>>> +        pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
>>>>           p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
>>>> -        pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
>>>> -        pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % p_localstatedir
>>>> -        pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd
>>>> -        pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp
>>>>           pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
>>>>
>>>>           rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
>>>>
>>>>
>>>>
>>>> On 02.05.2018 17:51, Ricardo Ribalda Delgado wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> I just got hit by this one. It is specially nasty because nfsroot
>>>>> fails to boot if the uids are wrong.
>>>>>
>>>>> What is the status on this?
>>>>>
>>>>> On Mon, Jan 22, 2018 at 11:39 AM, Volker Vogelhuber
>>>>> <v.vogelhuber@digitalendoscopy.de> wrote:
>>>>>>
>>>>>> I finally found out, what's the reason why the included recovery rootfs
>>>>>> has
>>>>>> the wrong UIDs, while the main image has the correct one. The reason
>>>>>> seems
>>>>>> to be the PSEUDO_LOCALSTATEDIR variable that is set to the state dir of
>>>>>> the
>>>>>> main image in both cases.
>>>>>> I debugged all the calls up to the environment setup in partition.py's
>>>>>> prepare_rootfs method where a check for an existing PSEUDO_LOCALSTATEDIR
>>>>>> environment variable is done. That seems to be the problem.
>>>>>> Instead of using the correct value passed to the prepare_rootfs method,
>>>>>> an
>>>>>> existing ENV value is used that points to the state dir of the main image
>>>>>> instead of the recovery one's. I guess the reason is that in bitbake.conf
>>>>>> the PSEUDO_LOCALSTATEDIR is set to the image currently build (which is
>>>>>> the
>>>>>> main image and not the recovery image only referenced by the main image).
>>>>>> So
>>>>>> because that environment variable is already set, the call to mkfs.ext4
>>>>>> for
>>>>>> the recovery image uses the wrong PSEUDO_LOCALSTATEDIR and does not apply
>>>>>> the correct UID.
>>>>>>
>>>>>> Any ideas how to fix that? I tend to just remove the patch introduced by
>>>>>> Ed
>>>>>> Bartosh three years ago (https://patchwork.openembedded.org/patch/90419/)
>>>>>> because I don't need a custom PSEUDO_PREFIX. But maybe not everyone
>>>>>> agrees.
>>>>>>
>>>>>> On 19.01.2018 17:00, Volker Vogelhuber wrote:
>>>>>>>
>>>>>>>
>>>>>>> I'm currently trying to create a multi RootFS WIC image as mentioned in
>>>>>>> directdisk-multi-rootfs.wks. For that I have two image recipes. One that
>>>>>>> is creating only an ext4 image (image-recovery), and the second that is
>>>>>>> also
>>>>>>> creating a WIC image (image-main). I used the IMAGE_FSTYPES variable for
>>>>>>> that. The WKS file for the second image is integrating the recovery
>>>>>>> image by
>>>>>>> specifying --rootfs-dir=image-recovery in it's part description.
>>>>>>>
>>>>>>> # primary / recovery image
>>>>>>> part / --source rootfs --rootfs-dir=image-main --exclude-path mnt/data/
>>>>>>> mnt/data2/ --fstype=ext4 --label primary_rootfs --align 1024 --size 700
>>>>>>> --overhead-factor=1.0
>>>>>>> part /recovery --source rootfs --rootfs-dir=image-recovery --fstype=ext4
>>>>>>> --label recovery_rootfs --align 1024 --size 640 --overhead-factor=1.0
>>>>>>>
>>>>>>> The UIDs of the second rootfs (image-main) are correctly set to 0 within
>>>>>>> the file system when calling mkfs.ext4 during prepare_rootfs_ext. For
>>>>>>> the
>>>>>>> recovery rootfs the UID is always set to my own (host) one which is of
>>>>>>> course not valid for the image where that UID does not exist.
>>>>>>> I tried calling the mkfs.ext4 command myself from a terminal and for
>>>>>>> whatever reason an image created out of the rootfs folder of the second
>>>>>>> image (image-main) recipe is deployed with the correct UID 0, while the
>>>>>>> rootfs folder of the first image (image-recovery) recipe always uses the
>>>>>>> UID
>>>>>>> of the source folder/files.
>>>>>>>
>>>>>>> I search the code of e2fsprogs for the line that sets the UID and added
>>>>>>> a
>>>>>>> printf in set_inode_extra. There I can see clearly that the source UID
>>>>>>> for
>>>>>>> the file is 0 for the rootfs of the image-main rootfs folder while it is
>>>>>>> 10000 (my own UID) for the image-recovery. I wonder how the UID of the
>>>>>>> image-main rootfs folder can be zero when I don't call any command with
>>>>>>> root
>>>>>>> permissions. I searched for a preparation step where the UIDs are
>>>>>>> managed in
>>>>>>> the scripts folder of Yocto, but didn't found any hint for the whole
>>>>>>> behavior. So while it is good that the rootfs partition of the main
>>>>>>> rootfs
>>>>>>> has the UID set correctly to zero, I can't understand why it happens. On
>>>>>>> the
>>>>>>> other side I can understand why the UID of the recovery rootfs is set to
>>>>>>> my
>>>>>>> own one, but it stops me from booting that rootfs because the UIDs of
>>>>>>> the
>>>>>>> files and folders are set to a user that does not exist on the target
>>>>>>> system.
>>>>>>>
>>>>>>> Can someone please explain to me, how that UID handling is meant to be
>>>>>>> done?
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Ricardo Ribalda
>>
>>
>>
>> --
>> Ricardo Ribalda
> 
> 
> 

Re: Question about multi rootfs UIDs when using wic

[Ricardo Ribalda Delgado]

Hello

So this is what I have tried out:

I have downloaded master from openembedded-core
(f04e6bd144deb0c8fe2742f66b18904b6619a502)
then:
bitbake core-image-minimal core-image-base syslinux

Create a file  test.wks
part / --source rootfs --fstype=ext4 --rootfs-dir=core-image-base
part /mnt/data1/ --fstype=ext4 --source rootfs
--rootfs-dir=core-image-minimal
part /mnt/data2/ --fstype=ext4 --source rootfs
--rootfs-dir=core-image-minimal

bootloader --timeout=0


Then:
losetup --partscan --find --read-only --show test*direct
sudo mount /dev/loop0p1 /mnt
ls -lan /mnt
total 40
drwxr-xr-x 18    0    0  1024 May 11 16:11 .
drwxr-xr-x 25    0    0  4096 May  6 06:31 ..
drwxr-xr-x  2 1000 1000  3072 May 11 15:51 bin
drwxr-xr-x  2 1000 1000  1024 May 11 13:59 boot
drwxr-xr-x  2 1000 1000  1024 May 11 13:59 dev
drwxr-xr-x 25 1000 1000  3072 May 11 15:51 etc
drwxr-xr-x  3 1000 1000  1024 May 11 13:59 home
drwxr-xr-x  6 1000 1000  3072 May 11 15:51 lib
drwx------  2    0    0 12288 May 11 16:11 lost+found
drwxr-xr-x  2 1000 1000  1024 May 11 13:59 media
drwxr-xr-x  3 1000 1000  1024 May 11 15:50 mnt
drwxr-xr-x  2 1000 1000  1024 May 11 13:59 proc
drwxr-xr-x  2 1000 1000  1024 May 11 15:51 run
drwxr-xr-x  2 1000 1000  3072 May 11 15:51 sbin
drwxr-xr-x  2 1000 1000  1024 May 11 13:59 sys
drwxr-xr-t  2 1000 1000  1024 May 11 13:59 tmp
drwxr-xr-x 10 1000 1000  1024 May 11 14:54 usr
drwxr-xr-x  8 1000 1000  1024 May 11 14:55 var


With my patch:
ricardo@neopili:/tmp/openembedded-core/build$ ls -lan /mnt
total 40
drwxr-xr-x 18 0 0  1024 May 11 16:18 .
drwxr-xr-x 25 0 0  4096 May  6 06:31 ..
drwxr-xr-x  2 0 0  3072 May 11 15:51 bin
drwxr-xr-x  2 0 0  1024 May 11 13:59 boot
drwxr-xr-x  2 0 0  1024 May 11 13:59 dev
drwxr-xr-x 25 0 0  3072 May 11 15:51 etc
drwxr-xr-x  3 0 0  1024 May 11 13:59 home
drwxr-xr-x  6 0 0  3072 May 11 15:51 lib
drwx------  2 0 0 12288 May 11 16:18 lost+found
drwxr-xr-x  2 0 0  1024 May 11 13:59 media
drwxr-xr-x  3 0 0  1024 May 11 15:50 mnt
drwxr-xr-x  2 0 0  1024 May 11 13:59 proc
drwxr-xr-x  2 0 0  1024 May 11 15:51 run
drwxr-xr-x  2 0 0  3072 May 11 15:51 sbin
drwxr-xr-x  2 0 0  1024 May 11 13:59 sys
drwxrwxrwt  2 0 0  1024 May 11 13:59 tmp
drwxr-xr-x 10 0 0  1024 May 11 14:54 usr
drwxr-xr-x  8 0 0  1024 May 11 14:55 var


I think that I am going to send a formal patch to the mailing list and see
if we get some feedback


Cheers!

On Tue, May 8, 2018 at 3:39 PM Volker Vogelhuber <
v.vogelhuber@digitalendoscopy.de> wrote:

> Hi Ricardo,

> On 07.05.2018 15:22, Ricardo Ribalda Delgado wrote:
> > In my case this patch does the trick
> Unfortunately your patch results in having the recovery image's UID and
> GID settings to be set to my current user instead of root. So no, your
> patch does not seem to work for me.
> I didn't have the time to check in detail what's wrong with your version.

> >
> > commit 269334f2400c6ee0c7b4fec7d9bf0f701e50c329 (HEAD -> europa,
origin/europa)
> > Author: Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>
> > Date:   Mon May 7 13:54:15 2018 +0200
> >
> >      wic: Fix partition files UIDs on multi rootfs images
> >
> > diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> > index 84fe85d62b..49921e7494 100644
> > --- a/scripts/lib/wic/partition.py
> > +++ b/scripts/lib/wic/partition.py
> > @@ -206,7 +206,7 @@ class Partition():
> >           """
> >           p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" %
native_sysroot)
> >           p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
> > -                                         "%s/../pseudo" %
> > get_bitbake_var("IMAGE_ROOTFS"))
> > +                                         "%s/../pseudo" %  rootfs_dir)
> >           p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
> >           p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
> >           pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
> > diff --git a/scripts/lib/wic/plugins/imager/direct.py
> > b/scripts/lib/wic/plugins/imager/direct.py
> > index da1c061063..a90a47c926 100644
> > --- a/scripts/lib/wic/plugins/imager/direct.py
> > +++ b/scripts/lib/wic/plugins/imager/direct.py
> > @@ -121,6 +121,10 @@ class DirectPlugin(ImagerPlugin):
> >           if self._update_fstab(fstab_lines, self.parts):
> >               # copy rootfs dir to workdir to update fstab
> >               # as rootfs can be used by other tasks and can't be
modified
> > +            new_pseudo = os.path.realpath(os.path.join(self.workdir,
"pseudo"))
> > +            from_dir = os.path.join(os.path.join(image_rootfs, ".."),
"pseudo")
> > +            from_dir = os.path.realpath(from_dir)
> > +            copyhardlinktree(from_dir, new_pseudo)
> >               new_rootfs = os.path.realpath(os.path.join(self.workdir,
> > "rootfs_copy"))
> >               copyhardlinktree(image_rootfs, new_rootfs)
> >               fstab_path = os.path.join(new_rootfs, 'etc/fstab')
> >
> >
> >
> > Can you try it on your setup? If it also works for you I will work on
> > getting it merged.
> >
> >
> > Cheers!
> >
> > On Wed, May 2, 2018 at 8:25 PM, Ricardo Ribalda Delgado
> > <ricardo.ribalda@gmail.com> wrote:
> >> cc: ed
> >>
> >> Is it a very bad idea to revert your patch?
> >>
> >> Thanks!
> >>
> >> On Wed, May 2, 2018 at 8:17 PM, Ricardo Ribalda Delgado
> >> <ricardo.ribalda@gmail.com> wrote:
> >>> Hi
> >>>
> >>> I do not know how many people is actively using wic. We have started
> >>> using it from last week, it is hard to replace old scripts that work
> >>> :)
> >>>
> >>> Try sending it as a proper patch, I can help you if you need it.
> >>>
> >>> Regards
> >>>
> >>> On Wed, May 2, 2018 at 6:31 PM, Volker Vogelhuber
> >>> <v.vogelhuber@digitalendoscopy.de> wrote:
> >>>> Hi,
> >>>>
> >>>> I never got any response to my message (until now). So AFAIK I
solved the
> >>>> problem with the following patch:
> >>>>
> >>>> diff --git a/scripts/lib/wic/partition.py
b/scripts/lib/wic/partition.py
> >>>> index 84fe85d62b..66ab757aec 100644
> >>>> --- a/scripts/lib/wic/partition.py
> >>>> +++ b/scripts/lib/wic/partition.py
> >>>> @@ -204,15 +204,10 @@ class Partition():
> >>>>
> >>>>           Currently handles ext2/3/4, btrfs and vfat.
> >>>>           """
> >>>> -        p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" %
> >>>> native_sysroot)
> >>>> -        p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
> >>>> -                                         "%s/../pseudo" %
> >>>> get_bitbake_var("IMAGE_ROOTFS"))
> >>>> -        p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
> >>>> +        pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
> >>>> +        pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" %
> >>>> self.rootfs_dir
> >>>> +        pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
> >>>>           p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
> >>>> -        pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
> >>>> -        pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" %
p_localstatedir
> >>>> -        pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd
> >>>> -        pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp
> >>>>           pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
> >>>>
> >>>>           rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
> >>>>
> >>>>
> >>>>
> >>>> On 02.05.2018 17:51, Ricardo Ribalda Delgado wrote:
> >>>>>
> >>>>> Hi
> >>>>>
> >>>>> I just got hit by this one. It is specially nasty because nfsroot
> >>>>> fails to boot if the uids are wrong.
> >>>>>
> >>>>> What is the status on this?
> >>>>>
> >>>>> On Mon, Jan 22, 2018 at 11:39 AM, Volker Vogelhuber
> >>>>> <v.vogelhuber@digitalendoscopy.de> wrote:
> >>>>>>
> >>>>>> I finally found out, what's the reason why the included recovery
rootfs
> >>>>>> has
> >>>>>> the wrong UIDs, while the main image has the correct one. The
reason
> >>>>>> seems
> >>>>>> to be the PSEUDO_LOCALSTATEDIR variable that is set to the state
dir of
> >>>>>> the
> >>>>>> main image in both cases.
> >>>>>> I debugged all the calls up to the environment setup in
partition.py's
> >>>>>> prepare_rootfs method where a check for an existing
PSEUDO_LOCALSTATEDIR
> >>>>>> environment variable is done. That seems to be the problem.
> >>>>>> Instead of using the correct value passed to the prepare_rootfs
method,
> >>>>>> an
> >>>>>> existing ENV value is used that points to the state dir of the
main image
> >>>>>> instead of the recovery one's. I guess the reason is that in
bitbake.conf
> >>>>>> the PSEUDO_LOCALSTATEDIR is set to the image currently build
(which is
> >>>>>> the
> >>>>>> main image and not the recovery image only referenced by the main
image).
> >>>>>> So
> >>>>>> because that environment variable is already set, the call to
mkfs.ext4
> >>>>>> for
> >>>>>> the recovery image uses the wrong PSEUDO_LOCALSTATEDIR and does
not apply
> >>>>>> the correct UID.
> >>>>>>
> >>>>>> Any ideas how to fix that? I tend to just remove the patch
introduced by
> >>>>>> Ed
> >>>>>> Bartosh three years ago (
https://patchwork.openembedded.org/patch/90419/)
> >>>>>> because I don't need a custom PSEUDO_PREFIX. But maybe not everyone
> >>>>>> agrees.
> >>>>>>
> >>>>>> On 19.01.2018 17:00, Volker Vogelhuber wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>> I'm currently trying to create a multi RootFS WIC image as
mentioned in
> >>>>>>> directdisk-multi-rootfs.wks. For that I have two image recipes.
One that
> >>>>>>> is creating only an ext4 image (image-recovery), and the second
that is
> >>>>>>> also
> >>>>>>> creating a WIC image (image-main). I used the IMAGE_FSTYPES
variable for
> >>>>>>> that. The WKS file for the second image is integrating the
recovery
> >>>>>>> image by
> >>>>>>> specifying --rootfs-dir=image-recovery in it's part description.
> >>>>>>>
> >>>>>>> # primary / recovery image
> >>>>>>> part / --source rootfs --rootfs-dir=image-main --exclude-path
mnt/data/
> >>>>>>> mnt/data2/ --fstype=ext4 --label primary_rootfs --align 1024
--size 700
> >>>>>>> --overhead-factor=1.0
> >>>>>>> part /recovery --source rootfs --rootfs-dir=image-recovery
--fstype=ext4
> >>>>>>> --label recovery_rootfs --align 1024 --size 640
--overhead-factor=1.0
> >>>>>>>
> >>>>>>> The UIDs of the second rootfs (image-main) are correctly set to 0
within
> >>>>>>> the file system when calling mkfs.ext4 during prepare_rootfs_ext.
For
> >>>>>>> the
> >>>>>>> recovery rootfs the UID is always set to my own (host) one which
is of
> >>>>>>> course not valid for the image where that UID does not exist.
> >>>>>>> I tried calling the mkfs.ext4 command myself from a terminal and
for
> >>>>>>> whatever reason an image created out of the rootfs folder of the
second
> >>>>>>> image (image-main) recipe is deployed with the correct UID 0,
while the
> >>>>>>> rootfs folder of the first image (image-recovery) recipe always
uses the
> >>>>>>> UID
> >>>>>>> of the source folder/files.
> >>>>>>>
> >>>>>>> I search the code of e2fsprogs for the line that sets the UID and
added
> >>>>>>> a
> >>>>>>> printf in set_inode_extra. There I can see clearly that the
source UID
> >>>>>>> for
> >>>>>>> the file is 0 for the rootfs of the image-main rootfs folder
while it is
> >>>>>>> 10000 (my own UID) for the image-recovery. I wonder how the UID
of the
> >>>>>>> image-main rootfs folder can be zero when I don't call any
command with
> >>>>>>> root
> >>>>>>> permissions. I searched for a preparation step where the UIDs are
> >>>>>>> managed in
> >>>>>>> the scripts folder of Yocto, but didn't found any hint for the
whole
> >>>>>>> behavior. So while it is good that the rootfs partition of the
main
> >>>>>>> rootfs
> >>>>>>> has the UID set correctly to zero, I can't understand why it
happens. On
> >>>>>>> the
> >>>>>>> other side I can understand why the UID of the recovery rootfs is
set to
> >>>>>>> my
> >>>>>>> own one, but it stops me from booting that rootfs because the
UIDs of
> >>>>>>> the
> >>>>>>> files and folders are set to a user that does not exist on the
target
> >>>>>>> system.
> >>>>>>>
> >>>>>>> Can someone please explain to me, how that UID handling is meant
to be
> >>>>>>> done?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Ricardo Ribalda
> >>
> >>
> >>
> >> --
> >> Ricardo Ribalda
> >
> >
> >



--
Ricardo Ribalda