[Draft design][RFC] Running postinst at rootfs generation time

[Draft design][RFC] Running postinst at rootfs generation time

[Cui, Dexuan]

Hi all, below is an initial investigation about the task and we'll continue to further look into it.

In poky we have 2 types of postinst scripts:  one (type-1) can be (and has already been) run at rootfs generation time and the other (type-2) has to be delayed to the first-boot of target device. Type-2 makes target device's first-boot slow and it would be great if we can fix it and convert it to type-1.

We can instrument a first-boot with minimal/sato first to see which postinstalls take the most time and then prioritise those ones to fix.

I figurerd out a list of 33 recipes in total(recipes with the same name but with different versions are counted once) we possibly need to fix.
For the recipes, we need try to find recipe-specific ways(use appropriately modified native utilities to generate caches, files, etc as necessary on the target filesystem).

11 recipes: these could be easily fixed if we add the properly-adjusted utilities "adduser, addgroup, pwconv, etc". Scott is actually adding the utilites: http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=sgarman/useradd-rebased&id=99e54d9696104ed38ec1e3464e17aa1f9b8d98ac
meta/recipes-devtools/distcc/distcc_2.18.3.bb
meta/recipes-extended/cronie/cronie_1.4.7.bb
meta/recipes-extended/at/at_3.1.12.bb:47
meta/recipes-support/hal/hal.inc:45
meta/recipes-core/dbus/dbus.inc:49
meta/recipes-connectivity/openssh/openssh_5.8p2.bb
meta/recipes-connectivity/ppp-dialin/ppp-dialin_0.1.bb
meta/recipes-graphics/x11-common/xserver-nodm-init.bb
meta/recipes-multimedia/pulseaudio/pulseaudio.inc:87
meta/recipes-extended/shadow/shadow_4.1.4.3.bb:125
meta/classes/libc-package.bbclass

6  recipes: these should be easily fixed since the scripts are not related to special native utilites.
meta/recipes-extended/sudo/sudo.inc
meta/recipes-extended/sysklogd/sysklogd.inc
meta/classes/update-rc.d.bbclass
meta/recipes-connectivity/ppp/ppp_2.4.5.bb
meta/recipes-graphics/pango/pango.inc
meta/recipes-gnome/gtk+/gtk+.inc
 
4 recipes: we may need to add gtk-update-icon-cache-native.
meta/classes/gtk-icon-cache.bbclass
meta/recipes-gnome/librsvg/librsvg_2.32.1.bb
meta/recipes-gnome/gdk-pixbuf/gdk-pixbuf_2.22.1.bb
meta/recipes-sato/sato-icon-theme/sato-icon-theme.inc

3 recipes: need to add gconftool-2-native?
meta/classes/gconf.bbclass
meta/recipes-graphics/mutter/mutter.inc
meta/recipes-sato/matchbox-sato/matchbox-session-sato_0.1.bb

3 recipes: "dpkg --configure, opkg-cl configure": looks it's possible to fix them if we specify proper parematers?
meta/recipes-devtools/dpkg/dpkg.inc
meta/recipes-devtools/opkg/opkg_svn.bb
meta/recipes-devtools/opkg/opkg_0.1.8.bb

1 recipe: prelink: we could propablly fix it, but I'm not sure yet.
meta/recipes-devtools/prelink/prelink_git.bb

1 recipe: "/etc/init.d/populate-volatile.sh update ; DBUSPID=`pidof dbus-daemon`": We can't fix this one.
meta/recipes-connectivity/wpa-supplicant/wpa-supplicant-0.7.inc

The below 4 recipes need the related utilities and need more investigation. 

1 recipe: update-modules
meta/recipes-kernel/update-modules/update-modules_1.0.bb

1 recipe: systemctl
meta/recipes-connectivity/avahi/avahi.inc

1 recipe: fc-cache
meta/recipes-graphics/ttf-fonts/liberation-fonts_1.04.bb:37

1 recipe: gtk-query-immodules-2.0
meta/recipes-sato/matchbox-keyboard/matchbox-keyboard_git.bb

Thanks,
-- Dexuan

Re: [Draft design][RFC] Running postinst at rootfs generation time

[Mark Hatle]

On 6/27/11 9:09 PM, Cui, Dexuan wrote:
> Hi all, below is an initial investigation about the task and we'll continue to further look into it.
> 
> In poky we have 2 types of postinst scripts:  one (type-1) can be (and has already been) run at rootfs generation time and the other (type-2) has to be delayed to the first-boot of target device. Type-2 makes target device's first-boot slow and it would be great if we can fix it and convert it to type-1.
> 
> We can instrument a first-boot with minimal/sato first to see which postinstalls take the most time and then prioritise those ones to fix.
> 
> I figurerd out a list of 33 recipes in total(recipes with the same name but with different versions are counted once) we possibly need to fix.
> For the recipes, we need try to find recipe-specific ways(use appropriately modified native utilities to generate caches, files, etc as necessary on the target filesystem).
> 

...

> 1 recipe: prelink: we could propablly fix it, but I'm not sure yet.
> meta/recipes-devtools/prelink/prelink_git.bb

There is nothing to do for the prelinker.  The "image-prelink.bbclass" handles
everything needed to prelink during image creation.  The script is only there
for on-target field upgrades.  So you can remove this from your list.

--Mark

...

> 
> Thanks,
> -- Dexuan
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [Draft design][RFC] Running postinst at rootfs generation time

[Richard Purdie]

On Mon, 2011-06-27 at 23:11 -0500, Mark Hatle wrote:
> On 6/27/11 9:09 PM, Cui, Dexuan wrote:
> > Hi all, below is an initial investigation about the task and we'll continue to further look into it.
> > 
> > In poky we have 2 types of postinst scripts:  one (type-1) can be (and has already been) run at rootfs generation time and the other (type-2) has to be delayed to the first-boot of target device. Type-2 makes target device's first-boot slow and it would be great if we can fix it and convert it to type-1.
> > 
> > We can instrument a first-boot with minimal/sato first to see which postinstalls take the most time and then prioritise those ones to fix.
> > 
> > I figurerd out a list of 33 recipes in total(recipes with the same name but with different versions are counted once) we possibly need to fix.
> > For the recipes, we need try to find recipe-specific ways(use appropriately modified native utilities to generate caches, files, etc as necessary on the target filesystem).
> > 
> 
> ...
> 
> > 1 recipe: prelink: we could propablly fix it, but I'm not sure yet.
> > meta/recipes-devtools/prelink/prelink_git.bb
> 
> There is nothing to do for the prelinker.  The "image-prelink.bbclass" handles
> everything needed to prelink during image creation.  The script is only there
> for on-target field upgrades.  So you can remove this from your list.

Mark, are you 100% sure about this?

It looks like if we install prelink into an image it adds a post install
which runs "prelink -a" on the target device at first boot.

This would happen regardless of whether the cross prelinker did or did
not prelink the image :/.

Cheers,

Richard

Re: [Draft design][RFC] Running postinst at rootfs generation time

[Richard Purdie]

Hi Dexuan,

On Tue, 2011-06-28 at 10:09 +0800, Cui, Dexuan wrote:
> Hi all, below is an initial investigation about the task and we'll
> continue to further look into it.
> 
> In poky we have 2 types of postinst scripts:  one (type-1) can be (and
> has already been) run at rootfs generation time and the other (type-2)
> has to be delayed to the first-boot of target device. Type-2 makes
> target device's first-boot slow and it would be great if we can fix it
> and convert it to type-1.
> 
> We can instrument a first-boot with minimal/sato first to see which
> postinstalls take the most time and then prioritise those ones to fix.
> 
> I figurerd out a list of 33 recipes in total(recipes with the same
> name but with different versions are counted once) we possibly need to
> fix.
> For the recipes, we need try to find recipe-specific ways(use
> appropriately modified native utilities to generate caches, files, etc
> as necessary on the target filesystem).

Out of interest, how long were these different groups of postinstall
taking on the target device?

> 11 recipes: these could be easily fixed if we add the properly-adjusted utilities "adduser, addgroup, pwconv, etc". Scott is actually adding the utilites: http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=sgarman/useradd-rebased&id=99e54d9696104ed38ec1e3464e17aa1f9b8d98ac
> meta/recipes-devtools/distcc/distcc_2.18.3.bb
> meta/recipes-extended/cronie/cronie_1.4.7.bb
> meta/recipes-extended/at/at_3.1.12.bb:47
> meta/recipes-support/hal/hal.inc:45
> meta/recipes-core/dbus/dbus.inc:49
> meta/recipes-connectivity/openssh/openssh_5.8p2.bb
> meta/recipes-connectivity/ppp-dialin/ppp-dialin_0.1.bb
> meta/recipes-graphics/x11-common/xserver-nodm-init.bb
> meta/recipes-multimedia/pulseaudio/pulseaudio.inc:87
> meta/recipes-extended/shadow/shadow_4.1.4.3.bb:125
> meta/classes/libc-package.bbclass

We should definitely fix these now we have the user code.

> 6  recipes: these should be easily fixed since the scripts are not related to special native utilites.
> meta/recipes-extended/sudo/sudo.inc
> meta/recipes-extended/sysklogd/sysklogd.inc
> meta/classes/update-rc.d.bbclass
> meta/recipes-connectivity/ppp/ppp_2.4.5.bb
> meta/recipes-graphics/pango/pango.inc
> meta/recipes-gnome/gtk+/gtk+.inc
>  
> 4 recipes: we may need to add gtk-update-icon-cache-native.
> meta/classes/gtk-icon-cache.bbclass
> meta/recipes-gnome/librsvg/librsvg_2.32.1.bb
> meta/recipes-gnome/gdk-pixbuf/gdk-pixbuf_2.22.1.bb
> meta/recipes-sato/sato-icon-theme/sato-icon-theme.inc
> 
> 3 recipes: need to add gconftool-2-native?
> meta/classes/gconf.bbclass
> meta/recipes-graphics/mutter/mutter.inc
> meta/recipes-sato/matchbox-sato/matchbox-session-sato_0.1.bb

I suspect this category and the once above (icon-cache) are the slowest
on the target device?

> 3 recipes: "dpkg --configure, opkg-cl configure": looks it's possible to fix them if we specify proper parematers?
> meta/recipes-devtools/dpkg/dpkg.inc
> meta/recipes-devtools/opkg/opkg_svn.bb
> meta/recipes-devtools/opkg/opkg_0.1.8.bb
> 
> 1 recipe: prelink: we could propablly fix it, but I'm not sure yet.
> meta/recipes-devtools/prelink/prelink_git.bb
> 
> 1 recipe: "/etc/init.d/populate-volatile.sh update ; DBUSPID=`pidof dbus-daemon`": We can't fix this one.
> meta/recipes-connectivity/wpa-supplicant/wpa-supplicant-0.7.inc
> 
> The below 4 recipes need the related utilities and need more investigation. 
> 
> 1 recipe: update-modules
> meta/recipes-kernel/update-modules/update-modules_1.0.bb
> 
> 1 recipe: systemctl
> meta/recipes-connectivity/avahi/avahi.inc
> 
> 1 recipe: fc-cache
> meta/recipes-graphics/ttf-fonts/liberation-fonts_1.04.bb:37
> 
> 1 recipe: gtk-query-immodules-2.0
> meta/recipes-sato/matchbox-keyboard/matchbox-keyboard_git.bb

This looks like a good start to me but I'd be interested to see the
relative lengths of time these postinstalls take...

Cheers,

Richard

Re: [Draft design][RFC] Running postinst at rootfs generation time

[Mark Hatle]

On 6/28/11 8:15 AM, Richard Purdie wrote:
> On Mon, 2011-06-27 at 23:11 -0500, Mark Hatle wrote:
>> On 6/27/11 9:09 PM, Cui, Dexuan wrote:
>>> Hi all, below is an initial investigation about the task and we'll continue to further look into it.
>>>
>>> In poky we have 2 types of postinst scripts:  one (type-1) can be (and has already been) run at rootfs generation time and the other (type-2) has to be delayed to the first-boot of target device. Type-2 makes target device's first-boot slow and it would be great if we can fix it and convert it to type-1.
>>>
>>> We can instrument a first-boot with minimal/sato first to see which postinstalls take the most time and then prioritise those ones to fix.
>>>
>>> I figurerd out a list of 33 recipes in total(recipes with the same name but with different versions are counted once) we possibly need to fix.
>>> For the recipes, we need try to find recipe-specific ways(use appropriately modified native utilities to generate caches, files, etc as necessary on the target filesystem).
>>>
>>
>> ...
>>
>>> 1 recipe: prelink: we could propablly fix it, but I'm not sure yet.
>>> meta/recipes-devtools/prelink/prelink_git.bb
>>
>> There is nothing to do for the prelinker.  The "image-prelink.bbclass" handles
>> everything needed to prelink during image creation.  The script is only there
>> for on-target field upgrades.  So you can remove this from your list.
> 
> Mark, are you 100% sure about this?
> 
> It looks like if we install prelink into an image it adds a post install
> which runs "prelink -a" on the target device at first boot.
> 
> This would happen regardless of whether the cross prelinker did or did
> not prelink the image :/.

That is certainly not the intention of this code.  If it does, it should be a
fairly quick bug to fix -- and improve boot-time.  (Of course, this only should
be skipped if we've done image-prelink.bbclass.)

I'll attempt to look into this shortly....

--Mark

> Cheers,
> 
> Richard
> 
> 
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [Draft design][RFC] Running postinst at rootfs generation time

[Hauser, Wolfgang (external)]

Beside the discussed changes, the postinst scripts should be executed in
the dependency order.
At the time, the scripts are executed in alphabetic order, which breaks
the image generation if depended packages are not in alphabetic order.

e.g. busybox and busybox subpackages (busybox-mdev).

Regards
Wolfgang Hauser 

Re: [Draft design][RFC] Running postinst at rootfs generation time

[Saul Wold]

On 06/27/2011 07:09 PM, Cui, Dexuan wrote:
> Hi all, below is an initial investigation about the task and we'll continue to further look into it.
>
> In poky we have 2 types of postinst scripts:  one (type-1) can be (and has already been) run at rootfs generation time and the other (type-2) has to be delayed to the first-boot of target device. Type-2 makes target device's first-boot slow and it would be great if we can fix it and convert it to type-1.
>
> We can instrument a first-boot with minimal/sato first to see which postinstalls take the most time and then prioritise those ones to fix.
>
> I figurerd out a list of 33 recipes in total(recipes with the same name but with different versions are counted once) we possibly need to fix.
> For the recipes, we need try to find recipe-specific ways(use appropriately modified native utilities to generate caches, files, etc as necessary on the target filesystem).
>
Dexuan,

Great start on this list, can you break this down to what's sato and 
minimal.

> 11 recipes: these could be easily fixed if we add the properly-adjusted utilities "adduser, addgroup, pwconv, etc". Scott is actually adding the utilites: http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=sgarman/useradd-rebased&id=99e54d9696104ed38ec1e3464e17aa1f9b8d98ac
> meta/recipes-devtools/distcc/distcc_2.18.3.bb
> meta/recipes-extended/cronie/cronie_1.4.7.bb
> meta/recipes-extended/at/at_3.1.12.bb:47
> meta/recipes-support/hal/hal.inc:45
> meta/recipes-core/dbus/dbus.inc:49
> meta/recipes-connectivity/openssh/openssh_5.8p2.bb
> meta/recipes-connectivity/ppp-dialin/ppp-dialin_0.1.bb
> meta/recipes-graphics/x11-common/xserver-nodm-init.bb
> meta/recipes-multimedia/pulseaudio/pulseaudio.inc:87
> meta/recipes-extended/shadow/shadow_4.1.4.3.bb:125
> meta/classes/libc-package.bbclass
>
As RP noted, the useradd code from Scott is very close to being pull, 
will you modify and test these recipes?

Sau!

> 6  recipes: these should be easily fixed since the scripts are not related to special native utilites.
> meta/recipes-extended/sudo/sudo.inc
> meta/recipes-extended/sysklogd/sysklogd.inc
> meta/classes/update-rc.d.bbclass
> meta/recipes-connectivity/ppp/ppp_2.4.5.bb
> meta/recipes-graphics/pango/pango.inc
> meta/recipes-gnome/gtk+/gtk+.inc
>
> 4 recipes: we may need to add gtk-update-icon-cache-native.
> meta/classes/gtk-icon-cache.bbclass
> meta/recipes-gnome/librsvg/librsvg_2.32.1.bb
> meta/recipes-gnome/gdk-pixbuf/gdk-pixbuf_2.22.1.bb
> meta/recipes-sato/sato-icon-theme/sato-icon-theme.inc
>
> 3 recipes: need to add gconftool-2-native?
> meta/classes/gconf.bbclass
> meta/recipes-graphics/mutter/mutter.inc
> meta/recipes-sato/matchbox-sato/matchbox-session-sato_0.1.bb
>
> 3 recipes: "dpkg --configure, opkg-cl configure": looks it's possible to fix them if we specify proper parematers?
> meta/recipes-devtools/dpkg/dpkg.inc
> meta/recipes-devtools/opkg/opkg_svn.bb
> meta/recipes-devtools/opkg/opkg_0.1.8.bb
>
> 1 recipe: prelink: we could propablly fix it, but I'm not sure yet.
> meta/recipes-devtools/prelink/prelink_git.bb
>
> 1 recipe: "/etc/init.d/populate-volatile.sh update ; DBUSPID=`pidof dbus-daemon`": We can't fix this one.
> meta/recipes-connectivity/wpa-supplicant/wpa-supplicant-0.7.inc
>
> The below 4 recipes need the related utilities and need more investigation.
>
> 1 recipe: update-modules
> meta/recipes-kernel/update-modules/update-modules_1.0.bb
>
> 1 recipe: systemctl
> meta/recipes-connectivity/avahi/avahi.inc
>
> 1 recipe: fc-cache
> meta/recipes-graphics/ttf-fonts/liberation-fonts_1.04.bb:37
>
> 1 recipe: gtk-query-immodules-2.0
> meta/recipes-sato/matchbox-keyboard/matchbox-keyboard_git.bb
>
> Thanks,
> -- Dexuan
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>

Re: [Draft design][RFC] Running postinst at rootfs generation time

[Cui, Dexuan]

Hauser, Wolfgang (external) wrote:
> Beside the discussed changes, the postinst scripts should be executed
> in the dependency order.
> At the time, the scripts are executed in alphabetic order, which
> breaks the image generation if depended packages are not in
> alphabetic order. 
> 
> e.g. busybox and busybox subpackages (busybox-mdev).
> 
> Regards
> Wolfgang Hauser
> 

Thank all for the suggestions!
I created a wiki page to summarize the mail threads:
https://wiki.yoctoproject.org/wiki/Run_postinst_during_rootfs_generation

Thanks,
-- Dexuan

Re: [Draft design][RFC] Running postinst at rootfs generation time

[Tom Parkin]

I'm interested to read about this initiative as we've recently
stumbled over the issue of postinst scripts in trying to port our
build to OE.

So far it seems the primary concern is about the boot-time impact of
postinst on the target.  But for the use-cases I'm interested in, we
need to capture all install processes as a part of the build in order
to output a fully-provisioned read-only rootfs image (e.g. squashfs).
Our boxes don't typically modify their filesystems at runtime, other
than to take full updates (completely new rootfs images).

I think the reduction, or even elimination, of target-run postinst
scripts would help a lot with the read-only rootfs case.

Tom

On 4 July 2011 01:23, Cui, Dexuan <dexuan.cui@intel.com> wrote:
> Hauser, Wolfgang (external) wrote:
>> Beside the discussed changes, the postinst scripts should be executed
>> in the dependency order.
>> At the time, the scripts are executed in alphabetic order, which
>> breaks the image generation if depended packages are not in
>> alphabetic order.
>>
>> e.g. busybox and busybox subpackages (busybox-mdev).
>>
>> Regards
>> Wolfgang Hauser
>>
>
> Thank all for the suggestions!
> I created a wiki page to summarize the mail threads:
> https://wiki.yoctoproject.org/wiki/Run_postinst_during_rootfs_generation
>
> Thanks,
> -- Dexuan
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>



-- 
Tom Parkin
www.thesecretdogproject.com
Morality, like art, means drawing a line someplace /Wilde/

Re: [Draft design][RFC] Running postinst at rootfs generation time

[Hauser, Wolfgang (external)]

To build a read-only image, we have to set up an extra layer where we modified all packages they cause a runtime modification.

Especially we have to cover some adduser/addgroup issues, modify the volatile file handling for our needs, and for omap3 graphics the handling of the GLES version.
A failed offline postinst causing a runtime configuration. So the image generation log has to be checked for such failing scripts.

Some of the packages (e.g. dropbear) are prepared for supporting read-only filesystems. Most packages are not.

The problem with the alphabetic order of the postinst scripts we managed by renaming some of the packages to fit the needed order. An ugly solution, but in case of missing python skills, the easiest.

Nice to see someone having the same needs we have to fit.
 
Regards
Wolfgang Hauser

-----Ursprüngliche Nachricht-----
Von: Tom Parkin [mailto:tom.parkin@gmail.com] 
Gesendet: Dienstag, 5. Juli 2011 10:42
An: Patches and discussions about the oe-core layer
Cc: Hauser, Wolfgang (external)
Betreff: Re: [OE-core] [Draft design][RFC] Running postinst at rootfs generation time

I'm interested to read about this initiative as we've recently
stumbled over the issue of postinst scripts in trying to port our
build to OE.

So far it seems the primary concern is about the boot-time impact of
postinst on the target.  But for the use-cases I'm interested in, we
need to capture all install processes as a part of the build in order
to output a fully-provisioned read-only rootfs image (e.g. squashfs).
Our boxes don't typically modify their filesystems at runtime, other
than to take full updates (completely new rootfs images).

I think the reduction, or even elimination, of target-run postinst
scripts would help a lot with the read-only rootfs case.

Tom

On 4 July 2011 01:23, Cui, Dexuan <dexuan.cui@intel.com> wrote:
> Hauser, Wolfgang (external) wrote:
>> Beside the discussed changes, the postinst scripts should be executed
>> in the dependency order.
>> At the time, the scripts are executed in alphabetic order, which
>> breaks the image generation if depended packages are not in
>> alphabetic order.
>>
>> e.g. busybox and busybox subpackages (busybox-mdev).
>>
>> Regards
>> Wolfgang Hauser
>>
>
> Thank all for the suggestions!
> I created a wiki page to summarize the mail threads:
> https://wiki.yoctoproject.org/wiki/Run_postinst_during_rootfs_generation
>
> Thanks,
> -- Dexuan
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>



-- 
Tom Parkin
www.thesecretdogproject.com
Morality, like art, means drawing a line someplace /Wilde/

Re: [Draft design][RFC] Running postinst at rootfs generation time

[Phil Blundell]

On Tue, 2011-07-05 at 12:06 +0200, Hauser, Wolfgang (external) wrote:
> Especially we have to cover some adduser/addgroup issues, modify the volatile file handling for our needs, and for omap3 graphics the handling of the GLES version.
> A failed offline postinst causing a runtime configuration. So the image generation log has to be checked for such failing scripts.

Nowadays you can set IMAGE_FEATURES = "read-only-rootfs", which will
result in a fatal error if any packages are still unconfigured after
rootfs construction is finished.

p.

Re: [Draft design][RFC] Running postinst at rootfs generation time

[Hauser, Wolfgang (external)]

>Nowadays you can set IMAGE_FEATURES = "read-only-rootfs", which will
>result in a fatal error if any packages are still unconfigured after
>rootfs construction is finished.

This helps to identify the packages to be modified, but it may be better to enhance the packages to be "read-only save". 
May be a official guide line may help here. 

Regards
Wolfgang Hauser
 
-----Ursprüngliche Nachricht-----
Von: openembedded-core-bounces@lists.openembedded.org [mailto:openembedded-core-bounces@lists.openembedded.org] Im Auftrag von Phil Blundell
Gesendet: Dienstag, 5. Juli 2011 12:23
An: Patches and discussions about the oe-core layer
Betreff: Re: [OE-core] [Draft design][RFC] Running postinst at rootfs generation time

On Tue, 2011-07-05 at 12:06 +0200, Hauser, Wolfgang (external) wrote:
> Especially we have to cover some adduser/addgroup issues, modify the volatile file handling for our needs, and for omap3 graphics the handling of the GLES version.
> A failed offline postinst causing a runtime configuration. So the image generation log has to be checked for such failing scripts.

Nowadays you can set IMAGE_FEATURES = "read-only-rootfs", which will
result in a fatal error if any packages are still unconfigured after
rootfs construction is finished.

p.



_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [Draft design][RFC] Running postinst at rootfs generation time

[Phil Blundell]

On Tue, 2011-07-05 at 12:30 +0200, Hauser, Wolfgang (external) wrote:
> >Nowadays you can set IMAGE_FEATURES = "read-only-rootfs", which will
> >result in a fatal error if any packages are still unconfigured after
> >rootfs construction is finished.
> 
> This helps to identify the packages to be modified, but it may be better to enhance the packages to be "read-only save". 

Yes, indeed.  The IMAGE_FEATURES thing doesn't magically make the
postinsts work in offline mode, it just avoids any unconfigured packages
leaking into the generated image.  Fundamentally the recipes do need
fixing to either eliminate the postinsts altogether or, if that's not
feasible, make them work in offline root mode.

> May be a official guide line may help here. 

What sort of guidance are you looking for?  If there are particular
recipes with issues that you don't know how to solve, post them here and
we can discuss them.

p.

Re: [Draft design][RFC] Running postinst at rootfs generation time

[Mark Hatle]

On 7/5/11 5:06 AM, Hauser, Wolfgang (external) wrote:
> To build a read-only image, we have to set up an extra layer where we modified all packages they cause a runtime modification.
> 
> Especially we have to cover some adduser/addgroup issues, modify the volatile file handling for our needs, and for omap3 graphics the handling of the GLES version.
> A failed offline postinst causing a runtime configuration. So the image generation log has to be checked for such failing scripts.

Just an FYI, with the new adduser/group code.. that should no longer be one of
the contributing factors.. we can now add users and groups during recipe /
rootfs construction.

> Some of the packages (e.g. dropbear) are prepared for supporting read-only filesystems. Most packages are not.
> 
> The problem with the alphabetic order of the postinst scripts we managed by renaming some of the packages to fit the needed order. An ugly solution, but in case of missing python skills, the easiest.

Ya, this one bothers me slightly... and I'm not really sure how to solve it...

--Mark

> Nice to see someone having the same needs we have to fit.
>  
> Regards
> Wolfgang Hauser
> 
> -----Ursprüngliche Nachricht-----
> Von: Tom Parkin [mailto:tom.parkin@gmail.com] 
> Gesendet: Dienstag, 5. Juli 2011 10:42
> An: Patches and discussions about the oe-core layer
> Cc: Hauser, Wolfgang (external)
> Betreff: Re: [OE-core] [Draft design][RFC] Running postinst at rootfs generation time
> 
> I'm interested to read about this initiative as we've recently
> stumbled over the issue of postinst scripts in trying to port our
> build to OE.
> 
> So far it seems the primary concern is about the boot-time impact of
> postinst on the target.  But for the use-cases I'm interested in, we
> need to capture all install processes as a part of the build in order
> to output a fully-provisioned read-only rootfs image (e.g. squashfs).
> Our boxes don't typically modify their filesystems at runtime, other
> than to take full updates (completely new rootfs images).
> 
> I think the reduction, or even elimination, of target-run postinst
> scripts would help a lot with the read-only rootfs case.
> 
> Tom
> 
> On 4 July 2011 01:23, Cui, Dexuan <dexuan.cui@intel.com> wrote:
>> Hauser, Wolfgang (external) wrote:
>>> Beside the discussed changes, the postinst scripts should be executed
>>> in the dependency order.
>>> At the time, the scripts are executed in alphabetic order, which
>>> breaks the image generation if depended packages are not in
>>> alphabetic order.
>>>
>>> e.g. busybox and busybox subpackages (busybox-mdev).
>>>
>>> Regards
>>> Wolfgang Hauser
>>>
>>
>> Thank all for the suggestions!
>> I created a wiki page to summarize the mail threads:
>> https://wiki.yoctoproject.org/wiki/Run_postinst_during_rootfs_generation
>>
>> Thanks,
>> -- Dexuan
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>
> 
> 
> 

Re: [Draft design][RFC] Running postinst at rootfs generation time

[Hauser, Wolfgang (external)]

>> The problem with the alphabetic order of the postinst scripts we
managed by renaming some of the packages to fit the needed order. An
ugly solution, but in case of missing python skills, the easiest.

>Ya, this one bothers me slightly... and I'm not really sure how to
solve it...

>--Mark

The packages are installed by opkg in their dependency order and are
echoed into the log file, as I know the common package managers (rpm,
dpkg, ...) provide an output of the dependency order of packages or have
a verbose installing output. It may possible to recycle this output as a
control list for the postinst loop instead of using the directory
listing (using tee to get an intermediate listing).

Regards
Wolfgang Hauser

Re: [Draft design][RFC] Running postinst at rootfs generation time

[Phil Blundell]

On Tue, 2011-07-12 at 12:07 -0500, Mark Hatle wrote:
> On 7/5/11 5:06 AM, Hauser, Wolfgang (external) wrote:
> > The problem with the alphabetic order of the postinst scripts we managed by renaming some of the packages to fit the needed order. An ugly solution, but in case of missing python skills, the easiest.
> 
> Ya, this one bothers me slightly... and I'm not really sure how to solve it...

I guess the obvious answer is just to capture the ordering of the
postinsts at the point where opkg-native tries to run them in offline
root mode.  That is, something like:

+	rm -f ${IMAGE_ROOTFS}${opkglibdir}/unconfigured
	runtime_script_required=0
	for i in ${IMAGE_ROOTFS}${opkglibdir}/info/*.preinst; do
		if [ -f $i ] && ! sh $i; then
		     	runtime_script_required=1
-			opkg-cl ${IPKG_ARGS} flag unpacked `basename $i .postinst`
+			pkg=`basename $i .postinst`
+			opkg-cl ${IPKG_ARGS} flag unpacked $pkg
+			echo $pkg >> ${IMAGE_ROOTFS}${opkglibdir}/unconfigured

[...]

	if ${@base_contains("IMAGE_FEATURES", "package-management", "false", "true", d)}; then
		if [ $runtime_script_required -eq 0 ]; then
			# All packages were successfully configured.
			# update-rc.d, base-passwd are no further use, remove them now
			opkg-cl ${IPKG_ARGS} --force-depends remove update-rc.d base-passwd || true

			# Also delete the status files
			rm -rf ${IMAGE_ROOTFS}${opkglibdir}
		fi
+	else
+		rm -f ${IMAGE_ROOTFS}${opkglibdir}/unconfigured
	fi

Or, well, you get the idea.

p.