[PATCH v2 0/3] Permit to install files from an overlay in images recipes

[PATCH v2 0/3] Permit to install files from an overlay in images recipes

[Geoffrey Levillain]

Hi,

These patches introduce a capability to install overlay into image after
fetching them.

There is actually a need to add image-specific configurations files into
image recipes, and these patches are a way to standardize this process as
well as simplify it by re-enabling fetching when image_overlay is
inherited, and by providing a task that copy every files in directories
listed in OVERLAY_ROOT_DIRS into IMAGE_ROOTFS directory.

Geoffrey Levillain (3):
  image.bbclass: No need to force image licence on MIT
  image_overlay.bbclass: Add possibility to install overlays to image
  rm_work.bbclass: Do not remove fetching and install_overlay stamps

 meta/classes/image.bbclass         |  2 +-
 meta/classes/image_overlay.bbclass | 24 ++++++++++++++++++++++++
 meta/classes/rm_work.bbclass       |  4 ++++
 3 files changed, 29 insertions(+), 1 deletion(-)
 create mode 100644 meta/classes/image_overlay.bbclass

Regards,
Geoffrey Levillain
--

[PATCH v2 1/3] image.bbclass: No need to force image licence on MIT

[Geoffrey Levillain]

If there is a user-defined licence there is no need to enforce MIT
licence instead. Therefor we set MIT if there is no licence defined.

Note : It also permit to set the licence before inherit image
in overlayable images (where a licence is needed because of the files
associated to the image).

Signed-off-by: Geoffrey Levillain <geoffrey.levillain@smile.fr>
---
 meta/classes/image.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index b5a4fb4a33..75c2860a75 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -9,7 +9,7 @@ TOOLCHAIN_TARGET_TASK += "${PACKAGE_INSTALL}"
 TOOLCHAIN_TARGET_TASK_ATTEMPTONLY += "${PACKAGE_INSTALL_ATTEMPTONLY}"
 POPULATE_SDK_POST_TARGET_COMMAND += "rootfs_sysroot_relativelinks; "
 
-LICENSE = "MIT"
+LICENSE ?= "MIT"
 PACKAGES = ""
 DEPENDS += "${MLPREFIX}qemuwrapper-cross depmodwrapper-cross"
 RDEPENDS += "${PACKAGE_INSTALL} ${LINGUAS_INSTALL}"
-- 
2.11.0

[PATCH v2 2/3] image_overlay.bbclass: Add possibility to install overlays to image

[Geoffrey Levillain]

There is often a need to add configuration files specific to an image,
to simplify this process we can use a task.

Fetching is set as noexec for images, but there is a need to fetch files
here. This class permit to use do_fetch, do_unpack and do_patch in order
to get your image-specific overlays to install.

Signed-off-by: Geoffrey Levillain <geoffrey.levillain@smile.fr>
---
 meta/classes/image_overlay.bbclass | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
 create mode 100644 meta/classes/image_overlay.bbclass

diff --git a/meta/classes/image_overlay.bbclass b/meta/classes/image_overlay.bbclass
new file mode 100644
index 0000000000..93c1ea6f9f
--- /dev/null
+++ b/meta/classes/image_overlay.bbclass
@@ -0,0 +1,24 @@
+# Enable SRC_URI for images and use it to install overlays
+
+# Let these task be executed to fetch overlay
+unset do_fetch[noexec]
+unset do_unpack[noexec]
+unset do_patch[noexec]
+
+# Add a task to install the overlay
+# (content of folders listed in OVERLAY_ROOT_DIRS is copied in rootfs)
+addtask install_overlay after do_unpack do_patch do_rootfs before do_image
+fakeroot do_install_overlay() {
+    for dir in ${OVERLAY_ROOT_DIRS}; do
+        if [ ! -d "${WORKDIR}/${dir}" ] ; then
+            bbfatal "${dir} not found, please check your OVERLAY_ROOT_DIRS and SRC_URI variables."
+        fi
+        cp -dr --no-preserve=ownership "${WORKDIR}/${dir}"/* "${IMAGE_ROOTFS}"
+    done
+}
+do_install_overlay[depends] += "virtual/fakeroot-native:do_populate_sysroot"
+do_install_overlay[dirs] = "${IMAGE_ROOTFS}"
+do_install_overlay[umask] = "022"
+do_install_overlay[vardeps] += "OVERLAY_ROOT_DIRS"
+
+SSTATETASKS += "do_install_overlay"
-- 
2.11.0

[PATCH v2 3/3] rm_work.bbclass: Do not remove fetching and install_overlay stamps

[Geoffrey Levillain]

In order to keep do_install_overlay to reexecute it's needed to keep
stamps of fetching tasks.

Without that do_install_overlay would reexecute and do_image as well but
do_image will crash since there is no rootfs.

Signed-off-by: Geoffrey Levillain <geoffrey.levillain@smile.fr>
---
 meta/classes/rm_work.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes/rm_work.bbclass b/meta/classes/rm_work.bbclass
index 054c93716e..c0e7c2a0a6 100644
--- a/meta/classes/rm_work.bbclass
+++ b/meta/classes/rm_work.bbclass
@@ -71,6 +71,10 @@ do_rm_work () {
                 i=dummy
                 break
                 ;;
+            *do_fetch*|*do_unpack*|*do_patch*|*install_overlay*)
+                i=dummy
+                break
+                ;;
             *do_rootfs*|*do_image*|*do_bootimg*|*do_bootdirectdisk*|*do_vmimg*|*do_write_qemuboot_conf*)
                 i=dummy
                 break
-- 
2.11.0

Re: [PATCH v2 0/3] Permit to install files from an overlay in images recipes

[Gary Thomas]

On 2017-02-15 18:33, Geoffrey Levillain wrote:
> Hi,
>
> These patches introduce a capability to install overlay into image after
> fetching them.
>
> There is actually a need to add image-specific configurations files into
> image recipes, and these patches are a way to standardize this process as
> well as simplify it by re-enabling fetching when image_overlay is
> inherited, and by providing a task that copy every files in directories
> listed in OVERLAY_ROOT_DIRS into IMAGE_ROOTFS directory.
>
> Geoffrey Levillain (3):
>   image.bbclass: No need to force image licence on MIT
>   image_overlay.bbclass: Add possibility to install overlays to image
>   rm_work.bbclass: Do not remove fetching and install_overlay stamps
>
>  meta/classes/image.bbclass         |  2 +-
>  meta/classes/image_overlay.bbclass | 24 ++++++++++++++++++++++++
>  meta/classes/rm_work.bbclass       |  4 ++++
>  3 files changed, 29 insertions(+), 1 deletion(-)
>  create mode 100644 meta/classes/image_overlay.bbclass

Can you give an example where this approach is better than just
having the image include some [possibly optional] packages that
contain these image-specific configuration files?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: [PATCH v2 2/3] image_overlay.bbclass: Add possibility to install overlays to image

[Richard Purdie]

On Wed, 2017-02-15 at 18:33 +0100, Geoffrey Levillain wrote:
> There is often a need to add configuration files specific to an
> image,
> to simplify this process we can use a task.
> 
> Fetching is set as noexec for images, but there is a need to fetch
> files
> here. This class permit to use do_fetch, do_unpack and do_patch in
> order
> to get your image-specific overlays to install.
> 
> Signed-off-by: Geoffrey Levillain <geoffrey.levillain@smile.fr>
> ---
>  meta/classes/image_overlay.bbclass | 24 ++++++++++++++++++++++++
>  1 file changed, 24 insertions(+)
>  create mode 100644 meta/classes/image_overlay.bbclass
> 
> diff --git a/meta/classes/image_overlay.bbclass
> b/meta/classes/image_overlay.bbclass
> new file mode 100644
> index 0000000000..93c1ea6f9f
> --- /dev/null
> +++ b/meta/classes/image_overlay.bbclass
> @@ -0,0 +1,24 @@
> +# Enable SRC_URI for images and use it to install overlays
> +
> +# Let these task be executed to fetch overlay
> +unset do_fetch[noexec]
> +unset do_unpack[noexec]
> +unset do_patch[noexec]

At the same time as you're submitting this, I'm seriously considering a
change where instead of noexec, we'd deltask these tasks. The reason
being I'm not very keen on subverting the package manager and I'm not
convinced this is a particularly desirable way to be altering the
rootfs.

> +# Add a task to install the overlay
> +# (content of folders listed in OVERLAY_ROOT_DIRS is copied in
> rootfs)
> +addtask install_overlay after do_unpack do_patch do_rootfs before
> do_image
> +fakeroot do_install_overlay() {
> +    for dir in ${OVERLAY_ROOT_DIRS}; do
> +        if [ ! -d "${WORKDIR}/${dir}" ] ; then
> +            bbfatal "${dir} not found, please check your
> OVERLAY_ROOT_DIRS and SRC_URI variables."
> +        fi
> +        cp -dr --no-preserve=ownership "${WORKDIR}/${dir}"/*
> "${IMAGE_ROOTFS}"
> +    done
> +}
> +do_install_overlay[depends] += "virtual/fakeroot-
> native:do_populate_sysroot"
> +do_install_overlay[dirs] = "${IMAGE_ROOTFS}"
> +do_install_overlay[umask] = "022"
> +do_install_overlay[vardeps] += "OVERLAY_ROOT_DIRS"
> +
> +SSTATETASKS += "do_install_overlay"

Except this isn't an sstate task? There is no setscene task variant of
the main task. There are no sstate variables set either...

Cheers,

Richard

Re: [PATCH v2 3/3] rm_work.bbclass: Do not remove fetching and install_overlay stamps

[Richard Purdie]

On Wed, 2017-02-15 at 18:33 +0100, Geoffrey Levillain wrote:
> In order to keep do_install_overlay to reexecute it's needed to keep
> stamps of fetching tasks.
> 
> Without that do_install_overlay would reexecute and do_image as well
> but
> do_image will crash since there is no rootfs.
> 
> Signed-off-by: Geoffrey Levillain <geoffrey.levillain@smile.fr>
> ---
>  meta/classes/rm_work.bbclass | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/meta/classes/rm_work.bbclass
> b/meta/classes/rm_work.bbclass
> index 054c93716e..c0e7c2a0a6 100644
> --- a/meta/classes/rm_work.bbclass
> +++ b/meta/classes/rm_work.bbclass
> @@ -71,6 +71,10 @@ do_rm_work () {
>                  i=dummy
>                  break
>                  ;;
> +            *do_fetch*|*do_unpack*|*do_patch*|*install_overlay*)
> +                i=dummy
> +                break
> +                ;;

This patch whitelists not just the install_overlay stamps but the
fetch, patch and unpack stamps for *everything* in all recipes? That
clearly isn't correct...

Cheers,

Richard

Re: [PATCH v2 2/3] image_overlay.bbclass: Add possibility to install overlays to image

[Jérémy Rosen]

[-- Attachment #1: Type: text/plain, Size: 4065 bytes --]



On 16/02/2017 05:44, Richard Purdie wrote:
> On Wed, 2017-02-15 at 18:33 +0100, Geoffrey Levillain wrote:
>> There is often a need to add configuration files specific to an
>> image,
>> to simplify this process we can use a task.
>>
>> Fetching is set as noexec for images, but there is a need to fetch
>> files
>> here. This class permit to use do_fetch, do_unpack and do_patch in
>> order
>> to get your image-specific overlays to install.
>>
>> Signed-off-by: Geoffrey Levillain <geoffrey.levillain@smile.fr>
>> ---
>>   meta/classes/image_overlay.bbclass | 24 ++++++++++++++++++++++++
>>   1 file changed, 24 insertions(+)
>>   create mode 100644 meta/classes/image_overlay.bbclass
>>
>> diff --git a/meta/classes/image_overlay.bbclass
>> b/meta/classes/image_overlay.bbclass
>> new file mode 100644
>> index 0000000000..93c1ea6f9f
>> --- /dev/null
>> +++ b/meta/classes/image_overlay.bbclass
>> @@ -0,0 +1,24 @@
>> +# Enable SRC_URI for images and use it to install overlays
>> +
>> +# Let these task be executed to fetch overlay
>> +unset do_fetch[noexec]
>> +unset do_unpack[noexec]
>> +unset do_patch[noexec]
> At the same time as you're submitting this, I'm seriously considering a
> change where instead of noexec, we'd deltask these tasks. The reason
> being I'm not very keen on subverting the package manager and I'm not
> convinced this is a particularly desirable way to be altering the
> rootfs.
There are various files in images that are provided by the package 
manager and that are ok to modify. most files in /etc are in that category.

The idea here is to be able to have generic (image independant) 
packages, but to override the package configuration with custom ones at 
image-generation time.

on a normal (desktop) distribution, the user writes his configuration 
files, but in a deep-embedded useage, there is no user.

A bbappend (overriding the config file at package-build time) would seem 
a better solution at first, but this doesn't work because it forces to 
have a different package for each image type. Again we are trying to 
particularize each image, while keeping the distribution itself generic.

I hope this helps clarify why we do this at the image level and not the 
package. packages are shared between images, but configuration is 
altered per-image.

Jérémy

>> +# Add a task to install the overlay
>> +# (content of folders listed in OVERLAY_ROOT_DIRS is copied in
>> rootfs)
>> +addtask install_overlay after do_unpack do_patch do_rootfs before
>> do_image
>> +fakeroot do_install_overlay() {
>> +    for dir in ${OVERLAY_ROOT_DIRS}; do
>> +        if [ ! -d "${WORKDIR}/${dir}" ] ; then
>> +            bbfatal "${dir} not found, please check your
>> OVERLAY_ROOT_DIRS and SRC_URI variables."
>> +        fi
>> +        cp -dr --no-preserve=ownership "${WORKDIR}/${dir}"/*
>> "${IMAGE_ROOTFS}"
>> +    done
>> +}
>> +do_install_overlay[depends] += "virtual/fakeroot-
>> native:do_populate_sysroot"
>> +do_install_overlay[dirs] = "${IMAGE_ROOTFS}"
>> +do_install_overlay[umask] = "022"
>> +do_install_overlay[vardeps] += "OVERLAY_ROOT_DIRS"
>> +
>> +SSTATETASKS += "do_install_overlay"
> Except this isn't an sstate task? There is no setscene task variant of
> the main task. There are no sstate variables set either...
>
> Cheers,
>
> Richard
>
>
>

-- 
Logo <http://www.smile.fr/>

20 rue des Jardins
92600 Asnières-sur-Seine
www.smile.fr <http://www.smile.fr/> 	
*Jérémy ROSEN*
Architecte technique
Email : jeremy.rosen@smile.fr <mailto:jeremy.rosen@smile.fr>
Tel : +33141402967

Facebook <https://www.facebook.com/smileopensource> Google%2B 
<http://fr.slideshare.net/SmileOpenSource/presentations> LinkedIn 
<https://www.linkedin.com/company/smile> Twitter 
<https://twitter.com/GroupeSmile>


bandeaux_mail 
<http://www.smile.fr/Offres-services/Offres/Ingenierie?utm_source=signature&utm_medium=email&utm_campaign=signature>

eco Pour la planète, n'imprimez ce mail que si c'est nécessaire

[-- Attachment #2.1: Type: text/html, Size: 7802 bytes --]

[-- Attachment #2.2: Logo.png --]
[-- Type: image/png, Size: 3775 bytes --]

[-- Attachment #2.3: Facebook.png --]
[-- Type: image/png, Size: 835 bytes --]

[-- Attachment #2.4: Slideshare.png --]
[-- Type: image/png, Size: 967 bytes --]

[-- Attachment #2.5: LinkedIn.png --]
[-- Type: image/png, Size: 816 bytes --]

[-- Attachment #2.6: Twitter.png --]
[-- Type: image/png, Size: 989 bytes --]

[-- Attachment #2.7: OW_devient_Smile_ECS.png --]
[-- Type: image/png, Size: 83679 bytes --]

[-- Attachment #2.8: eco.png --]
[-- Type: image/png, Size: 267 bytes --]

Re: [PATCH v2 0/3] Permit to install files from an overlay in images recipes

[Jérémy Rosen]

[-- Attachment #1: Type: text/plain, Size: 2358 bytes --]



On 16/02/2017 03:18, Gary Thomas wrote:
> On 2017-02-15 18:33, Geoffrey Levillain wrote:
>> Hi,
>>
>> These patches introduce a capability to install overlay into image after
>> fetching them.
>>
>> There is actually a need to add image-specific configurations files into
>> image recipes, and these patches are a way to standardize this 
>> process as
>> well as simplify it by re-enabling fetching when image_overlay is
>> inherited, and by providing a task that copy every files in directories
>> listed in OVERLAY_ROOT_DIRS into IMAGE_ROOTFS directory.
>>
>> Geoffrey Levillain (3):
>>   image.bbclass: No need to force image licence on MIT
>>   image_overlay.bbclass: Add possibility to install overlays to image
>>   rm_work.bbclass: Do not remove fetching and install_overlay stamps
>>
>>  meta/classes/image.bbclass         |  2 +-
>>  meta/classes/image_overlay.bbclass | 24 ++++++++++++++++++++++++
>>  meta/classes/rm_work.bbclass       |  4 ++++
>>  3 files changed, 29 insertions(+), 1 deletion(-)
>>  create mode 100644 meta/classes/image_overlay.bbclass
>
> Can you give an example where this approach is better than just
> having the image include some [possibly optional] packages that
> contain these image-specific configuration files?
>
The problem is that most configuration files are provided by a package 
and thus can not be overriden by another package.
(a bbappend for each package we want to modify is a big amount of 
tweaking and would force a variant of the package for each image with a 
different configuration)
A "configuration package" can only add new conf files, it can't alter 
existing files.

Regards
Jérémy

-- 
Logo <http://www.smile.fr/>

20 rue des Jardins
92600 Asnières-sur-Seine
www.smile.fr <http://www.smile.fr/> 	
*Jérémy ROSEN*
Architecte technique
Email : jeremy.rosen@smile.fr <mailto:jeremy.rosen@smile.fr>
Tel : +33141402967

Facebook <https://www.facebook.com/smileopensource> Google%2B 
<http://fr.slideshare.net/SmileOpenSource/presentations> LinkedIn 
<https://www.linkedin.com/company/smile> Twitter 
<https://twitter.com/GroupeSmile>


bandeaux_mail 
<http://www.smile.fr/Offres-services/Offres/Ingenierie?utm_source=signature&utm_medium=email&utm_campaign=signature>

eco Pour la planète, n'imprimez ce mail que si c'est nécessaire

[-- Attachment #2.1: Type: text/html, Size: 6238 bytes --]

[-- Attachment #2.2: Logo.png --]
[-- Type: image/png, Size: 3775 bytes --]

[-- Attachment #2.3: Facebook.png --]
[-- Type: image/png, Size: 835 bytes --]

[-- Attachment #2.4: Slideshare.png --]
[-- Type: image/png, Size: 967 bytes --]

[-- Attachment #2.5: LinkedIn.png --]
[-- Type: image/png, Size: 816 bytes --]

[-- Attachment #2.6: Twitter.png --]
[-- Type: image/png, Size: 989 bytes --]

[-- Attachment #2.7: OW_devient_Smile_ECS.png --]
[-- Type: image/png, Size: 83679 bytes --]

[-- Attachment #2.8: eco.png --]
[-- Type: image/png, Size: 267 bytes --]