All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 0/3] Support template files for wic
@ 2016-05-23 20:34 Christopher Larson
  2016-05-23 20:34 ` [PATCH 1/3] image.bbclass: append to prefuncs/postfuncs for do_image_* Christopher Larson
                   ` (2 more replies)
  0 siblings, 3 replies; 9+ messages in thread
From: Christopher Larson @ 2016-05-23 20:34 UTC (permalink / raw)
  To: openembedded-core; +Cc: Christopher Larson

From: Christopher Larson <chris_larson@mentor.com>

This adds support for .wks.in files as WKS_FILE.

This is a re-send of the RFC PATCH series as a regular PATCH series, to
attempt to get the feature merged. There's a clear value-add, there were no
major objections to the RFC series, and further improvements can be added on
from here.

These files are treated as the contents of a bitbake variable, so usual
bitbake variable references are supported. I considered using another
templating mechanism, for example the one used by yocto-layer, but then we'd
end up largely mapping metadata variables to template fields anyway, which is
a pointless indirection. Let bitbake expand the variables directly instead.

This feature lets us, for example, reference ${APPEND} in --append, and avoid
hardcoding the serial console tty in the wks file, and let the user's changes
to APPEND affect wic the way they do the other image construction mechanisms.

The template is read in and set in a variable at parse time, so changes to the
variables referenced by the template will result in rebuilding the image.

It's been suggested that the name should better reflect how its contents are
used, i.e. .bbin or so. I've also considered including something like 'bbv' or
similar, to indicate its contents are that of a bitbake value. I think this
should be discussed.

I'd love to hear questions or comments on this. I think the value is clear,
and it's something we need to solve, but I'd like to hear if others agree or
disagree on that, and whether this implementation is appropriate.


The following changes since commit f6b0b260ce18a30d04edfb0afb7942b9f9a5480b:

  lib/oe/rootfs: Fix DEBUGFS generation, without openssl (2016-05-20 10:20:42 +0100)

are available in the git repository at:

  git://github.com/kergoth/openembedded-core wks.in
  https://github.com/kergoth/openembedded-core/tree/wks.in

Christopher Larson (3):
  image.bbclass: append to prefuncs/postfuncs for do_image_*
  image_types.bbclass: add intermediate var for WKS_FILE_CHECKSUM
  image_types.bbclass: support template .wks.in files for wic

 meta/classes/image.bbclass       |  4 ++--
 meta/classes/image_types.bbclass | 38 +++++++++++++++++++++++++++++++++++++-
 2 files changed, 39 insertions(+), 3 deletions(-)

-- 
2.8.0



^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH 1/3] image.bbclass: append to prefuncs/postfuncs for do_image_*
  2016-05-23 20:34 [PATCH 0/3] Support template files for wic Christopher Larson
@ 2016-05-23 20:34 ` Christopher Larson
  2016-05-23 20:34 ` [PATCH 2/3] image_types.bbclass: add intermediate var for WKS_FILE_CHECKSUM Christopher Larson
  2016-05-23 20:34 ` [PATCH 3/3] image_types.bbclass: support template .wks.in files for wic Christopher Larson
  2 siblings, 0 replies; 9+ messages in thread
From: Christopher Larson @ 2016-05-23 20:34 UTC (permalink / raw)
  To: openembedded-core; +Cc: Christopher Larson

From: Christopher Larson <chris_larson@mentor.com>

These are list style variables, so append to them rather than blowing away any
existing value.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/image.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 6b9f979..95dbb82 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -436,9 +436,9 @@ python () {
         d.setVar('do_image_%s' % t, '\n'.join(cmds))
         d.setVarFlag('do_image_%s' % t, 'func', '1')
         d.setVarFlag('do_image_%s' % t, 'fakeroot', '1')
-        d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size')
-        d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks')
         d.setVarFlag('do_image_%s' % t, 'subimages', ' '.join(subimages))
+        d.appendVarFlag('do_image_%s' % t, 'prefuncs', ' ' + debug + 'set_image_size')
+        d.appendVarFlag('do_image_%s' % t, 'postfuncs', ' create_symlinks')
         d.appendVarFlag('do_image_%s' % t, 'vardeps', ' '.join(vardeps))
         d.appendVarFlag('do_image_%s' % t, 'vardepsexclude', 'DATETIME')
 
-- 
2.8.0



^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH 2/3] image_types.bbclass: add intermediate var for WKS_FILE_CHECKSUM
  2016-05-23 20:34 [PATCH 0/3] Support template files for wic Christopher Larson
  2016-05-23 20:34 ` [PATCH 1/3] image.bbclass: append to prefuncs/postfuncs for do_image_* Christopher Larson
@ 2016-05-23 20:34 ` Christopher Larson
  2016-05-23 20:34 ` [PATCH 3/3] image_types.bbclass: support template .wks.in files for wic Christopher Larson
  2 siblings, 0 replies; 9+ messages in thread
From: Christopher Larson @ 2016-05-23 20:34 UTC (permalink / raw)
  To: openembedded-core; +Cc: Christopher Larson

From: Christopher Larson <chris_larson@mentor.com>

This is a bit nicer to work with, and easier to override.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/image_types.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index ea45809..dc681ae 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -208,7 +208,8 @@ IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES"
 
 # Rebuild when the wks file or vars in WICVARS change
 USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '', d)}"
-do_image_wic[file-checksums] += "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
+WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
+do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}"
 
 EXTRA_IMAGECMD = ""
 
-- 
2.8.0



^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH 3/3] image_types.bbclass: support template .wks.in files for wic
  2016-05-23 20:34 [PATCH 0/3] Support template files for wic Christopher Larson
  2016-05-23 20:34 ` [PATCH 1/3] image.bbclass: append to prefuncs/postfuncs for do_image_* Christopher Larson
  2016-05-23 20:34 ` [PATCH 2/3] image_types.bbclass: add intermediate var for WKS_FILE_CHECKSUM Christopher Larson
@ 2016-05-23 20:34 ` Christopher Larson
  2016-05-24  9:13   ` Richard Purdie
  2 siblings, 1 reply; 9+ messages in thread
From: Christopher Larson @ 2016-05-23 20:34 UTC (permalink / raw)
  To: openembedded-core; +Cc: Christopher Larson

From: Christopher Larson <chris_larson@mentor.com>

These files are treated as the contents of a bitbake variable, so usual
bitbake variable references are supported. I considered using another
templating mechanism, for example the one used by yocto-layer, but then we'd
end up largely mapping metadata variables to template fields anyway, which is
a pointless indirection. Let bitbake expand the variables directly instead.

This feature lets us, for example, reference ${APPEND} in --append, and avoid
hardcoding the serial console tty in the wks file, and let the user's changes
to APPEND affect wic the way they do the other image construction mechanisms.

The template is read in and set in a variable at parse time, so changes to the
variables referenced by the template will result in rebuilding the image.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/image_types.bbclass | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index dc681ae..caf8757 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -206,6 +206,16 @@ IMAGE_CMD_wic () {
 }
 IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES"
 
+python process_wks_template () {
+    """Write out expanded template contents to WKS_FULL_PATH."""
+    template_body = d.getVar('_WKS_TEMPLATE', True)
+    if template_body:
+        wks_file = d.getVar('WKS_FULL_PATH', True)
+        with open(wks_file, 'w') as f:
+            f.write(template_body)
+}
+do_image_wic[prefuncs] += 'process_wks_template'
+
 # Rebuild when the wks file or vars in WICVARS change
 USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '', d)}"
 WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
@@ -302,3 +312,28 @@ IMAGE_TYPES_MASKED ?= ""
 # The WICVARS variable is used to define list of bitbake variables used in wic code
 # variables from this list is written to <image>.env file
 WICVARS ?= "BBLAYERS DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS"
+
+python () {
+    """Read in and set up wks file template for wic."""
+    if d.getVar('USING_WIC', True):
+        wks_file_u = d.getVar('WKS_FULL_PATH', False)
+        wks_file = d.expand(wks_file_u)
+        base, ext = os.path.splitext(wks_file)
+        if ext == '.in' and os.path.exists(wks_file):
+            wks_out_file = os.path.join(d.getVar('WORKDIR', True), os.path.basename(base))
+            d.setVar('WKS_FULL_PATH', wks_out_file)
+            d.setVar('WKS_TEMPLATE_PATH', wks_file_u)
+            d.setVar('WKS_FILE_CHECKSUM', '${WKS_TEMPLATE_PATH}:True')
+
+            try:
+                with open(wks_file, 'r') as f:
+                    body = f.read()
+            except (IOError, OSError) as exc:
+                pass
+            else:
+                # Previously, I used expandWithRefs to get the dependency list
+                # and add it to WICVARS, but there's no point re-parsing the
+                # file in process_wks_template as well, so just put it in
+                # a variable and let the metadata deal with the deps.
+                d.setVar('_WKS_TEMPLATE', body)
+}
-- 
2.8.0



^ permalink raw reply related	[flat|nested] 9+ messages in thread

* Re: [PATCH 3/3] image_types.bbclass: support template .wks.in files for wic
  2016-05-23 20:34 ` [PATCH 3/3] image_types.bbclass: support template .wks.in files for wic Christopher Larson
@ 2016-05-24  9:13   ` Richard Purdie
  2016-05-24 14:26     ` Christopher Larson
  0 siblings, 1 reply; 9+ messages in thread
From: Richard Purdie @ 2016-05-24  9:13 UTC (permalink / raw)
  To: Christopher Larson, openembedded-core; +Cc: Christopher Larson

On Mon, 2016-05-23 at 13:34 -0700, Christopher Larson wrote:
> From: Christopher Larson <chris_larson@mentor.com>
> 
> These files are treated as the contents of a bitbake variable, so
> usual
> bitbake variable references are supported. I considered using another
> templating mechanism, for example the one used by yocto-layer, but
> then we'd
> end up largely mapping metadata variables to template fields anyway,
> which is
> a pointless indirection. Let bitbake expand the variables directly
> instead.
> 
> This feature lets us, for example, reference ${APPEND} in --append,
> and avoid
> hardcoding the serial console tty in the wks file, and let the user's
> changes
> to APPEND affect wic the way they do the other image construction
> mechanisms.
> 
> The template is read in and set in a variable at parse time, so
> changes to the
> variables referenced by the template will result in rebuilding the
> image.
> 
> Signed-off-by: Christopher Larson <chris_larson@mentor.com>
> ---
>  meta/classes/image_types.bbclass | 35
> +++++++++++++++++++++++++++++++++++
>  1 file changed, 35 insertions(+)
> 
> diff --git a/meta/classes/image_types.bbclass
> b/meta/classes/image_types.bbclass
> index dc681ae..caf8757 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -206,6 +206,16 @@ IMAGE_CMD_wic () {
>  }
>  IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES"
>  
> +python process_wks_template () {
> +    """Write out expanded template contents to WKS_FULL_PATH."""
> +    template_body = d.getVar('_WKS_TEMPLATE', True)
> +    if template_body:
> +        wks_file = d.getVar('WKS_FULL_PATH', True)
> +        with open(wks_file, 'w') as f:
> +            f.write(template_body)
> +}
> +do_image_wic[prefuncs] += 'process_wks_template'
> +
>  # Rebuild when the wks file or vars in WICVARS change
>  USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + '
> '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '',
> d)}"
>  WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' %
> os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
> @@ -302,3 +312,28 @@ IMAGE_TYPES_MASKED ?= ""
>  # The WICVARS variable is used to define list of bitbake variables
> used in wic code
>  # variables from this list is written to <image>.env file
>  WICVARS ?= "BBLAYERS DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME
> IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES
> INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR
> STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS"
> +
> +python () {
> +    """Read in and set up wks file template for wic."""
> +    if d.getVar('USING_WIC', True):
> +        wks_file_u = d.getVar('WKS_FULL_PATH', False)
> +        wks_file = d.expand(wks_file_u)
> +        base, ext = os.path.splitext(wks_file)
> +        if ext == '.in' and os.path.exists(wks_file):
> +            wks_out_file = os.path.join(d.getVar('WORKDIR', True),
> os.path.basename(base))
> +            d.setVar('WKS_FULL_PATH', wks_out_file)
> +            d.setVar('WKS_TEMPLATE_PATH', wks_file_u)
> +            d.setVar('WKS_FILE_CHECKSUM',
> '${WKS_TEMPLATE_PATH}:True')
> +
> +            try:
> +                with open(wks_file, 'r') as f:
> +                    body = f.read()
> +            except (IOError, OSError) as exc:
> +                pass

I'm a little nervous about determinism here. Shouldn't it either find
the referenced file or error?

Cheers,

Richard




^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH 3/3] image_types.bbclass: support template .wks.in files for wic
  2016-05-24  9:13   ` Richard Purdie
@ 2016-05-24 14:26     ` Christopher Larson
  2016-05-24 16:07       ` Christopher Larson
  0 siblings, 1 reply; 9+ messages in thread
From: Christopher Larson @ 2016-05-24 14:26 UTC (permalink / raw)
  To: Richard Purdie; +Cc: Patches and discussions about the oe-core layer

[-- Attachment #1: Type: text/plain, Size: 4473 bytes --]

On Tue, May 24, 2016 at 2:13 AM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Mon, 2016-05-23 at 13:34 -0700, Christopher Larson wrote:
> > From: Christopher Larson <chris_larson@mentor.com>
> >
> > These files are treated as the contents of a bitbake variable, so
> > usual
> > bitbake variable references are supported. I considered using another
> > templating mechanism, for example the one used by yocto-layer, but
> > then we'd
> > end up largely mapping metadata variables to template fields anyway,
> > which is
> > a pointless indirection. Let bitbake expand the variables directly
> > instead.
> >
> > This feature lets us, for example, reference ${APPEND} in --append,
> > and avoid
> > hardcoding the serial console tty in the wks file, and let the user's
> > changes
> > to APPEND affect wic the way they do the other image construction
> > mechanisms.
> >
> > The template is read in and set in a variable at parse time, so
> > changes to the
> > variables referenced by the template will result in rebuilding the
> > image.
> >
> > Signed-off-by: Christopher Larson <chris_larson@mentor.com>
> > ---
> >  meta/classes/image_types.bbclass | 35
> > +++++++++++++++++++++++++++++++++++
> >  1 file changed, 35 insertions(+)
> >
> > diff --git a/meta/classes/image_types.bbclass
> > b/meta/classes/image_types.bbclass
> > index dc681ae..caf8757 100644
> > --- a/meta/classes/image_types.bbclass
> > +++ b/meta/classes/image_types.bbclass
> > @@ -206,6 +206,16 @@ IMAGE_CMD_wic () {
> >  }
> >  IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES"
> >
> > +python process_wks_template () {
> > +    """Write out expanded template contents to WKS_FULL_PATH."""
> > +    template_body = d.getVar('_WKS_TEMPLATE', True)
> > +    if template_body:
> > +        wks_file = d.getVar('WKS_FULL_PATH', True)
> > +        with open(wks_file, 'w') as f:
> > +            f.write(template_body)
> > +}
> > +do_image_wic[prefuncs] += 'process_wks_template'
> > +
> >  # Rebuild when the wks file or vars in WICVARS change
> >  USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + '
> > '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '',
> > d)}"
> >  WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' %
> > os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
> > @@ -302,3 +312,28 @@ IMAGE_TYPES_MASKED ?= ""
> >  # The WICVARS variable is used to define list of bitbake variables
> > used in wic code
> >  # variables from this list is written to <image>.env file
> >  WICVARS ?= "BBLAYERS DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME
> > IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES
> > INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR
> > STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS"
> > +
> > +python () {
> > +    """Read in and set up wks file template for wic."""
> > +    if d.getVar('USING_WIC', True):
> > +        wks_file_u = d.getVar('WKS_FULL_PATH', False)
> > +        wks_file = d.expand(wks_file_u)
> > +        base, ext = os.path.splitext(wks_file)
> > +        if ext == '.in' and os.path.exists(wks_file):
> > +            wks_out_file = os.path.join(d.getVar('WORKDIR', True),
> > os.path.basename(base))
> > +            d.setVar('WKS_FULL_PATH', wks_out_file)
> > +            d.setVar('WKS_TEMPLATE_PATH', wks_file_u)
> > +            d.setVar('WKS_FILE_CHECKSUM',
> > '${WKS_TEMPLATE_PATH}:True')
> > +
> > +            try:
> > +                with open(wks_file, 'r') as f:
> > +                    body = f.read()
> > +            except (IOError, OSError) as exc:
> > +                pass
>
> I'm a little nervous about determinism here. Shouldn't it either find
> the referenced file or error?


That's a fair point, I think I was concerned about parse time errors, since
they tend to blow up into a lot of unpleasantness for the user, since parse
time errors in classes can cause errors for multiple recipes. If no file is
read in at parse time, there'll be no contents to write out, and we could
then error when trying to build the image with wic, as no wks file will
exist.

Of course, the file being missing will change checksums for the image
tasks, but I think that's reasonable.

Thoughts?
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 5903 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH 3/3] image_types.bbclass: support template .wks.in files for wic
  2016-05-24 14:26     ` Christopher Larson
@ 2016-05-24 16:07       ` Christopher Larson
  2016-05-24 17:45         ` Christopher Larson
  0 siblings, 1 reply; 9+ messages in thread
From: Christopher Larson @ 2016-05-24 16:07 UTC (permalink / raw)
  To: Richard Purdie; +Cc: Patches and discussions about the oe-core layer

[-- Attachment #1: Type: text/plain, Size: 4767 bytes --]

On Tue, May 24, 2016 at 7:26 AM, Christopher Larson <kergoth@gmail.com>
wrote:

>
> On Tue, May 24, 2016 at 2:13 AM, Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
>
>> On Mon, 2016-05-23 at 13:34 -0700, Christopher Larson wrote:
>> > From: Christopher Larson <chris_larson@mentor.com>
>> >
>> > These files are treated as the contents of a bitbake variable, so
>> > usual
>> > bitbake variable references are supported. I considered using another
>> > templating mechanism, for example the one used by yocto-layer, but
>> > then we'd
>> > end up largely mapping metadata variables to template fields anyway,
>> > which is
>> > a pointless indirection. Let bitbake expand the variables directly
>> > instead.
>> >
>> > This feature lets us, for example, reference ${APPEND} in --append,
>> > and avoid
>> > hardcoding the serial console tty in the wks file, and let the user's
>> > changes
>> > to APPEND affect wic the way they do the other image construction
>> > mechanisms.
>> >
>> > The template is read in and set in a variable at parse time, so
>> > changes to the
>> > variables referenced by the template will result in rebuilding the
>> > image.
>> >
>> > Signed-off-by: Christopher Larson <chris_larson@mentor.com>
>> > ---
>> >  meta/classes/image_types.bbclass | 35
>> > +++++++++++++++++++++++++++++++++++
>> >  1 file changed, 35 insertions(+)
>> >
>> > diff --git a/meta/classes/image_types.bbclass
>> > b/meta/classes/image_types.bbclass
>> > index dc681ae..caf8757 100644
>> > --- a/meta/classes/image_types.bbclass
>> > +++ b/meta/classes/image_types.bbclass
>> > @@ -206,6 +206,16 @@ IMAGE_CMD_wic () {
>> >  }
>> >  IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES"
>> >
>> > +python process_wks_template () {
>> > +    """Write out expanded template contents to WKS_FULL_PATH."""
>> > +    template_body = d.getVar('_WKS_TEMPLATE', True)
>> > +    if template_body:
>> > +        wks_file = d.getVar('WKS_FULL_PATH', True)
>> > +        with open(wks_file, 'w') as f:
>> > +            f.write(template_body)
>> > +}
>> > +do_image_wic[prefuncs] += 'process_wks_template'
>> > +
>> >  # Rebuild when the wks file or vars in WICVARS change
>> >  USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + '
>> > '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '',
>> > d)}"
>> >  WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' %
>> > os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
>> > @@ -302,3 +312,28 @@ IMAGE_TYPES_MASKED ?= ""
>> >  # The WICVARS variable is used to define list of bitbake variables
>> > used in wic code
>> >  # variables from this list is written to <image>.env file
>> >  WICVARS ?= "BBLAYERS DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME
>> > IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES
>> > INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR
>> > STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS"
>> > +
>> > +python () {
>> > +    """Read in and set up wks file template for wic."""
>> > +    if d.getVar('USING_WIC', True):
>> > +        wks_file_u = d.getVar('WKS_FULL_PATH', False)
>> > +        wks_file = d.expand(wks_file_u)
>> > +        base, ext = os.path.splitext(wks_file)
>> > +        if ext == '.in' and os.path.exists(wks_file):
>> > +            wks_out_file = os.path.join(d.getVar('WORKDIR', True),
>> > os.path.basename(base))
>> > +            d.setVar('WKS_FULL_PATH', wks_out_file)
>> > +            d.setVar('WKS_TEMPLATE_PATH', wks_file_u)
>> > +            d.setVar('WKS_FILE_CHECKSUM',
>> > '${WKS_TEMPLATE_PATH}:True')
>> > +
>> > +            try:
>> > +                with open(wks_file, 'r') as f:
>> > +                    body = f.read()
>> > +            except (IOError, OSError) as exc:
>> > +                pass
>>
>> I'm a little nervous about determinism here. Shouldn't it either find
>> the referenced file or error?
>
>
> That's a fair point, I think I was concerned about parse time errors,
> since they tend to blow up into a lot of unpleasantness for the user, since
> parse time errors in classes can cause errors for multiple recipes. If no
> file is read in at parse time, there'll be no contents to write out, and we
> could then error when trying to build the image with wic, as no wks file
> will exist.
>
> Of course, the file being missing will change checksums for the image
> tasks, but I think that's reasonable.
>
> Thoughts?
>

I'll do a bit more testing on the behaviors and get back to you on that
later. Thanks.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 6364 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH 3/3] image_types.bbclass: support template .wks.in files for wic
  2016-05-24 16:07       ` Christopher Larson
@ 2016-05-24 17:45         ` Christopher Larson
  2016-05-24 19:28           ` Christopher Larson
  0 siblings, 1 reply; 9+ messages in thread
From: Christopher Larson @ 2016-05-24 17:45 UTC (permalink / raw)
  To: Richard Purdie; +Cc: Patches and discussions about the oe-core layer

[-- Attachment #1: Type: text/plain, Size: 5958 bytes --]

On Tue, May 24, 2016 at 9:07 AM, Christopher Larson <kergoth@gmail.com>
wrote:

>
> On Tue, May 24, 2016 at 7:26 AM, Christopher Larson <kergoth@gmail.com>
> wrote:
>
>>
>> On Tue, May 24, 2016 at 2:13 AM, Richard Purdie <
>> richard.purdie@linuxfoundation.org> wrote:
>>
>>> On Mon, 2016-05-23 at 13:34 -0700, Christopher Larson wrote:
>>> > From: Christopher Larson <chris_larson@mentor.com>
>>> >
>>> > These files are treated as the contents of a bitbake variable, so
>>> > usual
>>> > bitbake variable references are supported. I considered using another
>>> > templating mechanism, for example the one used by yocto-layer, but
>>> > then we'd
>>> > end up largely mapping metadata variables to template fields anyway,
>>> > which is
>>> > a pointless indirection. Let bitbake expand the variables directly
>>> > instead.
>>> >
>>> > This feature lets us, for example, reference ${APPEND} in --append,
>>> > and avoid
>>> > hardcoding the serial console tty in the wks file, and let the user's
>>> > changes
>>> > to APPEND affect wic the way they do the other image construction
>>> > mechanisms.
>>> >
>>> > The template is read in and set in a variable at parse time, so
>>> > changes to the
>>> > variables referenced by the template will result in rebuilding the
>>> > image.
>>> >
>>> > Signed-off-by: Christopher Larson <chris_larson@mentor.com>
>>> > ---
>>> >  meta/classes/image_types.bbclass | 35
>>> > +++++++++++++++++++++++++++++++++++
>>> >  1 file changed, 35 insertions(+)
>>> >
>>> > diff --git a/meta/classes/image_types.bbclass
>>> > b/meta/classes/image_types.bbclass
>>> > index dc681ae..caf8757 100644
>>> > --- a/meta/classes/image_types.bbclass
>>> > +++ b/meta/classes/image_types.bbclass
>>> > @@ -206,6 +206,16 @@ IMAGE_CMD_wic () {
>>> >  }
>>> >  IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES"
>>> >
>>> > +python process_wks_template () {
>>> > +    """Write out expanded template contents to WKS_FULL_PATH."""
>>> > +    template_body = d.getVar('_WKS_TEMPLATE', True)
>>> > +    if template_body:
>>> > +        wks_file = d.getVar('WKS_FULL_PATH', True)
>>> > +        with open(wks_file, 'w') as f:
>>> > +            f.write(template_body)
>>> > +}
>>> > +do_image_wic[prefuncs] += 'process_wks_template'
>>> > +
>>> >  # Rebuild when the wks file or vars in WICVARS change
>>> >  USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + '
>>> > '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '',
>>> > d)}"
>>> >  WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' %
>>> > os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
>>> > @@ -302,3 +312,28 @@ IMAGE_TYPES_MASKED ?= ""
>>> >  # The WICVARS variable is used to define list of bitbake variables
>>> > used in wic code
>>> >  # variables from this list is written to <image>.env file
>>> >  WICVARS ?= "BBLAYERS DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME
>>> > IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES
>>> > INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR
>>> > STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS"
>>> > +
>>> > +python () {
>>> > +    """Read in and set up wks file template for wic."""
>>> > +    if d.getVar('USING_WIC', True):
>>> > +        wks_file_u = d.getVar('WKS_FULL_PATH', False)
>>> > +        wks_file = d.expand(wks_file_u)
>>> > +        base, ext = os.path.splitext(wks_file)
>>> > +        if ext == '.in' and os.path.exists(wks_file):
>>> > +            wks_out_file = os.path.join(d.getVar('WORKDIR', True),
>>> > os.path.basename(base))
>>> > +            d.setVar('WKS_FULL_PATH', wks_out_file)
>>> > +            d.setVar('WKS_TEMPLATE_PATH', wks_file_u)
>>> > +            d.setVar('WKS_FILE_CHECKSUM',
>>> > '${WKS_TEMPLATE_PATH}:True')
>>> > +
>>> > +            try:
>>> > +                with open(wks_file, 'r') as f:
>>> > +                    body = f.read()
>>> > +            except (IOError, OSError) as exc:
>>> > +                pass
>>>
>>> I'm a little nervous about determinism here. Shouldn't it either find
>>> the referenced file or error?
>>
>>
>> That's a fair point, I think I was concerned about parse time errors,
>> since they tend to blow up into a lot of unpleasantness for the user, since
>> parse time errors in classes can cause errors for multiple recipes. If
>> no file is read in at parse time, there'll be no contents to write out,
>> and we could then error when trying to build the image with wic, as no
>> wks file will exist.
>>
>> Of course, the file being missing will change checksums for the image
>> tasks, but I think that's reasonable.
>>
>> Thoughts?
>>
>
> I'll do a bit more testing on the behaviors and get back to you on that
> later. Thanks.


If the WKS_FILE doesn't exist, whether we're using templates or not, it'll
fail in do_image_wic:

ERROR: core-image-sato-1.0-r0 do_image_wic: No kickstart files from
WKS_FILES were found: /scratch/dogwood/minnowmax/build/mkefidiskf.wks.in
core-image-sato.wks. Please set WKS_FILE or WKS_FILES appropriately.
ERROR: core-image-sato-1.0-r0 do_image_wic: Function failed: do_image_wic
(log file is located at
/scratch/dogwood/minnowmax/build/tmp/work/intel_corei7_64-mel-linux/core-image-sato/1.0-r0/temp/log.do_image_wic.12181)

If the template doesn't exist, then no template handling will be done at
all, so do_image_wic won't depend on any of the variables referenced in the
template, and the task will fail. If it does, then we will depend on its
contents and referenced variables, and the task will not fail.

I don't really see a problem with determinism there. Either the file exists
at parse time or it doesn't, and the behavior seems reasonable in either
case.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 8442 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH 3/3] image_types.bbclass: support template .wks.in files for wic
  2016-05-24 17:45         ` Christopher Larson
@ 2016-05-24 19:28           ` Christopher Larson
  0 siblings, 0 replies; 9+ messages in thread
From: Christopher Larson @ 2016-05-24 19:28 UTC (permalink / raw)
  To: Richard Purdie; +Cc: Patches and discussions about the oe-core layer

[-- Attachment #1: Type: text/plain, Size: 6361 bytes --]

On Tue, May 24, 2016 at 10:45 AM, Christopher Larson <kergoth@gmail.com>
wrote:

>
> On Tue, May 24, 2016 at 9:07 AM, Christopher Larson <kergoth@gmail.com>
> wrote:
>
>>
>> On Tue, May 24, 2016 at 7:26 AM, Christopher Larson <kergoth@gmail.com>
>> wrote:
>>
>>>
>>> On Tue, May 24, 2016 at 2:13 AM, Richard Purdie <
>>> richard.purdie@linuxfoundation.org> wrote:
>>>
>>>> On Mon, 2016-05-23 at 13:34 -0700, Christopher Larson wrote:
>>>> > From: Christopher Larson <chris_larson@mentor.com>
>>>> >
>>>> > These files are treated as the contents of a bitbake variable, so
>>>> > usual
>>>> > bitbake variable references are supported. I considered using another
>>>> > templating mechanism, for example the one used by yocto-layer, but
>>>> > then we'd
>>>> > end up largely mapping metadata variables to template fields anyway,
>>>> > which is
>>>> > a pointless indirection. Let bitbake expand the variables directly
>>>> > instead.
>>>> >
>>>> > This feature lets us, for example, reference ${APPEND} in --append,
>>>> > and avoid
>>>> > hardcoding the serial console tty in the wks file, and let the user's
>>>> > changes
>>>> > to APPEND affect wic the way they do the other image construction
>>>> > mechanisms.
>>>> >
>>>> > The template is read in and set in a variable at parse time, so
>>>> > changes to the
>>>> > variables referenced by the template will result in rebuilding the
>>>> > image.
>>>> >
>>>> > Signed-off-by: Christopher Larson <chris_larson@mentor.com>
>>>> > ---
>>>> >  meta/classes/image_types.bbclass | 35
>>>> > +++++++++++++++++++++++++++++++++++
>>>> >  1 file changed, 35 insertions(+)
>>>> >
>>>> > diff --git a/meta/classes/image_types.bbclass
>>>> > b/meta/classes/image_types.bbclass
>>>> > index dc681ae..caf8757 100644
>>>> > --- a/meta/classes/image_types.bbclass
>>>> > +++ b/meta/classes/image_types.bbclass
>>>> > @@ -206,6 +206,16 @@ IMAGE_CMD_wic () {
>>>> >  }
>>>> >  IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES"
>>>> >
>>>> > +python process_wks_template () {
>>>> > +    """Write out expanded template contents to WKS_FULL_PATH."""
>>>> > +    template_body = d.getVar('_WKS_TEMPLATE', True)
>>>> > +    if template_body:
>>>> > +        wks_file = d.getVar('WKS_FULL_PATH', True)
>>>> > +        with open(wks_file, 'w') as f:
>>>> > +            f.write(template_body)
>>>> > +}
>>>> > +do_image_wic[prefuncs] += 'process_wks_template'
>>>> > +
>>>> >  # Rebuild when the wks file or vars in WICVARS change
>>>> >  USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + '
>>>> > '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '',
>>>> > d)}"
>>>> >  WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' %
>>>> > os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
>>>> > @@ -302,3 +312,28 @@ IMAGE_TYPES_MASKED ?= ""
>>>> >  # The WICVARS variable is used to define list of bitbake variables
>>>> > used in wic code
>>>> >  # variables from this list is written to <image>.env file
>>>> >  WICVARS ?= "BBLAYERS DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME
>>>> > IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES
>>>> > INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR
>>>> > STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS"
>>>> > +
>>>> > +python () {
>>>> > +    """Read in and set up wks file template for wic."""
>>>> > +    if d.getVar('USING_WIC', True):
>>>> > +        wks_file_u = d.getVar('WKS_FULL_PATH', False)
>>>> > +        wks_file = d.expand(wks_file_u)
>>>> > +        base, ext = os.path.splitext(wks_file)
>>>> > +        if ext == '.in' and os.path.exists(wks_file):
>>>> > +            wks_out_file = os.path.join(d.getVar('WORKDIR', True),
>>>> > os.path.basename(base))
>>>> > +            d.setVar('WKS_FULL_PATH', wks_out_file)
>>>> > +            d.setVar('WKS_TEMPLATE_PATH', wks_file_u)
>>>> > +            d.setVar('WKS_FILE_CHECKSUM',
>>>> > '${WKS_TEMPLATE_PATH}:True')
>>>> > +
>>>> > +            try:
>>>> > +                with open(wks_file, 'r') as f:
>>>> > +                    body = f.read()
>>>> > +            except (IOError, OSError) as exc:
>>>> > +                pass
>>>>
>>>> I'm a little nervous about determinism here. Shouldn't it either find
>>>> the referenced file or error?
>>>
>>>
>>> That's a fair point, I think I was concerned about parse time errors,
>>> since they tend to blow up into a lot of unpleasantness for the user, since
>>> parse time errors in classes can cause errors for multiple recipes. If
>>> no file is read in at parse time, there'll be no contents to write out,
>>> and we could then error when trying to build the image with wic, as no
>>> wks file will exist.
>>>
>>> Of course, the file being missing will change checksums for the image
>>> tasks, but I think that's reasonable.
>>>
>>> Thoughts?
>>>
>>
>> I'll do a bit more testing on the behaviors and get back to you on that
>> later. Thanks.
>
>
> If the WKS_FILE doesn't exist, whether we're using templates or not, it'll
> fail in do_image_wic:
>
> ERROR: core-image-sato-1.0-r0 do_image_wic: No kickstart files from
> WKS_FILES were found: /scratch/dogwood/minnowmax/build/mkefidiskf.wks.in
> core-image-sato.wks. Please set WKS_FILE or WKS_FILES appropriately.
> ERROR: core-image-sato-1.0-r0 do_image_wic: Function failed: do_image_wic
> (log file is located at
> /scratch/dogwood/minnowmax/build/tmp/work/intel_corei7_64-mel-linux/core-image-sato/1.0-r0/temp/log.do_image_wic.12181)
>
> If the template doesn't exist, then no template handling will be done at
> all, so do_image_wic won't depend on any of the variables referenced in the
> template, and the task will fail. If it does, then we will depend on its
> contents and referenced variables, and the task will not fail.
>
> I don't really see a problem with determinism there. Either the file
> exists at parse time or it doesn't, and the behavior seems reasonable in
> either case.
>

Update: I've occasionally seen a taskhash mismatch, so clearly I've missed
something. Hold off on this until further notice. Thanks again for the
feedback.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 8858 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2016-05-24 19:28 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-05-23 20:34 [PATCH 0/3] Support template files for wic Christopher Larson
2016-05-23 20:34 ` [PATCH 1/3] image.bbclass: append to prefuncs/postfuncs for do_image_* Christopher Larson
2016-05-23 20:34 ` [PATCH 2/3] image_types.bbclass: add intermediate var for WKS_FILE_CHECKSUM Christopher Larson
2016-05-23 20:34 ` [PATCH 3/3] image_types.bbclass: support template .wks.in files for wic Christopher Larson
2016-05-24  9:13   ` Richard Purdie
2016-05-24 14:26     ` Christopher Larson
2016-05-24 16:07       ` Christopher Larson
2016-05-24 17:45         ` Christopher Larson
2016-05-24 19:28           ` Christopher Larson

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.