[PATCH] image.bbclass: Do not expand variables in IMAGE_CMD vardepsexclude

[PATCH] image.bbclass: Do not expand variables in IMAGE_CMD vardepsexclude

[Jonathan Liu]

This fixes tashhash mismatch errors creating images when IMAGE_CMD
references a variable whose value is always changing even though the
variable is specified in IMAGE_CMD vardepsexclude.

Signed-off-by: Jonathan Liu <net147@gmail.com>
---
 meta/classes/image.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 8c8b8a2..f85ea86 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -441,6 +441,10 @@ python () {
         localdata.delVar('DATETIME')
         localdata.delVar('TMPDIR')
 
+        vardepsexclude = d.getVarFlag('IMAGE_CMD', 'vardepsexclude', True) or ''
+        vardepsexclude += ' ' + (d.getVarFlag('IMAGE_CMD_' + realt, 'vardepsexclude', True) or '')
+        for var in vardepsexclude.split():
+            localdata.delVar(var)
         image_cmd = localdata.getVar("IMAGE_CMD", True)
         vardeps.add('IMAGE_CMD_' + realt)
         if image_cmd:
-- 
2.9.3

Re: [PATCH] image.bbclass: Do not expand variables in IMAGE_CMD vardepsexclude

[Richard Purdie]

On Sun, 2016-09-11 at 22:14 +1000, Jonathan Liu wrote:
> This fixes tashhash mismatch errors creating images when IMAGE_CMD
> references a variable whose value is always changing even though the
> variable is specified in IMAGE_CMD vardepsexclude.
> 
> Signed-off-by: Jonathan Liu <net147@gmail.com>
> ---
>  meta/classes/image.bbclass | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 8c8b8a2..f85ea86 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -441,6 +441,10 @@ python () {
>          localdata.delVar('DATETIME')
>          localdata.delVar('TMPDIR')
>  
> +        vardepsexclude = d.getVarFlag('IMAGE_CMD', 'vardepsexclude',
> True) or ''
> +        vardepsexclude += ' ' + (d.getVarFlag('IMAGE_CMD_' + realt,
> 'vardepsexclude', True) or '')
> +        for var in vardepsexclude.split():
> +            localdata.delVar(var)
>          image_cmd = localdata.getVar("IMAGE_CMD", True)
>          vardeps.add('IMAGE_CMD_' + realt)
>          if image_cmd:

I think this is complicating an already fragile situation and it
worries me. Can you be more specific about which variable is causing
problems here?

I'm starting to wonder if we should have an explicit flag set here for
"problem" variables and we'd default to the values "DATETIME TMPDIR"
which are already handled in the code. You'd then just need to extend
that value. Using vardepsexclude seems to partially but not completely
match?

Incidentally, I really want to complete some of the rework in this area
so we get to "proper" functions and can stop playing games like this. I
just haven't had the time to do that and this point in the release
cycle isn't the right time to do that.

Cheers,

Richard

Re: [PATCH] image.bbclass: Do not expand variables in IMAGE_CMD vardepsexclude

[Jonathan Liu]

Hi Richard,

On 12 September 2016 at 20:19, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Sun, 2016-09-11 at 22:14 +1000, Jonathan Liu wrote:
>> This fixes tashhash mismatch errors creating images when IMAGE_CMD
>> references a variable whose value is always changing even though the
>> variable is specified in IMAGE_CMD vardepsexclude.
>>
>> Signed-off-by: Jonathan Liu <net147@gmail.com>
>> ---
>>  meta/classes/image.bbclass | 4 ++++
>>  1 file changed, 4 insertions(+)
>>
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index 8c8b8a2..f85ea86 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -441,6 +441,10 @@ python () {
>>          localdata.delVar('DATETIME')
>>          localdata.delVar('TMPDIR')
>>
>> +        vardepsexclude = d.getVarFlag('IMAGE_CMD', 'vardepsexclude',
>> True) or ''
>> +        vardepsexclude += ' ' + (d.getVarFlag('IMAGE_CMD_' + realt,
>> 'vardepsexclude', True) or '')
>> +        for var in vardepsexclude.split():
>> +            localdata.delVar(var)
>>          image_cmd = localdata.getVar("IMAGE_CMD", True)
>>          vardeps.add('IMAGE_CMD_' + realt)
>>          if image_cmd:
>
> I think this is complicating an already fragile situation and it
> worries me. Can you be more specific about which variable is causing
> problems here?

See IMAGEDATESTAMP variable in:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-raspberrypi/tree/classes/sdcard_image-rpi.bbclass
https://github.com/linux-sunxi/meta-sunxi/blob/master/classes/sdcard_image-sunxi.bbclass

You can test by changing to:
IMAGEDATESTAMP = "${@time.clock()}"

>
> I'm starting to wonder if we should have an explicit flag set here for
> "problem" variables and we'd default to the values "DATETIME TMPDIR"
> which are already handled in the code. You'd then just need to extend
> that value. Using vardepsexclude seems to partially but not completely
> match?
>
> Incidentally, I really want to complete some of the rework in this area
> so we get to "proper" functions and can stop playing games like this. I
> just haven't had the time to do that and this point in the release
> cycle isn't the right time to do that.
>
> Cheers,
>
> Richard
>
>

Regards,
Jonathan

Re: [PATCH] image.bbclass: Do not expand variables in IMAGE_CMD vardepsexclude

[Richard Purdie]

On Tue, 2016-09-13 at 00:05 +1000, Jonathan Liu wrote:
> Hi Richard,
> 
> On 12 September 2016 at 20:19, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > 
> > On Sun, 2016-09-11 at 22:14 +1000, Jonathan Liu wrote:
> > > 
> > > This fixes tashhash mismatch errors creating images when
> > > IMAGE_CMD
> > > references a variable whose value is always changing even though
> > > the
> > > variable is specified in IMAGE_CMD vardepsexclude.
> > > 
> > > Signed-off-by: Jonathan Liu <net147@gmail.com>
> > > ---
> > >  meta/classes/image.bbclass | 4 ++++
> > >  1 file changed, 4 insertions(+)
> > > 
> > > diff --git a/meta/classes/image.bbclass
> > > b/meta/classes/image.bbclass
> > > index 8c8b8a2..f85ea86 100644
> > > --- a/meta/classes/image.bbclass
> > > +++ b/meta/classes/image.bbclass
> > > @@ -441,6 +441,10 @@ python () {
> > >          localdata.delVar('DATETIME')
> > >          localdata.delVar('TMPDIR')
> > > 
> > > +        vardepsexclude = d.getVarFlag('IMAGE_CMD',
> > > 'vardepsexclude',
> > > True) or ''
> > > +        vardepsexclude += ' ' + (d.getVarFlag('IMAGE_CMD_' +
> > > realt,
> > > 'vardepsexclude', True) or '')
> > > +        for var in vardepsexclude.split():
> > > +            localdata.delVar(var)
> > >          image_cmd = localdata.getVar("IMAGE_CMD", True)
> > >          vardeps.add('IMAGE_CMD_' + realt)
> > >          if image_cmd:
> > I think this is complicating an already fragile situation and it
> > worries me. Can you be more specific about which variable is
> > causing
> > problems here?
> See IMAGEDATESTAMP variable in:
> https://git.yoctoproject.org/cgit/cgit.cgi/meta-raspberrypi/tree/clas
> ses/sdcard_image-rpi.bbclass
> https://github.com/linux-sunxi/meta-sunxi/blob/master/classes/sdcard_
> image-sunxi.bbclass
> 
> You can test by changing to:
> IMAGEDATESTAMP = "${@time.clock()}"

I appreciate this doesn't solve the underlying problem but is there a
reason they can't just use ${DATE} there?

I mention this since DATE/TIME are handled by bitbake specially so that
for example if your build rolls over midnight, it doesn't explode
badly. If your images is generated pre midnight but them processed
afterwards, I can see things working badly for that code...

Cheers,

Richard

Re: [PATCH] image.bbclass: Do not expand variables in IMAGE_CMD vardepsexclude

[Jonathan Liu]

Hi Richard,

On 13 September 2016 at 02:43, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Tue, 2016-09-13 at 00:05 +1000, Jonathan Liu wrote:
>> Hi Richard,
>>
>> On 12 September 2016 at 20:19, Richard Purdie
>> <richard.purdie@linuxfoundation.org> wrote:
>> >
>> > On Sun, 2016-09-11 at 22:14 +1000, Jonathan Liu wrote:
>> > >
>> > > This fixes tashhash mismatch errors creating images when
>> > > IMAGE_CMD
>> > > references a variable whose value is always changing even though
>> > > the
>> > > variable is specified in IMAGE_CMD vardepsexclude.
>> > >
>> > > Signed-off-by: Jonathan Liu <net147@gmail.com>
>> > > ---
>> > >  meta/classes/image.bbclass | 4 ++++
>> > >  1 file changed, 4 insertions(+)
>> > >
>> > > diff --git a/meta/classes/image.bbclass
>> > > b/meta/classes/image.bbclass
>> > > index 8c8b8a2..f85ea86 100644
>> > > --- a/meta/classes/image.bbclass
>> > > +++ b/meta/classes/image.bbclass
>> > > @@ -441,6 +441,10 @@ python () {
>> > >          localdata.delVar('DATETIME')
>> > >          localdata.delVar('TMPDIR')
>> > >
>> > > +        vardepsexclude = d.getVarFlag('IMAGE_CMD',
>> > > 'vardepsexclude',
>> > > True) or ''
>> > > +        vardepsexclude += ' ' + (d.getVarFlag('IMAGE_CMD_' +
>> > > realt,
>> > > 'vardepsexclude', True) or '')
>> > > +        for var in vardepsexclude.split():
>> > > +            localdata.delVar(var)
>> > >          image_cmd = localdata.getVar("IMAGE_CMD", True)
>> > >          vardeps.add('IMAGE_CMD_' + realt)
>> > >          if image_cmd:
>> > I think this is complicating an already fragile situation and it
>> > worries me. Can you be more specific about which variable is
>> > causing
>> > problems here?
>> See IMAGEDATESTAMP variable in:
>> https://git.yoctoproject.org/cgit/cgit.cgi/meta-raspberrypi/tree/clas
>> ses/sdcard_image-rpi.bbclass
>> https://github.com/linux-sunxi/meta-sunxi/blob/master/classes/sdcard_
>> image-sunxi.bbclass
>>
>> You can test by changing to:
>> IMAGEDATESTAMP = "${@time.clock()}"
>
> I appreciate this doesn't solve the underlying problem but is there a
> reason they can't just use ${DATE} there?

I have submitted patches to meta-raspberrypi and meta-sunxi to remove
IMAGEDATESTAMP since it is using IMAGE_NAME which already includes
date and time.

>
> I mention this since DATE/TIME are handled by bitbake specially so that
> for example if your build rolls over midnight, it doesn't explode
> badly. If your images is generated pre midnight but them processed
> afterwards, I can see things working badly for that code...
>
> Cheers,
>
> Richard

Regards,
Jonathan