using d.expand() in a python function

using d.expand() in a python function

[Rasmus Villemoes]

I have a do_configure() python function which is essentially

python do_configure() {
    import fileinput
    with open("foo.txt", "w") as out:
        for line in fileinput.input(files=d.getVar("FOO_INPUT").split()):
            line = d.expand(line)
            out.write(line)
}

So some of those input files can contain ${VARIABLE} and it gets
replaced appropriately. But to my surprise, when I then change VARIABLE,
the recipe doesn't get rebuilt, so it seems that bitbake doesn't
automatically realize that the recipe (or task) depends on VARIABLE.

So digging a bit into the code, it seems that d.expand() calls
expandWithRefs(), which returns a VariableParse object, but then only
uses the .value property, throwing away .references. So I assume that
what I really should be doing is call expandWithRefs and then somehow
manually pass .references to... something? Or, there must be some other
bitbake interface for doing this properly?

Rasmus

Re: [yocto] using d.expand() in a python function

[Richard Purdie]

On Wed, 2020-01-22 at 13:55 +0000, Rasmus Villemoes wrote:
> I have a do_configure() python function which is essentially
> 
> python do_configure() {
>     import fileinput
>     with open("foo.txt", "w") as out:
>         for line in
> fileinput.input(files=d.getVar("FOO_INPUT").split()):
>             line = d.expand(line)
>             out.write(line)
> }
> 
> So some of those input files can contain ${VARIABLE} and it gets
> replaced appropriately. But to my surprise, when I then change
> VARIABLE,
> the recipe doesn't get rebuilt, so it seems that bitbake doesn't
> automatically realize that the recipe (or task) depends on VARIABLE.
> 
> So digging a bit into the code, it seems that d.expand() calls
> expandWithRefs(), which returns a VariableParse object, but then only
> uses the .value property, throwing away .references. So I assume that
> what I really should be doing is call expandWithRefs and then somehow
> manually pass .references to... something? Or, there must be some
> other
> bitbake interface for doing this properly?

Bitbake can only detect direct references to variables. The code above
only runs at configure time, not at parse time so its not really
surprising that bitbake can't know what its doing.

You'd need to read the data from this file at parse time for bitbake to
stand some chance of seeing it, or manually set the variables it
depends upon using the vardeps flag.

Something like:

do_configure[vardeps] = "${@my_depends_function(d)}"

where my_depends_function would return a list of variables it
references.

Reading files like that at parse time doesn't work out well for
performance.

Cheers,

Richard

Re: [yocto] using d.expand() in a python function

[Rasmus Villemoes]

On 22/01/2020 15.35, Richard Purdie wrote:
> On Wed, 2020-01-22 at 13:55 +0000, Rasmus Villemoes wrote:
>> I have a do_configure() python function which is essentially
>>
>> python do_configure() {
>>     import fileinput
>>     with open("foo.txt", "w") as out:
>>         for line in
>> fileinput.input(files=d.getVar("FOO_INPUT").split()):
>>             line = d.expand(line)
>>             out.write(line)
>> }
>>
>> So some of those input files can contain ${VARIABLE} and it gets
>> replaced appropriately. But to my surprise, when I then change
>> VARIABLE,
>> the recipe doesn't get rebuilt, so it seems that bitbake doesn't
>> automatically realize that the recipe (or task) depends on VARIABLE.
>>
>> So digging a bit into the code, it seems that d.expand() calls
>> expandWithRefs(), which returns a VariableParse object, but then only
>> uses the .value property, throwing away .references. So I assume that
>> what I really should be doing is call expandWithRefs and then somehow
>> manually pass .references to... something? Or, there must be some
>> other
>> bitbake interface for doing this properly?
> 
> Bitbake can only detect direct references to variables. The code above
> only runs at configure time, not at parse time so its not really
> surprising that bitbake can't know what its doing.

Well, I thought that as well at first. But there must be something done
at run-time to feed back the variable dependencies, because if I add a
completely dumb

python do_configure() {
    import fileinput
+    d.getVar("VARIABLE") # silly
    with open("foo.txt", "w") as out:
        for line in fileinput.input(files=d.getVar("FOO_INPUT").split()):
            line = d.expand(line)
            out.write(line)
}

build the recipe (because now it has of course changed), and then change
VARIABLE, the recipe does get re-built. Or does bitbake specifically
look for d.getVar(<literal>) when it parses a python function?

> You'd need to read the data from this file at parse time for bitbake to
> stand some chance of seeing it, or manually set the variables it
> depends upon using the vardeps flag.
> 
> Something like:
> 
> do_configure[vardeps] = "${@my_depends_function(d)}"
> 
> where my_depends_function would return a list of variables it
> references.
> 
> Reading files like that at parse time doesn't work out well for
> performance.

Yeah, so I'd very much like to avoid that, and the above observation
suggests that I might be able to do the expansion manually, just looking
for the \$\{...\} regexp and calling getVar to find the replacement. But
that seems like something that should be solved already.

Thanks,
Rasmus

Re: [yocto] using d.expand() in a python function

[Richard Purdie]

On Wed, 2020-01-22 at 15:46 +0000, Rasmus Villemoes wrote:
> On 22/01/2020 15.35, Richard Purdie wrote:
> > Bitbake can only detect direct references to variables. The code
> > above only runs at configure time, not at parse time so its not
> > really surprising that bitbake can't know what its doing.
> 
> Well, I thought that as well at first. But there must be something
> done at run-time to feed back the variable dependencies, because if I
> add a completely dumb
> 
> python do_configure() {
>     import fileinput
> +    d.getVar("VARIABLE") # silly
>     with open("foo.txt", "w") as out:
>         for line in
> fileinput.input(files=d.getVar("FOO_INPUT").split()):
>             line = d.expand(line)
>             out.write(line)
> }
> 
> build the recipe (because now it has of course changed), and then
> change VARIABLE, the recipe does get re-built. Or does bitbake
> specifically look for d.getVar(<literal>) when it parses a python
> function?

Bitbake has ASTs for all its python fragments and can search those for
getVar() calls. It will spot getVar("X") but can't know what is in
d.expand(line).

> > You'd need to read the data from this file at parse time for
> > bitbake to
> > stand some chance of seeing it, or manually set the variables it
> > depends upon using the vardeps flag.
> > 
> > Something like:
> > 
> > do_configure[vardeps] = "${@my_depends_function(d)}"
> > 
> > where my_depends_function would return a list of variables it
> > references.
> > 
> > Reading files like that at parse time doesn't work out well for
> > performance.
> 
> Yeah, so I'd very much like to avoid that, and the above observation
> suggests that I might be able to do the expansion manually, just
> looking for the \$\{...\} regexp and calling getVar to find the
> replacement. But that seems like something that should be solved
> already.

Well, if you have the expansion, just put it in a variable the function
references...

Cheers,

Richard

Re: [yocto] using d.expand() in a python function

[Rasmus Villemoes]

On 22/01/2020 16.50, Richard Purdie wrote:
> On Wed, 2020-01-22 at 15:46 +0000, Rasmus Villemoes wrote:
>> On 22/01/2020 15.35, Richard Purdie wrote:
>>> Bitbake can only detect direct references to variables. The code
>>> above only runs at configure time, not at parse time so its not
>>> really surprising that bitbake can't know what its doing.
>>
>> Well, I thought that as well at first. But there must be something
>> done at run-time to feed back the variable dependencies, because if I
>> add a completely dumb
>>
>> python do_configure() {
>>     import fileinput
>> +    d.getVar("VARIABLE") # silly
>>     with open("foo.txt", "w") as out:
>>         for line in
>> fileinput.input(files=d.getVar("FOO_INPUT").split()):
>>             line = d.expand(line)
>>             out.write(line)
>> }
>>
>> build the recipe (because now it has of course changed), and then
>> change VARIABLE, the recipe does get re-built. Or does bitbake
>> specifically look for d.getVar(<literal>) when it parses a python
>> function?
> 
> Bitbake has ASTs for all its python fragments and can search those for
> getVar() calls. It will spot getVar("X") but can't know what is in
> d.expand(line).

Yeah, I just tried changing it to

  v1 = "VAR"
  v2 = "IABLE"
  d.getVar(v1 + v2)

and bitbake no longer knew about the dependency on VARIABLE :(

I can't be the first one to want to substitute bitbake variables in
various source files, without necessarily knowing upfront which
variables might appear.

So thinking out loud (and apologies if this is very silly): Would it be
possible to implement some auto-deps class that a recipe playing the
above game could inherit. Something like

AUTO_DEPS_TASKS = "configure"
python() {
  for t in AUTO_DEPS_TASKS:
    f = (tmpdir/cachedir/somewhere)/autodeps/pn/t.inc
    if f does not exist:
       make sure the recipe gets rebuilt
       return
    include f
}

python write_autodeps(d, t):
  logic to write a "do_t[vardeps] += ..." line to f, called at the end
of t (preferably automatically, but manually if need be)

python expand_autodeps(d, s):
  helper to be used within an autodeps task, recording any variable used
in the expansion of s

Rasmus

Re: [yocto] using d.expand() in a python function

[Richard Purdie]

On Wed, 2020-01-22 at 16:39 +0000, Rasmus Villemoes wrote:
> On 22/01/2020 16.50, Richard Purdie wrote:
> > On Wed, 2020-01-22 at 15:46 +0000, Rasmus Villemoes wrote:
> > > On 22/01/2020 15.35, Richard Purdie wrote:
> > > > Bitbake can only detect direct references to variables. The
> > > > code
> > > > above only runs at configure time, not at parse time so its not
> > > > really surprising that bitbake can't know what its doing.
> > > 
> > > Well, I thought that as well at first. But there must be
> > > something
> > > done at run-time to feed back the variable dependencies, because
> > > if I
> > > add a completely dumb
> > > 
> > > python do_configure() {
> > >     import fileinput
> > > +    d.getVar("VARIABLE") # silly
> > >     with open("foo.txt", "w") as out:
> > >         for line in
> > > fileinput.input(files=d.getVar("FOO_INPUT").split()):
> > >             line = d.expand(line)
> > >             out.write(line)
> > > }
> > > 
> > > build the recipe (because now it has of course changed), and then
> > > change VARIABLE, the recipe does get re-built. Or does bitbake
> > > specifically look for d.getVar(<literal>) when it parses a python
> > > function?
> > 
> > Bitbake has ASTs for all its python fragments and can search those
> > for
> > getVar() calls. It will spot getVar("X") but can't know what is in
> > d.expand(line).
> 
> Yeah, I just tried changing it to
> 
>   v1 = "VAR"
>   v2 = "IABLE"
>   d.getVar(v1 + v2)
> 
> and bitbake no longer knew about the dependency on VARIABLE :(
> 
> I can't be the first one to want to substitute bitbake variables in
> various source files, without necessarily knowing upfront which
> variables might appear.
> 
> So thinking out loud (and apologies if this is very silly): Would it
> be
> possible to implement some auto-deps class that a recipe playing the
> above game could inherit. Something like
> 
> AUTO_DEPS_TASKS = "configure"
> python() {
>   for t in AUTO_DEPS_TASKS:
>     f = (tmpdir/cachedir/somewhere)/autodeps/pn/t.inc
>     if f does not exist:
>        make sure the recipe gets rebuilt
>        return
>     include f
> }
> 
> python write_autodeps(d, t):
>   logic to write a "do_t[vardeps] += ..." line to f, called at the
> end
> of t (preferably automatically, but manually if need be)
> 
> python expand_autodeps(d, s):
>   helper to be used within an autodeps task, recording any variable
> used
> in the expansion of s

Why not do what I already suggested?

I suspect rather than a class you're looking for a helper function in
lib/oe to call into? Just move my suggested function somewhere there?

Cheers,

Richard

Re: [yocto] using d.expand() in a python function

[Rasmus Villemoes]

On 22/01/2020 17.39, Rasmus Villemoes via Lists.Yoctoproject.Org wrote:
> On 22/01/2020 16.50, Richard Purdie wrote:
>> On Wed, 2020-01-22 at 15:46 +0000, Rasmus Villemoes wrote:
>>> On 22/01/2020 15.35, Richard Purdie wrote:
>>>> Bitbake can only detect direct references to variables. The code
>>>> above only runs at configure time, not at parse time so its not
>>>> really surprising that bitbake can't know what its doing.
>>>
>>> Well, I thought that as well at first. But there must be something
>>> done at run-time to feed back the variable dependencies, because if I
>>> add a completely dumb
>>>
>>> python do_configure() {
>>>     import fileinput
>>> +    d.getVar("VARIABLE") # silly
>>>     with open("foo.txt", "w") as out:
>>>         for line in
>>> fileinput.input(files=d.getVar("FOO_INPUT").split()):
>>>             line = d.expand(line)
>>>             out.write(line)
>>> }
>>>
> So thinking out loud (and apologies if this is very silly): Would it be
> possible to implement some auto-deps class

Alternatively, as I will only need this for files I provide via file://
SRC_URIs, perhaps that fetcher could be taught a ";varexpand=1"
parameter. Bitbake must be doing md5sum or similar on local files anyway
(? how else are changes in those detected), so having varexpand=1 mean
"grep all ${foo} references and add foo to do_unpack[vardeps]" during
parsing shouldn't add that much.

And of course having bitbake automatically actually do the expansion as
part of placing the file in WORKDIR means one can avoid that logic
altogether in the recipe (so my do_configure could become a simple shell
function that does "cat ${FOO_INPUT} > foo.txt").

Rasmus

Re: [yocto] using d.expand() in a python function

[Rasmus Villemoes]

On 22/01/2020 17.59, Richard Purdie wrote:
> On Wed, 2020-01-22 at 16:39 +0000, Rasmus Villemoes wrote:
>> On 22/01/2020 16.50, Richard Purdie wrote:
>>> On Wed, 2020-01-22 at 15:46 +0000, Rasmus Villemoes wrote:
>>>> On 22/01/2020 15.35, Richard Purdie wrote:
>>
>> I can't be the first one to want to substitute bitbake variables in
>> various source files, without necessarily knowing upfront which
>> variables might appear.
>>
>> So thinking out loud (and apologies if this is very silly): Would it
>> be
>> possible to implement some auto-deps class 
> 
> Why not do what I already suggested?

Sorry, I'm probably a bit slow. Do you mean this part

  Well, if you have the expansion, just put it in a variable the
  function references...

? That was in response to something that I since discovered wouldn't
work [because d.getVar(line) doesn't magically feed back the deps info
as my naive experiment led me to believe], so I must admit I didn't
think too hard about it. But, I don't see how slurping in the file(s),
doing the expansion, and putting the result in some variable that I then
do a (dummy?) reference to would help teach bitbake which variables were
used during the expansion.

> I suspect rather than a class you're looking for a helper function in
> lib/oe to call into? Just move my suggested function somewhere there?

Well, yes, I'm certainly looking for or asking for a library function
that will do what I want. But I'm not really seeing how to implement
that. I do now see how the variable dependencies must be known at parse
time.

Rasmus