Incremental rebuild doesn't pick up FILES changes.

Incremental rebuild doesn't pick up FILES changes.

[Kaz Kylheku (poky)]

Hi,

I'm working in a Yocto 2.5 environment.

In my own layer, I added a bbappend targeting python which does this:

   FILES_${PN}_remove = "${libdir}/python2.7/encodings"

Now this didn't take effect as I expected. I had to do a "bitbake -c 
cleanall python" to rebuild the package.

In any case, all I have in the image now, as I expected, is these four 
files, vastly reduced from dozens:

   /usr/lib/python2.7/encodings/{aliases,utf-8}.{py,pyc}

Now, if I comment out this FILES_${PN}_remove variable assignment, I 
cannot get the previously removed encoding modules to re-appear in my 
image! I tried "bitbake -c cleanall" on python, and blowing away its 
build directory. It didn't work; still there are just those four files.

How do we get Yocto to react to FILES changes, short of blowing away the 
entire build tree?

Re: Incremental rebuild doesn't pick up FILES changes.

[Richard Purdie]

On Tue, 2019-03-05 at 18:28 -0800, Kaz Kylheku (poky) wrote:
> Hi,
> 
> I'm working in a Yocto 2.5 environment.
> 
> In my own layer, I added a bbappend targeting python which does this:
> 
>    FILES_${PN}_remove = "${libdir}/python2.7/encodings"
> 
> Now this didn't take effect as I expected. I had to do a "bitbake -c 
> cleanall python" to rebuild the package.
> 
> In any case, all I have in the image now, as I expected, is these
> four 
> files, vastly reduced from dozens:
> 
>    /usr/lib/python2.7/encodings/{aliases,utf-8}.{py,pyc}
> 
> Now, if I comment out this FILES_${PN}_remove variable assignment, I 
> cannot get the previously removed encoding modules to re-appear in
> my 
> image! I tried "bitbake -c cleanall" on python, and blowing away its 
> build directory. It didn't work; still there are just those four
> files.
> 
> How do we get Yocto to react to FILES changes, short of blowing away
> the entire build tree?

Maybe related to this change:

http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/data_smart.py?id=f7f5e30667e1ad8e1ca76ee331be2843f2976bfa

?

Cheers,

Richard

Re: Incremental rebuild doesn't pick up FILES changes.

[Kaz Kylheku (poky)]

On 2019-03-06 10:23, Richard Purdie wrote:
> On Tue, 2019-03-05 at 18:28 -0800, Kaz Kylheku (poky) wrote:
>> How do we get Yocto to react to FILES changes, short of blowing away
>> the entire build tree?
> 
> Maybe related to this change:
> 
> http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/data_smart.py?id=f7f5e30667e1ad8e1ca76ee331be2843f2976bfa

Actually, the problem was that I had an accidental local change applied
to the .json manifest file; that's where the 
"${libdir}/python2.7/encodings" was
being deleted.

So, the issues are now is that

FILES_${PN}_remove = "${libdir}/python2.7/encodings"

in fact does not work, as I believed.  It doesn't register at all. If I 
run "bitbake -e", to find the places that influence the value of $FILES, 
this _remove assignment doesn't appear anywhere. In fact, the name of my 
.bbpappend file doesn't even appear anywhere in that listing.

If I change it to just FILES_remove = ...  then it does show up in 
"bitbake -e" (early, before all the base recipe manipulations of 
$FILES). It doesn't work; the encodings do not disappear:


      $FILES [527 operations]
     #   set /mnt/sdb4/kaz/proj/poky/meta/conf/bitbake.conf:288
     #     ""
     #   set /mnt/sdb4/kaz/proj/poky/meta/conf/documentation.conf:169
     #     [doc] "The list of directories or files that are placed in 
packages."
     #   _remove 
/mnt/sdb4/kaz/proj/meta-custom/common/recipes-devtools/python/python_%.bbappend:15
     #     "${libdir}/python2.7/encodings"

     [ big snip ]

     #   override[python-core]:append python_2.7.15.bb:220 
[__anon_238__mnt_sdb4_kaz_y25_wp76_poky_meta_recipes_devtools_python_python_2_7_15_bb]
     #     " ${libdir}/python2.7/encodings"

I think this override might be winning over the _remove or something. 
The output never shows the final, settled value of FILES. At the bottom 
of this section about the $FILES variable we just have:

     # pre-expansion value:
     #   ""
     FILES=""

which is uninformative; obviously there is a populated FILES because the 
package has files in it.

Should I just add some install append steps to "rm -f" the unwanted 
material and be done with it?

I feel there should be a sane way to customize the FILES variable (and 
any other) emanating from a base recipe, though.

Re: Incremental rebuild doesn't pick up FILES changes.

[Burton, Ross]

On Wed, 6 Mar 2019 at 20:14, Kaz Kylheku (poky)
<442-103-8455@kylheku.com> wrote:
> So, the issues are now is that
>
> FILES_${PN}_remove = "${libdir}/python2.7/encodings"
>
> in fact does not work, as I believed.

Have a look at python.bb.  Specifically:

PACKAGES_remove = "${PN}"

There is no PN being packaged.

What are you actually trying to do?

Ross

Re: Incremental rebuild doesn't pick up FILES changes.

[Kaz Kylheku (poky)]

On 2019-03-06 12:29, Burton, Ross wrote:
> On Wed, 6 Mar 2019 at 20:14, Kaz Kylheku (poky)
> <442-103-8455@kylheku.com> wrote:
>> So, the issues are now is that
>> 
>> FILES_${PN}_remove = "${libdir}/python2.7/encodings"
>> 
>> in fact does not work, as I believed.
> 
> Have a look at python.bb.  Specifically:
> 
> PACKAGES_remove = "${PN}"
> 
> There is no PN being packaged.

Ah, okay; this is because it's been split into various sub-packages. So 
we have to operate on "-core".
The .json manifest brings these into the "core" package:

             "${libdir}/python2.7/encodings",
             "${libdir}/python2.7/encodings/aliases.py",
             "${libdir}/python2.7/encodings/utf_8.py",

I don't want all of encoders, just the utf_8.py and the aliases 
dictionary.

So I'm guessing this should be

   FILES_${PN}-core_remove =

Re: Incremental rebuild doesn't pick up FILES changes.

[Burton, Ross]

On Wed, 6 Mar 2019 at 22:27, Kaz Kylheku (poky)
<442-103-8455@kylheku.com> wrote:
> Ah, okay; this is because it's been split into various sub-packages. So
> we have to operate on "-core".
> The .json manifest brings these into the "core" package:
>
>              "${libdir}/python2.7/encodings",
>              "${libdir}/python2.7/encodings/aliases.py",
>              "${libdir}/python2.7/encodings/utf_8.py",
>
> I don't want all of encoders, just the utf_8.py and the aliases
> dictionary.
>
> So I'm guessing this should be
>
>    FILES_${PN}-core_remove =

Yes.  It's too late to think about parse ordering because of the
complexity of Python's packaging but that *should* work.

Worst case: provide your own manifest if the packaging isn't what you want.

Ross

Re: Incremental rebuild doesn't pick up FILES changes.

[Kaz Kylheku (poky)]

On 2019-03-06 14:26, Kaz Kylheku (poky) wrote:
> So I'm guessing this should be
> 
>   FILES_${PN}-core_remove =

No dice. FILES_${PN}-core_remove = "${libdir}/python-2.7/encodings" does 
not register anywhere. In the "bitbake -e" output, no _remove event 
appears for FILES_python-core, and the image still contains that 
encodings directory.

I'm afraid I'm outsmarted here by the Smart Dictionary.

This ugly fallback is doing what I want though:

   do_install_append() {
     encodings=${D}/usr/lib/python2.7/encodings

     if [ -e "$encodings" ] ; then
       find "$encodings" -type f | grep -v -E 
'/(utf_8|aliases)\.(py|pyc)$' | xargs rm -f
     fi
   }