[PATCH 0/1] bitbake-whatchanged: print what is about to happen

[PATCH 0/1] bitbake-whatchanged: print what is about to happen

[Robert Yang]

The following changes since commit c86d26cb976e665b1516e72153f3f686f62dedf9:

  subversion: Add missing build dependency on sqlite3 (2012-08-06 16:14:06 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib robert/whatchanged
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/whatchanged

Robert Yang (1):
  bitbake-whatchanged: print what is about to happen

 bitbake/bin/bitbake-whatchanged | 334 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 334 insertions(+)
 create mode 100755 bitbake/bin/bitbake-whatchanged

-- 
1.7.11.2

[PATCH 1/1] bitbake-whatchanged: print what is about to happen

[Robert Yang]

* Contents:
  - Summary
  - Usage
  - Implementation summary
  - Output
  - TODO

* Summary:
  This is used for printing out what is about to happen between the
  current and last builds, for example:

  $ bitbake core-image-sato
  # Edit some recipes
  $ bitbake-whatchanged core-image-sato

  The changes will be printed.

* Usage:
  bitbake-whatchanged [[opts] recipe]

* Implementation summary:
  - Move the tmp/stamps to tmp/stamps.old
  - Use the "bitbake -S recipe" to generate the new stamps

  - When "the -v" (verbose) is not specified:
    > Figure out the newly added tasks
    > Figure out the PV (including PE) and PR changeds "PN", I think
      that we the tasks details here, since all the task will change
      when PV or PR changes.
    > The left tasks are the ones that the "DEPENDS"(Or dependency might
      be better?) changed ones.

  - When "-v" is specified:
    > Figure out the newly added tasks
    > Use bb.siggen.compare_sigfiles to figure out the details

* Output, for example (between core-image-sato and meta-toolchain-sdk,
  and with recipes upgraded):
  - without "-v":
    Figuring out the TMPDIR ...
    Moving the stamps to stamps.old ...
    Generating the new stamps ... (need several minutes)
    === Newly added tasks: (807 tasks)
      libpthread-stubs-nativesdk: do_install do_build do_fetch do_package_write_rpm [snip]
      telepathy-mission-control: do_unpack do_build do_package_write_rpm do_package [snip]
      [snip]

    === PV changed: (48 tasks)
      bison-native: 2.5 -> 2.5.1
      atk: 2.2.0 -> 2.4.0
      [snip]

    === PR changed: (276 tasks)
      gdk-pixbuf-native: r3 -> r4
      db-native: r7 -> r8
      [snip]

    === DEPENDS changed: (2522 tasks)
      speex: do_package_write_rpm do_configure do_package do_populate_sysroot [snip]
      libgpg-error-native: do_build do_package_write do_package do_package_write_rpm [snip]
      [snip]

    === Summary: (3653 changed, 1927 unchanged)
    Newly added: 807, PV changed: 48, PR changed: 276, Depends changed: 2522

    Removing the newly generated stamps ...
    Moving the stamps.old back to stamps ...

  - with "-v": (No PV or PR changes summary)
    === Newly added tasks: (807 tasks)
      libpthread-stubs-nativesdk: do_install do_build do_fetch do_package_write_rpm [snip]
      telepathy-mission-control: do_unpack do_build do_package_write_rpm do_package [snip]
      [snip]

    === The verbose changes of libice:
    Dependency on task bash_4.2.bb.do_package_write was added with hash d03d35205f8c897b2d37c1ebf6249fbf
    Dependency on task bash_3.2.48.bb.do_package_write was removed with hash 4bdef77351789cb078742430fd034212
    [snip]

    === The verbose changes of bash:
    basehash changed from 2a3301fad820eaad859ae738b2cdb613 to 8086533b1abb4ab5b8507ff5e4a3046f
    List of dependencies for variable sbindir changed from set(['exec_prefix']) to set([])
    List of dependencies for variable bindir changed from set(['exec_prefix']) to set([])
    [snip]

    === Summary: (3653 changed, 1927 unchanged)
    Newly added: 807, Depends changed: 2846

    Removing the newly generated stamps ...
    Moving the stamps.old back to stamps ...

* TODO
  - It seems that the "bitabke -S core-image-sato" has bugs, it would always
    report errors, but doesn't fatal errors

  - "bitbake -S" always generate the nativesdk stamps regardless of what
     we build.

  - The gcc-cross' stamps are in tmp/stamps/work-shared, but the
    "bitbake -S" doesn't put the stamps in work-shared. This doesn't
     matter to the "bitbake-whatchanged" since it uses the os.walk.

  - Print the ones which can be installed from the sstate.

[YOCTO #1659]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/bin/bitbake-whatchanged | 334 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 334 insertions(+)
 create mode 100755 bitbake/bin/bitbake-whatchanged

diff --git a/bitbake/bin/bitbake-whatchanged b/bitbake/bin/bitbake-whatchanged
new file mode 100755
index 0000000..6e2f71b
--- /dev/null
+++ b/bitbake/bin/bitbake-whatchanged
@@ -0,0 +1,334 @@
+#!/usr/bin/env python
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+
+# Copyright (c) 2012 Wind River Systems, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+# See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+from __future__ import print_function
+import os
+import sys
+import getopt
+import shutil
+import re
+from optparse import OptionParser
+
+# Need the bb.process.run and bb.siggen
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib'))
+import bb.process
+import bb.siggen
+
+# Match the stamp's filename
+# group(1): PN
+# group(2): PE_PV (may no PE)
+# group(3): PR
+# group(4): TASK
+# group(5): HASH (not used here)
+fre = re.compile("(?P<pn>.*)-(?P<pv>.*)-(?P<pr>r\d+)\.(?P<task>do_\w+)\.sigdata\.(?P<hash>.*)")
+fre_sdk = re.compile("(.*)-nativesdk-(.*)|(.*)-crosssdk-(.*)")
+
+def gen_dict(stamps, sdk):
+    """
+    Generate the dict from the stamps dir.
+    The output dict format is:
+    {FN: {pn: PN, pv: PV, pr: PR, task: TASK, path: PATH}}
+    """
+    # The member of the sub dict (a "path" will be added later)
+    sub_mem = ("pn", "pv", "pr", "task")
+    d = {}
+    for dirpath, _, files in os.walk(stamps):
+        for fn in files:
+            subdict = {}
+            if fre.match(fn):
+                tmp = fre.match(fn)
+                # Nothing especially should be done if sdk, just add it to the dict.
+                if sdk:
+                    for i in sub_mem:
+                        subdict[i] = tmp.group(i)
+                elif not fre_sdk.match(fn):
+                    for i in sub_mem:
+                        subdict[i] = tmp.group(i)
+                if len(subdict) != 0:
+                    # The path will be used by os.stat() and bb.siggen
+                    subdict['path'] = dirpath
+                    d[fn] = subdict
+    return d
+
+# Re-construct the dict
+def recon_dict(dict_in):
+    """
+    The output dict format is:
+    {PN_TASK: {fn: FN, pv: PV, pr: PR, path: PATH}}
+    """
+    dict_out = {}
+    for k in dict_in.keys():
+        subdict = {}
+        # The key
+        pn_task = "%s_%s" % (dict_in.get(k).get('pn'), dict_in.get(k).get('task'))
+        # If more than one stamps are found, use the latest one.
+        if pn_task in dict_out:
+            full_path_pre = "%s/%s" % (dict_in.get(dict_out.get(pn_task).get(fn)).get('path'), k)
+            full_path_cur = "%s/%s" % (dict_in.get(k).get('path'), k)
+            if os.stat(full_path_pre).st_mtime > os.stat(full_path_cur).st_mtime:
+                continue
+        subdict['fn'] = k
+        subdict['pv'] = dict_in.get(k).get('pv')
+        subdict['pr'] = dict_in.get(k).get('pr')
+        subdict['path'] = dict_in.get(k).get('path')
+        dict_out[pn_task] = subdict
+
+    return dict_out
+
+def split_pntask(s):
+    """
+    Split the pn_task in to (pn, task) and return it
+    """
+    tmp = re.match("(.*)_(do_.*)", s)
+    return (tmp.group(1), tmp.group(2))
+
+
+def print_added(d_new = None, d_old = None):
+    """
+    Print the newly added tasks
+    """
+    added = {}
+    for k in d_new.keys():
+        if k not in d_old:
+            # Add the new one to added dict, and remove it from
+            # d_new, so the remaining ones are the changed ones
+            added[k] = d_new.get(k)
+            del(d_new[k])
+
+    if not added:
+        return 0
+
+    # Format the output, the dict format is:
+    # {pn: task1, task2 ...}
+    added_format = {}
+    counter = 0
+    for k in added.keys():
+        pn, task = split_pntask(k)
+        if pn in added_format:
+            # Append the value
+            added_format[pn] = "%s %s" % (added_format.get(pn), task)
+        else:
+            added_format[pn] = task
+        counter += 1
+    print("=== Newly added tasks: (%s tasks)" % counter)
+    for k in added_format.keys():
+        print("  %s: %s" % (k, added_format.get(k)))
+
+    return counter
+
+def print_vrchanged(d_new = None, d_old = None, vr = None):
+    """
+    Print the pv or pr changed tasks.
+    The arg "vr" is "pv" or "pr"
+    """
+    pvchanged = {}
+    counter = 0
+    for k in d_new.keys():
+        if d_new.get(k).get(vr) != d_old.get(k).get(vr):
+            counter += 1
+            pn, task = split_pntask(k)
+            if pn not in pvchanged:
+                # Format the output, we only print pn (no task) since
+                # all the tasks would be changed when pn or pr changed,
+                # the dict format is:
+                # {pn: pv/pr_old -> pv/pr_new}
+                pvchanged[pn] = "%s -> %s" % (d_old.get(k).get(vr), d_new.get(k).get(vr))
+            del(d_new[k])
+
+    if not pvchanged:
+        return 0
+
+    print("\n=== %s changed: (%s tasks)" % (vr.upper(), counter))
+    for k in pvchanged.keys():
+        print("  %s: %s" % (k, pvchanged.get(k)))
+
+    return counter
+
+def print_depchanged(d_new = None, d_old = None, verbose = False):
+    """
+    Print the dependency changes
+    """
+    depchanged = {}
+    counter = 0
+    for k in d_new.keys():
+        counter += 1
+        pn, task = split_pntask(k)
+        if (verbose):
+            full_path_old = "%s/%s" % (d_old.get(k).get("path"), d_old.get(k).get('fn'))
+            full_path_new = "%s/%s" % (d_new.get(k).get("path"), d_new.get(k).get('fn'))
+            # No counter since it is not ready here
+            print("\n=== The verbose changes of %s:" % pn)
+            bb.siggen.compare_sigfiles(full_path_old, full_path_new)
+        else:
+            # Format the output, the format is:
+            # {pn: task1, task2, ...}
+            if pn in depchanged:
+                depchanged[pn] = "%s %s" % (depchanged.get(pn), task)
+            else:
+                depchanged[pn] = task
+
+    if len(depchanged) > 0:
+        print("\n=== DEPENDS changed: (%s tasks)" % counter)
+        for k in depchanged.keys():
+            print("  %s: %s" % (k, depchanged[k]))
+
+    return counter
+
+def main():
+    """
+    Print what will be done between the current and last builds:
+    1) Save the stamps to stamps.old
+    2) Run "bitbake -S recipe" to re-generate the stamps
+    3) Figure out what are newly added and changed, can't figure out
+       what are removed since we can't know the previous stamps
+       clearly, for example, if there are several builds, we can't know
+       exactly which stamps the last build has used.
+    4) Use bb.siggen.compare_sigfiles to diff the old and new stamps
+    """
+
+    parser = OptionParser(
+        version = "1.0",
+        usage = """%prog [options] [package ...]
+print what will be done between the current and last builds, for example:
+
+    $ bitbake core-image-sato
+    # Edit the recipes
+    $ bitbake-whatchanged core-image-sato
+
+The changes will be printed"
+
+Note: The amount of tasks is not accurate when the task is "do_build" since
+      it usually depends on other tasks
+"""
+)
+    parser.add_option("-v", "--verbose", help = "print the verbose changes",
+               action = "store_true", dest = "verbose")
+
+    options, args = parser.parse_args(sys.argv)
+
+    verbose = options.verbose
+
+    if len(args) != 2:
+        parser.error("Incorrect number of arguments")
+    else:
+        recipe = args[1]
+
+    # FIXME
+    # The bitbake -S will always generate the stamp for the nativesdk,
+    # so we should check whether it really builds sdk or not:
+    if recipe.endswith('sdk'):
+        sdk = True
+    else:
+        sdk = False
+
+    # Get TMPDIR
+    print("Figuring out the TMPDIR ...")
+    cmdline = "bitbake -e %s | sed -ne 's/^TMPDIR=\"\(.*\)\"/\\1/p'" % recipe
+    tmpdir, err = bb.process.run(cmdline)
+    if not tmpdir:
+        print("No TMPDIR found for '%s'" % recipe, file=sys.stderr)
+        return 2
+
+    # Get the top stamps dir, TMPDIR/stamps
+    new_stamps = tmpdir.rstrip("\n") + "/stamps"
+    if not os.path.isdir(new_stamps):
+        print("ERROR: no stamps directory found!", file=sys.stderr)
+        return 2
+
+    old_stamps = new_stamps + ".old"
+    if os.path.exists(old_stamps):
+        print ("ERROR: %s is not empty!" % old_stamps, file=sys.stderr)
+        return 2
+
+    try:
+        # Rename the stamps to stamps.old:
+        print("Moving the stamps to stamps.old ...")
+        os.rename(new_stamps, old_stamps)
+
+        # Generate the new stamps dir
+        print("Generating the new stamps ... (need several minutes)")
+        cmdline = "bitbake -S %s" % recipe
+        # FIXME
+        # The "bitbake -S" may fail, not fatal error, the stamps will still
+        # be generated, This might be a bug of "bitbake -S".
+        try:
+            bb.process.run(cmdline)
+        except:
+            pass
+
+        # The dict for the new and old stamps.
+        new_dict = gen_dict(new_stamps, sdk)
+        old_dict = gen_dict(old_stamps, sdk)
+
+        # Remove the same one from both stamps.
+        cnt_unchanged = 0
+        for k in new_dict.keys():
+            if k in old_dict:
+                cnt_unchanged += 1
+                del(new_dict[k])
+                del(old_dict[k])
+
+        # Re-construct the dict to easily find out what is added or changed.
+        # The dict format is:
+        # {PN_TASK: {fn: FN, pv: PV, pr: PR, path:PATH}}
+        new_recon = recon_dict(new_dict)
+        old_recon = recon_dict(old_dict)
+
+        del new_dict
+        del old_dict
+
+        # Figure out what are changed, the new_recon would be changed
+        # by the print_xxx function.
+        # Newly added
+        cnt_added = print_added(new_recon, old_recon)
+
+        # PV (including PE) and PR changed
+        # Let the bb.siggen handle them if verbose
+        cnt_rv = {}
+        if not verbose:
+            for i in ('pv', 'pr'):
+               cnt_rv[i] = print_vrchanged(new_recon, old_recon, i)
+
+        # Depends changed (use bitbake-diffsigs)
+        cnt_dep = print_depchanged(new_recon, old_recon, verbose)
+
+        total_changed = cnt_added + (cnt_rv.get('pv') or 0) + (cnt_rv.get('pr') or 0) + cnt_dep
+
+        print("\n=== Summary: (%s changed, %s unchanged)" % (total_changed, cnt_unchanged))
+        if verbose:
+            print("Newly added: %s, Depends changed: %s\n" % \
+                (cnt_added, cnt_dep))
+        else:
+            print("Newly added: %s, PV changed: %s, PR changed: %s, Depends changed: %s\n" % \
+                (cnt_added, cnt_rv.get('pv') or 0, cnt_rv.get('pr') or 0, cnt_dep))
+    except:
+        print("ERROR occurred!!! Recovering the stamps dir ...", file=sys.stderr)
+        raise
+    finally:
+        # Remove the newly generated stamps dir
+        print("Removing the newly generated stamps ...")
+        shutil.rmtree(new_stamps)
+
+        # Rename the stamps.old back
+        print("Moving the stamps.old back to stamps ...")
+        os.rename(old_stamps, new_stamps)
+
+if __name__ == "__main__":
+    sys.exit(main())
+
-- 
1.7.11.2

Re: [PATCH 0/1] bitbake-whatchanged: print what is about to happen

[Paul Eggleton]

On Tuesday 07 August 2012 23:48:55 Robert Yang wrote:
> The following changes since commit c86d26cb976e665b1516e72153f3f686f62dedf9:
> 
>   subversion: Add missing build dependency on sqlite3 (2012-08-06 16:14:06
> +0100)
> 
> are available in the git repository at:
> 
>   git://git.pokylinux.org/poky-contrib robert/whatchanged
>   http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/whatchanged
> 
> Robert Yang (1):
>   bitbake-whatchanged: print what is about to happen
> 
>  bitbake/bin/bitbake-whatchanged | 334
> ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 334 insertions(+)
>  create mode 100755 bitbake/bin/bitbake-whatchanged

I tried this on top of latest master, unfortunately what I got was the
following:

-------------- snip ----------------
Figuring out the TMPDIR ...
Moving the stamps to stamps.old ...
Generating the new stamps ... (need several minutes)
ERROR occurred!!! Recovering the stamps dir ...
Removing the newly generated stamps ...
Moving the stamps.old back to stamps ...
Traceback (most recent call last):
  File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 333, in <module>
    sys.exit(main())
  File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 291, in main
    old_recon = recon_dict(old_dict)
  File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 82, in recon_dict
    full_path_pre = "%s/%s" % (dict_in.get(dict_out.get(pn_task).get(fn)).get('path'), k)
NameError: global name 'fn' is not defined
-------------- snip ----------------

A couple of other things:

1) We ought to be able to assume that TMPDIR is the same regardless of 
the recipe specified; this avoids having to parse all of the recipes just to
get the value of this variable.

2) I'm a little concerned with the general approach - is there no way of
avoiding having to copy and move around the stamps directory? It seems
a little risky if nothing else.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/1] bitbake-whatchanged: print what is about to happen

[Richard Purdie]

On Tue, 2012-08-07 at 18:12 +0100, Paul Eggleton wrote:
> On Tuesday 07 August 2012 23:48:55 Robert Yang wrote:
> > The following changes since commit c86d26cb976e665b1516e72153f3f686f62dedf9:
> > 
> >   subversion: Add missing build dependency on sqlite3 (2012-08-06 16:14:06
> > +0100)
> > 
> > are available in the git repository at:
> > 
> >   git://git.pokylinux.org/poky-contrib robert/whatchanged
> >   http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/whatchanged
> > 
> > Robert Yang (1):
> >   bitbake-whatchanged: print what is about to happen
> > 
> >  bitbake/bin/bitbake-whatchanged | 334
> > ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 334 insertions(+)
> >  create mode 100755 bitbake/bin/bitbake-whatchanged
> 
> I tried this on top of latest master, unfortunately what I got was the
> following:
> 
> -------------- snip ----------------
> Figuring out the TMPDIR ...
> Moving the stamps to stamps.old ...
> Generating the new stamps ... (need several minutes)
> ERROR occurred!!! Recovering the stamps dir ...
> Removing the newly generated stamps ...
> Moving the stamps.old back to stamps ...
> Traceback (most recent call last):
>   File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 333, in <module>
>     sys.exit(main())
>   File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 291, in main
>     old_recon = recon_dict(old_dict)
>   File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 82, in recon_dict
>     full_path_pre = "%s/%s" % (dict_in.get(dict_out.get(pn_task).get(fn)).get('path'), k)
> NameError: global name 'fn' is not defined
> -------------- snip ----------------
> 
> A couple of other things:
> 
> 1) We ought to be able to assume that TMPDIR is the same regardless of 
> the recipe specified; this avoids having to parse all of the recipes just to
> get the value of this variable.
> 
> 2) I'm a little concerned with the general approach - is there no way of
> avoiding having to copy and move around the stamps directory? It seems
> a little risky if nothing else.

I think adding a parameter to -S would be a good move for this, its
something people likely want in conjunction with that.

Cheers,

Richard

Re: [PATCH 0/1] bitbake-whatchanged: print what is about to happen

[Robert Yang]

On 08/08/2012 05:01 AM, Richard Purdie wrote:
> On Tue, 2012-08-07 at 18:12 +0100, Paul Eggleton wrote:
>> On Tuesday 07 August 2012 23:48:55 Robert Yang wrote:
>>> The following changes since commit c86d26cb976e665b1516e72153f3f686f62dedf9:
>>>
>>>    subversion: Add missing build dependency on sqlite3 (2012-08-06 16:14:06
>>> +0100)
>>>
>>> are available in the git repository at:
>>>
>>>    git://git.pokylinux.org/poky-contrib robert/whatchanged
>>>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/whatchanged
>>>
>>> Robert Yang (1):
>>>    bitbake-whatchanged: print what is about to happen
>>>
>>>   bitbake/bin/bitbake-whatchanged | 334
>>> ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 334 insertions(+)
>>>   create mode 100755 bitbake/bin/bitbake-whatchanged
>>
>> I tried this on top of latest master, unfortunately what I got was the
>> following:
>>
>> -------------- snip ----------------
>> Figuring out the TMPDIR ...
>> Moving the stamps to stamps.old ...
>> Generating the new stamps ... (need several minutes)
>> ERROR occurred!!! Recovering the stamps dir ...
>> Removing the newly generated stamps ...
>> Moving the stamps.old back to stamps ...
>> Traceback (most recent call last):
>>    File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 333, in <module>
>>      sys.exit(main())
>>    File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 291, in main
>>      old_recon = recon_dict(old_dict)
>>    File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 82, in recon_dict
>>      full_path_pre = "%s/%s" % (dict_in.get(dict_out.get(pn_task).get(fn)).get('path'), k)
>> NameError: global name 'fn' is not defined

Thanks Paul, I've fixed this one, my testing didn't get here before, it should
be the string 'fn', not the variable fn.

>> -------------- snip ----------------
>>
>> A couple of other things:
>>
>> 1) We ought to be able to assume that TMPDIR is the same regardless of
>> the recipe specified; this avoids having to parse all of the recipes just to
>> get the value of this variable.
>>

I'm sorry, I don't understand what did you mean here. it seems that what
I did is the same as you said: Use "bitbake -e" to figure out the TMPDIR at
the beginning, then use it elsewhere.

>> 2) I'm a little concerned with the general approach - is there no way of
>> avoiding having to copy and move around the stamps directory? It seems
>> a little risky if nothing else.
>
> I think adding a parameter to -S would be a good move for this, its
> something people likely want in conjunction with that.
>

Yes, add a parameter to "bitbake -S recipe" would be the correct way, but
as far as I know, the "-S" is a bool option currently, it doesn't accept
an argument, I think that we have the following 2 solutions:

1) Modify the "-S" to accept an argument, but this may break the the usage
    of the "bitbake -S", the currently usage is:

    bitbake -S <recipe>

    We may change it to:

    bitbake -S <tmpdir>(or stampsdir) <recipe>

    But it seems that it's not easy differentiate the argument behind "-S".

2) Use "TMPDIR(or STAMP)=<path> bitbake -S recipe", but we don't support it
    currently, but we can add an os.getenv("TMPDIR") in bitbake to achieve it,
    the BB_TMPDIR or BB_STAMP would be better, but I'm not sure whether it will
    cause other problems.

I'd like to send a patch for 2) if you are OK with it.

// Robert


> Cheers,
>
> Richard
>
>
>
>
>
>

Re: [PATCH 0/1] bitbake-whatchanged: print what is about to happen

[Paul Eggleton]

On Wednesday 08 August 2012 11:40:14 Robert Yang wrote:
> On 08/08/2012 05:01 AM, Richard Purdie wrote:
> > On Tue, 2012-08-07 at 18:12 +0100, Paul Eggleton wrote:
> >> A couple of other things:
> >> 
> >> 1) We ought to be able to assume that TMPDIR is the same regardless of
> >> the recipe specified; this avoids having to parse all of the recipes just
> >> to get the value of this variable.
> 
> I'm sorry, I don't understand what did you mean here. it seems that what
> I did is the same as you said: Use "bitbake -e" to figure out the TMPDIR at
> the beginning, then use it elsewhere.

Sorry, I wasn't being very clear. If you specify the recipe with bitbake -e, 
bitbake has to go through a parse of the recipes (retrieving from the cache if 
available of course, but even that still takes a few seconds). I'm suggesting 
you don't specify the recipe as TMPDIR shouldn't be recipe-specific, and save 
quite a bit of time.

> >> 2) I'm a little concerned with the general approach - is there no way of
> >> avoiding having to copy and move around the stamps directory? It seems
> >> a little risky if nothing else.
> > 
> > I think adding a parameter to -S would be a good move for this, its
> > something people likely want in conjunction with that.
> 
> Yes, add a parameter to "bitbake -S recipe" would be the correct way, but
> as far as I know, the "-S" is a bool option currently, it doesn't accept
> an argument, I think that we have the following 2 solutions:
> 
> 1) Modify the "-S" to accept an argument, but this may break the the usage
>     of the "bitbake -S", the currently usage is:
> 
>     bitbake -S <recipe>
> 
>     We may change it to:
> 
>     bitbake -S <tmpdir>(or stampsdir) <recipe>
> 
>     But it seems that it's not easy differentiate the argument behind "-S".

I have to admit I'm not sure what -S is currently being used for; so it's hard 
for me to comment on what might get broken if we change this. I suspect it's 
not being used very much at all though.
 
> 2) Use "TMPDIR(or STAMP)=<path> bitbake -S recipe", but we don't support it
>     currently, but we can add an os.getenv("TMPDIR") in bitbake to achieve
> it, the BB_TMPDIR or BB_STAMP would be better, but I'm not sure whether it
> will cause other problems.
> 
> I'd like to send a patch for 2) if you are OK with it.

We already have a mechanism to allow through variables from the environment - 
BB_ENV_WHITELIST / BB_ENV_EXTRAWHITE, and we should make use of that in 
preference to os.getenv(). I think we would not want this change in stamp 
writing behaviour to take effect unless -S is being used though, which suggests 
it ought to be a new variable that is not checked unless -S has been specified.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/1] bitbake-whatchanged: print what is about to happen

[Robert Yang]

On 08/08/2012 11:40 AM, Robert Yang wrote:
>
> On 08/08/2012 05:01 AM, Richard Purdie wrote:
>> On Tue, 2012-08-07 at 18:12 +0100, Paul Eggleton wrote:
>>> On Tuesday 07 August 2012 23:48:55 Robert Yang wrote:
>>>> The following changes since commit c86d26cb976e665b1516e72153f3f686f62dedf9:
>>>>
>>>>    subversion: Add missing build dependency on sqlite3 (2012-08-06 16:14:06
>>>> +0100)
>>>>
>>>> are available in the git repository at:
>>>>
>>>>    git://git.pokylinux.org/poky-contrib robert/whatchanged
>>>>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/whatchanged
>>>>
>>>> Robert Yang (1):
>>>>    bitbake-whatchanged: print what is about to happen
>>>>
>>>>   bitbake/bin/bitbake-whatchanged | 334
>>>> ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 334 insertions(+)
>>>>   create mode 100755 bitbake/bin/bitbake-whatchanged
>>>
>>> I tried this on top of latest master, unfortunately what I got was the
>>> following:
>>>
>>> -------------- snip ----------------
>>> Figuring out the TMPDIR ...
>>> Moving the stamps to stamps.old ...
>>> Generating the new stamps ... (need several minutes)
>>> ERROR occurred!!! Recovering the stamps dir ...
>>> Removing the newly generated stamps ...
>>> Moving the stamps.old back to stamps ...
>>> Traceback (most recent call last):
>>>    File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 333, in
>>> <module>
>>>      sys.exit(main())
>>>    File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 291, in
>>> main
>>>      old_recon = recon_dict(old_dict)
>>>    File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 82, in
>>> recon_dict
>>>      full_path_pre = "%s/%s" %
>>> (dict_in.get(dict_out.get(pn_task).get(fn)).get('path'), k)
>>> NameError: global name 'fn' is not defined
>
> Thanks Paul, I've fixed this one, my testing didn't get here before, it should
> be the string 'fn', not the variable fn.
>
>>> -------------- snip ----------------
>>>
>>> A couple of other things:
>>>
>>> 1) We ought to be able to assume that TMPDIR is the same regardless of
>>> the recipe specified; this avoids having to parse all of the recipes just to
>>> get the value of this variable.
>>>
>
> I'm sorry, I don't understand what did you mean here. it seems that what
> I did is the same as you said: Use "bitbake -e" to figure out the TMPDIR at
> the beginning, then use it elsewhere.
>
>>> 2) I'm a little concerned with the general approach - is there no way of
>>> avoiding having to copy and move around the stamps directory? It seems
>>> a little risky if nothing else.
>>
>> I think adding a parameter to -S would be a good move for this, its
>> something people likely want in conjunction with that.
>>

I just notices that added the variable STAMP to BB_ENV_EXTRAWHITE would make
it work, thanks, I missed this before, I will send an updated sooner.

// Robert

>
> Yes, add a parameter to "bitbake -S recipe" would be the correct way, but
> as far as I know, the "-S" is a bool option currently, it doesn't accept
> an argument, I think that we have the following 2 solutions:
>
> 1) Modify the "-S" to accept an argument, but this may break the the usage
>     of the "bitbake -S", the currently usage is:
>
>     bitbake -S <recipe>
>
>     We may change it to:
>
>     bitbake -S <tmpdir>(or stampsdir) <recipe>
>
>     But it seems that it's not easy differentiate the argument behind "-S".
>
> 2) Use "TMPDIR(or STAMP)=<path> bitbake -S recipe", but we don't support it
>     currently, but we can add an os.getenv("TMPDIR") in bitbake to achieve it,
>     the BB_TMPDIR or BB_STAMP would be better, but I'm not sure whether it will
>     cause other problems.
>
> I'd like to send a patch for 2) if you are OK with it.
>
> // Robert
>
>
>> Cheers,
>>
>> Richard
>>
>>
>>
>>
>>
>>
>
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
>
>

Re: [PATCH 0/1] bitbake-whatchanged: print what is about to happen

[Richard Purdie]

On Wed, 2012-08-08 at 11:40 +0800, Robert Yang wrote:
> >> 2) I'm a little concerned with the general approach - is there no way of
> >> avoiding having to copy and move around the stamps directory? It seems
> >> a little risky if nothing else.
> >
> > I think adding a parameter to -S would be a good move for this, its
> > something people likely want in conjunction with that.
> >
> 
> Yes, add a parameter to "bitbake -S recipe" would be the correct way, but
> as far as I know, the "-S" is a bool option currently, it doesn't accept
> an argument, I think that we have the following 2 solutions:
> 
> 1) Modify the "-S" to accept an argument, but this may break the the usage
>     of the "bitbake -S", the currently usage is:
> 
>     bitbake -S <recipe>
> 
>     We may change it to:
> 
>     bitbake -S <tmpdir>(or stampsdir) <recipe>
> 
>     But it seems that it's not easy differentiate the argument behind "-S".

Hmm, we probably could change this option as long as we update the
manuals too. Ideally I would like some way to say "use the default
stamps directory" without having to put a full path in.

> 2) Use "TMPDIR(or STAMP)=<path> bitbake -S recipe", but we don't support it
>     currently, but we can add an os.getenv("TMPDIR") in bitbake to achieve it,
>     the BB_TMPDIR or BB_STAMP would be better, but I'm not sure whether it will
>     cause other problems.
> 
> I'd like to send a patch for 2) if you are OK with it.

Please don't use TMPDIR, bitbake has no knowledge of that variable, nor
should it have and it will change the cache directory. I'd suggest using
STAMP, that is what the variable is designed for. We just need to allow
it from the environment which Paul mentions and this is what I
originally proposed.

I'm leaning towards 2) and STAMP with BB_ENV_WHITELIST (or whatever the
variable is called) updated accordingly.

Cheers,

Richard

Re: [PATCH 0/1] bitbake-whatchanged: print what is about to happen

[Robert Yang]

On 08/08/2012 05:12 PM, Paul Eggleton wrote:
> On Wednesday 08 August 2012 11:40:14 Robert Yang wrote:
>> On 08/08/2012 05:01 AM, Richard Purdie wrote:
>>> On Tue, 2012-08-07 at 18:12 +0100, Paul Eggleton wrote:
>>>> A couple of other things:
>>>>
>>>> 1) We ought to be able to assume that TMPDIR is the same regardless of
>>>> the recipe specified; this avoids having to parse all of the recipes just
>>>> to get the value of this variable.
>>
>> I'm sorry, I don't understand what did you mean here. it seems that what
>> I did is the same as you said: Use "bitbake -e" to figure out the TMPDIR at
>> the beginning, then use it elsewhere.
>
> Sorry, I wasn't being very clear. If you specify the recipe with bitbake -e,
> bitbake has to go through a parse of the recipes (retrieving from the cache if
> available of course, but even that still takes a few seconds). I'm suggesting
> you don't specify the recipe as TMPDIR shouldn't be recipe-specific, and save
> quite a bit of time.
>
>>>> 2) I'm a little concerned with the general approach - is there no way of
>>>> avoiding having to copy and move around the stamps directory? It seems
>>>> a little risky if nothing else.
>>>
>>> I think adding a parameter to -S would be a good move for this, its
>>> something people likely want in conjunction with that.
>>
>> Yes, add a parameter to "bitbake -S recipe" would be the correct way, but
>> as far as I know, the "-S" is a bool option currently, it doesn't accept
>> an argument, I think that we have the following 2 solutions:
>>

Got it, thanks.

>> 1) Modify the "-S" to accept an argument, but this may break the the usage
>>      of the "bitbake -S", the currently usage is:
>>
>>      bitbake -S <recipe>
>>
>>      We may change it to:
>>
>>      bitbake -S <tmpdir>(or stampsdir) <recipe>
>>
>>      But it seems that it's not easy differentiate the argument behind "-S".
>
> I have to admit I'm not sure what -S is currently being used for; so it's hard
> for me to comment on what might get broken if we change this. I suspect it's
> not being used very much at all though.
>
>> 2) Use "TMPDIR(or STAMP)=<path> bitbake -S recipe", but we don't support it
>>      currently, but we can add an os.getenv("TMPDIR") in bitbake to achieve
>> it, the BB_TMPDIR or BB_STAMP would be better, but I'm not sure whether it
>> will cause other problems.
>>
>> I'd like to send a patch for 2) if you are OK with it.
>
> We already have a mechanism to allow through variables from the environment -
> BB_ENV_WHITELIST / BB_ENV_EXTRAWHITE, and we should make use of that in
> preference to os.getenv(). I think we would not want this change in stamp
> writing behaviour to take effect unless -S is being used though, which suggests
> it ought to be a new variable that is not checked unless -S has been specified.
>

Thank you very much, I will add a BB_DUMP_STAMP.

// Robert

> Cheers,
> Paul
>

Re: [PATCH 0/1] bitbake-whatchanged: print what is about to happen

[Robert Yang]

On 08/08/2012 05:21 PM, Richard Purdie wrote:
> On Wed, 2012-08-08 at 11:40 +0800, Robert Yang wrote:
>>>> 2) I'm a little concerned with the general approach - is there no way of
>>>> avoiding having to copy and move around the stamps directory? It seems
>>>> a little risky if nothing else.
>>>
>>> I think adding a parameter to -S would be a good move for this, its
>>> something people likely want in conjunction with that.
>>>
>>
>> Yes, add a parameter to "bitbake -S recipe" would be the correct way, but
>> as far as I know, the "-S" is a bool option currently, it doesn't accept
>> an argument, I think that we have the following 2 solutions:
>>
>> 1) Modify the "-S" to accept an argument, but this may break the the usage
>>      of the "bitbake -S", the currently usage is:
>>
>>      bitbake -S <recipe>
>>
>>      We may change it to:
>>
>>      bitbake -S <tmpdir>(or stampsdir) <recipe>
>>
>>      But it seems that it's not easy differentiate the argument behind "-S".
>
> Hmm, we probably could change this option as long as we update the
> manuals too. Ideally I would like some way to say "use the default
> stamps directory" without having to put a full path in.
>
>> 2) Use "TMPDIR(or STAMP)=<path> bitbake -S recipe", but we don't support it
>>      currently, but we can add an os.getenv("TMPDIR") in bitbake to achieve it,
>>      the BB_TMPDIR or BB_STAMP would be better, but I'm not sure whether it will
>>      cause other problems.
>>
>> I'd like to send a patch for 2) if you are OK with it.
>
> Please don't use TMPDIR, bitbake has no knowledge of that variable, nor
> should it have and it will change the cache directory. I'd suggest using
> STAMP, that is what the variable is designed for. We just need to allow
> it from the environment which Paul mentions and this is what I
> originally proposed.
>
> I'm leaning towards 2) and STAMP with BB_ENV_WHITELIST (or whatever the
> variable is called) updated accordingly.
>

Thank you very much, the STAMP is:

STAMP = "${TMPDIR}/stamps/${MULTIMACH_TARGET_SYS}/${PF}"

How about add a variable STAMP_DIR, it would be:

STAMP_DIR ?= "${TMPDIR}/stamps/"
STAMP = "${STAMP_DIR}/${MULTIMACH_TARGET_SYS}/${PF}"

Then add STAMP_DIR to BB_ENV_WHITELIST and we can run bitbake -S like:
STAMP_DIR=/what/we/want bitbake -S

Thanks

// Robert


> Cheers,
>
> Richard
>
>
>