[PATCH v2 0/4] [RFC] Shell func backtracing

[PATCH v2 0/4] [RFC] Shell func backtracing

[Chris Laplante]

This patch series replaces the one I sent Friday. It implements two
backtraces when a Bash shell function fails:

1) A backtrace within the .run file that BB generated.
2) A computed backtrace of the actual metadata locations (e.g. bb or
    bbclass) where each shell function is defined.

At one point I had both backtraces implemented entirely in the shell
traps. I moved (2) to build.py because it is easier to implement in
Python. One nice thing about leaving it in the shell trap would be that
it would still work even if a user was running a .run script directly,
but it seems like not many people do that. Still, if it's something
we're interested in, know that it's possible.

P.S. Sorry for all the v2, v3's I've been sending.

Chris Laplante (4):
  data: emit filename/lineno information for shell functions
  build: print a backtrace when a Bash shell function fails
  build: print a backtrace with the original metadata locations of Bash
    shell funcs
  build: add back the Bash EXIT trap to handle cases that ERR doesn't

 lib/bb/build.py   | 116 +++++++++++++++++++++++++++++++++++++++++-----
 lib/bb/data.py    |   6 +++
 lib/bb/process.py |   3 +-
 3 files changed, 113 insertions(+), 12 deletions(-)

--
2.17.1

[PATCH v2 1/4] data: emit filename/lineno information for shell functions

[Chris Laplante]

Make it easier for users to debug shell task failure by including
some breadcrumbs in the emitted .run file that (hopefully) points
to the .bb/.bbclass file where the shell function was defined.

Unfortunately this won't work with functions with _append
or _prepends, since BitBake wipes the filename/lineno information.
This shouldn't be too hard to fix; for now, you'll just see
comments like this for such functions:

[YOCTO #7877]

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
---
 lib/bb/data.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/lib/bb/data.py b/lib/bb/data.py
index b0683c51..97022853 100644
--- a/lib/bb/data.py
+++ b/lib/bb/data.py
@@ -161,6 +161,12 @@ def emit_var(var, o=sys.__stdout__, d = init(), all=False):
         return True
 
     if func:
+        # Write a comment indicating where the shell function came from (line number and filename) to make it easier
+        # for the user to diagnose task failures. This comment is also used by build.py to determine the metadata
+        # location of shell functions.
+        o.write("# line: {0}, file: {1}\n".format(
+            d.getVarFlag(var, "lineno", False),
+            d.getVarFlag(var, "filename", False)))
         # NOTE: should probably check for unbalanced {} within the var
         val = val.rstrip('\n')
         o.write("%s() {\n%s\n}\n" % (varExpanded, val))
-- 
2.17.1

[PATCH v2 2/4] build: print a backtrace when a Bash shell function fails

[Chris Laplante]

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
---
 lib/bb/build.py | 36 +++++++++++++++++++++++++-----------
 1 file changed, 25 insertions(+), 11 deletions(-)

diff --git a/lib/bb/build.py b/lib/bb/build.py
index 977b02fc..ad445aa9 100644
--- a/lib/bb/build.py
+++ b/lib/bb/build.py
@@ -304,19 +304,33 @@ def exec_func_python(func, d, runfile, cwd=None):
 def shell_trap_code():
     return '''#!/bin/sh\n
 # Emit a useful diagnostic if something fails:
-bb_exit_handler() {
+bb_sh_exit_handler() {
     ret=$?
-    case $ret in
-    0)  ;;
-    *)  case $BASH_VERSION in
-        "") echo "WARNING: exit code $ret from a shell command.";;
-        *)  echo "WARNING: ${BASH_SOURCE[0]}:${BASH_LINENO[0]} exit $ret from '$BASH_COMMAND'";;
-        esac
-        exit $ret
-    esac
+    if [ "$ret" != 0 ]; then
+        echo "WARNING: exit code $ret from a shell command."
+    fi
+    exit $ret
 }
-trap 'bb_exit_handler' 0
-set -e
+
+bb_bash_err_handler() {
+    ret=$?
+    echo "WARNING: ${BASH_SOURCE[0]}:${BASH_LINENO[0]} exit $ret from '$BASH_COMMAND'"
+
+    echo "WARNING: Backtrace (BB generated script): "
+    for ((i=1; i<${#FUNCNAME[@]}; i++)); do
+        echo "\t#$((i-1)): ${FUNCNAME[$i]}, ${BASH_SOURCE[$((i-1))]}, line ${BASH_LINENO[$((i-1))]}"
+    done
+
+    exit $ret
+}
+
+case $BASH_VERSION in
+"") trap 'bb_sh_exit_handler' 0
+    set -e
+    ;;
+*)  trap 'bb_bash_err_handler' ERR
+    set -eE
+esac
 '''
 
 def create_progress_handler(func, progress, logfile, d):
-- 
2.17.1

[PATCH v2 3/4] build: print a backtrace with the original metadata locations of Bash shell funcs

[Chris Laplante]

Leverage the comments that emit_var writes and the backtrace that
the shell func writes to generate an additional metadata-relative
backtrace. This will help the user troubleshoot shell funcs much
more easily.

E.g.:

| /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.87873: line 168: no_exist: command not found
| WARNING: /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.87873:168 exit 127 from 'no_exist'
| WARNING: Backtrace (BB generated script):
| 	#0: myclass_do_something, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.87873, line 168
| 	#1: do_something, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.87873, line 157
| 	#2: actually_fail, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.87873, line 150
| 	#3: my_compile_extra, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.87873, line 152
| 	#4: do_compile, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.87873, line 138
| 	#5: main, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.87873, line 181
|
| Backtrace (metadata-relative locations):
| 	#0: myclass_do_something, /home/laplante/repos/oe-core/meta/classes/myclass.bbclass, line 2
| 	#1: do_something, autogenerated, line 2
| 	#2: actually_fail, /home/laplante/repos/oe-core/meta/recipes-extended/libsolv/libsolv_0.7.14.bb, line 36
| 	#3: my_compile_extra, /home/laplante/repos/oe-core/meta/recipes-extended/libsolv/libsolv_0.7.14.bb, line 38
| 	#4: do_compile, autogenerated, line 3

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
---
 lib/bb/build.py   | 58 +++++++++++++++++++++++++++++++++++++++++++++++
 lib/bb/process.py |  3 ++-
 2 files changed, 60 insertions(+), 1 deletion(-)

diff --git a/lib/bb/build.py b/lib/bb/build.py
index ad445aa9..834078fd 100644
--- a/lib/bb/build.py
+++ b/lib/bb/build.py
@@ -16,7 +16,9 @@ import os
 import sys
 import logging
 import glob
+import itertools
 import time
+import re
 import stat
 import bb
 import bb.msg
@@ -470,6 +472,62 @@ exit $ret
             bb.debug(2, "Executing shell function %s" % func)
             with open(os.devnull, 'r+') as stdin, logfile:
                 bb.process.run(cmd, shell=False, stdin=stdin, log=logfile, extrafiles=[(fifo,readfifo)])
+        except bb.process.ExecutionError as exe:
+            # Find the backtrace that the shell trap generated
+            backtrace_marker_regex = re.compile(r"WARNING: Backtrace \(BB generated script\)")
+            stdout_lines = (exe.stdout or "").split("\n")
+            backtrace_start_line = None
+            for i, line in enumerate(reversed(stdout_lines)):
+                if backtrace_marker_regex.search(line):
+                    backtrace_start_line = len(stdout_lines) - i
+                    break
+
+            # Read the backtrace frames, starting at the location we just found
+            backtrace_entry_regex = re.compile(r"#(?P<frameno>\d+): (?P<funcname>[^\s]+), (?P<file>.+?), line ("
+                                               r"?P<lineno>\d+)")
+            backtrace_frames = []
+            if backtrace_start_line:
+                for line in itertools.islice(stdout_lines, backtrace_start_line, None):
+                    match = backtrace_entry_regex.search(line)
+                    if match:
+                        backtrace_frames.append(match.groupdict())
+
+            with open(runfile, "r") as script:
+                script_lines = [line.rstrip() for line in script.readlines()]
+
+            # For each backtrace frame, search backwards in the script (from the line number called out by the frame),
+            # to find the comment that emit_vars injected when it wrote the script. This will give us the metadata
+            # filename (e.g. .bb or .bbclass) and line number where the shell function was originally defined.
+            script_metadata_comment_regex = re.compile(r"# line: (?P<lineno>\d+), file: (?P<file>.+)")
+            better_frames = []
+            # Skip the very last frame since it's just the call to the shell task in the body of the script
+            for frame in backtrace_frames[:-1]:
+                # Check whether the frame corresponds to a function defined in the script vs external script.
+                if os.path.samefile(frame["file"], runfile):
+                    # Search backwards from the frame lineno to locate the comment that BB injected
+                    i = int(frame["lineno"]) - 1
+                    while i >= 0:
+                        match = script_metadata_comment_regex.match(script_lines[i])
+                        if match:
+                            # Calculate the relative line in the function itself
+                            relative_line_in_function = int(frame["lineno"]) - i - 2
+                            # Calculate line in the function as declared in the metadata
+                            metadata_function_line = relative_line_in_function + int(match["lineno"])
+                            better_frames.append("#{frameno}: {funcname}, {file}, line {lineno}".format(
+                                frameno=frame["frameno"],
+                                funcname=frame["funcname"],
+                                file=match["file"],
+                                lineno=metadata_function_line
+                            ))
+                            break
+                        i -= 1
+                else:
+                    better_frames.append("#{frameno}: {funcname}, {file}, line {lineno}".format(**frame))
+
+            if better_frames:
+                better_frames = ("\t{0}".format(frame) for frame in better_frames)
+                exe.extra_message = "\nBacktrace (metadata-relative locations):\n{0}".format("\n".join(better_frames))
+            raise
         finally:
             os.unlink(fifopath)
 
diff --git a/lib/bb/process.py b/lib/bb/process.py
index 2dc472a8..f36c929d 100644
--- a/lib/bb/process.py
+++ b/lib/bb/process.py
@@ -41,6 +41,7 @@ class ExecutionError(CmdError):
         self.exitcode = exitcode
         self.stdout = stdout
         self.stderr = stderr
+        self.extra_message = None
 
     def __str__(self):
         message = ""
@@ -51,7 +52,7 @@ class ExecutionError(CmdError):
         if message:
             message = ":\n" + message
         return (CmdError.__str__(self) +
-                " with exit code %s" % self.exitcode + message)
+                " with exit code %s" % self.exitcode + message + (self.extra_message or ""))
 
 class Popen(subprocess.Popen):
     defaults = {
-- 
2.17.1

[PATCH v2 4/4] build: add back the Bash EXIT trap to handle cases that ERR doesn't

[Chris Laplante]

ERR unfortunately doesn't trigger for e.g. 'exit 1'. Unfortunately
the backtrace we generate in the EXIT trap won't be able to get the
correct line number for the first frame.

See http://gnu-bash.2382.n7.nabble.com/trap-echo-quot-trap-exit-on-LINENO-quot-EXIT-gt-wrong-linenumber-td3666.html

Therefore the metadata-relative backtrace will just not have the
first frame. Example:

| WARNING: /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.85057:1 exit 1 from 'exit 1'
| WARNING: Backtrace (BB generated script):
| 	#1: myclass_do_something, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.85057, line ?
| 	#2: do_something, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.85057, line 156
| 	#3: actually_fail, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.85057, line 144
| 	#4: my_compile_extra, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.85057, line 146
| 	#5: do_compile, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.85057, line 137
| 	#6: main, /home/laplante/repos/oe-core/build/tmp-glibc/work/core2-64-oe-linux/libsolv/0.7.14-r0/temp/run.do_compile.85057, line 180
|
| Backtrace (metadata-relative locations):
| 	#2: do_something, autogenerated, line 2
| 	#3: actually_fail, /home/laplante/repos/oe-core/meta/recipes-extended/libsolv/libsolv_0.7.14.bb, line 36
| 	#4: my_compile_extra, /home/laplante/repos/oe-core/meta/recipes-extended/libsolv/libsolv_0.7.14.bb, line 38
| 	#5: do_compile, autogenerated, line 3

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
---
 lib/bb/build.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/lib/bb/build.py b/lib/bb/build.py
index 834078fd..a6b7c5de 100644
--- a/lib/bb/build.py
+++ b/lib/bb/build.py
@@ -314,6 +314,26 @@ bb_sh_exit_handler() {
     exit $ret
 }
 
+bb_bash_exit_handler() {
+    ret=$?
+    if [ "$ret" != 0 ]; then
+        echo "WARNING: ${BASH_SOURCE[0]}:${BASH_LINENO[0]} exit $ret from '$BASH_COMMAND'"
+
+        echo "WARNING: Backtrace (BB generated script): "
+        for ((i=1; i<${#FUNCNAME[@]}; i++)); do
+            if [ $i -eq 1 ]; then
+                # TODO: maybe manually track LINENO with a DEBUG trap?
+                # LINENO / BASH_LINENO for the first frame in an EXIT trap is wrong :/
+                # http://gnu-bash.2382.n7.nabble.com/trap-echo-quot-trap-exit-on-LINENO-quot-EXIT-gt-wrong-linenumber-td3666.html
+                echo "\t#$((i)): ${FUNCNAME[$i]}, ${BASH_SOURCE[$((i-1))]}, line ?"
+            else
+                echo "\t#$((i)): ${FUNCNAME[$i]}, ${BASH_SOURCE[$((i-1))]}, line ${BASH_LINENO[$((i-1))]}"
+            fi
+        done
+    fi
+    exit $ret
+}
+
 bb_bash_err_handler() {
     ret=$?
     echo "WARNING: ${BASH_SOURCE[0]}:${BASH_LINENO[0]} exit $ret from '$BASH_COMMAND'"
@@ -323,6 +343,7 @@ bb_bash_err_handler() {
         echo "\t#$((i-1)): ${FUNCNAME[$i]}, ${BASH_SOURCE[$((i-1))]}, line ${BASH_LINENO[$((i-1))]}"
     done
 
+    trap "" EXIT
     exit $ret
 }
 
@@ -331,6 +352,7 @@ case $BASH_VERSION in
     set -e
     ;;
 *)  trap 'bb_bash_err_handler' ERR
+    trap 'bb_bash_exit_handler' 0
     set -eE
 esac
 '''
-- 
2.17.1

Re: [bitbake-devel] [PATCH v2 0/4] [RFC] Shell func backtracing

[Richard Purdie]

On Sun, 2020-08-02 at 10:35 -0400, Chris Laplante via
lists.openembedded.org wrote:
> This patch series replaces the one I sent Friday. It implements two
> backtraces when a Bash shell function fails:
> 
> 1) A backtrace within the .run file that BB generated.
> 2) A computed backtrace of the actual metadata locations (e.g. bb or
>     bbclass) where each shell function is defined.
> 
> At one point I had both backtraces implemented entirely in the shell
> traps. I moved (2) to build.py because it is easier to implement in
> Python. One nice thing about leaving it in the shell trap would be
> that
> it would still work even if a user was running a .run script
> directly,
> but it seems like not many people do that. Still, if it's something
> we're interested in, know that it's possible.

This does look really interesting/useful, thanks. I have a long
standing bug to add more information like this to functions!

The one piece I can't quite see there is whether things like appends to
a shell function are handled?

I do sometimes run the scripts by hand myself. Another long standing
idea I keep meaning to sort out is adding the correct pseudo
environment to those functions, even as comments the user could enable
for debugging.

Did you finish the other shell trap code? I think we're probably ok
without that but if you had it working it might be good to put in the
archives?

I'm a bit behind on patch review but will get these queued/tested.
Thanks again!

Cheers,

Richard

Re: [bitbake-devel] [PATCH v2 0/4] [RFC] Shell func backtracing

[Chris Laplante]

Hi Richard,

> This does look really interesting/useful, thanks. I have a long standing bug to
> add more information like this to functions!
> 
> The one piece I can't quite see there is whether things like appends to a shell
> function are handled?

Unfortunately not, since the parser resets the lineno/func to "1, autogenerated". 
I'll have to teach the parser to maintain the original location information, probably
as some sort of array since you can of course have multiple appends.

Even then, it won't be able to trace where people do d.appendVar("do_compile", ...
I have some half-baked ideas on how to handle that but it would probably require 
changes to data_smart. 

> 
> I do sometimes run the scripts by hand myself. Another long standing idea I
> keep meaning to sort out is adding the correct pseudo environment to those
> functions, even as comments the user could enable for debugging.
> 
> Did you finish the other shell trap code? I think we're probably ok without that
> but if you had it working it might be good to put in the archives?

I nearly finished it. I'll submit a v3 with it. My original design for the shell trap 
was that it would locate the comment by determining where the function is 
defined (by using "shopt -s extdebug" and "define -F") and then just grab the
comment by subtracting one from the line number. But it doesn’t quite work 
for nested shell functions. So I think I'll just have to do what I do in build.py,
scan backwards from the frame line number to locate the comment.

> 
> I'm a bit behind on patch review but will get these queued/tested.
> Thanks again!
> 
> Cheers,
> 
> Richard

Re: [bitbake-devel] [PATCH v2 0/4] [RFC] Shell func backtracing

[Richard Purdie]

On Fri, 2020-08-07 at 12:31 +0000, chris.laplante@agilent.com wrote:
> Hi Richard,
> 
> > This does look really interesting/useful, thanks. I have a long
> > standing bug to
> > add more information like this to functions!
> > 
> > The one piece I can't quite see there is whether things like
> > appends to a shell
> > function are handled?
> 
> Unfortunately not, since the parser resets the lineno/func to "1,
> autogenerated". 
> I'll have to teach the parser to maintain the original location
> information, probably
> as some sort of array since you can of course have multiple appends.
> 
> Even then, it won't be able to trace where people do
> d.appendVar("do_compile", ...
> I have some half-baked ideas on how to handle that but it would
> probably require 
> changes to data_smart. 

Have a look at what the d.varhistory.emit(var, oval, val, o, d) call
does in emit() in data.py. The hard part is we have to have variable
tracking turned on for that to work. You can see the output with
"bitbake -e". We don't turn on variable tracking by default as its a
heavy load for general parsing and not needed. You may be able to
enable that in worker context though.

> > I do sometimes run the scripts by hand myself. Another long
> > standing idea I
> > keep meaning to sort out is adding the correct pseudo environment
> > to those
> > functions, even as comments the user could enable for debugging.
> > 
> > Did you finish the other shell trap code? I think we're probably ok
> > without that
> > but if you had it working it might be good to put in the archives?
> 
> I nearly finished it. I'll submit a v3 with it. My original design
> for the shell trap 
> was that it would locate the comment by determining where the
> function is 
> defined (by using "shopt -s extdebug" and "define -F") and then just
> grab the
> comment by subtracting one from the line number. But it doesn’t quite
> work 
> for nested shell functions. So I think I'll just have to do what I do
> in build.py,
> scan backwards from the frame line number to locate the comment.

Don't worry about it too much, it just sounded quite neat and I don't
like work like that to get lost!

Cheers,

Richard

Re: [bitbake-devel] [PATCH v2 0/4] [RFC] Shell func backtracing

[Chris Laplante]

> > Unfortunately not, since the parser resets the lineno/func to "1,
> > autogenerated".
> > I'll have to teach the parser to maintain the original location
> > information, probably as some sort of array since you can of course
> > have multiple appends.
> >
> > Even then, it won't be able to trace where people do
> > d.appendVar("do_compile", ...
> > I have some half-baked ideas on how to handle that but it would
> > probably require changes to data_smart.
> 
> Have a look at what the d.varhistory.emit(var, oval, val, o, d) call does in emit()
> in data.py. The hard part is we have to have variable tracking turned on for
> that to work. You can see the output with "bitbake -e". We don't turn on
> variable tracking by default as its a heavy load for general parsing and not
> needed. You may be able to enable that in worker context though.

"bitbake -e" doesn’t seem to print the history information for tasks though. Maybe I skimmed it too fast but I don't remember it all the other times I've used "bitbake -e" either. 

> > > I do sometimes run the scripts by hand myself. Another long standing
> > > idea I keep meaning to sort out is adding the correct pseudo
> > > environment to those functions, even as comments the user could
> > > enable for debugging.
> > >
> > > Did you finish the other shell trap code? I think we're probably ok
> > > without that but if you had it working it might be good to put in
> > > the archives?
> >
> > I nearly finished it. I'll submit a v3 with it. My original design for
> > the shell trap was that it would locate the comment by determining
> > where the function is defined (by using "shopt -s extdebug" and
> > "define -F") and then just grab the comment by subtracting one from
> > the line number. But it doesn’t quite work for nested shell functions.
> > So I think I'll just have to do what I do in build.py, scan backwards
> > from the frame line number to locate the comment.
> 
> Don't worry about it too much, it just sounded quite neat and I don't like work
> like that to get lost!

I sent the rough WIP version I had for your Friday reading pleasure :)

> 
> Cheers,
> 
> Richard
> 

Re: [bitbake-devel] [PATCH v2 0/4] [RFC] Shell func backtracing

[Richard Purdie]

On Fri, 2020-08-07 at 14:30 +0000, chris.laplante@agilent.com wrote:
> > > Unfortunately not, since the parser resets the lineno/func to "1,
> > > autogenerated".
> > > I'll have to teach the parser to maintain the original location
> > > information, probably as some sort of array since you can of
> > > course
> > > have multiple appends.
> > > 
> > > Even then, it won't be able to trace where people do
> > > d.appendVar("do_compile", ...
> > > I have some half-baked ideas on how to handle that but it would
> > > probably require changes to data_smart.
> > 
> > Have a look at what the d.varhistory.emit(var, oval, val, o, d)
> > call does in emit()
> > in data.py. The hard part is we have to have variable tracking
> > turned on for
> > that to work. You can see the output with "bitbake -e". We don't
> > turn on
> > variable tracking by default as its a heavy load for general
> > parsing and not
> > needed. You may be able to enable that in worker context though.
> 
> "bitbake -e" doesn’t seem to print the history information for tasks
> though. Maybe I skimmed it too fast but I don't remember it all the
> other times I've used "bitbake -e" either. 

Hmm. I wonder why we don't show that for functions. Its easy to turn
on:

diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py
index b0683c51803..06be35fb149 100644
--- a/bitbake/lib/bb/data.py
+++ b/bitbake/lib/bb/data.py
@@ -185,7 +185,7 @@ def emit_env(o=sys.__stdout__, d = init(), all=False):
     grouped = groupby(keys, isfunc)
     for isfunc, keys in grouped:
         for key in sorted(keys):
-            emit_var(key, o, d, all and not isfunc) and o.write('\n')
+            emit_var(key, o, d, all) and o.write('\n')
 
 def exported_keys(d):
     return (key for key in d.keys() if not key.startswith('__') and

Perhaps we should do that, really not sure why we don't.

My point is the data is there (functions are treated the same as any
other variable internally in most code paths, they just have an extra
varFlag set).

> > Don't worry about it too much, it just sounded quite neat and I
> > don't like work
> > like that to get lost!
> 
> I sent the rough WIP version I had for your Friday reading pleasure
> :)

Thanks!

Cheers,

Richard

Re: [bitbake-devel] [PATCH v2 0/4] [RFC] Shell func backtracing

[Richard Purdie]

On Sun, 2020-08-02 at 10:35 -0400, Chris Laplante via
lists.openembedded.org wrote:
> This patch series replaces the one I sent Friday. It implements two
> backtraces when a Bash shell function fails:
> 
> 1) A backtrace within the .run file that BB generated.
> 2) A computed backtrace of the actual metadata locations (e.g. bb or
>     bbclass) where each shell function is defined.
> 
> At one point I had both backtraces implemented entirely in the shell
> traps. I moved (2) to build.py because it is easier to implement in
> Python. One nice thing about leaving it in the shell trap would be
> that
> it would still work even if a user was running a .run script
> directly,
> but it seems like not many people do that. Still, if it's something
> we're interested in, know that it's possible.

This didn't work well in testing:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1239/steps/10/logs/step1b
from
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1239

I did grab the failing script:

rpurdie@debian10-ty-1:~$ cat /home/pokybuild/yocto-worker/a-full/build/build-renamed/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/temp/run.fixup_distcc_mirror_tarball.9175 
#!/bin/sh

# Emit a useful diagnostic if something fails:
bb_sh_exit_handler() {
    ret=$?
    if [ "$ret" != 0 ]; then
        echo "WARNING: exit code $ret from a shell command."
    fi
    exit $ret
}

bb_bash_exit_handler() {
    ret=$?
    if [ "$ret" != 0 ]; then
        echo "WARNING: ${BASH_SOURCE[0]}:${BASH_LINENO[0]} exit $ret from '$BASH_COMMAND'"

        echo "WARNING: Backtrace (BB generated script): "
        for ((i=1; i<${#FUNCNAME[@]}; i++)); do
            if [ $i -eq 1 ]; then
                # TODO: maybe manually track LINENO with a DEBUG trap?
                # LINENO / BASH_LINENO for the first frame in an EXIT trap is wrong :/
                # http://gnu-bash.2382.n7.nabble.com/trap-echo-quot-trap-exit-on-LINENO-quot-EXIT-gt-wrong-linenumber-td3666.html
                echo "	#$((i)): ${FUNCNAME[$i]}, ${BASH_SOURCE[$((i-1))]}, line ?"
            else
                echo "	#$((i)): ${FUNCNAME[$i]}, ${BASH_SOURCE[$((i-1))]}, line ${BASH_LINENO[$((i-1))]}"
            fi
        done
    fi
    exit $ret
}

bb_bash_err_handler() {
    ret=$?
    echo "WARNING: ${BASH_SOURCE[0]}:${BASH_LINENO[0]} exit $ret from '$BASH_COMMAND'"

    echo "WARNING: Backtrace (BB generated script): "
    for ((i=1; i<${#FUNCNAME[@]}; i++)); do
        echo "	#$((i-1)): ${FUNCNAME[$i]}, ${BASH_SOURCE[$((i-1))]}, line ${BASH_LINENO[$((i-1))]}"
    done

    trap "" EXIT
    exit $ret
}

case $BASH_VERSION in
"") trap 'bb_sh_exit_handler' 0
    set -e
    ;;
*)  trap 'bb_bash_err_handler' ERR
    trap 'bb_bash_exit_handler' 0
    set -eE
esac
export AR="i686-poky-linux-gcc-ar"
export AS="i686-poky-linux-as  "
export BUILD_AR="ar"
export BUILD_AS="as "
export BUILD_CC="gcc "
export BUILD_CCLD="gcc "
export BUILD_CFLAGS="-isystem/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/usr/include -O2 -pipe"
export BUILD_CPP="gcc  -E"
export BUILD_CPPFLAGS="-isystem/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/usr/include"
export BUILD_CXX="g++ "
export BUILD_CXXFLAGS="-isystem/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/usr/include -O2 -pipe"
export BUILD_FC="gfortran "
export BUILD_LD="ld "
export BUILD_LDFLAGS="-L/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/usr/lib                         -L/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/lib                         -Wl,--enable-new-dtags                         -Wl,-rpath-link,/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/usr/lib                         -Wl,-rpath-link,/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/lib                         -Wl,-rpath,/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/usr/lib                         -Wl,-rpath,/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/lib                         -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/home/pokybuild/yocto-worker/a-full/build/build/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2"
export BUILD_NM="nm"
export BUILD_RANLIB="ranlib"
export BUILD_STRIP="strip"
export CC="i686-poky-linux-gcc  -m32 -march=core2 -mtune=core2 -msse3 -mfpmath=sse -fstack-protector-strong  -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot"
export CCLD="i686-poky-linux-gcc  -m32 -march=core2 -mtune=core2 -msse3 -mfpmath=sse -fstack-protector-strong  -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot"
export CC_FOR_BUILD="gcc "
export CFLAGS=" -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0=/usr/src/debug/distcc/3.3.3-r0                      -fdebug-prefix-map=/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0=/usr/src/debug/distcc/3.3.3-r0                      -fdebug-prefix-map=/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native= "
export CFLAGS_FOR_BUILD="-isystem/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/usr/include -O2 -pipe"
export CPP="i686-poky-linux-gcc -E --sysroot=/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot  -m32 -march=core2 -mtune=core2 -msse3 -mfpmath=sse -fstack-protector-strong  -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security"
export CPPFLAGS=""
export CPPFLAGS_FOR_BUILD="-isystem/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/usr/include"
export CPP_FOR_BUILD="gcc  -E"
export CXX="i686-poky-linux-g++  -m32 -march=core2 -mtune=core2 -msse3 -mfpmath=sse -fstack-protector-strong  -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot"
export CXXFLAGS=" -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0=/usr/src/debug/distcc/3.3.3-r0                      -fdebug-prefix-map=/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0=/usr/src/debug/distcc/3.3.3-r0                      -fdebug-prefix-map=/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native=  -fvisibility-inlines-hidden"
export CXXFLAGS_FOR_BUILD="-isystem/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/usr/include -O2 -pipe"
export CXX_FOR_BUILD="g++ "
unset DISTRO
export FC="i686-poky-linux-gfortran  -m32 -march=core2 -mtune=core2 -msse3 -mfpmath=sse -fstack-protector-strong  -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot"
export GIT_CEILING_DIRECTORIES="/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0"
export HOME="/home/pokybuild"
export LC_ALL="en_US.UTF-8"
export LD="i686-poky-linux-ld --sysroot=/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot  "
export LDFLAGS="-Wl,-O1 -Wl,--hash-style=gnu  -Wl,-z,relro,-z,now"
export LDFLAGS_FOR_BUILD="-L/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/usr/lib                         -L/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/lib                         -Wl,--enable-new-dtags                         -Wl,-rpath-link,/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/usr/lib                         -Wl,-rpath-link,/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/lib                         -Wl,-rpath,/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/usr/lib                         -Wl,-rpath,/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/lib                         -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/home/pokybuild/yocto-worker/a-full/build/build/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2"
export LD_FOR_BUILD="ld "
export LOGNAME="pokybuild"
unset MACHINE
export MAKE="make"
export NM="i686-poky-linux-nm"
export OBJCOPY="i686-poky-linux-objcopy"
export OBJDUMP="i686-poky-linux-objdump"
export PATH="/home/pokybuild/yocto-worker/a-full/build/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-worker/a-full/build/scripts:/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/usr/bin/i686-poky-linux:/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/sbin:/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot-native/bin:/home/pokybuild/yocto-worker/a-full/build/bitbake/bin:/home/pokybuild/yocto-worker/a-full/build/build/tmp/hosttools"
export PERL_HASH_SEED="0"
export PKG_CONFIG_DIR="/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot/usr/lib/pkgconfig"
export PKG_CONFIG_DISABLE_UNINSTALLED="yes"
export PKG_CONFIG_LIBDIR="/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot/usr/lib/pkgconfig"
export PKG_CONFIG_PATH="/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot/usr/lib/pkgconfig:/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot/usr/share/pkgconfig"
export PKG_CONFIG_SYSROOT_DIR="/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/recipe-sysroot"
export PKG_CONFIG_SYSTEM_INCLUDE_PATH="/usr/include"
export PKG_CONFIG_SYSTEM_LIBRARY_PATH="/lib:/usr/lib"
export PSEUDO_DISABLED="1"
export PSEUDO_UNLOAD="1"
export PYTHONHASHSEED="0"
export RANLIB="i686-poky-linux-gcc-ranlib"
export READELF="i686-poky-linux-readelf"
unset SHELL
export SOURCE_DATE_EPOCH="0"
export STRINGS="i686-poky-linux-strings"
export STRIP="i686-poky-linux-strip"
unset TARGET_ARCH
export TZ="UTC"
export USER="pokybuild"
export base_bindir="/bin"
export base_libdir="/lib"
export base_prefix=""
export base_sbindir="/sbin"
export bindir="/usr/bin"
export datadir="/usr/share"
export docdir="/usr/share/doc"
export exec_prefix="/usr"
export includedir="/usr/include"
export infodir="/usr/share/info"
export libdir="/usr/lib"
export libexecdir="/usr/libexec"
export localstatedir="/var"
export lt_cv_sys_lib_dlsearch_path_spec="/usr/lib /lib"
export mandir="/usr/share/man"
export nonarch_base_libdir="/lib"
export nonarch_libdir="/usr/lib"
export oldincludedir="/usr/include"
export prefix="/usr"
export sbindir="/usr/sbin"
export servicedir="/srv"
export sharedstatedir="/com"
export sysconfdir="/etc"
export systemd_system_unitdir="/lib/systemd/system"
export systemd_unitdir="/lib/systemd"
export systemd_user_unitdir="/usr/lib/systemd/user"

# line: 72, file: /home/pokybuild/yocto-worker/a-full/build/meta/recipes-devtools/distcc/distcc_3.3.3.bb
fixup_distcc_mirror_tarball() {
	TBALL=/srv/autobuilder/autobuilder.yoctoproject.org/current_sources/git2_github.com.distcc.distcc.git.tar.gz
	if [ -f $TBALL ]; then
		TDIR=`mktemp -d`
		cd $TDIR
		tar -xzf $TBALL
		set +e
		git rev-parse --verify 3.2
		if [ "$?" != "0" ]; then
			git branch 3.2 d8b18df3e9dcbe4f092bed565835d3975e99432c
			tar -czf $TBALL *
		fi
		set -e
		rm -rf $TDIR/*
	fi
}

fixup_distcc_mirror_tarball

# cleanup
ret=$?
trap '' 0
exit $ret

Re: [bitbake-devel] [PATCH v2 0/4] [RFC] Shell func backtracing

[Chris Laplante]

> This didn't work well in testing:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1239/ste
> ps/10/logs/step1b
> from
> https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1239

How do I view the log file it talks about, /home/pokybuild/yocto-worker/a-full/build/build/tmp/log/error-report/error_report_20200807202843.txt ?

Thanks,
Chris

Re: [bitbake-devel] [PATCH v2 0/4] [RFC] Shell func backtracing

[Richard Purdie]

On Fri, 2020-08-07 at 21:24 +0000, chris.laplante@agilent.com wrote:
> > This didn't work well in testing:
> > 
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1239/ste
> > ps/10/logs/step1b
> > from
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1239
> 
> How do I view the log file it talks about, /home/pokybuild/yocto-worker/a-full/build/build/tmp/log/error-report/error_report_20200807202843.txt ?

That would be the data uploaded onto errors.yoctoproject.org but the
above logs should have all the data too.

Cheers,

Richard

Re: [bitbake-devel] [PATCH v2 0/4] [RFC] Shell func backtracing

[Chris Laplante]

> > > This didn't work well in testing:
> > >
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/12
> > > 39/ste
> > > ps/10/logs/step1b
> > > from
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/12
> > > 39
> >
> > How do I view the log file it talks about, /home/pokybuild/yocto-worker/a-
> full/build/build/tmp/log/error-report/error_report_20200807202843.txt ?
> 
> That would be the data uploaded onto errors.yoctoproject.org but the above
> logs should have all the data too.

/home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/temp/run.fixup_distcc_mirror_tarball.9175: 18: /home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-poky-linux/distcc/3.3.3-r0/temp/run.fixup_distcc_mirror_tarball.9175: Syntax error: Bad for loop variable

So I guess it's a 'sh' problem. I thought I had the code set up correctly to handle sh vs bash (specifically, basically all of this is turned off for sh since it doesn’t have the backtrace capability). But I guess not.

Can you confirm that the autobuilder is using sh and not bash?

Chris 

Re: [bitbake-devel] [PATCH v2 0/4] [RFC] Shell func backtracing

[Richard Purdie]

On Sat, 2020-08-08 at 00:19 +0000, chris.laplante@agilent.com wrote:
> > > > This didn't work well in testing:
> > > > 
> > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/12
> > > > 39/ste
> > > > ps/10/logs/step1b
> > > > from
> > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/12
> > > > 39
> > > 
> > > How do I view the log file it talks about, /home/pokybuild/yocto-
> > > worker/a-
> > full/build/build/tmp/log/error-
> > report/error_report_20200807202843.txt ?
> > 
> > That would be the data uploaded onto errors.yoctoproject.org but
> > the above
> > logs should have all the data too.
> 
> /home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-32-
> poky-linux/distcc/3.3.3-r0/temp/run.fixup_distcc_mirror_tarball.9175: 
> 18: /home/pokybuild/yocto-worker/a-full/build/build/tmp/work/core2-
> 32-poky-linux/distcc/3.3.3-
> r0/temp/run.fixup_distcc_mirror_tarball.9175: Syntax error: Bad for
> loop variable
> 
> So I guess it's a 'sh' problem. I thought I had the code set up
> correctly to handle sh vs bash (specifically, basically all of this
> is turned off for sh since it doesn’t have the backtrace capability).
> But I guess not.
> 
> Can you confirm that the autobuilder is using sh and not bash?

That worker has dash:

rpurdie@debian10-ty-1:~$ ls -la /bin/sh
lrwxrwxrwx 1 root root 4 Jul 10  2019 /bin/sh -> dash

Cheers,

Richard

Re: [bitbake-devel] [PATCH v2 0/4] [RFC] Shell func backtracing

[Chris Laplante]

> That worker has dash:
> 
> rpurdie@debian10-ty-1:~$ ls -la /bin/sh
> lrwxrwxrwx 1 root root 4 Jul 10  2019 /bin/sh -> dash

OK. Even though dash isn't executing the bash part of the code, it is still choking on the syntax. I'll have to change the 'for' loop syntax to be POSIX-compliant.

Of course, dash and vanilla sh lack pretty much every piece of functionality I need to implement the backtracing. So once I fix the syntax, you still won't actually get any interesting output by using the autobuilder to test the patches.

Thanks,
Chris

Re: [bitbake-devel] [PATCH v2 0/4] [RFC] Shell func backtracing

[Richard Purdie]

On Sat, 2020-08-08 at 18:40 +0000, chris.laplante@agilent.com wrote:
> > That worker has dash:
> > 
> > rpurdie@debian10-ty-1:~$ ls -la /bin/sh
> > lrwxrwxrwx 1 root root 4 Jul 10  2019 /bin/sh -> dash
> 
> OK. Even though dash isn't executing the bash part of the code, it is
> still choking on the syntax. I'll have to change the 'for' loop
> syntax to be POSIX-compliant.
> 
> Of course, dash and vanilla sh lack pretty much every piece of
> functionality I need to implement the backtracing. So once I fix the
> syntax, you still won't actually get any interesting output by using
> the autobuilder to test the patches.

Some of the autobuilder workers have bash as /bin/sh, others don't, the
cluster is deliberately non-homogeneous. So it should see some testing.
Obviously we can't break dash users though...

Cheers,

Richard

Re: [bitbake-devel] [PATCH v2 0/4] [RFC] Shell func backtracing

[Chris Laplante]

Hi Richard,

> On Fri, 2020-08-07 at 14:30 +0000, chris.laplante@agilent.com wrote:
> > > > Unfortunately not, since the parser resets the lineno/func to "1,
> > > > autogenerated".
> > > > I'll have to teach the parser to maintain the original location
> > > > information, probably as some sort of array since you can of
> > > > course have multiple appends.
> > > >
> > > > Even then, it won't be able to trace where people do
> > > > d.appendVar("do_compile", ...
> > > > I have some half-baked ideas on how to handle that but it would
> > > > probably require changes to data_smart.
> > >
> > > Have a look at what the d.varhistory.emit(var, oval, val, o, d) call
> > > does in emit() in data.py. The hard part is we have to have variable
> > > tracking turned on for that to work. You can see the output with
> > > "bitbake -e". We don't turn on variable tracking by default as its a
> > > heavy load for general parsing and not needed. You may be able to
> > > enable that in worker context though.
> >
> > "bitbake -e" doesn’t seem to print the history information for tasks
> > though. Maybe I skimmed it too fast but I don't remember it all the
> > other times I've used "bitbake -e" either.
> 
> Hmm. I wonder why we don't show that for functions. Its easy to turn
> on:
> 
> diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py index
> b0683c51803..06be35fb149 100644
> --- a/bitbake/lib/bb/data.py
> +++ b/bitbake/lib/bb/data.py
> @@ -185,7 +185,7 @@ def emit_env(o=sys.__stdout__, d = init(), all=False):
>      grouped = groupby(keys, isfunc)
>      for isfunc, keys in grouped:
>          for key in sorted(keys):
> -            emit_var(key, o, d, all and not isfunc) and o.write('\n')
> +            emit_var(key, o, d, all) and o.write('\n')
> 
>  def exported_keys(d):
>      return (key for key in d.keys() if not key.startswith('__') and
> 
> Perhaps we should do that, really not sure why we don't.
> 
> My point is the data is there (functions are treated the same as any other
> variable internally in most code paths, they just have an extra varFlag set).
> 
> > > Don't worry about it too much, it just sounded quite neat and I
> > > don't like work like that to get lost!
> >
> > I sent the rough WIP version I had for your Friday reading pleasure
> > :)

What do you think if we changed data_smart to always track variable history for lowercase variables (i.e. functions)? This would let me generate backtraces for even appended/prepended functions. 

Chris