Re: Autobuilder data collection for intermittent bugs

["Sakib Sajal" <sakib.sajal@windriver.com>]

On 2021-03-26 1:50 p.m., Richard Purdie wrote:
> [Please note: This e-mail is from an EXTERNAL e-mail address]
>
> On Thu, 2021-03-25 at 20:00 -0400, Randy MacLeod wrote:
>> Thanks for the update Sakib.
>>
>> We're planning to follow Steve's example of getting commit access to
>> contrib branches of y-a-h and poky and using the YP AB to start custom
>> workers/job initially.  We'll also keep working to duplicate the
>> AB behaviour locally. Trevor has asked Konrad for an additional system
>> or two since when jobs are running, the web UI is
>> painfully slow since it's runnig on the same node right now.
>>
>> The one part that Sakib didn't ask about is how to get all the workers
>> to do the data collection but that's not urgent since we'll be doing
>> some smaller scale experiments tomorrow. If you can point out a helpful
>> example, that'd be good.
> The easiest is the fact that the workers all share an NFS mount which
> is at BASE_SHAREDDIR in config.json. See also WEBPUBLISH_DIR and
> WEBPUBLISH_URL.
Ok. We have not gone there, yet.
>
> We use this to share the reproducible build failures and to share the
> reproducible build failures/diffoscope and test results, e.g.:
> OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/
>
> or perhaps more useful is scripts/collect-results:
>
> """
> #!/bin/bash
> WORKDIR=$1
> DEST=$2
> target=$3
>
> RESFILE=$WORKDIR/tmp/log/oeqa/testresults.json
>
> if [ -e $RESFILE ]; then
>      mkdir -p $DEST/$target
>      cp $WORKDIR/tmp/log/oeqa/testresults.json  $DEST/$target/
> fi
> """
>
> would let you do something similar and DEST here is specific to a given
> build (date stamp + number). An example would be:
>
> https://autobuilder.yocto.io/pub/non-release/20210129-3/
>
> and then you could create a folder per target. You may need to
> include the worker name in the log.

We were able to follow the example and collect results in:

https://autobuilder.yocto.io/pub/non-release/20210331-18/testresults/qa-extras2/2021-03-31--20-32/

host_stats.0

host_stats.1

host_stats.2

We are renaming the files to host_stats_<step>.txt so that it would open 
on the browser.


We were looking at generate-testresult-index.py and would like to append 
a new column called "AB INT" and hyperlink each file in the folder where 
"top" was triggered. Normally there should not be any links. 
Occasionally, we should see 1 or 2 but not more unless things go really 
bad. Do you agree this is a reasonable approach? Who do we contact about 
testing the changes? Trevor has an internal instance running, we could 
try it out there.


Below is the diff of the changes we plan to make for the index. We do 
not completely understand the script yet, we are working on it.

autobuilder/yocto-autobuilder-helper$ git diff 
./scripts/generate-testresult-index.py
diff --git a/scripts/generate-testresult-index.py 
b/scripts/generate-testresult-index.py
index d9d577e..27fe06f 100755
--- a/scripts/generate-testresult-index.py
+++ b/scripts/generate-testresult-index.py
@@ -33,6 +33,7 @@ index_templpate = """
    <th>Performance Reports</th>
    <th>ptest Logs</th>
    <th>Buildhistory</th>
+  <th>AB INT</th>
  </tr>
  </thead>
  <tdata>
@@ -57,6 +58,11 @@ index_templpate = """
       <a href="{{bh[0]}}">{{bh[1]}}</a>
     {% endfor %}
     </td>
+   <td>
+   {% for abint in entry[8] %}
+     <a href="{{abint[0]}}">{{abint[1]}}</a>
+   {% endfor %}
+   </td>
  </tr>
  {% endfor %}
  </tdata>
@@ -145,6 +151,11 @@ for build in sorted(os.listdir(path), key=keygen, 
reverse=True):
      if os.path.exists(buildpath + "/qemuarm/buildhistory.txt"):
          buildhistory.append((reldir + 
"testresults/qemuarm/buildhistory.txt", "qemuarm"))

+    abint = []
+    for p in glob.glob(buildpath + "/ab-int*/host_stats*"):
+        abintname = os.path.basename(os.path.dirname(p))
+        abintreports.append((reldir + "testresults/" + abintname + "/" 
+ os.path.basename(p), perfname.replace("buildperf-",""))) #todo
+
      branch = get_build_branch(buildpath)


Regards,

Sakib, Randy

>
> Cheers,
>
> Richard
>