Autobuilder data collection for intermittent bugs

Autobuilder data collection for intermittent bugs

[Sakib Sajal]

Hi Richard,

I have been trying, with help from Randy, Trevor and Steve, to run some 
experiments on an internal autobuilder instance to collect host data 
during builds for signs of io latency.

Initially i modified yocto-autobuilder-helper/config.json and added the 
following 3 lines to existing templates (and created a commit) that were 
being used so that the builds would start collecting the data.


+                "BB_HEARTBEAT_EVENT = '5'",
+                "BB_LOG_HOST_STAT_ON_INTERVAL = '1'",
+                "BB_LOG_HOST_STAT_CMDS = 'pwd; ls'"

I also created a template in yocto-autobuilder-helper/config.json

+        "data-collect" : {
+            "BUILDINFO" : true,
+            "BBTARGETS" : "core-image-minimal",
+            "extravars" : [
+                "BB_HEARTBEAT_EVENT = '5'",
+                "BB_LOG_HOST_STAT_ON_INTERVAL = '1'",
+                "BB_LOG_HOST_STAT_CMDS = 'pwd; ls'"
+            ]
+        },

and modified a builder to use the template:

         "beaglebone-alt" : {
-            "MACHINE" : "beaglebone-yocto",
-            "TEMPLATE" : "altcfg-hw"
+            "MACHINE" : "qemux86-64",
+            "TEMPLATE" : "data-collect"
          },

Restarted both the controller and the worker, and started the 
beaglebone-alt build. Once the build started, i checked the conf file in 
the build directory but those lines were not appended to the local.conf 
file and hence no log files.

In summary, I have been trying to get the 3 lines from the "extravars" 
above to appear in one of the local.conf files. However none of the 
builds seems to have been altered.

I am wondering if the changes that I've made are even being picked up by 
the controller. Am I missing something?

Is there an easier way to update the conf files?

Initially I'd like collect data on a single node and scale it to all the 
nodes. Any ideas/suggestions are welcome!


Sincerely,

Sakib Sajal

Re: Autobuilder data collection for intermittent bugs

[Richard Purdie]

Hi Sakib,

On Thu, 2021-03-25 at 18:11 -0400, Sakib Sajal wrote:
> I have been trying, with help from Randy, Trevor and Steve, to run some 
> experiments on an internal autobuilder instance to collect host data 
> during builds for signs of io latency.
> 
> Initially i modified yocto-autobuilder-helper/config.json and added the 
> following 3 lines to existing templates (and created a commit) that were 
> being used so that the builds would start collecting the data.
> 
> 
> +                "BB_HEARTBEAT_EVENT = '5'",
> +                "BB_LOG_HOST_STAT_ON_INTERVAL = '1'",
> +                "BB_LOG_HOST_STAT_CMDS = 'pwd; ls'"
> 
> I also created a template in yocto-autobuilder-helper/config.json
> 
> +        "data-collect" : {
> +            "BUILDINFO" : true,
> +            "BBTARGETS" : "core-image-minimal",
> +            "extravars" : [
> +                "BB_HEARTBEAT_EVENT = '5'",
> +                "BB_LOG_HOST_STAT_ON_INTERVAL = '1'",
> +                "BB_LOG_HOST_STAT_CMDS = 'pwd; ls'"
> +            ]
> +        },
> 
> and modified a builder to use the template:
> 
>          "beaglebone-alt" : {
> -            "MACHINE" : "beaglebone-yocto",
> -            "TEMPLATE" : "altcfg-hw"
> +            "MACHINE" : "qemux86-64",
> +            "TEMPLATE" : "data-collect"
>           },
> 
> Restarted both the controller and the worker, and started the 
> beaglebone-alt build. Once the build started, i checked the conf file in 
> the build directory but those lines were not appended to the local.conf 
> file and hence no log files.

You shouldn't need to restart the controller, the worker should pull and 
use the configuration from the helper branch specified.

I'd note that it writes the config to auto.conf, not local.conf.

It should also give output in the logs about what its doing. For example:

https://autobuilder.yoctoproject.org/typhoon/#/builders/106/builds/2144/steps/10/logs/stdio

is the "Write config" step of beaglebone-alt on the autobuilder where it
clearly says:

Writing /home/pokybuild/yocto-worker/beaglebone-alt/build/build/conf/auto.conf with contents:

What do your logs say?

> In summary, I have been trying to get the 3 lines from the "extravars" 
> above to appear in one of the local.conf files. However none of the 
> builds seems to have been altered.
> 
> I am wondering if the changes that I've made are even being picked up by 
> the controller. Am I missing something?
> 
> Is there an easier way to update the conf files?
> 
> Initially I'd like collect data on a single node and scale it to all the 
> nodes. Any ideas/suggestions are welcome!

What you're doing should work. Its hard to comment more without seeing to logs
but check you're looking at auto.conf.

Cheers,

Richard

Re: Autobuilder data collection for intermittent bugs

[Randy MacLeod]

On 2021-03-25 6:23 p.m., Richard Purdie wrote:
> Hi Sakib,
> 
> On Thu, 2021-03-25 at 18:11 -0400, Sakib Sajal wrote:
>> I have been trying, with help from Randy, Trevor and Steve, to run some
>> experiments on an internal autobuilder instance to collect host data
>> during builds for signs of io latency.
>>
>> Initially i modified yocto-autobuilder-helper/config.json and added the
>> following 3 lines to existing templates (and created a commit) that were
>> being used so that the builds would start collecting the data.
>>
>>
>> +                "BB_HEARTBEAT_EVENT = '5'",
>> +                "BB_LOG_HOST_STAT_ON_INTERVAL = '1'",
>> +                "BB_LOG_HOST_STAT_CMDS = 'pwd; ls'"
>>
>> I also created a template in yocto-autobuilder-helper/config.json
>>
>> +        "data-collect" : {
>> +            "BUILDINFO" : true,
>> +            "BBTARGETS" : "core-image-minimal",
>> +            "extravars" : [
>> +                "BB_HEARTBEAT_EVENT = '5'",
>> +                "BB_LOG_HOST_STAT_ON_INTERVAL = '1'",
>> +                "BB_LOG_HOST_STAT_CMDS = 'pwd; ls'"
>> +            ]
>> +        },
>>
>> and modified a builder to use the template:
>>
>>           "beaglebone-alt" : {
>> -            "MACHINE" : "beaglebone-yocto",
>> -            "TEMPLATE" : "altcfg-hw"
>> +            "MACHINE" : "qemux86-64",
>> +            "TEMPLATE" : "data-collect"
>>            },
>>
>> Restarted both the controller and the worker, and started the
>> beaglebone-alt build. Once the build started, i checked the conf file in
>> the build directory but those lines were not appended to the local.conf
>> file and hence no log files.
> 
> You shouldn't need to restart the controller, the worker should pull and
> use the configuration from the helper branch specified.
> 
> I'd note that it writes the config to auto.conf, not local.conf.
> 
> It should also give output in the logs about what its doing. For example:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/106/builds/2144/steps/10/logs/stdio
> 
> is the "Write config" step of beaglebone-alt on the autobuilder where it
> clearly says:
> 
> Writing /home/pokybuild/yocto-worker/beaglebone-alt/build/build/conf/auto.conf with contents:
> 
> What do your logs say?
> 
>> In summary, I have been trying to get the 3 lines from the "extravars"
>> above to appear in one of the local.conf files. However none of the
>> builds seems to have been altered.
>>
>> I am wondering if the changes that I've made are even being picked up by
>> the controller. Am I missing something?
>>
>> Is there an easier way to update the conf files?
>>
>> Initially I'd like collect data on a single node and scale it to all the
>> nodes. Any ideas/suggestions are welcome!
> 
> What you're doing should work. Its hard to comment more without seeing to logs
> but check you're looking at auto.conf.

Thanks for the update Sakib.

We're planning to follow Steve's example of getting commit access to
contrib branches of y-a-h and poky and using the YP AB to start custom 
workers/job initially.  We'll also keep working to duplicate the
AB behaviour locally. Trevor has asked Konrad for an additional system
or two since when jobs are running, the web UI is
painfully slow since it's runnig on the same node right now.

The one part that Sakib didn't ask about is how to get all the workers
to do the data collection but that's not urgent since we'll be doing
some smaller scale experiments tomorrow. If you can point out a helpful
example, that'd be good.

Hopefully we'll have a good collection of data showing what processes
are running when the IO latency is > 5 seconds sometime next week.

../Randy

> 
> Cheers,
> 
> Richard
> 
> 


-- 
# Randy MacLeod
# Wind River Linux

Re: Autobuilder data collection for intermittent bugs

[Richard Purdie]

On Thu, 2021-03-25 at 20:00 -0400, Randy MacLeod wrote:
> Thanks for the update Sakib.
> 
> We're planning to follow Steve's example of getting commit access to
> contrib branches of y-a-h and poky and using the YP AB to start custom 
> workers/job initially.  We'll also keep working to duplicate the
> AB behaviour locally. Trevor has asked Konrad for an additional system
> or two since when jobs are running, the web UI is
> painfully slow since it's runnig on the same node right now.
> 
> The one part that Sakib didn't ask about is how to get all the workers
> to do the data collection but that's not urgent since we'll be doing
> some smaller scale experiments tomorrow. If you can point out a helpful
> example, that'd be good.

The easiest is the fact that the workers all share an NFS mount which
is at BASE_SHAREDDIR in config.json. See also WEBPUBLISH_DIR and 
WEBPUBLISH_URL.

We use this to share the reproducible build failures and to share the 
reproducible build failures/diffoscope and test results, e.g.:
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/

or perhaps more useful is scripts/collect-results:

"""
#!/bin/bash
WORKDIR=$1
DEST=$2
target=$3

RESFILE=$WORKDIR/tmp/log/oeqa/testresults.json 

if [ -e $RESFILE ]; then
    mkdir -p $DEST/$target
    cp $WORKDIR/tmp/log/oeqa/testresults.json  $DEST/$target/
fi
"""

would let you do something similar and DEST here is specific to a given 
build (date stamp + number). An example would be:

https://autobuilder.yocto.io/pub/non-release/20210129-3/

and then you could create a folder per target. You may need to 
include the worker name in the log.

Cheers,

Richard

Re: Autobuilder data collection for intermittent bugs

[Sakib Sajal]

On 2021-03-26 1:50 p.m., Richard Purdie wrote:
> [Please note: This e-mail is from an EXTERNAL e-mail address]
>
> On Thu, 2021-03-25 at 20:00 -0400, Randy MacLeod wrote:
>> Thanks for the update Sakib.
>>
>> We're planning to follow Steve's example of getting commit access to
>> contrib branches of y-a-h and poky and using the YP AB to start custom
>> workers/job initially.  We'll also keep working to duplicate the
>> AB behaviour locally. Trevor has asked Konrad for an additional system
>> or two since when jobs are running, the web UI is
>> painfully slow since it's runnig on the same node right now.
>>
>> The one part that Sakib didn't ask about is how to get all the workers
>> to do the data collection but that's not urgent since we'll be doing
>> some smaller scale experiments tomorrow. If you can point out a helpful
>> example, that'd be good.
> The easiest is the fact that the workers all share an NFS mount which
> is at BASE_SHAREDDIR in config.json. See also WEBPUBLISH_DIR and
> WEBPUBLISH_URL.
Ok. We have not gone there, yet.
>
> We use this to share the reproducible build failures and to share the
> reproducible build failures/diffoscope and test results, e.g.:
> OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/
>
> or perhaps more useful is scripts/collect-results:
>
> """
> #!/bin/bash
> WORKDIR=$1
> DEST=$2
> target=$3
>
> RESFILE=$WORKDIR/tmp/log/oeqa/testresults.json
>
> if [ -e $RESFILE ]; then
>      mkdir -p $DEST/$target
>      cp $WORKDIR/tmp/log/oeqa/testresults.json  $DEST/$target/
> fi
> """
>
> would let you do something similar and DEST here is specific to a given
> build (date stamp + number). An example would be:
>
> https://autobuilder.yocto.io/pub/non-release/20210129-3/
>
> and then you could create a folder per target. You may need to
> include the worker name in the log.

We were able to follow the example and collect results in:

https://autobuilder.yocto.io/pub/non-release/20210331-18/testresults/qa-extras2/2021-03-31--20-32/

host_stats.0

host_stats.1

host_stats.2

We are renaming the files to host_stats_<step>.txt so that it would open 
on the browser.


We were looking at generate-testresult-index.py and would like to append 
a new column called "AB INT" and hyperlink each file in the folder where 
"top" was triggered. Normally there should not be any links. 
Occasionally, we should see 1 or 2 but not more unless things go really 
bad. Do you agree this is a reasonable approach? Who do we contact about 
testing the changes? Trevor has an internal instance running, we could 
try it out there.


Below is the diff of the changes we plan to make for the index. We do 
not completely understand the script yet, we are working on it.

autobuilder/yocto-autobuilder-helper$ git diff 
./scripts/generate-testresult-index.py
diff --git a/scripts/generate-testresult-index.py 
b/scripts/generate-testresult-index.py
index d9d577e..27fe06f 100755
--- a/scripts/generate-testresult-index.py
+++ b/scripts/generate-testresult-index.py
@@ -33,6 +33,7 @@ index_templpate = """
    <th>Performance Reports</th>
    <th>ptest Logs</th>
    <th>Buildhistory</th>
+  <th>AB INT</th>
  </tr>
  </thead>
  <tdata>
@@ -57,6 +58,11 @@ index_templpate = """
       <a href="{{bh[0]}}">{{bh[1]}}</a>
     {% endfor %}
     </td>
+   <td>
+   {% for abint in entry[8] %}
+     <a href="{{abint[0]}}">{{abint[1]}}</a>
+   {% endfor %}
+   </td>
  </tr>
  {% endfor %}
  </tdata>
@@ -145,6 +151,11 @@ for build in sorted(os.listdir(path), key=keygen, 
reverse=True):
      if os.path.exists(buildpath + "/qemuarm/buildhistory.txt"):
          buildhistory.append((reldir + 
"testresults/qemuarm/buildhistory.txt", "qemuarm"))

+    abint = []
+    for p in glob.glob(buildpath + "/ab-int*/host_stats*"):
+        abintname = os.path.basename(os.path.dirname(p))
+        abintreports.append((reldir + "testresults/" + abintname + "/" 
+ os.path.basename(p), perfname.replace("buildperf-",""))) #todo
+
      branch = get_build_branch(buildpath)


Regards,

Sakib, Randy

>
> Cheers,
>
> Richard
>

Re: Autobuilder data collection for intermittent bugs

[Richard Purdie]

On Wed, 2021-03-31 at 17:45 -0400, Sakib Sajal wrote:
> We were able to follow the example and collect results in:
> 
> https://autobuilder.yocto.io/pub/non-release/20210331-18/testresults/qa-extras2/2021-03-31--20-32/
> 
> host_stats.0
> 
> host_stats.1
> 
> host_stats.2
> 
> We are renaming the files to host_stats_<step>.txt so that it would open 
> on the browser.
> 
> 
> We were looking at generate-testresult-index.py and would like to append 
> a new column called "AB INT" and hyperlink each file in the folder where 
> "top" was triggered. Normally there should not be any links. 
> Occasionally, we should see 1 or 2 but not more unless things go really 
> bad. Do you agree this is a reasonable approach? Who do we contact about 
> testing the changes? Trevor has an internal instance running, we could 
> try it out there.
> 
> 
> Below is the diff of the changes we plan to make for the index. We do 
> not completely understand the script yet, we are working on it.
> 
> autobuilder/yocto-autobuilder-helper$ git diff 
> ./scripts/generate-testresult-index.py
> diff --git a/scripts/generate-testresult-index.py 
> b/scripts/generate-testresult-index.py
> index d9d577e..27fe06f 100755
> --- a/scripts/generate-testresult-index.py
> +++ b/scripts/generate-testresult-index.py
> @@ -33,6 +33,7 @@ index_templpate = """
>     <th>Performance Reports</th>
>     <th>ptest Logs</th>
>     <th>Buildhistory</th>
> +  <th>AB INT</th>
>   </tr>
>   </thead>
>   <tdata>
> @@ -57,6 +58,11 @@ index_templpate = """
>        <a href="{{bh[0]}}">{{bh[1]}}</a>
>      {% endfor %}
>      </td>
> +   <td>
> +   {% for abint in entry[8] %}
> +     <a href="{{abint[0]}}">{{abint[1]}}</a>
> +   {% endfor %}
> +   </td>
>   </tr>
>   {% endfor %}
>   </tdata>
> @@ -145,6 +151,11 @@ for build in sorted(os.listdir(path), key=keygen, 
> reverse=True):
>       if os.path.exists(buildpath + "/qemuarm/buildhistory.txt"):
>           buildhistory.append((reldir + 
> "testresults/qemuarm/buildhistory.txt", "qemuarm"))
> 
> +    abint = []
> +    for p in glob.glob(buildpath + "/ab-int*/host_stats*"):
> +        abintname = os.path.basename(os.path.dirname(p))
> +        abintreports.append((reldir + "testresults/" + abintname + "/" 
> + os.path.basename(p), perfname.replace("buildperf-",""))) #todo
> +
>       branch = get_build_branch(buildpath)
> 

Rather than messing with the main index which is "production", could you just create 
your own for now for testing? :)

FWIW I added tmpfs testing for qemu images into master-next (needs ab-helper 
master-next too) so it will be interesting to compare builds running with that
with the previous build bug trends.

Cheers,

Richard

Re: [OE-core] Autobuilder data collection for intermittent bugs

[Randy MacLeod]

On 2021-03-31 6:01 p.m., Richard Purdie wrote:
> On Wed, 2021-03-31 at 17:45 -0400, Sakib Sajal wrote:
>> We were able to follow the example and collect results in:
>>
>> https://autobuilder.yocto.io/pub/non-release/20210331-18/testresults/qa-extras2/2021-03-31--20-32/
>>
>> host_stats.0
>>
>> host_stats.1
>>
>> host_stats.2
>>
>> We are renaming the files to host_stats_<step>.txt so that it would open
>> on the browser.
>>
>>
>> We were looking at generate-testresult-index.py and would like to append
>> a new column called "AB INT" and hyperlink each file in the folder where
>> "top" was triggered. Normally there should not be any links.
>> Occasionally, we should see 1 or 2 but not more unless things go really
>> bad. Do you agree this is a reasonable approach? Who do we contact about
>> testing the changes? Trevor has an internal instance running, we could
>> try it out there.
>>
>>
>> Below is the diff of the changes we plan to make for the index. We do
>> not completely understand the script yet, we are working on it.
>>
>> autobuilder/yocto-autobuilder-helper$ git diff
>> ./scripts/generate-testresult-index.py
>> diff --git a/scripts/generate-testresult-index.py
>> b/scripts/generate-testresult-index.py
>> index d9d577e..27fe06f 100755
>> --- a/scripts/generate-testresult-index.py
>> +++ b/scripts/generate-testresult-index.py
>> @@ -33,6 +33,7 @@ index_templpate = """
>>      <th>Performance Reports</th>
>>      <th>ptest Logs</th>
>>      <th>Buildhistory</th>
>> +  <th>AB INT</th>
>>    </tr>
>>    </thead>
>>    <tdata>
>> @@ -57,6 +58,11 @@ index_templpate = """
>>         <a href="{{bh[0]}}">{{bh[1]}}</a>
>>       {% endfor %}
>>       </td>
>> +   <td>
>> +   {% for abint in entry[8] %}
>> +     <a href="{{abint[0]}}">{{abint[1]}}</a>
>> +   {% endfor %}
>> +   </td>
>>    </tr>
>>    {% endfor %}
>>    </tdata>
>> @@ -145,6 +151,11 @@ for build in sorted(os.listdir(path), key=keygen,
>> reverse=True):
>>        if os.path.exists(buildpath + "/qemuarm/buildhistory.txt"):
>>            buildhistory.append((reldir +
>> "testresults/qemuarm/buildhistory.txt", "qemuarm"))
>>
>> +    abint = []
>> +    for p in glob.glob(buildpath + "/ab-int*/host_stats*"):
>> +        abintname = os.path.basename(os.path.dirname(p))
>> +        abintreports.append((reldir + "testresults/" + abintname + "/"
>> + os.path.basename(p), perfname.replace("buildperf-",""))) #todo
>> +
>>        branch = get_build_branch(buildpath)
>>
> 
> Rather than messing with the main index which is "production", could you just create
> your own for now for testing? :)

Yes, we've figured all that out, thanks.
We'll send you a patch once we've completed testing on our instance of
the YP AB, early this week.

> 
> FWIW I added tmpfs testing for qemu images into master-next (needs ab-helper
> master-next too) so it will be interesting to compare builds running with that
> with the previous build bug trends.

Any conclusions so far?

../Randy

> 
> Cheers,
> 
> Richard
> 
> 
> 
> 
> 


-- 
# Randy MacLeod
# Wind River Linux

Re: [OE-core] Autobuilder data collection for intermittent bugs

[Richard Purdie]

On Sun, 2021-04-04 at 15:56 -0400, Randy MacLeod wrote:
> > 
> > 
> > Rather than messing with the main index which is "production", could you just create
> > your own for now for testing? :)
> 
> Yes, we've figured all that out, thanks.
> We'll send you a patch once we've completed testing on our instance of
> the YP AB, early this week.
> 
> > 
> > FWIW I added tmpfs testing for qemu images into master-next (needs ab-helper
> > master-next too) so it will be interesting to compare builds running with that
> > with the previous build bug trends.
> 
> Any conclusions so far?

I think it helps make the runtime testing more consistent but we are seeing 
a number of "qemu didn't start in 120s" messages instead. That is probably
more desirable than random runtime failures in qemu.

We have still seen one valgrind ptest failure with the tmpfs patch too so it
doesn't remove all of them.

It also won't fix the bitbake server starting timeout problem.

I'm leaning to merging it.

I did note that in the 120s qemu timeout, we did get a list of other
processes that were running:

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1973

and it shows webkit being built. Would be interesting to correlate against
other failures, see what the pattern of running tasks is.

Cheers,

Richard