Re: Autobuilder data collection for intermittent bugs

["Richard Purdie" <richard.purdie@linuxfoundation.org>]

Hi Sakib,

On Thu, 2021-03-25 at 18:11 -0400, Sakib Sajal wrote:
> I have been trying, with help from Randy, Trevor and Steve, to run some 
> experiments on an internal autobuilder instance to collect host data 
> during builds for signs of io latency.
> 
> Initially i modified yocto-autobuilder-helper/config.json and added the 
> following 3 lines to existing templates (and created a commit) that were 
> being used so that the builds would start collecting the data.
> 
> 
> +                "BB_HEARTBEAT_EVENT = '5'",
> +                "BB_LOG_HOST_STAT_ON_INTERVAL = '1'",
> +                "BB_LOG_HOST_STAT_CMDS = 'pwd; ls'"
> 
> I also created a template in yocto-autobuilder-helper/config.json
> 
> +        "data-collect" : {
> +            "BUILDINFO" : true,
> +            "BBTARGETS" : "core-image-minimal",
> +            "extravars" : [
> +                "BB_HEARTBEAT_EVENT = '5'",
> +                "BB_LOG_HOST_STAT_ON_INTERVAL = '1'",
> +                "BB_LOG_HOST_STAT_CMDS = 'pwd; ls'"
> +            ]
> +        },
> 
> and modified a builder to use the template:
> 
>          "beaglebone-alt" : {
> -            "MACHINE" : "beaglebone-yocto",
> -            "TEMPLATE" : "altcfg-hw"
> +            "MACHINE" : "qemux86-64",
> +            "TEMPLATE" : "data-collect"
>           },
> 
> Restarted both the controller and the worker, and started the 
> beaglebone-alt build. Once the build started, i checked the conf file in 
> the build directory but those lines were not appended to the local.conf 
> file and hence no log files.

You shouldn't need to restart the controller, the worker should pull and 
use the configuration from the helper branch specified.

I'd note that it writes the config to auto.conf, not local.conf.

It should also give output in the logs about what its doing. For example:

https://autobuilder.yoctoproject.org/typhoon/#/builders/106/builds/2144/steps/10/logs/stdio

is the "Write config" step of beaglebone-alt on the autobuilder where it
clearly says:

Writing /home/pokybuild/yocto-worker/beaglebone-alt/build/build/conf/auto.conf with contents:

What do your logs say?

> In summary, I have been trying to get the 3 lines from the "extravars" 
> above to appear in one of the local.conf files. However none of the 
> builds seems to have been altered.
> 
> I am wondering if the changes that I've made are even being picked up by 
> the controller. Am I missing something?
> 
> Is there an easier way to update the conf files?
> 
> Initially I'd like collect data on a single node and scale it to all the 
> nodes. Any ideas/suggestions are welcome!

What you're doing should work. Its hard to comment more without seeing to logs
but check you're looking at auto.conf.

Cheers,

Richard