Re: [PATCH v3] bitfield.h: add FIELD_MAX() and field_max()

[Maxim Kuvyrkov <maxim.kuvyrkov@linaro.org>]

> On 2 Apr 2020, at 03:26, Nick Desaulniers <ndesaulniers@google.com> wrote:
> 
> On Wed, Apr 1, 2020 at 4:18 PM Alex Elder <elder@linaro.org> wrote:
>> 
>> On 4/1/20 5:26 PM, Nick Desaulniers wrote:
>>> 
>>> mainline is hosed for aarch64 due to some dtc failures.  I'm not sure
>>> how TCWG's CI chooses the bisection starting point, but if mainline
>>> was broken, and it jumped back say 300 commits, then the automated
>>> bisection may have converged on your first patch, but not the second.
>> 
>> This is similar to the situation I discussed with Maxim this
>> morning.  A different failure (yes, DTC related) led to an
>> automated bisect process, which landed on my commit. And my
>> commit unfortunately has the the known issue that was later
>> corrected.
>> 
>> Maxim said this was what started the automated bisect:
>> ===
>> +# 00:01:41 make[2]: *** [arch/arm64/boot/dts/ti/k3-am654-base-board.dtb] Error 2
>> +# 00:01:41 make[2]: *** [arch/arm64/boot/dts/ti/k3-j721e-common-proc-board.dtb] Error 2
>> +# 00:01:41 make[1]: *** [arch/arm64/boot/dts/ti] Error 2
>> +# 00:01:41 make: *** [dtbs] Error 2
> 
> DTC thread:
> https://lore.kernel.org/linux-arm-kernel/20200401223500.224253-1-ndesaulniers@google.com/
> 
> Maxim, can you describe how the last known good sha is chosen?  If you
> persist anything between builds, like ccache dir, maybe you could
> propagate a sha of the last successful build, updating it if no
> regression occurred?  Then that can always be a precise last known
> good sha.  Though I don't know if the merge commits complicate this.

Well, since you asked, the simplified version is …

Bisection is done between “baseline" commit and “regressed” commit, not between "last known-good” commit and “bad” commit.  Each build has a metric, and regression happens when metric for new build is worse than metric for baseline build.  For tcwg_kernel jobs the metric is the number of .o files produced in the kernel build (i.e., build with 18555 .o files is worse than build with 18560 .o files).

If a new build hasn’t regressed compared to the baseline build, then baseline metric is set to that of the new build, and baseline commit is set to sha1 of the new build.

If the new build has regressed, then bisection between baseline sha1 and new sha1 is triggered.  Once bisection identifies the first bad commit, CI notification is emailed, and baseline is reset to the first bad commit — so that we detect even worse regressions when they occur.

The baseline state is recorded in git repos (with one branch per CI configuration):
- metric / artifacts: https://git.linaro.org/toolchain/ci/base-artifacts.git/refs/heads
- linux kernel: https://git.linaro.org/toolchain/ci/linux.git/refs/heads
- llvm: https://git.linaro.org/toolchain/ci/llvm-project.git/refs/heads

--
Maxim Kuvyrkov
https://www.linaro.org