Re: kvm-unit-tests : Kconfigs and extra kernel args for full coverage

[Dan Rue <dan.rue@linaro.org>]

On Mon, Feb 24, 2020 at 09:30:33AM -0800, Sean Christopherson wrote:
> On Mon, Feb 24, 2020 at 10:36:54PM +0530, Naresh Kamboju wrote:
> > Hi Paolo,
> > 
> > On Mon, 24 Feb 2020 at 18:36, Paolo Bonzini <pbonzini@redhat.com> wrote:
> > >
> > > On 24/02/20 13:53, Naresh Kamboju wrote:
> > > > FAIL  vmx (408624 tests, 3 unexpected failures, 2 expected
> > > > failures, 5 skipped)
> > >
> > > This could be fixed in a more recent kernel.
> > 
> > I will keep running these tests on most recent kernels.
> > 
> > My two cents,
> > OTOH, It would be great if we have monthly tag release for kvm-unit-tests.
> > 
> > LKFT plan to keep track of metadata / release tag version of each test suites
> > and kernel branches and versions details.
> > 
> > Currently LKFT sending out kselftests results test summary on
> > each linux-next release tag for x86_64, i386, arm and arm64 devices.
> > 
> > The next plan is to enable kvm-unit-tests results reporting from LKFT.
> 
> Rather than monthly tags, what about tagging a release for each major
> kernel version?  E.g. for v5.5, v5.6, etc...  That way the compatibility
> is embedded in the tag itself, i.e. there's no need to cross reference
> release dates against kernel/KVM releases to figure out why version of
> kvm-unit-tests should be run.
> 
> Paolo more or less agreed to the idea[*], it's just never been implemented.
> 
> [*] https://lkml.kernel.org/r/dc5ff4ed-c6dd-74ea-03ae-4f65c5d58073@redhat.com

The behavior of kvm in LTS kernels will change over time. In general, as
I wrote in that original thread as well, we would much prefer to use the
latest version of kvm-unit-tests against older kernels.

I think this is a valid example (right?): v4.19 (Oct 22 2018) until now
shows 245 kvm-related patches backported:

    $ git log --oneline v4.19..v4.19.106 | grep -i kvm | wc -l
    245

Just for curiosity I took a look at patch counts per recent releases,
and it seems to average around 250 or so. This means that a 6-year
extended LTS kernel branch will likely receive several releases worth of
fixes.

    $ git log --oneline v5.1..v5.2 | grep -i kvm | wc -l
    238
    $ git log --oneline v5.2..v5.3 | grep -i kvm | wc -l
    239
    $ git log --oneline v5.3..v5.4 | grep -i kvm | wc -l
    246
    $ git log --oneline v5.4..v5.5 | grep -i kvm | wc -l
    172

Indeed, 4.9 has received almost two releases worth of changes since Dec
2016, and is scheduled to still be supported until 2023.

    $ git log --oneline v4.9..v4.9.214 | grep -i kvm | wc -l
    387

We would also like to be able to verify the additional test coverage
that may be added to kvm-unit-tests against older kernels (I assume it's
not all just new features that tests are added for?)

Dan

-- 
Linaro LKFT
https://lkft.linaro.org