Re: Regression tests for stable releases from companies involved in DPDK

[Luca Boccassi <bluca@debian.org>]

On Fri, 2018-06-01 at 10:17 +0200, Marco Varlese wrote:
> Hi Luca,
> 
> On Thu, 2018-05-31 at 11:26 +0100, Luca Boccassi wrote:
> > Hello all,
> > 
> > At this morning's release meeting (minutes coming soon from John),
> > we
> > briefly discussed the state of the regression testing for stable
> > releases and agreed we need to formalise the process.
> > 
> > At the moment we have a firm commitment from Intel and Mellanox to
> > test
> > all stable branches (and if I heard correctly from NXP as well?
> > Please
> > confirm!). AT&T committed to run regressions on the 16.11 branch.
> > 
> > Here's what we need in order to improve the quality of the stable
> > releases process:
> > 
> > 1) More commitments to help from other companies involved in the
> > DPDK
> > community. At the cost of re-stating the obvious, improving the
> > quality
> > of stable releases is for everyone's benefit, as a lot of customers
> > and
> > projects rely on the stable or LTS releases for their production
> > environments.
> 
> Do you have a list of steps (test-cases?) which are carried out for
> stable
> regression? I think it is necessary in order to understand the effort
> involved
> before committing to it.

This is left intentionally vague - apart from the unit tests, we don't
have a public formal "production" test suite, so each company has its
own, that in general is tailored to test their own product.

For example, AT&T builds a virtual router, and the regression tests
cover the functionality of that product that are implemented via DPDK.

I imagine companies developing NICs and PMDs will have regression tests
that exercise those network cards in production. Likewise an enterprise
distribution like Ubuntu can test their build of OVS, for example.

So what we are asking is, for those companies/groups that use DPDK, if
they can help us test that the stable candidate releases are working
correctly with their products/hardware/projects. The hope is that among
everybody there is enough coverage of various functionalities and PMDs.
I'd love to have a formalised, public, all-encompassing test suite
(something that was discussed in Dublin the year before last I believe)
but we have to work with what we've got.

Does this make sense?

> > 
> > 2) A formalised deadline - the current proposal is 10 days from the
> > "xx.yy patches review and test" email, which was just sent for
> > 16.11.
> > For the involved companies, please let us know if 10 days is
> > enough. In
> > terms of scheduling, this period will always start within a week
> > from
> > the mainline final release. Again, the signal is the "xx.yy patches
> > review and test" appearing in the inbox, which will detail the
> > deadline.
> 
> Again, I believe it depends on what needs to be tested (and how) in
> order to
> comment on "how much time is required".

See above - it will vary from stakeholder to stakeholder, and that's
why I asked if that timeframe is workable.

-- 
Kind regards,
Luca Boccassi