Regression tests for stable releases from companies involved in DPDK

Regression tests for stable releases from companies involved in DPDK

[Luca Boccassi]

Hello all,

At this morning's release meeting (minutes coming soon from John), we
briefly discussed the state of the regression testing for stable
releases and agreed we need to formalise the process.

At the moment we have a firm commitment from Intel and Mellanox to test
all stable branches (and if I heard correctly from NXP as well? Please
confirm!). AT&T committed to run regressions on the 16.11 branch.

Here's what we need in order to improve the quality of the stable
releases process:

1) More commitments to help from other companies involved in the DPDK
community. At the cost of re-stating the obvious, improving the quality
of stable releases is for everyone's benefit, as a lot of customers and
projects rely on the stable or LTS releases for their production
environments.

2) A formalised deadline - the current proposal is 10 days from the
"xx.yy patches review and test" email, which was just sent for 16.11.
For the involved companies, please let us know if 10 days is enough. In
terms of scheduling, this period will always start within a week from
the mainline final release. Again, the signal is the "xx.yy patches
review and test" appearing in the inbox, which will detail the
deadline.

Comments?

-- 
Kind regards,
Luca Boccassi

Re: [dpdk-stable] Regression tests for stable releases from companies involved in DPDK

[Christian Ehrhardt]

On Thu, May 31, 2018 at 12:26 PM, Luca Boccassi <bluca@debian.org> wrote:

> Hello all,
>
> At this morning's release meeting (minutes coming soon from John), we
> briefly discussed the state of the regression testing for stable
> releases and agreed we need to formalise the process.
>
> At the moment we have a firm commitment from Intel and Mellanox to test
> all stable branches (and if I heard correctly from NXP as well? Please
> confirm!). AT&T committed to run regressions on the 16.11 branch.
>
> Here's what we need in order to improve the quality of the stable
> releases process:
>
> 1) More commitments to help from other companies involved in the DPDK
> community. At the cost of re-stating the obvious, improving the quality
> of stable releases is for everyone's benefit, as a lot of customers and
> projects rely on the stable or LTS releases for their production
> environments.
>
> 2) A formalised deadline - the current proposal is 10 days from the
> "xx.yy patches review and test" email, which was just sent for 16.11.
> For the involved companies, please let us know if 10 days is enough. In
> terms of scheduling, this period will always start within a week from
> the mainline final release. Again, the signal is the "xx.yy patches
> review and test" appearing in the inbox, which will detail the
> deadline.
>
>
Hi Luca,
I discussed with Thomas about it.
I don't know how much extra effort for the stable maintainers it would be,
but I wonder if there could be a XX.YY.z-rc tarball.
That would be
a) a more clear sign what people are used to test
b) easier to integrate as I assume quite a bunch of tests will usually
start rebasing on tarballs instead of directly from git.

If you think everyone can derive from git easily I'm fine, I just wondered
if a proper -rc tarball might be more comfortable for the testing entities.

cu
Christian

Re: Regression tests for stable releases from companies involved in DPDK

[Marco Varlese]

Hi Luca,

On Thu, 2018-05-31 at 11:26 +0100, Luca Boccassi wrote:
> Hello all,
> 
> At this morning's release meeting (minutes coming soon from John), we
> briefly discussed the state of the regression testing for stable
> releases and agreed we need to formalise the process.
> 
> At the moment we have a firm commitment from Intel and Mellanox to test
> all stable branches (and if I heard correctly from NXP as well? Please
> confirm!). AT&T committed to run regressions on the 16.11 branch.
> 
> Here's what we need in order to improve the quality of the stable
> releases process:
> 
> 1) More commitments to help from other companies involved in the DPDK
> community. At the cost of re-stating the obvious, improving the quality
> of stable releases is for everyone's benefit, as a lot of customers and
> projects rely on the stable or LTS releases for their production
> environments.

Do you have a list of steps (test-cases?) which are carried out for stable
regression? I think it is necessary in order to understand the effort involved
before committing to it.

> 
> 2) A formalised deadline - the current proposal is 10 days from the
> "xx.yy patches review and test" email, which was just sent for 16.11.
> For the involved companies, please let us know if 10 days is enough. In
> terms of scheduling, this period will always start within a week from
> the mainline final release. Again, the signal is the "xx.yy patches
> review and test" appearing in the inbox, which will detail the
> deadline.

Again, I believe it depends on what needs to be tested (and how) in order to
comment on "how much time is required".

> 
> Comments?
> 
Regards,
-- 
Marco V

SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg

Re: Regression tests for stable releases from companies involved in DPDK

[Luca Boccassi]

On Fri, 2018-06-01 at 10:17 +0200, Marco Varlese wrote:
> Hi Luca,
> 
> On Thu, 2018-05-31 at 11:26 +0100, Luca Boccassi wrote:
> > Hello all,
> > 
> > At this morning's release meeting (minutes coming soon from John),
> > we
> > briefly discussed the state of the regression testing for stable
> > releases and agreed we need to formalise the process.
> > 
> > At the moment we have a firm commitment from Intel and Mellanox to
> > test
> > all stable branches (and if I heard correctly from NXP as well?
> > Please
> > confirm!). AT&T committed to run regressions on the 16.11 branch.
> > 
> > Here's what we need in order to improve the quality of the stable
> > releases process:
> > 
> > 1) More commitments to help from other companies involved in the
> > DPDK
> > community. At the cost of re-stating the obvious, improving the
> > quality
> > of stable releases is for everyone's benefit, as a lot of customers
> > and
> > projects rely on the stable or LTS releases for their production
> > environments.
> 
> Do you have a list of steps (test-cases?) which are carried out for
> stable
> regression? I think it is necessary in order to understand the effort
> involved
> before committing to it.

This is left intentionally vague - apart from the unit tests, we don't
have a public formal "production" test suite, so each company has its
own, that in general is tailored to test their own product.

For example, AT&T builds a virtual router, and the regression tests
cover the functionality of that product that are implemented via DPDK.

I imagine companies developing NICs and PMDs will have regression tests
that exercise those network cards in production. Likewise an enterprise
distribution like Ubuntu can test their build of OVS, for example.

So what we are asking is, for those companies/groups that use DPDK, if
they can help us test that the stable candidate releases are working
correctly with their products/hardware/projects. The hope is that among
everybody there is enough coverage of various functionalities and PMDs.
I'd love to have a formalised, public, all-encompassing test suite
(something that was discussed in Dublin the year before last I believe)
but we have to work with what we've got.

Does this make sense?

> > 
> > 2) A formalised deadline - the current proposal is 10 days from the
> > "xx.yy patches review and test" email, which was just sent for
> > 16.11.
> > For the involved companies, please let us know if 10 days is
> > enough. In
> > terms of scheduling, this period will always start within a week
> > from
> > the mainline final release. Again, the signal is the "xx.yy patches
> > review and test" appearing in the inbox, which will detail the
> > deadline.
> 
> Again, I believe it depends on what needs to be tested (and how) in
> order to
> comment on "how much time is required".

See above - it will vary from stakeholder to stakeholder, and that's
why I asked if that timeframe is workable.

-- 
Kind regards,
Luca Boccassi

Re: [dpdk-stable] Regression tests for stable releases from companies involved in DPDK

[Luca Boccassi]

On Fri, 2018-06-01 at 06:38 +0200, Christian Ehrhardt wrote:
> On Thu, May 31, 2018 at 12:26 PM, Luca Boccassi <bluca@debian.org>
> wrote:
> 
> > Hello all,
> > 
> > At this morning's release meeting (minutes coming soon from John),
> > we
> > briefly discussed the state of the regression testing for stable
> > releases and agreed we need to formalise the process.
> > 
> > At the moment we have a firm commitment from Intel and Mellanox to
> > test
> > all stable branches (and if I heard correctly from NXP as well?
> > Please
> > confirm!). AT&T committed to run regressions on the 16.11 branch.
> > 
> > Here's what we need in order to improve the quality of the stable
> > releases process:
> > 
> > 1) More commitments to help from other companies involved in the
> > DPDK
> > community. At the cost of re-stating the obvious, improving the
> > quality
> > of stable releases is for everyone's benefit, as a lot of customers
> > and
> > projects rely on the stable or LTS releases for their production
> > environments.
> > 
> > 2) A formalised deadline - the current proposal is 10 days from the
> > "xx.yy patches review and test" email, which was just sent for
> > 16.11.
> > For the involved companies, please let us know if 10 days is
> > enough. In
> > terms of scheduling, this period will always start within a week
> > from
> > the mainline final release. Again, the signal is the "xx.yy patches
> > review and test" appearing in the inbox, which will detail the
> > deadline.
> > 
> > 
> 
> Hi Luca,
> I discussed with Thomas about it.
> I don't know how much extra effort for the stable maintainers it
> would be,
> but I wonder if there could be a XX.YY.z-rc tarball.
> That would be
> a) a more clear sign what people are used to test
> b) easier to integrate as I assume quite a bunch of tests will
> usually
> start rebasing on tarballs instead of directly from git.
> 
> If you think everyone can derive from git easily I'm fine, I just
> wondered
> if a proper -rc tarball might be more comfortable for the testing
> entities.
> 
> cu
> Christian

I think that's a good idea, and something we can consider for the next
release cycle - the tools to push rc to mainline should work just the
same for the stable repo.

-- 
Kind regards,
Luca Boccassi

Re: Regression tests for stable releases from companies involved in DPDK

[Marco Varlese]

On Fri, 2018-06-01 at 10:56 +0100, Luca Boccassi wrote:
> On Fri, 2018-06-01 at 10:17 +0200, Marco Varlese wrote:
> > Hi Luca,
> > 
> > On Thu, 2018-05-31 at 11:26 +0100, Luca Boccassi wrote:
> > > Hello all,
> > > 
> > > At this morning's release meeting (minutes coming soon from John),
> > > we
> > > briefly discussed the state of the regression testing for stable
> > > releases and agreed we need to formalise the process.
> > > 
> > > At the moment we have a firm commitment from Intel and Mellanox to
> > > test
> > > all stable branches (and if I heard correctly from NXP as well?
> > > Please
> > > confirm!). AT&T committed to run regressions on the 16.11 branch.
> > > 
> > > Here's what we need in order to improve the quality of the stable
> > > releases process:
> > > 
> > > 1) More commitments to help from other companies involved in the
> > > DPDK
> > > community. At the cost of re-stating the obvious, improving the
> > > quality
> > > of stable releases is for everyone's benefit, as a lot of customers
> > > and
> > > projects rely on the stable or LTS releases for their production
> > > environments.
> > 
> > Do you have a list of steps (test-cases?) which are carried out for
> > stable
> > regression? I think it is necessary in order to understand the effort
> > involved
> > before committing to it.
> 
> This is left intentionally vague - apart from the unit tests, we don't
> have a public formal "production" test suite, so each company has its
> own, that in general is tailored to test their own product.
> 
> For example, AT&T builds a virtual router, and the regression tests
> cover the functionality of that product that are implemented via DPDK.
> 
> I imagine companies developing NICs and PMDs will have regression tests
> that exercise those network cards in production. Likewise an enterprise
> distribution like Ubuntu can test their build of OVS, for example.
> 
> So what we are asking is, for those companies/groups that use DPDK, if
> they can help us test that the stable candidate releases are working
> correctly with their products/hardware/projects. The hope is that among
> everybody there is enough coverage of various functionalities and PMDs.
> I'd love to have a formalised, public, all-encompassing test suite
> (something that was discussed in Dublin the year before last I believe)
> but we have to work with what we've got.
> 
> Does this make sense?
Thanks for taking the time to explain it. Now, it is much clearer in my mind.
> 
> > > 
> > > 2) A formalised deadline - the current proposal is 10 days from the
> > > "xx.yy patches review and test" email, which was just sent for
> > > 16.11.
> > > For the involved companies, please let us know if 10 days is
> > > enough. In
> > > terms of scheduling, this period will always start within a week
> > > from
> > > the mainline final release. Again, the signal is the "xx.yy patches
> > > review and test" appearing in the inbox, which will detail the
> > > deadline.
> > 
> > Again, I believe it depends on what needs to be tested (and how) in
> > order to
> > comment on "how much time is required".
> 
> See above - it will vary from stakeholder to stakeholder, and that's
> why I asked if that timeframe is workable.
Undestood.
> 


Cheers,
-- 
Marco V

SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg

Re: Regression tests for stable releases from companies involved in DPDK

[Shreyansh Jain]

Hello Luca

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Luca Boccassi
> Sent: Thursday, May 31, 2018 3:57 PM
> To: dev@dpdk.org
> Cc: stable@dpdk.org; thomas@monjalon.net
> Subject: [dpdk-dev] Regression tests for stable releases from companies
> involved in DPDK
> 
> Hello all,
> 
> At this morning's release meeting (minutes coming soon from John), we
> briefly discussed the state of the regression testing for stable
> releases and agreed we need to formalise the process.
> 
> At the moment we have a firm commitment from Intel and Mellanox to test
> all stable branches (and if I heard correctly from NXP as well? Please

Yes, I confirmed that on call on behalf of NXP. But...

> confirm!). AT&T committed to run regressions on the 16.11 branch.

Not until 17.05 did NXP's first driver started appearing in the DPDK upstream releases. Somehow, I misunderstood your request for 16.11 with 17.11 stable.

[...]

-
Shreyansh

Re: [dpdk-stable] Regression tests for stable releases from companies involved in DPDK

[Luca Boccassi]

On Mon, 2018-06-04 at 05:24 +0000, Shreyansh Jain wrote:
> Hello Luca
> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Luca Boccassi
> > Sent: Thursday, May 31, 2018 3:57 PM
> > To: dev@dpdk.org
> > Cc: stable@dpdk.org; thomas@monjalon.net
> > Subject: [dpdk-dev] Regression tests for stable releases from
> > companies
> > involved in DPDK
> > 
> > Hello all,
> > 
> > At this morning's release meeting (minutes coming soon from John),
> > we
> > briefly discussed the state of the regression testing for stable
> > releases and agreed we need to formalise the process.
> > 
> > At the moment we have a firm commitment from Intel and Mellanox to
> > test
> > all stable branches (and if I heard correctly from NXP as well?
> > Please
> 
> Yes, I confirmed that on call on behalf of NXP. But...
> 
> > confirm!). AT&T committed to run regressions on the 16.11 branch.
> 
> Not until 17.05 did NXP's first driver started appearing in the DPDK
> upstream releases. Somehow, I misunderstood your request for 16.11
> with 17.11 stable.

That's fine - the 17.11 branch is also getting ready for release (see
relevant threads), and 18.02 will soon as well.

-- 
Kind regards,
Luca Boccassi