From mboxrd@z Thu Jan 1 00:00:00 1970 From: Marco Varlese Subject: Re: Regression tests for stable releases from companies involved in DPDK Date: Fri, 01 Jun 2018 13:04:14 +0200 Message-ID: References: <1527762399.6997.44.camel@debian.org> <1527847000.6997.66.camel@debian.org> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit Cc: stable@dpdk.org, thomas@monjalon.net To: Luca Boccassi , dev@dpdk.org Return-path: In-Reply-To: <1527847000.6997.66.camel@debian.org> List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Fri, 2018-06-01 at 10:56 +0100, Luca Boccassi wrote: > On Fri, 2018-06-01 at 10:17 +0200, Marco Varlese wrote: > > Hi Luca, > > > > On Thu, 2018-05-31 at 11:26 +0100, Luca Boccassi wrote: > > > Hello all, > > > > > > At this morning's release meeting (minutes coming soon from John), > > > we > > > briefly discussed the state of the regression testing for stable > > > releases and agreed we need to formalise the process. > > > > > > At the moment we have a firm commitment from Intel and Mellanox to > > > test > > > all stable branches (and if I heard correctly from NXP as well? > > > Please > > > confirm!). AT&T committed to run regressions on the 16.11 branch. > > > > > > Here's what we need in order to improve the quality of the stable > > > releases process: > > > > > > 1) More commitments to help from other companies involved in the > > > DPDK > > > community. At the cost of re-stating the obvious, improving the > > > quality > > > of stable releases is for everyone's benefit, as a lot of customers > > > and > > > projects rely on the stable or LTS releases for their production > > > environments. > > > > Do you have a list of steps (test-cases?) which are carried out for > > stable > > regression? I think it is necessary in order to understand the effort > > involved > > before committing to it. > > This is left intentionally vague - apart from the unit tests, we don't > have a public formal "production" test suite, so each company has its > own, that in general is tailored to test their own product. > > For example, AT&T builds a virtual router, and the regression tests > cover the functionality of that product that are implemented via DPDK. > > I imagine companies developing NICs and PMDs will have regression tests > that exercise those network cards in production. Likewise an enterprise > distribution like Ubuntu can test their build of OVS, for example. > > So what we are asking is, for those companies/groups that use DPDK, if > they can help us test that the stable candidate releases are working > correctly with their products/hardware/projects. The hope is that among > everybody there is enough coverage of various functionalities and PMDs. > I'd love to have a formalised, public, all-encompassing test suite > (something that was discussed in Dublin the year before last I believe) > but we have to work with what we've got. > > Does this make sense? Thanks for taking the time to explain it. Now, it is much clearer in my mind. > > > > > > > 2) A formalised deadline - the current proposal is 10 days from the > > > "xx.yy patches review and test" email, which was just sent for > > > 16.11. > > > For the involved companies, please let us know if 10 days is > > > enough. In > > > terms of scheduling, this period will always start within a week > > > from > > > the mainline final release. Again, the signal is the "xx.yy patches > > > review and test" appearing in the inbox, which will detail the > > > deadline. > > > > Again, I believe it depends on what needs to be tested (and how) in > > order to > > comment on "how much time is required". > > See above - it will vary from stakeholder to stakeholder, and that's > why I asked if that timeframe is workable. Undestood. > Cheers, -- Marco V SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg