Re: [SPDK] Add Jenkins job for periodic (nightly?) test run against upstream DPDK stable?

* Re: [SPDK] Add Jenkins job for periodic (nightly?) test run against upstream DPDK stable?
@ 2019-02-15 21:19 Lance Hartmann ORACLE
  0 siblings, 0 replies; 9+ messages in thread
From: Lance Hartmann ORACLE @ 2019-02-15 21:19 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 10671 bytes --]

> On Feb 15, 2019, at 1:46 PM, Harris, James R <james.r.harris(a)intel.com> wrote:
> 
> Hi Lance,
> 
> In this case, "DPDK master" means the tip of DPDK.  So to summarize, we have three cases here:

And for anyone else who may not be entirely clear on this, by "DPDK master", Jim is referring to the master of upstream DPDK (dpdk.org) -- not our fork of the DPDK we have as a git-submodule.

> 
> 1) SPDK master v. DPDK master
> 2) SPDK master v. DPDK last stable release
> 3) SPDK last stable release v. DPDK master
> 
> Since #3 is stable from an SPDK point of view, nightly tests here are fine.
> 
> I think #1 has to be nightly as well.  Since DPDK master is constantly in flux, we don't want our SPDK patches to get blocked due to DPDK bugs that get checked in between releases.
> 
> It's #2 that's the question.  I talked with Darek about it this morning.  Here are the pros and cons as I see them:
> 
> Per-patch testing:
> Pros:
> * immediately find incompatibilities between DPDK last stable release and DPDK submodule
> Cons:
> * only a subset of SPDK tests run against DPDK last stable release - running all SPDK tests (or at least that don't have known current incompatibilities) against both DPDK last stable release and DPDK submodule would require doubling the CI systems
> 
> Nightly testing:
> Pros:
> * we run all SPDK tests against DPDK last stable release
> Cons:
> * patches could potentially get merged to master which expose an incompatibility between DPDK stable and the DPDK submodule
> 
> I think to start, we get the nightly testing in place ASAP.  If a nightly test fails it's a critical issue that likely leads to an immediate revert.  We can still add per-patch testing on top of it - maybe on just a very limited number of SPDK tests - but we'll still need to rely on the nightly testing for full coverage.  I'd be interested in hearing other opinions on this.

So the risk on nightly-testing for #2 is the scenario that a submitted patch passed against the DPDK submodule, was quickly reviewed and then merged before we would find out later -- overnight -- that the same patch against the DPDK last stable did not.   I don't know how often the case exists where a newly submitted patch runs through the test-pool, gets two maintainer's reviews and merged all in a single day.   That said, do you fear that by always holding off one day before doing a merge, we could inspect the overnight test against DPDK stable first?

I'd also like to pose my question again with respect to "DPDK last stable".   Right now as I compose this per http://core.dpdk.org/download/, they identify "Latest Stable" as 18.05.1 (which I find surprising and wonder if that's correct).   They identify "Latest Major" as 19.02 (released 2019, February 1st), and the release prior to that was DPDK 18.11.0 (LTS).    So, given those three (3) versions of the upstream DPDK, which of them aligns with our use of the term "DPDK last stable release"?  

--
Lance Hartmann

> 
> -Jim
> 
> 
> On 2/15/19, 11:25 AM, "Lance Hartmann ORACLE" <lance.hartmann(a)oracle.com> wrote:
> 
> 
>    Responses interlined below:
> 
> 
> 
>> On Feb 15, 2019, at 8:26 AM, Latecki, Karol <karol.latecki(a)intel.com> wrote:
>> 
>> Hi everyone,
>> We had a talk about this here and we have following proposal about testing vs DPDK.
>> We'd like to start with 3 nightly job configurations, with e-mail notifications enabled:
> 
> 
>    Just nightly?   I think Jim had initially expressed an interest in doing a run per-patch I believe with the hope of identifying and addressing any issues more quickly.   I'll be eager to hear his (and others') feedback on this.
> 
> 
>> * SPDK master vs DPDK master
> 
>    When you write "DPDK master" above -- you mean the SPDK's DPDK submodule?   If so, maybe we might refer to that as "sDPDK".
> 
>> * SPDK master vs DPDK last stable release
> 
>    Again, clarification:  By "DPDK last stable release", you certainly mean the *upstream* DPDK's most recently cut stable release (suggested notation, "uDPDK").    However, a new uDPDK stable release is cut one month after each SPDK release (based on the current respective release schedules).   So would someone (or some automated task?) poll for a newly cut uDPDK stable and then change the Jenkins job so it started building against that? 
> 
> 
> 
>> * SPDK last stable release vs DPDK master
> 
> 
>    By "DPDK master", you mean the current master for the SPDK's DPDK submodule (sDPDK)?
> 
>> That way we'd know fast enough about new issues between SPDK and DPDK. 
>> 
>> I think that your point Jim still applies for these configurations - we'd have to configure them not to test things that we know are failing. That also means that we would have to re-evaluate the configs from time to time.
> 
> 
>    --
>    Lance Hartmann
> 
> 
> 
>> 
>> Karol
>> 
>> -----Original Message-----
>> From: Harris, James R 
>> Sent: Thursday, February 14, 2019 3:32 PM
>> To: Storage Performance Development Kit <spdk(a)lists.01.org>; Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com>; Latecki, Karol <karol.latecki(a)intel.com>
>> Subject: Re: [SPDK] Add Jenkins job for periodic (nightly?) test run against upstream DPDK stable?
>> 
>> We could also just switch some of our existing jobs to build/run against the latest stable DPDK release.  We can be careful about picking jobs that don't have dependencies on patches from our DPDK submodule.
>> 
>> On 2/14/19, 6:39 AM, "SPDK on behalf of Luse, Paul E" <spdk-bounces(a)lists.01.org on behalf of paul.e.luse(a)intel.com> wrote:
>> 
>>   So we talked a lot about this in last night's  community meeting and everyone agreed as a first step we need to:
>> 
>>   * create new Jenkins subjob for per-patch testing that tests against last stable DPDK release
>>   * configure the job to not test things that we know will fail (like crypto due to a patch we pushed upstream that won't land until 19.05).  There may be others like this well (some secondary process tests, etc) so it may take a little experimentation and investigation to get the job right.
>>   * We'll need to review job settings at each release to enable/disable features to test based on the contents of both the SPDK and DPDK releases at the time.
>> 
>>   Darek/Karol, can you guys get started on this and keep everyone posted?
>> 
>>   Thanks! 
>>   Paul
>> 
>>   PS: Thanks Lance for driving this discussion
>> 
>>   -----Original Message-----
>>   From: Luse, Paul E 
>>   Sent: Tuesday, February 12, 2019 5:10 AM
>>   To: Storage Performance Development Kit <spdk(a)lists.01.org>
>>   Subject: RE: [SPDK] Add Jenkins job for periodic (nightly?) test run against upstream DPDK stable?
>> 
>>   I agree that would be a great addition! For some reason I thought that was already being done somewhere by someone, no matter though, setting up a Jenkins job is the right way to do it now. Karol, can you take a look at this and let everyone know your thoughts?
>> 
>>   Thanks
>>   Paul
>> 
>>   -----Original Message-----
>>   From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Lance Hartmann ORACLE
>>   Sent: Monday, February 11, 2019 4:44 PM
>>   To: spdk(a)lists.01.org
>>   Subject: [SPDK] Add Jenkins job for periodic (nightly?) test run against upstream DPDK stable?
>> 
>> 
>>   Group,
>> 
>>   I recently had a sideband chat with Jim regarding our SPDK testing with the DPDK, and he urged I open this up to the email list for more discussion.
>> 
>>   When a Linux distro produces SPDK package rpms, they satisfy the DPDK dependency via a specified stable DPDK release from upstream that has been packaged. However, on the run up to each SPDK quarterly release, we are constantly building and testing against the SPDK's fork'd copy of the DPDK (as a git-submodule) on which we often (virtually always?) have our own patches.   While this works fine for a development perspective as we work toward the next SPDK release, it's problematic when the time arrives to produce new SPDK packages because they will be built against a pristine DPDK stable release which very likely is not identical with what we have been testing against.   When I described this with Jim, he agreed that it would be an excellent idea to augment our SPDK Jenkins setup with some tests which would run against the latest "pristine" (i.e. upstream stable) DPDK release.   With that in place, when the time arrives to produce new SPDK packages, we will have gained the testing c
>>    onfidence of having validated running with the exact version of the DPDK against which the SPDK packages will have been built (and will run alongside at runtime).
>> 
>>   Having said all that, I did float another idea on how we could, if we really needed to, produce SPDK package rpms based on the combination of a stable DPDK upstream release and our own collection of patches to it.   The rpm packaging system natively supports such a build, but as you might quickly deduce it comes at the cost of additional maintenance efforts which can become very unpalatable, esp. when/if some of our patches are never accepted/merged into the official upstream DPDK tree.
>> 
>> 
>>   --
>>   Lance Hartmann
>> 
>> 
>> 
>>   _______________________________________________
>>   SPDK mailing list
>>   SPDK(a)lists.01.org
>>   https://lists.01.org/mailman/listinfo/spdk
>>   _______________________________________________
>>   SPDK mailing list
>>   SPDK(a)lists.01.org
>>   https://lists.01.org/mailman/listinfo/spdk
>> 
>> 
>> --------------------------------------------------------------------
>> 
>> Intel Technology Poland sp. z o.o.
>> ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
>> 
>> Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
>> przegladanie lub rozpowszechnianie jest zabronione.
>> This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
>> others is strictly prohibited.
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
> 
> 
> 

^ permalink raw reply	[flat|nested] 9+ messages in thread