Re: [SPDK] For the Chandler testing pool

[Harris, James R <james.r.harris at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 6187 bytes --]

The Chandler test pool is definitely slower than the Jenkins pool.  Jenkins can do two jobs in parallel (mostly) and will also abort early if a patch fails basic compilation and unit tests.  It seems Jenkins runs about 10 minutes per patch while the Chandler pool is about 15 minutes.  (12 minutes test time plus 3 minutes overhead to copy results, etc.)

I don't see a public version of https://ci.spdk.io/spdk/status/ for the Jenkins test pool though on http://spdk.io/ci.  Is there a public version of that available?  All of the Jenkins data I'm pulling is from the internal (to Intel) version of the status page.

Something that everyone can help with is looking at how to speed up the tests.  Each successful test run includes a flamegraph.  For example:

1) Look at a patch that has run through the test pool.  Here's one from today from the Chandler test pool:  http://spdk.intel.com/public/spdk/builds/review/171498ede29337a2fd52ad13c6ff613cff3bf13b.1543288927/.

2) Find which system took the longest to execute.  On the Chandler test pool, this is consistently fedora-03 (12+ minutes).  Since the Chandler CI doesn't run jobs in parallel, you'll only save time by shortening the longest test.  Here's the results for fedora-03 for that patch.  http://spdk.intel.com/public/spdk/builds/review/171498ede29337a2fd52ad13c6ff613cff3bf13b.1543288927/fedora-03/.

3) Open the flamegraph (the SVG file).  http://spdk.intel.com/public/spdk/builds/review/171498ede29337a2fd52ad13c6ff613cff3bf13b.1543288927/fedora-03/timing.svg

4)  Look where the time is being spent - see for areas that can be optimized, etc.

Thanks,

-Jim





On 11/26/18, 7:14 PM, "SPDK on behalf of Luse, Paul E" <spdk-bounces(a)lists.01.org on behalf of paul.e.luse(a)intel.com> wrote:

    Ok thanks. Agree on the requirement. Let’s figure out how to measure throughout time for you with Jenkins. If that’s still too slow we can expand the pool and/or look at some other options maybe including some local test systems. We need to wrap up the Jenkins tradition first though and your input helps define some real need around getting it done sooner than later so thanks.  
    
    Karol curious for your input on getting publicalky available  throughout times made available. I don’t mean exactly like the ch pool but maybe something on the status page after the test is done that shows submission and completion times
    
    -from my iPhone 
    
    > On Nov 26, 2018, at 7:02 PM, Yang, Ziye <ziye.yang(a)intel.com> wrote:
    > 
    > Hi Paul,
    > 
    > For the wait times in the Jenkins test pool, I cannot have the estimation since currently I cannot see the list of patches and also the execution time statistics. My request is that: Either what kinds of test pool use,  it should be friendly to users.  Since we are in different time zones,  I hope that we will not wait so much time for the results from the test pool in our own time zone, thus it is not efficient. 
    > 
    > Thanks.
    > 
    > 
    > 
    > 
    > Best Regards
    > Ziye Yang 
    > 
    > -----Original Message-----
    > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
    > Sent: Tuesday, November 27, 2018 9:56 AM
    > To: Storage Performance Development Kit <spdk(a)lists.01.org>
    > Subject: Re: [SPDK] For the Chandler testing pool
    > 
    > Hi Ziye,
    > 
    > That's a good observation. Can I ask real quick what your wait times typically look like with the Jenkins test pool?
    > 
    > One of the reasons for the transition to Janina is to improve scalability and I'm thinking the way it was implemented throughout time should already be better but I'd like to hear your perspective. 
    > 
    > We are going to retire the chandler pool soon once the final Jenkins issues are worked out. We have not been in a big hurry to make the move though because we don't want to make a mistake and disrupt things for an arbitrary deadline. 
    > 
    > Once this happens though it will be a full replacement. I may not be understanding your question about local testing but for a global fully transparent open community a solid extendible continuous integration system is not optional. Is that what you were asking or were you just wondering why we haven't totally moved to Jenkins yet?
    > 
    > Thx
    > Paul
    > 
    > -from my iPhone 
    > 
    >> On Nov 26, 2018, at 6:43 PM, Yang, Ziye <ziye.yang(a)intel.com> wrote:
    >> 
    >> Hi all,
    >> 
    >> These days, I found that the Chandler testing pool is very busy. When in PRC time zone, there are lots of patches are waiting to be tested. For example today,  from now, I see 27 patches are in the list. According to the testing process, each will waste 12 minutes, it means that the total time will be about 324 minutes,  which means that if you submit a patch this time, you need to wait at least 5.5 hours (after 14:30, you can see the results). Since we are global team, Poland team will nearly start work at 4:00 PM. So it left less time for PRC in the daily time. Usually, I will get the results of my patches after my working time.  So my question is that: Can we reduce this time since currently the testing time is too long?
    >> 
    >> PS: Why we need the Chandler test pool? Since it has different environments, you cannot verify your patch in the single local test environment.
    >> 
    >> Thanks.
    >> 
    >> 
    >> 
    >> 
    >> Best Regards
    >> Ziye Yang
    >> 
    >> _______________________________________________
    >> SPDK mailing list
    >> SPDK(a)lists.01.org
    >> https://lists.01.org/mailman/listinfo/spdk
    > _______________________________________________
    > SPDK mailing list
    > SPDK(a)lists.01.org
    > https://lists.01.org/mailman/listinfo/spdk
    > _______________________________________________
    > SPDK mailing list
    > SPDK(a)lists.01.org
    > https://lists.01.org/mailman/listinfo/spdk
    _______________________________________________
    SPDK mailing list
    SPDK(a)lists.01.org
    https://lists.01.org/mailman/listinfo/spdk