[SPDK] Long test times and Python interpreter

[SPDK] Long test times and Python interpreter

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 2010 bytes --]

I’ve been digging into the NVMe-oF target tests today – starting with looking at how long it takes to run all of the per-patch tests.  I’d like to consider a new strategy for using rpc.py from our test scripts to reduce the number of times we start the Python interpreter.  There’s a non-trivial amount of overhead just related to starting the interpreter – and tests like nvmf/shutdown do 30 RPCs (10 subsystems * 3 RPCs to set up each subsystem).  If it’s only 100ms overhead per RPC, that’s 3 seconds wasted*.

There are two approaches I can see:


  1.  Save the configuration in JSON RPC format, and check in the file into the test directory.  Then we just have a single RPC to call (load_config).
  2.  Add a new top level function to rpc.py called “run_from_file”.  This is not an RPC – it just loads a file, and runs each line as if it is was its own rpc.py invocation.
     *   A variation could also be piping this file through stdin

I think #2 is probably better.  At least for nvmf/shutdown, we use a different number of subsystems based on whether we’re testing with a real RDMA NIC or with soft-roce.  In those cases, we’d need two separate JSON RPC format files.

This would also be helpful for adding more stressful tests – for example, create 1000 subsystems or 1000 logical volumes.  The SPDK application itself can do this pretty quickly, but I think our current strategy for actually building that config via RPC would force it be run only nightly due to length of time.

I’ll probably try putting something together tomorrow, but wanted to see if anyone else had any additional thoughts or comments.

Thanks,

-Jim

* I know 3 seconds doesn’t sound like much.  It’s probably more like 5 or 6 seconds in this case.  We probably run about 8 patches per hour through the test pool – so every 7-8 seconds saved is a 1% improvement in test pool throughput.  There are several other places we could use this – especially vhost and iscsi.

Re: [SPDK] Long test times and Python interpreter

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 3764 bytes --]

Hi Pawel,

The rpc.py changes have been made and are merged to master.  We ended up doing a slight variation on #2 - just pipe the 'script' into rpc.py instead of an explicit top-level function.  The nvmf shutdown tests are now using this functionality too to speed up the execution time - see test/nvmf/target/shutdown.sh.

There are lots of opportunities to use this elsewhere in our tests across SPDK though.

Thanks,

-Jim


On 5/7/19, 6:31 AM, "SPDK on behalf of Kaminski, PawelX" <spdk-bounces(a)lists.01.org on behalf of pawelx.kaminski(a)intel.com> wrote:

    Hi Jim,
    
    I can work on this. I could try second solution and check the tests execution time.
    I think we could also try to create multiline rpc command similar to test/spdkcli/spdkcli_job.py.
    For this script we can pass many commands as one argument.
    
    Pawel
    
    > -----Original Message-----
    > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
    > Sent: Wednesday, May 1, 2019 2:44 AM
    > To: Storage Performance Development Kit <spdk(a)lists.01.org>
    > Subject: [SPDK] Long test times and Python interpreter
    > 
    > I’ve been digging into the NVMe-oF target tests today – starting with looking
    > at how long it takes to run all of the per-patch tests.  I’d like to consider a
    > new strategy for using rpc.py from our test scripts to reduce the number of
    > times we start the Python interpreter.  There’s a non-trivial amount of
    > overhead just related to starting the interpreter – and tests like
    > nvmf/shutdown do 30 RPCs (10 subsystems * 3 RPCs to set up each
    > subsystem).  If it’s only 100ms overhead per RPC, that’s 3 seconds wasted*.
    > 
    > There are two approaches I can see:
    > 
    > 
    >   1.  Save the configuration in JSON RPC format, and check in the file into the
    > test directory.  Then we just have a single RPC to call (load_config).
    >   2.  Add a new top level function to rpc.py called “run_from_file”.  This is not
    > an RPC – it just loads a file, and runs each line as if it is was its own rpc.py
    > invocation.
    >      *   A variation could also be piping this file through stdin
    > 
    > I think #2 is probably better.  At least for nvmf/shutdown, we use a different
    > number of subsystems based on whether we’re testing with a real RDMA
    > NIC or with soft-roce.  In those cases, we’d need two separate JSON RPC
    > format files.
    > 
    > This would also be helpful for adding more stressful tests – for example,
    > create 1000 subsystems or 1000 logical volumes.  The SPDK application itself
    > can do this pretty quickly, but I think our current strategy for actually building
    > that config via RPC would force it be run only nightly due to length of time.
    > 
    > I’ll probably try putting something together tomorrow, but wanted to see if
    > anyone else had any additional thoughts or comments.
    > 
    > Thanks,
    > 
    > -Jim
    > 
    > * I know 3 seconds doesn’t sound like much.  It’s probably more like 5 or 6
    > seconds in this case.  We probably run about 8 patches per hour through the
    > test pool – so every 7-8 seconds saved is a 1% improvement in test pool
    > throughput.  There are several other places we could use this – especially
    > vhost and iscsi.
    > _______________________________________________
    > SPDK mailing list
    > SPDK(a)lists.01.org
    > https://lists.01.org/mailman/listinfo/spdk
    _______________________________________________
    SPDK mailing list
    SPDK(a)lists.01.org
    https://lists.01.org/mailman/listinfo/spdk
    

Re: [SPDK] Long test times and Python interpreter

[Kaminski, PawelX]

[-- Attachment #1: Type: text/plain, Size: 2794 bytes --]

Hi Jim,

I can work on this. I could try second solution and check the tests execution time.
I think we could also try to create multiline rpc command similar to test/spdkcli/spdkcli_job.py.
For this script we can pass many commands as one argument.

Pawel

> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
> Sent: Wednesday, May 1, 2019 2:44 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Long test times and Python interpreter
> 
> I’ve been digging into the NVMe-oF target tests today – starting with looking
> at how long it takes to run all of the per-patch tests.  I’d like to consider a
> new strategy for using rpc.py from our test scripts to reduce the number of
> times we start the Python interpreter.  There’s a non-trivial amount of
> overhead just related to starting the interpreter – and tests like
> nvmf/shutdown do 30 RPCs (10 subsystems * 3 RPCs to set up each
> subsystem).  If it’s only 100ms overhead per RPC, that’s 3 seconds wasted*.
> 
> There are two approaches I can see:
> 
> 
>   1.  Save the configuration in JSON RPC format, and check in the file into the
> test directory.  Then we just have a single RPC to call (load_config).
>   2.  Add a new top level function to rpc.py called “run_from_file”.  This is not
> an RPC – it just loads a file, and runs each line as if it is was its own rpc.py
> invocation.
>      *   A variation could also be piping this file through stdin
> 
> I think #2 is probably better.  At least for nvmf/shutdown, we use a different
> number of subsystems based on whether we’re testing with a real RDMA
> NIC or with soft-roce.  In those cases, we’d need two separate JSON RPC
> format files.
> 
> This would also be helpful for adding more stressful tests – for example,
> create 1000 subsystems or 1000 logical volumes.  The SPDK application itself
> can do this pretty quickly, but I think our current strategy for actually building
> that config via RPC would force it be run only nightly due to length of time.
> 
> I’ll probably try putting something together tomorrow, but wanted to see if
> anyone else had any additional thoughts or comments.
> 
> Thanks,
> 
> -Jim
> 
> * I know 3 seconds doesn’t sound like much.  It’s probably more like 5 or 6
> seconds in this case.  We probably run about 8 patches per hour through the
> test pool – so every 7-8 seconds saved is a 1% improvement in test pool
> throughput.  There are several other places we could use this – especially
> vhost and iscsi.
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk