Re: [SPDK] Multiple start/stop of reactor and add/remove of bdev

[Harris, James R <james.r.harris at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 4251 bytes --]

Hi Michael,

The reactors definitely have a cost if they are polling with no real work to do.

Ben Walker has been working on a patch series [1] that would effectively decouple and “spdk_thread” from an “spdk_reactor”.  Currently the relationship is always one-to-one.  Vishal Verma pushed some patches earlier this year [2] that track how much real work a reactor is doing by tracking whether a poller did any real work.  Combining these two would enable dynamic load balancing.  It sounds like that might meet the vhost use case you’ve described.

This is why I’m interested in understanding more about Shahar’s request to start and stop reactors and reinitialize the bdev layer, and whether it is also based on a desire to do this kind of load balancing.

Thanks,

-Jim

[1] Patch series starts here: https://review.gerrithub.io/#/c/spdk/spdk/+/417784/
[2] Main patch is: https://review.gerrithub.io/#/c/spdk/spdk/+/412695/

On 10/8/18, 12:24 PM, "SPDK on behalf of Michael Haeuptle" <spdk-bounces(a)lists.01.org on behalf of michaelhaeuptle(a)gmail.com> wrote:

    I can't comment on Shahar's use case but it would be helpful in cases where
    I want to flex out reactors based on the # of devices which are not fixed.
    
    In my use case, I don't really want to have reactors running if I don't
    need them since they do incur some cost as far as I understand it (maybe
    max_delay_us could be used).
    
    For example, it may be good enough to have 1 reactor serving 2
    vhost_scsi_controllers starting out. However, if I need to add more
    controllers (since I want to add more devices), then I'd like to start
    another reactor for the new vhost_scsi_controllers. Reducing controllers is
    tricky but I'm not sure this happens very often in my use case.
    
    -- Michael
    
    
    
    On Mon, Oct 8, 2018 at 11:00 AM Harris, James R <james.r.harris(a)intel.com>
    wrote:
    
    > Hi Shahar,
    >
    > Can you describe why you require starting/stopping the reactor on demand?
    >
    > Thanks,
    >
    > -Jim
    >
    >
    >
    >
    > On 10/8/18, 1:07 AM, "SPDK on behalf of Shahar Salzman" <
    > spdk-bounces(a)lists.01.org on behalf of shahar.salzman(a)kaminario.com>
    > wrote:
    >
    >     Hi,
    >
    >
    >     I am now working with my passthrough bdev directly from my
    > application, and require adding/removing them dynamically. In addition, I
    > also require starting/stopping the reactor on demand without tearing down
    > the entire system (e.g. dpdk stays initialized). This mostly works out of
    > the box, but there are a few globals and bdev internal fields which require
    > re-init or tear down.
    >
    >     In addition, the copy engine did not io_unregister itself, so I also
    > added this in the patchset, allowing me the following application life span:
    >
    >
    >     - env init (init config, spdk_env_init)
    >
    >     - reactors init
    >
    >     - while (application is alive)
    >
    >       subsystem_init + reactor start
    >
    >       ...
    >
    >       Do some NVMeF stuff
    >
    >       ...
    >
    >       subsysem_fini + reactors stop
    >
    >     - reactors_fini
    >
    >     - rpc_finish
    >
    >
    >
    >     I submitted the following patches for review:
    >
    >     https://review.gerrithub.io/c/spdk/spdk/+/428305
    >
    >     https://review.gerrithub.io/c/spdk/spdk/+/428304
    >
    >     https://review.gerrithub.io/c/spdk/spdk/+/428303
    >
    >     https://review.gerrithub.io/c/spdk/spdk/+/428302
    >
    >
    >     WDYT?
    >
    >
    >     Shahar
    >     _______________________________________________
    >     SPDK mailing list
    >     SPDK(a)lists.01.org
    >     https://lists.01.org/mailman/listinfo/spdk
    >
    >
    > _______________________________________________
    > SPDK mailing list
    > SPDK(a)lists.01.org
    > https://lists.01.org/mailman/listinfo/spdk
    >
    _______________________________________________
    SPDK mailing list
    SPDK(a)lists.01.org
    https://lists.01.org/mailman/listinfo/spdk