Hi,


My use case is a distributed highly available storage appliance with features such as dedup, snapshots etc.

We have both FC and iSCSI targets implemented in the kernel, and basically using the same cores as SPDK for receiving IO. Reactor polling comes at a cost which I pay happily as long as there is work for the NVMeF target. If the customer does not use the system with NVMeF, this is just taking away from FC/iSCSI performance.

With the patches bellow, I can start/stop without tearing down memory etc.

It should be noted that I am not using the spdk app, since I require this fine granularity, so I embedded the init/run/destroy flow in our app.


Shahar

________________________________
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Harris, James R <james.r.harris(a)intel.com>
Sent: Tuesday, October 9, 2018 3:28:13 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] Multiple start/stop of reactor and add/remove of bdev

Hi Michael,

The reactors definitely have a cost if they are polling with no real work to do.

Ben Walker has been working on a patch series [1] that would effectively decouple and “spdk_thread” from an “spdk_reactor”.  Currently the relationship is always one-to-one.  Vishal Verma pushed some patches earlier this year [2] that track how much real work a reactor is doing by tracking whether a poller did any real work.  Combining these two would enable dynamic load balancing.  It sounds like that might meet the vhost use case you’ve described.

This is why I’m interested in understanding more about Shahar’s request to start and stop reactors and reinitialize the bdev layer, and whether it is also based on a desire to do this kind of load balancing.

Thanks,

-Jim

[1] Patch series starts here: https://review.gerrithub.io/#/c/spdk/spdk/+/417784/
[2] Main patch is: https://review.gerrithub.io/#/c/spdk/spdk/+/412695/

On 10/8/18, 12:24 PM, "SPDK on behalf of Michael Haeuptle" <spdk-bounces(a)lists.01.org on behalf of michaelhaeuptle(a)gmail.com> wrote:

    I can't comment on Shahar's use case but it would be helpful in cases where
    I want to flex out reactors based on the # of devices which are not fixed.

    In my use case, I don't really want to have reactors running if I don't
    need them since they do incur some cost as far as I understand it (maybe
    max_delay_us could be used).

    For example, it may be good enough to have 1 reactor serving 2
    vhost_scsi_controllers starting out. However, if I need to add more
    controllers (since I want to add more devices), then I'd like to start
    another reactor for the new vhost_scsi_controllers. Reducing controllers is
    tricky but I'm not sure this happens very often in my use case.

    -- Michael



    On Mon, Oct 8, 2018 at 11:00 AM Harris, James R <james.r.harris(a)intel.com>
    wrote:

    > Hi Shahar,
    >
    > Can you describe why you require starting/stopping the reactor on demand?
    >
    > Thanks,
    >
    > -Jim
    >
    >
    >
    >
    > On 10/8/18, 1:07 AM, "SPDK on behalf of Shahar Salzman" <
    > spdk-bounces(a)lists.01.org on behalf of shahar.salzman(a)kaminario.com>
    > wrote:
    >
    >     Hi,
    >
    >
    >     I am now working with my passthrough bdev directly from my
    > application, and require adding/removing them dynamically. In addition, I
    > also require starting/stopping the reactor on demand without tearing down
    > the entire system (e.g. dpdk stays initialized). This mostly works out of
    > the box, but there are a few globals and bdev internal fields which require
    > re-init or tear down.
    >
    >     In addition, the copy engine did not io_unregister itself, so I also
    > added this in the patchset, allowing me the following application life span:
    >
    >
    >     - env init (init config, spdk_env_init)
    >
    >     - reactors init
    >
    >     - while (application is alive)
    >
    >       subsystem_init + reactor start
    >
    >       ...
    >
    >       Do some NVMeF stuff
    >
    >       ...
    >
    >       subsysem_fini + reactors stop
    >
    >     - reactors_fini
    >
    >     - rpc_finish
    >
    >
    >
    >     I submitted the following patches for review:
    >
    >     https://review.gerrithub.io/c/spdk/spdk/+/428305
    >
    >     https://review.gerrithub.io/c/spdk/spdk/+/428304
    >
    >     https://review.gerrithub.io/c/spdk/spdk/+/428303
    >
    >     https://review.gerrithub.io/c/spdk/spdk/+/428302
    >
    >
    >     WDYT?
    >
    >
    >     Shahar
    >     _______________________________________________
    >     SPDK mailing list
    >     SPDK(a)lists.01.org
    >     https://lists.01.org/mailman/listinfo/spdk
    >
    >
    > _______________________________________________
    > SPDK mailing list
    > SPDK(a)lists.01.org
    > https://lists.01.org/mailman/listinfo/spdk
    >
    _______________________________________________
    SPDK mailing list
    SPDK(a)lists.01.org
    https://lists.01.org/mailman/listinfo/spdk


_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk