Re: [SPDK] Multiple start/stop of reactor and add/remove of bdev

[Michael Haeuptle <michaelhaeuptle at gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 5366 bytes --]

Hi Jim,

thanks for the reply! These patches actually solve another issue that we're
currently having. We have a cooperative threading environment and I had to
modify the current reactor loop to insert a yield to give other threads a
chance to run. Making the poller and message processing functions
accessible allows me to write our own reactor loop more easily.

As you stated correctly, my use case is to load balance based on the
current # of devices that are configured by a user, which changes over
time. Once the patches are merged, I need to take a closer look and do some
experiments.

-- Michael

On Mon, Oct 8, 2018 at 6:28 PM Harris, James R <james.r.harris(a)intel.com>
wrote:

> Hi Michael,
>
> The reactors definitely have a cost if they are polling with no real work
> to do.
>
> Ben Walker has been working on a patch series [1] that would effectively
> decouple and “spdk_thread” from an “spdk_reactor”.  Currently the
> relationship is always one-to-one.  Vishal Verma pushed some patches
> earlier this year [2] that track how much real work a reactor is doing by
> tracking whether a poller did any real work.  Combining these two would
> enable dynamic load balancing.  It sounds like that might meet the vhost
> use case you’ve described.
>
> This is why I’m interested in understanding more about Shahar’s request to
> start and stop reactors and reinitialize the bdev layer, and whether it is
> also based on a desire to do this kind of load balancing.
>
> Thanks,
>
> -Jim
>
> [1] Patch series starts here:
> https://review.gerrithub.io/#/c/spdk/spdk/+/417784/
> [2] Main patch is: https://review.gerrithub.io/#/c/spdk/spdk/+/412695/
>
> On 10/8/18, 12:24 PM, "SPDK on behalf of Michael Haeuptle" <
> spdk-bounces(a)lists.01.org on behalf of michaelhaeuptle(a)gmail.com> wrote:
>
>     I can't comment on Shahar's use case but it would be helpful in cases
> where
>     I want to flex out reactors based on the # of devices which are not
> fixed.
>
>     In my use case, I don't really want to have reactors running if I don't
>     need them since they do incur some cost as far as I understand it
> (maybe
>     max_delay_us could be used).
>
>     For example, it may be good enough to have 1 reactor serving 2
>     vhost_scsi_controllers starting out. However, if I need to add more
>     controllers (since I want to add more devices), then I'd like to start
>     another reactor for the new vhost_scsi_controllers. Reducing
> controllers is
>     tricky but I'm not sure this happens very often in my use case.
>
>     -- Michael
>
>
>
>     On Mon, Oct 8, 2018 at 11:00 AM Harris, James R <
> james.r.harris(a)intel.com>
>     wrote:
>
>     > Hi Shahar,
>     >
>     > Can you describe why you require starting/stopping the reactor on
> demand?
>     >
>     > Thanks,
>     >
>     > -Jim
>     >
>     >
>     >
>     >
>     > On 10/8/18, 1:07 AM, "SPDK on behalf of Shahar Salzman" <
>     > spdk-bounces(a)lists.01.org on behalf of shahar.salzman(a)kaminario.com>
>     > wrote:
>     >
>     >     Hi,
>     >
>     >
>     >     I am now working with my passthrough bdev directly from my
>     > application, and require adding/removing them dynamically. In
> addition, I
>     > also require starting/stopping the reactor on demand without tearing
> down
>     > the entire system (e.g. dpdk stays initialized). This mostly works
> out of
>     > the box, but there are a few globals and bdev internal fields which
> require
>     > re-init or tear down.
>     >
>     >     In addition, the copy engine did not io_unregister itself, so I
> also
>     > added this in the patchset, allowing me the following application
> life span:
>     >
>     >
>     >     - env init (init config, spdk_env_init)
>     >
>     >     - reactors init
>     >
>     >     - while (application is alive)
>     >
>     >       subsystem_init + reactor start
>     >
>     >       ...
>     >
>     >       Do some NVMeF stuff
>     >
>     >       ...
>     >
>     >       subsysem_fini + reactors stop
>     >
>     >     - reactors_fini
>     >
>     >     - rpc_finish
>     >
>     >
>     >
>     >     I submitted the following patches for review:
>     >
>     >     https://review.gerrithub.io/c/spdk/spdk/+/428305
>     >
>     >     https://review.gerrithub.io/c/spdk/spdk/+/428304
>     >
>     >     https://review.gerrithub.io/c/spdk/spdk/+/428303
>     >
>     >     https://review.gerrithub.io/c/spdk/spdk/+/428302
>     >
>     >
>     >     WDYT?
>     >
>     >
>     >     Shahar
>     >     _______________________________________________
>     >     SPDK mailing list
>     >     SPDK(a)lists.01.org
>     >     https://lists.01.org/mailman/listinfo/spdk
>     >
>     >
>     > _______________________________________________
>     > SPDK mailing list
>     > SPDK(a)lists.01.org
>     > https://lists.01.org/mailman/listinfo/spdk
>     >
>     _______________________________________________
>     SPDK mailing list
>     SPDK(a)lists.01.org
>     https://lists.01.org/mailman/listinfo/spdk
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>