From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============0532319746984058515==" MIME-Version: 1.0 From: Harris, James R Subject: Re: [SPDK] Multiple start/stop of reactor and add/remove of bdev Date: Tue, 09 Oct 2018 21:15:37 +0000 Message-ID: <622AFE0B-2865-42A7-8F78-277DDACFC522@intel.com> In-Reply-To: AM6PR04MB51279022A5F6EA2DD41C1CA089E70@AM6PR04MB5127.eurprd04.prod.outlook.com List-ID: To: spdk@lists.01.org --===============0532319746984058515== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Shahar, Thanks for the clarifications =E2=80=93 specifically that you are embedding= the library init/fini calls directly in your app. I looked at your patches =E2=80=93 overall they seem reasonable. I=E2=80= =99ve posted some comments on them in GerritHub =E2=80=93 we should be able= to get them merged pretty easily. Thanks, -Jim On 10/9/18, 12:47 AM, "SPDK on behalf of Shahar Salzman" wrote: Hi, = = My use case is a distributed highly available storage appliance with fe= atures such as dedup, snapshots etc. = We have both FC and iSCSI targets implemented in the kernel, and basica= lly using the same cores as SPDK for receiving IO. Reactor polling comes at= a cost which I pay happily as long as there is work for the NVMeF target. = If the customer does not use the system with NVMeF, this is just taking awa= y from FC/iSCSI performance. = With the patches bellow, I can start/stop without tearing down memory e= tc. = It should be noted that I am not using the spdk app, since I require th= is fine granularity, so I embedded the init/run/destroy flow in our app. = = Shahar = ________________________________ From: SPDK on behalf of Harris, James R <= james.r.harris(a)intel.com> Sent: Tuesday, October 9, 2018 3:28:13 AM To: Storage Performance Development Kit Subject: Re: [SPDK] Multiple start/stop of reactor and add/remove of bd= ev = Hi Michael, = The reactors definitely have a cost if they are polling with no real wo= rk to do. = Ben Walker has been working on a patch series [1] that would effectivel= y decouple and =E2=80=9Cspdk_thread=E2=80=9D from an =E2=80=9Cspdk_reactor= =E2=80=9D. Currently the relationship is always one-to-one. Vishal Verma = pushed some patches earlier this year [2] that track how much real work a r= eactor is doing by tracking whether a poller did any real work. Combining = these two would enable dynamic load balancing. It sounds like that might m= eet the vhost use case you=E2=80=99ve described. = This is why I=E2=80=99m interested in understanding more about Shahar= =E2=80=99s request to start and stop reactors and reinitialize the bdev lay= er, and whether it is also based on a desire to do this kind of load balanc= ing. = Thanks, = -Jim = [1] Patch series starts here: https://review.gerrithub.io/#/c/spdk/spdk= /+/417784/ [2] Main patch is: https://review.gerrithub.io/#/c/spdk/spdk/+/412695/ = On 10/8/18, 12:24 PM, "SPDK on behalf of Michael Haeuptle" wrote: = I can't comment on Shahar's use case but it would be helpful in cas= es where I want to flex out reactors based on the # of devices which are not= fixed. = In my use case, I don't really want to have reactors running if I d= on't need them since they do incur some cost as far as I understand it (= maybe max_delay_us could be used). = For example, it may be good enough to have 1 reactor serving 2 vhost_scsi_controllers starting out. However, if I need to add more controllers (since I want to add more devices), then I'd like to st= art another reactor for the new vhost_scsi_controllers. Reducing contro= llers is tricky but I'm not sure this happens very often in my use case. = -- Michael = = = On Mon, Oct 8, 2018 at 11:00 AM Harris, James R wrote: = > Hi Shahar, > > Can you describe why you require starting/stopping the reactor on= demand? > > Thanks, > > -Jim > > > > > On 10/8/18, 1:07 AM, "SPDK on behalf of Shahar Salzman" < > spdk-bounces(a)lists.01.org on behalf of shahar.salzman(a)kaminar= io.com> > wrote: > > Hi, > > > I am now working with my passthrough bdev directly from my > application, and require adding/removing them dynamically. In add= ition, I > also require starting/stopping the reactor on demand without tear= ing down > the entire system (e.g. dpdk stays initialized). This mostly work= s out of > the box, but there are a few globals and bdev internal fields whi= ch require > re-init or tear down. > > In addition, the copy engine did not io_unregister itself, so= I also > added this in the patchset, allowing me the following application= life span: > > > - env init (init config, spdk_env_init) > > - reactors init > > - while (application is alive) > > subsystem_init + reactor start > > ... > > Do some NVMeF stuff > > ... > > subsysem_fini + reactors stop > > - reactors_fini > > - rpc_finish > > > > I submitted the following patches for review: > > https://review.gerrithub.io/c/spdk/spdk/+/428305 > > https://review.gerrithub.io/c/spdk/spdk/+/428304 > > https://review.gerrithub.io/c/spdk/spdk/+/428303 > > https://review.gerrithub.io/c/spdk/spdk/+/428302 > > > WDYT? > > > Shahar > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > > > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk = = _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk = --===============0532319746984058515==--