[SPDK] Re: Nvme target and Multi processes

* [SPDK] Re: Nvme target and Multi processes
@ 2019-11-28 10:57 Avner Taieb
  0 siblings, 0 replies; 10+ messages in thread
From: Avner Taieb @ 2019-11-28 10:57 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 4726 bytes --]

Thanks Jim.
Are the limitations that you listed , hold also for the case the the nvme
target (in our case) doesn't use the underneath subsystem ?
The single reason we set it up to secondary,  is because it communicates
through the SPDK/DPDK shared memory (pools and ring) to communicate with
other processes ?

Avner

On Wed, Nov 27, 2019 at 5:12 PM Harris, James R <james.r.harris(a)intel.com>
wrote:

> Hi Gang and Avner,
>
> The only component in SPDK that has any multi-process support today is the
> SPDK NVMe driver.  The NVMe target does not support multi-process.  You can
> start multiple NVMe target applications on the same system, each as a
> primary process.
>
> Supporting multi-process is way harder than just naming for the memzones.
> Here are just a few areas of complexity (there are many more):
>
> * the bdev layer requires coordination across all threads for certain
> operations such as reset and QoS; this thread iteration is not designed to
> work across threads in multiple processes
> * the NVMe target requires coordination across all threads associated with
> a subsystem; so two processes could not each have connections to the same
> subsystem
> * the processes share state via shared memory; a bug or crash in one
> process may silently affect the state of the other processes
>
> We've been doing a lot of work with SPDK and DPDK to reduce the amount of
> time it takes for a process to initialize.  The goal here is to be able to
> more quickly restart a target process when necessary, rather than trying to
> rely on multi-process for availability.
>
> -Jim
>
>
>
> On 11/27/19, 1:46 AM, "Cao, Gang" <gang.cao(a)intel.com> wrote:
>
>     One workaround for the 2nd question is that if the name is unique and
> the process who allocate it is exited unexpectedly, the other alive process
> (in the multi processes mode) could try to free it through the name lookup
> to determine whether the allocated memzone is freed or not.
>
>     For the question, I think we coded the common name there in some cases
> and not using the options. May just need to change the code in this case.
>
>     -----Original Message-----
>     From: Avner Taieb <avner(a)reduxio.com>
>     Sent: Wednesday, November 27, 2019 16:36
>     To: Storage Performance Development Kit <spdk(a)lists.01.org>
>     Subject: [SPDK] Re: Nvme target and Multi processes
>
>     Thanks.
>     Is it possible without modifying the spdk code (e.g. through options
> when
>     initialize) ?
>     Also for my second questions , using each time a process re-start a
> new memory zone, will cause memory leakage.
>
>     On Wed, Nov 27, 2019 at 10:22 AM Cao, Gang <gang.cao(a)intel.com> wrote:
>
>     > If creating the same memory zone, it will fail as duplicate name.
> You
>     > may just lookup an existing zone, or create a unique zone per
> process.
>     >
>     > -----Original Message-----
>     > From: Avner Taieb <avner(a)reduxio.com>
>     > Sent: Wednesday, November 27, 2019 14:52
>     > To: Storage Performance Development Kit <spdk(a)lists.01.org>
>     > Subject: [SPDK] Nvme target and Multi processes
>     >
>     > Hi All,
>     > Using spdk 19.07, NVME target over TCP, in multi process environment
>     >
>     > The zone name that is reserved for nvme target is not unique (i.e.
>     > "spdk_nvmf_TCP_data") :
>     > 1.  Does it mean that two nvme target processes running as secondary
>     > can not run simultaneously ?
>     > 2. In case of only one secondary as nvme target , if the nvme target
>     > process exits unexpectedly, it looks like it will not be able to
>     > create memory zone, because it already exist from a previous run.
>     >
>     > Am I right ? and if yes, is there any simple workaround  ?
>     >
>     > Thanks
>     > _______________________________________________
>     > SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email
> to
>     > spdk-leave(a)lists.01.org
>     > _______________________________________________
>     > SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email
> to
>     > spdk-leave(a)lists.01.org
>     >
>     _______________________________________________
>     SPDK mailing list -- spdk(a)lists.01.org
>     To unsubscribe send an email to spdk-leave(a)lists.01.org
>     _______________________________________________
>     SPDK mailing list -- spdk(a)lists.01.org
>     To unsubscribe send an email to spdk-leave(a)lists.01.org
>
>
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
>

^ permalink raw reply	[flat|nested] 10+ messages in thread