Re: [SPDK] SPDK concurrent access

Re: [SPDK] SPDK concurrent access

[Nabarro, Tom]

[-- Attachment #1: Type: text/plain, Size: 7599 bytes --]

Looks like fio w/ plugin is not reading shm_id from flags on commandline as below, when I supply in the fio config it works, fio_plugin.c comments suggest this is a bug
[ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk1 --base-virtaddr=0x200000000000 --proc-type=auto ]

Regards,
Tom Nabarro 
M: +44 (0)7786 260986
Skype: tom.nabarro


-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Nabarro, Tom
Sent: Tuesday, October 2, 2018 3:59 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK concurrent access

Continuing with investigation:

LD_PRELOAD=...snip.../spdk/fio_plugin/fio_plugin ...snip.../bin/fio --filename="trtype=PCIe traddr=0000.81.00.0 ns=1 shm_id=2 enable_sgl=1" --ioengine=spdk ...snip.../share/spdk/fio_plugin/example_config.fio

resulting output doesn't seem to print the assigned shm_id, should it?
[ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid35355 ]

Regards,
Tom Nabarro 
M: +44 (0)7786 260986
Skype: tom.nabarro


-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Nabarro, Tom
Sent: Tuesday, October 2, 2018 2:38 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK concurrent access

Thanks yes can see some implementation in examples/nvme/perf

Regards,
Tom Nabarro 
M: +44 (0)7786 260986
Skype: tom.nabarro


-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Wodkowski, PawelX
Sent: Tuesday, October 2, 2018 2:34 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK concurrent access

Try this https://spdk.io/doc/app_overview.html#cmd_arg_single_file_segments

Take a look at '--shm-id'.

Pawel

> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Nabarro, 
> Tom
> Sent: Tuesday, October 2, 2018 3:27 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] SPDK concurrent access
> 
> Great thanks for the link, any advice on how to pass the necessary EAL 
> parameters using SPDK (maybe in the init())?
> 
> Regards,
> Tom Nabarro
> M: +44 (0)7786 260986
> Skype: tom.nabarro
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul 
> E
> Sent: Tuesday, October 2, 2018 2:01 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] SPDK concurrent access
> 
> Hi Tom,
> 
> Great question! Check out the DPDK docs for the complete answer:
> https://doc.dpdk.org/guides/prog_guide/multi_proc_support.html
> 
> Thx
> Paul
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Nabarro, 
> Tom
> Sent: Tuesday, October 2, 2018 5:45 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] SPDK concurrent access
> 
> Hello
> 
> Is it correct that 2 processes cannot access SPDK simultaneously, for 
> example
> 
> Ex:.
> Start 2 terminals (root)
> on terminal 1 run nvme_manage example, select #1 to list controllers 
> but don't exit on terminal 2 run identify example. Result EAL: FATAL: 
> Cannot get hugepage information.
> if we end the nvme_manage process then identity completes okay
> 
> The context is that I'm using Golang to interact with SPDK (using 
> simple bindings to execute routines similar to the examples) and 
> expected that I could run different tasks (sequentially) from 
> go-routines. Example tasks include firmware update and executing fio 
> workloads with the spdk fio_plugin example. What I'm finding is that 
> even when executed within a go-routine (implemented with ULTs), the 
> necessary resources are not released afterwards and then I can't issue 
> a subsequent spdk task because I receive similar FATAL failures as listed above.
> 
> One solution might be to fork separate processes to perform the tasks 
> require and build executables to do so but I don't want to have to 
> resort to that.
> 
> Best regards,
> Tom Nabarro BEng (hons) MIET
> Intel Corporation
> Software Engineer
> E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
> M: +44 (0)7786 260986
> Skype: tom.nabarro
> 
> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47
> 
> This e-mail and any attachments may contain confidential material for 
> the sole use of the intended recipient(s). Any review or distribution 
> by others is strictly prohibited. If you are not the intended 
> recipient, please contact the sender and delete all copies.
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47
> 
> This e-mail and any attachments may contain confidential material for 
> the sole use of the intended recipient(s). Any review or distribution 
> by others is strictly prohibited. If you are not the intended 
> recipient, please contact the sender and delete all copies.
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: [SPDK] SPDK concurrent access

[Nabarro, Tom]

[-- Attachment #1: Type: text/plain, Size: 6271 bytes --]

Do you have the SPDK NVMe driver loaded into your application's process space?

> I'm calling libspdk.so from go-spdk bindings that link to it, the bindings are imported in the application, so yes as it is statically linked

Are you running something like the SPDK NVMe-oF or iSCSI target? Most SPDK applications have extensive JSON-RPC interfaces now, which is why I ask if you are directly embedding the SPDK NVMe driver (just a library, no RPC) or if you are using a full application.

> I'm not using the JSON-RPC interface as I'm pulling the C structs representing the Controllers and Namespaces across the c-go boundary and using the translated information in the go app

Another interesting program we use for testing is "bdev_svc", which loads up SPDK's block device layer and then sits there and waits for commands via JSON- RPC. It's in test/app/bdev_svc. One thing you might want to take a look at is starting that up when your application starts up, and then interacting with it via JSON-RPC. We're even in the process of adding a way to send NVMe commands via JSON-RPC to the devices, so you could do firmware update and such that way.

> That sounds really interesting, so we are using blobstore interaction elsewhere in another part of our application but that looks like a great development tool, will take a look. Firmware update seems to be working fine through bindings for me.

The only problem will be when you go to run fio. I think when you run fio, you should stop any background SPDK service or unload the NVMe driver from your local process, and then run fio independently. That benchmark is really not designed to be run in a shared environment right now.

> right, when you say unload the driver, is that just by calling detach on the controller or running the "setup.sh reset" command or something else? I couldn't find a "de_init_env" function!
> I think I'm almost there with running the burn-in as a secondary process (the primary won't be doing much with the devices) but will be wary and probably heed your advice

Regards,
Tom Nabarro 
M: +44 (0)7786 260986
Skype: tom.nabarro


-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
Sent: Tuesday, October 2, 2018 5:51 PM
To: spdk(a)lists.01.org
Subject: Re: [SPDK] SPDK concurrent access

Looping back to the original question to clarify a few things

On Tue, 2018-10-02 at 12:45 +0000, Nabarro, Tom wrote:
> Hello
> 
> Is it correct that 2 processes cannot access SPDK simultaneously, for 
> example
> 
> Ex:.
> Start 2 terminals (root)
> on terminal 1 run nvme_manage example, select #1 to list controllers 
> but don't exit on terminal 2 run identify example. Result EAL: FATAL: 
> Cannot get hugepage information.
> if we end the nvme_manage process then identity completes okay

The SPDK NVMe driver is multi-process aware. The rest of the components in SPDK are not.

> 
> The context is that I'm using Golang to interact with SPDK (using 
> simple bindings to execute routines similar to the examples) and 
> expected that I could run different tasks (sequentially) from 
> go-routines. Example tasks include firmware update and executing fio 
> workloads with the spdk fio_plugin example. What I'm finding is that 
> even when executed within a go-routine (implemented with ULTs), the 
> necessary resources are not released afterwards and then I can't issue 
> a subsequent spdk task because I receive similar FATAL failures as listed above.

Do you have the SPDK NVMe driver loaded into your application's process space?
Are you running something like the SPDK NVMe-oF or iSCSI target? Most SPDK applications have extensive JSON-RPC interfaces now, which is why I ask if you are directly embedding the SPDK NVMe driver (just a library, no RPC) or if you are using a full application.

Another interesting program we use for testing is "bdev_svc", which loads up SPDK's block device layer and then sits there and waits for commands via JSON- RPC. It's in test/app/bdev_svc. One thing you might want to take a look at is starting that up when your application starts up, and then interacting with it via JSON-RPC. We're even in the process of adding a way to send NVMe commands via JSON-RPC to the devices, so you could do firmware update and such that way.

The only problem will be when you go to run fio. I think when you run fio, you should stop any background SPDK service or unload the NVMe driver from your local process, and then run fio independently. That benchmark is really not designed to be run in a shared environment right now.

> 
> One solution might be to fork separate processes to perform the tasks 
> require and build executables to do so but I don't want to have to resort to that.
> 
> Best regards,
> Tom Nabarro BEng (hons) MIET
> Intel Corporation
> Software Engineer
> E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
> M: +44 (0)7786 260986
> Skype: tom.nabarro
> 
> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47
> 
> This e-mail and any attachments may contain confidential material for 
> the sole use of the intended recipient(s). Any review or distribution 
> by others is strictly prohibited. If you are not the intended 
> recipient, please contact the sender and delete all copies.
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: [SPDK] SPDK concurrent access

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 3308 bytes --]

Looping back to the original question to clarify a few things

On Tue, 2018-10-02 at 12:45 +0000, Nabarro, Tom wrote:
> Hello
> 
> Is it correct that 2 processes cannot access SPDK simultaneously, for example
> 
> Ex:.
> Start 2 terminals (root)
> on terminal 1 run nvme_manage example, select #1 to list controllers but don't
> exit
> on terminal 2 run identify example. Result EAL: FATAL: Cannot get hugepage
> information.
> if we end the nvme_manage process then identity completes okay

The SPDK NVMe driver is multi-process aware. The rest of the components in SPDK
are not.

> 
> The context is that I'm using Golang to interact with SPDK (using simple
> bindings to execute routines similar to the examples) and expected that I
> could run different tasks (sequentially) from go-routines. Example tasks
> include firmware update and executing fio workloads with the spdk fio_plugin
> example. What I'm finding is that even when executed within a go-routine
> (implemented with ULTs), the necessary resources are not released afterwards
> and then I can't issue a subsequent spdk task because I receive similar FATAL
> failures as listed above.

Do you have the SPDK NVMe driver loaded into your application's process space?
Are you running something like the SPDK NVMe-oF or iSCSI target? Most SPDK
applications have extensive JSON-RPC interfaces now, which is why I ask if you
are directly embedding the SPDK NVMe driver (just a library, no RPC) or if you
are using a full application.

Another interesting program we use for testing is "bdev_svc", which loads up
SPDK's block device layer and then sits there and waits for commands via JSON-
RPC. It's in test/app/bdev_svc. One thing you might want to take a look at is
starting that up when your application starts up, and then interacting with it
via JSON-RPC. We're even in the process of adding a way to send NVMe commands
via JSON-RPC to the devices, so you could do firmware update and such that way.

The only problem will be when you go to run fio. I think when you run fio, you
should stop any background SPDK service or unload the NVMe driver from your
local process, and then run fio independently. That benchmark is really not
designed to be run in a shared environment right now.

> 
> One solution might be to fork separate processes to perform the tasks require
> and build executables to do so but I don't want to have to resort to that.
> 
> Best regards,
> Tom Nabarro BEng (hons) MIET
> Intel Corporation
> Software Engineer
> E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
> M: +44 (0)7786 260986
> Skype: tom.nabarro
> 
> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ
> VAT No: 860 2173 47
> 
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] SPDK concurrent access

[Nabarro, Tom]

[-- Attachment #1: Type: text/plain, Size: 6383 bytes --]

Continuing with investigation:

LD_PRELOAD=...snip.../spdk/fio_plugin/fio_plugin ...snip.../bin/fio --filename="trtype=PCIe traddr=0000.81.00.0 ns=1 shm_id=2 enable_sgl=1" --ioengine=spdk ...snip.../share/spdk/fio_plugin/example_config.fio

resulting output doesn't seem to print the assigned shm_id, should it?
[ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid35355 ]

Regards,
Tom Nabarro 
M: +44 (0)7786 260986
Skype: tom.nabarro


-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Nabarro, Tom
Sent: Tuesday, October 2, 2018 2:38 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK concurrent access

Thanks yes can see some implementation in examples/nvme/perf

Regards,
Tom Nabarro 
M: +44 (0)7786 260986
Skype: tom.nabarro


-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Wodkowski, PawelX
Sent: Tuesday, October 2, 2018 2:34 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK concurrent access

Try this https://spdk.io/doc/app_overview.html#cmd_arg_single_file_segments

Take a look at '--shm-id'.

Pawel

> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Nabarro, 
> Tom
> Sent: Tuesday, October 2, 2018 3:27 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] SPDK concurrent access
> 
> Great thanks for the link, any advice on how to pass the necessary EAL 
> parameters using SPDK (maybe in the init())?
> 
> Regards,
> Tom Nabarro
> M: +44 (0)7786 260986
> Skype: tom.nabarro
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul 
> E
> Sent: Tuesday, October 2, 2018 2:01 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] SPDK concurrent access
> 
> Hi Tom,
> 
> Great question! Check out the DPDK docs for the complete answer:
> https://doc.dpdk.org/guides/prog_guide/multi_proc_support.html
> 
> Thx
> Paul
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Nabarro, 
> Tom
> Sent: Tuesday, October 2, 2018 5:45 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] SPDK concurrent access
> 
> Hello
> 
> Is it correct that 2 processes cannot access SPDK simultaneously, for 
> example
> 
> Ex:.
> Start 2 terminals (root)
> on terminal 1 run nvme_manage example, select #1 to list controllers 
> but don't exit on terminal 2 run identify example. Result EAL: FATAL: 
> Cannot get hugepage information.
> if we end the nvme_manage process then identity completes okay
> 
> The context is that I'm using Golang to interact with SPDK (using 
> simple bindings to execute routines similar to the examples) and 
> expected that I could run different tasks (sequentially) from 
> go-routines. Example tasks include firmware update and executing fio 
> workloads with the spdk fio_plugin example. What I'm finding is that 
> even when executed within a go-routine (implemented with ULTs), the 
> necessary resources are not released afterwards and then I can't issue 
> a subsequent spdk task because I receive similar FATAL failures as listed above.
> 
> One solution might be to fork separate processes to perform the tasks 
> require and build executables to do so but I don't want to have to 
> resort to that.
> 
> Best regards,
> Tom Nabarro BEng (hons) MIET
> Intel Corporation
> Software Engineer
> E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
> M: +44 (0)7786 260986
> Skype: tom.nabarro
> 
> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47
> 
> This e-mail and any attachments may contain confidential material for 
> the sole use of the intended recipient(s). Any review or distribution 
> by others is strictly prohibited. If you are not the intended 
> recipient, please contact the sender and delete all copies.
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47
> 
> This e-mail and any attachments may contain confidential material for 
> the sole use of the intended recipient(s). Any review or distribution 
> by others is strictly prohibited. If you are not the intended 
> recipient, please contact the sender and delete all copies.
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: [SPDK] SPDK concurrent access

[Nabarro, Tom]

[-- Attachment #1: Type: text/plain, Size: 5060 bytes --]

Thanks yes can see some implementation in examples/nvme/perf

Regards,
Tom Nabarro 
M: +44 (0)7786 260986
Skype: tom.nabarro


-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Wodkowski, PawelX
Sent: Tuesday, October 2, 2018 2:34 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK concurrent access

Try this https://spdk.io/doc/app_overview.html#cmd_arg_single_file_segments

Take a look at '--shm-id'.

Pawel

> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Nabarro, 
> Tom
> Sent: Tuesday, October 2, 2018 3:27 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] SPDK concurrent access
> 
> Great thanks for the link, any advice on how to pass the necessary EAL 
> parameters using SPDK (maybe in the init())?
> 
> Regards,
> Tom Nabarro
> M: +44 (0)7786 260986
> Skype: tom.nabarro
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul 
> E
> Sent: Tuesday, October 2, 2018 2:01 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] SPDK concurrent access
> 
> Hi Tom,
> 
> Great question! Check out the DPDK docs for the complete answer:
> https://doc.dpdk.org/guides/prog_guide/multi_proc_support.html
> 
> Thx
> Paul
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Nabarro, 
> Tom
> Sent: Tuesday, October 2, 2018 5:45 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] SPDK concurrent access
> 
> Hello
> 
> Is it correct that 2 processes cannot access SPDK simultaneously, for 
> example
> 
> Ex:.
> Start 2 terminals (root)
> on terminal 1 run nvme_manage example, select #1 to list controllers 
> but don't exit on terminal 2 run identify example. Result EAL: FATAL: 
> Cannot get hugepage information.
> if we end the nvme_manage process then identity completes okay
> 
> The context is that I'm using Golang to interact with SPDK (using 
> simple bindings to execute routines similar to the examples) and 
> expected that I could run different tasks (sequentially) from 
> go-routines. Example tasks include firmware update and executing fio 
> workloads with the spdk fio_plugin example. What I'm finding is that 
> even when executed within a go-routine (implemented with ULTs), the 
> necessary resources are not released afterwards and then I can't issue 
> a subsequent spdk task because I receive similar FATAL failures as listed above.
> 
> One solution might be to fork separate processes to perform the tasks 
> require and build executables to do so but I don't want to have to 
> resort to that.
> 
> Best regards,
> Tom Nabarro BEng (hons) MIET
> Intel Corporation
> Software Engineer
> E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
> M: +44 (0)7786 260986
> Skype: tom.nabarro
> 
> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47
> 
> This e-mail and any attachments may contain confidential material for 
> the sole use of the intended recipient(s). Any review or distribution 
> by others is strictly prohibited. If you are not the intended 
> recipient, please contact the sender and delete all copies.
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47
> 
> This e-mail and any attachments may contain confidential material for 
> the sole use of the intended recipient(s). Any review or distribution 
> by others is strictly prohibited. If you are not the intended 
> recipient, please contact the sender and delete all copies.
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: [SPDK] SPDK concurrent access

[Wodkowski, PawelX]

[-- Attachment #1: Type: text/plain, Size: 4028 bytes --]

Try this https://spdk.io/doc/app_overview.html#cmd_arg_single_file_segments

Take a look at '--shm-id'.

Pawel

> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Nabarro, Tom
> Sent: Tuesday, October 2, 2018 3:27 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] SPDK concurrent access
> 
> Great thanks for the link, any advice on how to pass the necessary EAL
> parameters using SPDK (maybe in the init())?
> 
> Regards,
> Tom Nabarro
> M: +44 (0)7786 260986
> Skype: tom.nabarro
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
> Sent: Tuesday, October 2, 2018 2:01 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] SPDK concurrent access
> 
> Hi Tom,
> 
> Great question! Check out the DPDK docs for the complete answer:
> https://doc.dpdk.org/guides/prog_guide/multi_proc_support.html
> 
> Thx
> Paul
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Nabarro, Tom
> Sent: Tuesday, October 2, 2018 5:45 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] SPDK concurrent access
> 
> Hello
> 
> Is it correct that 2 processes cannot access SPDK simultaneously, for example
> 
> Ex:.
> Start 2 terminals (root)
> on terminal 1 run nvme_manage example, select #1 to list controllers but
> don't exit on terminal 2 run identify example. Result EAL: FATAL: Cannot get
> hugepage information.
> if we end the nvme_manage process then identity completes okay
> 
> The context is that I'm using Golang to interact with SPDK (using simple
> bindings to execute routines similar to the examples) and expected that I
> could run different tasks (sequentially) from go-routines. Example tasks
> include firmware update and executing fio workloads with the spdk
> fio_plugin example. What I'm finding is that even when executed within a
> go-routine (implemented with ULTs), the necessary resources are not
> released afterwards and then I can't issue a subsequent spdk task because I
> receive similar FATAL failures as listed above.
> 
> One solution might be to fork separate processes to perform the tasks
> require and build executables to do so but I don't want to have to resort to
> that.
> 
> Best regards,
> Tom Nabarro BEng (hons) MIET
> Intel Corporation
> Software Engineer
> E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
> M: +44 (0)7786 260986
> Skype: tom.nabarro
> 
> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47
> 
> This e-mail and any attachments may contain confidential material for the
> sole use of the intended recipient(s). Any review or distribution by others is
> strictly prohibited. If you are not the intended recipient, please contact the
> sender and delete all copies.
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ
> VAT No: 860 2173 47
> 
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] SPDK concurrent access

[Nabarro, Tom]

[-- Attachment #1: Type: text/plain, Size: 3315 bytes --]

Great thanks for the link, any advice on how to pass the necessary EAL parameters using SPDK (maybe in the init())?

Regards,
Tom Nabarro 
M: +44 (0)7786 260986
Skype: tom.nabarro

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Tuesday, October 2, 2018 2:01 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK concurrent access

Hi Tom,

Great question! Check out the DPDK docs for the complete answer: https://doc.dpdk.org/guides/prog_guide/multi_proc_support.html 

Thx
Paul

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Nabarro, Tom
Sent: Tuesday, October 2, 2018 5:45 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] SPDK concurrent access

Hello

Is it correct that 2 processes cannot access SPDK simultaneously, for example

Ex:.
Start 2 terminals (root)
on terminal 1 run nvme_manage example, select #1 to list controllers but don't exit on terminal 2 run identify example. Result EAL: FATAL: Cannot get hugepage information.
if we end the nvme_manage process then identity completes okay

The context is that I'm using Golang to interact with SPDK (using simple bindings to execute routines similar to the examples) and expected that I could run different tasks (sequentially) from go-routines. Example tasks include firmware update and executing fio workloads with the spdk fio_plugin example. What I'm finding is that even when executed within a go-routine (implemented with ULTs), the necessary resources are not released afterwards and then I can't issue a subsequent spdk task because I receive similar FATAL failures as listed above.

One solution might be to fork separate processes to perform the tasks require and build executables to do so but I don't want to have to resort to that.

Best regards,
Tom Nabarro BEng (hons) MIET
Intel Corporation
Software Engineer
E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
M: +44 (0)7786 260986
Skype: tom.nabarro

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: [SPDK] SPDK concurrent access

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 2271 bytes --]

Hi Tom,

Great question! Check out the DPDK docs for the complete answer: https://doc.dpdk.org/guides/prog_guide/multi_proc_support.html 

Thx
Paul

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Nabarro, Tom
Sent: Tuesday, October 2, 2018 5:45 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] SPDK concurrent access

Hello

Is it correct that 2 processes cannot access SPDK simultaneously, for example

Ex:.
Start 2 terminals (root)
on terminal 1 run nvme_manage example, select #1 to list controllers but don't exit
on terminal 2 run identify example. Result EAL: FATAL: Cannot get hugepage information.
if we end the nvme_manage process then identity completes okay

The context is that I'm using Golang to interact with SPDK (using simple bindings to execute routines similar to the examples) and expected that I could run different tasks (sequentially) from go-routines. Example tasks include firmware update and executing fio workloads with the spdk fio_plugin example. What I'm finding is that even when executed within a go-routine (implemented with ULTs), the necessary resources are not released afterwards and then I can't issue a subsequent spdk task because I receive similar FATAL failures as listed above.

One solution might be to fork separate processes to perform the tasks require and build executables to do so but I don't want to have to resort to that.

Best regards,
Tom Nabarro BEng (hons) MIET
Intel Corporation
Software Engineer
E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
M: +44 (0)7786 260986
Skype: tom.nabarro

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

[SPDK] SPDK concurrent access

[Nabarro, Tom]

[-- Attachment #1: Type: text/plain, Size: 1734 bytes --]

Hello

Is it correct that 2 processes cannot access SPDK simultaneously, for example

Ex:.
Start 2 terminals (root)
on terminal 1 run nvme_manage example, select #1 to list controllers but don't exit
on terminal 2 run identify example. Result EAL: FATAL: Cannot get hugepage information.
if we end the nvme_manage process then identity completes okay

The context is that I'm using Golang to interact with SPDK (using simple bindings to execute routines similar to the examples) and expected that I could run different tasks (sequentially) from go-routines. Example tasks include firmware update and executing fio workloads with the spdk fio_plugin example. What I'm finding is that even when executed within a go-routine (implemented with ULTs), the necessary resources are not released afterwards and then I can't issue a subsequent spdk task because I receive similar FATAL failures as listed above.

One solution might be to fork separate processes to perform the tasks require and build executables to do so but I don't want to have to resort to that.

Best regards,
Tom Nabarro BEng (hons) MIET
Intel Corporation
Software Engineer
E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
M: +44 (0)7786 260986
Skype: tom.nabarro

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.