Re: [SPDK] core_mask and spawning new threads

Re: [SPDK] core_mask and spawning new threads

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 3706 bytes --]

You'll want to check this one out too:

http://www.spdk.io/doc/concurrency.html

You only need to change the way your application spawns threads if you are using the SPDK event framework (spdk/event.h). Many (most?) SPDK components can be used without buying into the whole framework at this point. The two documents linked provide a good overview of the system.


-------- Original message --------
From: "Stojaczyk, DariuszX" <dariuszx.stojaczyk(a)intel.com>
Date: 2/21/18 9:09 AM (GMT-07:00)
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] core_mask and spawning new threads

For integrating your application further with SPDK, you might also want to check the following document:

http://www.spdk.io/doc/event.html

Regards,
D.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Stojaczyk, DariuszX
Sent: Wednesday, February 21, 2018 4:34 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] core_mask and spawning new threads

> in case I'm integrating the SPDK into my existing application, I have to change the way I'm spawning thread in the whole application, right?
That's correct.
Regards,
D.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ernest Zed
Sent: Wednesday, February 21, 2018 1:18 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] core_mask and spawning new threads

Thanks for the explanation. Just to make it clear, in case I'm integrating the SPDK into my existing application, I have to change the way I'm spawning thread in the whole application, right?
Sincerely,
E.

On Wed, Feb 21, 2018 at 1:43 PM, Stojaczyk, DariuszX <dariuszx.stojaczyk(a)intel.com<mailto:dariuszx.stojaczyk(a)intel.com>> wrote:
Hi Ernest,
This is expected behavior. SPDK starts a fixed number of affinitized threads on startup, and the core mask parameter you've mentioned is what specifies the affinity and number of those. Each thread is bound to a single CPU. The one thread that called `spdk_env_init` becomes a master thread and is affinitized to the `opts.master_core`. To spawn additional POSIX threads from any SPDK-affinitized thread, you should use `spdk_call_unaffinitized()` (include/spdk/env.h).

(There's also an `spdk_unaffinitize_thread()`, but it's restricted to very specific workloads and probably shouldn't be used at all.)
Best regards,
D.

From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Ernest Zed
Sent: Wednesday, February 21, 2018 10:47 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] core_mask and spawning new threads

Hi all,
I see that all threads I create within my application are running on the same core. Looks like core_mask is the right parameter to adjust, however I dont see core usage affected by changing the core_mask on spdk_env_opts
The code:
    spdk_env_opts opts;
    spdk_env_opts_init(&opts);
    opts.name<http://opts.name> = "blah";
    opts.core_mask = "0xFFFF";
    opts.master_core = 1;
    opts.shm_id = 0;
    if (spdk_env_init(&opts) < 0) {
        throw std::runtime_error("Unable to initialize SPDK env");
    }
Whatever value I assign to the core_mask everything runs on the same core defined in master_core. If I set affinity to my threads manually everything works as expected. Is it a spdk_env_init bug? Am I doing something wrong?
Sincerely,
Ernest

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 14222 bytes --]

Re: [SPDK] core_mask and spawning new threads

[Wodkowski, PawelX]

[-- Attachment #1: Type: text/plain, Size: 4849 bytes --]

This sound little complicated. Let me explain SPDK core_mask from other side. This is the mask SPDK use to launch own threads called reactors. You can execute any C++ code on those threads by using single shot events (spdk_event_call) or periodically by using pollers (spdk_poller_register). The  rule of thumb is that you must not block/sleep on those threads. So you have following options:


1.       Use SPDK event/poll mode framework

2.       If you really want to have separate C++ thread you can launch them from app startup handler (see spdk_app_start())

3.       Create separate thread in which you call spdk_app_start – this thread will block.

4.       Use combination of 1-2 or 1-3 but this probably introduce an overhead in synchronization.

Hope I was helpful :)
Pawel

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of E.W.Z.
Sent: Wednesday, February 21, 2018 6:41 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] core_mask and spawning new threads

I think I’ve found a way to overcome this. In my test application when thread which were created to produce I/O were spawned from the “main” thread in which the SPDK was initialized. Of course the cpu mask given to SPDK environment was affecting affinity of thread I was creating. So I tried to do the following, create a temporary thread, in this thread create an instance of my object which instantiates the SPDK environment, and then the temporary thread is destroyed and I have fully working SPDK machinery (which in turn spawned its own threads affected by affinity set in SPDK initialization) with threads doing IO created from main thread, which, in turn, knows nothing about SPDK machinery and its affinity. It successfully completed the run. Looks like a hack, but working as intended. What do you say?

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Stojaczyk, DariuszX<mailto:dariuszx.stojaczyk(a)intel.com>
Sent: Wednesday, February 21, 2018 5:33 PM
To: Storage Performance Development Kit<mailto:spdk(a)lists.01.org>
Subject: Re: [SPDK] core_mask and spawning new threads

> in case I'm integrating the SPDK into my existing application, I have to change the way I'm spawning thread in the whole application, right?
That's correct.
Regards,
D.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ernest Zed
Sent: Wednesday, February 21, 2018 1:18 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] core_mask and spawning new threads

Thanks for the explanation. Just to make it clear, in case I'm integrating the SPDK into my existing application, I have to change the way I'm spawning thread in the whole application, right?
Sincerely,
E.

On Wed, Feb 21, 2018 at 1:43 PM, Stojaczyk, DariuszX <dariuszx.stojaczyk(a)intel.com<mailto:dariuszx.stojaczyk(a)intel.com>> wrote:
Hi Ernest,
This is expected behavior. SPDK starts a fixed number of affinitized threads on startup, and the core mask parameter you've mentioned is what specifies the affinity and number of those. Each thread is bound to a single CPU. The one thread that called `spdk_env_init` becomes a master thread and is affinitized to the `opts.master_core`. To spawn additional POSIX threads from any SPDK-affinitized thread, you should use `spdk_call_unaffinitized()` (include/spdk/env.h).

(There's also an `spdk_unaffinitize_thread()`, but it's restricted to very specific workloads and probably shouldn't be used at all.)
Best regards,
D.

From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Ernest Zed
Sent: Wednesday, February 21, 2018 10:47 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] core_mask and spawning new threads

Hi all,
I see that all threads I create within my application are running on the same core. Looks like core_mask is the right parameter to adjust, however I dont see core usage affected by changing the core_mask on spdk_env_opts
The code:
    spdk_env_opts opts;
    spdk_env_opts_init(&opts);
    opts.name<http://opts.name> = "blah";
    opts.core_mask = "0xFFFF";
    opts.master_core = 1;
    opts.shm_id = 0;
    if (spdk_env_init(&opts) < 0) {
        throw std::runtime_error("Unable to initialize SPDK env");
    }
Whatever value I assign to the core_mask everything runs on the same core defined in master_core. If I set affinity to my threads manually everything works as expected. Is it a spdk_env_init bug? Am I doing something wrong?
Sincerely,
Ernest

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 21933 bytes --]

Re: [SPDK] core_mask and spawning new threads

[E.W.Z.]

[-- Attachment #1: Type: text/plain, Size: 3548 bytes --]

I think I’ve found a way to overcome this. In my test application when thread which were created to produce I/O were spawned from the “main” thread in which the SPDK was initialized. Of course the cpu mask given to SPDK environment was affecting affinity of thread I was creating. So I tried to do the following, create a temporary thread, in this thread create an instance of my object which instantiates the SPDK environment, and then the temporary thread is destroyed and I have fully working SPDK machinery (which in turn spawned its own threads affected by affinity set in SPDK initialization) with threads doing IO created from main thread, which, in turn, knows nothing about SPDK machinery and its affinity. It successfully completed the run. Looks like a hack, but working as intended. What do you say?

Sent from Mail for Windows 10

From: Stojaczyk, DariuszX
Sent: Wednesday, February 21, 2018 5:33 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] core_mask and spawning new threads

> in case I'm integrating the SPDK into my existing application, I have to change the way I'm spawning thread in the whole application, right?
That's correct.
Regards,
D.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ernest Zed
Sent: Wednesday, February 21, 2018 1:18 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] core_mask and spawning new threads

Thanks for the explanation. Just to make it clear, in case I'm integrating the SPDK into my existing application, I have to change the way I'm spawning thread in the whole application, right?
Sincerely,
E.

On Wed, Feb 21, 2018 at 1:43 PM, Stojaczyk, DariuszX <dariuszx.stojaczyk(a)intel.com> wrote:
Hi Ernest,
This is expected behavior. SPDK starts a fixed number of affinitized threads on startup, and the core mask parameter you've mentioned is what specifies the affinity and number of those. Each thread is bound to a single CPU. The one thread that called `spdk_env_init` becomes a master thread and is affinitized to the `opts.master_core`. To spawn additional POSIX threads from any SPDK-affinitized thread, you should use `spdk_call_unaffinitized()` (include/spdk/env.h). 
 
(There's also an `spdk_unaffinitize_thread()`, but it's restricted to very specific workloads and probably shouldn't be used at all.)
Best regards,
D.
 
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ernest Zed
Sent: Wednesday, February 21, 2018 10:47 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] core_mask and spawning new threads
 
Hi all,
I see that all threads I create within my application are running on the same core. Looks like core_mask is the right parameter to adjust, however I dont see core usage affected by changing the core_mask on spdk_env_opts
The code:
    spdk_env_opts opts;
    spdk_env_opts_init(&opts);
    opts.name = "blah";
    opts.core_mask = "0xFFFF";
    opts.master_core = 1;
    opts.shm_id = 0;
    if (spdk_env_init(&opts) < 0) {
        throw std::runtime_error("Unable to initialize SPDK env");
    }
Whatever value I assign to the core_mask everything runs on the same core defined in master_core. If I set affinity to my threads manually everything works as expected. Is it a spdk_env_init bug? Am I doing something wrong?
Sincerely,
Ernest

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 13009 bytes --]

Re: [SPDK] core_mask and spawning new threads

[E.W.Z.]

[-- Attachment #1: Type: text/plain, Size: 3317 bytes --]

Thanks, I already checked this document, as well as the one Benjamin provided. I resorted to more C++-ish solution using promise/future, which was mentioned in Bejamin’s link


Sent from Mail for Windows 10

From: Stojaczyk, DariuszX
Sent: Wednesday, February 21, 2018 6:09 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] core_mask and spawning new threads

For integrating your application further with SPDK, you might also want to check the following document:

http://www.spdk.io/doc/event.html

Regards,
D.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Stojaczyk, DariuszX
Sent: Wednesday, February 21, 2018 4:34 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] core_mask and spawning new threads

> in case I'm integrating the SPDK into my existing application, I have to change the way I'm spawning thread in the whole application, right?
That's correct.
Regards,
D.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ernest Zed
Sent: Wednesday, February 21, 2018 1:18 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] core_mask and spawning new threads

Thanks for the explanation. Just to make it clear, in case I'm integrating the SPDK into my existing application, I have to change the way I'm spawning thread in the whole application, right?
Sincerely,
E.

On Wed, Feb 21, 2018 at 1:43 PM, Stojaczyk, DariuszX <dariuszx.stojaczyk(a)intel.com> wrote:
Hi Ernest,
This is expected behavior. SPDK starts a fixed number of affinitized threads on startup, and the core mask parameter you've mentioned is what specifies the affinity and number of those. Each thread is bound to a single CPU. The one thread that called `spdk_env_init` becomes a master thread and is affinitized to the `opts.master_core`. To spawn additional POSIX threads from any SPDK-affinitized thread, you should use `spdk_call_unaffinitized()` (include/spdk/env.h). 
 
(There's also an `spdk_unaffinitize_thread()`, but it's restricted to very specific workloads and probably shouldn't be used at all.)
Best regards,
D.
 
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ernest Zed
Sent: Wednesday, February 21, 2018 10:47 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] core_mask and spawning new threads
 
Hi all,
I see that all threads I create within my application are running on the same core. Looks like core_mask is the right parameter to adjust, however I dont see core usage affected by changing the core_mask on spdk_env_opts
The code:
    spdk_env_opts opts;
    spdk_env_opts_init(&opts);
    opts.name = "blah";
    opts.core_mask = "0xFFFF";
    opts.master_core = 1;
    opts.shm_id = 0;
    if (spdk_env_init(&opts) < 0) {
        throw std::runtime_error("Unable to initialize SPDK env");
    }
Whatever value I assign to the core_mask everything runs on the same core defined in master_core. If I set affinity to my threads manually everything works as expected. Is it a spdk_env_init bug? Am I doing something wrong?
Sincerely,
Ernest

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 13863 bytes --]

Re: [SPDK] core_mask and spawning new threads

[Stojaczyk, DariuszX]

[-- Attachment #1: Type: text/plain, Size: 3080 bytes --]

For integrating your application further with SPDK, you might also want to check the following document:

http://www.spdk.io/doc/event.html

Regards,
D.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Stojaczyk, DariuszX
Sent: Wednesday, February 21, 2018 4:34 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] core_mask and spawning new threads

> in case I'm integrating the SPDK into my existing application, I have to change the way I'm spawning thread in the whole application, right?
That's correct.
Regards,
D.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ernest Zed
Sent: Wednesday, February 21, 2018 1:18 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] core_mask and spawning new threads

Thanks for the explanation. Just to make it clear, in case I'm integrating the SPDK into my existing application, I have to change the way I'm spawning thread in the whole application, right?
Sincerely,
E.

On Wed, Feb 21, 2018 at 1:43 PM, Stojaczyk, DariuszX <dariuszx.stojaczyk(a)intel.com<mailto:dariuszx.stojaczyk(a)intel.com>> wrote:
Hi Ernest,
This is expected behavior. SPDK starts a fixed number of affinitized threads on startup, and the core mask parameter you've mentioned is what specifies the affinity and number of those. Each thread is bound to a single CPU. The one thread that called `spdk_env_init` becomes a master thread and is affinitized to the `opts.master_core`. To spawn additional POSIX threads from any SPDK-affinitized thread, you should use `spdk_call_unaffinitized()` (include/spdk/env.h).

(There's also an `spdk_unaffinitize_thread()`, but it's restricted to very specific workloads and probably shouldn't be used at all.)
Best regards,
D.

From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Ernest Zed
Sent: Wednesday, February 21, 2018 10:47 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] core_mask and spawning new threads

Hi all,
I see that all threads I create within my application are running on the same core. Looks like core_mask is the right parameter to adjust, however I dont see core usage affected by changing the core_mask on spdk_env_opts
The code:
    spdk_env_opts opts;
    spdk_env_opts_init(&opts);
    opts.name<http://opts.name> = "blah";
    opts.core_mask = "0xFFFF";
    opts.master_core = 1;
    opts.shm_id = 0;
    if (spdk_env_init(&opts) < 0) {
        throw std::runtime_error("Unable to initialize SPDK env");
    }
Whatever value I assign to the core_mask everything runs on the same core defined in master_core. If I set affinity to my threads manually everything works as expected. Is it a spdk_env_init bug? Am I doing something wrong?
Sincerely,
Ernest

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 15921 bytes --]

Re: [SPDK] core_mask and spawning new threads

[Stojaczyk, DariuszX]

[-- Attachment #1: Type: text/plain, Size: 2646 bytes --]

> in case I'm integrating the SPDK into my existing application, I have to change the way I'm spawning thread in the whole application, right?

That's correct.
Regards,
D.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ernest Zed
Sent: Wednesday, February 21, 2018 1:18 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] core_mask and spawning new threads

Thanks for the explanation. Just to make it clear, in case I'm integrating the SPDK into my existing application, I have to change the way I'm spawning thread in the whole application, right?
Sincerely,
E.

On Wed, Feb 21, 2018 at 1:43 PM, Stojaczyk, DariuszX <dariuszx.stojaczyk(a)intel.com<mailto:dariuszx.stojaczyk(a)intel.com>> wrote:
Hi Ernest,
This is expected behavior. SPDK starts a fixed number of affinitized threads on startup, and the core mask parameter you've mentioned is what specifies the affinity and number of those. Each thread is bound to a single CPU. The one thread that called `spdk_env_init` becomes a master thread and is affinitized to the `opts.master_core`. To spawn additional POSIX threads from any SPDK-affinitized thread, you should use `spdk_call_unaffinitized()` (include/spdk/env.h).

(There's also an `spdk_unaffinitize_thread()`, but it's restricted to very specific workloads and probably shouldn't be used at all.)
Best regards,
D.

From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Ernest Zed
Sent: Wednesday, February 21, 2018 10:47 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] core_mask and spawning new threads

Hi all,
I see that all threads I create within my application are running on the same core. Looks like core_mask is the right parameter to adjust, however I dont see core usage affected by changing the core_mask on spdk_env_opts
The code:
    spdk_env_opts opts;
    spdk_env_opts_init(&opts);
    opts.name<http://opts.name> = "blah";
    opts.core_mask = "0xFFFF";
    opts.master_core = 1;
    opts.shm_id = 0;
    if (spdk_env_init(&opts) < 0) {
        throw std::runtime_error("Unable to initialize SPDK env");
    }
Whatever value I assign to the core_mask everything runs on the same core defined in master_core. If I set affinity to my threads manually everything works as expected. Is it a spdk_env_init bug? Am I doing something wrong?
Sincerely,
Ernest

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 13119 bytes --]

Re: [SPDK] core_mask and spawning new threads

[Ernest Zed]

[-- Attachment #1: Type: text/plain, Size: 2262 bytes --]

Thanks for the explanation. Just to make it clear, in case I'm integrating
the SPDK into my existing application, I have to change the way I'm
spawning thread in the whole application, right?
Sincerely,
E.

On Wed, Feb 21, 2018 at 1:43 PM, Stojaczyk, DariuszX <
dariuszx.stojaczyk(a)intel.com> wrote:

> Hi Ernest,
>
> This is expected behavior. SPDK starts a fixed number of affinitized
> threads on startup, and the core mask parameter you've mentioned is what
> specifies the affinity and number of those. Each thread is bound to a
> single CPU. The one thread that called `spdk_env_init` becomes a master
> thread and is affinitized to the `opts.master_core`. To spawn additional
> POSIX threads from any SPDK-affinitized thread, you should use
> `spdk_call_unaffinitized()` (include/spdk/env.h).
>
>
>
> (There's also an `spdk_unaffinitize_thread()`, but it's restricted to very
> specific workloads and probably shouldn't be used at all.)
>
> Best regards,
>
> D.
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Ernest Zed
> *Sent:* Wednesday, February 21, 2018 10:47 AM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* [SPDK] core_mask and spawning new threads
>
>
>
> Hi all,
>
> I see that all threads I create within my application are running on the
> same core. Looks like core_mask is the right parameter to adjust, however I
> dont see core usage affected by changing the core_mask on spdk_env_opts
>
> The code:
>
>     spdk_env_opts opts;
>
>     spdk_env_opts_init(&opts);
>
>     opts.name = "blah";
>
>     opts.core_mask = "0xFFFF";
>
>     opts.master_core = 1;
>
>     opts.shm_id = 0;
>
>     if (spdk_env_init(&opts) < 0) {
>
>         throw std::runtime_error("Unable to initialize SPDK env");
>
>     }
>
> Whatever value I assign to the core_mask everything runs on the same core
> defined in master_core. If I set affinity to my threads manually everything
> works as expected. Is it a spdk_env_init bug? Am I doing something wrong?
>
> Sincerely,
>
> Ernest
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 7149 bytes --]

Re: [SPDK] core_mask and spawning new threads

[Stojaczyk, DariuszX]

[-- Attachment #1: Type: text/plain, Size: 1656 bytes --]

Hi Ernest,
This is expected behavior. SPDK starts a fixed number of affinitized threads on startup, and the core mask parameter you've mentioned is what specifies the affinity and number of those. Each thread is bound to a single CPU. The one thread that called `spdk_env_init` becomes a master thread and is affinitized to the `opts.master_core`. To spawn additional POSIX threads from any SPDK-affinitized thread, you should use `spdk_call_unaffinitized()` (include/spdk/env.h).

(There's also an `spdk_unaffinitize_thread()`, but it's restricted to very specific workloads and probably shouldn't be used at all.)
Best regards,
D.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ernest Zed
Sent: Wednesday, February 21, 2018 10:47 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] core_mask and spawning new threads

Hi all,
I see that all threads I create within my application are running on the same core. Looks like core_mask is the right parameter to adjust, however I dont see core usage affected by changing the core_mask on spdk_env_opts
The code:
    spdk_env_opts opts;
    spdk_env_opts_init(&opts);
    opts.name<http://opts.name> = "blah";
    opts.core_mask = "0xFFFF";
    opts.master_core = 1;
    opts.shm_id = 0;
    if (spdk_env_init(&opts) < 0) {
        throw std::runtime_error("Unable to initialize SPDK env");
    }
Whatever value I assign to the core_mask everything runs on the same core defined in master_core. If I set affinity to my threads manually everything works as expected. Is it a spdk_env_init bug? Am I doing something wrong?
Sincerely,
Ernest

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 7598 bytes --]

[SPDK] core_mask and spawning new threads

[Ernest Zed]

[-- Attachment #1: Type: text/plain, Size: 766 bytes --]

Hi all,
I see that all threads I create within my application are running on the
same core. Looks like core_mask is the right parameter to adjust, however I
dont see core usage affected by changing the core_mask on spdk_env_opts
The code:
    spdk_env_opts opts;
    spdk_env_opts_init(&opts);
    opts.name = "blah";
    opts.core_mask = "0xFFFF";
    opts.master_core = 1;
    opts.shm_id = 0;
    if (spdk_env_init(&opts) < 0) {
        throw std::runtime_error("Unable to initialize SPDK env");
    }
Whatever value I assign to the core_mask everything runs on the same core
defined in master_core. If I set affinity to my threads manually everything
works as expected. Is it a spdk_env_init bug? Am I doing something wrong?
Sincerely,
Ernest

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 1011 bytes --]