* Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
@ 2018-09-07 14:00 Luse, Paul E
0 siblings, 0 replies; 15+ messages in thread
From: Luse, Paul E @ 2018-09-07 14:00 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 3261 bytes --]
That's a great question! I haven't used fio in forever so am not familiar with the startup sequence, I'll take a look in a bit.
Thx
Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Wodkowski, PawelX
Sent: Friday, September 7, 2018 6:57 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
Are you sure rte_vdev_init() is called after DPDK is initialized or you (fio) is calling rte_vdev_init() from non-DPDK thread?
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul
> E
> Sent: Friday, September 7, 2018 3:27 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] socket_id difference under fio plugin vs bdevio or
> bdevperf
>
>
> In the crypto patch when I init the aesni virtual device I've been
> passing in NULL for optional args and it's worked fine. When running
> w/fio however, the aesni driver took the null args to mean "socket
> any" which should be fine but for whatever reason its arg parser doesn't like that value.
>
> So this works in bdevio/bdevperf but does not in FIO
>
> rc = rte_vdev_init(AESNI_MB, NULL);
>
> Hardcoding the only arg I care about, socket_id, because it's the one
> that the aesni driver doesn't like not being specified works in fio as well as our apps:
>
> rc = rte_vdev_init(AESNI_MB, "socket_id=0");
>
> So using spdk calls I tried this:
>
> snprintf(socket_id, 32, "socket_id=%d",
> spdk_env_get_socket_id(spdk_env_get_current_core()));
> rc = rte_vdev_init(AESNI_MB, socket_id);
>
> and once again woks with bdevio and fails with fio. Looking at the
> differences in the code here what I see:
>
> running with bdevio:
>
> │173 rte_lcore_to_socket_id(unsigned lcore_id)
> │
> │174 { │
> >│175 return lcore_config[lcore_id].socket_id;
> │
> │176 }
>
> (gdb) p lcore_config[0]
> $1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0},
> pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
> socket_id = 0, core_id = 0, core_index = 0, cpuset = {__bits = {1, 0
> <repeats
> 15 times>}}, core_role = 0 '\000'}
>
> and running with fio:
>
> $1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0},
> pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
> socket_id = 4294967295, core_id = 0, core_index = 0, cpuset =
> {__bits = {18446744073709551615, 16777215, 0 <repeats 14 times>}},
> core_role = 0 '\000'}
>
> Notice the difference. Any idea why running with fio plugin vs bdevio
> or bdevperf would cause this??
>
> Thanks!
> Paul
>
>
>
>
>
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
@ 2018-09-09 23:41 Luse, Paul E
0 siblings, 0 replies; 15+ messages in thread
From: Luse, Paul E @ 2018-09-09 23:41 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 17549 bytes --]
Actually, I have a fix for this just in the crypto code by not using rte_cryptodev_socket_id() to set the socket_id for queue config but instead letting the value of -1 'fall through' by not specifying a socket at all when I call rte_vdev_init() avoids storing it in that uint8 and things work.
So, this is no longer an issue with some minor changes that I'm about to push.
-Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Sunday, September 9, 2018 10:00 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
Thinking about this some more, maybe this is actually how it should behave. If we are going to unaffinitize and pass in a cpuset that includes cpus that span numa nodes then I think it would make sense that the lcore_config structure specify SOCKET_ID_ANY right?
Assuming that's the case, I looked into how the crypto init code is handling this value for socket because ultimately it’s a call to rte_zmalloc_socket() that is failing when I don't specify a valid socket #. That API is supposed to ignore the socket parm if its set to -1 however it's not making it through as -1 because one of the interim data structures, shown below, stores the socket_id as a uint8 so by the time it makes it to the zmalloc call it looks like 255. Whether my assumption up above is correct or not this looks like either a bug or a specific choice not to support SOCKET_ID_ANY for configuring crypto devices. I’ll follow-up with the DPDK crypto folks on this after I hear back from at least one of you guys wrt your thoughts on this. Thanks!!
rte_cryptodev_data Struct Reference
#include <rte_cryptodev.h<http://doc.dpdk.org/api/rte__cryptodev_8h_source.html>>
Data Fields
uint8_t
dev_id<http://doc.dpdk.org/api/structrte__cryptodev__data.html#a360603720ae8b2c0e4931dd0f648c504>
uint8_t
socket_id<http://doc.dpdk.org/api/structrte__cryptodev__data.html#ac122e8ef3accab126b7ba007a38c1bbd>
char
name<http://doc.dpdk.org/api/structrte__cryptodev__data.html#af602850a0b59884cee4faafde1369298> [RTE_CRYPTODEV_NAME_MAX_LEN<http://doc.dpdk.org/api/rte__cryptodev_8h.html#ab743b8ca147b449954c2a9549c29a6ed>]
__extension__ uint8_t
dev_started<http://doc.dpdk.org/api/structrte__cryptodev__data.html#ac287d497d80fa9ddfa9e11cd3a29ed36>: 1
struct rte_mempool<http://doc.dpdk.org/api/structrte__mempool.html> *
session_pool<http://doc.dpdk.org/api/structrte__cryptodev__data.html#aa4b418564936ee79df48540804b72907>
void **
queue_pairs<http://doc.dpdk.org/api/structrte__cryptodev__data.html#a1ceecb8f75e74de1c403e2f67dc1fa7f>
uint16_t
nb_queue_pairs<http://doc.dpdk.org/api/structrte__cryptodev__data.html#ab6b5a902514f586f964c368ef5993cd7>
void *
dev_private<http://doc.dpdk.org/api/structrte__cryptodev__data.html#a69817252ce0a09bc3ddd99d9b3dd1a16>
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Sunday, September 9, 2018 9:19 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
Darek/Jim,
This isn't going to be the funnest/easiet to follow email to read but it does explain how the socket_id is being set different between FIO and BDEVIO. Can't say as I fully understand why yet but please take a shot at interpreting my walk-thru and let me know what you think. Don't yet know if this is a bug or maybe something that we're missing.
Thanks!!
Paul
DPDK init: (and what happens with bdevio, etc):
----------------------------------------------------------------
So in short the DPDK global struct lcore_config (where the socket_id in question is ultimately being retrieved from) is set up 'correctly' or, as expected, by rte_eal_cpu_init (). As a side note, because there's some code called later that's relevant, also as part of the DPDK init, rte_thread_set_affinity() is called on the cpuset that is part of the lcore_config. That cpusetp includes cpu 0 only.
rte_thread_set_affinity() then calls eal_cpuset_socket_id() which loops through all cpus and gets their socket node info from /sys which, based on this cpusetp is cpu 0. Bottom line is that following DPDK init, lcore_config is accurate.
With FIO:
-------------
After DPDK int, when the fio plugin calls spdk_unaffinitize_thread(), the function rte_thread_set_affinity() is called again but with a cpusetp that includes all cpus. Below is a snippet from eal_cpuset_socket_id() (called from rte_thread_set_affinity ) and since this time, as opposed to DPDK init time, the cpuset includes all cpus, when we get to cpu #22 (on my system) we move over to socket 1 instead of socket 0 so return at line 58 with a socketed of SOCKET_ID_ANY
│49 do { │
│50 if (!CPU_ISSET(cpu, cpusetp)) │
│51 continue; │
│52 │
│53 if (socket_id == SOCKET_ID_ANY) │
│54 socket_id = eal_cpu_socket_id(cpu); │
│55 │
│56 sid = eal_cpu_socket_id(cpu); │
│57 if (socket_id != sid) { │
B+>│58 socket_id = SOCKET_ID_ANY; │
│59 break; │
│60 } │
│61 │
> │62 } while (++cpu < RTE_MAX_LCORE); │
│63 │
│64 return socket_id;
This snippet is from the calling code for the function above (called from line 84) and because RTE_PER_LCORE(_socket_id) is set to SOCKET_ID_ANY, the global structure in question for locre 0 is then updated from its previous value of 0 (from the DPDK init code) to what I'm seeing at line 93. I don't know what the comment on line 92 actually means but at first glance I think it's maybe saying "set to ANY for now because something somewhere will update this later?" I dunno. This is how the socket_id is getting set this way though. I'll see if I can dig more into that comment or jump on the DPDK dist list if nobody here has any tips.
│82 /* store socket_id in TLS for quick access */ │
│83 RTE_PER_LCORE(_socket_id) = │
│84 eal_cpuset_socket_id(cpusetp); │
│85 │
│86 /* store cpuset in TLS for quick access */ │
>│87 memmove(&RTE_PER_LCORE(_cpuset), cpusetp, │
│88 sizeof(rte_cpuset_t)); │
│89 │
│90 lcore_id = rte_lcore_id(); │
│91 if (lcore_id != (unsigned)LCORE_ID_ANY) { │
│92 /* EAL thread will update lcore_config */ │
│93 lcore_config[lcore_id].socket_id = RTE_PER_LCORE(_socket_id); │
│94 memmove(&lcore_config[lcore_id].cpuset, cpusetp, │
│95 sizeof(rte_cpuset_t)); │
│96 }
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Sunday, September 9, 2018 6:42 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
Hi Darek,
I can close on this from my end, thanks though!
-Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Stojaczyk, Dariusz
Sent: Saturday, September 8, 2018 4:10 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
Apparently the socket id == -1 you were getting did not mean unitialized. It was implicitly set to SOCKET_ID_ANY by rte_thread_set_affinity() in spdk_unaffinitize_thread(). Since DPDK allows SOCKET_ID_ANY as an possible socket id value, then it must be the vdev or the crypto stuff that's broken. Do you mind sharing the exact error log? I don't have the setup to reproduce this myself.
D.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, September 7, 2018 5:07 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
For sure, I'll wait for Derek's response on what he was planning on doing tomorrow and pick anything up from there...
Thx
Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, September 7, 2018 8:05 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
I like Darek’s suggestions. I do think this is a DPDK bug though. Other DPDK calls like rte_malloc() work from a non-DPDK thread. Paul – can you dig into this further?
-Jim
On 9/7/18, 7:39 AM, "SPDK on behalf of Luse, Paul E" <spdk-bounces(a)lists.01.org on behalf of paul.e.luse(a)intel.com> wrote:
That's really good to know, thanks! I'm hardcoded to 0 now which is fine for the remainder of testing I think, let me know how it goes with get_mempolicy
Thx,
Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 7:31 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
I wasn't registered on the mailing list with this email before, so I registered just now and am resending this message.
D.
-----Original Message-----
From: Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 4:04 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: RE: socket_id difference under fio plugin vs bdevio or bdevperf
HI Paul,
Unlike bdevperf and bdevio, fio_plugin doesn't really use DPDK abstraction for multi-threading. It uses standard pthreads for most of its work and that's why you're seeing socket id 4294967295 (unsigned -1). This value is simply uninitialized. To get socket id in a portable way, you might want to use get_mempolicy. In fact, we should probably use it somewhere within our SPDK APIs. I'll look into adding it tomorrow.
D.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, September 7, 2018 3:27 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
In the crypto patch when I init the aesni virtual device I've been passing in NULL for optional args and it's worked fine. When running w/fio however, the aesni driver took the null args to mean "socket any" which should be fine but for whatever reason its arg parser doesn't like that value.
So this works in bdevio/bdevperf but does not in FIO
rc = rte_vdev_init(AESNI_MB, NULL);
Hardcoding the only arg I care about, socket_id, because it's the one that the aesni driver doesn't like not being specified works in fio as well as our apps:
rc = rte_vdev_init(AESNI_MB, "socket_id=0");
So using spdk calls I tried this:
snprintf(socket_id, 32, "socket_id=%d",
spdk_env_get_socket_id(spdk_env_get_current_core()));
rc = rte_vdev_init(AESNI_MB, socket_id);
and once again woks with bdevio and fails with fio. Looking at the differences in the code here what I see:
running with bdevio:
│173 rte_lcore_to_socket_id(unsigned lcore_id) │
│174 { │
>│175 return lcore_config[lcore_id].socket_id; │
│176 }
(gdb) p lcore_config[0]
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 0, core_id = 0, core_index = 0, cpuset = {__bits = {1, 0 <repeats 15 times>}}, core_role = 0 '\000'}
and running with fio:
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 4294967295, core_id = 0, core_index = 0, cpuset = {__bits = {18446744073709551615, 16777215, 0 <repeats 14 times>}},
core_role = 0 '\000'}
Notice the difference. Any idea why running with fio plugin vs bdevio or bdevperf would cause this??
Thanks!
Paul
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
@ 2018-09-09 17:00 Luse, Paul E
0 siblings, 0 replies; 15+ messages in thread
From: Luse, Paul E @ 2018-09-09 17:00 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 16736 bytes --]
Thinking about this some more, maybe this is actually how it should behave. If we are going to unaffinitize and pass in a cpuset that includes cpus that span numa nodes then I think it would make sense that the lcore_config structure specify SOCKET_ID_ANY right?
Assuming that's the case, I looked into how the crypto init code is handling this value for socket because ultimately it’s a call to rte_zmalloc_socket() that is failing when I don't specify a valid socket #. That API is supposed to ignore the socket parm if its set to -1 however it's not making it through as -1 because one of the interim data structures, shown below, stores the socket_id as a uint8 so by the time it makes it to the zmalloc call it looks like 255. Whether my assumption up above is correct or not this looks like either a bug or a specific choice not to support SOCKET_ID_ANY for configuring crypto devices. I’ll follow-up with the DPDK crypto folks on this after I hear back from at least one of you guys wrt your thoughts on this. Thanks!!
rte_cryptodev_data Struct Reference
#include <rte_cryptodev.h<http://doc.dpdk.org/api/rte__cryptodev_8h_source.html>>
Data Fields
uint8_t
dev_id<http://doc.dpdk.org/api/structrte__cryptodev__data.html#a360603720ae8b2c0e4931dd0f648c504>
uint8_t
socket_id<http://doc.dpdk.org/api/structrte__cryptodev__data.html#ac122e8ef3accab126b7ba007a38c1bbd>
char
name<http://doc.dpdk.org/api/structrte__cryptodev__data.html#af602850a0b59884cee4faafde1369298> [RTE_CRYPTODEV_NAME_MAX_LEN<http://doc.dpdk.org/api/rte__cryptodev_8h.html#ab743b8ca147b449954c2a9549c29a6ed>]
__extension__ uint8_t
dev_started<http://doc.dpdk.org/api/structrte__cryptodev__data.html#ac287d497d80fa9ddfa9e11cd3a29ed36>: 1
struct rte_mempool<http://doc.dpdk.org/api/structrte__mempool.html> *
session_pool<http://doc.dpdk.org/api/structrte__cryptodev__data.html#aa4b418564936ee79df48540804b72907>
void **
queue_pairs<http://doc.dpdk.org/api/structrte__cryptodev__data.html#a1ceecb8f75e74de1c403e2f67dc1fa7f>
uint16_t
nb_queue_pairs<http://doc.dpdk.org/api/structrte__cryptodev__data.html#ab6b5a902514f586f964c368ef5993cd7>
void *
dev_private<http://doc.dpdk.org/api/structrte__cryptodev__data.html#a69817252ce0a09bc3ddd99d9b3dd1a16>
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Sunday, September 9, 2018 9:19 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
Darek/Jim,
This isn't going to be the funnest/easiet to follow email to read but it does explain how the socket_id is being set different between FIO and BDEVIO. Can't say as I fully understand why yet but please take a shot at interpreting my walk-thru and let me know what you think. Don't yet know if this is a bug or maybe something that we're missing.
Thanks!!
Paul
DPDK init: (and what happens with bdevio, etc):
----------------------------------------------------------------
So in short the DPDK global struct lcore_config (where the socket_id in question is ultimately being retrieved from) is set up 'correctly' or, as expected, by rte_eal_cpu_init (). As a side note, because there's some code called later that's relevant, also as part of the DPDK init, rte_thread_set_affinity() is called on the cpuset that is part of the lcore_config. That cpusetp includes cpu 0 only.
rte_thread_set_affinity() then calls eal_cpuset_socket_id() which loops through all cpus and gets their socket node info from /sys which, based on this cpusetp is cpu 0. Bottom line is that following DPDK init, lcore_config is accurate.
With FIO:
-------------
After DPDK int, when the fio plugin calls spdk_unaffinitize_thread(), the function rte_thread_set_affinity() is called again but with a cpusetp that includes all cpus. Below is a snippet from eal_cpuset_socket_id() (called from rte_thread_set_affinity ) and since this time, as opposed to DPDK init time, the cpuset includes all cpus, when we get to cpu #22 (on my system) we move over to socket 1 instead of socket 0 so return at line 58 with a socketed of SOCKET_ID_ANY
│49 do { │
│50 if (!CPU_ISSET(cpu, cpusetp)) │
│51 continue; │
│52 │
│53 if (socket_id == SOCKET_ID_ANY) │
│54 socket_id = eal_cpu_socket_id(cpu); │
│55 │
│56 sid = eal_cpu_socket_id(cpu); │
│57 if (socket_id != sid) { │
B+>│58 socket_id = SOCKET_ID_ANY; │
│59 break; │
│60 } │
│61 │
> │62 } while (++cpu < RTE_MAX_LCORE); │
│63 │
│64 return socket_id;
This snippet is from the calling code for the function above (called from line 84) and because RTE_PER_LCORE(_socket_id) is set to SOCKET_ID_ANY, the global structure in question for locre 0 is then updated from its previous value of 0 (from the DPDK init code) to what I'm seeing at line 93. I don't know what the comment on line 92 actually means but at first glance I think it's maybe saying "set to ANY for now because something somewhere will update this later?" I dunno. This is how the socket_id is getting set this way though. I'll see if I can dig more into that comment or jump on the DPDK dist list if nobody here has any tips.
│82 /* store socket_id in TLS for quick access */ │
│83 RTE_PER_LCORE(_socket_id) = │
│84 eal_cpuset_socket_id(cpusetp); │
│85 │
│86 /* store cpuset in TLS for quick access */ │
>│87 memmove(&RTE_PER_LCORE(_cpuset), cpusetp, │
│88 sizeof(rte_cpuset_t)); │
│89 │
│90 lcore_id = rte_lcore_id(); │
│91 if (lcore_id != (unsigned)LCORE_ID_ANY) { │
│92 /* EAL thread will update lcore_config */ │
│93 lcore_config[lcore_id].socket_id = RTE_PER_LCORE(_socket_id); │
│94 memmove(&lcore_config[lcore_id].cpuset, cpusetp, │
│95 sizeof(rte_cpuset_t)); │
│96 }
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Sunday, September 9, 2018 6:42 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
Hi Darek,
I can close on this from my end, thanks though!
-Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Stojaczyk, Dariusz
Sent: Saturday, September 8, 2018 4:10 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
Apparently the socket id == -1 you were getting did not mean unitialized. It was implicitly set to SOCKET_ID_ANY by rte_thread_set_affinity() in spdk_unaffinitize_thread(). Since DPDK allows SOCKET_ID_ANY as an possible socket id value, then it must be the vdev or the crypto stuff that's broken. Do you mind sharing the exact error log? I don't have the setup to reproduce this myself.
D.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, September 7, 2018 5:07 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
For sure, I'll wait for Derek's response on what he was planning on doing tomorrow and pick anything up from there...
Thx
Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, September 7, 2018 8:05 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
I like Darek’s suggestions. I do think this is a DPDK bug though. Other DPDK calls like rte_malloc() work from a non-DPDK thread. Paul – can you dig into this further?
-Jim
On 9/7/18, 7:39 AM, "SPDK on behalf of Luse, Paul E" <spdk-bounces(a)lists.01.org on behalf of paul.e.luse(a)intel.com> wrote:
That's really good to know, thanks! I'm hardcoded to 0 now which is fine for the remainder of testing I think, let me know how it goes with get_mempolicy
Thx,
Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 7:31 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
I wasn't registered on the mailing list with this email before, so I registered just now and am resending this message.
D.
-----Original Message-----
From: Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 4:04 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: RE: socket_id difference under fio plugin vs bdevio or bdevperf
HI Paul,
Unlike bdevperf and bdevio, fio_plugin doesn't really use DPDK abstraction for multi-threading. It uses standard pthreads for most of its work and that's why you're seeing socket id 4294967295 (unsigned -1). This value is simply uninitialized. To get socket id in a portable way, you might want to use get_mempolicy. In fact, we should probably use it somewhere within our SPDK APIs. I'll look into adding it tomorrow.
D.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, September 7, 2018 3:27 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
In the crypto patch when I init the aesni virtual device I've been passing in NULL for optional args and it's worked fine. When running w/fio however, the aesni driver took the null args to mean "socket any" which should be fine but for whatever reason its arg parser doesn't like that value.
So this works in bdevio/bdevperf but does not in FIO
rc = rte_vdev_init(AESNI_MB, NULL);
Hardcoding the only arg I care about, socket_id, because it's the one that the aesni driver doesn't like not being specified works in fio as well as our apps:
rc = rte_vdev_init(AESNI_MB, "socket_id=0");
So using spdk calls I tried this:
snprintf(socket_id, 32, "socket_id=%d",
spdk_env_get_socket_id(spdk_env_get_current_core()));
rc = rte_vdev_init(AESNI_MB, socket_id);
and once again woks with bdevio and fails with fio. Looking at the differences in the code here what I see:
running with bdevio:
│173 rte_lcore_to_socket_id(unsigned lcore_id) │
│174 { │
>│175 return lcore_config[lcore_id].socket_id; │
│176 }
(gdb) p lcore_config[0]
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 0, core_id = 0, core_index = 0, cpuset = {__bits = {1, 0 <repeats 15 times>}}, core_role = 0 '\000'}
and running with fio:
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 4294967295, core_id = 0, core_index = 0, cpuset = {__bits = {18446744073709551615, 16777215, 0 <repeats 14 times>}},
core_role = 0 '\000'}
Notice the difference. Any idea why running with fio plugin vs bdevio or bdevperf would cause this??
Thanks!
Paul
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
@ 2018-09-09 16:18 Luse, Paul E
0 siblings, 0 replies; 15+ messages in thread
From: Luse, Paul E @ 2018-09-09 16:18 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 13710 bytes --]
Darek/Jim,
This isn't going to be the funnest/easiet to follow email to read but it does explain how the socket_id is being set different between FIO and BDEVIO. Can't say as I fully understand why yet but please take a shot at interpreting my walk-thru and let me know what you think. Don't yet know if this is a bug or maybe something that we're missing.
Thanks!!
Paul
DPDK init: (and what happens with bdevio, etc):
----------------------------------------------------------------
So in short the DPDK global struct lcore_config (where the socket_id in question is ultimately being retrieved from) is set up 'correctly' or, as expected, by rte_eal_cpu_init (). As a side note, because there's some code called later that's relevant, also as part of the DPDK init, rte_thread_set_affinity() is called on the cpuset that is part of the lcore_config. That cpusetp includes cpu 0 only.
rte_thread_set_affinity() then calls eal_cpuset_socket_id() which loops through all cpus and gets their socket node info from /sys which, based on this cpusetp is cpu 0. Bottom line is that following DPDK init, lcore_config is accurate.
With FIO:
-------------
After DPDK int, when the fio plugin calls spdk_unaffinitize_thread(), the function rte_thread_set_affinity() is called again but with a cpusetp that includes all cpus. Below is a snippet from eal_cpuset_socket_id() (called from rte_thread_set_affinity ) and since this time, as opposed to DPDK init time, the cpuset includes all cpus, when we get to cpu #22 (on my system) we move over to socket 1 instead of socket 0 so return at line 58 with a socketed of SOCKET_ID_ANY
│49 do { │
│50 if (!CPU_ISSET(cpu, cpusetp)) │
│51 continue; │
│52 │
│53 if (socket_id == SOCKET_ID_ANY) │
│54 socket_id = eal_cpu_socket_id(cpu); │
│55 │
│56 sid = eal_cpu_socket_id(cpu); │
│57 if (socket_id != sid) { │
B+>│58 socket_id = SOCKET_ID_ANY; │
│59 break; │
│60 } │
│61 │
> │62 } while (++cpu < RTE_MAX_LCORE); │
│63 │
│64 return socket_id;
This snippet is from the calling code for the function above (called from line 84) and because RTE_PER_LCORE(_socket_id) is set to SOCKET_ID_ANY, the global structure in question for locre 0 is then updated from its previous value of 0 (from the DPDK init code) to what I'm seeing at line 93. I don't know what the comment on line 92 actually means but at first glance I think it's maybe saying "set to ANY for now because something somewhere will update this later?" I dunno. This is how the socket_id is getting set this way though. I'll see if I can dig more into that comment or jump on the DPDK dist list if nobody here has any tips.
│82 /* store socket_id in TLS for quick access */ │
│83 RTE_PER_LCORE(_socket_id) = │
│84 eal_cpuset_socket_id(cpusetp); │
│85 │
│86 /* store cpuset in TLS for quick access */ │
>│87 memmove(&RTE_PER_LCORE(_cpuset), cpusetp, │
│88 sizeof(rte_cpuset_t)); │
│89 │
│90 lcore_id = rte_lcore_id(); │
│91 if (lcore_id != (unsigned)LCORE_ID_ANY) { │
│92 /* EAL thread will update lcore_config */ │
│93 lcore_config[lcore_id].socket_id = RTE_PER_LCORE(_socket_id); │
│94 memmove(&lcore_config[lcore_id].cpuset, cpusetp, │
│95 sizeof(rte_cpuset_t)); │
│96 }
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Sunday, September 9, 2018 6:42 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
Hi Darek,
I can close on this from my end, thanks though!
-Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Stojaczyk, Dariusz
Sent: Saturday, September 8, 2018 4:10 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
Apparently the socket id == -1 you were getting did not mean unitialized. It was implicitly set to SOCKET_ID_ANY by rte_thread_set_affinity() in spdk_unaffinitize_thread(). Since DPDK allows SOCKET_ID_ANY as an possible socket id value, then it must be the vdev or the crypto stuff that's broken. Do you mind sharing the exact error log? I don't have the setup to reproduce this myself.
D.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, September 7, 2018 5:07 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
For sure, I'll wait for Derek's response on what he was planning on doing tomorrow and pick anything up from there...
Thx
Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, September 7, 2018 8:05 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
I like Darek’s suggestions. I do think this is a DPDK bug though. Other DPDK calls like rte_malloc() work from a non-DPDK thread. Paul – can you dig into this further?
-Jim
On 9/7/18, 7:39 AM, "SPDK on behalf of Luse, Paul E" <spdk-bounces(a)lists.01.org on behalf of paul.e.luse(a)intel.com> wrote:
That's really good to know, thanks! I'm hardcoded to 0 now which is fine for the remainder of testing I think, let me know how it goes with get_mempolicy
Thx,
Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 7:31 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
I wasn't registered on the mailing list with this email before, so I registered just now and am resending this message.
D.
-----Original Message-----
From: Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 4:04 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: RE: socket_id difference under fio plugin vs bdevio or bdevperf
HI Paul,
Unlike bdevperf and bdevio, fio_plugin doesn't really use DPDK abstraction for multi-threading. It uses standard pthreads for most of its work and that's why you're seeing socket id 4294967295 (unsigned -1). This value is simply uninitialized. To get socket id in a portable way, you might want to use get_mempolicy. In fact, we should probably use it somewhere within our SPDK APIs. I'll look into adding it tomorrow.
D.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, September 7, 2018 3:27 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
In the crypto patch when I init the aesni virtual device I've been passing in NULL for optional args and it's worked fine. When running w/fio however, the aesni driver took the null args to mean "socket any" which should be fine but for whatever reason its arg parser doesn't like that value.
So this works in bdevio/bdevperf but does not in FIO
rc = rte_vdev_init(AESNI_MB, NULL);
Hardcoding the only arg I care about, socket_id, because it's the one that the aesni driver doesn't like not being specified works in fio as well as our apps:
rc = rte_vdev_init(AESNI_MB, "socket_id=0");
So using spdk calls I tried this:
snprintf(socket_id, 32, "socket_id=%d",
spdk_env_get_socket_id(spdk_env_get_current_core()));
rc = rte_vdev_init(AESNI_MB, socket_id);
and once again woks with bdevio and fails with fio. Looking at the differences in the code here what I see:
running with bdevio:
│173 rte_lcore_to_socket_id(unsigned lcore_id) │
│174 { │
>│175 return lcore_config[lcore_id].socket_id; │
│176 }
(gdb) p lcore_config[0]
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 0, core_id = 0, core_index = 0, cpuset = {__bits = {1, 0 <repeats 15 times>}}, core_role = 0 '\000'}
and running with fio:
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 4294967295, core_id = 0, core_index = 0, cpuset = {__bits = {18446744073709551615, 16777215, 0 <repeats 14 times>}},
core_role = 0 '\000'}
Notice the difference. Any idea why running with fio plugin vs bdevio or bdevperf would cause this??
Thanks!
Paul
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
@ 2018-09-09 13:41 Luse, Paul E
0 siblings, 0 replies; 15+ messages in thread
From: Luse, Paul E @ 2018-09-09 13:41 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 6659 bytes --]
Hi Darek,
I can close on this from my end, thanks though!
-Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Stojaczyk, Dariusz
Sent: Saturday, September 8, 2018 4:10 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
Apparently the socket id == -1 you were getting did not mean unitialized. It was implicitly set to SOCKET_ID_ANY by rte_thread_set_affinity() in spdk_unaffinitize_thread(). Since DPDK allows SOCKET_ID_ANY as an possible socket id value, then it must be the vdev or the crypto stuff that's broken. Do you mind sharing the exact error log? I don't have the setup to reproduce this myself.
D.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, September 7, 2018 5:07 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
For sure, I'll wait for Derek's response on what he was planning on doing tomorrow and pick anything up from there...
Thx
Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, September 7, 2018 8:05 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
I like Darek’s suggestions. I do think this is a DPDK bug though. Other DPDK calls like rte_malloc() work from a non-DPDK thread. Paul – can you dig into this further?
-Jim
On 9/7/18, 7:39 AM, "SPDK on behalf of Luse, Paul E" <spdk-bounces(a)lists.01.org on behalf of paul.e.luse(a)intel.com> wrote:
That's really good to know, thanks! I'm hardcoded to 0 now which is fine for the remainder of testing I think, let me know how it goes with get_mempolicy
Thx,
Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 7:31 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
I wasn't registered on the mailing list with this email before, so I registered just now and am resending this message.
D.
-----Original Message-----
From: Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 4:04 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: RE: socket_id difference under fio plugin vs bdevio or bdevperf
HI Paul,
Unlike bdevperf and bdevio, fio_plugin doesn't really use DPDK abstraction for multi-threading. It uses standard pthreads for most of its work and that's why you're seeing socket id 4294967295 (unsigned -1). This value is simply uninitialized. To get socket id in a portable way, you might want to use get_mempolicy. In fact, we should probably use it somewhere within our SPDK APIs. I'll look into adding it tomorrow.
D.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, September 7, 2018 3:27 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
In the crypto patch when I init the aesni virtual device I've been passing in NULL for optional args and it's worked fine. When running w/fio however, the aesni driver took the null args to mean "socket any" which should be fine but for whatever reason its arg parser doesn't like that value.
So this works in bdevio/bdevperf but does not in FIO
rc = rte_vdev_init(AESNI_MB, NULL);
Hardcoding the only arg I care about, socket_id, because it's the one that the aesni driver doesn't like not being specified works in fio as well as our apps:
rc = rte_vdev_init(AESNI_MB, "socket_id=0");
So using spdk calls I tried this:
snprintf(socket_id, 32, "socket_id=%d",
spdk_env_get_socket_id(spdk_env_get_current_core()));
rc = rte_vdev_init(AESNI_MB, socket_id);
and once again woks with bdevio and fails with fio. Looking at the differences in the code here what I see:
running with bdevio:
│173 rte_lcore_to_socket_id(unsigned lcore_id) │
│174 { │
>│175 return lcore_config[lcore_id].socket_id; │
│176 }
(gdb) p lcore_config[0]
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 0, core_id = 0, core_index = 0, cpuset = {__bits = {1, 0 <repeats 15 times>}}, core_role = 0 '\000'}
and running with fio:
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 4294967295, core_id = 0, core_index = 0, cpuset = {__bits = {18446744073709551615, 16777215, 0 <repeats 14 times>}},
core_role = 0 '\000'}
Notice the difference. Any idea why running with fio plugin vs bdevio or bdevperf would cause this??
Thanks!
Paul
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
@ 2018-09-08 23:09 Stojaczyk, Dariusz
0 siblings, 0 replies; 15+ messages in thread
From: Stojaczyk, Dariusz @ 2018-09-08 23:09 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 6155 bytes --]
Apparently the socket id == -1 you were getting did not mean unitialized. It was implicitly set to SOCKET_ID_ANY by rte_thread_set_affinity() in spdk_unaffinitize_thread(). Since DPDK allows SOCKET_ID_ANY as an possible socket id value, then it must be the vdev or the crypto stuff that's broken. Do you mind sharing the exact error log? I don't have the setup to reproduce this myself.
D.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, September 7, 2018 5:07 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
For sure, I'll wait for Derek's response on what he was planning on doing tomorrow and pick anything up from there...
Thx
Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, September 7, 2018 8:05 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
I like Darek’s suggestions. I do think this is a DPDK bug though. Other DPDK calls like rte_malloc() work from a non-DPDK thread. Paul – can you dig into this further?
-Jim
On 9/7/18, 7:39 AM, "SPDK on behalf of Luse, Paul E" <spdk-bounces(a)lists.01.org on behalf of paul.e.luse(a)intel.com> wrote:
That's really good to know, thanks! I'm hardcoded to 0 now which is fine for the remainder of testing I think, let me know how it goes with get_mempolicy
Thx,
Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 7:31 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
I wasn't registered on the mailing list with this email before, so I registered just now and am resending this message.
D.
-----Original Message-----
From: Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 4:04 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: RE: socket_id difference under fio plugin vs bdevio or bdevperf
HI Paul,
Unlike bdevperf and bdevio, fio_plugin doesn't really use DPDK abstraction for multi-threading. It uses standard pthreads for most of its work and that's why you're seeing socket id 4294967295 (unsigned -1). This value is simply uninitialized. To get socket id in a portable way, you might want to use get_mempolicy. In fact, we should probably use it somewhere within our SPDK APIs. I'll look into adding it tomorrow.
D.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, September 7, 2018 3:27 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
In the crypto patch when I init the aesni virtual device I've been passing in NULL for optional args and it's worked fine. When running w/fio however, the aesni driver took the null args to mean "socket any" which should be fine but for whatever reason its arg parser doesn't like that value.
So this works in bdevio/bdevperf but does not in FIO
rc = rte_vdev_init(AESNI_MB, NULL);
Hardcoding the only arg I care about, socket_id, because it's the one that the aesni driver doesn't like not being specified works in fio as well as our apps:
rc = rte_vdev_init(AESNI_MB, "socket_id=0");
So using spdk calls I tried this:
snprintf(socket_id, 32, "socket_id=%d",
spdk_env_get_socket_id(spdk_env_get_current_core()));
rc = rte_vdev_init(AESNI_MB, socket_id);
and once again woks with bdevio and fails with fio. Looking at the differences in the code here what I see:
running with bdevio:
│173 rte_lcore_to_socket_id(unsigned lcore_id) │
│174 { │
>│175 return lcore_config[lcore_id].socket_id; │
│176 }
(gdb) p lcore_config[0]
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 0, core_id = 0, core_index = 0, cpuset = {__bits = {1, 0 <repeats 15 times>}}, core_role = 0 '\000'}
and running with fio:
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 4294967295, core_id = 0, core_index = 0, cpuset = {__bits = {18446744073709551615, 16777215, 0 <repeats 14 times>}},
core_role = 0 '\000'}
Notice the difference. Any idea why running with fio plugin vs bdevio or bdevperf would cause this??
Thanks!
Paul
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
@ 2018-09-07 15:08 Stojaczyk, Dariusz
0 siblings, 0 replies; 15+ messages in thread
From: Stojaczyk, Dariusz @ 2018-09-07 15:08 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 5790 bytes --]
Ack, I'll let you know.
D.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, September 7, 2018 5:07 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
For sure, I'll wait for Derek's response on what he was planning on doing tomorrow and pick anything up from there...
Thx
Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, September 7, 2018 8:05 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
I like Darek’s suggestions. I do think this is a DPDK bug though. Other DPDK calls like rte_malloc() work from a non-DPDK thread. Paul – can you dig into this further?
-Jim
On 9/7/18, 7:39 AM, "SPDK on behalf of Luse, Paul E" <spdk-bounces(a)lists.01.org on behalf of paul.e.luse(a)intel.com> wrote:
That's really good to know, thanks! I'm hardcoded to 0 now which is fine for the remainder of testing I think, let me know how it goes with get_mempolicy
Thx,
Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 7:31 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
I wasn't registered on the mailing list with this email before, so I registered just now and am resending this message.
D.
-----Original Message-----
From: Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 4:04 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: RE: socket_id difference under fio plugin vs bdevio or bdevperf
HI Paul,
Unlike bdevperf and bdevio, fio_plugin doesn't really use DPDK abstraction for multi-threading. It uses standard pthreads for most of its work and that's why you're seeing socket id 4294967295 (unsigned -1). This value is simply uninitialized. To get socket id in a portable way, you might want to use get_mempolicy. In fact, we should probably use it somewhere within our SPDK APIs. I'll look into adding it tomorrow.
D.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, September 7, 2018 3:27 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
In the crypto patch when I init the aesni virtual device I've been passing in NULL for optional args and it's worked fine. When running w/fio however, the aesni driver took the null args to mean "socket any" which should be fine but for whatever reason its arg parser doesn't like that value.
So this works in bdevio/bdevperf but does not in FIO
rc = rte_vdev_init(AESNI_MB, NULL);
Hardcoding the only arg I care about, socket_id, because it's the one that the aesni driver doesn't like not being specified works in fio as well as our apps:
rc = rte_vdev_init(AESNI_MB, "socket_id=0");
So using spdk calls I tried this:
snprintf(socket_id, 32, "socket_id=%d",
spdk_env_get_socket_id(spdk_env_get_current_core()));
rc = rte_vdev_init(AESNI_MB, socket_id);
and once again woks with bdevio and fails with fio. Looking at the differences in the code here what I see:
running with bdevio:
│173 rte_lcore_to_socket_id(unsigned lcore_id) │
│174 { │
>│175 return lcore_config[lcore_id].socket_id; │
│176 }
(gdb) p lcore_config[0]
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 0, core_id = 0, core_index = 0, cpuset = {__bits = {1, 0 <repeats 15 times>}}, core_role = 0 '\000'}
and running with fio:
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 4294967295, core_id = 0, core_index = 0, cpuset = {__bits = {18446744073709551615, 16777215, 0 <repeats 14 times>}},
core_role = 0 '\000'}
Notice the difference. Any idea why running with fio plugin vs bdevio or bdevperf would cause this??
Thanks!
Paul
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
@ 2018-09-07 15:06 Luse, Paul E
0 siblings, 0 replies; 15+ messages in thread
From: Luse, Paul E @ 2018-09-07 15:06 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 5336 bytes --]
For sure, I'll wait for Derek's response on what he was planning on doing tomorrow and pick anything up from there...
Thx
Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, September 7, 2018 8:05 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
I like Darek’s suggestions. I do think this is a DPDK bug though. Other DPDK calls like rte_malloc() work from a non-DPDK thread. Paul – can you dig into this further?
-Jim
On 9/7/18, 7:39 AM, "SPDK on behalf of Luse, Paul E" <spdk-bounces(a)lists.01.org on behalf of paul.e.luse(a)intel.com> wrote:
That's really good to know, thanks! I'm hardcoded to 0 now which is fine for the remainder of testing I think, let me know how it goes with get_mempolicy
Thx,
Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 7:31 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
I wasn't registered on the mailing list with this email before, so I registered just now and am resending this message.
D.
-----Original Message-----
From: Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 4:04 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: RE: socket_id difference under fio plugin vs bdevio or bdevperf
HI Paul,
Unlike bdevperf and bdevio, fio_plugin doesn't really use DPDK abstraction for multi-threading. It uses standard pthreads for most of its work and that's why you're seeing socket id 4294967295 (unsigned -1). This value is simply uninitialized. To get socket id in a portable way, you might want to use get_mempolicy. In fact, we should probably use it somewhere within our SPDK APIs. I'll look into adding it tomorrow.
D.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, September 7, 2018 3:27 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
In the crypto patch when I init the aesni virtual device I've been passing in NULL for optional args and it's worked fine. When running w/fio however, the aesni driver took the null args to mean "socket any" which should be fine but for whatever reason its arg parser doesn't like that value.
So this works in bdevio/bdevperf but does not in FIO
rc = rte_vdev_init(AESNI_MB, NULL);
Hardcoding the only arg I care about, socket_id, because it's the one that the aesni driver doesn't like not being specified works in fio as well as our apps:
rc = rte_vdev_init(AESNI_MB, "socket_id=0");
So using spdk calls I tried this:
snprintf(socket_id, 32, "socket_id=%d",
spdk_env_get_socket_id(spdk_env_get_current_core()));
rc = rte_vdev_init(AESNI_MB, socket_id);
and once again woks with bdevio and fails with fio. Looking at the differences in the code here what I see:
running with bdevio:
│173 rte_lcore_to_socket_id(unsigned lcore_id) │
│174 { │
>│175 return lcore_config[lcore_id].socket_id; │
│176 }
(gdb) p lcore_config[0]
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 0, core_id = 0, core_index = 0, cpuset = {__bits = {1, 0 <repeats 15 times>}}, core_role = 0 '\000'}
and running with fio:
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 4294967295, core_id = 0, core_index = 0, cpuset = {__bits = {18446744073709551615, 16777215, 0 <repeats 14 times>}},
core_role = 0 '\000'}
Notice the difference. Any idea why running with fio plugin vs bdevio or bdevperf would cause this??
Thanks!
Paul
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
@ 2018-09-07 15:04 Harris, James R
0 siblings, 0 replies; 15+ messages in thread
From: Harris, James R @ 2018-09-07 15:04 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 4776 bytes --]
I like Darek’s suggestions. I do think this is a DPDK bug though. Other DPDK calls like rte_malloc() work from a non-DPDK thread. Paul – can you dig into this further?
-Jim
On 9/7/18, 7:39 AM, "SPDK on behalf of Luse, Paul E" <spdk-bounces(a)lists.01.org on behalf of paul.e.luse(a)intel.com> wrote:
That's really good to know, thanks! I'm hardcoded to 0 now which is fine for the remainder of testing I think, let me know how it goes with get_mempolicy
Thx,
Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 7:31 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
I wasn't registered on the mailing list with this email before, so I registered just now and am resending this message.
D.
-----Original Message-----
From: Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 4:04 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: RE: socket_id difference under fio plugin vs bdevio or bdevperf
HI Paul,
Unlike bdevperf and bdevio, fio_plugin doesn't really use DPDK abstraction for multi-threading. It uses standard pthreads for most of its work and that's why you're seeing socket id 4294967295 (unsigned -1). This value is simply uninitialized. To get socket id in a portable way, you might want to use get_mempolicy. In fact, we should probably use it somewhere within our SPDK APIs. I'll look into adding it tomorrow.
D.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, September 7, 2018 3:27 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
In the crypto patch when I init the aesni virtual device I've been passing in NULL for optional args and it's worked fine. When running w/fio however, the aesni driver took the null args to mean "socket any" which should be fine but for whatever reason its arg parser doesn't like that value.
So this works in bdevio/bdevperf but does not in FIO
rc = rte_vdev_init(AESNI_MB, NULL);
Hardcoding the only arg I care about, socket_id, because it's the one that the aesni driver doesn't like not being specified works in fio as well as our apps:
rc = rte_vdev_init(AESNI_MB, "socket_id=0");
So using spdk calls I tried this:
snprintf(socket_id, 32, "socket_id=%d",
spdk_env_get_socket_id(spdk_env_get_current_core()));
rc = rte_vdev_init(AESNI_MB, socket_id);
and once again woks with bdevio and fails with fio. Looking at the differences in the code here what I see:
running with bdevio:
│173 rte_lcore_to_socket_id(unsigned lcore_id) │
│174 { │
>│175 return lcore_config[lcore_id].socket_id; │
│176 }
(gdb) p lcore_config[0]
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 0, core_id = 0, core_index = 0, cpuset = {__bits = {1, 0 <repeats 15 times>}}, core_role = 0 '\000'}
and running with fio:
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 4294967295, core_id = 0, core_index = 0, cpuset = {__bits = {18446744073709551615, 16777215, 0 <repeats 14 times>}},
core_role = 0 '\000'}
Notice the difference. Any idea why running with fio plugin vs bdevio or bdevperf would cause this??
Thanks!
Paul
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
@ 2018-09-07 14:39 Luse, Paul E
0 siblings, 0 replies; 15+ messages in thread
From: Luse, Paul E @ 2018-09-07 14:39 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 4210 bytes --]
Thanks Andrey!
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
Sent: Friday, September 7, 2018 7:09 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
Fio doesn't use dpdk threads, so everything rte remains uninitialized. In
particular, rte_core stuff is ynusable with fio plugin.
Regards,
Andrey
On Fri, Sep 7, 2018, 17:00 Luse, Paul E <paul.e.luse(a)intel.com> wrote:
> That's a great question! I haven't used fio in forever so am not familiar
> with the startup sequence, I'll take a look in a bit.
>
> Thx
> Paul
>
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Wodkowski,
> PawelX
> Sent: Friday, September 7, 2018 6:57 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or
> bdevperf
>
> Are you sure rte_vdev_init() is called after DPDK is initialized or you
> (fio) is calling rte_vdev_init() from non-DPDK thread?
>
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul
> > E
> > Sent: Friday, September 7, 2018 3:27 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: [SPDK] socket_id difference under fio plugin vs bdevio or
> > bdevperf
> >
> >
> > In the crypto patch when I init the aesni virtual device I've been
> > passing in NULL for optional args and it's worked fine. When running
> > w/fio however, the aesni driver took the null args to mean "socket
> > any" which should be fine but for whatever reason its arg parser doesn't
> like that value.
> >
> > So this works in bdevio/bdevperf but does not in FIO
> >
> > rc = rte_vdev_init(AESNI_MB, NULL);
> >
> > Hardcoding the only arg I care about, socket_id, because it's the one
> > that the aesni driver doesn't like not being specified works in fio as
> well as our apps:
> >
> > rc = rte_vdev_init(AESNI_MB, "socket_id=0");
> >
> > So using spdk calls I tried this:
> >
> > snprintf(socket_id, 32, "socket_id=%d",
> > spdk_env_get_socket_id(spdk_env_get_current_core()));
> > rc = rte_vdev_init(AESNI_MB, socket_id);
> >
> > and once again woks with bdevio and fails with fio. Looking at the
> > differences in the code here what I see:
> >
> > running with bdevio:
> >
> > │173 rte_lcore_to_socket_id(unsigned lcore_id)
> > │
> > │174 {
> │
> > >│175 return lcore_config[lcore_id].socket_id;
> > │
> > │176 }
> >
> > (gdb) p lcore_config[0]
> > $1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0},
> > pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state =
> WAIT,
> > socket_id = 0, core_id = 0, core_index = 0, cpuset = {__bits = {1, 0
> > <repeats
> > 15 times>}}, core_role = 0 '\000'}
> >
> > and running with fio:
> >
> > $1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0},
> > pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state =
> WAIT,
> > socket_id = 4294967295, core_id = 0, core_index = 0, cpuset =
> > {__bits = {18446744073709551615, 16777215, 0 <repeats 14 times>}},
> > core_role = 0 '\000'}
> >
> > Notice the difference. Any idea why running with fio plugin vs bdevio
> > or bdevperf would cause this??
> >
> > Thanks!
> > Paul
> >
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
--
Regards,
Andrey
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
@ 2018-09-07 14:39 Luse, Paul E
0 siblings, 0 replies; 15+ messages in thread
From: Luse, Paul E @ 2018-09-07 14:39 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 3953 bytes --]
That's really good to know, thanks! I'm hardcoded to 0 now which is fine for the remainder of testing I think, let me know how it goes with get_mempolicy
Thx,
Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 7:31 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
I wasn't registered on the mailing list with this email before, so I registered just now and am resending this message.
D.
-----Original Message-----
From: Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 4:04 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: RE: socket_id difference under fio plugin vs bdevio or bdevperf
HI Paul,
Unlike bdevperf and bdevio, fio_plugin doesn't really use DPDK abstraction for multi-threading. It uses standard pthreads for most of its work and that's why you're seeing socket id 4294967295 (unsigned -1). This value is simply uninitialized. To get socket id in a portable way, you might want to use get_mempolicy. In fact, we should probably use it somewhere within our SPDK APIs. I'll look into adding it tomorrow.
D.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, September 7, 2018 3:27 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
In the crypto patch when I init the aesni virtual device I've been passing in NULL for optional args and it's worked fine. When running w/fio however, the aesni driver took the null args to mean "socket any" which should be fine but for whatever reason its arg parser doesn't like that value.
So this works in bdevio/bdevperf but does not in FIO
rc = rte_vdev_init(AESNI_MB, NULL);
Hardcoding the only arg I care about, socket_id, because it's the one that the aesni driver doesn't like not being specified works in fio as well as our apps:
rc = rte_vdev_init(AESNI_MB, "socket_id=0");
So using spdk calls I tried this:
snprintf(socket_id, 32, "socket_id=%d",
spdk_env_get_socket_id(spdk_env_get_current_core()));
rc = rte_vdev_init(AESNI_MB, socket_id);
and once again woks with bdevio and fails with fio. Looking at the differences in the code here what I see:
running with bdevio:
│173 rte_lcore_to_socket_id(unsigned lcore_id) │
│174 { │
>│175 return lcore_config[lcore_id].socket_id; │
│176 }
(gdb) p lcore_config[0]
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 0, core_id = 0, core_index = 0, cpuset = {__bits = {1, 0 <repeats 15 times>}}, core_role = 0 '\000'}
and running with fio:
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 4294967295, core_id = 0, core_index = 0, cpuset = {__bits = {18446744073709551615, 16777215, 0 <repeats 14 times>}},
core_role = 0 '\000'}
Notice the difference. Any idea why running with fio plugin vs bdevio or bdevperf would cause this??
Thanks!
Paul
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
@ 2018-09-07 14:30 Stojaczyk, Dariusz
0 siblings, 0 replies; 15+ messages in thread
From: Stojaczyk, Dariusz @ 2018-09-07 14:30 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 3352 bytes --]
I wasn't registered on the mailing list with this email before, so I registered just now and am resending this message.
D.
-----Original Message-----
From: Stojaczyk, Dariusz
Sent: Friday, September 7, 2018 4:04 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: RE: socket_id difference under fio plugin vs bdevio or bdevperf
HI Paul,
Unlike bdevperf and bdevio, fio_plugin doesn't really use DPDK abstraction for multi-threading. It uses standard pthreads for most of its work and that's why you're seeing socket id 4294967295 (unsigned -1). This value is simply uninitialized. To get socket id in a portable way, you might want to use get_mempolicy. In fact, we should probably use it somewhere within our SPDK APIs. I'll look into adding it tomorrow.
D.
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, September 7, 2018 3:27 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
In the crypto patch when I init the aesni virtual device I've been passing in NULL for optional args and it's worked fine. When running w/fio however, the aesni driver took the null args to mean "socket any" which should be fine but for whatever reason its arg parser doesn't like that value.
So this works in bdevio/bdevperf but does not in FIO
rc = rte_vdev_init(AESNI_MB, NULL);
Hardcoding the only arg I care about, socket_id, because it's the one that the aesni driver doesn't like not being specified works in fio as well as our apps:
rc = rte_vdev_init(AESNI_MB, "socket_id=0");
So using spdk calls I tried this:
snprintf(socket_id, 32, "socket_id=%d",
spdk_env_get_socket_id(spdk_env_get_current_core()));
rc = rte_vdev_init(AESNI_MB, socket_id);
and once again woks with bdevio and fails with fio. Looking at the differences in the code here what I see:
running with bdevio:
│173 rte_lcore_to_socket_id(unsigned lcore_id) │
│174 { │
>│175 return lcore_config[lcore_id].socket_id; │
│176 }
(gdb) p lcore_config[0]
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 0, core_id = 0, core_index = 0, cpuset = {__bits = {1, 0 <repeats 15 times>}}, core_role = 0 '\000'}
and running with fio:
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 4294967295, core_id = 0, core_index = 0, cpuset = {__bits = {18446744073709551615, 16777215, 0 <repeats 14 times>}},
core_role = 0 '\000'}
Notice the difference. Any idea why running with fio plugin vs bdevio or bdevperf would cause this??
Thanks!
Paul
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
@ 2018-09-07 14:09 Andrey Kuzmin
0 siblings, 0 replies; 15+ messages in thread
From: Andrey Kuzmin @ 2018-09-07 14:09 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 3768 bytes --]
Fio doesn't use dpdk threads, so everything rte remains uninitialized. In
particular, rte_core stuff is ynusable with fio plugin.
Regards,
Andrey
On Fri, Sep 7, 2018, 17:00 Luse, Paul E <paul.e.luse(a)intel.com> wrote:
> That's a great question! I haven't used fio in forever so am not familiar
> with the startup sequence, I'll take a look in a bit.
>
> Thx
> Paul
>
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Wodkowski,
> PawelX
> Sent: Friday, September 7, 2018 6:57 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] socket_id difference under fio plugin vs bdevio or
> bdevperf
>
> Are you sure rte_vdev_init() is called after DPDK is initialized or you
> (fio) is calling rte_vdev_init() from non-DPDK thread?
>
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul
> > E
> > Sent: Friday, September 7, 2018 3:27 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: [SPDK] socket_id difference under fio plugin vs bdevio or
> > bdevperf
> >
> >
> > In the crypto patch when I init the aesni virtual device I've been
> > passing in NULL for optional args and it's worked fine. When running
> > w/fio however, the aesni driver took the null args to mean "socket
> > any" which should be fine but for whatever reason its arg parser doesn't
> like that value.
> >
> > So this works in bdevio/bdevperf but does not in FIO
> >
> > rc = rte_vdev_init(AESNI_MB, NULL);
> >
> > Hardcoding the only arg I care about, socket_id, because it's the one
> > that the aesni driver doesn't like not being specified works in fio as
> well as our apps:
> >
> > rc = rte_vdev_init(AESNI_MB, "socket_id=0");
> >
> > So using spdk calls I tried this:
> >
> > snprintf(socket_id, 32, "socket_id=%d",
> > spdk_env_get_socket_id(spdk_env_get_current_core()));
> > rc = rte_vdev_init(AESNI_MB, socket_id);
> >
> > and once again woks with bdevio and fails with fio. Looking at the
> > differences in the code here what I see:
> >
> > running with bdevio:
> >
> > │173 rte_lcore_to_socket_id(unsigned lcore_id)
> > │
> > │174 {
> │
> > >│175 return lcore_config[lcore_id].socket_id;
> > │
> > │176 }
> >
> > (gdb) p lcore_config[0]
> > $1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0},
> > pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state =
> WAIT,
> > socket_id = 0, core_id = 0, core_index = 0, cpuset = {__bits = {1, 0
> > <repeats
> > 15 times>}}, core_role = 0 '\000'}
> >
> > and running with fio:
> >
> > $1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0},
> > pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state =
> WAIT,
> > socket_id = 4294967295, core_id = 0, core_index = 0, cpuset =
> > {__bits = {18446744073709551615, 16777215, 0 <repeats 14 times>}},
> > core_role = 0 '\000'}
> >
> > Notice the difference. Any idea why running with fio plugin vs bdevio
> > or bdevperf would cause this??
> >
> > Thanks!
> > Paul
> >
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
--
Regards,
Andrey
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
@ 2018-09-07 13:56 Wodkowski, PawelX
0 siblings, 0 replies; 15+ messages in thread
From: Wodkowski, PawelX @ 2018-09-07 13:56 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 2669 bytes --]
Are you sure rte_vdev_init() is called after DPDK is initialized or you (fio) is calling rte_vdev_init() from non-DPDK thread?
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
> Sent: Friday, September 7, 2018 3:27 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
>
>
> In the crypto patch when I init the aesni virtual device I've been passing in
> NULL for optional args and it's worked fine. When running w/fio however,
> the aesni driver took the null args to mean "socket any" which should be fine
> but for whatever reason its arg parser doesn't like that value.
>
> So this works in bdevio/bdevperf but does not in FIO
>
> rc = rte_vdev_init(AESNI_MB, NULL);
>
> Hardcoding the only arg I care about, socket_id, because it's the one that the
> aesni driver doesn't like not being specified works in fio as well as our apps:
>
> rc = rte_vdev_init(AESNI_MB, "socket_id=0");
>
> So using spdk calls I tried this:
>
> snprintf(socket_id, 32, "socket_id=%d",
> spdk_env_get_socket_id(spdk_env_get_current_core()));
> rc = rte_vdev_init(AESNI_MB, socket_id);
>
> and once again woks with bdevio and fails with fio. Looking at the
> differences in the code here what I see:
>
> running with bdevio:
>
> │173 rte_lcore_to_socket_id(unsigned lcore_id)
> │
> │174 { │
> >│175 return lcore_config[lcore_id].socket_id;
> │
> │176 }
>
> (gdb) p lcore_config[0]
> $1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0},
> pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
> socket_id = 0, core_id = 0, core_index = 0, cpuset = {__bits = {1, 0 <repeats
> 15 times>}}, core_role = 0 '\000'}
>
> and running with fio:
>
> $1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0},
> pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
> socket_id = 4294967295, core_id = 0, core_index = 0, cpuset = {__bits =
> {18446744073709551615, 16777215, 0 <repeats 14 times>}},
> core_role = 0 '\000'}
>
> Notice the difference. Any idea why running with fio plugin vs bdevio or
> bdevperf would cause this??
>
> Thanks!
> Paul
>
>
>
>
>
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 15+ messages in thread
* [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf
@ 2018-09-07 13:27 Luse, Paul E
0 siblings, 0 replies; 15+ messages in thread
From: Luse, Paul E @ 2018-09-07 13:27 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 2132 bytes --]
In the crypto patch when I init the aesni virtual device I've been passing in NULL for optional args and it's worked fine. When running w/fio however, the aesni driver took the null args to mean "socket any" which should be fine but for whatever reason its arg parser doesn't like that value.
So this works in bdevio/bdevperf but does not in FIO
rc = rte_vdev_init(AESNI_MB, NULL);
Hardcoding the only arg I care about, socket_id, because it's the one that the aesni driver doesn't like not being specified works in fio as well as our apps:
rc = rte_vdev_init(AESNI_MB, "socket_id=0");
So using spdk calls I tried this:
snprintf(socket_id, 32, "socket_id=%d",
spdk_env_get_socket_id(spdk_env_get_current_core()));
rc = rte_vdev_init(AESNI_MB, socket_id);
and once again woks with bdevio and fails with fio. Looking at the differences in the code here what I see:
running with bdevio:
│173 rte_lcore_to_socket_id(unsigned lcore_id) │
│174 { │
>│175 return lcore_config[lcore_id].socket_id; │
│176 }
(gdb) p lcore_config[0]
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 0, core_id = 0, core_index = 0, cpuset = {__bits = {1, 0 <repeats 15 times>}}, core_role = 0 '\000'}
and running with fio:
$1 = {detected = 1, thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT,
socket_id = 4294967295, core_id = 0, core_index = 0, cpuset = {__bits = {18446744073709551615, 16777215, 0 <repeats 14 times>}},
core_role = 0 '\000'}
Notice the difference. Any idea why running with fio plugin vs bdevio or bdevperf would cause this??
Thanks!
Paul
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2018-09-09 23:41 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-09-07 14:00 [SPDK] socket_id difference under fio plugin vs bdevio or bdevperf Luse, Paul E
-- strict thread matches above, loose matches on Subject: below --
2018-09-09 23:41 Luse, Paul E
2018-09-09 17:00 Luse, Paul E
2018-09-09 16:18 Luse, Paul E
2018-09-09 13:41 Luse, Paul E
2018-09-08 23:09 Stojaczyk, Dariusz
2018-09-07 15:08 Stojaczyk, Dariusz
2018-09-07 15:06 Luse, Paul E
2018-09-07 15:04 Harris, James R
2018-09-07 14:39 Luse, Paul E
2018-09-07 14:39 Luse, Paul E
2018-09-07 14:30 Stojaczyk, Dariusz
2018-09-07 14:09 Andrey Kuzmin
2018-09-07 13:56 Wodkowski, PawelX
2018-09-07 13:27 Luse, Paul E
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.