Re: [SPDK] bdev/fio_plugin coremask

Re: [SPDK] bdev/fio_plugin coremask

[Kariuki, John K]

[-- Attachment #1: Type: text/plain, Size: 1852 bytes --]

Andrey,
The core mask is set correctly but not reported correctly in the logged message.
On my system I see the message below, which reports that the DPDK EAL –c parameter has a value of 0x1 just like you saw on your system.

fio-3.3
Starting 2 threads
Starting SPDK v18.04 / DPDK 18.02.0 initialization...
[ DPDK EAL parameters: fio -c 0x1 --file-prefix=spdk_pid133108 ]

But I can see the CPU cores being used are CPU core 28 and 29 per my fio configuration file.
[cid:image001.png(a)01D427ED.8227DD10]

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
Sent: Sunday, July 29, 2018 1:47 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] bdev/fio_plugin coremask

Not sure if it's a bogus related to how fio_plugin initializes DPDK environment, but I'm confused by the discrepancy between what I have in the SPDK config file used to run FIO plugin, which is
...
  # Users can restrict work items to only run on certain cores by
  #  specifying a ReactorMask.  Default is to allow work items to run
  #  on core 0.
  ReactorMask 0x7
...
and what FIO tells me on startup:
...
fio-3.8-21-ga8712
Starting 1 thread
Starting SPDK v18.04.1 / DPDK 18.02.0 initialization...
[ DPDK EAL parameters: fio -c 0x1 --file-prefix=spdk_pid6744 ]
EAL: Detected 4 lcore(s)
EAL: Multi-process socket /var/run/.spdk_pid6744_unix
EAL: Probing VFIO support...
...

If I run bdevperf with the same SPDK config file, I'm quite the contrary getting exactly what I asked for:
...
Starting SPDK v18.04.1 / DPDK 18.02.0 initialization...
[ DPDK EAL parameters: bdevtest -c 0x7 -m 1024 --file-prefix=spdk_pid6915 ]
EAL: Detected 4 lcore(s)
EAL: Multi-process socket /var/run/.spdk_pid6915_unix
EAL: Probing VFIO support...
...

Comments will be appreciated.

Regards,
Andrey

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 7381 bytes --]

[-- Attachment #3: image001.png --]
[-- Type: image/png, Size: 25926 bytes --]

Re: [SPDK] bdev/fio_plugin coremask

[Andrey Kuzmin]

[-- Attachment #1: Type: text/plain, Size: 5217 bytes --]

On Tue, Jul 31, 2018 at 12:20 AM Walker, Benjamin <benjamin.walker(a)intel.com>
wrote:

> On Mon, 2018-07-30 at 22:58 +0300, Andrey Kuzmin wrote:
> > I believe my fio threads also affinitize anbd run as requested. What I'm
> > concerned with is whether spdk pinned threads are started as requested
> by the
> > core mask, and it looks like they aren't. Not something really
> concerning as
> > fio plugin runs atop fio threads, but still a discrepancy (and likeley
> worth a
> > note in the plugin code/docs).
>
> Making DPDK and fio play together is very tricky. The main problem is that
> both
> DPDK and fio want to be in charge of spawning and affinitizing the threads
> (they're both frameworks). There is simply no way to "hook" the creation
> of a
> thread in fio to force it to use an existing one, nor is there a way to
> force a
> thread to become a DPDK thread (there is no equivalent of
> spdk_allocate_thread(), for example).


Ironically, that's sort of a platform approach (SPDK took for good) revenge
;).  When I first laid my hands on fio + spdk some three two and a half
year back (could be traced down to exact fio commit that introduced fio
hooks for I/O memory initialization I needed to get fio use SPDK memory
buffers for I/O), I got spdk nvme library code integrated into fio plugin
in a pretty straightforward manner (fio way), with very good performance
results.


> Fortunately, all of the DPDK data
> structures and API calls we depend on can be called safely from a non-DPDK
> thread. Some of them have worse performance in this case (the big one is
> that
> you lose access to the mempool's per-thread cache), but they all at least
> function correctly. So the strategy I took here was to spawn DPDK on just
> one
> thread ("the initialization thread"), unaffinitize it, and do nothing on
> it.
> Then do all operations from the fio threads which are spawned according to
> your
> fio configuration file.
>

OK, this both makes sense and explains my observation.


> When we first implemented this, we did notice some fairly significant
> performance hits when pulling bdev_io out of the central pool. However, Jim
> added some code that made SPDK's bdev layer manage its own per-SPDK-thread
> bdev_io caches a few months ago, so this is no longer a concern. When SPDK
> threads become "lightweight" threads that can get rescheduled onto various
> real
> threads at any time, changes like that will become increasingly important
> anyway
> - you want the cache to follow the "lightweight" thread around.
>

Sure. I'd also like to get better educated on the lightweight threading
model, so any pointers will be appreciated.

Thanks for spending time on going into the details,
Andrey

>
> Thanks,
> Ben
>
>
> >
> > Regards,
> > Andrey
> > >
> > > fio-3.3
> > > Starting 2 threads
> > > Starting SPDK v18.04 / DPDK 18.02.0 initialization...
> > > [ DPDK EAL parameters: fio -c 0x1 --file-prefix=spdk_pid133108 ]
> > >
> > > But I can see the CPU cores being used are CPU core 28 and 29 per my
> fio
> > > configuration file.
> > >
> > >
> > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey
> Kuzmin
> > > Sent: Sunday, July 29, 2018 1:47 PM
> > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > Subject: [SPDK] bdev/fio_plugin coremask
> > >
> > > Not sure if it's a bogus related to how fio_plugin initializes DPDK
> > > environment, but I'm confused by the discrepancy between what I have
> in the
> > > SPDK config file used to run FIO plugin, which is
> > > ...
> > >   # Users can restrict work items to only run on certain cores by
> > >   #  specifying a ReactorMask.  Default is to allow work items to run
> > >   #  on core 0.
> > >   ReactorMask 0x7
> > > ...
> > > and what FIO tells me on startup:
> > > ...
> > > fio-3.8-21-ga8712
> > > Starting 1 thread
> > > Starting SPDK v18.04.1 / DPDK 18.02.0 initialization...
> > > [ DPDK EAL parameters: fio -c 0x1 --file-prefix=spdk_pid6744 ]
> > > EAL: Detected 4 lcore(s)
> > > EAL: Multi-process socket /var/run/.spdk_pid6744_unix
> > > EAL: Probing VFIO support...
> > > ...
> > >
> > > If I run bdevperf with the same SPDK config file, I'm quite the
> contrary
> > > getting exactly what I asked for:
> > > ...
> > > Starting SPDK v18.04.1 / DPDK 18.02.0 initialization...
> > > [ DPDK EAL parameters: bdevtest -c 0x7 -m 1024
> --file-prefix=spdk_pid6915 ]
> > > EAL: Detected 4 lcore(s)
> > > EAL: Multi-process socket /var/run/.spdk_pid6915_unix
> > > EAL: Probing VFIO support...
> > > ...
> > >
> > > Comments will be appreciated.
> > >
> > > Regards,
> > > Andrey
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> >
> > --
> > Regards,
> > Andrey
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 7283 bytes --]

Re: [SPDK] bdev/fio_plugin coremask

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 4044 bytes --]

On Mon, 2018-07-30 at 22:58 +0300, Andrey Kuzmin wrote:
> I believe my fio threads also affinitize anbd run as requested. What I'm
> concerned with is whether spdk pinned threads are started as requested by the
> core mask, and it looks like they aren't. Not something really concerning as
> fio plugin runs atop fio threads, but still a discrepancy (and likeley worth a
> note in the plugin code/docs).

Making DPDK and fio play together is very tricky. The main problem is that both
DPDK and fio want to be in charge of spawning and affinitizing the threads
(they're both frameworks). There is simply no way to "hook" the creation of a
thread in fio to force it to use an existing one, nor is there a way to force a
thread to become a DPDK thread (there is no equivalent of
spdk_allocate_thread(), for example). Fortunately, all of the DPDK data
structures and API calls we depend on can be called safely from a non-DPDK
thread. Some of them have worse performance in this case (the big one is that
you lose access to the mempool's per-thread cache), but they all at least
function correctly. So the strategy I took here was to spawn DPDK on just one
thread ("the initialization thread"), unaffinitize it, and do nothing on it.
Then do all operations from the fio threads which are spawned according to your
fio configuration file.

When we first implemented this, we did notice some fairly significant
performance hits when pulling bdev_io out of the central pool. However, Jim
added some code that made SPDK's bdev layer manage its own per-SPDK-thread
bdev_io caches a few months ago, so this is no longer a concern. When SPDK
threads become "lightweight" threads that can get rescheduled onto various real
threads at any time, changes like that will become increasingly important anyway
- you want the cache to follow the "lightweight" thread around.

Thanks,
Ben


> 
> Regards,
> Andrey
> >  
> > fio-3.3
> > Starting 2 threads
> > Starting SPDK v18.04 / DPDK 18.02.0 initialization...
> > [ DPDK EAL parameters: fio -c 0x1 --file-prefix=spdk_pid133108 ]
> >  
> > But I can see the CPU cores being used are CPU core 28 and 29 per my fio
> > configuration file.
> > 
> >  
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
> > Sent: Sunday, July 29, 2018 1:47 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: [SPDK] bdev/fio_plugin coremask
> >  
> > Not sure if it's a bogus related to how fio_plugin initializes DPDK
> > environment, but I'm confused by the discrepancy between what I have in the
> > SPDK config file used to run FIO plugin, which is
> > ...
> >   # Users can restrict work items to only run on certain cores by
> >   #  specifying a ReactorMask.  Default is to allow work items to run
> >   #  on core 0.
> >   ReactorMask 0x7
> > ...
> > and what FIO tells me on startup:
> > ...
> > fio-3.8-21-ga8712
> > Starting 1 thread
> > Starting SPDK v18.04.1 / DPDK 18.02.0 initialization...
> > [ DPDK EAL parameters: fio -c 0x1 --file-prefix=spdk_pid6744 ]
> > EAL: Detected 4 lcore(s)
> > EAL: Multi-process socket /var/run/.spdk_pid6744_unix
> > EAL: Probing VFIO support...
> > ...
> >  
> > If I run bdevperf with the same SPDK config file, I'm quite the contrary
> > getting exactly what I asked for:
> > ...
> > Starting SPDK v18.04.1 / DPDK 18.02.0 initialization...
> > [ DPDK EAL parameters: bdevtest -c 0x7 -m 1024 --file-prefix=spdk_pid6915 ]
> > EAL: Detected 4 lcore(s)
> > EAL: Multi-process socket /var/run/.spdk_pid6915_unix
> > EAL: Probing VFIO support...
> > ...
> >  
> > Comments will be appreciated.
> >  
> > Regards,
> > Andrey
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> 
> -- 
> Regards,
> Andrey
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] bdev/fio_plugin coremask

[Andrey Kuzmin]

[-- Attachment #1: Type: text/plain, Size: 2695 bytes --]

On Mon, Jul 30, 2018, 20:10 Kariuki, John K <john.k.kariuki(a)intel.com>
wrote:

> Andrey,
>
> The core mask is set correctly but not reported correctly in the logged
> message.
>
> On my system I see the message below, which reports that the DPDK EAL –c
> parameter has a value of 0x1 just like you saw on your system.
>
Thanks John, but I believe my fio threads also affinitize anbd run as
requested. What I'm concerned with is whether spdk pinned threads are
started as requested by the core mask, and it looks like they aren't. Not
something really concerning as fio plugin runs atop fio threads, but still
a discrepancy (and likeley worth a note in the plugin code/docs).

Regards,
Andrey

>
>
> fio-3.3
>
> Starting 2 threads
>
> Starting SPDK v18.04 / DPDK 18.02.0 initialization...
>
> [ DPDK EAL parameters: fio -c 0x1 --file-prefix=spdk_pid133108 ]
>
>
>
> But I can see the CPU cores being used are CPU core 28 and 29 per my fio
> configuration file.
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Andrey
> Kuzmin
> *Sent:* Sunday, July 29, 2018 1:47 PM
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* [SPDK] bdev/fio_plugin coremask
>
>
>
> Not sure if it's a bogus related to how fio_plugin initializes DPDK
> environment, but I'm confused by the discrepancy between what I have in the
> SPDK config file used to run FIO plugin, which is
>
> ...
>
>   # Users can restrict work items to only run on certain cores by
>
>   #  specifying a ReactorMask.  Default is to allow work items to run
>
>   #  on core 0.
>
>   *ReactorMask 0x7*
>
> ...
>
> and what FIO tells me on startup:
>
> ...
>
> fio-3.8-21-ga8712
>
> Starting 1 thread
>
> Starting SPDK v18.04.1 / DPDK 18.02.0 initialization...
>
> [ DPDK EAL parameters: fio *-c 0x1* --file-prefix=spdk_pid6744 ]
>
> EAL: Detected 4 lcore(s)
>
> EAL: Multi-process socket /var/run/.spdk_pid6744_unix
>
> EAL: Probing VFIO support...
>
> ...
>
>
>
> If I run bdevperf with the same SPDK config file, I'm quite the contrary
> getting exactly what I asked for:
>
> ...
>
> Starting SPDK v18.04.1 / DPDK 18.02.0 initialization...
>
> [ DPDK EAL parameters: bdevtest -c 0x7 -m 1024 --file-prefix=spdk_pid6915 ]
>
> EAL: Detected 4 lcore(s)
>
> EAL: Multi-process socket /var/run/.spdk_pid6915_unix
>
> EAL: Probing VFIO support...
>
> ...
>
>
>
> Comments will be appreciated.
>
>
>
> Regards,
> Andrey
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
-- 

Regards,
Andrey

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 7513 bytes --]

[-- Attachment #3: image001.png --]
[-- Type: image/png, Size: 25926 bytes --]

[SPDK] bdev/fio_plugin coremask

[Andrey Kuzmin]

[-- Attachment #1: Type: text/plain, Size: 1119 bytes --]

Not sure if it's a bogus related to how fio_plugin initializes DPDK
environment, but I'm confused by the discrepancy between what I have in the
SPDK config file used to run FIO plugin, which is
...
  # Users can restrict work items to only run on certain cores by
  #  specifying a ReactorMask.  Default is to allow work items to run
  #  on core 0.
  *ReactorMask 0x7*
...
and what FIO tells me on startup:
...
fio-3.8-21-ga8712
Starting 1 thread
Starting SPDK v18.04.1 / DPDK 18.02.0 initialization...
[ DPDK EAL parameters: fio *-c 0x1* --file-prefix=spdk_pid6744 ]
EAL: Detected 4 lcore(s)
EAL: Multi-process socket /var/run/.spdk_pid6744_unix
EAL: Probing VFIO support...
...

If I run bdevperf with the same SPDK config file, I'm quite the contrary
getting exactly what I asked for:
...
Starting SPDK v18.04.1 / DPDK 18.02.0 initialization...
[ DPDK EAL parameters: bdevtest -c 0x7 -m 1024 --file-prefix=spdk_pid6915 ]
EAL: Detected 4 lcore(s)
EAL: Multi-process socket /var/run/.spdk_pid6915_unix
EAL: Probing VFIO support...
...

Comments will be appreciated.

Regards,
Andrey

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 1559 bytes --]