From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============8332999458680839334==" MIME-Version: 1.0 From: Walker, Benjamin Subject: Re: [SPDK] bdev/fio_plugin coremask Date: Mon, 30 Jul 2018 21:20:15 +0000 Message-ID: <539c3c86ceba1d5f65106dbcad232ff65f2fadd3.camel@intel.com> In-Reply-To: CANvN+ek5xMHJKHbNSy8X+h2prrajgeG5XUicqFtoDZxt1fissg@mail.gmail.com List-ID: To: spdk@lists.01.org --===============8332999458680839334== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Mon, 2018-07-30 at 22:58 +0300, Andrey Kuzmin wrote: > I believe my fio threads also affinitize anbd run as requested. What I'm > concerned with is whether spdk pinned threads are started as requested by= the > core mask, and it looks like they aren't. Not something really concerning= as > fio plugin runs atop fio threads, but still a discrepancy (and likeley wo= rth a > note in the plugin code/docs). Making DPDK and fio play together is very tricky. The main problem is that = both DPDK and fio want to be in charge of spawning and affinitizing the threads (they're both frameworks). There is simply no way to "hook" the creation of= a thread in fio to force it to use an existing one, nor is there a way to for= ce a thread to become a DPDK thread (there is no equivalent of spdk_allocate_thread(), for example). Fortunately, all of the DPDK data structures and API calls we depend on can be called safely from a non-DPDK thread. Some of them have worse performance in this case (the big one is th= at you lose access to the mempool's per-thread cache), but they all at least function correctly. So the strategy I took here was to spawn DPDK on just o= ne thread ("the initialization thread"), unaffinitize it, and do nothing on it. Then do all operations from the fio threads which are spawned according to = your fio configuration file. When we first implemented this, we did notice some fairly significant performance hits when pulling bdev_io out of the central pool. However, Jim added some code that made SPDK's bdev layer manage its own per-SPDK-thread bdev_io caches a few months ago, so this is no longer a concern. When SPDK threads become "lightweight" threads that can get rescheduled onto various = real threads at any time, changes like that will become increasingly important a= nyway - you want the cache to follow the "lightweight" thread around. Thanks, Ben > = > Regards, > Andrey > > = > > fio-3.3 > > Starting 2 threads > > Starting SPDK v18.04 / DPDK 18.02.0 initialization... > > [ DPDK EAL parameters: fio -c 0x1 --file-prefix=3Dspdk_pid133108 ] > > = > > But I can see the CPU cores being used are CPU core 28 and 29 per my fio > > configuration file. > > = > > = > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuz= min > > Sent: Sunday, July 29, 2018 1:47 PM > > To: Storage Performance Development Kit > > Subject: [SPDK] bdev/fio_plugin coremask > > = > > Not sure if it's a bogus related to how fio_plugin initializes DPDK > > environment, but I'm confused by the discrepancy between what I have in= the > > SPDK config file used to run FIO plugin, which is > > ... > > # Users can restrict work items to only run on certain cores by > > # specifying a ReactorMask. Default is to allow work items to run > > # on core 0. > > ReactorMask 0x7 > > ... > > and what FIO tells me on startup: > > ... > > fio-3.8-21-ga8712 > > Starting 1 thread > > Starting SPDK v18.04.1 / DPDK 18.02.0 initialization... > > [ DPDK EAL parameters: fio -c 0x1 --file-prefix=3Dspdk_pid6744 ] > > EAL: Detected 4 lcore(s) > > EAL: Multi-process socket /var/run/.spdk_pid6744_unix > > EAL: Probing VFIO support... > > ... > > = > > If I run bdevperf with the same SPDK config file, I'm quite the contrary > > getting exactly what I asked for: > > ... > > Starting SPDK v18.04.1 / DPDK 18.02.0 initialization... > > [ DPDK EAL parameters: bdevtest -c 0x7 -m 1024 --file-prefix=3Dspdk_pid= 6915 ] > > EAL: Detected 4 lcore(s) > > EAL: Multi-process socket /var/run/.spdk_pid6915_unix > > EAL: Probing VFIO support... > > ... > > = > > Comments will be appreciated. > > = > > Regards, > > Andrey > > _______________________________________________ > > SPDK mailing list > > SPDK(a)lists.01.org > > https://lists.01.org/mailman/listinfo/spdk > = > -- = > Regards, > Andrey > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============8332999458680839334==--