NCQ Priority Support and FIO

NCQ Priority Support and FIO

[Brandon Schwartz]

Hi,

In the latest kernel 4.10 there has been support added for NCQ
Priority commands.  I can test this functionality by using the option
prioclass=1 as in the following FIO script (100% RR) and I can see
that the prio bit is set on the 8k RR commands, but not the 64k RR
(expected).

[global]
direct=1
filename=/dev/sdb
runtime=2m
ioengine=libaio
norandommap

[High-Prio-8kRR-q8]
prioclass=1
iodepth=8
rw=randread
bs=8k

[Low-Prio-64kRR-q8]
prioclass=0
iodepth=8
rw=randread
bs=64k

Would it be possible to add something in FIO that would give finer
control over the split of high priority commands to low priority?
Ideally I would like to be able to test something like:

60% RandomRead / 40% RandomWrite but of those reads, have 15% be high priority

Or is there a way to get this sort of priority testing done with
current FIO?

Regards,
Brandon

Re: NCQ Priority Support and FIO

[Sitsofe Wheeler]

Hi,

Perhaps you could use multiple threads and utilise the flow
(http://fio.readthedocs.io/en/sphinx-doc/fio_doc.html#cmdoption-arg-flow
) control on pairs of threads?

On 2 February 2017 at 22:29, Brandon Schwartz <schwartz.xn@gmail.com> wrote:
>
> 60% RandomRead / 40% RandomWrite but of those reads, have 15% be high priority
>
> Or is there a way to get this sort of priority testing done with
> current FIO?

-- 
Sitsofe | http://sucs.org/~sits/

Re: NCQ Priority Support and FIO

[Brandon Schwartz]

Thanks for the info, I'll play around with that option and see if I
can make that work.  Is there anything that would allow me to control
the amount of priority commands from within a single thread?

Regards,
Brandon

On Thu, Feb 2, 2017 at 3:36 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> Hi,
>
> Perhaps you could use multiple threads and utilise the flow
> (http://fio.readthedocs.io/en/sphinx-doc/fio_doc.html#cmdoption-arg-flow
> ) control on pairs of threads?
>
> On 2 February 2017 at 22:29, Brandon Schwartz <schwartz.xn@gmail.com> wrote:
>>
>> 60% RandomRead / 40% RandomWrite but of those reads, have 15% be high priority
>>
>> Or is there a way to get this sort of priority testing done with
>> current FIO?
>
> --
> Sitsofe | http://sucs.org/~sits/

Re: NCQ Priority Support and FIO

[Sitsofe Wheeler]

Hi,

I can't think of anything if you mean "I want the same thread/single
job to send both normal AND high priority commands" so perhaps it
would have to be done as new patch to the pvsync2 ioengine...

On 4 February 2017 at 00:03, Brandon Schwartz <schwartz.xn@gmail.com> wrote:
> Thanks for the info, I'll play around with that option and see if I
> can make that work.  Is there anything that would allow me to control
> the amount of priority commands from within a single thread?
>
> Regards,
> Brandon
>
> On Thu, Feb 2, 2017 at 3:36 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>> Hi,
>>
>> Perhaps you could use multiple threads and utilise the flow
>> (http://fio.readthedocs.io/en/sphinx-doc/fio_doc.html#cmdoption-arg-flow
>> ) control on pairs of threads?
>>
>> On 2 February 2017 at 22:29, Brandon Schwartz <schwartz.xn@gmail.com> wrote:
>>>
>>> 60% RandomRead / 40% RandomWrite but of those reads, have 15% be high priority
>>>
>>> Or is there a way to get this sort of priority testing done with
>>> current FIO?
>>
>> --
>> Sitsofe | http://sucs.org/~sits/



-- 
Sitsofe | http://sucs.org/~sits/

Re: NCQ Priority Support and FIO

[Sitsofe Wheeler]

Hi Brandon,

Looks like someone went on to implement a design based on what you
were after https://github.com/xahmad/fio/commit/c45abdaa857bd30ccf4f20ed59340686fc3e9103
. It's not in mainline fio but perhaps if it was cleaned up it could
be submitted for inclusion?

On 4 February 2017 at 07:37, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>
> I can't think of anything if you mean "I want the same thread/single
> job to send both normal AND high priority commands" so perhaps it
> would have to be done as new patch to the pvsync2 ioengine...
>
> On 4 February 2017 at 00:03, Brandon Schwartz <schwartz.xn@gmail.com> wrote:
>> Thanks for the info, I'll play around with that option and see if I
>> can make that work.  Is there anything that would allow me to control
>> the amount of priority commands from within a single thread?
>>
>> On Thu, Feb 2, 2017 at 3:36 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>>
>>> Perhaps you could use multiple threads and utilise the flow
>>> (http://fio.readthedocs.io/en/sphinx-doc/fio_doc.html#cmdoption-arg-flow
>>> ) control on pairs of threads?
>>>
>>> On 2 February 2017 at 22:29, Brandon Schwartz <schwartz.xn@gmail.com> wrote:
>>>>
>>>> 60% RandomRead / 40% RandomWrite but of those reads, have 15% be high priority
>>>>
>>>> Or is there a way to get this sort of priority testing done with
>>>> current FIO?

-- 
Sitsofe | http://sucs.org/~sits/

Re: NCQ Priority Support and FIO

[Brandon Schwartz]

Hi Sitsofe,

Building off of the above change to control the priority bit, I'm
trying to add the ability to track completion latencies for high and
low prio read commands separately, but I'm not sure what I added to FIO is
exactly correct.  It seems to be very accurate for qdepths up to 8,
but at 16 and 32 I start seeing the FIO output for the high prio
commands be higher than what I see when I take a bus trace and parse
that (low prio latencies are what I would expect however).  What I am
trying to do is track the priority of the IO when it is set in
libaio.c via the following (notice the io_us[0]->priorityBit
additions):

https://github.com/brschwartz/fio/blob/10fdb93663e52beefa1f5dd836927d1e85c354d7/engines/libaio.c#L280

And then I grab that value again when the IO is complete and add the
associated latency to high and low prio read data structures which
then get outputted in the json output.

https://github.com/brschwartz/fio/commit/b7ad9003feb83fca57ec7438f5532426f597cec6
https://github.com/brschwartz/fio/blob/10fdb93663e52beefa1f5dd836927d1e85c354d7/stat.c#L1056

Does this seem right?  Or should I be tracking it a different way?
Thanks for any help/input!

Regards,
Brandon



On Sat, Apr 8, 2017 at 3:06 AM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>
> Hi Brandon,
>
> Looks like someone went on to implement a design based on what you
> were after https://github.com/xahmad/fio/commit/c45abdaa857bd30ccf4f20ed59340686fc3e9103
> . It's not in mainline fio but perhaps if it was cleaned up it could
> be submitted for inclusion?
>
> On 4 February 2017 at 07:37, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> >
> > I can't think of anything if you mean "I want the same thread/single
> > job to send both normal AND high priority commands" so perhaps it
> > would have to be done as new patch to the pvsync2 ioengine...
> >
> > On 4 February 2017 at 00:03, Brandon Schwartz <schwartz.xn@gmail.com> wrote:
> >> Thanks for the info, I'll play around with that option and see if I
> >> can make that work.  Is there anything that would allow me to control
> >> the amount of priority commands from within a single thread?
> >>
> >> On Thu, Feb 2, 2017 at 3:36 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> >>>
> >>> Perhaps you could use multiple threads and utilise the flow
> >>> (http://fio.readthedocs.io/en/sphinx-doc/fio_doc.html#cmdoption-arg-flow
> >>> ) control on pairs of threads?
> >>>
> >>> On 2 February 2017 at 22:29, Brandon Schwartz <schwartz.xn@gmail.com> wrote:
> >>>>
> >>>> 60% RandomRead / 40% RandomWrite but of those reads, have 15% be high priority
> >>>>
> >>>> Or is there a way to get this sort of priority testing done with
> >>>> current FIO?
>
> --
> Sitsofe | http://sucs.org/~sits/

Re: NCQ Priority Support and FIO

[Sitsofe Wheeler]

Hi,

On 5 September 2017 at 19:45, Brandon Schwartz <schwartz.xn@gmail.com> wrote:
>
> Building off of the above change to control the priority bit, I'm
> trying to add the ability to track completion latencies for high and
> low prio read commands separately, but I'm not sure what I added to FIO is
> exactly correct.  It seems to be very accurate for qdepths up to 8,
> but at 16 and 32 I start seeing the FIO output for the high prio
> commands be higher than what I see when I take a bus trace and parse
> that (low prio latencies are what I would expect however).  What I am
> trying to do is track the priority of the IO when it is set in
> libaio.c via the following (notice the io_us[0]->priorityBit
> additions):
>
> https://github.com/brschwartz/fio/blob/10fdb93663e52beefa1f5dd836927d1e85c354d7/engines/libaio.c#L280

This is going to get confusing fast as you are changing the priority
of the fio thread/process periodically rather than tagging the I/O
with a priority which it keeps until completion. Are you saying
batches of I/O all have the same priority and if so why aren't you
tagging all the I/Os that comprise your batch? What if you submit an
I/O then while waiting for completion you send another I/O off but
change your I/O priority - what does that mean for the first I/O when
it completes? My understanding is that ioprio_set only controls CFQ
I/O scheduler priorities and nothing else BTW (Jens said you would
need a new kernel interface to pass NCQ priorities from fio over on
https://www.spinics.net/lists/fio/msg06102.html ).

> rand()%100

You may want to use fio's pseudorandom generator. See
get_rand_file_size() in filesetup.c for an example.

> And then I grab that value again when the IO is complete and add the
> associated latency to high and low prio read data structures which
> then get outputted in the json output.
>
> https://github.com/brschwartz/fio/commit/b7ad9003feb83fca57ec7438f5532426f597cec6
> https://github.com/brschwartz/fio/blob/10fdb93663e52beefa1f5dd836927d1e85c354d7/stat.c#L1056
>
> Does this seem right?  Or should I be tracking it a different way?
> Thanks for any help/input!

I find the premise a bit confusing because unless you're using
direct=1 coupled with the appropriate batch options
(http://fio.readthedocs.io/en/latest/fio_doc.html#i-o-depth ) so you
maintain the same priority across the whole lifetime of an I/O.
Without at least that you don't know if the I/Os are sitting in a
queue in while you repeatedly flip priorities of the thread/process
and at that point what priority should the completed I/O be attributed
to?

-- 
Sitsofe | http://sucs.org/~sits/