buffer-cache builds up with invalidate=1 too

buffer-cache builds up with invalidate=1 too

[Paolo Valente]

Hi,
in a time_based test with direct=0, a small file happened to be read several times. Yet I/O was actually done only when the file was read for the first time, even if invalidate was set.

So, if I'm not mistaken, the buffer cache still builds up even if invalidate is set, right?

Then, in a time_based test, the only way to make sure that I/O is constantly done, is to use direct=1, isn't it? Of course, unless the file is so large that one knows that no portion of the file will be found in the buffer cache when read again.

Thanks,
Paolo

Re: buffer-cache builds up with invalidate=1 too

[Sitsofe Wheeler]

Hi,

On 23 October 2017 at 11:02, Paolo Valente <paolo.valente@linaro.org> wrote:
>
> in a time_based test with direct=0, a small file happened to be read several times. Yet I/O was actually done only when the file was read for the first time, even if invalidate was set.
>
> So, if I'm not mistaken, the buffer cache still builds up even if invalidate is set, right?

Correct - invalidate just invalidates the cache at the "beginning"
(see http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-invalidate
). See below where I agree with using your direct=1 suggestion if you
want to avoid the cache. See
http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-direct
for documentation on direct.

> Then, in a time_based test, the only way to make sure that I/O is constantly done, is to use direct=1, isn't it? Of course, unless the file is so large that one knows that no portion of the file will be found in the buffer cache when read again.

If you mean "the only way for fio to try and avoid all I/O being
cached/being only serviced by Linux's buffer cache is to use direct=1"
then yes. Also as you suggest you can do I/O over a region big enough
that a significant amount of I/O must not be cacheable therefore the
average speed would better reflect that of the underlying disk (that's
why you hear the old bonnie benchmark rule that you had to use a file
size at least twice memory etc as it had no cache bypass mode). If you
do a lot of I/O to only a small region then this is clearly still
cachable though.

-- 
Sitsofe | http://sucs.org/~sits/

Re: buffer-cache builds up with invalidate=1 too

[Paolo Valente]

> Il giorno 23 ott 2017, alle ore 20:26, Sitsofe Wheeler <sitsofe@gmail.com> ha scritto:
> 
> Hi,
> 
> On 23 October 2017 at 11:02, Paolo Valente <paolo.valente@linaro.org> wrote:
>> 
>> in a time_based test with direct=0, a small file happened to be read several times. Yet I/O was actually done only when the file was read for the first time, even if invalidate was set.
>> 
>> So, if I'm not mistaken, the buffer cache still builds up even if invalidate is set, right?
> 
> Correct - invalidate just invalidates the cache at the "beginning"
> (see http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-invalidate
> ).

Hi,
thank you very much for answering this and my other question
thoroughly!

Maybe I would specify, in the description of invalidate, that the
invalidation is not repeated in case the I/O on a file restarts from
the beginning go the file.

Thanks,
Paolo

> See below where I agree with using your direct=1 suggestion if you
> want to avoid the cache. See
> http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-direct
> for documentation on direct.
> 
>> Then, in a time_based test, the only way to make sure that I/O is constantly done, is to use direct=1, isn't it? Of course, unless the file is so large that one knows that no portion of the file will be found in the buffer cache when read again.
> 
> If you mean "the only way for fio to try and avoid all I/O being
> cached/being only serviced by Linux's buffer cache is to use direct=1"
> then yes. Also as you suggest you can do I/O over a region big enough
> that a significant amount of I/O must not be cacheable therefore the
> average speed would better reflect that of the underlying disk (that's
> why you hear the old bonnie benchmark rule that you had to use a file
> size at least twice memory etc as it had no cache bypass mode). If you
> do a lot of I/O to only a small region then this is clearly still
> cachable though.
> 
> -- 
> Sitsofe | http://sucs.org/~sits/

Re: buffer-cache builds up with invalidate=1 too

[Sitsofe Wheeler]

Hi,

If memory serves it's actually slightly more complicated. If you are
using loops=<number> then I *think* (you'll have to check) you will
find that invalidation happens once per each loop start. However when
you use time_based to do the repetition there is essentially only one
"loop" (even though the job goes on forever) so loop actions only
happen right at the start of the job with that option (that's why I
put the scare quotes around "beginning" ;-).

On 24 October 2017 at 04:23, Paolo Valente <paolo.valente@linaro.org> wrote:
>
> thank you very much for answering this and my other question
> thoroughly!
>
> Maybe I would specify, in the description of invalidate, that the
> invalidation is not repeated in case the I/O on a file restarts from
> the beginning go the file.

-- 
Sitsofe | http://sucs.org/~sits/

Re: buffer-cache builds up with invalidate=1 too

[Paolo Valente]

> Il giorno 24 ott 2017, alle ore 08:28, Sitsofe Wheeler <sitsofe@gmail.com> ha scritto:
> 
> Hi,
> 
> If memory serves it's actually slightly more complicated. If you are
> using loops=<number> then I *think* (you'll have to check) you will
> find that invalidation happens once per each loop start. However when
> you use time_based to do the repetition there is essentially only one
> "loop" (even though the job goes on forever) so loop actions only
> happen right at the start of the job with that option (that's why I
> put the scare quotes around "beginning" ;-).
> 

Thanks for this additional, useful piece of information.  Actually,
this further, possibly different caching behavior makes me think that
some extra comment in the manpage might be helpful.

Thanks,
Paolo

> On 24 October 2017 at 04:23, Paolo Valente <paolo.valente@linaro.org> wrote:
>> 
>> thank you very much for answering this and my other question
>> thoroughly!
>> 
>> Maybe I would specify, in the description of invalidate, that the
>> invalidation is not repeated in case the I/O on a file restarts from
>> the beginning go the file.
> 
> -- 
> Sitsofe | http://sucs.org/~sits/

Re: buffer-cache builds up with invalidate=1 too

[Jens Axboe]

On 10/24/2017 08:10 AM, Paolo Valente wrote:
> 
>> Il giorno 24 ott 2017, alle ore 08:28, Sitsofe Wheeler <sitsofe@gmail.com> ha scritto:
>>
>> Hi,
>>
>> If memory serves it's actually slightly more complicated. If you are
>> using loops=<number> then I *think* (you'll have to check) you will
>> find that invalidation happens once per each loop start. However when
>> you use time_based to do the repetition there is essentially only one
>> "loop" (even though the job goes on forever) so loop actions only
>> happen right at the start of the job with that option (that's why I
>> put the scare quotes around "beginning" ;-).
>>
> 
> Thanks for this additional, useful piece of information.  Actually,
> this further, possibly different caching behavior makes me think that
> some extra comment in the manpage might be helpful.

Would probably make sense to change 'invalidate' to be a range of
possible values:

0	As it is now, never invalidate
1	As it is now, invalidate initially
once	Same as '1', invalidate initially / once
open	New value, invalidate on every open
close	New value, invalidate on close

as I can definitely see reasons why you would want to invalidate every
time you open the file.

To do that, the 'invalidate' option should be changed from a
FIO_OPT_BOOL to a FIO_OPT_STR, and the above possible values should be
added as posval[] for that option.

Compliment that with the an enum of ranges for the ovals:

enum {
	FIO_FILE_INVALIDATE_OFF	= 0,
	FIO_FILE_INVALIDATE_ONCE,
	FIO_FILE_INVALIDATE_OPEN,
	FIO_FILE_INVALIDATE_CLOSE
};

Hope this makes sense, should be trivial to add as most of the work is
already documented in this email :-). The remaining bits is just calling
file_invalidate_cache() in the proper locations,
td_io_{open,close}_file() would be prime candidates.

-- 
Jens Axboe

Re: buffer-cache builds up with invalidate=1 too

[Jens Axboe]

On 10/25/2017 10:32 PM, Jens Axboe wrote:
> On 10/24/2017 08:10 AM, Paolo Valente wrote:
>>
>>> Il giorno 24 ott 2017, alle ore 08:28, Sitsofe Wheeler <sitsofe@gmail.com> ha scritto:
>>>
>>> Hi,
>>>
>>> If memory serves it's actually slightly more complicated. If you are
>>> using loops=<number> then I *think* (you'll have to check) you will
>>> find that invalidation happens once per each loop start. However when
>>> you use time_based to do the repetition there is essentially only one
>>> "loop" (even though the job goes on forever) so loop actions only
>>> happen right at the start of the job with that option (that's why I
>>> put the scare quotes around "beginning" ;-).
>>>
>>
>> Thanks for this additional, useful piece of information.  Actually,
>> this further, possibly different caching behavior makes me think that
>> some extra comment in the manpage might be helpful.
> 
> Would probably make sense to change 'invalidate' to be a range of
> possible values:
> 
> 0	As it is now, never invalidate
> 1	As it is now, invalidate initially
> once	Same as '1', invalidate initially / once
> open	New value, invalidate on every open
> close	New value, invalidate on close
> 
> as I can definitely see reasons why you would want to invalidate every
> time you open the file.
> 
> To do that, the 'invalidate' option should be changed from a
> FIO_OPT_BOOL to a FIO_OPT_STR, and the above possible values should be
> added as posval[] for that option.
> 
> Compliment that with the an enum of ranges for the ovals:
> 
> enum {
> 	FIO_FILE_INVALIDATE_OFF	= 0,
> 	FIO_FILE_INVALIDATE_ONCE,
> 	FIO_FILE_INVALIDATE_OPEN,
> 	FIO_FILE_INVALIDATE_CLOSE
> };
> 
> Hope this makes sense, should be trivial to add as most of the work is
> already documented in this email :-). The remaining bits is just calling
> file_invalidate_cache() in the proper locations,
> td_io_{open,close}_file() would be prime candidates.

On a plane, so decided to take a look at this. Looking at how it's
implemented, fio actually does do cache invalidation at open time,
if you look at td_io_open_file(). I traced it, and it's invoked just
fine by default, and if you use loops=X (with X > 1).

For time_based, we don't open and close the file, so we don't trigger
that at all. We just loop/reset the offsets. Something like the below
should do the trick.


diff --git a/io_u.c b/io_u.c
index fb4180a3bc35..34d8cf7e986d 100644
--- a/io_u.c
+++ b/io_u.c
@@ -323,6 +323,15 @@ fetch:
 	goto fetch;
 }
 
+static void loop_cache_invalidate(struct thread_data *td, struct fio_file *f)
+{
+	if (td->o.invalidate_cache) {
+		int fio_unused ret;
+
+		ret = file_invalidate_cache(td, f);
+	}
+}
+
 static int get_next_rand_block(struct thread_data *td, struct fio_file *f,
 			       enum fio_ddir ddir, uint64_t *b)
 {
@@ -334,6 +343,7 @@ static int get_next_rand_block(struct thread_data *td, struct fio_file *f,
 		fio_file_reset(td, f);
 		if (!get_next_rand_offset(td, f, ddir, b))
 			return 0;
+		loop_cache_invalidate(td, f);
 	}
 
 	dprint(FD_IO, "%s: rand offset failed, last=%llu, size=%llu\n",
@@ -358,6 +368,8 @@ static int get_next_seq_offset(struct thread_data *td, struct fio_file *f,
 			f->last_pos[ddir] = 0;
 		else
 			f->last_pos[ddir] = f->last_pos[ddir] - io_size;
+
+		loop_cache_invalidate(td, f);
 	}
 
 	if (f->last_pos[ddir] < f->real_file_size) {

-- 
Jens Axboe

Re: buffer-cache builds up with invalidate=1 too

[Sitsofe Wheeler]

On 26 October 2017 at 19:06, Jens Axboe <axboe@kernel.dk> wrote:
> On 10/25/2017 10:32 PM, Jens Axboe wrote:
>> On 10/24/2017 08:10 AM, Paolo Valente wrote:
>>>
>>>> Il giorno 24 ott 2017, alle ore 08:28, Sitsofe Wheeler <sitsofe@gmail.com> ha scritto:
>>>>
>>>> Hi,
>>>>
>>>> If memory serves it's actually slightly more complicated. If you are
>>>> using loops=<number> then I *think* (you'll have to check) you will
>>>> find that invalidation happens once per each loop start. However when
>>>> you use time_based to do the repetition there is essentially only one
>>>> "loop" (even though the job goes on forever) so loop actions only
>>>> happen right at the start of the job with that option (that's why I
>>>> put the scare quotes around "beginning" ;-).
>>>>
>>>
>>> Thanks for this additional, useful piece of information.  Actually,
>>> this further, possibly different caching behavior makes me think that
>>> some extra comment in the manpage might be helpful.
>>
>> Would probably make sense to change 'invalidate' to be a range of
>> possible values:
>>
>> 0     As it is now, never invalidate
>> 1     As it is now, invalidate initially
>> once  Same as '1', invalidate initially / once
>> open  New value, invalidate on every open
>> close New value, invalidate on close
>>
>> as I can definitely see reasons why you would want to invalidate every
>> time you open the file.
>>
>> To do that, the 'invalidate' option should be changed from a
>> FIO_OPT_BOOL to a FIO_OPT_STR, and the above possible values should be
>> added as posval[] for that option.
>>
>> Compliment that with the an enum of ranges for the ovals:
>>
>> enum {
>>       FIO_FILE_INVALIDATE_OFF = 0,
>>       FIO_FILE_INVALIDATE_ONCE,
>>       FIO_FILE_INVALIDATE_OPEN,
>>       FIO_FILE_INVALIDATE_CLOSE
>> };
>>
>> Hope this makes sense, should be trivial to add as most of the work is
>> already documented in this email :-). The remaining bits is just calling
>> file_invalidate_cache() in the proper locations,
>> td_io_{open,close}_file() would be prime candidates.
>
> On a plane, so decided to take a look at this. Looking at how it's
> implemented, fio actually does do cache invalidation at open time,
> if you look at td_io_open_file(). I traced it, and it's invoked just
> fine by default, and if you use loops=X (with X > 1).

I was going to say - when I did the "windowsaio invalidate" I noticed
invalidation seemed linked to open.

> For time_based, we don't open and close the file, so we don't trigger
> that at all. We just loop/reset the offsets. Something like the below
> should do the trick.

Unfortunately I don't think it's easy to do effective invalidation
with open file handles on Windows so I think windowaio will have to
stay as is when using time_based. Is it OK that we don't change that
ioengine for now?

-- 
Sitsofe | http://sucs.org/~sits/

Re: buffer-cache builds up with invalidate=1 too

[Jens Axboe]

On 10/26/2017 01:05 PM, Sitsofe Wheeler wrote:
> On 26 October 2017 at 19:06, Jens Axboe <axboe@kernel.dk> wrote:
>> On 10/25/2017 10:32 PM, Jens Axboe wrote:
>>> On 10/24/2017 08:10 AM, Paolo Valente wrote:
>>>>
>>>>> Il giorno 24 ott 2017, alle ore 08:28, Sitsofe Wheeler <sitsofe@gmail.com> ha scritto:
>>>>>
>>>>> Hi,
>>>>>
>>>>> If memory serves it's actually slightly more complicated. If you are
>>>>> using loops=<number> then I *think* (you'll have to check) you will
>>>>> find that invalidation happens once per each loop start. However when
>>>>> you use time_based to do the repetition there is essentially only one
>>>>> "loop" (even though the job goes on forever) so loop actions only
>>>>> happen right at the start of the job with that option (that's why I
>>>>> put the scare quotes around "beginning" ;-).
>>>>>
>>>>
>>>> Thanks for this additional, useful piece of information.  Actually,
>>>> this further, possibly different caching behavior makes me think that
>>>> some extra comment in the manpage might be helpful.
>>>
>>> Would probably make sense to change 'invalidate' to be a range of
>>> possible values:
>>>
>>> 0     As it is now, never invalidate
>>> 1     As it is now, invalidate initially
>>> once  Same as '1', invalidate initially / once
>>> open  New value, invalidate on every open
>>> close New value, invalidate on close
>>>
>>> as I can definitely see reasons why you would want to invalidate every
>>> time you open the file.
>>>
>>> To do that, the 'invalidate' option should be changed from a
>>> FIO_OPT_BOOL to a FIO_OPT_STR, and the above possible values should be
>>> added as posval[] for that option.
>>>
>>> Compliment that with the an enum of ranges for the ovals:
>>>
>>> enum {
>>>       FIO_FILE_INVALIDATE_OFF = 0,
>>>       FIO_FILE_INVALIDATE_ONCE,
>>>       FIO_FILE_INVALIDATE_OPEN,
>>>       FIO_FILE_INVALIDATE_CLOSE
>>> };
>>>
>>> Hope this makes sense, should be trivial to add as most of the work is
>>> already documented in this email :-). The remaining bits is just calling
>>> file_invalidate_cache() in the proper locations,
>>> td_io_{open,close}_file() would be prime candidates.
>>
>> On a plane, so decided to take a look at this. Looking at how it's
>> implemented, fio actually does do cache invalidation at open time,
>> if you look at td_io_open_file(). I traced it, and it's invoked just
>> fine by default, and if you use loops=X (with X > 1).
> 
> I was going to say - when I did the "windowsaio invalidate" I noticed
> invalidation seemed linked to open.
> 
>> For time_based, we don't open and close the file, so we don't trigger
>> that at all. We just loop/reset the offsets. Something like the below
>> should do the trick.
> 
> Unfortunately I don't think it's easy to do effective invalidation
> with open file handles on Windows so I think windowaio will have to
> stay as is when using time_based. Is it OK that we don't change that
> ioengine for now?

It would be cleaner if we just closed/opened the files... But that's
easier said than done, since we'd have to quiesce IO and wait for
references to the file to go away.

And even then it won't necessarily work on Windows, since it only
works if that instance is the only opener of it.

Haven't we documented that invalidation on windows is best effort?
It already is, even from the initial run.

-- 
Jens Axboe

Re: buffer-cache builds up with invalidate=1 too

[Paolo Valente]

[-- Attachment #1: Type: text/plain, Size: 2016 bytes --]

Il 26/ott/2017 06:32 AM, "Jens Axboe" <axboe@kernel.dk> ha scritto:

On 10/24/2017 08:10 AM, Paolo Valente wrote:
>
>> Il giorno 24 ott 2017, alle ore 08:28, Sitsofe Wheeler <sitsofe@gmail.com>
ha scritto:
>>
>> Hi,
>>
>> If memory serves it's actually slightly more complicated. If you are
>> using loops=<number> then I *think* (you'll have to check) you will
>> find that invalidation happens once per each loop start. However when
>> you use time_based to do the repetition there is essentially only one
>> "loop" (even though the job goes on forever) so loop actions only
>> happen right at the start of the job with that option (that's why I
>> put the scare quotes around "beginning" ;-).
>>
>
> Thanks for this additional, useful piece of information.  Actually,
> this further, possibly different caching behavior makes me think that
> some extra comment in the manpage might be helpful.

Would probably make sense to change 'invalidate' to be a range of
possible values:

0       As it is now, never invalidate
1       As it is now, invalidate initially
once    Same as '1', invalidate initially / once
open    New value, invalidate on every open
close   New value, invalidate on close

as I can definitely see reasons why you would want to invalidate every
time you open the file.

To do that, the 'invalidate' option should be changed from a
FIO_OPT_BOOL to a FIO_OPT_STR, and the above possible values should be
added as posval[] for that option.

Compliment that with the an enum of ranges for the ovals:

enum {
        FIO_FILE_INVALIDATE_OFF = 0,
        FIO_FILE_INVALIDATE_ONCE,
        FIO_FILE_INVALIDATE_OPEN,
        FIO_FILE_INVALIDATE_CLOSE
};

Hope this makes sense, should be trivial to add as most of the work is
already documented in this email :-). The remaining bits is just calling
file_invalidate_cache() in the proper locations,
td_io_{open,close}_file() would be prime candidates.


IMO this solution would make things both clearer and more flexible

Thanks,
Paolo


--
Jens Axboe

[-- Attachment #2: Type: text/html, Size: 3185 bytes --]

Re: buffer-cache builds up with invalidate=1 too

[Paolo Valente]

[RESENDING, BECAUSE REJECTED BY THE VGER]

> Il giorno 27 ott 2017, alle ore 08:22, Paolo Valente <paolo.valente@linaro.org> ha scritto:
> 
> 
> 
> Il 26/ott/2017 06:32 AM, "Jens Axboe" <axboe@kernel.dk> ha scritto:
> On 10/24/2017 08:10 AM, Paolo Valente wrote:
> >
> >> Il giorno 24 ott 2017, alle ore 08:28, Sitsofe Wheeler <sitsofe@gmail.com> ha scritto:
> >>
> >> Hi,
> >>
> >> If memory serves it's actually slightly more complicated. If you are
> >> using loops=<number> then I *think* (you'll have to check) you will
> >> find that invalidation happens once per each loop start. However when
> >> you use time_based to do the repetition there is essentially only one
> >> "loop" (even though the job goes on forever) so loop actions only
> >> happen right at the start of the job with that option (that's why I
> >> put the scare quotes around "beginning" ;-).
> >>
> >
> > Thanks for this additional, useful piece of information.  Actually,
> > this further, possibly different caching behavior makes me think that
> > some extra comment in the manpage might be helpful.
> 
> Would probably make sense to change 'invalidate' to be a range of
> possible values:
> 
> 0       As it is now, never invalidate
> 1       As it is now, invalidate initially
> once    Same as '1', invalidate initially / once
> open    New value, invalidate on every open
> close   New value, invalidate on close
> 
> as I can definitely see reasons why you would want to invalidate every
> time you open the file.
> 
> To do that, the 'invalidate' option should be changed from a
> FIO_OPT_BOOL to a FIO_OPT_STR, and the above possible values should be
> added as posval[] for that option.
> 
> Compliment that with the an enum of ranges for the ovals:
> 
> enum {
>         FIO_FILE_INVALIDATE_OFF = 0,
>         FIO_FILE_INVALIDATE_ONCE,
>         FIO_FILE_INVALIDATE_OPEN,
>         FIO_FILE_INVALIDATE_CLOSE
> };
> 
> Hope this makes sense, should be trivial to add as most of the work is
> already documented in this email :-). The remaining bits is just calling
> file_invalidate_cache() in the proper locations,
> td_io_{open,close}_file() would be prime candidates.
> 
> IMO this solution would make things both clearer and more flexible
> 
> Thanks,
> Paolo
> 
> --
> Jens Axboe

Re: buffer-cache builds up with invalidate=1 too

[Jens Axboe]

On 10/27/2017 12:52 AM, Paolo Valente wrote:
> [RESENDING, BECAUSE REJECTED BY THE VGER]
> 
>> Il giorno 27 ott 2017, alle ore 08:22, Paolo Valente <paolo.valente@linaro.org> ha scritto:
>>
>>
>>
>> Il 26/ott/2017 06:32 AM, "Jens Axboe" <axboe@kernel.dk> ha scritto:
>> On 10/24/2017 08:10 AM, Paolo Valente wrote:
>>>
>>>> Il giorno 24 ott 2017, alle ore 08:28, Sitsofe Wheeler <sitsofe@gmail.com> ha scritto:
>>>>
>>>> Hi,
>>>>
>>>> If memory serves it's actually slightly more complicated. If you are
>>>> using loops=<number> then I *think* (you'll have to check) you will
>>>> find that invalidation happens once per each loop start. However when
>>>> you use time_based to do the repetition there is essentially only one
>>>> "loop" (even though the job goes on forever) so loop actions only
>>>> happen right at the start of the job with that option (that's why I
>>>> put the scare quotes around "beginning" ;-).
>>>>
>>>
>>> Thanks for this additional, useful piece of information.  Actually,
>>> this further, possibly different caching behavior makes me think that
>>> some extra comment in the manpage might be helpful.
>>
>> Would probably make sense to change 'invalidate' to be a range of
>> possible values:
>>
>> 0       As it is now, never invalidate
>> 1       As it is now, invalidate initially
>> once    Same as '1', invalidate initially / once
>> open    New value, invalidate on every open
>> close   New value, invalidate on close
>>
>> as I can definitely see reasons why you would want to invalidate every
>> time you open the file.
>>
>> To do that, the 'invalidate' option should be changed from a
>> FIO_OPT_BOOL to a FIO_OPT_STR, and the above possible values should be
>> added as posval[] for that option.
>>
>> Compliment that with the an enum of ranges for the ovals:
>>
>> enum {
>>         FIO_FILE_INVALIDATE_OFF = 0,
>>         FIO_FILE_INVALIDATE_ONCE,
>>         FIO_FILE_INVALIDATE_OPEN,
>>         FIO_FILE_INVALIDATE_CLOSE
>> };
>>
>> Hope this makes sense, should be trivial to add as most of the work is
>> already documented in this email :-). The remaining bits is just calling
>> file_invalidate_cache() in the proper locations,
>> td_io_{open,close}_file() would be prime candidates.
>>
>> IMO this solution would make things both clearer and more flexible

See my followup, fio already does invalidates for each open. The
problem with time_based was that we just reset the file, we don't
close and re-open it. That was fixed in git:

commit 0bcf41cdc22dfee6b3f3b2ba9a533b4b103c70c2
Author: Jens Axboe <axboe@kernel.dk>
Date:   Thu Oct 26 12:08:20 2017 -0600

    io_u: re-invalidate cache when looping around without file open/close

so current git should work for your test case. Please test.

-- 
Jens Axboe

Re: buffer-cache builds up with invalidate=1 too

[Paolo Valente]

> Il giorno 27 ott 2017, alle ore 16:21, Jens Axboe <axboe@kernel.dk> ha scritto:
> 
> On 10/27/2017 12:52 AM, Paolo Valente wrote:
>> [RESENDING, BECAUSE REJECTED BY THE VGER]
>> 
>>> Il giorno 27 ott 2017, alle ore 08:22, Paolo Valente <paolo.valente@linaro.org> ha scritto:
>>> 
>>> 
>>> 
>>> Il 26/ott/2017 06:32 AM, "Jens Axboe" <axboe@kernel.dk> ha scritto:
>>> On 10/24/2017 08:10 AM, Paolo Valente wrote:
>>>> 
>>>>> Il giorno 24 ott 2017, alle ore 08:28, Sitsofe Wheeler <sitsofe@gmail.com> ha scritto:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> If memory serves it's actually slightly more complicated. If you are
>>>>> using loops=<number> then I *think* (you'll have to check) you will
>>>>> find that invalidation happens once per each loop start. However when
>>>>> you use time_based to do the repetition there is essentially only one
>>>>> "loop" (even though the job goes on forever) so loop actions only
>>>>> happen right at the start of the job with that option (that's why I
>>>>> put the scare quotes around "beginning" ;-).
>>>>> 
>>>> 
>>>> Thanks for this additional, useful piece of information.  Actually,
>>>> this further, possibly different caching behavior makes me think that
>>>> some extra comment in the manpage might be helpful.
>>> 
>>> Would probably make sense to change 'invalidate' to be a range of
>>> possible values:
>>> 
>>> 0       As it is now, never invalidate
>>> 1       As it is now, invalidate initially
>>> once    Same as '1', invalidate initially / once
>>> open    New value, invalidate on every open
>>> close   New value, invalidate on close
>>> 
>>> as I can definitely see reasons why you would want to invalidate every
>>> time you open the file.
>>> 
>>> To do that, the 'invalidate' option should be changed from a
>>> FIO_OPT_BOOL to a FIO_OPT_STR, and the above possible values should be
>>> added as posval[] for that option.
>>> 
>>> Compliment that with the an enum of ranges for the ovals:
>>> 
>>> enum {
>>>        FIO_FILE_INVALIDATE_OFF = 0,
>>>        FIO_FILE_INVALIDATE_ONCE,
>>>        FIO_FILE_INVALIDATE_OPEN,
>>>        FIO_FILE_INVALIDATE_CLOSE
>>> };
>>> 
>>> Hope this makes sense, should be trivial to add as most of the work is
>>> already documented in this email :-). The remaining bits is just calling
>>> file_invalidate_cache() in the proper locations,
>>> td_io_{open,close}_file() would be prime candidates.
>>> 
>>> IMO this solution would make things both clearer and more flexible
> 
> See my followup, fio already does invalidates for each open. The
> problem with time_based was that we just reset the file, we don't
> close and re-open it. That was fixed in git:
> 
> commit 0bcf41cdc22dfee6b3f3b2ba9a533b4b103c70c2
> Author: Jens Axboe <axboe@kernel.dk>
> Date:   Thu Oct 26 12:08:20 2017 -0600
> 
>    io_u: re-invalidate cache when looping around without file open/close
> 
> so current git should work for your test case. Please test.
> 

Tested, it does solve the problem.  As a side note, and if useful for
you, the throughput is much higher with sequential reads and direct=0
(4.14-rc5, virtual disk on an SSD).  It happens because of merges,
which seem to not occur with direct=1.  I thought direct I/O skipped
buffering, but still enjoyed features as request merging, but probably
I'm just wrong.

Thanks for addressing this caching issue,
Paolo

> -- 
> Jens Axboe

Re: buffer-cache builds up with invalidate=1 too

[Paolo Valente]

> Il giorno 28 ott 2017, alle ore 10:37, Paolo Valente <paolo.valente@linaro.org> ha scritto:
> 
>> 
>> Il giorno 27 ott 2017, alle ore 16:21, Jens Axboe <axboe@kernel.dk> ha scritto:
>> 
>> On 10/27/2017 12:52 AM, Paolo Valente wrote:
>>> [RESENDING, BECAUSE REJECTED BY THE VGER]
>>> 
>>>> Il giorno 27 ott 2017, alle ore 08:22, Paolo Valente <paolo.valente@linaro.org> ha scritto:
>>>> 
>>>> 
>>>> 
>>>> Il 26/ott/2017 06:32 AM, "Jens Axboe" <axboe@kernel.dk> ha scritto:
>>>> On 10/24/2017 08:10 AM, Paolo Valente wrote:
>>>>> 
>>>>>> Il giorno 24 ott 2017, alle ore 08:28, Sitsofe Wheeler <sitsofe@gmail.com> ha scritto:
>>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> If memory serves it's actually slightly more complicated. If you are
>>>>>> using loops=<number> then I *think* (you'll have to check) you will
>>>>>> find that invalidation happens once per each loop start. However when
>>>>>> you use time_based to do the repetition there is essentially only one
>>>>>> "loop" (even though the job goes on forever) so loop actions only
>>>>>> happen right at the start of the job with that option (that's why I
>>>>>> put the scare quotes around "beginning" ;-).
>>>>>> 
>>>>> 
>>>>> Thanks for this additional, useful piece of information.  Actually,
>>>>> this further, possibly different caching behavior makes me think that
>>>>> some extra comment in the manpage might be helpful.
>>>> 
>>>> Would probably make sense to change 'invalidate' to be a range of
>>>> possible values:
>>>> 
>>>> 0       As it is now, never invalidate
>>>> 1       As it is now, invalidate initially
>>>> once    Same as '1', invalidate initially / once
>>>> open    New value, invalidate on every open
>>>> close   New value, invalidate on close
>>>> 
>>>> as I can definitely see reasons why you would want to invalidate every
>>>> time you open the file.
>>>> 
>>>> To do that, the 'invalidate' option should be changed from a
>>>> FIO_OPT_BOOL to a FIO_OPT_STR, and the above possible values should be
>>>> added as posval[] for that option.
>>>> 
>>>> Compliment that with the an enum of ranges for the ovals:
>>>> 
>>>> enum {
>>>>       FIO_FILE_INVALIDATE_OFF = 0,
>>>>       FIO_FILE_INVALIDATE_ONCE,
>>>>       FIO_FILE_INVALIDATE_OPEN,
>>>>       FIO_FILE_INVALIDATE_CLOSE
>>>> };
>>>> 
>>>> Hope this makes sense, should be trivial to add as most of the work is
>>>> already documented in this email :-). The remaining bits is just calling
>>>> file_invalidate_cache() in the proper locations,
>>>> td_io_{open,close}_file() would be prime candidates.
>>>> 
>>>> IMO this solution would make things both clearer and more flexible
>> 
>> See my followup, fio already does invalidates for each open. The
>> problem with time_based was that we just reset the file, we don't
>> close and re-open it. That was fixed in git:
>> 
>> commit 0bcf41cdc22dfee6b3f3b2ba9a533b4b103c70c2
>> Author: Jens Axboe <axboe@kernel.dk>
>> Date:   Thu Oct 26 12:08:20 2017 -0600
>> 
>>   io_u: re-invalidate cache when looping around without file open/close
>> 
>> so current git should work for your test case. Please test.
>> 
> 
> Tested, it does solve the problem.  As a side note, and if useful for
> you, the throughput is much higher with sequential reads and direct=0
> (4.14-rc5, virtual disk on an SSD).  It happens because of merges,
> which seem to not occur with direct=1.  I thought direct I/O skipped
> buffering, but still enjoyed features as request merging, but probably
> I'm just wrong.
> 

Just to be clear, by much higher I mean ten times higher.

Thanks,
Paolo

> Thanks for addressing this caching issue,
> Paolo
> 
>> -- 
>> Jens Axboe

Re: buffer-cache builds up with invalidate=1 too

[Jens Axboe]

On 10/28/2017 02:37 AM, Paolo Valente wrote:
> 
>> Il giorno 27 ott 2017, alle ore 16:21, Jens Axboe <axboe@kernel.dk> ha scritto:
>>
>> On 10/27/2017 12:52 AM, Paolo Valente wrote:
>>> [RESENDING, BECAUSE REJECTED BY THE VGER]
>>>
>>>> Il giorno 27 ott 2017, alle ore 08:22, Paolo Valente <paolo.valente@linaro.org> ha scritto:
>>>>
>>>>
>>>>
>>>> Il 26/ott/2017 06:32 AM, "Jens Axboe" <axboe@kernel.dk> ha scritto:
>>>> On 10/24/2017 08:10 AM, Paolo Valente wrote:
>>>>>
>>>>>> Il giorno 24 ott 2017, alle ore 08:28, Sitsofe Wheeler <sitsofe@gmail.com> ha scritto:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> If memory serves it's actually slightly more complicated. If you are
>>>>>> using loops=<number> then I *think* (you'll have to check) you will
>>>>>> find that invalidation happens once per each loop start. However when
>>>>>> you use time_based to do the repetition there is essentially only one
>>>>>> "loop" (even though the job goes on forever) so loop actions only
>>>>>> happen right at the start of the job with that option (that's why I
>>>>>> put the scare quotes around "beginning" ;-).
>>>>>>
>>>>>
>>>>> Thanks for this additional, useful piece of information.  Actually,
>>>>> this further, possibly different caching behavior makes me think that
>>>>> some extra comment in the manpage might be helpful.
>>>>
>>>> Would probably make sense to change 'invalidate' to be a range of
>>>> possible values:
>>>>
>>>> 0       As it is now, never invalidate
>>>> 1       As it is now, invalidate initially
>>>> once    Same as '1', invalidate initially / once
>>>> open    New value, invalidate on every open
>>>> close   New value, invalidate on close
>>>>
>>>> as I can definitely see reasons why you would want to invalidate every
>>>> time you open the file.
>>>>
>>>> To do that, the 'invalidate' option should be changed from a
>>>> FIO_OPT_BOOL to a FIO_OPT_STR, and the above possible values should be
>>>> added as posval[] for that option.
>>>>
>>>> Compliment that with the an enum of ranges for the ovals:
>>>>
>>>> enum {
>>>>        FIO_FILE_INVALIDATE_OFF = 0,
>>>>        FIO_FILE_INVALIDATE_ONCE,
>>>>        FIO_FILE_INVALIDATE_OPEN,
>>>>        FIO_FILE_INVALIDATE_CLOSE
>>>> };
>>>>
>>>> Hope this makes sense, should be trivial to add as most of the work is
>>>> already documented in this email :-). The remaining bits is just calling
>>>> file_invalidate_cache() in the proper locations,
>>>> td_io_{open,close}_file() would be prime candidates.
>>>>
>>>> IMO this solution would make things both clearer and more flexible
>>
>> See my followup, fio already does invalidates for each open. The
>> problem with time_based was that we just reset the file, we don't
>> close and re-open it. That was fixed in git:
>>
>> commit 0bcf41cdc22dfee6b3f3b2ba9a533b4b103c70c2
>> Author: Jens Axboe <axboe@kernel.dk>
>> Date:   Thu Oct 26 12:08:20 2017 -0600
>>
>>    io_u: re-invalidate cache when looping around without file open/close
>>
>> so current git should work for your test case. Please test.
>>
> 
> Tested, it does solve the problem.  As a side note, and if useful for
> you, the throughput is much higher with sequential reads and direct=0
> (4.14-rc5, virtual disk on an SSD).  It happens because of merges,
> which seem to not occur with direct=1.  I thought direct I/O skipped
> buffering, but still enjoyed features as request merging, but probably
> I'm just wrong.

What does your job file look like? If you have something like bs=4k,
then it's readahead saving your bacon with buffered. For O_DIRECT
and bs=4k, each IO is sync, so it's hard/impossible to get merging.
You need batched submission for that, through libaio for instance.

-- 
Jens Axboe

Re: buffer-cache builds up with invalidate=1 too

[Sitsofe Wheeler]

On 28 October 2017 at 09:37, Paolo Valente <paolo.valente@linaro.org> wrote:
>
> Tested, it does solve the problem.  As a side note, and if useful for
> you, the throughput is much higher with sequential reads and direct=0
> (4.14-rc5, virtual disk on an SSD).  It happens because of merges,
> which seem to not occur with direct=1.  I thought direct I/O skipped
> buffering, but still enjoyed features as request merging, but probably
> I'm just wrong.
>
> Thanks for addressing this caching issue,
> Paolo

I think the window for merging is smaller but non-zero with direct=1 -
remember all the I/Os must arrive close enough together to be merged
and if fio is doing its submit one in each batch/send one new as soon
as one completes default behaviour the odds of that happening are very
small. Do you have an iodepth greater than one and are you using an
async I/O engine? For example:

dd if=/dev/zero of=/mnt/iotrace/fio.tmp bs=1M count=1 oflag=sync
fio --filename /mnt/iotrace/fio.tmp --size=1M --rw=write
--ioengine=libaio --direct=1 --bs=512 --iodepth 32 --name=merge
--time_based --runtime=15s --iodepth=32 --iodepth_batch=32
--iodepth_low=1
[...]
Disk stats (read/write):
  sdb: ios=0/172944, merge=0/14619, ticks=0/286476, in_queue=286200, util=97.74%

-- 
Sitsofe | http://sucs.org/~sits/

Re: buffer-cache builds up with invalidate=1 too

[Paolo Valente]

> Il giorno 28 ott 2017, alle ore 16:20, Jens Axboe <axboe@kernel.dk> ha scritto:
> 
> On 10/28/2017 02:37 AM, Paolo Valente wrote:
>> 
>>> Il giorno 27 ott 2017, alle ore 16:21, Jens Axboe <axboe@kernel.dk> ha scritto:
>>> 
>>> On 10/27/2017 12:52 AM, Paolo Valente wrote:
>>>> [RESENDING, BECAUSE REJECTED BY THE VGER]
>>>> 
>>>>> Il giorno 27 ott 2017, alle ore 08:22, Paolo Valente <paolo.valente@linaro.org> ha scritto:
>>>>> 
>>>>> 
>>>>> 
>>>>> Il 26/ott/2017 06:32 AM, "Jens Axboe" <axboe@kernel.dk> ha scritto:
>>>>> On 10/24/2017 08:10 AM, Paolo Valente wrote:
>>>>>> 
>>>>>>> Il giorno 24 ott 2017, alle ore 08:28, Sitsofe Wheeler <sitsofe@gmail.com> ha scritto:
>>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> If memory serves it's actually slightly more complicated. If you are
>>>>>>> using loops=<number> then I *think* (you'll have to check) you will
>>>>>>> find that invalidation happens once per each loop start. However when
>>>>>>> you use time_based to do the repetition there is essentially only one
>>>>>>> "loop" (even though the job goes on forever) so loop actions only
>>>>>>> happen right at the start of the job with that option (that's why I
>>>>>>> put the scare quotes around "beginning" ;-).
>>>>>>> 
>>>>>> 
>>>>>> Thanks for this additional, useful piece of information.  Actually,
>>>>>> this further, possibly different caching behavior makes me think that
>>>>>> some extra comment in the manpage might be helpful.
>>>>> 
>>>>> Would probably make sense to change 'invalidate' to be a range of
>>>>> possible values:
>>>>> 
>>>>> 0       As it is now, never invalidate
>>>>> 1       As it is now, invalidate initially
>>>>> once    Same as '1', invalidate initially / once
>>>>> open    New value, invalidate on every open
>>>>> close   New value, invalidate on close
>>>>> 
>>>>> as I can definitely see reasons why you would want to invalidate every
>>>>> time you open the file.
>>>>> 
>>>>> To do that, the 'invalidate' option should be changed from a
>>>>> FIO_OPT_BOOL to a FIO_OPT_STR, and the above possible values should be
>>>>> added as posval[] for that option.
>>>>> 
>>>>> Compliment that with the an enum of ranges for the ovals:
>>>>> 
>>>>> enum {
>>>>>       FIO_FILE_INVALIDATE_OFF = 0,
>>>>>       FIO_FILE_INVALIDATE_ONCE,
>>>>>       FIO_FILE_INVALIDATE_OPEN,
>>>>>       FIO_FILE_INVALIDATE_CLOSE
>>>>> };
>>>>> 
>>>>> Hope this makes sense, should be trivial to add as most of the work is
>>>>> already documented in this email :-). The remaining bits is just calling
>>>>> file_invalidate_cache() in the proper locations,
>>>>> td_io_{open,close}_file() would be prime candidates.
>>>>> 
>>>>> IMO this solution would make things both clearer and more flexible
>>> 
>>> See my followup, fio already does invalidates for each open. The
>>> problem with time_based was that we just reset the file, we don't
>>> close and re-open it. That was fixed in git:
>>> 
>>> commit 0bcf41cdc22dfee6b3f3b2ba9a533b4b103c70c2
>>> Author: Jens Axboe <axboe@kernel.dk>
>>> Date:   Thu Oct 26 12:08:20 2017 -0600
>>> 
>>>   io_u: re-invalidate cache when looping around without file open/close
>>> 
>>> so current git should work for your test case. Please test.
>>> 
>> 
>> Tested, it does solve the problem.  As a side note, and if useful for
>> you, the throughput is much higher with sequential reads and direct=0
>> (4.14-rc5, virtual disk on an SSD).  It happens because of merges,
>> which seem to not occur with direct=1.  I thought direct I/O skipped
>> buffering, but still enjoyed features as request merging, but probably
>> I'm just wrong.
> 
> What does your job file look like? If you have something like bs=4k,
> then it's readahead saving your bacon with buffered. For O_DIRECT
> and bs=4k, each IO is sync, so it's hard/impossible to get merging.
> You need batched submission for that, through libaio for instance.
> 

Yes.  I tried several configurations in my job file, to thoroughly
test the cache issue I reported, and I ended up setting sync and 4k.
Then I reported this throughput gap, just as a possibly useful piece
of information, but without first getting to the bottom of it.  Your
feedback will help me bear in mind this other important difference
between direct and buffered: direct doesn't enjoy readahead.

Thanks,
Paolo

> -- 
> Jens Axboe

Re: buffer-cache builds up with invalidate=1 too

[Paolo Valente]

> Il giorno 28 ott 2017, alle ore 17:04, Sitsofe Wheeler <sitsofe@gmail.com> ha scritto:
> 
> On 28 October 2017 at 09:37, Paolo Valente <paolo.valente@linaro.org> wrote:
>> 
>> Tested, it does solve the problem.  As a side note, and if useful for
>> you, the throughput is much higher with sequential reads and direct=0
>> (4.14-rc5, virtual disk on an SSD).  It happens because of merges,
>> which seem to not occur with direct=1.  I thought direct I/O skipped
>> buffering, but still enjoyed features as request merging, but probably
>> I'm just wrong.
>> 
>> Thanks for addressing this caching issue,
>> Paolo
> 
> I think the window for merging is smaller but non-zero with direct=1 -
> remember all the I/Os must arrive close enough together to be merged
> and if fio is doing its submit one in each batch/send one new as soon
> as one completes default behaviour the odds of that happening are very
> small.

Exactly, as I wrote in my previous reply, I have of course seen this
problem with sync, and reported it too soon, sorry.

Thanks for your help,
Paolo



> Do you have an iodepth greater than one and are you using an
> async I/O engine? For example:
> 
> dd if=/dev/zero of=/mnt/iotrace/fio.tmp bs=1M count=1 oflag=sync
> fio --filename /mnt/iotrace/fio.tmp --size=1M --rw=write
> --ioengine=libaio --direct=1 --bs=512 --iodepth 32 --name=merge
> --time_based --runtime=15s --iodepth=32 --iodepth_batch=32
> --iodepth_low=1
> [...]
> Disk stats (read/write):
>  sdb: ios=0/172944, merge=0/14619, ticks=0/286476, in_queue=286200, util=97.74%
> 
> -- 
> Sitsofe | http://sucs.org/~sits/

Re: buffer-cache builds up with invalidate=1 too

[Paolo Valente]

> Il giorno 28 ott 2017, alle ore 16:20, Jens Axboe <axboe@kernel.dk> ha scritto:
> 
> On 10/28/2017 02:37 AM, Paolo Valente wrote:
>> 
>>> Il giorno 27 ott 2017, alle ore 16:21, Jens Axboe <axboe@kernel.dk> ha scritto:
>>> 
>>> On 10/27/2017 12:52 AM, Paolo Valente wrote:
>>>> [RESENDING, BECAUSE REJECTED BY THE VGER]
>>>> 
>>>>> Il giorno 27 ott 2017, alle ore 08:22, Paolo Valente <paolo.valente@linaro.org> ha scritto:
>>>>> 
>>>>> 
>>>>> 
>>>>> Il 26/ott/2017 06:32 AM, "Jens Axboe" <axboe@kernel.dk> ha scritto:
>>>>> On 10/24/2017 08:10 AM, Paolo Valente wrote:
>>>>>> 
>>>>>>> Il giorno 24 ott 2017, alle ore 08:28, Sitsofe Wheeler <sitsofe@gmail.com> ha scritto:
>>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> If memory serves it's actually slightly more complicated. If you are
>>>>>>> using loops=<number> then I *think* (you'll have to check) you will
>>>>>>> find that invalidation happens once per each loop start. However when
>>>>>>> you use time_based to do the repetition there is essentially only one
>>>>>>> "loop" (even though the job goes on forever) so loop actions only
>>>>>>> happen right at the start of the job with that option (that's why I
>>>>>>> put the scare quotes around "beginning" ;-).
>>>>>>> 
>>>>>> 
>>>>>> Thanks for this additional, useful piece of information.  Actually,
>>>>>> this further, possibly different caching behavior makes me think that
>>>>>> some extra comment in the manpage might be helpful.
>>>>> 
>>>>> Would probably make sense to change 'invalidate' to be a range of
>>>>> possible values:
>>>>> 
>>>>> 0       As it is now, never invalidate
>>>>> 1       As it is now, invalidate initially
>>>>> once    Same as '1', invalidate initially / once
>>>>> open    New value, invalidate on every open
>>>>> close   New value, invalidate on close
>>>>> 
>>>>> as I can definitely see reasons why you would want to invalidate every
>>>>> time you open the file.
>>>>> 
>>>>> To do that, the 'invalidate' option should be changed from a
>>>>> FIO_OPT_BOOL to a FIO_OPT_STR, and the above possible values should be
>>>>> added as posval[] for that option.
>>>>> 
>>>>> Compliment that with the an enum of ranges for the ovals:
>>>>> 
>>>>> enum {
>>>>>      FIO_FILE_INVALIDATE_OFF = 0,
>>>>>      FIO_FILE_INVALIDATE_ONCE,
>>>>>      FIO_FILE_INVALIDATE_OPEN,
>>>>>      FIO_FILE_INVALIDATE_CLOSE
>>>>> };
>>>>> 
>>>>> Hope this makes sense, should be trivial to add as most of the work is
>>>>> already documented in this email :-). The remaining bits is just calling
>>>>> file_invalidate_cache() in the proper locations,
>>>>> td_io_{open,close}_file() would be prime candidates.
>>>>> 
>>>>> IMO this solution would make things both clearer and more flexible
>>> 
>>> See my followup, fio already does invalidates for each open. The
>>> problem with time_based was that we just reset the file, we don't
>>> close and re-open it. That was fixed in git:
>>> 
>>> commit 0bcf41cdc22dfee6b3f3b2ba9a533b4b103c70c2
>>> Author: Jens Axboe <axboe@kernel.dk>
>>> Date:   Thu Oct 26 12:08:20 2017 -0600
>>> 
>>>  io_u: re-invalidate cache when looping around without file open/close
>>> 
>>> so current git should work for your test case. Please test.
>>> 
>> 
>> Tested, it does solve the problem.  As a side note, and if useful for
>> you, the throughput is much higher with sequential reads and direct=0
>> (4.14-rc5, virtual disk on an SSD).  It happens because of merges,
>> which seem to not occur with direct=1.  I thought direct I/O skipped
>> buffering, but still enjoyed features as request merging, but probably
>> I'm just wrong.
> 
> What does your job file look like? If you have something like bs=4k,
> then it's readahead saving your bacon with buffered. For O_DIRECT
> and bs=4k, each IO is sync, so it's hard/impossible to get merging.
> You need batched submission for that, through libaio for instance.
> 

Yes.  I tried several configurations in my job file, to thoroughly
test the cache issue I reported, and I ended up setting sync and 4k.
Then I reported this throughput gap, just as a possibly useful piece
of information, but without first getting to the bottom of it.  Your
feedback will help me bear in mind this other important difference
between direct and buffered: direct doesn't enjoy readahead.

Thanks,
Paolo

> -- 
> Jens Axboe