How to do strict synchronous i/o on Windows?

How to do strict synchronous i/o on Windows?

[Greg Sullivan]

I need to simulate strict synchronous, round robin i/o to a group of
files. I am on Windows 7 32-bit.
fio is very nearly working, except that even with a queue depth of 1,
it is still resulting in a disk queue that is > 1, because the
"iodepth" parameter is not global - it is per thread. (correct?)

I've tried using the "sync" engine, however that doesn't work at all -
just spews out errors.

Any assistance much appreciated.

Greg.

Re: How to do strict synchronous i/o on Windows?

[Greg Sullivan]

The error I receive when attempting to use the "sync" io engine in Windows 7:

"fio: pid=4904, err=5/file:backend.c:633, func=full resid,
error=Input/output error"

I tried psync - that ran, but didn't seem to do the proper io. vsync
didn't work at all.

If there are no synchronous engines available at the moment for
Windows, I suggest adding an option to simulate synchronous io when
using the Windows AIO engine.

Greg.


On 14 August 2012 16:24, Greg Sullivan <greg.sullivan@sullivang.net> wrote:
> I need to simulate strict synchronous, round robin i/o to a group of
> files. I am on Windows 7 32-bit.
> fio is very nearly working, except that even with a queue depth of 1,
> it is still resulting in a disk queue that is > 1, because the
> "iodepth" parameter is not global - it is per thread. (correct?)
>
> I've tried using the "sync" engine, however that doesn't work at all -
> just spews out errors.
>
> Any assistance much appreciated.
>
> Greg.

Re: How to do strict synchronous i/o on Windows?

[Jens Axboe]

On 08/14/2012 08:24 AM, Greg Sullivan wrote:
> I need to simulate strict synchronous, round robin i/o to a group of
> files. I am on Windows 7 32-bit.
> fio is very nearly working, except that even with a queue depth of 1,
> it is still resulting in a disk queue that is > 1, because the
> "iodepth" parameter is not global - it is per thread. (correct?)
> 
> I've tried using the "sync" engine, however that doesn't work at all -
> just spews out errors.

That'll be the case for ANY platform and IO engine. If you have more
than 1 thread or process going, you can have > 1 depth at the device
side. The definition of a sync IO call is that the call doesn't return
until the IO is done. If you have overlapped calls due to more than 1
thread, then that is no longer true.

What you are looking for is outside the scope of an application. You
would have to limit the queue depth on the operating system side to
achieve that. Or artificially limit fio in some way, which would not
make a lot of sense imho.

-- 
Jens Axboe

Re: How to do strict synchronous i/o on Windows?

[Greg Sullivan]

[-- Attachment #1: Type: text/plain, Size: 1401 bytes --]

Thanks Jens. I do in fact have an application that reads in exactly the
manner I described. I have monitored the queue depth - it does not rise
above 1.  It is a real time musical sample streamer.

Please consider this a new feature request for fio - thankyou.

Greg.
 On Aug 14, 2012 11:06 PM, "Jens Axboe" <axboe@kernel.dk> wrote:

> On 08/14/2012 08:24 AM, Greg Sullivan wrote:
> > I need to simulate strict synchronous, round robin i/o to a group of
> > files. I am on Windows 7 32-bit.
> > fio is very nearly working, except that even with a queue depth of 1,
> > it is still resulting in a disk queue that is > 1, because the
> > "iodepth" parameter is not global - it is per thread. (correct?)
> >
> > I've tried using the "sync" engine, however that doesn't work at all -
> > just spews out errors.
>
> That'll be the case for ANY platform and IO engine. If you have more
> than 1 thread or process going, you can have > 1 depth at the device
> side. The definition of a sync IO call is that the call doesn't return
> until the IO is done. If you have overlapped calls due to more than 1
> thread, then that is no longer true.
>
> What you are looking for is outside the scope of an application. You
> would have to limit the queue depth on the operating system side to
> achieve that. Or artificially limit fio in some way, which would not
> make a lot of sense imho.
>
> --
> Jens Axboe
>
>

[-- Attachment #2: Type: text/html, Size: 1792 bytes --]

Re: How to do strict synchronous i/o on Windows?

[Greg Sullivan]

Thanks Jens. I do in fact have an application that reads in exactly the
manner I described. I have monitored the queue depth - it does not rise
above 1.  It is a real time musical sample streamer.

Please consider this a new feature request for fio - thankyou.

Greg.


On Aug 14, 2012 11:06 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
>
> On 08/14/2012 08:24 AM, Greg Sullivan wrote:
> > I need to simulate strict synchronous, round robin i/o to a group of
> > files. I am on Windows 7 32-bit.
> > fio is very nearly working, except that even with a queue depth of 1,
> > it is still resulting in a disk queue that is > 1, because the
> > "iodepth" parameter is not global - it is per thread. (correct?)
> >
> > I've tried using the "sync" engine, however that doesn't work at all -
> > just spews out errors.
>
> That'll be the case for ANY platform and IO engine. If you have more
> than 1 thread or process going, you can have > 1 depth at the device
> side. The definition of a sync IO call is that the call doesn't return
> until the IO is done. If you have overlapped calls due to more than 1
> thread, then that is no longer true.
>
> What you are looking for is outside the scope of an application. You
> would have to limit the queue depth on the operating system side to
> achieve that. Or artificially limit fio in some way, which would not
> make a lot of sense imho.
>
> --
> Jens Axboe
>

Re: How to do strict synchronous i/o on Windows?

[Greg Sullivan]

How exactly does fio behave at the moment for round-robin file access,
when using synchronous io? I would have expected it to behave as I
wanted - the documentation says that the ios are farmed out to each
file in a round-robin fashion, so surely if the io is done
synchronously, it means that the thread issuing that io will wait
until that io has completed! And if it does that, the round-robin
allocator would not trigger the next "thread" to issue it's read until
the current io had completed, which is exactly what I want! :)

Greg.

On 14 August 2012 23:19, Greg Sullivan <greg.sullivan@sullivang.net> wrote:
> Thanks Jens. I do in fact have an application that reads in exactly the
> manner I described. I have monitored the queue depth - it does not rise
> above 1.  It is a real time musical sample streamer.
>
> Please consider this a new feature request for fio - thankyou.
>
> Greg.
>
>
> On Aug 14, 2012 11:06 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>
>> On 08/14/2012 08:24 AM, Greg Sullivan wrote:
>> > I need to simulate strict synchronous, round robin i/o to a group of
>> > files. I am on Windows 7 32-bit.
>> > fio is very nearly working, except that even with a queue depth of 1,
>> > it is still resulting in a disk queue that is > 1, because the
>> > "iodepth" parameter is not global - it is per thread. (correct?)
>> >
>> > I've tried using the "sync" engine, however that doesn't work at all -
>> > just spews out errors.
>>
>> That'll be the case for ANY platform and IO engine. If you have more
>> than 1 thread or process going, you can have > 1 depth at the device
>> side. The definition of a sync IO call is that the call doesn't return
>> until the IO is done. If you have overlapped calls due to more than 1
>> thread, then that is no longer true.
>>
>> What you are looking for is outside the scope of an application. You
>> would have to limit the queue depth on the operating system side to
>> achieve that. Or artificially limit fio in some way, which would not
>> make a lot of sense imho.
>>
>> --
>> Jens Axboe
>>

Re: How to do strict synchronous i/o on Windows?

[Martin Steigerwald]

Hi Greg, 

Please quote (http://learn.to/quote click "English")

Am Dienstag, 14. August 2012 schrieb Greg Sullivan:

> On Aug 14, 2012 11:06 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
> >
> > On 08/14/2012 08:24 AM, Greg Sullivan wrote:
> > > I need to simulate strict synchronous, round robin i/o to a group of
> > > files. I am on Windows 7 32-bit.
> > > fio is very nearly working, except that even with a queue depth of 1,
> > > it is still resulting in a disk queue that is > 1, because the
> > > "iodepth" parameter is not global - it is per thread. (correct?)
> > >
> > > I've tried using the "sync" engine, however that doesn't work at all -
> > > just spews out errors.
> >
> > That'll be the case for ANY platform and IO engine. If you have more
> > than 1 thread or process going, you can have > 1 depth at the device
> > side. The definition of a sync IO call is that the call doesn't return
> > until the IO is done. If you have overlapped calls due to more than 1
> > thread, then that is no longer true.
> >
> > What you are looking for is outside the scope of an application. You
> > would have to limit the queue depth on the operating system side to
> > achieve that. Or artificially limit fio in some way, which would not
> > make a lot of sense imho.

> Thanks Jens. I do in fact have an application that reads in exactly the
> manner I described. I have monitored the queue depth - it does not rise
> above 1.  It is a real time musical sample streamer.
> 
> Please consider this a new feature request for fio - thankyou.

Is this application multithreaded? If so, are mutiple threads doing I/O
at the same time? If not I�d suggest just testing with one job.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: How to do strict synchronous i/o on Windows?

[Greg Sullivan]

On 15 August 2012 03:36, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> Hi Greg,
>
> Please quote (http://learn.to/quote click "English")
>
> Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
>
>> On Aug 14, 2012 11:06 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
>> >
>> > On 08/14/2012 08:24 AM, Greg Sullivan wrote:
>> > > I need to simulate strict synchronous, round robin i/o to a group of
>> > > files. I am on Windows 7 32-bit.
>> > > fio is very nearly working, except that even with a queue depth of 1,
>> > > it is still resulting in a disk queue that is > 1, because the
>> > > "iodepth" parameter is not global - it is per thread. (correct?)
>> > >
>> > > I've tried using the "sync" engine, however that doesn't work at all -
>> > > just spews out errors.
>> >
>> > That'll be the case for ANY platform and IO engine. If you have more
>> > than 1 thread or process going, you can have > 1 depth at the device
>> > side. The definition of a sync IO call is that the call doesn't return
>> > until the IO is done. If you have overlapped calls due to more than 1
>> > thread, then that is no longer true.
>> >
>> > What you are looking for is outside the scope of an application. You
>> > would have to limit the queue depth on the operating system side to
>> > achieve that. Or artificially limit fio in some way, which would not
>> > make a lot of sense imho.
>
>> Thanks Jens. I do in fact have an application that reads in exactly the
>> manner I described. I have monitored the queue depth - it does not rise
>> above 1.  It is a real time musical sample streamer.
>>
>> Please consider this a new feature request for fio - thankyou.
>
> Is this application multithreaded? If so, are mutiple threads doing I/O
> at the same time? If not I�d suggest just testing with one job.

I don't know whether it is multithreaded or not. All I know is that it
reads many files sequentially and in a round-robin fashion, without
causing any disk queuing.

Is it possible to read from more than file in a single job, in a
round-robin fashion? I tried putting more than one file in a single
job, but it only opened one file. If you mean to just do random reads
in a single file - I've tried that, and the throughput is
unrealistically low. I suspect it's because the read-ahead buffer
cannot be effective for random accesses.  Of course, reading
sequentially from a single file will result in a throughput that is
far too high to simulate the application.

Reading a single file using the "skip" option would be a reasonable
compromise, if there were a way to get it to loop back to the
beginning of the file, and start at the next block into the file, such
that the entire file is read. This would simulate reading sequentially
from many files. (if it can be configured to do this, please advise)

Greg.

Re: How to do strict synchronous i/o on Windows?

[Martin Steigerwald]

Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> On 15 August 2012 03:36, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> > Hi Greg,
[…]
> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> >
> >> On Aug 14, 2012 11:06 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
> >> >
> >> > On 08/14/2012 08:24 AM, Greg Sullivan wrote:
> >> > > I need to simulate strict synchronous, round robin i/o to a group of
> >> > > files. I am on Windows 7 32-bit.
> >> > > fio is very nearly working, except that even with a queue depth of 1,
> >> > > it is still resulting in a disk queue that is > 1, because the
> >> > > "iodepth" parameter is not global - it is per thread. (correct?)
> >> > >
> >> > > I've tried using the "sync" engine, however that doesn't work at all -
> >> > > just spews out errors.
> >> >
> >> > That'll be the case for ANY platform and IO engine. If you have more
> >> > than 1 thread or process going, you can have > 1 depth at the device
> >> > side. The definition of a sync IO call is that the call doesn't return
> >> > until the IO is done. If you have overlapped calls due to more than 1
> >> > thread, then that is no longer true.
> >> >
> >> > What you are looking for is outside the scope of an application. You
> >> > would have to limit the queue depth on the operating system side to
> >> > achieve that. Or artificially limit fio in some way, which would not
> >> > make a lot of sense imho.
> >
> >> Thanks Jens. I do in fact have an application that reads in exactly the
> >> manner I described. I have monitored the queue depth - it does not rise
> >> above 1.  It is a real time musical sample streamer.
> >>
> >> Please consider this a new feature request for fio - thankyou.
> >
> > Is this application multithreaded? If so, are mutiple threads doing I/O
> > at the same time? If not I´d suggest just testing with one job.
> 
> I don't know whether it is multithreaded or not. All I know is that it
> reads many files sequentially and in a round-robin fashion, without
> causing any disk queuing.
> 
> Is it possible to read from more than file in a single job, in a
> round-robin fashion? I tried putting more than one file in a single
> job, but it only opened one file. If you mean to just do random reads
> in a single file - I've tried that, and the throughput is
> unrealistically low. I suspect it's because the read-ahead buffer
> cannot be effective for random accesses.  Of course, reading
> sequentially from a single file will result in a throughput that is
> far too high to simulate the application.

Have you tried

       nrfiles=int
              Number of files to use for this job.  Default: 1.

       openfiles=int
              Number of files to keep open at the same time.  Default:
              nrfiles.

       file_service_type=str
              Defines  how files to service are selected.  The follow‐
              ing types are defined:

                     random Choose a file at random

                     roundrobin
                            Round robin  over  open  files  (default).
                            sequential Do each file in the set sequen‐
                            tially.

              The number of I/Os to issue before switching a new  file
              can  be  specified  by  appending  `:int' to the service
              type.

? (see fio manpage).

It seems to me that all you need is nrfiles. I´d bet that fio distributes
the I/O size given among those files, but AFAIR there is something about
that in fio documentation as well.

Use the doc! ;)

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: How to do strict synchronous i/o on Windows?

[Greg Sullivan]

On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
>> On 15 August 2012 03:36, Martin Steigerwald <Martin@lichtvoll.de> wrote:
>> > Hi Greg,
> […]
>> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
>> >
>> >> On Aug 14, 2012 11:06 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
>> >> >
>> >> > On 08/14/2012 08:24 AM, Greg Sullivan wrote:
>> >> > > I need to simulate strict synchronous, round robin i/o to a group
>> >> > > of
>> >> > > files. I am on Windows 7 32-bit.
>> >> > > fio is very nearly working, except that even with a queue depth of
>> >> > > 1,
>> >> > > it is still resulting in a disk queue that is > 1, because the
>> >> > > "iodepth" parameter is not global - it is per thread. (correct?)
>> >> > >
>> >> > > I've tried using the "sync" engine, however that doesn't work at
>> >> > > all -
>> >> > > just spews out errors.
>> >> >
>> >> > That'll be the case for ANY platform and IO engine. If you have more
>> >> > than 1 thread or process going, you can have > 1 depth at the device
>> >> > side. The definition of a sync IO call is that the call doesn't
>> >> > return
>> >> > until the IO is done. If you have overlapped calls due to more than
>> >> > 1
>> >> > thread, then that is no longer true.
>> >> >
>> >> > What you are looking for is outside the scope of an application. You
>> >> > would have to limit the queue depth on the operating system side to
>> >> > achieve that. Or artificially limit fio in some way, which would not
>> >> > make a lot of sense imho.
>> >
>> >> Thanks Jens. I do in fact have an application that reads in exactly
>> >> the
>> >> manner I described. I have monitored the queue depth - it does not
>> >> rise
>> >> above 1.  It is a real time musical sample streamer.
>> >>
>> >> Please consider this a new feature request for fio - thankyou.
>> >
>> > Is this application multithreaded? If so, are mutiple threads doing I/O
>> > at the same time? If not I´d suggest just testing with one job.
>>
>> I don't know whether it is multithreaded or not. All I know is that it
>> reads many files sequentially and in a round-robin fashion, without
>> causing any disk queuing.
>>
>> Is it possible to read from more than file in a single job, in a
>> round-robin fashion? I tried putting more than one file in a single
>> job, but it only opened one file. If you mean to just do random reads
>> in a single file - I've tried that, and the throughput is
>> unrealistically low. I suspect it's because the read-ahead buffer
>> cannot be effective for random accesses.  Of course, reading
>> sequentially from a single file will result in a throughput that is
>> far too high to simulate the application.
>
> Have you tried
>
>        nrfiles=int
>               Number of files to use for this job.  Default: 1.
>
>        openfiles=int
>               Number of files to keep open at the same time.  Default:
>               nrfiles.
>
>        file_service_type=str
>               Defines  how files to service are selected.  The follow‐
>               ing types are defined:
>
>                      random Choose a file at random
>
>                      roundrobin
>                             Round robin  over  open  files  (default).
>                             sequential Do each file in the set sequen‐
>                             tially.
>
>               The number of I/Os to issue before switching a new  file
>               can  be  specified  by  appending  `:int' to the service
>               type.
>
> ? (see fio manpage).
>
> It seems to me that all you need is nrfiles. I´d bet that fio distributes
> the I/O size given among those files, but AFAIR there is something about
> that in fio documentation as well.
>
> Use the doc! ;)
>
> Ciao,
> --
> Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
> GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
>

Yes, I have tried all that, and it works, except that it causes disk
queuing, as I stated in my first post. I thought you meant to put all
the files into a single [job name] section of the ini file, to enforce
single threaded io.

Thanks,
Greg.

Re: How to do strict synchronous i/o on Windows?

[Martin Steigerwald]

Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> >> On 15 August 2012 03:36, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> >> > Hi Greg,
> > […]
> >> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> >> >
> >> >> On Aug 14, 2012 11:06 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
> >> >> >
> >> >> > On 08/14/2012 08:24 AM, Greg Sullivan wrote:
> >> >> > > I need to simulate strict synchronous, round robin i/o to a group
> >> >> > > of
> >> >> > > files. I am on Windows 7 32-bit.
> >> >> > > fio is very nearly working, except that even with a queue depth of
> >> >> > > 1,
> >> >> > > it is still resulting in a disk queue that is > 1, because the
> >> >> > > "iodepth" parameter is not global - it is per thread. (correct?)
> >> >> > >
> >> >> > > I've tried using the "sync" engine, however that doesn't work at
> >> >> > > all -
> >> >> > > just spews out errors.
> >> >> >
> >> >> > That'll be the case for ANY platform and IO engine. If you have more
> >> >> > than 1 thread or process going, you can have > 1 depth at the device
> >> >> > side. The definition of a sync IO call is that the call doesn't
> >> >> > return
> >> >> > until the IO is done. If you have overlapped calls due to more than
> >> >> > 1
> >> >> > thread, then that is no longer true.
> >> >> >
> >> >> > What you are looking for is outside the scope of an application. You
> >> >> > would have to limit the queue depth on the operating system side to
> >> >> > achieve that. Or artificially limit fio in some way, which would not
> >> >> > make a lot of sense imho.
> >> >
> >> >> Thanks Jens. I do in fact have an application that reads in exactly
> >> >> the
> >> >> manner I described. I have monitored the queue depth - it does not
> >> >> rise
> >> >> above 1.  It is a real time musical sample streamer.
> >> >>
> >> >> Please consider this a new feature request for fio - thankyou.
> >> >
> >> > Is this application multithreaded? If so, are mutiple threads doing I/O
> >> > at the same time? If not I´d suggest just testing with one job.
> >>
> >> I don't know whether it is multithreaded or not. All I know is that it
> >> reads many files sequentially and in a round-robin fashion, without
> >> causing any disk queuing.
> >>
> >> Is it possible to read from more than file in a single job, in a
> >> round-robin fashion? I tried putting more than one file in a single
> >> job, but it only opened one file. If you mean to just do random reads
> >> in a single file - I've tried that, and the throughput is
> >> unrealistically low. I suspect it's because the read-ahead buffer
> >> cannot be effective for random accesses.  Of course, reading
> >> sequentially from a single file will result in a throughput that is
> >> far too high to simulate the application.
> >
> > Have you tried
> >
> >        nrfiles=int
> >               Number of files to use for this job.  Default: 1.
> >
> >        openfiles=int
> >               Number of files to keep open at the same time.  Default:
> >               nrfiles.
> >
> >        file_service_type=str
[…]
> > ? (see fio manpage).
> >
> > It seems to me that all you need is nrfiles. I´d bet that fio distributes
> > the I/O size given among those files, but AFAIR there is something about
> > that in fio documentation as well.
> >
> > Use the doc! ;)
[…]
> Yes, I have tried all that, and it works, except that it causes disk
> queuing, as I stated in my first post. I thought you meant to put all
> the files into a single [job name] section of the ini file, to enforce
> single threaded io.

With just one job running at once?

Can you post an example job file?

Did you try the sync=1 / direct=1 suggestion from Bruce Chan?

I only know the behaviour of fio on Linux where I/O depth of greater
than one is only possible with libaio and direct=1. The manpage hints
at I/O depth is one for all synchronous I/O engines, so I´d bet that
refers to Windows as well.

Other than that I have no idea.

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: How to do strict synchronous i/o on Windows?

[Greg Sullivan]

On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
>> On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
>> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
>> >> On 15 August 2012 03:36, Martin Steigerwald <Martin@lichtvoll.de>
>> >> wrote:
>> >> > Hi Greg,
>> > [�]
>> >> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
>> >> >
>> >> >> On Aug 14, 2012 11:06 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
>> >> >> >
>> >> >> > On 08/14/2012 08:24 AM, Greg Sullivan wrote:
>> >> >> > > I need to simulate strict synchronous, round robin i/o to a
>> >> >> > > group
>> >> >> > > of
>> >> >> > > files. I am on Windows 7 32-bit.
>> >> >> > > fio is very nearly working, except that even with a queue depth
>> >> >> > > of
>> >> >> > > 1,
>> >> >> > > it is still resulting in a disk queue that is > 1, because the
>> >> >> > > "iodepth" parameter is not global - it is per thread.
>> >> >> > > (correct?)
>> >> >> > >
>> >> >> > > I've tried using the "sync" engine, however that doesn't work
>> >> >> > > at
>> >> >> > > all -
>> >> >> > > just spews out errors.
>> >> >> >
>> >> >> > That'll be the case for ANY platform and IO engine. If you have
>> >> >> > more
>> >> >> > than 1 thread or process going, you can have > 1 depth at the
>> >> >> > device
>> >> >> > side. The definition of a sync IO call is that the call doesn't
>> >> >> > return
>> >> >> > until the IO is done. If you have overlapped calls due to more
>> >> >> > than
>> >> >> > 1
>> >> >> > thread, then that is no longer true.
>> >> >> >
>> >> >> > What you are looking for is outside the scope of an application.
>> >> >> > You
>> >> >> > would have to limit the queue depth on the operating system side
>> >> >> > to
>> >> >> > achieve that. Or artificially limit fio in some way, which would
>> >> >> > not
>> >> >> > make a lot of sense imho.
>> >> >
>> >> >> Thanks Jens. I do in fact have an application that reads in exactly
>> >> >> the
>> >> >> manner I described. I have monitored the queue depth - it does not
>> >> >> rise
>> >> >> above 1.  It is a real time musical sample streamer.
>> >> >>
>> >> >> Please consider this a new feature request for fio - thankyou.
>> >> >
>> >> > Is this application multithreaded? If so, are mutiple threads doing
>> >> > I/O
>> >> > at the same time? If not I�d suggest just testing with one job.
>> >>
>> >> I don't know whether it is multithreaded or not. All I know is that it
>> >> reads many files sequentially and in a round-robin fashion, without
>> >> causing any disk queuing.
>> >>
>> >> Is it possible to read from more than file in a single job, in a
>> >> round-robin fashion? I tried putting more than one file in a single
>> >> job, but it only opened one file. If you mean to just do random reads
>> >> in a single file - I've tried that, and the throughput is
>> >> unrealistically low. I suspect it's because the read-ahead buffer
>> >> cannot be effective for random accesses.  Of course, reading
>> >> sequentially from a single file will result in a throughput that is
>> >> far too high to simulate the application.
>> >
>> > Have you tried
>> >
>> >        nrfiles=int
>> >               Number of files to use for this job.  Default: 1.
>> >
>> >        openfiles=int
>> >               Number of files to keep open at the same time.  Default:
>> >               nrfiles.
>> >
>> >        file_service_type=str
> [�]
>> > ? (see fio manpage).
>> >
>> > It seems to me that all you need is nrfiles. I�d bet that fio
>> > distributes
>> > the I/O size given among those files, but AFAIR there is something
>> > about
>> > that in fio documentation as well.
>> >
>> > Use the doc! ;)
> [�]
>> Yes, I have tried all that, and it works, except that it causes disk
>> queuing, as I stated in my first post. I thought you meant to put all
>> the files into a single [job name] section of the ini file, to enforce
>> single threaded io.
>
> With just one job running at once?
>
> Can you post an example job file?
>
> Did you try the sync=1 / direct=1 suggestion from Bruce Chan?
>
> I only know the behaviour of fio on Linux where I/O depth of greater
> than one is only possible with libaio and direct=1. The manpage hints
> at I/O depth is one for all synchronous I/O engines, so I�d bet that
> refers to Windows as well.
>
> Other than that I have no idea.
>
> --

One INI file, but a seperate [job name] section for each file, yes.
According to Jens, because each [job name] is a seperate thread, and
iodepth acts at the thread level, there will still be queuing at the
device level. If there were a way to do what I want I think Jens would
have told me, unfortunately.   ;)

direct io does at least allow me to do cache-less reads though - thankyou.

Greg.

Re: How to do strict synchronous i/o on Windows?

[Martin Steigerwald]

Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> >> On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> >> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> >> >> On 15 August 2012 03:36, Martin Steigerwald <Martin@lichtvoll.de>
> >> >> wrote:
> >> >> > Hi Greg,
> >> > [�]
> >> >> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> >> >> >
> >> >> >> On Aug 14, 2012 11:06 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
> >> >> >> >
> >> >> >> > On 08/14/2012 08:24 AM, Greg Sullivan wrote:
[�]
> >> >> Is it possible to read from more than file in a single job, in a
> >> >> round-robin fashion? I tried putting more than one file in a single
> >> >> job, but it only opened one file. If you mean to just do random reads
> >> >> in a single file - I've tried that, and the throughput is
> >> >> unrealistically low. I suspect it's because the read-ahead buffer
> >> >> cannot be effective for random accesses.  Of course, reading
> >> >> sequentially from a single file will result in a throughput that is
> >> >> far too high to simulate the application.
> >> >
> >> > Have you tried
> >> >
> >> >        nrfiles=int
> >> >               Number of files to use for this job.  Default: 1.
> >> >
> >> >        openfiles=int
> >> >               Number of files to keep open at the same time.  Default:
> >> >               nrfiles.
> >> >
> >> >        file_service_type=str
> > [�]
> >> > ? (see fio manpage).
> >> >
> >> > It seems to me that all you need is nrfiles. I�d bet that fio
> >> > distributes
> >> > the I/O size given among those files, but AFAIR there is something
> >> > about
> >> > that in fio documentation as well.
> >> >
> >> > Use the doc! ;)
> > [�]
> >> Yes, I have tried all that, and it works, except that it causes disk
> >> queuing, as I stated in my first post. I thought you meant to put all
> >> the files into a single [job name] section of the ini file, to enforce
> >> single threaded io.
> >
> > With just one job running at once?
> >
> > Can you post an example job file?
> >
> > Did you try the sync=1 / direct=1 suggestion from Bruce Chan?
> >
> > I only know the behaviour of fio on Linux where I/O depth of greater
> > than one is only possible with libaio and direct=1. The manpage hints
> > at I/O depth is one for all synchronous I/O engines, so I�d bet that
> > refers to Windows as well.
> >
> > Other than that I have no idea.
[�]
> One INI file, but a seperate [job name] section for each file, yes.
> According to Jens, because each [job name] is a seperate thread, and
> iodepth acts at the thread level, there will still be queuing at the
> device level. If there were a way to do what I want I think Jens would
> have told me, unfortunately.   ;)
> 
> direct io does at least allow me to do cache-less reads though - thankyou.

My suggestion is to use one job with several files.

martin@merkaba:/tmp> cat severalfiles.job 
[global]
size=1G
nrfiles=100

[read]
rw=read

[write]
stonewall
rw=write

(now these are two jobs, but stonewall lets the write job run after the
read one with cache invalidation if not disabled and if supported by OS)

martin@merkaba:/tmp> fio severalfiles.job
read: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
write: (g=1): rw=write, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
2.0.8
Starting 2 processes
read: Laying out IO file(s) (100 file(s) / 1023MB)
write: Laying out IO file(s) (100 file(s) / 1023MB)
Jobs: 1 (f=100)
read: (groupid=0, jobs=1): err= 0: pid=23377
[� lots of fast due to /tmp being a RAM-based filesystem � tmpfs �]


martin@merkaba:/tmp> ls -lh read.1.* | head
-rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.0
-rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.1
-rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.10
-rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.11
-rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.12
-rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.13
-rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.14
-rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.15
-rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.16
-rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.17
[� only first ten displayed �]

martin@merkaba:/tmp> find -name "read.1*" 2>/dev/null | wc -l
100

100 files a 11M, due to rounding issues that may nicely add up to
the one GiB.

Raw sizes are:

martin@merkaba:/tmp> ls -l read.1.* | head                  
-rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.0
-rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.1
-rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.10
-rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.11
-rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.12
-rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.13
-rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.14
-rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.15
-rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.16
-rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.17

Note: When I used filename, fio just created one files regardless of the
nrfiles setting. I would have expected it to use the filename as a prefix.
There might be some way to have it do that.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: How to do strict synchronous i/o on Windows?

[Greg Sullivan]

On 15 August 2012 07:24, Martin Steigerwald <Martin@lichtvoll.de> wrote:
>
> Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> > On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> > > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> > >> On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> > >> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> > >> >> On 15 August 2012 03:36, Martin Steigerwald <Martin@lichtvoll.de>
> > >> >> wrote:
> > >> >> > Hi Greg,
> > >> > [�]
> > >> >> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> > >> >> >
> > >> >> >> On Aug 14, 2012 11:06 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
> > >> >> >> >
> > >> >> >> > On 08/14/2012 08:24 AM, Greg Sullivan wrote:
> [�]
> > >> >> Is it possible to read from more than file in a single job, in a
> > >> >> round-robin fashion? I tried putting more than one file in a single
> > >> >> job, but it only opened one file. If you mean to just do random reads
> > >> >> in a single file - I've tried that, and the throughput is
> > >> >> unrealistically low. I suspect it's because the read-ahead buffer
> > >> >> cannot be effective for random accesses.  Of course, reading
> > >> >> sequentially from a single file will result in a throughput that is
> > >> >> far too high to simulate the application.
> > >> >
> > >> > Have you tried
> > >> >
> > >> >        nrfiles=int
> > >> >               Number of files to use for this job.  Default: 1.
> > >> >
> > >> >        openfiles=int
> > >> >               Number of files to keep open at the same time.  Default:
> > >> >               nrfiles.
> > >> >
> > >> >        file_service_type=str
> > > [�]
> > >> > ? (see fio manpage).
> > >> >
> > >> > It seems to me that all you need is nrfiles. I�d bet that fio
> > >> > distributes
> > >> > the I/O size given among those files, but AFAIR there is something
> > >> > about
> > >> > that in fio documentation as well.
> > >> >
> > >> > Use the doc! ;)
> > > [�]
> > >> Yes, I have tried all that, and it works, except that it causes disk
> > >> queuing, as I stated in my first post. I thought you meant to put all
> > >> the files into a single [job name] section of the ini file, to enforce
> > >> single threaded io.
> > >
> > > With just one job running at once?
> > >
> > > Can you post an example job file?
> > >
> > > Did you try the sync=1 / direct=1 suggestion from Bruce Chan?
> > >
> > > I only know the behaviour of fio on Linux where I/O depth of greater
> > > than one is only possible with libaio and direct=1. The manpage hints
> > > at I/O depth is one for all synchronous I/O engines, so I�d bet that
> > > refers to Windows as well.
> > >
> > > Other than that I have no idea.
> [�]
> > One INI file, but a seperate [job name] section for each file, yes.
> > According to Jens, because each [job name] is a seperate thread, and
> > iodepth acts at the thread level, there will still be queuing at the
> > device level. If there were a way to do what I want I think Jens would
> > have told me, unfortunately.   ;)
> >
> > direct io does at least allow me to do cache-less reads though - thankyou.
>
> My suggestion is to use one job with several files.
>
> martin@merkaba:/tmp> cat severalfiles.job
> [global]
> size=1G
> nrfiles=100
>
> [read]
> rw=read
>
> [write]
> stonewall
> rw=write
>
> (now these are two jobs, but stonewall lets the write job run after the
> read one with cache invalidation if not disabled and if supported by OS)
>
> martin@merkaba:/tmp> fio severalfiles.job
> read: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
> write: (g=1): rw=write, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
> 2.0.8
> Starting 2 processes
> read: Laying out IO file(s) (100 file(s) / 1023MB)
> write: Laying out IO file(s) (100 file(s) / 1023MB)
> Jobs: 1 (f=100)
> read: (groupid=0, jobs=1): err= 0: pid=23377
> [� lots of fast due to /tmp being a RAM-based filesystem � tmpfs �]
>
>
> martin@merkaba:/tmp> ls -lh read.1.* | head
> -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.0
> -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.1
> -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.10
> -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.11
> -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.12
> -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.13
> -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.14
> -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.15
> -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.16
> -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.17
> [� only first ten displayed �]
>
> martin@merkaba:/tmp> find -name "read.1*" 2>/dev/null | wc -l
> 100
>
> 100 files a 11M, due to rounding issues that may nicely add up to
> the one GiB.
>
> Raw sizes are:
>
> martin@merkaba:/tmp> ls -l read.1.* | head
> -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.0
> -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.1
> -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.10
> -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.11
> -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.12
> -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.13
> -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.14
> -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.15
> -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.16
> -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.17
>
> Note: When I used filename, fio just created one files regardless of the
> nrfiles setting. I would have expected it to use the filename as a prefix.
> There might be some way to have it do that.
>
> Ciao,

Thanks - that runs, but it's still queuing. As I said before, I can't
use the sync engine - I receive an error. Is there a synchronous
engine available for Windows? Perhaps that's the only problem.
Can you check to see whether your system is queuing at the file
system/device level when you run that test?

I had attempted to put the files in a single job earlier - I think it
may have been successfully accessing both files, but I didn't notice
it in the output. I'm a raw beginner.

Greg.

Re: How to do strict synchronous i/o on Windows?

[Martin Steigerwald]

Am Mittwoch, 15. August 2012 schrieb Greg Sullivan:
> On 15 August 2012 07:24, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> > > On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> > > > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> > > >> On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> > > >> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> > > >> >> On 15 August 2012 03:36, Martin Steigerwald
> > > >> >> <Martin@lichtvoll.de>
> > > >> >> 
> > > >> >> wrote:
> > > >> >> > Hi Greg,
> > > >> > 
> > > >> > [�]
> > > >> > 
> > > >> >> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> > > >> >> >> On Aug 14, 2012 11:06 PM, "Jens Axboe" <axboe@kernel.dk> 
wrote:
> > > >> >> >> > On 08/14/2012 08:24 AM, Greg Sullivan wrote:
> > [�]
> > 
> > > >> >> Is it possible to read from more than file in a single job,
> > > >> >> in a round-robin fashion? I tried putting more than one file
> > > >> >> in a single job, but it only opened one file. If you mean to
> > > >> >> just do random reads in a single file - I've tried that, and
> > > >> >> the throughput is unrealistically low. I suspect it's
> > > >> >> because the read-ahead buffer cannot be effective for random
> > > >> >> accesses.  Of course, reading sequentially from a single
> > > >> >> file will result in a throughput that is far too high to
> > > >> >> simulate the application.
> > > >> > 
> > > >> > Have you tried
> > > >> > 
> > > >> >        nrfiles=int
> > > >> >        
> > > >> >               Number of files to use for this job.  Default:
> > > >> >               1.
> > > >> >        
> > > >> >        openfiles=int
> > > >> >        
> > > >> >               Number of files to keep open at the same time. 
> > > >> >               Default: nrfiles.
> > > >> >        
> > > >> >        file_service_type=str
> > > > 
> > > > [�]
> > > > 
> > > >> > ? (see fio manpage).
> > > >> > 
> > > >> > It seems to me that all you need is nrfiles. I�d bet that fio
> > > >> > distributes
> > > >> > the I/O size given among those files, but AFAIR there is
> > > >> > something about
> > > >> > that in fio documentation as well.
> > > >> > 
> > > >> > Use the doc! ;)
> > > > 
> > > > [�]
> > > > 
> > > >> Yes, I have tried all that, and it works, except that it causes
> > > >> disk queuing, as I stated in my first post. I thought you meant
> > > >> to put all the files into a single [job name] section of the
> > > >> ini file, to enforce single threaded io.
> > > > 
> > > > With just one job running at once?
> > > > 
> > > > Can you post an example job file?
> > > > 
> > > > Did you try the sync=1 / direct=1 suggestion from Bruce Chan?
> > > > 
> > > > I only know the behaviour of fio on Linux where I/O depth of
> > > > greater than one is only possible with libaio and direct=1. The
> > > > manpage hints at I/O depth is one for all synchronous I/O
> > > > engines, so I�d bet that refers to Windows as well.
> > > > 
> > > > Other than that I have no idea.
> > 
> > [�]
> > 
> > > One INI file, but a seperate [job name] section for each file, yes.
> > > According to Jens, because each [job name] is a seperate thread,
> > > and iodepth acts at the thread level, there will still be queuing
> > > at the device level. If there were a way to do what I want I think
> > > Jens would have told me, unfortunately.   ;)
> > > 
> > > direct io does at least allow me to do cache-less reads though -
> > > thankyou.
> > 
> > My suggestion is to use one job with several files.
> > 
> > martin@merkaba:/tmp> cat severalfiles.job
> > [global]
> > size=1G
> > nrfiles=100
> > 
> > [read]
> > rw=read
> > 
> > [write]
> > stonewall
> > rw=write
> > 
> > (now these are two jobs, but stonewall lets the write job run after
> > the read one with cache invalidation if not disabled and if
> > supported by OS)
> > 
> > martin@merkaba:/tmp> fio severalfiles.job
> > read: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
> > write: (g=1): rw=write, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
> > 2.0.8
> > Starting 2 processes
> > read: Laying out IO file(s) (100 file(s) / 1023MB)
> > write: Laying out IO file(s) (100 file(s) / 1023MB)
> > Jobs: 1 (f=100)
> > read: (groupid=0, jobs=1): err= 0: pid=23377
> > [� lots of fast due to /tmp being a RAM-based filesystem � tmpfs �]
> > 
> > 
> > martin@merkaba:/tmp> ls -lh read.1.* | head
> > -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.0
> > -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.1
[�]
> > [� only first ten displayed �]
> > 
> > martin@merkaba:/tmp> find -name "read.1*" 2>/dev/null | wc -l
> > 100
> > 
> > 100 files a 11M, due to rounding issues that may nicely add up to
> > the one GiB.
> > 
> > Raw sizes are:
> > 
> > martin@merkaba:/tmp> ls -l read.1.* | head
> > -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.0
> > -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.1
[�]
> > Note: When I used filename, fio just created one files regardless of
> > the nrfiles setting. I would have expected it to use the filename as
> > a prefix. There might be some way to have it do that.
> > 
> > Ciao,
> 
> Thanks - that runs, but it's still queuing. As I said before, I can't
> use the sync engine - I receive an error. Is there a synchronous
> engine available for Windows? Perhaps that's the only problem.
> Can you check to see whether your system is queuing at the file
> system/device level when you run that test?
> 
> I had attempted to put the files in a single job earlier - I think it
> may have been successfully accessing both files, but I didn't notice
> it in the output. I'm a raw beginner.

Did you try with

ioengine=windowsaio

+

iodepth=1 (should be default however I think)


Otherwise I have no idea. I never used fio on Windows so far.

It might help when you try to explain exactly which problem you want to 
solve by the fio measurements. Multimedia streaming. Is it to slow? What is 
it why you want to do these measurements?

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: How to do strict synchronous i/o on Windows?

[Greg Sullivan]

On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> Am Mittwoch, 15. August 2012 schrieb Greg Sullivan:
>> On 15 August 2012 07:24, Martin Steigerwald <Martin@lichtvoll.de> wrote:
>> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
>> > > On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
>> > > > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
>> > > >> On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
>> > > >> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
>> > > >> >> On 15 August 2012 03:36, Martin Steigerwald
>> > > >> >> <Martin@lichtvoll.de>
>> > > >> >>
>> > > >> >> wrote:
>> > > >> >> > Hi Greg,
>> > > >> >
>> > > >> > [�]
>> > > >> >
>> > > >> >> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
>> > > >> >> >> On Aug 14, 2012 11:06 PM, "Jens Axboe" <axboe@kernel.dk>
> wrote:
>> > > >> >> >> > On 08/14/2012 08:24 AM, Greg Sullivan wrote:
>> > [�]
>> >
>> > > >> >> Is it possible to read from more than file in a single job,
>> > > >> >> in a round-robin fashion? I tried putting more than one file
>> > > >> >> in a single job, but it only opened one file. If you mean to
>> > > >> >> just do random reads in a single file - I've tried that, and
>> > > >> >> the throughput is unrealistically low. I suspect it's
>> > > >> >> because the read-ahead buffer cannot be effective for random
>> > > >> >> accesses.  Of course, reading sequentially from a single
>> > > >> >> file will result in a throughput that is far too high to
>> > > >> >> simulate the application.
>> > > >> >
>> > > >> > Have you tried
>> > > >> >
>> > > >> >        nrfiles=int
>> > > >> >
>> > > >> >               Number of files to use for this job.  Default:
>> > > >> >               1.
>> > > >> >
>> > > >> >        openfiles=int
>> > > >> >
>> > > >> >               Number of files to keep open at the same time.
>> > > >> >               Default: nrfiles.
>> > > >> >
>> > > >> >        file_service_type=str
>> > > >
>> > > > [�]
>> > > >
>> > > >> > ? (see fio manpage).
>> > > >> >
>> > > >> > It seems to me that all you need is nrfiles. I�d bet that fio
>> > > >> > distributes
>> > > >> > the I/O size given among those files, but AFAIR there is
>> > > >> > something about
>> > > >> > that in fio documentation as well.
>> > > >> >
>> > > >> > Use the doc! ;)
>> > > >
>> > > > [�]
>> > > >
>> > > >> Yes, I have tried all that, and it works, except that it causes
>> > > >> disk queuing, as I stated in my first post. I thought you meant
>> > > >> to put all the files into a single [job name] section of the
>> > > >> ini file, to enforce single threaded io.
>> > > >
>> > > > With just one job running at once?
>> > > >
>> > > > Can you post an example job file?
>> > > >
>> > > > Did you try the sync=1 / direct=1 suggestion from Bruce Chan?
>> > > >
>> > > > I only know the behaviour of fio on Linux where I/O depth of
>> > > > greater than one is only possible with libaio and direct=1. The
>> > > > manpage hints at I/O depth is one for all synchronous I/O
>> > > > engines, so I�d bet that refers to Windows as well.
>> > > >
>> > > > Other than that I have no idea.
>> >
>> > [�]
>> >
>> > > One INI file, but a seperate [job name] section for each file, yes.
>> > > According to Jens, because each [job name] is a seperate thread,
>> > > and iodepth acts at the thread level, there will still be queuing
>> > > at the device level. If there were a way to do what I want I think
>> > > Jens would have told me, unfortunately.   ;)
>> > >
>> > > direct io does at least allow me to do cache-less reads though -
>> > > thankyou.
>> >
>> > My suggestion is to use one job with several files.
>> >
>> > martin@merkaba:/tmp> cat severalfiles.job
>> > [global]
>> > size=1G
>> > nrfiles=100
>> >
>> > [read]
>> > rw=read
>> >
>> > [write]
>> > stonewall
>> > rw=write
>> >
>> > (now these are two jobs, but stonewall lets the write job run after
>> > the read one with cache invalidation if not disabled and if
>> > supported by OS)
>> >
>> > martin@merkaba:/tmp> fio severalfiles.job
>> > read: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
>> > write: (g=1): rw=write, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
>> > 2.0.8
>> > Starting 2 processes
>> > read: Laying out IO file(s) (100 file(s) / 1023MB)
>> > write: Laying out IO file(s) (100 file(s) / 1023MB)
>> > Jobs: 1 (f=100)
>> > read: (groupid=0, jobs=1): err= 0: pid=23377
>> > [� lots of fast due to /tmp being a RAM-based filesystem � tmpfs �]
>> >
>> >
>> > martin@merkaba:/tmp> ls -lh read.1.* | head
>> > -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.0
>> > -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.1
> [�]
>> > [� only first ten displayed �]
>> >
>> > martin@merkaba:/tmp> find -name "read.1*" 2>/dev/null | wc -l
>> > 100
>> >
>> > 100 files a 11M, due to rounding issues that may nicely add up to
>> > the one GiB.
>> >
>> > Raw sizes are:
>> >
>> > martin@merkaba:/tmp> ls -l read.1.* | head
>> > -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.0
>> > -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.1
> [�]
>> > Note: When I used filename, fio just created one files regardless of
>> > the nrfiles setting. I would have expected it to use the filename as
>> > a prefix. There might be some way to have it do that.
>> >
>> > Ciao,
>>
>> Thanks - that runs, but it's still queuing. As I said before, I can't
>> use the sync engine - I receive an error. Is there a synchronous
>> engine available for Windows? Perhaps that's the only problem.
>> Can you check to see whether your system is queuing at the file
>> system/device level when you run that test?
>>
>> I had attempted to put the files in a single job earlier - I think it
>> may have been successfully accessing both files, but I didn't notice
>> it in the output. I'm a raw beginner.
>
> Did you try with
>
> ioengine=windowsaio
>
> +
>
> iodepth=1 (should be default however I think)
>
>
> Otherwise I have no idea. I never used fio on Windows so far.
>
> It might help when you try to explain exactly which problem you want to
> solve by the fio measurements. Multimedia streaming. Is it to slow? What is
>
> it why you want to do these measurements?
>

They are both defaults, and the output shows that both are being used.
If you could tell me whether your system is generating queuing it
would help, because if yours queues even when using the sync io
engine, it means I'm wasting my time and fio simply needs to be
augmented to support strict single threaded operation over multiple
files.

I am wanting to determine whether the application in question is
extracting a reasonable number of real time streams from any given
storage system.

Thanks,
Greg.

Re: How to do strict synchronous i/o on Windows?

[Bruce Cran]

On 15/08/2012 03:46, Greg Sullivan wrote:
> Thanks - that runs, but it's still queuing. As I said before, I can't
> use the sync engine - I receive an error. Is there a synchronous
> engine available for Windows? Perhaps that's the only problem.
> Can you check to see whether your system is queuing at the file
> system/device level when you run that test?

The sync ioengine doesn't work on Windows. You should be able to get the 
same behaviour by using sync=1 and direct=1 with windowsaio.

-- 
Bruce Cran

Re: How to do strict synchronous i/o on Windows?

[Greg Sullivan]

On 15/08/2012, Bruce Cran <bruce@cran.org.uk> wrote:
> On 15/08/2012 03:46, Greg Sullivan wrote:
>> Thanks - that runs, but it's still queuing. As I said before, I can't
>> use the sync engine - I receive an error. Is there a synchronous
>> engine available for Windows? Perhaps that's the only problem.
>> Can you check to see whether your system is queuing at the file
>> system/device level when you run that test?
>
> The sync ioengine doesn't work on Windows. You should be able to get the
> same behaviour by using sync=1 and direct=1 with windowsaio.
>
> --
> Bruce Cran
>
Thanks Bruce. The reason I hadn't tried sync=1 is because I thought it
only applied to writes, which I'm not interested in. I'll give it a
try though and report back.

Greg.

Re: How to do strict synchronous i/o on Windows?

[Greg Sullivan]

On 15 August 2012 17:58, Greg Sullivan <greg.sullivan@sullivang.net> wrote:
> On 15/08/2012, Bruce Cran <bruce@cran.org.uk> wrote:
>> On 15/08/2012 03:46, Greg Sullivan wrote:
>>> Thanks - that runs, but it's still queuing. As I said before, I can't
>>> use the sync engine - I receive an error. Is there a synchronous
>>> engine available for Windows? Perhaps that's the only problem.
>>> Can you check to see whether your system is queuing at the file
>>> system/device level when you run that test?
>>
>> The sync ioengine doesn't work on Windows. You should be able to get the
>> same behaviour by using sync=1 and direct=1 with windowsaio.
>>
>> --
>> Bruce Cran
>>
> Thanks Bruce. The reason I hadn't tried sync=1 is because I thought it
> only applied to writes, which I'm not interested in. I'll give it a
> try though and report back.
>
> Greg.

My profuse apologies Bruce and Martin - it IS working. I had not
actually tried to see whether direct=1   prevented queuing - I had
only tried it to prevent caching.  I'm really sorry for wasting your
time!  (sync=1 is not required for my read-only test though, it seems)

The throughput is exactly what I expected it should be, too, judging
by the throughput of the application under test.

This utility is absolutely fantastic. I can now experiment with long
term tests, using the rate command the ability to raise an alarm if
the rate cannot be maintained. I am so glad I found it. It took me a
while to find it actually.

All the best,
Greg.

Re: How to do strict synchronous i/o on Windows?

[Martin Steigerwald]

Am Mittwoch, 15. August 2012 schrieb Greg Sullivan:
> On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> > Am Mittwoch, 15. August 2012 schrieb Greg Sullivan:
> >> On 15 August 2012 07:24, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> >> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> >> > > On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> >> > > > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> >> > > >> On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> >> > > >> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> >> > > >> >> On 15 August 2012 03:36, Martin Steigerwald
> >> > > >> >> <Martin@lichtvoll.de>
> >> > > >> >> 
> >> > > >> >> wrote:
> >> > > >> >> > Hi Greg,
> >> > > >> > 
> >> > > >> > [�]
> >> > > >> > 
> >> > > >> >> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
> >> > > >> >> >> On Aug 14, 2012 11:06 PM, "Jens Axboe"
> >> > > >> >> >> <axboe@kernel.dk>
> > 
> > wrote:
> >> > > >> >> >> > On 08/14/2012 08:24 AM, Greg Sullivan wrote:
> >> > [�]
> >> > 
> >> > > >> >> Is it possible to read from more than file in a single
> >> > > >> >> job, in a round-robin fashion? I tried putting more than
> >> > > >> >> one file in a single job, but it only opened one file. If
> >> > > >> >> you mean to just do random reads in a single file - I've
> >> > > >> >> tried that, and the throughput is unrealistically low. I
> >> > > >> >> suspect it's because the read-ahead buffer cannot be
> >> > > >> >> effective for random accesses.  Of course, reading
> >> > > >> >> sequentially from a single file will result in a
> >> > > >> >> throughput that is far too high to simulate the
> >> > > >> >> application.
> >> > > >> > 
> >> > > >> > Have you tried
> >> > > >> > 
> >> > > >> >        nrfiles=int
> >> > > >> >        
> >> > > >> >               Number of files to use for this job. 
> >> > > >> >               Default: 1.
> >> > > >> >        
> >> > > >> >        openfiles=int
> >> > > >> >        
> >> > > >> >               Number of files to keep open at the same
> >> > > >> >               time. Default: nrfiles.
> >> > > >> >        
> >> > > >> >        file_service_type=str
> >> > > > 
> >> > > > [�]
> >> > > > 
> >> > > >> > ? (see fio manpage).
> >> > > >> > 
> >> > > >> > It seems to me that all you need is nrfiles. I�d bet that
> >> > > >> > fio distributes
> >> > > >> > the I/O size given among those files, but AFAIR there is
> >> > > >> > something about
> >> > > >> > that in fio documentation as well.
> >> > > >> > 
> >> > > >> > Use the doc! ;)
> >> > > > 
> >> > > > [�]
> >> > > > 
> >> > > >> Yes, I have tried all that, and it works, except that it
> >> > > >> causes disk queuing, as I stated in my first post. I thought
> >> > > >> you meant to put all the files into a single [job name]
> >> > > >> section of the ini file, to enforce single threaded io.
> >> > > > 
> >> > > > With just one job running at once?
> >> > > > 
> >> > > > Can you post an example job file?
> >> > > > 
> >> > > > Did you try the sync=1 / direct=1 suggestion from Bruce Chan?
> >> > > > 
> >> > > > I only know the behaviour of fio on Linux where I/O depth of
> >> > > > greater than one is only possible with libaio and direct=1.
> >> > > > The manpage hints at I/O depth is one for all synchronous I/O
> >> > > > engines, so I�d bet that refers to Windows as well.
> >> > > > 
> >> > > > Other than that I have no idea.
> >> > 
> >> > [�]
> >> > 
> >> > > One INI file, but a seperate [job name] section for each file,
> >> > > yes. According to Jens, because each [job name] is a seperate
> >> > > thread, and iodepth acts at the thread level, there will still
> >> > > be queuing at the device level. If there were a way to do what
> >> > > I want I think Jens would have told me, unfortunately.   ;)
> >> > > 
> >> > > direct io does at least allow me to do cache-less reads though -
> >> > > thankyou.
> >> > 
> >> > My suggestion is to use one job with several files.
> >> > 
> >> > martin@merkaba:/tmp> cat severalfiles.job
> >> > [global]
> >> > size=1G
> >> > nrfiles=100
> >> > 
> >> > [read]
> >> > rw=read
> >> > 
> >> > [write]
> >> > stonewall
> >> > rw=write
> >> > 
> >> > (now these are two jobs, but stonewall lets the write job run
> >> > after the read one with cache invalidation if not disabled and if
> >> > supported by OS)
> >> > 
> >> > martin@merkaba:/tmp> fio severalfiles.job
> >> > read: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
> >> > write: (g=1): rw=write, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
> >> > 2.0.8
> >> > Starting 2 processes
> >> > read: Laying out IO file(s) (100 file(s) / 1023MB)
> >> > write: Laying out IO file(s) (100 file(s) / 1023MB)
> >> > Jobs: 1 (f=100)
> >> > read: (groupid=0, jobs=1): err= 0: pid=23377
> >> > [� lots of fast due to /tmp being a RAM-based filesystem � tmpfs
> >> > �]
> >> > 
> >> > 
> >> > martin@merkaba:/tmp> ls -lh read.1.* | head
> >> > -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.0
> >> > -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.1
> > 
> > [�]
> > 
> >> > [� only first ten displayed �]
> >> > 
> >> > martin@merkaba:/tmp> find -name "read.1*" 2>/dev/null | wc -l
> >> > 100
> >> > 
> >> > 100 files a 11M, due to rounding issues that may nicely add up to
> >> > the one GiB.
> >> > 
> >> > Raw sizes are:
> >> > 
> >> > martin@merkaba:/tmp> ls -l read.1.* | head
> >> > -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.0
> >> > -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.1
> > 
> > [�]
> > 
> >> > Note: When I used filename, fio just created one files regardless
> >> > of the nrfiles setting. I would have expected it to use the
> >> > filename as a prefix. There might be some way to have it do that.
> >> > 
> >> > Ciao,
> >> 
> >> Thanks - that runs, but it's still queuing. As I said before, I
> >> can't use the sync engine - I receive an error. Is there a
> >> synchronous engine available for Windows? Perhaps that's the only
> >> problem. Can you check to see whether your system is queuing at the
> >> file system/device level when you run that test?
> >> 
> >> I had attempted to put the files in a single job earlier - I think
> >> it may have been successfully accessing both files, but I didn't
> >> notice it in the output. I'm a raw beginner.
> > 
> > Did you try with
> > 
> > ioengine=windowsaio
> > 
> > +
> > 
> > iodepth=1 (should be default however I think)
> > 
> > 
> > Otherwise I have no idea. I never used fio on Windows so far.
> > 
> > It might help when you try to explain exactly which problem you want
> > to solve by the fio measurements. Multimedia streaming. Is it to
> > slow? What is
> > 
> > it why you want to do these measurements?
> 
> They are both defaults, and the output shows that both are being used.
> If you could tell me whether your system is generating queuing it
> would help, because if yours queues even when using the sync io
> engine, it means I'm wasting my time and fio simply needs to be
> augmented to support strict single threaded operation over multiple
> files.
> 
> I am wanting to determine whether the application in question is
> extracting a reasonable number of real time streams from any given
> storage system.

Just for the record since you got it working on Windows as well � it works for me:

merkaba:/tmp> cat severalfiles.job 
[global]
size=1G
nrfiles=100

[read]
rw=read

merkaba:/tmp> fio severalfiles.job
read: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
2.0.8
Starting 1 process

read: (groupid=0, jobs=1): err= 0: pid=4579
  read : io=1023.9MB, bw=2409.7MB/s, iops=616705 , runt=   425msec
    clat (usec): min=0 , max=54 , avg= 1.08, stdev= 0.64
     lat (usec): min=0 , max=54 , avg= 1.13, stdev= 0.66
    clat percentiles (usec):
     |  1.00th=[    0],  5.00th=[    1], 10.00th=[    1], 20.00th=[    1],
     | 30.00th=[    1], 40.00th=[    1], 50.00th=[    1], 60.00th=[    1],
     | 70.00th=[    1], 80.00th=[    1], 90.00th=[    1], 95.00th=[    2],
     | 99.00th=[    2], 99.50th=[    2], 99.90th=[   14], 99.95th=[   16],
     | 99.99th=[   23]
    lat (usec) : 2=92.14%, 4=7.74%, 10=0.01%, 20=0.10%, 50=0.01%
    lat (usec) : 100=0.01%
  cpu          : usr=22.41%, sys=76.42%, ctx=421, majf=0, minf=36
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=262100/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=1023.9MB, aggrb=2409.7MB/s, minb=2409.7MB/s, maxb=2409.7MB/s, mint=425msec, maxt=425msec

(if you wonder about the figures � thats RAM testing � Linux tmpfs ;)

100% at IO depth 1.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: How to do strict synchronous i/o on Windows?

[Greg Sullivan]

On 15 August 2012 21:31, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> Am Mittwoch, 15. August 2012 schrieb Greg Sullivan:
>> On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
>> > Am Mittwoch, 15. August 2012 schrieb Greg Sullivan:
>> >> On 15 August 2012 07:24, Martin Steigerwald <Martin@lichtvoll.de> wrote:
>> >> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
>> >> > > On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
>> >> > > > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
>> >> > > >> On 15/08/2012, Martin Steigerwald <Martin@lichtvoll.de> wrote:
>> >> > > >> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
>> >> > > >> >> On 15 August 2012 03:36, Martin Steigerwald
>> >> > > >> >> <Martin@lichtvoll.de>
>> >> > > >> >>
>> >> > > >> >> wrote:
>> >> > > >> >> > Hi Greg,
>> >> > > >> >
>> >> > > >> > [�]
>> >> > > >> >
>> >> > > >> >> > Am Dienstag, 14. August 2012 schrieb Greg Sullivan:
>> >> > > >> >> >> On Aug 14, 2012 11:06 PM, "Jens Axboe"
>> >> > > >> >> >> <axboe@kernel.dk>
>> >
>> > wrote:
>> >> > > >> >> >> > On 08/14/2012 08:24 AM, Greg Sullivan wrote:
>> >> > [�]
>> >> >
>> >> > > >> >> Is it possible to read from more than file in a single
>> >> > > >> >> job, in a round-robin fashion? I tried putting more than
>> >> > > >> >> one file in a single job, but it only opened one file. If
>> >> > > >> >> you mean to just do random reads in a single file - I've
>> >> > > >> >> tried that, and the throughput is unrealistically low. I
>> >> > > >> >> suspect it's because the read-ahead buffer cannot be
>> >> > > >> >> effective for random accesses.  Of course, reading
>> >> > > >> >> sequentially from a single file will result in a
>> >> > > >> >> throughput that is far too high to simulate the
>> >> > > >> >> application.
>> >> > > >> >
>> >> > > >> > Have you tried
>> >> > > >> >
>> >> > > >> >        nrfiles=int
>> >> > > >> >
>> >> > > >> >               Number of files to use for this job.
>> >> > > >> >               Default: 1.
>> >> > > >> >
>> >> > > >> >        openfiles=int
>> >> > > >> >
>> >> > > >> >               Number of files to keep open at the same
>> >> > > >> >               time. Default: nrfiles.
>> >> > > >> >
>> >> > > >> >        file_service_type=str
>> >> > > >
>> >> > > > [�]
>> >> > > >
>> >> > > >> > ? (see fio manpage).
>> >> > > >> >
>> >> > > >> > It seems to me that all you need is nrfiles. I�d bet that
>> >> > > >> > fio distributes
>> >> > > >> > the I/O size given among those files, but AFAIR there is
>> >> > > >> > something about
>> >> > > >> > that in fio documentation as well.
>> >> > > >> >
>> >> > > >> > Use the doc! ;)
>> >> > > >
>> >> > > > [�]
>> >> > > >
>> >> > > >> Yes, I have tried all that, and it works, except that it
>> >> > > >> causes disk queuing, as I stated in my first post. I thought
>> >> > > >> you meant to put all the files into a single [job name]
>> >> > > >> section of the ini file, to enforce single threaded io.
>> >> > > >
>> >> > > > With just one job running at once?
>> >> > > >
>> >> > > > Can you post an example job file?
>> >> > > >
>> >> > > > Did you try the sync=1 / direct=1 suggestion from Bruce Chan?
>> >> > > >
>> >> > > > I only know the behaviour of fio on Linux where I/O depth of
>> >> > > > greater than one is only possible with libaio and direct=1.
>> >> > > > The manpage hints at I/O depth is one for all synchronous I/O
>> >> > > > engines, so I�d bet that refers to Windows as well.
>> >> > > >
>> >> > > > Other than that I have no idea.
>> >> >
>> >> > [�]
>> >> >
>> >> > > One INI file, but a seperate [job name] section for each file,
>> >> > > yes. According to Jens, because each [job name] is a seperate
>> >> > > thread, and iodepth acts at the thread level, there will still
>> >> > > be queuing at the device level. If there were a way to do what
>> >> > > I want I think Jens would have told me, unfortunately.   ;)
>> >> > >
>> >> > > direct io does at least allow me to do cache-less reads though -
>> >> > > thankyou.
>> >> >
>> >> > My suggestion is to use one job with several files.
>> >> >
>> >> > martin@merkaba:/tmp> cat severalfiles.job
>> >> > [global]
>> >> > size=1G
>> >> > nrfiles=100
>> >> >
>> >> > [read]
>> >> > rw=read
>> >> >
>> >> > [write]
>> >> > stonewall
>> >> > rw=write
>> >> >
>> >> > (now these are two jobs, but stonewall lets the write job run
>> >> > after the read one with cache invalidation if not disabled and if
>> >> > supported by OS)
>> >> >
>> >> > martin@merkaba:/tmp> fio severalfiles.job
>> >> > read: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
>> >> > write: (g=1): rw=write, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
>> >> > 2.0.8
>> >> > Starting 2 processes
>> >> > read: Laying out IO file(s) (100 file(s) / 1023MB)
>> >> > write: Laying out IO file(s) (100 file(s) / 1023MB)
>> >> > Jobs: 1 (f=100)
>> >> > read: (groupid=0, jobs=1): err= 0: pid=23377
>> >> > [� lots of fast due to /tmp being a RAM-based filesystem � tmpfs
>> >> > �]
>> >> >
>> >> >
>> >> > martin@merkaba:/tmp> ls -lh read.1.* | head
>> >> > -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.0
>> >> > -rw-r--r-- 1 martin martin 11M Aug 14 23:15 read.1.1
>> >
>> > [�]
>> >
>> >> > [� only first ten displayed �]
>> >> >
>> >> > martin@merkaba:/tmp> find -name "read.1*" 2>/dev/null | wc -l
>> >> > 100
>> >> >
>> >> > 100 files a 11M, due to rounding issues that may nicely add up to
>> >> > the one GiB.
>> >> >
>> >> > Raw sizes are:
>> >> >
>> >> > martin@merkaba:/tmp> ls -l read.1.* | head
>> >> > -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.0
>> >> > -rw-r--r-- 1 martin martin 10737418 Aug 14 23:20 read.1.1
>> >
>> > [�]
>> >
>> >> > Note: When I used filename, fio just created one files regardless
>> >> > of the nrfiles setting. I would have expected it to use the
>> >> > filename as a prefix. There might be some way to have it do that.
>> >> >
>> >> > Ciao,
>> >>
>> >> Thanks - that runs, but it's still queuing. As I said before, I
>> >> can't use the sync engine - I receive an error. Is there a
>> >> synchronous engine available for Windows? Perhaps that's the only
>> >> problem. Can you check to see whether your system is queuing at the
>> >> file system/device level when you run that test?
>> >>
>> >> I had attempted to put the files in a single job earlier - I think
>> >> it may have been successfully accessing both files, but I didn't
>> >> notice it in the output. I'm a raw beginner.
>> >
>> > Did you try with
>> >
>> > ioengine=windowsaio
>> >
>> > +
>> >
>> > iodepth=1 (should be default however I think)
>> >
>> >
>> > Otherwise I have no idea. I never used fio on Windows so far.
>> >
>> > It might help when you try to explain exactly which problem you want
>> > to solve by the fio measurements. Multimedia streaming. Is it to
>> > slow? What is
>> >
>> > it why you want to do these measurements?
>>
>> They are both defaults, and the output shows that both are being used.
>> If you could tell me whether your system is generating queuing it
>> would help, because if yours queues even when using the sync io
>> engine, it means I'm wasting my time and fio simply needs to be
>> augmented to support strict single threaded operation over multiple
>> files.
>>
>> I am wanting to determine whether the application in question is
>> extracting a reasonable number of real time streams from any given
>> storage system.
>
> Just for the record since you got it working on Windows as well � it works for me:
>
> merkaba:/tmp> cat severalfiles.job
> [global]
> size=1G
> nrfiles=100
>
> [read]
> rw=read
>
> merkaba:/tmp> fio severalfiles.job
> read: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
> 2.0.8
> Starting 1 process
>
> read: (groupid=0, jobs=1): err= 0: pid=4579
>   read : io=1023.9MB, bw=2409.7MB/s, iops=616705 , runt=   425msec
>     clat (usec): min=0 , max=54 , avg= 1.08, stdev= 0.64
>      lat (usec): min=0 , max=54 , avg= 1.13, stdev= 0.66
>     clat percentiles (usec):
>      |  1.00th=[    0],  5.00th=[    1], 10.00th=[    1], 20.00th=[    1],
>      | 30.00th=[    1], 40.00th=[    1], 50.00th=[    1], 60.00th=[    1],
>      | 70.00th=[    1], 80.00th=[    1], 90.00th=[    1], 95.00th=[    2],
>      | 99.00th=[    2], 99.50th=[    2], 99.90th=[   14], 99.95th=[   16],
>      | 99.99th=[   23]
>     lat (usec) : 2=92.14%, 4=7.74%, 10=0.01%, 20=0.10%, 50=0.01%
>     lat (usec) : 100=0.01%
>   cpu          : usr=22.41%, sys=76.42%, ctx=421, majf=0, minf=36
>   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>      issued    : total=r=262100/w=0/d=0, short=r=0/w=0/d=0
>
> Run status group 0 (all jobs):
>    READ: io=1023.9MB, aggrb=2409.7MB/s, minb=2409.7MB/s, maxb=2409.7MB/s, mint=425msec, maxt=425msec
>
> (if you wonder about the figures � thats RAM testing � Linux tmpfs ;)
>
> 100% at IO depth 1.
>
> Ciao,
> --
> Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
> GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Are you sure that the "100% at IO depth 1" actually means that there
is no queuing at the device? I.e - doesn't that IO depth simply mean
that each thread reached an IO depth of 1, without necessarily
referring to the disk queue?  I'm actually checking my physical disk
queue. Before I was using direct io, the io depths were 100% at 1 just
like yours, but ios were being queued at the device level. I now have
confidence that it probably is working for you, of course, but I'm not
yet convinced that you are doing the correct checking, that's all. ;^)

Greg.