Rate problems (Windows)

Rate problems (Windows)

[Greg Sullivan]

I'm finding that the throughput is varying wildly during a run - more
than I think is probably normal. It happens regardless of whether I
limit by data rate or io rate, although I think it's worse when I
limit by io.

Here is a test job followed by the output. The io rate fluctuated
wildly - I watched it in the Windows Performance Monitor. The io rate
should be extremely easy to maintain. The "CR=80" looks suspicious -
why is there an 80 there when I requested 40?

BEGIN>>>
[streams]
sync=1
direct=1
bs=4k
thread
size=10M
nrfiles=1
rate_iops=40
rw=read
END>>>

BEGIN>>>
D:\>fio test.ini
streams: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=windowsaio, iodepth=1
fio-2.0.8-19-g9576
Starting 1 thread
Jobs: 1 (f=1), CR=80/0 IOPS: [R] [100.0% done] [160K/0K /s] [40 /0
iops] [eta 00m:00s]
streams: (groupid=0, jobs=1): err= 0: pid=4888: Sun Aug 19 22:32:01 2012
  read : io=10240KB, bw=163898 B/s, iops=40 , runt= 63977msec
    slat (usec): min=13 , max=106 , avg=23.81, stdev= 7.43
    clat (usec): min=214 , max=26940 , avg=688.03, stdev=1920.57
     lat (usec): min=248 , max=26986 , avg=711.84, stdev=1921.02
    clat percentiles (usec):
     |  1.00th=[  239],  5.00th=[  262], 10.00th=[  266], 20.00th=[  270],
     | 30.00th=[  274], 40.00th=[  278], 50.00th=[  282], 60.00th=[  286],
     | 70.00th=[  290], 80.00th=[  302], 90.00th=[  354], 95.00th=[ 4960],
     | 99.00th=[ 5088], 99.50th=[18048], 99.90th=[19328], 99.95th=[19328],
     | 99.99th=[27008]
    bw (KB/s)  : min=    3, max=  961, per=100.00%, avg=218.90, stdev=207.65
    lat (usec) : 250=1.80%, 500=89.80%, 750=2.23%
    lat (msec) : 10=5.39%, 20=0.74%, 50=0.04%
  cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2560/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=10240KB, aggrb=160KB/s, minb=160KB/s, maxb=160KB/s,
mint=63977msec, maxt=63977msec

D:\>
END>>>

Windows 7 32-bit.

Greg.

Re: Rate problems (Windows)

[Greg Sullivan]

On 19 August 2012 22:39, Greg Sullivan <greg.sullivan@sullivang.net> wrote:
> I'm finding that the throughput is varying wildly during a run - more
> than I think is probably normal. It happens regardless of whether I
> limit by data rate or io rate, although I think it's worse when I
> limit by io.
>
> Here is a test job followed by the output. The io rate fluctuated
> wildly - I watched it in the Windows Performance Monitor. The io rate
> should be extremely easy to maintain. The "CR=80" looks suspicious -
> why is there an 80 there when I requested 40?
[stuff removed]

I tried the same test on Ubuntu, and I think it's working properly -
the iops figures displayed by fio are relatively constant as the test
runs.  (although I'm having trouble relating the the output of iostat
to the actual workload that should be generated) I still get the
CR=<double the requested rate> on Ubuntu, so I assume this is normal.
The iops figures over on the right of the output look kosher. (as they
do on Windows, except for the wide fluctuations)

Greg.

Re: Rate problems (Windows)

[Jens Axboe]

On 08/20/2012 03:42 AM, Greg Sullivan wrote:
> On 19 August 2012 22:39, Greg Sullivan <greg.sullivan@sullivang.net> wrote:
>> I'm finding that the throughput is varying wildly during a run - more
>> than I think is probably normal. It happens regardless of whether I
>> limit by data rate or io rate, although I think it's worse when I
>> limit by io.
>>
>> Here is a test job followed by the output. The io rate fluctuated
>> wildly - I watched it in the Windows Performance Monitor. The io rate
>> should be extremely easy to maintain. The "CR=80" looks suspicious -
>> why is there an 80 there when I requested 40?
> [stuff removed]
> 
> I tried the same test on Ubuntu, and I think it's working properly -
> the iops figures displayed by fio are relatively constant as the test
> runs.  (although I'm having trouble relating the the output of iostat
> to the actual workload that should be generated) I still get the
> CR=<double the requested rate> on Ubuntu, so I assume this is normal.
> The iops figures over on the right of the output look kosher. (as they
> do on Windows, except for the wide fluctuations)

The CR=x/y reading is as follows:

- x is the target read _and_ write commit rate. If your workload was a
  read/write workload, it'd be 40 from each. I should change that, it's
  confusing. But for now, suffice to say that it is to be expected and
  the 40 is parsed/understood by fio.
- y is the same, but the minimum rate (if that is set).

See below patch, should make it more easily understandable.

As to the Windows rate fluctuations - a quick guess would be that the
timing is too coarse on Windows.

diff --git a/eta.c b/eta.c
index 9114595..552845d 100644
--- a/eta.c
+++ b/eta.c
@@ -285,11 +285,18 @@ int calc_thread_status(struct jobs_eta *je, int force)
 		    || td->runstate == TD_FSYNCING
 		    || td->runstate == TD_PRE_READING) {
 			je->nr_running++;
-			je->t_rate += td->o.rate[0] + td->o.rate[1];
-			je->m_rate += td->o.ratemin[0] + td->o.ratemin[1];
-			je->t_iops += td->o.rate_iops[0] + td->o.rate_iops[1];
-			je->m_iops += td->o.rate_iops_min[0] +
-					td->o.rate_iops_min[1];
+			if (td_read(td)) {
+				je->t_rate += td->o.rate[DDIR_READ];
+				je->t_iops += td->o.rate_iops[DDIR_READ];
+				je->m_rate += td->o.ratemin[DDIR_READ];
+				je->m_iops += td->o.rate_iops_min[DDIR_READ];
+			}
+			if (td_write(td)) {
+				je->t_rate += td->o.rate[DDIR_WRITE];
+				je->t_iops += td->o.rate_iops[DDIR_WRITE];
+				je->m_rate += td->o.ratemin[DDIR_WRITE];
+				je->m_iops += td->o.rate_iops_min[DDIR_WRITE];
+			}
 			je->files_open += td->nr_open_files;
 		} else if (td->runstate == TD_RAMP) {
 			je->nr_running++;

-- 
Jens Axboe

Re: Rate problems (Windows)

[Greg Sullivan]

On 20 August 2012 22:43, Jens Axboe <axboe@kernel.dk> wrote:
> On 08/20/2012 03:42 AM, Greg Sullivan wrote:
>> On 19 August 2012 22:39, Greg Sullivan <greg.sullivan@sullivang.net> wrote:
>>> I'm finding that the throughput is varying wildly during a run - more
>>> than I think is probably normal. It happens regardless of whether I
>>> limit by data rate or io rate, although I think it's worse when I
>>> limit by io.
>>>
>>> Here is a test job followed by the output. The io rate fluctuated
>>> wildly - I watched it in the Windows Performance Monitor. The io rate
>>> should be extremely easy to maintain. The "CR=80" looks suspicious -
>>> why is there an 80 there when I requested 40?
>> [stuff removed]
>>
>> I tried the same test on Ubuntu, and I think it's working properly -
>> the iops figures displayed by fio are relatively constant as the test
>> runs.  (although I'm having trouble relating the the output of iostat
>> to the actual workload that should be generated) I still get the
>> CR=<double the requested rate> on Ubuntu, so I assume this is normal.
>> The iops figures over on the right of the output look kosher. (as they
>> do on Windows, except for the wide fluctuations)
>
> The CR=x/y reading is as follows:
>
> - x is the target read _and_ write commit rate. If your workload was a
>   read/write workload, it'd be 40 from each. I should change that, it's
>   confusing. But for now, suffice to say that it is to be expected and
>   the 40 is parsed/understood by fio.
> - y is the same, but the minimum rate (if that is set).
>
> See below patch, should make it more easily understandable.
>
> As to the Windows rate fluctuations - a quick guess would be that the
> timing is too coarse on Windows.
>
> diff --git a/eta.c b/eta.c
> index 9114595..552845d 100644
> --- a/eta.c
> +++ b/eta.c
> @@ -285,11 +285,18 @@ int calc_thread_status(struct jobs_eta *je, int force)
>                     || td->runstate == TD_FSYNCING
>                     || td->runstate == TD_PRE_READING) {
>                         je->nr_running++;
> -                       je->t_rate += td->o.rate[0] + td->o.rate[1];
> -                       je->m_rate += td->o.ratemin[0] + td->o.ratemin[1];
> -                       je->t_iops += td->o.rate_iops[0] + td->o.rate_iops[1];
> -                       je->m_iops += td->o.rate_iops_min[0] +
> -                                       td->o.rate_iops_min[1];
> +                       if (td_read(td)) {
> +                               je->t_rate += td->o.rate[DDIR_READ];
> +                               je->t_iops += td->o.rate_iops[DDIR_READ];
> +                               je->m_rate += td->o.ratemin[DDIR_READ];
> +                               je->m_iops += td->o.rate_iops_min[DDIR_READ];
> +                       }
> +                       if (td_write(td)) {
> +                               je->t_rate += td->o.rate[DDIR_WRITE];
> +                               je->t_iops += td->o.rate_iops[DDIR_WRITE];
> +                               je->m_rate += td->o.ratemin[DDIR_WRITE];
> +                               je->m_iops += td->o.rate_iops_min[DDIR_WRITE];
> +                       }
>                         je->files_open += td->nr_open_files;
>                 } else if (td->runstate == TD_RAMP) {
>                         je->nr_running++;
>
> --
> Jens Axboe
>

I have now done a basic test of the rate functionality on Ubuntu, and
it seems to be working fine. I am more confident now that there is
indeed a problem when running fio on Windows.

Greg.

Re: Rate problems (Windows)

[Bruce Cran]

On 21/08/2012 12:35, Greg Sullivan wrote:
> I have now done a basic test of the rate functionality on Ubuntu, and
> it seems to be working fine. I am more confident now that there is
> indeed a problem when running fio on Windows.

Are you running the test on a new filesystem on a separate disk, or in a 
directory on a disk that's got lots of other files on it?

-- 
Bruce Cran

Re: Rate problems (Windows)

[Greg Sullivan]

On 21 August 2012 21:41, Bruce Cran <bruce@cran.org.uk> wrote:
> On 21/08/2012 12:35, Greg Sullivan wrote:
>>
>> I have now done a basic test of the rate functionality on Ubuntu, and
>> it seems to be working fine. I am more confident now that there is
>> indeed a problem when running fio on Windows.
>
>
> Are you running the test on a new filesystem on a separate disk, or in a
> directory on a disk that's got lots of other files on it?
>
> --
> Bruce Cran

A dedicated disk. It has some files on it, but not much. There are no
other processes accessing the disk.  Are you not able to reproduce the
problem?

Thanks,
Greg.

Re: Rate problems (Windows)

[Greg Sullivan]

On 21 August 2012 21:44, Greg Sullivan <greg.sullivan@sullivang.net> wrote:
> On 21 August 2012 21:41, Bruce Cran <bruce@cran.org.uk> wrote:
>> On 21/08/2012 12:35, Greg Sullivan wrote:
>>>
>>> I have now done a basic test of the rate functionality on Ubuntu, and
>>> it seems to be working fine. I am more confident now that there is
>>> indeed a problem when running fio on Windows.
>>
>>
>> Are you running the test on a new filesystem on a separate disk, or in a
>> directory on a disk that's got lots of other files on it?
>>
>> --
>> Bruce Cran
>
> A dedicated disk. It has some files on it, but not much. There are no
> other processes accessing the disk.  Are you not able to reproduce the
> problem?
>
> Thanks,
> Greg.

(just btw, after sending my initial problem report, I lowered the io
rate even further - down to 10 iops. Same problem)

Greg.

Re: Rate problems (Windows)

[Bruce Cran]

On 21/08/2012 12:44, Greg Sullivan wrote:
> A dedicated disk. It has some files on it, but not much. There are no
> other processes accessing the disk.  Are you not able to reproduce the
> problem?

I saw the problem last week on an old installation of Server 2008 R2 on 
a disk that was over 40% fragmented and had other traffic to it but now, 
on Windows 8 the IOPS stay between 39 and 40.

-- 
Bruce

Re: Rate problems (Windows)

[Greg Sullivan]

On 21 August 2012 21:53, Bruce Cran <bruce@cran.org.uk> wrote:
> On 21/08/2012 12:44, Greg Sullivan wrote:
>>
>> A dedicated disk. It has some files on it, but not much. There are no
>> other processes accessing the disk.  Are you not able to reproduce the
>> problem?
>
>
> I saw the problem last week on an old installation of Server 2008 R2 on a
> disk that was over 40% fragmented and had other traffic to it but now, on
> Windows 8 the IOPS stay between 39 and 40.
>
> --
> Bruce

It's working sometimes for me. However, sometimes it fails miserably.
For example, I just ran a job with a target rate of 50 iops, and the
output was on zero iops for ages (maybe about 20 seconds), and then it
briefly flashed up 1118 iops.  It might be working better in buffered
mode than it is in direct mode. (not 100% sure yet)

Greg.