regression in re-read operation by iozone up to 20%

regression in re-read operation by iozone up to 20%

[Matus Kocka]

Dear All,

for some time we see ~20% regression in reread operation in iozone data.
Kernel 4.6.0-1 was OK, but since that, there is ~20% regression from
1MB file size on SATA SSD disks and also similar one on SAS HDD,
as you can see here:  https://bugzilla.kernel.org/show_bug.cgi?id=155821

links to some plots:
SAS HDD: https://bugzilla.kernel.org/attachment.cgi?id=242491
SATA SSD: https://bugzilla.kernel.org/attachment.cgi?id=242501

The plots are for the XFS file system, but it looks very similar for the Ext4.
The machine which I used for the generating these plots is 2 NUMA,
32GB RAM Lenovo x3550 M5
I used numactl to pin iozone only to socket 0.

We see this also on different machines.

Can you help us to find the cause of this regression?
Do you have any idea why it is only in reread?

Data from the results, showing percentage difference between 4.6 and
4.7 kernels for various file size.

SATA SSD drive, iozone re-read operation

File size [MB] | kernel 4.6 [MB] | err [%] | kernel 4.7 [MB] | err [%]
| difference [%]
----------------------------------------------------------------------------------------
        1.0 | 13269.10 | 5.0 | 10416.83 | 1.9 | -21.5
        2.0 | 13015.61 | 4.7 | 10024.62 | 0.2 | -23.0
        4.0 | 12745.33 | 4.5 | 10050.46 | 0.2 | -21.1
        8.0 | 12905.56 | 3.1 |  9934.37 | 1.4 | -23.0
      16.0 | 11294.55 | 6.7 |  9627.13 | 0.0 | -14.8
      32.0 |   7417.55 | 4.8 |  6454.81 | 1.8 | -13.0
      64.0 |   6789.36 | 3.6 |  5759.64 | 2.0 | -15.2
    128.0 |   6768.6  | 0.8 |  5930.25 | 1.7 | -12.4
    256.0 |   7199.7  | 0.0 |  6101.62 | 0.7 | -15.2
    512.0 |   7456.02 | 0.1 |  6234.62 | 0.4 | -16.4
  1024.0 |   7503.09 | 0.4 |  6230.33 | 0.2 | -17.0
  2048.0 |   7509.11 | 0.4 |  6177.36 | 0.1 | -17.7
  4096.0 |   7486.92 | 0.1 |  6219.58 | 0.5 | -16.9
  8192.0 |   6860.39 | 0.0 |  5760.61 | 0.3 | -16.0
16384.0 |   6702.57 | 0.1 |  5624.17 | 0.1 | -16.1
32768.0 |   1215.87 | 0.8 |    942.31 |12.8 | -22.5
65536.0 |     511.72 | 0.2 |   512.58 | 0.2 | 0.2

SATA SSD drive, iozone re-read operation

File size [MB] | kernel 4.6 [MB] | err [%] | kernel 4.7 [MB] | err [%]
| difference [%]
----------------------------------------------------------------------------------------
        1.0 |  9281.34 | 29.5 | 9949.28 | 2.3 | 7.2
        2.0 |  8245.63 | 25.0 | 9732.59 | 2.1 | 18.0
        4.0 | 10183.30 |  2.7 | 9454.31 | 1.2 | -7.2
        8.0 | 10329.19 | 11.3 | 9093.59 | 4.7 | -12.0
      16.0 | 10356.85 | 11.1 | 8913.64 | 2.5 | -13.9
      32.0 |  7495.62 |  5.6 | 6217.42 | 2.4 | -17.1
      64.0 |  6608.68 |  7.6 | 5749.62 | 2.1 | -13.0
    128.0 |  7012.76 |  5.0 | 5927.56 | 1.1 | -15.5
    256.0 |  7203.37 |  1.7 | 6130.53 | 0.9 | -14.9
    512.0 |  7429.33 |  0.5 | 6219.21 | 0.4 | -16.3
  1024.0 |  7470.16 |  0.4 | 6247.00 | 0.3 | -16.4
  2048.0 |  7468.32 |  0.2 | 6209.81 | 0.8 | -16.9
  4096.0 |  7475.38 |  0.1 | 6065.46 | 1.8 | -18.9
  8192.0 |  6854.90 |  0.3 | 5687.97 | 0.2 | -17.0
16384.0 |  6678.79 |  0.3 | 5594.29 | 0.4 | -16.2
32768.0 |   444.30 |  0.7 |  483.13 |10.3 | 8.7
65536.0 |   304.49 |  0.0 |  250.57 | 0.0 | -17.7

Thanks in advance for any informations or suggestions.

Best Regards
Matus Kocka

Re: regression in re-read operation by iozone up to 20%

[Chris Mason]

On 10/24/2016 08:38 AM, Matus Kocka wrote:
> Dear All,
>
> for some time we see ~20% regression in reread operation in iozone data.
> Kernel 4.6.0-1 was OK, but since that, there is ~20% regression from
> 1MB file size on SATA SSD disks and also similar one on SAS HDD,
> as you can see here:  https://bugzilla.kernel.org/show_bug.cgi?id=155821
>
> links to some plots:
> SAS HDD: https://bugzilla.kernel.org/attachment.cgi?id=242491
> SATA SSD: https://bugzilla.kernel.org/attachment.cgi?id=242501
>
> The plots are for the XFS file system, but it looks very similar for the Ext4.
> The machine which I used for the generating these plots is 2 NUMA,
> 32GB RAM Lenovo x3550 M5
> I used numactl to pin iozone only to socket 0.
>
> We see this also on different machines.
>
> Can you help us to find the cause of this regression?
> Do you have any idea why it is only in reread?
>
> Data from the results, showing percentage difference between 4.6 and
> 4.7 kernels for various file size.

How fast is the storage for real?  Since these graphs all have a cliff 
at 32GB, my guess is that we're really benchmarking how much of the file 
is kept in ram for all of the smaller sizes.

I'd suggest watching with blktrace (or even vmstat) to measure how much 
is read from disk during the smaller runs.  If older kernels were doing 
better at keeping things in ram, that might explain the regression.

It could also be that reads are fighting with buffered writeback for 
access to the storage, but again vmstat and/or blktrace would help show 
that.

-chris

Re: regression in re-read operation by iozone up to 20%

[Chris Mason]

On 03/07/2017 11:32 AM, Chris Mason wrote:
> On 10/24/2016 08:38 AM, Matus Kocka wrote:
>> Dear All,
>>
>> for some time we see ~20% regression in reread operation in iozone data.
>> Kernel 4.6.0-1 was OK, but since that, there is ~20% regression from
>> 1MB file size on SATA SSD disks and also similar one on SAS HDD,
>> as you can see here:  https://bugzilla.kernel.org/show_bug.cgi?id=155821
>>
>> links to some plots:
>> SAS HDD: https://bugzilla.kernel.org/attachment.cgi?id=242491
>> SATA SSD: https://bugzilla.kernel.org/attachment.cgi?id=242501
>>
>> The plots are for the XFS file system, but it looks very similar for
>> the Ext4.
>> The machine which I used for the generating these plots is 2 NUMA,
>> 32GB RAM Lenovo x3550 M5
>> I used numactl to pin iozone only to socket 0.
>>
>> We see this also on different machines.
>>
>> Can you help us to find the cause of this regression?
>> Do you have any idea why it is only in reread?
>>
>> Data from the results, showing percentage difference between 4.6 and
>> 4.7 kernels for various file size.
>
> How fast is the storage for real?  Since these graphs all have a cliff
> at 32GB, my guess is that we're really benchmarking how much of the file
> is kept in ram for all of the smaller sizes.
>
> I'd suggest watching with blktrace (or even vmstat) to measure how much
> is read from disk during the smaller runs.  If older kernels were doing
> better at keeping things in ram, that might explain the regression.
>
> It could also be that reads are fighting with buffered writeback for
> access to the storage, but again vmstat and/or blktrace would help show
> that.

And I apparently just clicked sort-by-subject and found an ancient 
thread.  Sorry ;)

-chris