Re: [RESEND] [PATCH] readahead:add blk_run_backing_dev

[Vladislav Bolkhovitin <vst@vlnb.net>]

Ronald Moesbergen, on 07/03/2009 01:14 PM wrote:
>>> OK, now I tend to agree on decreasing max_sectors_kb and increasing
>>> read_ahead_kb. But before actually trying to push that idea I'd like
>>> to
>>> - do more benchmarks
>>> - figure out why context readahead didn't help SCST performance
>>>  (previous traces show that context readahead is submitting perfect
>>>   large io requests, so I wonder if it's some io scheduler bug)
>> Because, as we found out, without your http://lkml.org/lkml/2009/5/21/319
>> patch read-ahead was nearly disabled, hence there were no difference which
>> algorithm was used?
>>
>> Ronald, can you run the following tests, please? This time with 2 hosts,
>> initiator (client) and target (server) connected using 1 Gbps iSCSI. It
>> would be the best if on the client vanilla 2.6.29 will be ran, but any other
>> kernel will be fine as well, only specify which. Blockdev-perftest should be
>> ran as before in buffered mode, i.e. with "-a" switch.
>>
>> 1. All defaults on the client, on the server vanilla 2.6.29 with Fengguang's
>> http://lkml.org/lkml/2009/5/21/319 patch with all default settings.
>>
>> 2. All defaults on the client, on the server vanilla 2.6.29 with Fengguang's
>> http://lkml.org/lkml/2009/5/21/319 patch with default RA size and 64KB
>> max_sectors_kb.
>>
>> 3. All defaults on the client, on the server vanilla 2.6.29 with Fengguang's
>> http://lkml.org/lkml/2009/5/21/319 patch with 2MB RA size and default
>> max_sectors_kb.
>>
>> 4. All defaults on the client, on the server vanilla 2.6.29 with Fengguang's
>> http://lkml.org/lkml/2009/5/21/319 patch with 2MB RA size and 64KB
>> max_sectors_kb.
>>
>> 5. All defaults on the client, on the server vanilla 2.6.29 with Fengguang's
>> http://lkml.org/lkml/2009/5/21/319 patch and with context RA patch. RA size
>> and max_sectors_kb are default. For your convenience I committed the
>> backported context RA patches into the SCST SVN repository.
>>
>> 6. All defaults on the client, on the server vanilla 2.6.29 with Fengguang's
>> http://lkml.org/lkml/2009/5/21/319 and context RA patches with default RA
>> size and 64KB max_sectors_kb.
>>
>> 7. All defaults on the client, on the server vanilla 2.6.29 with Fengguang's
>> http://lkml.org/lkml/2009/5/21/319 and context RA patches with 2MB RA size
>> and default max_sectors_kb.
>>
>> 8. All defaults on the client, on the server vanilla 2.6.29 with Fengguang's
>> http://lkml.org/lkml/2009/5/21/319 and context RA patches with 2MB RA size
>> and 64KB max_sectors_kb.
>>
>> 9. On the client default RA size and 64KB max_sectors_kb. On the server
>> vanilla 2.6.29 with Fengguang's http://lkml.org/lkml/2009/5/21/319 and
>> context RA patches with 2MB RA size and 64KB max_sectors_kb.
>>
>> 10. On the client 2MB RA size and default max_sectors_kb. On the server
>> vanilla 2.6.29 with Fengguang's http://lkml.org/lkml/2009/5/21/319 and
>> context RA patches with 2MB RA size and 64KB max_sectors_kb.
>>
>> 11. On the client 2MB RA size and 64KB max_sectors_kb. On the server vanilla
>> 2.6.29 with Fengguang's http://lkml.org/lkml/2009/5/21/319 and context RA
>> patches with 2MB RA size and 64KB max_sectors_kb.
> 
> Ok, done. Performance is pretty bad overall :(
> 
> The kernels I used:
> client kernel: 2.6.26-15lenny3 (debian)
> server kernel: 2.6.29.5 with blk_dev_run patch
> 
> And I adjusted the blockdev-perftest script to drop caches on both the
> server (via ssh) and the client.
> 
> The results:
> 
> 1) client: default, server: default
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  19.808   20.078   20.180   51.147    0.402    0.799
>  33554432  19.162   19.952   20.375   51.673    1.322    1.615
>  16777216  19.714   20.331   19.948   51.214    0.649    3.201
>   8388608  18.572   20.126   20.345   52.116    2.149    6.515
>   4194304  18.711   19.663   19.811   52.831    1.350   13.208
>   2097152  19.112   19.927   19.130   52.832    1.022   26.416
>   1048576  19.771   19.686   20.010   51.661    0.356   51.661
>    524288  19.585   19.940   19.483   52.065    0.515  104.131
>    262144  19.168   20.794   19.605   51.634    1.757  206.535
>    131072  19.077   20.776   20.271   51.160    1.849  409.282
>     65536  19.643   21.230   19.144   51.284    2.227  820.549
>     32768  19.702   20.869   19.686   51.020    1.380 1632.635
>     16384  21.218   20.222   20.221   49.846    1.121 3190.174
> 
> 2) client: default, server: 64 max_sectors_kb
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  20.881   20.102   21.689   49.065    1.522    0.767
>  33554432  20.329   19.938   20.522   50.543    0.609    1.579
>  16777216  20.247   19.744   20.912   50.468    1.185    3.154
>   8388608  19.739   20.184   21.032   50.433    1.318    6.304
>   4194304  19.968   18.748   20.230   52.174    1.750   13.043
>   2097152  19.633   20.068   19.858   51.584    0.462   25.792
>   1048576  20.552   20.618   20.974   49.437    0.440   49.437
>    524288  21.595   20.830   20.454   48.881    1.098   97.762
>    262144  21.720   20.602   20.176   49.201    1.515  196.805
>    131072  20.976   19.089   20.712   50.634    2.144  405.072
>     65536  20.661   19.952   19.312   51.303    1.414  820.854
>     32768  21.155   18.464   20.640   51.159    3.081 1637.090
>     16384  22.023   19.944   20.629   49.159    2.008 3146.205
> 
> 3) client: default, server: default max_sectors_kb, RA 2MB
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  21.709   19.315   18.319   52.028    3.631    0.813
>  33554432  20.745   19.209   19.048   52.142    1.976    1.629
>  16777216  19.762   19.175   19.485   52.591    0.649    3.287
>   8388608  19.812   19.142   19.574   52.498    0.749    6.562
>   4194304  19.931   19.786   19.505   51.877    0.466   12.969
>   2097152  19.473   19.208   19.438   52.859    0.322   26.430
>   1048576  19.524   19.033   19.477   52.941    0.610   52.941
>    524288  20.115   20.402   19.542   51.166    0.920  102.333
>    262144  19.291   19.715   21.016   51.249    1.844  204.996
>    131072  18.782   19.130   20.334   52.802    1.775  422.419
>     65536  19.030   19.233   20.328   52.475    1.504  839.599
>     32768  19.147   19.326   19.411   53.074    0.303 1698.357
>     16384  19.573   19.596   20.417   51.575    1.005 3300.788
> 
> 4) client: default, server: 64 max_sectors_kb, RA 2MB
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  22.604   21.707   20.721   47.298    1.683    0.739
>  33554432  21.654   20.812   21.162   48.293    0.784    1.509
>  16777216  20.461   19.782   21.160   50.068    1.377    3.129
>   8388608  20.886   20.434   21.512   48.914    1.028    6.114
>   4194304  22.154   20.512   21.433   47.974    1.517   11.993
>   2097152  22.258   20.971   20.738   48.071    1.478   24.035
>   1048576  19.953   21.294   19.662   50.497    1.731   50.497
>    524288  21.577   20.884   20.883   48.509    0.743   97.019
>    262144  20.959   20.749   20.256   49.587    0.712  198.347
>    131072  19.926   21.542   19.634   50.360    2.022  402.877
>     65536  20.973   22.546   20.840   47.793    1.685  764.690
>     32768  20.695   21.031   21.182   48.837    0.476 1562.791
>     16384  20.163   21.112   20.037   50.133    1.159 3208.481
> 
> 5) Server RA-context Patched, client: default, server: default
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  19.756   23.647   18.852   49.818    4.717    0.778
>  33554432  18.892   19.727   18.857   53.472    1.106    1.671
>  16777216  18.943   19.255   18.949   53.760    0.409    3.360
>   8388608  18.766   19.105   18.847   54.165    0.413    6.771
>   4194304  19.177   19.609   20.191   52.111    1.097   13.028
>   2097152  18.968   19.517   18.862   53.581    0.797   26.790
>   1048576  18.833   19.912   18.626   53.592    1.551   53.592
>    524288  19.128   19.379   19.134   53.298    0.324  106.596
>    262144  18.955   19.328   18.879   53.748    0.550  214.992
>    131072  18.401   19.642   18.928   53.961    1.439  431.691
>     65536  19.366   19.822   18.615   53.182    1.384  850.908
>     32768  19.252   19.229   18.752   53.683    0.653 1717.857
>     16384  21.373   19.507   19.162   51.282    2.415 3282.039
> 
> 6) Server RA-context Patched, client: default, server: 64
> max_sectors_kb, RA default
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  22.753   21.071   20.532   47.825    2.061    0.747
>  33554432  20.404   19.239   20.722   50.943    1.644    1.592
>  16777216  20.914   20.114   21.854   48.910    1.655    3.057
>   8388608  19.524   21.932   21.465   48.949    2.510    6.119
>   4194304  20.306   20.809   20.000   50.279    0.820   12.570
>   2097152  20.133   20.194   20.181   50.770    0.066   25.385
>   1048576  19.515   21.593   20.052   50.321    2.128   50.321
>    524288  20.231   20.502   20.299   50.335    0.284  100.670
>    262144  19.620   19.737   19.911   51.834    0.313  207.336
>    131072  20.486   21.138   22.339   48.089    1.711  384.714
>     65536  20.113   18.322   22.247   50.943    4.025  815.088
>     32768  23.341   23.328   20.809   45.659    2.511 1461.089
>     16384  20.962   21.839   23.405   46.496    2.100 2975.773
> 
> 7) Server RA-context Patched, client: default, server: default
> max_sectors_kb, RA 2MB
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  19.565   19.028   19.164   53.196    0.627    0.831
>  33554432  19.048   18.401   18.940   54.491    0.828    1.703
>  16777216  18.728   19.330   19.076   53.778    0.699    3.361
>   8388608  19.174   18.710   19.922   53.179    1.368    6.647
>   4194304  19.133   18.514   19.672   53.628    1.331   13.407
>   2097152  18.903   18.547   20.070   53.468    1.782   26.734
>   1048576  19.210   19.204   18.994   53.513    0.282   53.513
>    524288  18.978   18.723   20.839   52.596    2.464  105.192
>    262144  18.912   18.590   18.635   54.726    0.415  218.905
>    131072  18.732   18.578   19.797   53.837    1.505  430.694
>     65536  19.046   18.872   19.318   53.678    0.516  858.852
>     32768  18.490   18.582   20.374   53.583    2.353 1714.661
>     16384  19.138   19.215   20.602   52.167    1.744 3338.700
> 
>  8) Server RA-context Patched, client: default, server: 64
> max_sectors_kb, RA 2MB
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  21.029   21.654   21.093   48.177    0.630    0.753
>  33554432  21.174   19.759   20.659   49.918    1.435    1.560
>  16777216  20.385   20.235   22.145   49.026    1.976    3.064
>   8388608  19.053   20.162   20.158   51.778    1.391    6.472
>   4194304  20.123   23.173   20.073   48.696    3.188   12.174
>   2097152  19.401   20.824   20.326   50.778    1.500   25.389
>   1048576  21.821   21.401   21.026   47.825    0.724   47.825
>    524288  21.478   20.742   21.355   48.332    0.742   96.664
>    262144  20.290   20.183   20.980   50.004    0.853  200.015
>    131072  20.299   21.501   20.766   49.127    1.158  393.020
>     65536  21.087   19.340   20.867   50.193    1.959  803.092
>     32768  21.597   21.223   23.504   46.410    2.039 1485.132
>     16384  21.681   21.709   22.944   46.343    1.212 2965.967
> 
> 9) Server RA-context Patched, client: 64 max_sectors_kb, default RA.
> server: 64 max_sectors_kb, RA 2MB
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  42.767   40.615   41.188   24.672    0.535    0.386
>  33554432  41.204   42.294   40.514   24.780    0.437    0.774
>  16777216  39.774   42.809   41.804   24.720    0.762    1.545
>   8388608  42.292   41.799   40.386   24.689    0.486    3.086
>   4194304  41.784   39.037   41.830   25.073    0.819    6.268
>   2097152  41.983   41.145   44.115   24.164    0.703   12.082
>   1048576  41.468   43.495   41.640   24.276    0.520   24.276
>    524288  42.631   42.724   41.267   24.267    0.387   48.535
>    262144  41.930   41.954   41.975   24.408    0.011   97.634
>    131072  42.511   41.266   42.835   24.269    0.393  194.154
>     65536  41.307   41.544   40.746   24.857    0.203  397.704
>     32768  42.270   42.728   40.822   24.425    0.478  781.607
>     16384  39.307   40.044   40.259   25.686    0.264 1643.908
>      8192  41.258   40.879   40.969   24.955    0.098 3194.183
> 
> 10) Server RA-context Patched, client: default max_sectors_kb, 2MB RA.
> server: 64 max_sectors_kb, RA 2MB
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  26.160   26.878   25.790   38.982    0.666    0.609
>  33554432  25.832   25.362   25.695   39.956    0.309    1.249
>  16777216  26.119   24.769   25.526   40.221    0.876    2.514
>   8388608  25.660   26.257   25.106   39.898    0.730    4.987
>   4194304  26.603   25.404   25.271   39.773    0.910    9.943
>   2097152  26.012   24.815   26.064   39.973    0.914   19.986
>   1048576  25.256   27.073   25.153   39.693    1.323   39.693
>    524288  29.452   28.883   29.146   35.118    0.280   70.236
>    262144  26.559   27.315   26.837   38.067    0.440  152.268
>    131072  25.259   25.794   25.992   39.879    0.483  319.030
>     65536  26.417   25.205   26.177   39.503    0.808  632.047
>     32768  26.453   26.401   25.759   39.083    0.474 1250.669
>     16384  24.701   24.609   25.143   41.265    0.385 2640.945
> 
> 11) Server RA-context Patched, client: 64 max_sectors_kb, 2MB. RA
> server: 64 max_sectors_kb, RA 2MB
> blocksize       R        R        R   R(avg,    R(std        R
>   (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
>  67108864  29.629   31.703   30.407   33.513    0.930    0.524
>  33554432  29.768   29.598   30.717   34.111    0.553    1.066
>  16777216  30.054   30.640   30.102   33.837    0.295    2.115
>   8388608  29.906   29.744   31.394   33.762    0.813    4.220
>   4194304  30.708   30.797   30.418   33.420    0.177    8.355
>   2097152  31.364   29.646   30.712   33.511    0.781   16.755
>   1048576  30.757   30.600   30.470   33.455    0.128   33.455
>    524288  29.715   31.176   29.977   33.822    0.701   67.644
>    262144  30.533   30.218   30.259   33.755    0.155  135.021
>    131072  30.403   32.609   30.651   32.831    1.016  262.645
>     65536  30.846   30.208   32.116   32.993    0.835  527.889
>     32768  30.526   29.794   30.556   33.809    0.397 1081.878
>     16384  31.560   31.532   30.938   32.673    0.301 2091.092

Those are on the server without io_context-2.6.29 and readahead-2.6.29 
patches applied and with CFQ scheduler, correct?

Then we see how reorder of requests caused by many I/O threads 
submitting I/O in separate I/O contexts badly affect performance and no 
RA, especially with default 128KB RA size, can solve it. Less 
max_sectors_kb on the client => more requests it sends at once => more 
reorder on the server => worse throughput. Although, Fengguang, in 
theory, context RA with 2MB RA size should considerably help it, no?

Ronald, can you perform those tests again with both io_context-2.6.29 
and readahead-2.6.29 patches applied on the server, please?

Thanks,
Vlad