Re: Reduce latencies for syncronous writes and high I/O priority requests in deadline IO scheduler

[Aaron Carroll <aaronc@cse.unsw.edu.au>]

Corrado Zoccolo wrote:
> Hi,
> deadline I/O scheduler currently classifies all I/O requests in only 2
> classes, reads (always considered high priority) and writes (always
> lower).
> The attached patch, intended to reduce latencies for syncronous writes

Can be achieved by switching to sync/async rather than read/write.  No
one has shown results where this makes an improvement.  Let us know if
you have a good example.

> and high I/O priority requests, introduces more levels of priorities:
> * real time reads: highest priority and shortest deadline, can starve
> other levels
> * syncronous operations (either best effort reads or RT/BE writes),
> mid priority, starvation for lower level is prevented as usual
> * asyncronous operations (async writes and all IDLE class requests),
> lowest priority and longest deadline
> 
> The patch also introduces some new heuristics:
> * for non-rotational devices, reads (within a given priority level)
> are issued in FIFO order, to improve the latency perceived by readers

This might be a good idea.  Can you make this a separate patch?
Is there a good reason not to do the same for writes?

> * minimum batch timespan (time quantum): partners with fifo_batch to
> improve throughput, by sending more consecutive requests together. A
> given number of requests will not always take the same time (due to
> amount of seek needed), therefore fifo_batch must be tuned for worst
> cases, while in best cases, having longer batches would give a
> throughput boost.
> * batch start request is chosen fifo_batch/3 requests before the
> expired one, to improve fairness for requests with lower start sector,
> that otherwise have higher probability to miss a deadline than
> mid-sector requests.

I don't like the rest of it.  I use deadline because it's a simple,
no surprises, no bullshit scheduler with reasonably good performance
in all situations.  Is there some reason why CFQ won't work for you?

> I did few performance comparisons:
> * HDD, ext3 partition with data=writeback, tiotest with 32 threads,
> each writing 80MB of data
> 
> ** deadline-original
> Tiotest results for 32 concurrent io threads:
> ,----------------------------------------------------------------------.
> | Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
> +-----------------------+----------+--------------+----------+---------+
> | Write        2560 MBs |  103.0 s |  24.848 MB/s |  10.6 %  | 522.2 % |
> | Random Write  125 MBs |   98.8 s |   1.265 MB/s |  -1.6 %  |  16.1 % |
> | Read         2560 MBs |  166.2 s |  15.400 MB/s |   4.2 %  |  82.7 % |
> | Random Read   125 MBs |  193.3 s |   0.647 MB/s |  -0.8 %  |  14.5 % |
> `----------------------------------------------------------------------'
> Tiotest latency results:
> ,-------------------------------------------------------------------------.
> | Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |
> +--------------+-----------------+-----------------+----------+-----------+
> | Write        |        4.122 ms |    17922.920 ms |  0.07980 |   0.00061 |
> | Random Write |        0.599 ms |     1245.200 ms |  0.00000 |   0.00000 |
> | Read         |        8.032 ms |     1125.759 ms |  0.00000 |   0.00000 |
> | Random Read  |      181.968 ms |      972.657 ms |  0.00000 |   0.00000 |
> |--------------+-----------------+-----------------+----------+-----------|
> | Total        |       10.044 ms |    17922.920 ms |  0.03804 |   0.00029 |
> `--------------+-----------------+-----------------+----------+-----------'
> 
> ** cfq (2.6.30-rc2)
> Tiotest results for 32 concurrent io threads:
> ,----------------------------------------------------------------------.
> | Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
> +-----------------------+----------+--------------+----------+---------+
> | Write        2560 MBs |  132.4 s |  19.342 MB/s |   8.5 %  | 400.4 % |
> | Random Write  125 MBs |  107.8 s |   1.159 MB/s |  -1.6 %  |  16.8 % |
> | Read         2560 MBs |  107.6 s |  23.788 MB/s |   5.4 %  |  95.7 % |
> | Random Read   125 MBs |  158.4 s |   0.789 MB/s |   0.9 %  |   7.7 % |
> `----------------------------------------------------------------------'
> Tiotest latency results:
> ,-------------------------------------------------------------------------.
> | Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |
> +--------------+-----------------+-----------------+----------+-----------+
> | Write        |        5.362 ms |    21081.012 ms |  0.09811 |   0.00244 |
> | Random Write |       23.310 ms |    31865.095 ms |  0.13437 |   0.06250 |
> | Read         |        5.048 ms |     3694.001 ms |  0.15167 |   0.00000 |
> | Random Read  |      146.523 ms |     2880.409 ms |  0.52187 |   0.00000 |
> |--------------+-----------------+-----------------+----------+-----------|
> | Total        |        8.916 ms |    31865.095 ms |  0.13435 |   0.00262 |
> `--------------+-----------------+-----------------+----------+-----------'
> 
> ** deadline-patched
> Tiotest results for 32 concurrent io threads:
> ,----------------------------------------------------------------------.
> | Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
> +-----------------------+----------+--------------+----------+---------+
> | Write        2560 MBs |  105.3 s |  24.301 MB/s |  10.5 %  | 514.8 % |
> | Random Write  125 MBs |   95.9 s |   1.304 MB/s |  -1.8 %  |  17.3 % |
> | Read         2560 MBs |  165.1 s |  15.507 MB/s |   2.7 %  |  61.9 % |
> | Random Read   125 MBs |  110.6 s |   1.130 MB/s |   0.8 %  |  12.2 % |
> `----------------------------------------------------------------------'
> Tiotest latency results:
> ,-------------------------------------------------------------------------.
> | Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |
> +--------------+-----------------+-----------------+----------+-----------+
> | Write        |        4.131 ms |    17456.831 ms |  0.08041 |   0.00275 |
> | Random Write |        2.780 ms |     5073.180 ms |  0.07500 |   0.00000 |
> | Read         |        7.748 ms |      936.499 ms |  0.00000 |   0.00000 |
> | Random Read  |      104.849 ms |      695.192 ms |  0.00000 |   0.00000 |
> |--------------+-----------------+-----------------+----------+-----------|
> | Total        |        8.168 ms |    17456.831 ms |  0.04008 |   0.00131 |
> `--------------+-----------------+-----------------+----------+-----------'
> 
> * SD card, nilfs2 partition, tiotest with 16 threads, each writing 80MB of data
> ** cfq(2.6.30-rc2)
> Tiotest results for 16 concurrent io threads:
> ,----------------------------------------------------------------------.
> | Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
> +-----------------------+----------+--------------+----------+---------+
> | Write        1280 MBs |  217.8 s |   5.878 MB/s |   3.7 %  |  92.2 % |
> | Random Write   62 MBs |   18.2 s |   3.432 MB/s |  -2.3 %  |  28.7 % |
> | Read         1280 MBs |  114.7 s |  11.156 MB/s |   7.3 %  |  76.6 % |
> | Random Read    62 MBs |    3.4 s |  18.615 MB/s |  -5.4 %  | 274.2 % |
> `----------------------------------------------------------------------'
> Tiotest latency results:
> ,-------------------------------------------------------------------------.
> | Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |
> +--------------+-----------------+-----------------+----------+-----------+
> | Write        |        9.943 ms |    10223.581 ms |  0.14252 |   0.00488 |
> | Random Write |       12.287 ms |     5097.196 ms |  0.25625 |   0.00000 |
> | Read         |        5.352 ms |     1550.162 ms |  0.00000 |   0.00000 |
> | Random Read  |        3.051 ms |     1507.837 ms |  0.00000 |   0.00000 |
> |--------------+-----------------+-----------------+----------+-----------|
> | Total        |        7.649 ms |    10223.581 ms |  0.07391 |   0.00233 |
> `--------------+-----------------+-----------------+----------+-----------'
> 
> ** deadline-patched:
> Tiotest results for 16 concurrent io threads:
> ,----------------------------------------------------------------------.
> | Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
> +-----------------------+----------+--------------+----------+---------+
> | Write        1280 MBs |  220.9 s |   5.794 MB/s |   4.0 %  |  93.9 % |
> | Random Write   62 MBs |   20.5 s |   3.044 MB/s |  -2.2 %  |  24.9 % |
> | Read         1280 MBs |  113.2 s |  11.304 MB/s |   6.8 %  |  72.8 % |
> | Random Read    62 MBs |    2.9 s |  21.896 MB/s |   5.1 %  | 293.8 % |
> `----------------------------------------------------------------------'
> Tiotest latency results:
> ,-------------------------------------------------------------------------.
> | Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |
> +--------------+-----------------+-----------------+----------+-----------+
> | Write        |       10.078 ms |    13303.036 ms |  0.14160 |   0.00031 |
> | Random Write |       14.350 ms |     5265.088 ms |  0.40000 |   0.00000 |
> | Read         |        5.455 ms |      434.495 ms |  0.00000 |   0.00000 |
> | Random Read  |        2.685 ms |       12.652 ms |  0.00000 |   0.00000 |
> |--------------+-----------------+-----------------+----------+-----------|
> | Total        |        7.801 ms |    13303.036 ms |  0.07682 |   0.00015 |
> `--------------+-----------------+-----------------+----------+-----------'
> 
> * fsync-tester results, on HDD, empty ext3 partition, mounted with
> data=writeback
> ** deadline-original:
> fsync time: 0.7963
> fsync time: 4.5914
> fsync time: 4.2347
> fsync time: 1.1670
> fsync time: 0.8164
> fsync time: 1.9783
> fsync time: 4.9726
> fsync time: 2.4929
> fsync time: 2.5448
> fsync time: 3.9627
> ** cfq 2.6.30-rc2
> fsync time: 0.0288
> fsync time: 0.0528
> fsync time: 0.0299
> fsync time: 0.0397
> fsync time: 0.5720
> fsync time: 0.0409
> fsync time: 0.0876
> fsync time: 0.0294
> fsync time: 0.0485
> ** deadline-patched
> fsync time: 0.0772
> fsync time: 0.0381
> fsync time: 0.0604
> fsync time: 0.2923
> fsync time: 0.2488
> fsync time: 0.0924
> fsync time: 0.0144
> fsync time: 1.4824
> fsync time: 0.0789
> fsync time: 0.0565
> fsync time: 0.0550
> fsync time: 0.0421
> ** deadline-patched, ionice -c1:
> fsync time: 0.2569
> fsync time: 0.0500
> fsync time: 0.0681
> fsync time: 0.2863
> fsync time: 0.0140
> fsync time: 0.0171
> fsync time: 0.1198
> fsync time: 0.0530
> fsync time: 0.0503
> fsync time: 0.0462
> fsync time: 0.0484
> fsync time: 0.0328
> fsync time: 0.0562
> fsync time: 0.0451
> fsync time: 0.0576
> fsync time: 0.0444
> fsync time: 0.0469
> fsync time: 0.0368
> fsync time: 0.2865
> 
> Corrado
>