Re: [POC][PATCH] xfs: reduce ilock contention on buffered randrw workload

[Amir Goldstein <amir73il@gmail.com>]

> > I am going to go find a machine with slow disk to test the random rw
> > workload again on both xfs and ext4 pre and post invalidate_lock and
> > to try out the pre-warm page cache solution.
> >
> > The results could be:
> > a) ext4 random rw performance has been degraded by invalidate_lock
> > b) pre-warm page cache before taking IOLOCK is going to improve
> >     xfs random rw performance
> > c) A little bit of both

The correct answer is b. :)

>
> Well, numbers always beat the theory so I'm all for measuring it but let me
> say our kernel performance testing within SUSE didn't show significant hit
> being introduced by invalidate_lock for any major filesystem.
>

Here are the numbers produced on v5.10.109, on v5.19-rc3
and on v5.19-rc3+ which includes the pre-warn test patch [1].

The numbers are produced by a filebench workload [2] that runs
8 random reader threads and 8 random writer threads for 60 seconds
on a cold cache preallocated 5GB file.

Note that the machine I tested with has much faster storage than
the one that was used 3 years ago, but the performance impact
of IOLOCK is still very clear, even larger in this test.

If there are no other objections to the pre-warm concept,
I will go on to write and test a proper patch.

Thanks,
Amir.

[1] https://github.com/amir73il/linux/commit/70e94f3471739c442b1110ee46e8b59e5d5f5042
[2] https://github.com/amir73il/filebench/blob/overlayfs-devel/workloads/randomrw.f

--- EXT4 5.10 ---
 filebench randomrw (8 read threads, 8 write threads)
 kernel 5.10.109, ext4

Test #1:
rand-write1          3002127ops    50020ops/s 390.8mb/s    0.156ms/op
[0.002ms - 213.755ms]
rand-read1           31749234ops   528988ops/s 4132.7mb/s
0.010ms/op [0.001ms - 68.884ms]

Test #2:
rand-write1          3083679ops    51381ops/s 401.4mb/s    0.152ms/op
[0.002ms - 181.368ms]
rand-read1           32182118ops   536228ops/s 4189.3mb/s
0.010ms/op [0.001ms - 61.158ms]

--- EXT4 5.19 ---
 filebench randomrw (8 read threads, 8 write threads)
 kernel 5.19-rc3, ext4

Test #1:
rand-write1          2829917ops    47159ops/s 368.4mb/s    0.160ms/op
[0.002ms - 4709.167ms]
rand-read1           36997540ops   616542ops/s 4816.7mb/s
0.009ms/op [0.001ms - 4704.105ms]

Test #2:
rand-write1          2764486ops    46067ops/s 359.9mb/s    0.170ms/op
[0.002ms - 5042.597ms]
rand-read1           38893279ops   648118ops/s 5063.4mb/s
0.008ms/op [0.001ms - 5004.069ms]

--- XFS 5.10 ---
 filebench randomrw (8 read threads, 8 write threads)
 kernel 5.10.109, xfs

Test #1:
rand-write1          1049278ops    17485ops/s 136.6mb/s    0.456ms/op
[0.002ms - 224.062ms]
rand-read1           33325ops      555ops/s   4.3mb/s   14.392ms/op
[0.007ms - 224.833ms]

Test #2:
rand-write1          1127497ops    18788ops/s 146.8mb/s    0.424ms/op
[0.003ms - 445.810ms]
rand-read1           35341ops      589ops/s   4.6mb/s   13.566ms/op
[0.005ms - 445.529ms]

--- XFS 5.19 ---
 filebench randomrw (8 read threads, 8 write threads)
 kernel 5.19-rc3, xfs

Test #1:
rand-write1          3295934ops    54920ops/s 429.1mb/s    0.144ms/op
[0.003ms - 109.703ms]
rand-read1           86768ops     1446ops/s  11.3mb/s    5.520ms/op
[0.003ms - 372.000ms]

Test #2:
rand-write1          3246935ops    54106ops/s 422.7mb/s    0.146ms/op
[0.002ms - 103.505ms]
rand-read1           167018ops     2783ops/s  21.7mb/s    2.867ms/op
[0.003ms - 101.105ms]

--- XFS+ 5.19 ---
 filebench randomrw (8 read threads, 8 write threads)
 kernel 5.19-rc3+ (xfs page cache warmup patch)

Test #1:
rand-write1          3054567ops    50899ops/s 397.6mb/s    0.154ms/op
[0.002ms - 201.531ms]
rand-read1           38107333ops   634990ops/s 4960.9mb/s
0.008ms/op [0.001ms - 60.027ms]

Test #2:
rand-write1          2704416ops    45053ops/s 352.0mb/s    0.174ms/op
[0.002ms - 287.079ms]
rand-read1           38589737ops   642874ops/s 5022.4mb/s
0.008ms/op [0.001ms - 60.741ms]