* Some performance numbers for virtiofs, DAX and virtio-9p
@ 2020-12-10 16:11 ` Vivek Goyal
0 siblings, 0 replies; 20+ messages in thread
From: Vivek Goyal @ 2020-12-10 16:11 UTC (permalink / raw)
To: virtio-fs-list, qemu-devel
Cc: Miklos Szeredi, jose.carlos.venegas.munoz, Christian Schoenebeck,
Dr. David Alan Gilbert, Stefan Hajnoczi
Hi,
Now virtiofs DAX support has been merged in upstream kernel 5.10 (It
has yet to be merged qemu). So I decided to do some performance
benchmarking and see what's the impact of DAX on different workloads in
different caching modes. Hence I ran some performance numbers and
publishing the results here.
I was running these numbers for virtiofs, so thought of why not throw
virtio-9p also into the mix to get and idea how these two compare.
I have pushed logfiles and final test results and test-setup information
in virtiofs-tests git tree.
https://github.com/rhvgoyal/virtiofs-tests/tree/master/performance-results/dec-10-2020
Methodology
===========
I have basically run bunch of fio jobs to get a sense of speed of
various operations. I wrote a simple wrapper script to run fio jobs
3 times and take their average and report it. These scripts and fio
jobs are available here.
https://github.com/rhvgoyal/virtiofs-tests
I have an SSD on host where I setup a test directory and exported that
directory inside guest using virtiofs and virtio-9p and ran tests inside
guests in different configurations.
Used caching modes cache=none, cache=auto and cache=always for virtiofs
and cache=none, cache=mmap and cache=loose for virtio-9p. For virtiofs,
I also rand all caching test with DAX too to evaluate the impact of DAX.
Test Setup
-----------
- A fedora 29 host with 376Gi RAM, 2 sockets (20 cores per socket, 2
threads per core)
- Using SSD on host as backing store. 4 fio files of 4G each.
- Created a VM with 32 VCPUS and 64GB memory. Used a 16GB cache window for dax.
- For mounting virtio-9p, I used msize=16MB.
- For guest kernel, I used Miklos's for-next branch of fuse tree. It has
killpriv_v2 support patches which should go upstream in 5.11.
https://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse.git/log/?h=for-next
- For qemu, I used following qemu branch.
https://github.com/rhvgoyal/qemu/commits/dax-killpriv-thread-pool
Bunch of patches like DAX support, killpriv_v2 support, thread-pool-0 support
are not upstream yet. So applied these patches on this branch and used
for testing.
Also, carried a patch which improves virtio-9p performance to not use
coroutines. Hopefully this will make upstream someday.
virtiofsd command line
----------------------
- Used following options for virtiofsd.
./tools/virtiofsd/virtiofsd --socket-path=/tmp/vhostqemu -o source=/mnt/sdb/virtiofs-source -o no_posix_lock -o xattr -o cache=none --thread-pool-size=0 --daemonize
Test Results
------------
- Results in nine configurations have been reported.
virtio-fs: cache=none, cache=auto, cache=always,
cache=none + DAX, cache=auto + DAX, cache=always + DAX
virtio-9p: cache=none, cache=mmap, cache=loose
- Three fio ioengines psync, libaio and mmap have been used.
- I/O Workload of randread, radwrite, seqread and seqwrite have been run.
- Each file size is 4G. Block size 4K. iodepth=16
- "multi" means same operation was done with 4 jobs and each job is
operating on a file of size 4G.
- Some results are "0 (KiB/s)". That means that particular operation is
not supported in that configuration.
Conclusion
-----------
- virtiofs DAX seems to help a lot in many workloads.
Note, DAX performance well only if data fits in cache window. My total
data is 16G and cache window size is 16G as well. If data is larger
than DAX cache window, then performance of dax suffers a lot. Overhead
of reclaiming old mapping and setting up a new one is very high.
NAME WORKLOAD Bandwidth IOPS
9p-none seqread-psync 98.6mb 24.6k
9p-mmap seqread-psync 97.5mb 24.3k
9p-loose seqread-psync 91.6mb 22.9k
vtfs-none seqread-psync 98.4mb 24.6k
vtfs-none-dax seqread-psync 660.3mb 165.0k
vtfs-auto seqread-psync 650.0mb 162.5k
vtfs-auto-dax seqread-psync 703.1mb 175.7k
vtfs-always seqread-psync 671.3mb 167.8k
vtfs-always-dax seqread-psync 687.2mb 171.8k
9p-none seqread-psync-multi 397.6mb 99.4k
9p-mmap seqread-psync-multi 382.7mb 95.6k
9p-loose seqread-psync-multi 350.5mb 87.6k
vtfs-none seqread-psync-multi 360.0mb 90.0k
vtfs-none-dax seqread-psync-multi 2281.1mb 570.2k
vtfs-auto seqread-psync-multi 2530.7mb 632.6k
vtfs-auto-dax seqread-psync-multi 2423.9mb 605.9k
vtfs-always seqread-psync-multi 2535.7mb 633.9k
vtfs-always-dax seqread-psync-multi 2406.1mb 601.5k
9p-none seqread-mmap 64.3mb 16.0k
9p-mmap seqread-mmap 91.2mb 22.8k
9p-loose seqread-mmap 88.4mb 22.1k
vtfs-none seqread-mmap 0kb 0
vtfs-none-dax seqread-mmap 915.0mb 228.7k
vtfs-auto seqread-mmap 570.6mb 142.6k
vtfs-auto-dax seqread-mmap 916.9mb 229.2k
vtfs-always seqread-mmap 585.0mb 146.2k
vtfs-always-dax seqread-mmap 889.5mb 222.3k
9p-none seqread-mmap-multi 256.6mb 64.1k
9p-mmap seqread-mmap-multi 333.2mb 83.3k
9p-loose seqread-mmap-multi 352.3mb 88.0k
vtfs-none seqread-mmap-multi 0kb 0
vtfs-none-dax seqread-mmap-multi 2848.7mb 712.1k
vtfs-auto seqread-mmap-multi 2210.1mb 552.5k
vtfs-auto-dax seqread-mmap-multi 3106.7mb 776.6k
vtfs-always seqread-mmap-multi 2218.8mb 554.7k
vtfs-always-dax seqread-mmap-multi 3084.3mb 771.0k
9p-none seqread-libaio 93.1mb 23.2k
9p-mmap seqread-libaio 97.7mb 24.4k
9p-loose seqread-libaio 89.6mb 22.4k
vtfs-none seqread-libaio 259.3mb 64.8k
vtfs-none-dax seqread-libaio 406.3mb 101.5k
vtfs-auto seqread-libaio 303.3mb 75.8k
vtfs-auto-dax seqread-libaio 400.2mb 100.0k
vtfs-always seqread-libaio 274.7mb 68.6k
vtfs-always-dax seqread-libaio 415.4mb 103.8k
9p-none seqread-libaio-multi 349.4mb 87.3k
9p-mmap seqread-libaio-multi 353.5mb 88.3k
9p-loose seqread-libaio-multi 330.1mb 82.5k
vtfs-none seqread-libaio-multi 329.0mb 82.2k
vtfs-none-dax seqread-libaio-multi 1472.0mb 368.0k
vtfs-auto seqread-libaio-multi 1500.0mb 375.0k
vtfs-auto-dax seqread-libaio-multi 1491.2mb 372.8k
vtfs-always seqread-libaio-multi 1485.2mb 371.3k
vtfs-always-dax seqread-libaio-multi 1494.1mb 373.5k
9p-none randread-psync 101.5mb 25.3k
9p-mmap randread-psync 102.8mb 25.7k
9p-loose randread-psync 83.2mb 20.8k
vtfs-none randread-psync 104.9mb 26.2k
vtfs-none-dax randread-psync 662.5mb 165.6k
vtfs-auto randread-psync 77.9mb 19.4k
vtfs-auto-dax randread-psync 586.6mb 146.6k
vtfs-always randread-psync 79.3mb 19.8k
vtfs-always-dax randread-psync 634.1mb 158.5k
9p-none randread-psync-multi 383.3mb 95.8k
9p-mmap randread-psync-multi 372.6mb 93.1k
9p-loose randread-psync-multi 334.5mb 83.6k
vtfs-none randread-psync-multi 351.0mb 87.7k
vtfs-none-dax randread-psync-multi 519.9mb 129.9k
vtfs-auto randread-psync-multi 285.2mb 71.3k
vtfs-auto-dax randread-psync-multi 517.1mb 129.2k
vtfs-always randread-psync-multi 289.6mb 72.4k
vtfs-always-dax randread-psync-multi 507.9mb 126.9k
9p-none randread-mmap 68.5mb 17.1k
9p-mmap randread-mmap 64.4mb 16.1k
9p-loose randread-mmap 65.2mb 16.3k
vtfs-none randread-mmap 0kb 0
vtfs-none-dax randread-mmap 878.8mb 219.7k
vtfs-auto randread-mmap 57.7mb 14.4k
vtfs-auto-dax randread-mmap 870.6mb 217.6k
vtfs-always randread-mmap 58.5mb 14.6k
vtfs-always-dax randread-mmap 892.2mb 223.0k
9p-none randread-mmap-multi 248.0mb 62.0k
9p-mmap randread-mmap-multi 247.4mb 61.8k
9p-loose randread-mmap-multi 245.4mb 61.3k
vtfs-none randread-mmap-multi 0kb 0
vtfs-none-dax randread-mmap-multi 1055.3mb 263.8k
vtfs-auto randread-mmap-multi 243.5mb 60.8k
vtfs-auto-dax randread-mmap-multi 1030.9mb 257.7k
vtfs-always randread-mmap-multi 240.4mb 60.1k
vtfs-always-dax randread-mmap-multi 1052.0mb 263.0k
9p-none randread-libaio 91.0mb 22.7k
9p-mmap randread-libaio 92.1mb 23.0k
9p-loose randread-libaio 91.7mb 22.9k
vtfs-none randread-libaio 266.4mb 66.6k
vtfs-none-dax randread-libaio 387.2mb 96.8k
vtfs-auto randread-libaio 276.2mb 69.0k
vtfs-auto-dax randread-libaio 379.8mb 94.9k
vtfs-always randread-libaio 275.0mb 68.7k
vtfs-always-dax randread-libaio 399.5mb 99.8k
9p-none randread-libaio-multi 354.5mb 88.6k
9p-mmap randread-libaio-multi 359.5mb 89.8k
9p-loose randread-libaio-multi 348.5mb 87.1k
vtfs-none randread-libaio-multi 278.6mb 69.6k
vtfs-none-dax randread-libaio-multi 374.4mb 93.6k
vtfs-auto randread-libaio-multi 271.0mb 67.7k
vtfs-auto-dax randread-libaio-multi 367.0mb 91.7k
vtfs-always randread-libaio-multi 278.4mb 69.6k
vtfs-always-dax randread-libaio-multi 360.6mb 90.1k
9p-none seqwrite-psync 93.2mb 23.3k
9p-mmap seqwrite-psync 95.8mb 23.9k
9p-loose seqwrite-psync 69.3mb 17.3k
vtfs-none seqwrite-psync 95.3mb 23.8k
vtfs-none-dax seqwrite-psync 486.5mb 121.6k
vtfs-auto seqwrite-psync 78.6mb 19.6k
vtfs-auto-dax seqwrite-psync 470.9mb 117.7k
vtfs-always seqwrite-psync 75.9mb 18.9k
vtfs-always-dax seqwrite-psync 491.4mb 122.8k
9p-none seqwrite-psync-multi 339.9mb 84.9k
9p-mmap seqwrite-psync-multi 343.6mb 85.9k
9p-loose seqwrite-psync-multi 290.5mb 72.6k
vtfs-none seqwrite-psync-multi 321.3mb 80.3k
vtfs-none-dax seqwrite-psync-multi 1283.5mb 320.8k
vtfs-auto seqwrite-psync-multi 305.7mb 76.4k
vtfs-auto-dax seqwrite-psync-multi 1319.5mb 329.8k
vtfs-always seqwrite-psync-multi 307.5mb 76.8k
vtfs-always-dax seqwrite-psync-multi 1200.8mb 300.2k
9p-none seqwrite-mmap 0kb 0
9p-mmap seqwrite-mmap 101.3mb 25.3k
9p-loose seqwrite-mmap 112.4mb 28.1k
vtfs-none seqwrite-mmap 0kb 0
vtfs-none-dax seqwrite-mmap 1297.2mb 324.3k
vtfs-auto seqwrite-mmap 199.7mb 49.9k
vtfs-auto-dax seqwrite-mmap 1323.0mb 330.7k
vtfs-always seqwrite-mmap 181.7mb 45.4k
vtfs-always-dax seqwrite-mmap 1353.7mb 338.4k
9p-none seqwrite-mmap-multi 0kb 0
9p-mmap seqwrite-mmap-multi 274.7mb 68.6k
9p-loose seqwrite-mmap-multi 281.3mb 70.3k
vtfs-none seqwrite-mmap-multi 0kb 0
vtfs-none-dax seqwrite-mmap-multi 1806.2mb 451.5k
vtfs-auto seqwrite-mmap-multi 384.1mb 96.0k
vtfs-auto-dax seqwrite-mmap-multi 2023.4mb 505.8k
vtfs-always seqwrite-mmap-multi 384.5mb 96.1k
vtfs-always-dax seqwrite-mmap-multi 2008.9mb 502.2k
9p-none seqwrite-libaio 87.9mb 21.9k
9p-mmap seqwrite-libaio 89.3mb 22.3k
9p-loose seqwrite-libaio 42.4mb 10.5k
vtfs-none seqwrite-libaio 254.5mb 63.6k
vtfs-none-dax seqwrite-libaio 284.7mb 71.1k
vtfs-auto seqwrite-libaio 279.1mb 69.7k
vtfs-auto-dax seqwrite-libaio 299.6mb 74.9k
vtfs-always seqwrite-libaio 238.9mb 59.7k
vtfs-always-dax seqwrite-libaio 305.8mb 76.4k
9p-none seqwrite-libaio-multi 339.6mb 84.9k
9p-mmap seqwrite-libaio-multi 329.5mb 82.3k
9p-loose seqwrite-libaio-multi 165.8mb 41.4k
vtfs-none seqwrite-libaio-multi 320.6mb 80.1k
vtfs-none-dax seqwrite-libaio-multi 975.3mb 243.8k
vtfs-auto seqwrite-libaio-multi 306.4mb 76.6k
vtfs-auto-dax seqwrite-libaio-multi 974.4mb 243.6k
vtfs-always seqwrite-libaio-multi 312.6mb 78.1k
vtfs-always-dax seqwrite-libaio-multi 883.8mb 220.9k
9p-none randwrite-psync 94.9mb 23.7k
9p-mmap randwrite-psync 99.3mb 24.8k
9p-loose randwrite-psync 68.7mb 17.1k
vtfs-none randwrite-psync 97.2mb 24.3k
vtfs-none-dax randwrite-psync 480.3mb 120.0k
vtfs-auto randwrite-psync 82.2mb 20.5k
vtfs-auto-dax randwrite-psync 435.6mb 108.9k
vtfs-always randwrite-psync 77.3mb 19.3k
vtfs-always-dax randwrite-psync 451.1mb 112.7k
9p-none randwrite-psync-multi 336.0mb 84.0k
9p-mmap randwrite-psync-multi 338.5mb 84.6k
9p-loose randwrite-psync-multi 278.4mb 69.6k
vtfs-none randwrite-psync-multi 323.7mb 80.9k
vtfs-none-dax randwrite-psync-multi 519.2mb 129.8k
vtfs-auto randwrite-psync-multi 302.1mb 75.5k
vtfs-auto-dax randwrite-psync-multi 473.3mb 118.3k
vtfs-always randwrite-psync-multi 302.4mb 75.6k
vtfs-always-dax randwrite-psync-multi 512.9mb 128.2k
9p-none randwrite-mmap 0kb 0
9p-mmap randwrite-mmap 58.2mb 14.5k
9p-loose randwrite-mmap 63.5mb 15.8k
vtfs-none randwrite-mmap 0kb 0
vtfs-none-dax randwrite-mmap 872.3mb 218.0k
vtfs-auto randwrite-mmap 55.9mb 13.9k
vtfs-auto-dax randwrite-mmap 771.8mb 192.9k
vtfs-always randwrite-mmap 59.0mb 14.7k
vtfs-always-dax randwrite-mmap 828.8mb 207.2k
9p-none randwrite-mmap-multi 0kb 0
9p-mmap randwrite-mmap-multi 251.2mb 62.8k
9p-loose randwrite-mmap-multi 257.9mb 64.4k
vtfs-none randwrite-mmap-multi 0kb 0
vtfs-none-dax randwrite-mmap-multi 1119.1mb 279.7k
vtfs-auto randwrite-mmap-multi 208.9mb 52.2k
vtfs-auto-dax randwrite-mmap-multi 1043.9mb 260.9k
vtfs-always randwrite-mmap-multi 209.4mb 52.3k
vtfs-always-dax randwrite-mmap-multi 1091.4mb 272.8k
9p-none randwrite-libaio 85.9mb 21.4k
9p-mmap randwrite-libaio 87.8mb 21.9k
9p-loose randwrite-libaio 41.6mb 10.4k
vtfs-none randwrite-libaio 252.4mb 63.1k
vtfs-none-dax randwrite-libaio 282.0mb 70.5k
vtfs-auto randwrite-libaio 258.7mb 64.6k
vtfs-auto-dax randwrite-libaio 272.5mb 68.1k
vtfs-always randwrite-libaio 239.0mb 59.7k
vtfs-always-dax randwrite-libaio 256.4mb 64.1k
9p-none randwrite-libaio-multi 329.5mb 82.3k
9p-mmap randwrite-libaio-multi 324.4mb 81.1k
9p-loose randwrite-libaio-multi 161.1mb 40.2k
vtfs-none randwrite-libaio-multi 323.8mb 80.9k
vtfs-none-dax randwrite-libaio-multi 366.3mb 91.5k
vtfs-auto randwrite-libaio-multi 320.9mb 80.2k
vtfs-auto-dax randwrite-libaio-multi 351.2mb 87.8k
vtfs-always randwrite-libaio-multi 328.8mb 82.2k
vtfs-always-dax randwrite-libaio-multi 362.5mb 90.6k
9p-none randrw-psync 70.2mb/23.5mb 17.5k/6017
9p-mmap randrw-psync 74.5mb/24.9mb 18.6k/6398
9p-loose randrw-psync 59.2mb/19.8mb 14.8k/5077
vtfs-none randrw-psync 75.8mb/25.4mb 18.9k/6508
vtfs-none-dax randrw-psync 403.2mb/134.7mb 100.8k/33.6k
vtfs-auto randrw-psync 38.1mb/12.7mb 9775/3265
vtfs-auto-dax randrw-psync 382.4mb/127.8mb 95.6k/31.9k
vtfs-always randrw-psync 40.7mb/13.6mb 10.1k/3485
vtfs-always-dax randrw-psync 393.4mb/131.4mb 98.3k/32.8k
9p-none randrw-psync-multi 264.3mb/88.5mb 66.0k/22.1k
9p-mmap randrw-psync-multi 260.5mb/87.2mb 65.1k/21.8k
9p-loose randrw-psync-multi 232.6mb/77.9mb 58.1k/19.4k
vtfs-none randrw-psync-multi 260.7mb/87.3mb 65.1k/21.8k
vtfs-none-dax randrw-psync-multi 270.0mb/90.4mb 67.5k/22.6k
vtfs-auto randrw-psync-multi 151.4mb/50.7mb 37.8k/12.6k
vtfs-auto-dax randrw-psync-multi 251.2mb/84.1mb 62.8k/21.0k
vtfs-always randrw-psync-multi 152.9mb/51.2mb 38.2k/12.8k
vtfs-always-dax randrw-psync-multi 251.5mb/84.3mb 62.8k/21.0k
9p-none randrw-mmap 0kb/0kb 0/0
9p-mmap randrw-mmap 47.4mb/15.8mb 11.8k/4057
9p-loose randrw-mmap 48.4mb/16.1mb 12.1k/4143
vtfs-none randrw-mmap 0kb/0kb 0/0
vtfs-none-dax randrw-mmap 539.5mb/180.3mb 134.8k/45.0k
vtfs-auto randrw-mmap 44.4mb/14.8mb 11.1k/3806
vtfs-auto-dax randrw-mmap 544.2mb/181.8mb 136.0k/45.4k
vtfs-always randrw-mmap 43.1mb/14.4mb 10.7k/3691
vtfs-always-dax randrw-mmap 555.4mb/185.6mb 138.8k/46.4k
9p-none randrw-mmap-multi 0kb/0kb 0/0
9p-mmap randrw-mmap-multi 184.0mb/61.6mb 46.0k/15.4k
9p-loose randrw-mmap-multi 188.8mb/63.2mb 47.2k/15.8k
vtfs-none randrw-mmap-multi 0kb/0kb 0/0
vtfs-none-dax randrw-mmap-multi 573.8mb/191.9mb 143.4k/47.9k
vtfs-auto randrw-mmap-multi 179.3mb/60.0mb 44.8k/15.0k
vtfs-auto-dax randrw-mmap-multi 498.0mb/166.7mb 124.5k/41.6k
vtfs-always randrw-mmap-multi 180.2mb/60.4mb 45.0k/15.1k
vtfs-always-dax randrw-mmap-multi 525.0mb/175.7mb 131.2k/43.9k
9p-none randrw-libaio 68.4mb/22.8mb 17.1k/5858
9p-mmap randrw-libaio 67.4mb/22.5mb 16.8k/5777
9p-loose randrw-libaio 52.7mb/17.6mb 13.1k/4517
vtfs-none randrw-libaio 199.9mb/66.8mb 49.9k/16.7k
vtfs-none-dax randrw-libaio 256.9mb/85.8mb 64.2k/21.4k
vtfs-auto randrw-libaio 83.4mb/27.9mb 20.8k/7161
vtfs-auto-dax randrw-libaio 241.2mb/80.6mb 60.3k/20.1k
vtfs-always randrw-libaio 87.3mb/29.2mb 21.8k/7491
vtfs-always-dax randrw-libaio 252.5mb/84.3mb 63.1k/21.0k
9p-none randrw-libaio-multi 267.4mb/89.5mb 66.8k/22.3k
9p-mmap randrw-libaio-multi 256.2mb/85.8mb 64.0k/21.4k
9p-loose randrw-libaio-multi 200.1mb/67.0mb 50.0k/16.7k
vtfs-none randrw-libaio-multi 210.9mb/70.7mb 52.7k/17.6k
vtfs-none-dax randrw-libaio-multi 192.6mb/64.5mb 48.1k/16.1k
vtfs-auto randrw-libaio-multi 209.6mb/70.2mb 52.4k/17.5k
vtfs-auto-dax randrw-libaio-multi 189.1mb/63.3mb 47.2k/15.8k
vtfs-always randrw-libaio-multi 212.3mb/71.1mb 53.0k/17.7k
vtfs-always-dax randrw-libaio-multi 195.8mb/65.6mb 48.9k/16.4k
Thanks
Vivek
^ permalink raw reply [flat|nested] 20+ messages in thread
* [Virtio-fs] Some performance numbers for virtiofs, DAX and virtio-9p
@ 2020-12-10 16:11 ` Vivek Goyal
0 siblings, 0 replies; 20+ messages in thread
From: Vivek Goyal @ 2020-12-10 16:11 UTC (permalink / raw)
To: virtio-fs-list, qemu-devel
Cc: jose.carlos.venegas.munoz, Christian Schoenebeck
Hi,
Now virtiofs DAX support has been merged in upstream kernel 5.10 (It
has yet to be merged qemu). So I decided to do some performance
benchmarking and see what's the impact of DAX on different workloads in
different caching modes. Hence I ran some performance numbers and
publishing the results here.
I was running these numbers for virtiofs, so thought of why not throw
virtio-9p also into the mix to get and idea how these two compare.
I have pushed logfiles and final test results and test-setup information
in virtiofs-tests git tree.
https://github.com/rhvgoyal/virtiofs-tests/tree/master/performance-results/dec-10-2020
Methodology
===========
I have basically run bunch of fio jobs to get a sense of speed of
various operations. I wrote a simple wrapper script to run fio jobs
3 times and take their average and report it. These scripts and fio
jobs are available here.
https://github.com/rhvgoyal/virtiofs-tests
I have an SSD on host where I setup a test directory and exported that
directory inside guest using virtiofs and virtio-9p and ran tests inside
guests in different configurations.
Used caching modes cache=none, cache=auto and cache=always for virtiofs
and cache=none, cache=mmap and cache=loose for virtio-9p. For virtiofs,
I also rand all caching test with DAX too to evaluate the impact of DAX.
Test Setup
-----------
- A fedora 29 host with 376Gi RAM, 2 sockets (20 cores per socket, 2
threads per core)
- Using SSD on host as backing store. 4 fio files of 4G each.
- Created a VM with 32 VCPUS and 64GB memory. Used a 16GB cache window for dax.
- For mounting virtio-9p, I used msize=16MB.
- For guest kernel, I used Miklos's for-next branch of fuse tree. It has
killpriv_v2 support patches which should go upstream in 5.11.
https://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse.git/log/?h=for-next
- For qemu, I used following qemu branch.
https://github.com/rhvgoyal/qemu/commits/dax-killpriv-thread-pool
Bunch of patches like DAX support, killpriv_v2 support, thread-pool-0 support
are not upstream yet. So applied these patches on this branch and used
for testing.
Also, carried a patch which improves virtio-9p performance to not use
coroutines. Hopefully this will make upstream someday.
virtiofsd command line
----------------------
- Used following options for virtiofsd.
./tools/virtiofsd/virtiofsd --socket-path=/tmp/vhostqemu -o source=/mnt/sdb/virtiofs-source -o no_posix_lock -o xattr -o cache=none --thread-pool-size=0 --daemonize
Test Results
------------
- Results in nine configurations have been reported.
virtio-fs: cache=none, cache=auto, cache=always,
cache=none + DAX, cache=auto + DAX, cache=always + DAX
virtio-9p: cache=none, cache=mmap, cache=loose
- Three fio ioengines psync, libaio and mmap have been used.
- I/O Workload of randread, radwrite, seqread and seqwrite have been run.
- Each file size is 4G. Block size 4K. iodepth=16
- "multi" means same operation was done with 4 jobs and each job is
operating on a file of size 4G.
- Some results are "0 (KiB/s)". That means that particular operation is
not supported in that configuration.
Conclusion
-----------
- virtiofs DAX seems to help a lot in many workloads.
Note, DAX performance well only if data fits in cache window. My total
data is 16G and cache window size is 16G as well. If data is larger
than DAX cache window, then performance of dax suffers a lot. Overhead
of reclaiming old mapping and setting up a new one is very high.
NAME WORKLOAD Bandwidth IOPS
9p-none seqread-psync 98.6mb 24.6k
9p-mmap seqread-psync 97.5mb 24.3k
9p-loose seqread-psync 91.6mb 22.9k
vtfs-none seqread-psync 98.4mb 24.6k
vtfs-none-dax seqread-psync 660.3mb 165.0k
vtfs-auto seqread-psync 650.0mb 162.5k
vtfs-auto-dax seqread-psync 703.1mb 175.7k
vtfs-always seqread-psync 671.3mb 167.8k
vtfs-always-dax seqread-psync 687.2mb 171.8k
9p-none seqread-psync-multi 397.6mb 99.4k
9p-mmap seqread-psync-multi 382.7mb 95.6k
9p-loose seqread-psync-multi 350.5mb 87.6k
vtfs-none seqread-psync-multi 360.0mb 90.0k
vtfs-none-dax seqread-psync-multi 2281.1mb 570.2k
vtfs-auto seqread-psync-multi 2530.7mb 632.6k
vtfs-auto-dax seqread-psync-multi 2423.9mb 605.9k
vtfs-always seqread-psync-multi 2535.7mb 633.9k
vtfs-always-dax seqread-psync-multi 2406.1mb 601.5k
9p-none seqread-mmap 64.3mb 16.0k
9p-mmap seqread-mmap 91.2mb 22.8k
9p-loose seqread-mmap 88.4mb 22.1k
vtfs-none seqread-mmap 0kb 0
vtfs-none-dax seqread-mmap 915.0mb 228.7k
vtfs-auto seqread-mmap 570.6mb 142.6k
vtfs-auto-dax seqread-mmap 916.9mb 229.2k
vtfs-always seqread-mmap 585.0mb 146.2k
vtfs-always-dax seqread-mmap 889.5mb 222.3k
9p-none seqread-mmap-multi 256.6mb 64.1k
9p-mmap seqread-mmap-multi 333.2mb 83.3k
9p-loose seqread-mmap-multi 352.3mb 88.0k
vtfs-none seqread-mmap-multi 0kb 0
vtfs-none-dax seqread-mmap-multi 2848.7mb 712.1k
vtfs-auto seqread-mmap-multi 2210.1mb 552.5k
vtfs-auto-dax seqread-mmap-multi 3106.7mb 776.6k
vtfs-always seqread-mmap-multi 2218.8mb 554.7k
vtfs-always-dax seqread-mmap-multi 3084.3mb 771.0k
9p-none seqread-libaio 93.1mb 23.2k
9p-mmap seqread-libaio 97.7mb 24.4k
9p-loose seqread-libaio 89.6mb 22.4k
vtfs-none seqread-libaio 259.3mb 64.8k
vtfs-none-dax seqread-libaio 406.3mb 101.5k
vtfs-auto seqread-libaio 303.3mb 75.8k
vtfs-auto-dax seqread-libaio 400.2mb 100.0k
vtfs-always seqread-libaio 274.7mb 68.6k
vtfs-always-dax seqread-libaio 415.4mb 103.8k
9p-none seqread-libaio-multi 349.4mb 87.3k
9p-mmap seqread-libaio-multi 353.5mb 88.3k
9p-loose seqread-libaio-multi 330.1mb 82.5k
vtfs-none seqread-libaio-multi 329.0mb 82.2k
vtfs-none-dax seqread-libaio-multi 1472.0mb 368.0k
vtfs-auto seqread-libaio-multi 1500.0mb 375.0k
vtfs-auto-dax seqread-libaio-multi 1491.2mb 372.8k
vtfs-always seqread-libaio-multi 1485.2mb 371.3k
vtfs-always-dax seqread-libaio-multi 1494.1mb 373.5k
9p-none randread-psync 101.5mb 25.3k
9p-mmap randread-psync 102.8mb 25.7k
9p-loose randread-psync 83.2mb 20.8k
vtfs-none randread-psync 104.9mb 26.2k
vtfs-none-dax randread-psync 662.5mb 165.6k
vtfs-auto randread-psync 77.9mb 19.4k
vtfs-auto-dax randread-psync 586.6mb 146.6k
vtfs-always randread-psync 79.3mb 19.8k
vtfs-always-dax randread-psync 634.1mb 158.5k
9p-none randread-psync-multi 383.3mb 95.8k
9p-mmap randread-psync-multi 372.6mb 93.1k
9p-loose randread-psync-multi 334.5mb 83.6k
vtfs-none randread-psync-multi 351.0mb 87.7k
vtfs-none-dax randread-psync-multi 519.9mb 129.9k
vtfs-auto randread-psync-multi 285.2mb 71.3k
vtfs-auto-dax randread-psync-multi 517.1mb 129.2k
vtfs-always randread-psync-multi 289.6mb 72.4k
vtfs-always-dax randread-psync-multi 507.9mb 126.9k
9p-none randread-mmap 68.5mb 17.1k
9p-mmap randread-mmap 64.4mb 16.1k
9p-loose randread-mmap 65.2mb 16.3k
vtfs-none randread-mmap 0kb 0
vtfs-none-dax randread-mmap 878.8mb 219.7k
vtfs-auto randread-mmap 57.7mb 14.4k
vtfs-auto-dax randread-mmap 870.6mb 217.6k
vtfs-always randread-mmap 58.5mb 14.6k
vtfs-always-dax randread-mmap 892.2mb 223.0k
9p-none randread-mmap-multi 248.0mb 62.0k
9p-mmap randread-mmap-multi 247.4mb 61.8k
9p-loose randread-mmap-multi 245.4mb 61.3k
vtfs-none randread-mmap-multi 0kb 0
vtfs-none-dax randread-mmap-multi 1055.3mb 263.8k
vtfs-auto randread-mmap-multi 243.5mb 60.8k
vtfs-auto-dax randread-mmap-multi 1030.9mb 257.7k
vtfs-always randread-mmap-multi 240.4mb 60.1k
vtfs-always-dax randread-mmap-multi 1052.0mb 263.0k
9p-none randread-libaio 91.0mb 22.7k
9p-mmap randread-libaio 92.1mb 23.0k
9p-loose randread-libaio 91.7mb 22.9k
vtfs-none randread-libaio 266.4mb 66.6k
vtfs-none-dax randread-libaio 387.2mb 96.8k
vtfs-auto randread-libaio 276.2mb 69.0k
vtfs-auto-dax randread-libaio 379.8mb 94.9k
vtfs-always randread-libaio 275.0mb 68.7k
vtfs-always-dax randread-libaio 399.5mb 99.8k
9p-none randread-libaio-multi 354.5mb 88.6k
9p-mmap randread-libaio-multi 359.5mb 89.8k
9p-loose randread-libaio-multi 348.5mb 87.1k
vtfs-none randread-libaio-multi 278.6mb 69.6k
vtfs-none-dax randread-libaio-multi 374.4mb 93.6k
vtfs-auto randread-libaio-multi 271.0mb 67.7k
vtfs-auto-dax randread-libaio-multi 367.0mb 91.7k
vtfs-always randread-libaio-multi 278.4mb 69.6k
vtfs-always-dax randread-libaio-multi 360.6mb 90.1k
9p-none seqwrite-psync 93.2mb 23.3k
9p-mmap seqwrite-psync 95.8mb 23.9k
9p-loose seqwrite-psync 69.3mb 17.3k
vtfs-none seqwrite-psync 95.3mb 23.8k
vtfs-none-dax seqwrite-psync 486.5mb 121.6k
vtfs-auto seqwrite-psync 78.6mb 19.6k
vtfs-auto-dax seqwrite-psync 470.9mb 117.7k
vtfs-always seqwrite-psync 75.9mb 18.9k
vtfs-always-dax seqwrite-psync 491.4mb 122.8k
9p-none seqwrite-psync-multi 339.9mb 84.9k
9p-mmap seqwrite-psync-multi 343.6mb 85.9k
9p-loose seqwrite-psync-multi 290.5mb 72.6k
vtfs-none seqwrite-psync-multi 321.3mb 80.3k
vtfs-none-dax seqwrite-psync-multi 1283.5mb 320.8k
vtfs-auto seqwrite-psync-multi 305.7mb 76.4k
vtfs-auto-dax seqwrite-psync-multi 1319.5mb 329.8k
vtfs-always seqwrite-psync-multi 307.5mb 76.8k
vtfs-always-dax seqwrite-psync-multi 1200.8mb 300.2k
9p-none seqwrite-mmap 0kb 0
9p-mmap seqwrite-mmap 101.3mb 25.3k
9p-loose seqwrite-mmap 112.4mb 28.1k
vtfs-none seqwrite-mmap 0kb 0
vtfs-none-dax seqwrite-mmap 1297.2mb 324.3k
vtfs-auto seqwrite-mmap 199.7mb 49.9k
vtfs-auto-dax seqwrite-mmap 1323.0mb 330.7k
vtfs-always seqwrite-mmap 181.7mb 45.4k
vtfs-always-dax seqwrite-mmap 1353.7mb 338.4k
9p-none seqwrite-mmap-multi 0kb 0
9p-mmap seqwrite-mmap-multi 274.7mb 68.6k
9p-loose seqwrite-mmap-multi 281.3mb 70.3k
vtfs-none seqwrite-mmap-multi 0kb 0
vtfs-none-dax seqwrite-mmap-multi 1806.2mb 451.5k
vtfs-auto seqwrite-mmap-multi 384.1mb 96.0k
vtfs-auto-dax seqwrite-mmap-multi 2023.4mb 505.8k
vtfs-always seqwrite-mmap-multi 384.5mb 96.1k
vtfs-always-dax seqwrite-mmap-multi 2008.9mb 502.2k
9p-none seqwrite-libaio 87.9mb 21.9k
9p-mmap seqwrite-libaio 89.3mb 22.3k
9p-loose seqwrite-libaio 42.4mb 10.5k
vtfs-none seqwrite-libaio 254.5mb 63.6k
vtfs-none-dax seqwrite-libaio 284.7mb 71.1k
vtfs-auto seqwrite-libaio 279.1mb 69.7k
vtfs-auto-dax seqwrite-libaio 299.6mb 74.9k
vtfs-always seqwrite-libaio 238.9mb 59.7k
vtfs-always-dax seqwrite-libaio 305.8mb 76.4k
9p-none seqwrite-libaio-multi 339.6mb 84.9k
9p-mmap seqwrite-libaio-multi 329.5mb 82.3k
9p-loose seqwrite-libaio-multi 165.8mb 41.4k
vtfs-none seqwrite-libaio-multi 320.6mb 80.1k
vtfs-none-dax seqwrite-libaio-multi 975.3mb 243.8k
vtfs-auto seqwrite-libaio-multi 306.4mb 76.6k
vtfs-auto-dax seqwrite-libaio-multi 974.4mb 243.6k
vtfs-always seqwrite-libaio-multi 312.6mb 78.1k
vtfs-always-dax seqwrite-libaio-multi 883.8mb 220.9k
9p-none randwrite-psync 94.9mb 23.7k
9p-mmap randwrite-psync 99.3mb 24.8k
9p-loose randwrite-psync 68.7mb 17.1k
vtfs-none randwrite-psync 97.2mb 24.3k
vtfs-none-dax randwrite-psync 480.3mb 120.0k
vtfs-auto randwrite-psync 82.2mb 20.5k
vtfs-auto-dax randwrite-psync 435.6mb 108.9k
vtfs-always randwrite-psync 77.3mb 19.3k
vtfs-always-dax randwrite-psync 451.1mb 112.7k
9p-none randwrite-psync-multi 336.0mb 84.0k
9p-mmap randwrite-psync-multi 338.5mb 84.6k
9p-loose randwrite-psync-multi 278.4mb 69.6k
vtfs-none randwrite-psync-multi 323.7mb 80.9k
vtfs-none-dax randwrite-psync-multi 519.2mb 129.8k
vtfs-auto randwrite-psync-multi 302.1mb 75.5k
vtfs-auto-dax randwrite-psync-multi 473.3mb 118.3k
vtfs-always randwrite-psync-multi 302.4mb 75.6k
vtfs-always-dax randwrite-psync-multi 512.9mb 128.2k
9p-none randwrite-mmap 0kb 0
9p-mmap randwrite-mmap 58.2mb 14.5k
9p-loose randwrite-mmap 63.5mb 15.8k
vtfs-none randwrite-mmap 0kb 0
vtfs-none-dax randwrite-mmap 872.3mb 218.0k
vtfs-auto randwrite-mmap 55.9mb 13.9k
vtfs-auto-dax randwrite-mmap 771.8mb 192.9k
vtfs-always randwrite-mmap 59.0mb 14.7k
vtfs-always-dax randwrite-mmap 828.8mb 207.2k
9p-none randwrite-mmap-multi 0kb 0
9p-mmap randwrite-mmap-multi 251.2mb 62.8k
9p-loose randwrite-mmap-multi 257.9mb 64.4k
vtfs-none randwrite-mmap-multi 0kb 0
vtfs-none-dax randwrite-mmap-multi 1119.1mb 279.7k
vtfs-auto randwrite-mmap-multi 208.9mb 52.2k
vtfs-auto-dax randwrite-mmap-multi 1043.9mb 260.9k
vtfs-always randwrite-mmap-multi 209.4mb 52.3k
vtfs-always-dax randwrite-mmap-multi 1091.4mb 272.8k
9p-none randwrite-libaio 85.9mb 21.4k
9p-mmap randwrite-libaio 87.8mb 21.9k
9p-loose randwrite-libaio 41.6mb 10.4k
vtfs-none randwrite-libaio 252.4mb 63.1k
vtfs-none-dax randwrite-libaio 282.0mb 70.5k
vtfs-auto randwrite-libaio 258.7mb 64.6k
vtfs-auto-dax randwrite-libaio 272.5mb 68.1k
vtfs-always randwrite-libaio 239.0mb 59.7k
vtfs-always-dax randwrite-libaio 256.4mb 64.1k
9p-none randwrite-libaio-multi 329.5mb 82.3k
9p-mmap randwrite-libaio-multi 324.4mb 81.1k
9p-loose randwrite-libaio-multi 161.1mb 40.2k
vtfs-none randwrite-libaio-multi 323.8mb 80.9k
vtfs-none-dax randwrite-libaio-multi 366.3mb 91.5k
vtfs-auto randwrite-libaio-multi 320.9mb 80.2k
vtfs-auto-dax randwrite-libaio-multi 351.2mb 87.8k
vtfs-always randwrite-libaio-multi 328.8mb 82.2k
vtfs-always-dax randwrite-libaio-multi 362.5mb 90.6k
9p-none randrw-psync 70.2mb/23.5mb 17.5k/6017
9p-mmap randrw-psync 74.5mb/24.9mb 18.6k/6398
9p-loose randrw-psync 59.2mb/19.8mb 14.8k/5077
vtfs-none randrw-psync 75.8mb/25.4mb 18.9k/6508
vtfs-none-dax randrw-psync 403.2mb/134.7mb 100.8k/33.6k
vtfs-auto randrw-psync 38.1mb/12.7mb 9775/3265
vtfs-auto-dax randrw-psync 382.4mb/127.8mb 95.6k/31.9k
vtfs-always randrw-psync 40.7mb/13.6mb 10.1k/3485
vtfs-always-dax randrw-psync 393.4mb/131.4mb 98.3k/32.8k
9p-none randrw-psync-multi 264.3mb/88.5mb 66.0k/22.1k
9p-mmap randrw-psync-multi 260.5mb/87.2mb 65.1k/21.8k
9p-loose randrw-psync-multi 232.6mb/77.9mb 58.1k/19.4k
vtfs-none randrw-psync-multi 260.7mb/87.3mb 65.1k/21.8k
vtfs-none-dax randrw-psync-multi 270.0mb/90.4mb 67.5k/22.6k
vtfs-auto randrw-psync-multi 151.4mb/50.7mb 37.8k/12.6k
vtfs-auto-dax randrw-psync-multi 251.2mb/84.1mb 62.8k/21.0k
vtfs-always randrw-psync-multi 152.9mb/51.2mb 38.2k/12.8k
vtfs-always-dax randrw-psync-multi 251.5mb/84.3mb 62.8k/21.0k
9p-none randrw-mmap 0kb/0kb 0/0
9p-mmap randrw-mmap 47.4mb/15.8mb 11.8k/4057
9p-loose randrw-mmap 48.4mb/16.1mb 12.1k/4143
vtfs-none randrw-mmap 0kb/0kb 0/0
vtfs-none-dax randrw-mmap 539.5mb/180.3mb 134.8k/45.0k
vtfs-auto randrw-mmap 44.4mb/14.8mb 11.1k/3806
vtfs-auto-dax randrw-mmap 544.2mb/181.8mb 136.0k/45.4k
vtfs-always randrw-mmap 43.1mb/14.4mb 10.7k/3691
vtfs-always-dax randrw-mmap 555.4mb/185.6mb 138.8k/46.4k
9p-none randrw-mmap-multi 0kb/0kb 0/0
9p-mmap randrw-mmap-multi 184.0mb/61.6mb 46.0k/15.4k
9p-loose randrw-mmap-multi 188.8mb/63.2mb 47.2k/15.8k
vtfs-none randrw-mmap-multi 0kb/0kb 0/0
vtfs-none-dax randrw-mmap-multi 573.8mb/191.9mb 143.4k/47.9k
vtfs-auto randrw-mmap-multi 179.3mb/60.0mb 44.8k/15.0k
vtfs-auto-dax randrw-mmap-multi 498.0mb/166.7mb 124.5k/41.6k
vtfs-always randrw-mmap-multi 180.2mb/60.4mb 45.0k/15.1k
vtfs-always-dax randrw-mmap-multi 525.0mb/175.7mb 131.2k/43.9k
9p-none randrw-libaio 68.4mb/22.8mb 17.1k/5858
9p-mmap randrw-libaio 67.4mb/22.5mb 16.8k/5777
9p-loose randrw-libaio 52.7mb/17.6mb 13.1k/4517
vtfs-none randrw-libaio 199.9mb/66.8mb 49.9k/16.7k
vtfs-none-dax randrw-libaio 256.9mb/85.8mb 64.2k/21.4k
vtfs-auto randrw-libaio 83.4mb/27.9mb 20.8k/7161
vtfs-auto-dax randrw-libaio 241.2mb/80.6mb 60.3k/20.1k
vtfs-always randrw-libaio 87.3mb/29.2mb 21.8k/7491
vtfs-always-dax randrw-libaio 252.5mb/84.3mb 63.1k/21.0k
9p-none randrw-libaio-multi 267.4mb/89.5mb 66.8k/22.3k
9p-mmap randrw-libaio-multi 256.2mb/85.8mb 64.0k/21.4k
9p-loose randrw-libaio-multi 200.1mb/67.0mb 50.0k/16.7k
vtfs-none randrw-libaio-multi 210.9mb/70.7mb 52.7k/17.6k
vtfs-none-dax randrw-libaio-multi 192.6mb/64.5mb 48.1k/16.1k
vtfs-auto randrw-libaio-multi 209.6mb/70.2mb 52.4k/17.5k
vtfs-auto-dax randrw-libaio-multi 189.1mb/63.3mb 47.2k/15.8k
vtfs-always randrw-libaio-multi 212.3mb/71.1mb 53.0k/17.7k
vtfs-always-dax randrw-libaio-multi 195.8mb/65.6mb 48.9k/16.4k
Thanks
Vivek
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Some performance numbers for virtiofs, DAX and virtio-9p
2020-12-10 16:11 ` [Virtio-fs] " Vivek Goyal
@ 2020-12-10 19:29 ` Miklos Szeredi
-1 siblings, 0 replies; 20+ messages in thread
From: Miklos Szeredi @ 2020-12-10 19:29 UTC (permalink / raw)
To: Vivek Goyal
Cc: Venegas Munoz, Jose Carlos, Christian Schoenebeck,
QEMU Developers, Dr. David Alan Gilbert, virtio-fs-list,
Stefan Hajnoczi
On Thu, Dec 10, 2020 at 5:11 PM Vivek Goyal <vgoyal@redhat.com> wrote:
> Conclusion
> -----------
> - virtiofs DAX seems to help a lot in many workloads.
>
> Note, DAX performance well only if data fits in cache window. My total
> data is 16G and cache window size is 16G as well. If data is larger
> than DAX cache window, then performance of dax suffers a lot. Overhead
> of reclaiming old mapping and setting up a new one is very high.
Which begs the question: what is the optimal window size?
What is the cost per GB of window to the host and guest?
Could we measure at what point does a large window size actually make
performance worse?
>
> NAME WORKLOAD Bandwidth IOPS
> 9p-none seqread-psync 98.6mb 24.6k
> 9p-mmap seqread-psync 97.5mb 24.3k
> 9p-loose seqread-psync 91.6mb 22.9k
> vtfs-none seqread-psync 98.4mb 24.6k
> vtfs-none-dax seqread-psync 660.3mb 165.0k
> vtfs-auto seqread-psync 650.0mb 162.5k
> vtfs-auto-dax seqread-psync 703.1mb 175.7k
> vtfs-always seqread-psync 671.3mb 167.8k
> vtfs-always-dax seqread-psync 687.2mb 171.8k
>
> 9p-none seqread-psync-multi 397.6mb 99.4k
> 9p-mmap seqread-psync-multi 382.7mb 95.6k
> 9p-loose seqread-psync-multi 350.5mb 87.6k
> vtfs-none seqread-psync-multi 360.0mb 90.0k
> vtfs-none-dax seqread-psync-multi 2281.1mb 570.2k
> vtfs-auto seqread-psync-multi 2530.7mb 632.6k
> vtfs-auto-dax seqread-psync-multi 2423.9mb 605.9k
> vtfs-always seqread-psync-multi 2535.7mb 633.9k
> vtfs-always-dax seqread-psync-multi 2406.1mb 601.5k
Seems like in all the -multi tests 9p-none performs consistently
better than vtfs-none. Could that be due to the single queue?
Thanks,
Miklos
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Virtio-fs] Some performance numbers for virtiofs, DAX and virtio-9p
@ 2020-12-10 19:29 ` Miklos Szeredi
0 siblings, 0 replies; 20+ messages in thread
From: Miklos Szeredi @ 2020-12-10 19:29 UTC (permalink / raw)
To: Vivek Goyal
Cc: Venegas Munoz, Jose Carlos, Christian Schoenebeck,
QEMU Developers, virtio-fs-list
On Thu, Dec 10, 2020 at 5:11 PM Vivek Goyal <vgoyal@redhat.com> wrote:
> Conclusion
> -----------
> - virtiofs DAX seems to help a lot in many workloads.
>
> Note, DAX performance well only if data fits in cache window. My total
> data is 16G and cache window size is 16G as well. If data is larger
> than DAX cache window, then performance of dax suffers a lot. Overhead
> of reclaiming old mapping and setting up a new one is very high.
Which begs the question: what is the optimal window size?
What is the cost per GB of window to the host and guest?
Could we measure at what point does a large window size actually make
performance worse?
>
> NAME WORKLOAD Bandwidth IOPS
> 9p-none seqread-psync 98.6mb 24.6k
> 9p-mmap seqread-psync 97.5mb 24.3k
> 9p-loose seqread-psync 91.6mb 22.9k
> vtfs-none seqread-psync 98.4mb 24.6k
> vtfs-none-dax seqread-psync 660.3mb 165.0k
> vtfs-auto seqread-psync 650.0mb 162.5k
> vtfs-auto-dax seqread-psync 703.1mb 175.7k
> vtfs-always seqread-psync 671.3mb 167.8k
> vtfs-always-dax seqread-psync 687.2mb 171.8k
>
> 9p-none seqread-psync-multi 397.6mb 99.4k
> 9p-mmap seqread-psync-multi 382.7mb 95.6k
> 9p-loose seqread-psync-multi 350.5mb 87.6k
> vtfs-none seqread-psync-multi 360.0mb 90.0k
> vtfs-none-dax seqread-psync-multi 2281.1mb 570.2k
> vtfs-auto seqread-psync-multi 2530.7mb 632.6k
> vtfs-auto-dax seqread-psync-multi 2423.9mb 605.9k
> vtfs-always seqread-psync-multi 2535.7mb 633.9k
> vtfs-always-dax seqread-psync-multi 2406.1mb 601.5k
Seems like in all the -multi tests 9p-none performs consistently
better than vtfs-none. Could that be due to the single queue?
Thanks,
Miklos
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Some performance numbers for virtiofs, DAX and virtio-9p
2020-12-10 19:29 ` [Virtio-fs] " Miklos Szeredi
@ 2020-12-11 16:06 ` Vivek Goyal
-1 siblings, 0 replies; 20+ messages in thread
From: Vivek Goyal @ 2020-12-11 16:06 UTC (permalink / raw)
To: Miklos Szeredi
Cc: Venegas Munoz, Jose Carlos, Christian Schoenebeck,
QEMU Developers, Dr. David Alan Gilbert, virtio-fs-list,
Stefan Hajnoczi
On Thu, Dec 10, 2020 at 08:29:21PM +0100, Miklos Szeredi wrote:
> On Thu, Dec 10, 2020 at 5:11 PM Vivek Goyal <vgoyal@redhat.com> wrote:
>
> > Conclusion
> > -----------
> > - virtiofs DAX seems to help a lot in many workloads.
> >
> > Note, DAX performance well only if data fits in cache window. My total
> > data is 16G and cache window size is 16G as well. If data is larger
> > than DAX cache window, then performance of dax suffers a lot. Overhead
> > of reclaiming old mapping and setting up a new one is very high.
>
> Which begs the question: what is the optimal window size?
Yep. I will need to run some more tests with data size being constant
and varying DAX window size.
For now, I would say optimal window size is same as data size. But
knowing data size might be hard in advance. So a rough guideline
could be that it could be same as amount of RAM given to guest.
>
> What is the cost per GB of window to the host and guest?
Inside guest, I think two primary structures are allocated. There
will be "struct page" allocated per 4K page. Size of struct page
seems to be 64. And then there will be "struct fuse_dax_mapping"
allocated per 2MB. Size of "struct fuse_dax_mapping" is 112.
This means per 2MB of DAX window, memory needed in guest is.
memory per 2MB of DAX window = 112 + 64 * 512 = 32880 bytes.
memory per 1GB of DAX window = 32880 * 512 = 16834560 (16MB approx)
I think "struct page" allocation is biggest memory allocation
and that's roughly 1.56% (64/4096) of DAX window size. And that also
results in 16MB memory allocation per GB of dax window.
So if a guest has 4G RAM and 4G dax window, then 64MB will be
consumed in dax window struct pages. I will say no too bad.
I am looking at qemu code and its not obvious to me what memory
allocation will be needed 1GB of guest. Looks like it just
stores the cache window location and size and when mapping
request comes, it simply adds offset to cache window start. So
it might not be allocating memory per page of dax window.
mmap(cache_host + sm->c_offset[i], sm->len[i]....
David, you most likely have a better idea about this.
>
> Could we measure at what point does a large window size actually make
> performance worse?
Will do. Will run tests with varying window sizes (small to large)
and see how does it impact performance for same workload with
same guest memory.
>
> >
> > NAME WORKLOAD Bandwidth IOPS
> > 9p-none seqread-psync 98.6mb 24.6k
> > 9p-mmap seqread-psync 97.5mb 24.3k
> > 9p-loose seqread-psync 91.6mb 22.9k
> > vtfs-none seqread-psync 98.4mb 24.6k
> > vtfs-none-dax seqread-psync 660.3mb 165.0k
> > vtfs-auto seqread-psync 650.0mb 162.5k
> > vtfs-auto-dax seqread-psync 703.1mb 175.7k
> > vtfs-always seqread-psync 671.3mb 167.8k
> > vtfs-always-dax seqread-psync 687.2mb 171.8k
> >
> > 9p-none seqread-psync-multi 397.6mb 99.4k
> > 9p-mmap seqread-psync-multi 382.7mb 95.6k
> > 9p-loose seqread-psync-multi 350.5mb 87.6k
> > vtfs-none seqread-psync-multi 360.0mb 90.0k
> > vtfs-none-dax seqread-psync-multi 2281.1mb 570.2k
> > vtfs-auto seqread-psync-multi 2530.7mb 632.6k
> > vtfs-auto-dax seqread-psync-multi 2423.9mb 605.9k
> > vtfs-always seqread-psync-multi 2535.7mb 633.9k
> > vtfs-always-dax seqread-psync-multi 2406.1mb 601.5k
>
> Seems like in all the -multi tests 9p-none performs consistently
> better than vtfs-none. Could that be due to the single queue?
Not sure. In the past I had run -multi tests with shared thread pool
(cache=auto) and single thread seemed to perform better. I can
try shared pool and run -multi tests again and see if that helps.
Thanks
Vivek
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Virtio-fs] Some performance numbers for virtiofs, DAX and virtio-9p
@ 2020-12-11 16:06 ` Vivek Goyal
0 siblings, 0 replies; 20+ messages in thread
From: Vivek Goyal @ 2020-12-11 16:06 UTC (permalink / raw)
To: Miklos Szeredi
Cc: Venegas Munoz, Jose Carlos, Christian Schoenebeck,
QEMU Developers, virtio-fs-list
On Thu, Dec 10, 2020 at 08:29:21PM +0100, Miklos Szeredi wrote:
> On Thu, Dec 10, 2020 at 5:11 PM Vivek Goyal <vgoyal@redhat.com> wrote:
>
> > Conclusion
> > -----------
> > - virtiofs DAX seems to help a lot in many workloads.
> >
> > Note, DAX performance well only if data fits in cache window. My total
> > data is 16G and cache window size is 16G as well. If data is larger
> > than DAX cache window, then performance of dax suffers a lot. Overhead
> > of reclaiming old mapping and setting up a new one is very high.
>
> Which begs the question: what is the optimal window size?
Yep. I will need to run some more tests with data size being constant
and varying DAX window size.
For now, I would say optimal window size is same as data size. But
knowing data size might be hard in advance. So a rough guideline
could be that it could be same as amount of RAM given to guest.
>
> What is the cost per GB of window to the host and guest?
Inside guest, I think two primary structures are allocated. There
will be "struct page" allocated per 4K page. Size of struct page
seems to be 64. And then there will be "struct fuse_dax_mapping"
allocated per 2MB. Size of "struct fuse_dax_mapping" is 112.
This means per 2MB of DAX window, memory needed in guest is.
memory per 2MB of DAX window = 112 + 64 * 512 = 32880 bytes.
memory per 1GB of DAX window = 32880 * 512 = 16834560 (16MB approx)
I think "struct page" allocation is biggest memory allocation
and that's roughly 1.56% (64/4096) of DAX window size. And that also
results in 16MB memory allocation per GB of dax window.
So if a guest has 4G RAM and 4G dax window, then 64MB will be
consumed in dax window struct pages. I will say no too bad.
I am looking at qemu code and its not obvious to me what memory
allocation will be needed 1GB of guest. Looks like it just
stores the cache window location and size and when mapping
request comes, it simply adds offset to cache window start. So
it might not be allocating memory per page of dax window.
mmap(cache_host + sm->c_offset[i], sm->len[i]....
David, you most likely have a better idea about this.
>
> Could we measure at what point does a large window size actually make
> performance worse?
Will do. Will run tests with varying window sizes (small to large)
and see how does it impact performance for same workload with
same guest memory.
>
> >
> > NAME WORKLOAD Bandwidth IOPS
> > 9p-none seqread-psync 98.6mb 24.6k
> > 9p-mmap seqread-psync 97.5mb 24.3k
> > 9p-loose seqread-psync 91.6mb 22.9k
> > vtfs-none seqread-psync 98.4mb 24.6k
> > vtfs-none-dax seqread-psync 660.3mb 165.0k
> > vtfs-auto seqread-psync 650.0mb 162.5k
> > vtfs-auto-dax seqread-psync 703.1mb 175.7k
> > vtfs-always seqread-psync 671.3mb 167.8k
> > vtfs-always-dax seqread-psync 687.2mb 171.8k
> >
> > 9p-none seqread-psync-multi 397.6mb 99.4k
> > 9p-mmap seqread-psync-multi 382.7mb 95.6k
> > 9p-loose seqread-psync-multi 350.5mb 87.6k
> > vtfs-none seqread-psync-multi 360.0mb 90.0k
> > vtfs-none-dax seqread-psync-multi 2281.1mb 570.2k
> > vtfs-auto seqread-psync-multi 2530.7mb 632.6k
> > vtfs-auto-dax seqread-psync-multi 2423.9mb 605.9k
> > vtfs-always seqread-psync-multi 2535.7mb 633.9k
> > vtfs-always-dax seqread-psync-multi 2406.1mb 601.5k
>
> Seems like in all the -multi tests 9p-none performs consistently
> better than vtfs-none. Could that be due to the single queue?
Not sure. In the past I had run -multi tests with shared thread pool
(cache=auto) and single thread seemed to perform better. I can
try shared pool and run -multi tests again and see if that helps.
Thanks
Vivek
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Some performance numbers for virtiofs, DAX and virtio-9p
2020-12-11 16:06 ` [Virtio-fs] " Vivek Goyal
@ 2020-12-11 18:29 ` Dr. David Alan Gilbert
-1 siblings, 0 replies; 20+ messages in thread
From: Dr. David Alan Gilbert @ 2020-12-11 18:29 UTC (permalink / raw)
To: Vivek Goyal
Cc: Miklos Szeredi, Venegas Munoz, Jose Carlos,
Christian Schoenebeck, QEMU Developers, virtio-fs-list,
Stefan Hajnoczi
* Vivek Goyal (vgoyal@redhat.com) wrote:
> On Thu, Dec 10, 2020 at 08:29:21PM +0100, Miklos Szeredi wrote:
> > On Thu, Dec 10, 2020 at 5:11 PM Vivek Goyal <vgoyal@redhat.com> wrote:
> >
> > > Conclusion
> > > -----------
> > > - virtiofs DAX seems to help a lot in many workloads.
> > >
> > > Note, DAX performance well only if data fits in cache window. My total
> > > data is 16G and cache window size is 16G as well. If data is larger
> > > than DAX cache window, then performance of dax suffers a lot. Overhead
> > > of reclaiming old mapping and setting up a new one is very high.
> >
> > Which begs the question: what is the optimal window size?
>
> Yep. I will need to run some more tests with data size being constant
> and varying DAX window size.
>
> For now, I would say optimal window size is same as data size. But
> knowing data size might be hard in advance. So a rough guideline
> could be that it could be same as amount of RAM given to guest.
>
> >
> > What is the cost per GB of window to the host and guest?
>
> Inside guest, I think two primary structures are allocated. There
> will be "struct page" allocated per 4K page. Size of struct page
> seems to be 64. And then there will be "struct fuse_dax_mapping"
> allocated per 2MB. Size of "struct fuse_dax_mapping" is 112.
>
> This means per 2MB of DAX window, memory needed in guest is.
>
> memory per 2MB of DAX window = 112 + 64 * 512 = 32880 bytes.
> memory per 1GB of DAX window = 32880 * 512 = 16834560 (16MB approx)
>
> I think "struct page" allocation is biggest memory allocation
> and that's roughly 1.56% (64/4096) of DAX window size. And that also
> results in 16MB memory allocation per GB of dax window.
>
> So if a guest has 4G RAM and 4G dax window, then 64MB will be
> consumed in dax window struct pages. I will say no too bad.
>
> I am looking at qemu code and its not obvious to me what memory
> allocation will be needed 1GB of guest. Looks like it just
> stores the cache window location and size and when mapping
> request comes, it simply adds offset to cache window start. So
> it might not be allocating memory per page of dax window.
>
> mmap(cache_host + sm->c_offset[i], sm->len[i]....
>
> David, you most likely have a better idea about this.
No, I don't think we do any more; it might make sense of us to store a
per-mapping structure though at some point.
I'm assuming the host kernel is going to get some overhead as well.
> >
> > Could we measure at what point does a large window size actually make
> > performance worse?
>
> Will do. Will run tests with varying window sizes (small to large)
> and see how does it impact performance for same workload with
> same guest memory.
I wonder how realistic it is though; it makes some sense if you have a
scenario like a fairly small root filesystem - something tractable; but
if you have a large FS you're not realistically going to be able to set
the cache size to match it - that's why it's a cache!
Dave
> >
> > >
> > > NAME WORKLOAD Bandwidth IOPS
> > > 9p-none seqread-psync 98.6mb 24.6k
> > > 9p-mmap seqread-psync 97.5mb 24.3k
> > > 9p-loose seqread-psync 91.6mb 22.9k
> > > vtfs-none seqread-psync 98.4mb 24.6k
> > > vtfs-none-dax seqread-psync 660.3mb 165.0k
> > > vtfs-auto seqread-psync 650.0mb 162.5k
> > > vtfs-auto-dax seqread-psync 703.1mb 175.7k
> > > vtfs-always seqread-psync 671.3mb 167.8k
> > > vtfs-always-dax seqread-psync 687.2mb 171.8k
> > >
> > > 9p-none seqread-psync-multi 397.6mb 99.4k
> > > 9p-mmap seqread-psync-multi 382.7mb 95.6k
> > > 9p-loose seqread-psync-multi 350.5mb 87.6k
> > > vtfs-none seqread-psync-multi 360.0mb 90.0k
> > > vtfs-none-dax seqread-psync-multi 2281.1mb 570.2k
> > > vtfs-auto seqread-psync-multi 2530.7mb 632.6k
> > > vtfs-auto-dax seqread-psync-multi 2423.9mb 605.9k
> > > vtfs-always seqread-psync-multi 2535.7mb 633.9k
> > > vtfs-always-dax seqread-psync-multi 2406.1mb 601.5k
> >
> > Seems like in all the -multi tests 9p-none performs consistently
> > better than vtfs-none. Could that be due to the single queue?
>
> Not sure. In the past I had run -multi tests with shared thread pool
> (cache=auto) and single thread seemed to perform better. I can
> try shared pool and run -multi tests again and see if that helps.
>
> Thanks
> Vivek
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Virtio-fs] Some performance numbers for virtiofs, DAX and virtio-9p
@ 2020-12-11 18:29 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 20+ messages in thread
From: Dr. David Alan Gilbert @ 2020-12-11 18:29 UTC (permalink / raw)
To: Vivek Goyal
Cc: Venegas Munoz, Jose Carlos, Christian Schoenebeck,
QEMU Developers, virtio-fs-list
* Vivek Goyal (vgoyal@redhat.com) wrote:
> On Thu, Dec 10, 2020 at 08:29:21PM +0100, Miklos Szeredi wrote:
> > On Thu, Dec 10, 2020 at 5:11 PM Vivek Goyal <vgoyal@redhat.com> wrote:
> >
> > > Conclusion
> > > -----------
> > > - virtiofs DAX seems to help a lot in many workloads.
> > >
> > > Note, DAX performance well only if data fits in cache window. My total
> > > data is 16G and cache window size is 16G as well. If data is larger
> > > than DAX cache window, then performance of dax suffers a lot. Overhead
> > > of reclaiming old mapping and setting up a new one is very high.
> >
> > Which begs the question: what is the optimal window size?
>
> Yep. I will need to run some more tests with data size being constant
> and varying DAX window size.
>
> For now, I would say optimal window size is same as data size. But
> knowing data size might be hard in advance. So a rough guideline
> could be that it could be same as amount of RAM given to guest.
>
> >
> > What is the cost per GB of window to the host and guest?
>
> Inside guest, I think two primary structures are allocated. There
> will be "struct page" allocated per 4K page. Size of struct page
> seems to be 64. And then there will be "struct fuse_dax_mapping"
> allocated per 2MB. Size of "struct fuse_dax_mapping" is 112.
>
> This means per 2MB of DAX window, memory needed in guest is.
>
> memory per 2MB of DAX window = 112 + 64 * 512 = 32880 bytes.
> memory per 1GB of DAX window = 32880 * 512 = 16834560 (16MB approx)
>
> I think "struct page" allocation is biggest memory allocation
> and that's roughly 1.56% (64/4096) of DAX window size. And that also
> results in 16MB memory allocation per GB of dax window.
>
> So if a guest has 4G RAM and 4G dax window, then 64MB will be
> consumed in dax window struct pages. I will say no too bad.
>
> I am looking at qemu code and its not obvious to me what memory
> allocation will be needed 1GB of guest. Looks like it just
> stores the cache window location and size and when mapping
> request comes, it simply adds offset to cache window start. So
> it might not be allocating memory per page of dax window.
>
> mmap(cache_host + sm->c_offset[i], sm->len[i]....
>
> David, you most likely have a better idea about this.
No, I don't think we do any more; it might make sense of us to store a
per-mapping structure though at some point.
I'm assuming the host kernel is going to get some overhead as well.
> >
> > Could we measure at what point does a large window size actually make
> > performance worse?
>
> Will do. Will run tests with varying window sizes (small to large)
> and see how does it impact performance for same workload with
> same guest memory.
I wonder how realistic it is though; it makes some sense if you have a
scenario like a fairly small root filesystem - something tractable; but
if you have a large FS you're not realistically going to be able to set
the cache size to match it - that's why it's a cache!
Dave
> >
> > >
> > > NAME WORKLOAD Bandwidth IOPS
> > > 9p-none seqread-psync 98.6mb 24.6k
> > > 9p-mmap seqread-psync 97.5mb 24.3k
> > > 9p-loose seqread-psync 91.6mb 22.9k
> > > vtfs-none seqread-psync 98.4mb 24.6k
> > > vtfs-none-dax seqread-psync 660.3mb 165.0k
> > > vtfs-auto seqread-psync 650.0mb 162.5k
> > > vtfs-auto-dax seqread-psync 703.1mb 175.7k
> > > vtfs-always seqread-psync 671.3mb 167.8k
> > > vtfs-always-dax seqread-psync 687.2mb 171.8k
> > >
> > > 9p-none seqread-psync-multi 397.6mb 99.4k
> > > 9p-mmap seqread-psync-multi 382.7mb 95.6k
> > > 9p-loose seqread-psync-multi 350.5mb 87.6k
> > > vtfs-none seqread-psync-multi 360.0mb 90.0k
> > > vtfs-none-dax seqread-psync-multi 2281.1mb 570.2k
> > > vtfs-auto seqread-psync-multi 2530.7mb 632.6k
> > > vtfs-auto-dax seqread-psync-multi 2423.9mb 605.9k
> > > vtfs-always seqread-psync-multi 2535.7mb 633.9k
> > > vtfs-always-dax seqread-psync-multi 2406.1mb 601.5k
> >
> > Seems like in all the -multi tests 9p-none performs consistently
> > better than vtfs-none. Could that be due to the single queue?
>
> Not sure. In the past I had run -multi tests with shared thread pool
> (cache=auto) and single thread seemed to perform better. I can
> try shared pool and run -multi tests again and see if that helps.
>
> Thanks
> Vivek
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Some performance numbers for virtiofs, DAX and virtio-9p
2020-12-11 16:06 ` [Virtio-fs] " Vivek Goyal
@ 2020-12-11 18:47 ` Vivek Goyal
-1 siblings, 0 replies; 20+ messages in thread
From: Vivek Goyal @ 2020-12-11 18:47 UTC (permalink / raw)
To: Miklos Szeredi
Cc: Venegas Munoz, Jose Carlos, Christian Schoenebeck,
QEMU Developers, Dr. David Alan Gilbert, virtio-fs-list,
Stefan Hajnoczi
On Fri, Dec 11, 2020 at 11:06:03AM -0500, Vivek Goyal wrote:
[..]
> >
> > Could we measure at what point does a large window size actually make
> > performance worse?
>
> Will do. Will run tests with varying window sizes (small to large)
> and see how does it impact performance for same workload with
> same guest memory.
>
Ok, I ran some tests with virtiofs cache=auto and varied dax window
size from 1G to 64G. (1, 2, 4, 8, 16, 32, 64). Still using file size
4G each. And -multi test use 4 files of 4G each. Here are the test
results.
To run tests faster, I ran one iteration of each job (instead of running
3 iterations and taking average). This should be good enough to give
some idea.
I have also pushed raw data file and test results here.
https://github.com/rhvgoyal/virtiofs-tests/tree/master/performance-results/dec-10-2020
First line is vtfs cache auto without dax to see with what dax window
size dax helps as opposed to without dax.
Conclusion
----------
- Decreasing window size hurts tremendously for random read and random
write workloads.
- seqread/seqwrite still perform reasonably well even with smaller
window size.
- Increasing DAX window does not hurt.
I will try to spend some time with smaller dax window and random read
and ramdom write workloads and see if there is anything I can do to
reduce the performance drop.
Thanks
Vivek
NAME WORKLOAD Bandwidth IOPS
vtfs-auto seqread-psync 650.0mb 162.5k
vtfs-auto-dax-1G seqread-psync 653.1mb 163.2k
vtfs-auto-dax-2G seqread-psync 668.2mb 167.0k
vtfs-auto-dax-4G seqread-psync 657.8mb 164.4k
vtfs-auto-dax-8G seqread-psync 699.5mb 174.8k
vtfs-auto-dax-16G seqread-psync 674.3mb 168.5k
vtfs-auto-dax-32G seqread-psync 667.6mb 166.9k
vtfs-auto-dax-64G seqread-psync 696.1mb 174.0k
vtfs-auto seqread-psync-multi 2530.7mb 632.6k
vtfs-auto-dax-1G seqread-psync-multi 1931.1mb 482.7k
vtfs-auto-dax-2G seqread-psync-multi 2105.6mb 526.4k
vtfs-auto-dax-4G seqread-psync-multi 2142.2mb 535.5k
vtfs-auto-dax-8G seqread-psync-multi 2229.7mb 557.4k
vtfs-auto-dax-16G seqread-psync-multi 2324.3mb 581.0k
vtfs-auto-dax-32G seqread-psync-multi 2421.1mb 605.2k
vtfs-auto-dax-64G seqread-psync-multi 2469.7mb 617.4k
vtfs-auto seqread-mmap 570.6mb 142.6k
vtfs-auto-dax-1G seqread-mmap 823.9mb 205.9k
vtfs-auto-dax-2G seqread-mmap 798.1mb 199.5k
vtfs-auto-dax-4G seqread-mmap 874.4mb 218.6k
vtfs-auto-dax-8G seqread-mmap 960.5mb 240.1k
vtfs-auto-dax-16G seqread-mmap 917.1mb 229.2k
vtfs-auto-dax-32G seqread-mmap 884.4mb 221.1k
vtfs-auto-dax-64G seqread-mmap 855.6mb 213.9k
vtfs-auto seqread-mmap-multi 2210.1mb 552.5k
vtfs-auto-dax-1G seqread-mmap-multi 2722.9mb 680.7k
vtfs-auto-dax-2G seqread-mmap-multi 2828.2mb 707.0k
vtfs-auto-dax-4G seqread-mmap-multi 2777.4mb 694.3k
vtfs-auto-dax-8G seqread-mmap-multi 2880.4mb 720.1k
vtfs-auto-dax-16G seqread-mmap-multi 3159.2mb 789.8k
vtfs-auto-dax-32G seqread-mmap-multi 2989.2mb 747.3k
vtfs-auto-dax-64G seqread-mmap-multi 3007.3mb 751.8k
vtfs-auto seqread-libaio 303.3mb 75.8k
vtfs-auto-dax-1G seqread-libaio 396.4mb 99.1k
vtfs-auto-dax-2G seqread-libaio 391.9mb 97.9k
vtfs-auto-dax-4G seqread-libaio 422.9mb 105.7k
vtfs-auto-dax-8G seqread-libaio 419.0mb 104.7k
vtfs-auto-dax-16G seqread-libaio 411.2mb 102.8k
vtfs-auto-dax-32G seqread-libaio 436.1mb 109.0k
vtfs-auto-dax-64G seqread-libaio 418.0mb 104.5k
vtfs-auto seqread-libaio-multi 1500.0mb 375.0k
vtfs-auto-dax-1G seqread-libaio-multi 1271.7mb 317.9k
vtfs-auto-dax-2G seqread-libaio-multi 1338.1mb 334.5k
vtfs-auto-dax-4G seqread-libaio-multi 1306.6mb 326.6k
vtfs-auto-dax-8G seqread-libaio-multi 1416.8mb 354.2k
vtfs-auto-dax-16G seqread-libaio-multi 1461.9mb 365.4k
vtfs-auto-dax-32G seqread-libaio-multi 1531.9mb 382.9k
vtfs-auto-dax-64G seqread-libaio-multi 1495.4mb 373.8k
vtfs-auto randread-psync 77.9mb 19.4k
vtfs-auto-dax-1G randread-psync 18.7mb 4792
vtfs-auto-dax-2G randread-psync 25.2mb 6463
vtfs-auto-dax-4G randread-psync 59.9mb 14.9k
vtfs-auto-dax-8G randread-psync 693.7mb 173.4k
vtfs-auto-dax-16G randread-psync 646.0mb 161.5k
vtfs-auto-dax-32G randread-psync 635.4mb 158.8k
vtfs-auto-dax-64G randread-psync 663.4mb 165.8k
vtfs-auto randread-psync-multi 285.2mb 71.3k
vtfs-auto-dax-1G randread-psync-multi 28.7mb 7357
vtfs-auto-dax-2G randread-psync-multi 29.6mb 7596
vtfs-auto-dax-4G randread-psync-multi 35.0mb 8969
vtfs-auto-dax-8G randread-psync-multi 51.3mb 12.8k
vtfs-auto-dax-16G randread-psync-multi 497.1mb 124.2k
vtfs-auto-dax-32G randread-psync-multi 2092.9mb 523.2k
vtfs-auto-dax-64G randread-psync-multi 2051.0mb 512.7k
vtfs-auto randread-mmap 57.7mb 14.4k
vtfs-auto-dax-1G randread-mmap 15.1mb 3889
vtfs-auto-dax-2G randread-mmap 21.5mb 5506
vtfs-auto-dax-4G randread-mmap 59.4mb 14.8k
vtfs-auto-dax-8G randread-mmap 970.1mb 242.5k
vtfs-auto-dax-16G randread-mmap 883.1mb 220.7k
vtfs-auto-dax-32G randread-mmap 874.2mb 218.5k
vtfs-auto-dax-64G randread-mmap 954.3mb 238.5k
vtfs-auto randread-mmap-multi 243.5mb 60.8k
vtfs-auto-dax-1G randread-mmap-multi 11.7mb 3005
vtfs-auto-dax-2G randread-mmap-multi 20.4mb 5232
vtfs-auto-dax-4G randread-mmap-multi 23.1mb 5937
vtfs-auto-dax-8G randread-mmap-multi 35.3mb 9058
vtfs-auto-dax-16G randread-mmap-multi 1009.8mb 252.4k
vtfs-auto-dax-32G randread-mmap-multi 2612.6mb 653.1k
vtfs-auto-dax-64G randread-mmap-multi 2908.5mb 727.1k
vtfs-auto randread-libaio 276.2mb 69.0k
vtfs-auto-dax-1G randread-libaio 19.4mb 4968
vtfs-auto-dax-2G randread-libaio 23.3mb 5981
vtfs-auto-dax-4G randread-libaio 60.6mb 15.1k
vtfs-auto-dax-8G randread-libaio 413.5mb 103.3k
vtfs-auto-dax-16G randread-libaio 410.8mb 102.7k
vtfs-auto-dax-32G randread-libaio 392.0mb 98.0k
vtfs-auto-dax-64G randread-libaio 412.9mb 103.2k
vtfs-auto randread-libaio-multi 271.0mb 67.7k
vtfs-auto-dax-1G randread-libaio-multi 29.2mb 7479
vtfs-auto-dax-2G randread-libaio-multi 30.4mb 7803
vtfs-auto-dax-4G randread-libaio-multi 36.6mb 9387
vtfs-auto-dax-8G randread-libaio-multi 49.0mb 12.2k
vtfs-auto-dax-16G randread-libaio-multi 364.3mb 91.0k
vtfs-auto-dax-32G randread-libaio-multi 1435.0mb 358.7k
vtfs-auto-dax-64G randread-libaio-multi 1358.4mb 339.6k
vtfs-auto seqwrite-psync 78.6mb 19.6k
vtfs-auto-dax-1G seqwrite-psync 440.7mb 110.1k
vtfs-auto-dax-2G seqwrite-psync 428.6mb 107.1k
vtfs-auto-dax-4G seqwrite-psync 476.5mb 119.1k
vtfs-auto-dax-8G seqwrite-psync 547.8mb 136.9k
vtfs-auto-dax-16G seqwrite-psync 530.1mb 132.5k
vtfs-auto-dax-32G seqwrite-psync 486.6mb 121.6k
vtfs-auto-dax-64G seqwrite-psync 515.8mb 128.9k
vtfs-auto seqwrite-psync-multi 305.7mb 76.4k
vtfs-auto-dax-1G seqwrite-psync-multi 1405.0mb 351.2k
vtfs-auto-dax-2G seqwrite-psync-multi 1447.0mb 361.7k
vtfs-auto-dax-4G seqwrite-psync-multi 1329.4mb 332.3k
vtfs-auto-dax-8G seqwrite-psync-multi 1250.4mb 312.5k
vtfs-auto-dax-16G seqwrite-psync-multi 1264.7mb 316.1k
vtfs-auto-dax-32G seqwrite-psync-multi 1224.5mb 306.1k
vtfs-auto-dax-64G seqwrite-psync-multi 1165.6mb 291.4k
vtfs-auto seqwrite-mmap 199.7mb 49.9k
vtfs-auto-dax-1G seqwrite-mmap 1262.6mb 315.6k
vtfs-auto-dax-2G seqwrite-mmap 1369.4mb 342.3k
vtfs-auto-dax-4G seqwrite-mmap 1363.5mb 340.8k
vtfs-auto-dax-8G seqwrite-mmap 1368.9mb 342.2k
vtfs-auto-dax-16G seqwrite-mmap 1305.2mb 326.3k
vtfs-auto-dax-32G seqwrite-mmap 1299.4mb 324.8k
vtfs-auto-dax-64G seqwrite-mmap 1301.1mb 325.2k
vtfs-auto seqwrite-mmap-multi 384.1mb 96.0k
vtfs-auto-dax-1G seqwrite-mmap-multi 2162.9mb 540.7k
vtfs-auto-dax-2G seqwrite-mmap-multi 2191.8mb 547.9k
vtfs-auto-dax-4G seqwrite-mmap-multi 2045.1mb 511.2k
vtfs-auto-dax-8G seqwrite-mmap-multi 1973.0mb 493.2k
vtfs-auto-dax-16G seqwrite-mmap-multi 1911.3mb 477.8k
vtfs-auto-dax-32G seqwrite-mmap-multi 2263.9mb 565.9k
vtfs-auto-dax-64G seqwrite-mmap-multi 1718.4mb 429.6k
vtfs-auto seqwrite-libaio 279.1mb 69.7k
vtfs-auto-dax-1G seqwrite-libaio 294.2mb 73.5k
vtfs-auto-dax-2G seqwrite-libaio 308.7mb 77.1k
vtfs-auto-dax-4G seqwrite-libaio 308.3mb 77.0k
vtfs-auto-dax-8G seqwrite-libaio 339.0mb 84.7k
vtfs-auto-dax-16G seqwrite-libaio 323.3mb 80.8k
vtfs-auto-dax-32G seqwrite-libaio 311.4mb 77.8k
vtfs-auto-dax-64G seqwrite-libaio 334.2mb 83.5k
vtfs-auto seqwrite-libaio-multi 306.4mb 76.6k
vtfs-auto-dax-1G seqwrite-libaio-multi 968.7mb 242.1k
vtfs-auto-dax-2G seqwrite-libaio-multi 1050.7mb 262.6k
vtfs-auto-dax-4G seqwrite-libaio-multi 989.4mb 247.3k
vtfs-auto-dax-8G seqwrite-libaio-multi 1017.8mb 254.4k
vtfs-auto-dax-16G seqwrite-libaio-multi 1101.5mb 275.3k
vtfs-auto-dax-32G seqwrite-libaio-multi 915.0mb 228.7k
vtfs-auto-dax-64G seqwrite-libaio-multi 1067.5mb 266.8k
vtfs-auto randwrite-psync 82.2mb 20.5k
vtfs-auto-dax-1G randwrite-psync 20.4mb 5240
vtfs-auto-dax-2G randwrite-psync 29.6mb 7585
vtfs-auto-dax-4G randwrite-psync 65.4mb 16.3k
vtfs-auto-dax-8G randwrite-psync 502.6mb 125.6k
vtfs-auto-dax-16G randwrite-psync 501.6mb 125.4k
vtfs-auto-dax-32G randwrite-psync 461.6mb 115.4k
vtfs-auto-dax-64G randwrite-psync 481.2mb 120.2k
vtfs-auto randwrite-psync-multi 302.1mb 75.5k
vtfs-auto-dax-1G randwrite-psync-multi 30.9mb 7915
vtfs-auto-dax-2G randwrite-psync-multi 35.0mb 8968
vtfs-auto-dax-4G randwrite-psync-multi 36.7mb 9399
vtfs-auto-dax-8G randwrite-psync-multi 57.1mb 14.2k
vtfs-auto-dax-16G randwrite-psync-multi 499.0mb 124.7k
vtfs-auto-dax-32G randwrite-psync-multi 1123.0mb 280.7k
vtfs-auto-dax-64G randwrite-psync-multi 1058.3mb 264.5k
vtfs-auto randwrite-mmap 55.9mb 13.9k
vtfs-auto-dax-1G randwrite-mmap 16.4mb 4221
vtfs-auto-dax-2G randwrite-mmap 22.0mb 5648
vtfs-auto-dax-4G randwrite-mmap 60.6mb 15.1k
vtfs-auto-dax-8G randwrite-mmap 763.3mb 190.8k
vtfs-auto-dax-16G randwrite-mmap 859.2mb 214.8k
vtfs-auto-dax-32G randwrite-mmap 792.5mb 198.1k
vtfs-auto-dax-64G randwrite-mmap 807.7mb 201.9k
vtfs-auto randwrite-mmap-multi 208.9mb 52.2k
vtfs-auto-dax-1G randwrite-mmap-multi 12.4mb 3191
vtfs-auto-dax-2G randwrite-mmap-multi 19.8mb 5072
vtfs-auto-dax-4G randwrite-mmap-multi 22.3mb 5710
vtfs-auto-dax-8G randwrite-mmap-multi 33.2mb 8511
vtfs-auto-dax-16G randwrite-mmap-multi 1061.2mb 265.3k
vtfs-auto-dax-32G randwrite-mmap-multi 2113.2mb 528.3k
vtfs-auto-dax-64G randwrite-mmap-multi 2099.4mb 524.8k
vtfs-auto randwrite-libaio 258.7mb 64.6k
vtfs-auto-dax-1G randwrite-libaio 20.4mb 5231
vtfs-auto-dax-2G randwrite-libaio 26.4mb 6774
vtfs-auto-dax-4G randwrite-libaio 64.9mb 16.2k
vtfs-auto-dax-8G randwrite-libaio 326.7mb 81.6k
vtfs-auto-dax-16G randwrite-libaio 303.6mb 75.9k
vtfs-auto-dax-32G randwrite-libaio 298.2mb 74.5k
vtfs-auto-dax-64G randwrite-libaio 328.5mb 82.1k
vtfs-auto randwrite-libaio-multi 320.9mb 80.2k
vtfs-auto-dax-1G randwrite-libaio-multi 31.1mb 7977
vtfs-auto-dax-2G randwrite-libaio-multi 34.2mb 8772
vtfs-auto-dax-4G randwrite-libaio-multi 38.8mb 9945
vtfs-auto-dax-8G randwrite-libaio-multi 55.9mb 13.9k
vtfs-auto-dax-16G randwrite-libaio-multi 363.2mb 90.8k
vtfs-auto-dax-32G randwrite-libaio-multi 800.4mb 200.1k
vtfs-auto-dax-64G randwrite-libaio-multi 805.5mb 201.3k
vtfs-auto randrw-psync 38.1mb/12.7mb 9775/3265
vtfs-auto-dax-1G randrw-psync 15.0mb/5134kb 3855/1283
vtfs-auto-dax-2G randrw-psync 19.0mb/6490kb 4866/1622
vtfs-auto-dax-4G randrw-psync 41.9mb/14.0mb 10.4k/3595
vtfs-auto-dax-8G randrw-psync 422.2mb/141.1mb 105.5k/35.2k
vtfs-auto-dax-16G randrw-psync 395.9mb/132.3mb 98.9k/33.0k
vtfs-auto-dax-32G randrw-psync 378.2mb/126.4mb 94.5k/31.5k
vtfs-auto-dax-64G randrw-psync 419.2mb/140.1mb 104.8k/35.0k
vtfs-auto randrw-psync-multi 151.4mb/50.7mb 37.8k/12.6k
vtfs-auto-dax-1G randrw-psync-multi 23.5mb/8091kb 6020/2022
vtfs-auto-dax-2G randrw-psync-multi 24.7mb/8515kb 6343/2128
vtfs-auto-dax-4G randrw-psync-multi 27.9mb/9606kb 7148/2401
vtfs-auto-dax-8G randrw-psync-multi 38.6mb/12.9mb 9882/3309
vtfs-auto-dax-16G randrw-psync-multi 277.7mb/93.0mb 69.4k/23.2k
vtfs-auto-dax-32G randrw-psync-multi 1233.6mb/412.5mb308.4k/103.1k
vtfs-auto-dax-64G randrw-psync-multi 1447.4mb/483.9mb361.8k/120.9k
vtfs-auto randrw-mmap 44.4mb/14.8mb 11.1k/3806
vtfs-auto-dax-1G randrw-mmap 11.6mb/3978kb 2980/994
vtfs-auto-dax-2G randrw-mmap 15.0mb/5133kb 3854/1283
vtfs-auto-dax-4G randrw-mmap 36.1mb/12.0mb 9247/3088
vtfs-auto-dax-8G randrw-mmap 618.2mb/206.6mb 154.5k/51.6k
vtfs-auto-dax-16G randrw-mmap 638.2mb/213.3mb 159.5k/53.3k
vtfs-auto-dax-32G randrw-mmap 554.6mb/185.3mb 138.6k/46.3k
vtfs-auto-dax-64G randrw-mmap 608.5mb/203.3mb 152.1k/50.8k
vtfs-auto randrw-mmap-multi 179.3mb/60.0mb 44.8k/15.0k
vtfs-auto-dax-1G randrw-mmap-multi 11.7mb/4069kb 3007/1017
vtfs-auto-dax-2G randrw-mmap-multi 14.7mb/5088kb 3768/1272
vtfs-auto-dax-4G randrw-mmap-multi 16.8mb/5806kb 4307/1451
vtfs-auto-dax-8G randrw-mmap-multi 24.7mb/8515kb 6341/2128
vtfs-auto-dax-16G randrw-mmap-multi 479.4mb/160.4mb 119.8k/40.1k
vtfs-auto-dax-32G randrw-mmap-multi 1290.1mb/431.3mb322.5k/107.8k
vtfs-auto-dax-64G randrw-mmap-multi 1828.4mb/611.4mb457.1k/152.8k
vtfs-auto randrw-libaio 83.4mb/27.9mb 20.8k/7161
vtfs-auto-dax-1G randrw-libaio 13.9mb/4725kb 3565/1181
vtfs-auto-dax-2G randrw-libaio 17.1mb/5866kb 4396/1466
vtfs-auto-dax-4G randrw-libaio 38.1mb/12.7mb 9756/3259
vtfs-auto-dax-8G randrw-libaio 253.0mb/84.5mb 63.2k/21.1k
vtfs-auto-dax-16G randrw-libaio 252.5mb/84.4mb 63.1k/21.1k
vtfs-auto-dax-32G randrw-libaio 239.9mb/80.1mb 59.9k/20.0k
vtfs-auto-dax-64G randrw-libaio 255.7mb/85.4mb 63.9k/21.3k
vtfs-auto randrw-libaio-multi 209.6mb/70.2mb 52.4k/17.5k
vtfs-auto-dax-1G randrw-libaio-multi 23.8mb/8197kb 6105/2049
vtfs-auto-dax-2G randrw-libaio-multi 21.5mb/7418kb 5507/1854
vtfs-auto-dax-4G randrw-libaio-multi 28.0mb/9651kb 7186/2412
vtfs-auto-dax-8G randrw-libaio-multi 34.6mb/11.6mb 8873/2976
vtfs-auto-dax-16G randrw-libaio-multi 197.4mb/66.1mb 49.3k/16.5k
vtfs-auto-dax-32G randrw-libaio-multi 633.8mb/211.9mb 158.4k/52.9k
vtfs-auto-dax-64G randrw-libaio-multi 943.8mb/315.5mb 235.9k/78.8k
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Virtio-fs] Some performance numbers for virtiofs, DAX and virtio-9p
@ 2020-12-11 18:47 ` Vivek Goyal
0 siblings, 0 replies; 20+ messages in thread
From: Vivek Goyal @ 2020-12-11 18:47 UTC (permalink / raw)
To: Miklos Szeredi
Cc: Venegas Munoz, Jose Carlos, Christian Schoenebeck,
QEMU Developers, virtio-fs-list
On Fri, Dec 11, 2020 at 11:06:03AM -0500, Vivek Goyal wrote:
[..]
> >
> > Could we measure at what point does a large window size actually make
> > performance worse?
>
> Will do. Will run tests with varying window sizes (small to large)
> and see how does it impact performance for same workload with
> same guest memory.
>
Ok, I ran some tests with virtiofs cache=auto and varied dax window
size from 1G to 64G. (1, 2, 4, 8, 16, 32, 64). Still using file size
4G each. And -multi test use 4 files of 4G each. Here are the test
results.
To run tests faster, I ran one iteration of each job (instead of running
3 iterations and taking average). This should be good enough to give
some idea.
I have also pushed raw data file and test results here.
https://github.com/rhvgoyal/virtiofs-tests/tree/master/performance-results/dec-10-2020
First line is vtfs cache auto without dax to see with what dax window
size dax helps as opposed to without dax.
Conclusion
----------
- Decreasing window size hurts tremendously for random read and random
write workloads.
- seqread/seqwrite still perform reasonably well even with smaller
window size.
- Increasing DAX window does not hurt.
I will try to spend some time with smaller dax window and random read
and ramdom write workloads and see if there is anything I can do to
reduce the performance drop.
Thanks
Vivek
NAME WORKLOAD Bandwidth IOPS
vtfs-auto seqread-psync 650.0mb 162.5k
vtfs-auto-dax-1G seqread-psync 653.1mb 163.2k
vtfs-auto-dax-2G seqread-psync 668.2mb 167.0k
vtfs-auto-dax-4G seqread-psync 657.8mb 164.4k
vtfs-auto-dax-8G seqread-psync 699.5mb 174.8k
vtfs-auto-dax-16G seqread-psync 674.3mb 168.5k
vtfs-auto-dax-32G seqread-psync 667.6mb 166.9k
vtfs-auto-dax-64G seqread-psync 696.1mb 174.0k
vtfs-auto seqread-psync-multi 2530.7mb 632.6k
vtfs-auto-dax-1G seqread-psync-multi 1931.1mb 482.7k
vtfs-auto-dax-2G seqread-psync-multi 2105.6mb 526.4k
vtfs-auto-dax-4G seqread-psync-multi 2142.2mb 535.5k
vtfs-auto-dax-8G seqread-psync-multi 2229.7mb 557.4k
vtfs-auto-dax-16G seqread-psync-multi 2324.3mb 581.0k
vtfs-auto-dax-32G seqread-psync-multi 2421.1mb 605.2k
vtfs-auto-dax-64G seqread-psync-multi 2469.7mb 617.4k
vtfs-auto seqread-mmap 570.6mb 142.6k
vtfs-auto-dax-1G seqread-mmap 823.9mb 205.9k
vtfs-auto-dax-2G seqread-mmap 798.1mb 199.5k
vtfs-auto-dax-4G seqread-mmap 874.4mb 218.6k
vtfs-auto-dax-8G seqread-mmap 960.5mb 240.1k
vtfs-auto-dax-16G seqread-mmap 917.1mb 229.2k
vtfs-auto-dax-32G seqread-mmap 884.4mb 221.1k
vtfs-auto-dax-64G seqread-mmap 855.6mb 213.9k
vtfs-auto seqread-mmap-multi 2210.1mb 552.5k
vtfs-auto-dax-1G seqread-mmap-multi 2722.9mb 680.7k
vtfs-auto-dax-2G seqread-mmap-multi 2828.2mb 707.0k
vtfs-auto-dax-4G seqread-mmap-multi 2777.4mb 694.3k
vtfs-auto-dax-8G seqread-mmap-multi 2880.4mb 720.1k
vtfs-auto-dax-16G seqread-mmap-multi 3159.2mb 789.8k
vtfs-auto-dax-32G seqread-mmap-multi 2989.2mb 747.3k
vtfs-auto-dax-64G seqread-mmap-multi 3007.3mb 751.8k
vtfs-auto seqread-libaio 303.3mb 75.8k
vtfs-auto-dax-1G seqread-libaio 396.4mb 99.1k
vtfs-auto-dax-2G seqread-libaio 391.9mb 97.9k
vtfs-auto-dax-4G seqread-libaio 422.9mb 105.7k
vtfs-auto-dax-8G seqread-libaio 419.0mb 104.7k
vtfs-auto-dax-16G seqread-libaio 411.2mb 102.8k
vtfs-auto-dax-32G seqread-libaio 436.1mb 109.0k
vtfs-auto-dax-64G seqread-libaio 418.0mb 104.5k
vtfs-auto seqread-libaio-multi 1500.0mb 375.0k
vtfs-auto-dax-1G seqread-libaio-multi 1271.7mb 317.9k
vtfs-auto-dax-2G seqread-libaio-multi 1338.1mb 334.5k
vtfs-auto-dax-4G seqread-libaio-multi 1306.6mb 326.6k
vtfs-auto-dax-8G seqread-libaio-multi 1416.8mb 354.2k
vtfs-auto-dax-16G seqread-libaio-multi 1461.9mb 365.4k
vtfs-auto-dax-32G seqread-libaio-multi 1531.9mb 382.9k
vtfs-auto-dax-64G seqread-libaio-multi 1495.4mb 373.8k
vtfs-auto randread-psync 77.9mb 19.4k
vtfs-auto-dax-1G randread-psync 18.7mb 4792
vtfs-auto-dax-2G randread-psync 25.2mb 6463
vtfs-auto-dax-4G randread-psync 59.9mb 14.9k
vtfs-auto-dax-8G randread-psync 693.7mb 173.4k
vtfs-auto-dax-16G randread-psync 646.0mb 161.5k
vtfs-auto-dax-32G randread-psync 635.4mb 158.8k
vtfs-auto-dax-64G randread-psync 663.4mb 165.8k
vtfs-auto randread-psync-multi 285.2mb 71.3k
vtfs-auto-dax-1G randread-psync-multi 28.7mb 7357
vtfs-auto-dax-2G randread-psync-multi 29.6mb 7596
vtfs-auto-dax-4G randread-psync-multi 35.0mb 8969
vtfs-auto-dax-8G randread-psync-multi 51.3mb 12.8k
vtfs-auto-dax-16G randread-psync-multi 497.1mb 124.2k
vtfs-auto-dax-32G randread-psync-multi 2092.9mb 523.2k
vtfs-auto-dax-64G randread-psync-multi 2051.0mb 512.7k
vtfs-auto randread-mmap 57.7mb 14.4k
vtfs-auto-dax-1G randread-mmap 15.1mb 3889
vtfs-auto-dax-2G randread-mmap 21.5mb 5506
vtfs-auto-dax-4G randread-mmap 59.4mb 14.8k
vtfs-auto-dax-8G randread-mmap 970.1mb 242.5k
vtfs-auto-dax-16G randread-mmap 883.1mb 220.7k
vtfs-auto-dax-32G randread-mmap 874.2mb 218.5k
vtfs-auto-dax-64G randread-mmap 954.3mb 238.5k
vtfs-auto randread-mmap-multi 243.5mb 60.8k
vtfs-auto-dax-1G randread-mmap-multi 11.7mb 3005
vtfs-auto-dax-2G randread-mmap-multi 20.4mb 5232
vtfs-auto-dax-4G randread-mmap-multi 23.1mb 5937
vtfs-auto-dax-8G randread-mmap-multi 35.3mb 9058
vtfs-auto-dax-16G randread-mmap-multi 1009.8mb 252.4k
vtfs-auto-dax-32G randread-mmap-multi 2612.6mb 653.1k
vtfs-auto-dax-64G randread-mmap-multi 2908.5mb 727.1k
vtfs-auto randread-libaio 276.2mb 69.0k
vtfs-auto-dax-1G randread-libaio 19.4mb 4968
vtfs-auto-dax-2G randread-libaio 23.3mb 5981
vtfs-auto-dax-4G randread-libaio 60.6mb 15.1k
vtfs-auto-dax-8G randread-libaio 413.5mb 103.3k
vtfs-auto-dax-16G randread-libaio 410.8mb 102.7k
vtfs-auto-dax-32G randread-libaio 392.0mb 98.0k
vtfs-auto-dax-64G randread-libaio 412.9mb 103.2k
vtfs-auto randread-libaio-multi 271.0mb 67.7k
vtfs-auto-dax-1G randread-libaio-multi 29.2mb 7479
vtfs-auto-dax-2G randread-libaio-multi 30.4mb 7803
vtfs-auto-dax-4G randread-libaio-multi 36.6mb 9387
vtfs-auto-dax-8G randread-libaio-multi 49.0mb 12.2k
vtfs-auto-dax-16G randread-libaio-multi 364.3mb 91.0k
vtfs-auto-dax-32G randread-libaio-multi 1435.0mb 358.7k
vtfs-auto-dax-64G randread-libaio-multi 1358.4mb 339.6k
vtfs-auto seqwrite-psync 78.6mb 19.6k
vtfs-auto-dax-1G seqwrite-psync 440.7mb 110.1k
vtfs-auto-dax-2G seqwrite-psync 428.6mb 107.1k
vtfs-auto-dax-4G seqwrite-psync 476.5mb 119.1k
vtfs-auto-dax-8G seqwrite-psync 547.8mb 136.9k
vtfs-auto-dax-16G seqwrite-psync 530.1mb 132.5k
vtfs-auto-dax-32G seqwrite-psync 486.6mb 121.6k
vtfs-auto-dax-64G seqwrite-psync 515.8mb 128.9k
vtfs-auto seqwrite-psync-multi 305.7mb 76.4k
vtfs-auto-dax-1G seqwrite-psync-multi 1405.0mb 351.2k
vtfs-auto-dax-2G seqwrite-psync-multi 1447.0mb 361.7k
vtfs-auto-dax-4G seqwrite-psync-multi 1329.4mb 332.3k
vtfs-auto-dax-8G seqwrite-psync-multi 1250.4mb 312.5k
vtfs-auto-dax-16G seqwrite-psync-multi 1264.7mb 316.1k
vtfs-auto-dax-32G seqwrite-psync-multi 1224.5mb 306.1k
vtfs-auto-dax-64G seqwrite-psync-multi 1165.6mb 291.4k
vtfs-auto seqwrite-mmap 199.7mb 49.9k
vtfs-auto-dax-1G seqwrite-mmap 1262.6mb 315.6k
vtfs-auto-dax-2G seqwrite-mmap 1369.4mb 342.3k
vtfs-auto-dax-4G seqwrite-mmap 1363.5mb 340.8k
vtfs-auto-dax-8G seqwrite-mmap 1368.9mb 342.2k
vtfs-auto-dax-16G seqwrite-mmap 1305.2mb 326.3k
vtfs-auto-dax-32G seqwrite-mmap 1299.4mb 324.8k
vtfs-auto-dax-64G seqwrite-mmap 1301.1mb 325.2k
vtfs-auto seqwrite-mmap-multi 384.1mb 96.0k
vtfs-auto-dax-1G seqwrite-mmap-multi 2162.9mb 540.7k
vtfs-auto-dax-2G seqwrite-mmap-multi 2191.8mb 547.9k
vtfs-auto-dax-4G seqwrite-mmap-multi 2045.1mb 511.2k
vtfs-auto-dax-8G seqwrite-mmap-multi 1973.0mb 493.2k
vtfs-auto-dax-16G seqwrite-mmap-multi 1911.3mb 477.8k
vtfs-auto-dax-32G seqwrite-mmap-multi 2263.9mb 565.9k
vtfs-auto-dax-64G seqwrite-mmap-multi 1718.4mb 429.6k
vtfs-auto seqwrite-libaio 279.1mb 69.7k
vtfs-auto-dax-1G seqwrite-libaio 294.2mb 73.5k
vtfs-auto-dax-2G seqwrite-libaio 308.7mb 77.1k
vtfs-auto-dax-4G seqwrite-libaio 308.3mb 77.0k
vtfs-auto-dax-8G seqwrite-libaio 339.0mb 84.7k
vtfs-auto-dax-16G seqwrite-libaio 323.3mb 80.8k
vtfs-auto-dax-32G seqwrite-libaio 311.4mb 77.8k
vtfs-auto-dax-64G seqwrite-libaio 334.2mb 83.5k
vtfs-auto seqwrite-libaio-multi 306.4mb 76.6k
vtfs-auto-dax-1G seqwrite-libaio-multi 968.7mb 242.1k
vtfs-auto-dax-2G seqwrite-libaio-multi 1050.7mb 262.6k
vtfs-auto-dax-4G seqwrite-libaio-multi 989.4mb 247.3k
vtfs-auto-dax-8G seqwrite-libaio-multi 1017.8mb 254.4k
vtfs-auto-dax-16G seqwrite-libaio-multi 1101.5mb 275.3k
vtfs-auto-dax-32G seqwrite-libaio-multi 915.0mb 228.7k
vtfs-auto-dax-64G seqwrite-libaio-multi 1067.5mb 266.8k
vtfs-auto randwrite-psync 82.2mb 20.5k
vtfs-auto-dax-1G randwrite-psync 20.4mb 5240
vtfs-auto-dax-2G randwrite-psync 29.6mb 7585
vtfs-auto-dax-4G randwrite-psync 65.4mb 16.3k
vtfs-auto-dax-8G randwrite-psync 502.6mb 125.6k
vtfs-auto-dax-16G randwrite-psync 501.6mb 125.4k
vtfs-auto-dax-32G randwrite-psync 461.6mb 115.4k
vtfs-auto-dax-64G randwrite-psync 481.2mb 120.2k
vtfs-auto randwrite-psync-multi 302.1mb 75.5k
vtfs-auto-dax-1G randwrite-psync-multi 30.9mb 7915
vtfs-auto-dax-2G randwrite-psync-multi 35.0mb 8968
vtfs-auto-dax-4G randwrite-psync-multi 36.7mb 9399
vtfs-auto-dax-8G randwrite-psync-multi 57.1mb 14.2k
vtfs-auto-dax-16G randwrite-psync-multi 499.0mb 124.7k
vtfs-auto-dax-32G randwrite-psync-multi 1123.0mb 280.7k
vtfs-auto-dax-64G randwrite-psync-multi 1058.3mb 264.5k
vtfs-auto randwrite-mmap 55.9mb 13.9k
vtfs-auto-dax-1G randwrite-mmap 16.4mb 4221
vtfs-auto-dax-2G randwrite-mmap 22.0mb 5648
vtfs-auto-dax-4G randwrite-mmap 60.6mb 15.1k
vtfs-auto-dax-8G randwrite-mmap 763.3mb 190.8k
vtfs-auto-dax-16G randwrite-mmap 859.2mb 214.8k
vtfs-auto-dax-32G randwrite-mmap 792.5mb 198.1k
vtfs-auto-dax-64G randwrite-mmap 807.7mb 201.9k
vtfs-auto randwrite-mmap-multi 208.9mb 52.2k
vtfs-auto-dax-1G randwrite-mmap-multi 12.4mb 3191
vtfs-auto-dax-2G randwrite-mmap-multi 19.8mb 5072
vtfs-auto-dax-4G randwrite-mmap-multi 22.3mb 5710
vtfs-auto-dax-8G randwrite-mmap-multi 33.2mb 8511
vtfs-auto-dax-16G randwrite-mmap-multi 1061.2mb 265.3k
vtfs-auto-dax-32G randwrite-mmap-multi 2113.2mb 528.3k
vtfs-auto-dax-64G randwrite-mmap-multi 2099.4mb 524.8k
vtfs-auto randwrite-libaio 258.7mb 64.6k
vtfs-auto-dax-1G randwrite-libaio 20.4mb 5231
vtfs-auto-dax-2G randwrite-libaio 26.4mb 6774
vtfs-auto-dax-4G randwrite-libaio 64.9mb 16.2k
vtfs-auto-dax-8G randwrite-libaio 326.7mb 81.6k
vtfs-auto-dax-16G randwrite-libaio 303.6mb 75.9k
vtfs-auto-dax-32G randwrite-libaio 298.2mb 74.5k
vtfs-auto-dax-64G randwrite-libaio 328.5mb 82.1k
vtfs-auto randwrite-libaio-multi 320.9mb 80.2k
vtfs-auto-dax-1G randwrite-libaio-multi 31.1mb 7977
vtfs-auto-dax-2G randwrite-libaio-multi 34.2mb 8772
vtfs-auto-dax-4G randwrite-libaio-multi 38.8mb 9945
vtfs-auto-dax-8G randwrite-libaio-multi 55.9mb 13.9k
vtfs-auto-dax-16G randwrite-libaio-multi 363.2mb 90.8k
vtfs-auto-dax-32G randwrite-libaio-multi 800.4mb 200.1k
vtfs-auto-dax-64G randwrite-libaio-multi 805.5mb 201.3k
vtfs-auto randrw-psync 38.1mb/12.7mb 9775/3265
vtfs-auto-dax-1G randrw-psync 15.0mb/5134kb 3855/1283
vtfs-auto-dax-2G randrw-psync 19.0mb/6490kb 4866/1622
vtfs-auto-dax-4G randrw-psync 41.9mb/14.0mb 10.4k/3595
vtfs-auto-dax-8G randrw-psync 422.2mb/141.1mb 105.5k/35.2k
vtfs-auto-dax-16G randrw-psync 395.9mb/132.3mb 98.9k/33.0k
vtfs-auto-dax-32G randrw-psync 378.2mb/126.4mb 94.5k/31.5k
vtfs-auto-dax-64G randrw-psync 419.2mb/140.1mb 104.8k/35.0k
vtfs-auto randrw-psync-multi 151.4mb/50.7mb 37.8k/12.6k
vtfs-auto-dax-1G randrw-psync-multi 23.5mb/8091kb 6020/2022
vtfs-auto-dax-2G randrw-psync-multi 24.7mb/8515kb 6343/2128
vtfs-auto-dax-4G randrw-psync-multi 27.9mb/9606kb 7148/2401
vtfs-auto-dax-8G randrw-psync-multi 38.6mb/12.9mb 9882/3309
vtfs-auto-dax-16G randrw-psync-multi 277.7mb/93.0mb 69.4k/23.2k
vtfs-auto-dax-32G randrw-psync-multi 1233.6mb/412.5mb308.4k/103.1k
vtfs-auto-dax-64G randrw-psync-multi 1447.4mb/483.9mb361.8k/120.9k
vtfs-auto randrw-mmap 44.4mb/14.8mb 11.1k/3806
vtfs-auto-dax-1G randrw-mmap 11.6mb/3978kb 2980/994
vtfs-auto-dax-2G randrw-mmap 15.0mb/5133kb 3854/1283
vtfs-auto-dax-4G randrw-mmap 36.1mb/12.0mb 9247/3088
vtfs-auto-dax-8G randrw-mmap 618.2mb/206.6mb 154.5k/51.6k
vtfs-auto-dax-16G randrw-mmap 638.2mb/213.3mb 159.5k/53.3k
vtfs-auto-dax-32G randrw-mmap 554.6mb/185.3mb 138.6k/46.3k
vtfs-auto-dax-64G randrw-mmap 608.5mb/203.3mb 152.1k/50.8k
vtfs-auto randrw-mmap-multi 179.3mb/60.0mb 44.8k/15.0k
vtfs-auto-dax-1G randrw-mmap-multi 11.7mb/4069kb 3007/1017
vtfs-auto-dax-2G randrw-mmap-multi 14.7mb/5088kb 3768/1272
vtfs-auto-dax-4G randrw-mmap-multi 16.8mb/5806kb 4307/1451
vtfs-auto-dax-8G randrw-mmap-multi 24.7mb/8515kb 6341/2128
vtfs-auto-dax-16G randrw-mmap-multi 479.4mb/160.4mb 119.8k/40.1k
vtfs-auto-dax-32G randrw-mmap-multi 1290.1mb/431.3mb322.5k/107.8k
vtfs-auto-dax-64G randrw-mmap-multi 1828.4mb/611.4mb457.1k/152.8k
vtfs-auto randrw-libaio 83.4mb/27.9mb 20.8k/7161
vtfs-auto-dax-1G randrw-libaio 13.9mb/4725kb 3565/1181
vtfs-auto-dax-2G randrw-libaio 17.1mb/5866kb 4396/1466
vtfs-auto-dax-4G randrw-libaio 38.1mb/12.7mb 9756/3259
vtfs-auto-dax-8G randrw-libaio 253.0mb/84.5mb 63.2k/21.1k
vtfs-auto-dax-16G randrw-libaio 252.5mb/84.4mb 63.1k/21.1k
vtfs-auto-dax-32G randrw-libaio 239.9mb/80.1mb 59.9k/20.0k
vtfs-auto-dax-64G randrw-libaio 255.7mb/85.4mb 63.9k/21.3k
vtfs-auto randrw-libaio-multi 209.6mb/70.2mb 52.4k/17.5k
vtfs-auto-dax-1G randrw-libaio-multi 23.8mb/8197kb 6105/2049
vtfs-auto-dax-2G randrw-libaio-multi 21.5mb/7418kb 5507/1854
vtfs-auto-dax-4G randrw-libaio-multi 28.0mb/9651kb 7186/2412
vtfs-auto-dax-8G randrw-libaio-multi 34.6mb/11.6mb 8873/2976
vtfs-auto-dax-16G randrw-libaio-multi 197.4mb/66.1mb 49.3k/16.5k
vtfs-auto-dax-32G randrw-libaio-multi 633.8mb/211.9mb 158.4k/52.9k
vtfs-auto-dax-64G randrw-libaio-multi 943.8mb/315.5mb 235.9k/78.8k
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Some performance numbers for virtiofs, DAX and virtio-9p
2020-12-11 18:29 ` [Virtio-fs] " Dr. David Alan Gilbert
@ 2020-12-11 19:25 ` Vivek Goyal
-1 siblings, 0 replies; 20+ messages in thread
From: Vivek Goyal @ 2020-12-11 19:25 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Miklos Szeredi, Venegas Munoz, Jose Carlos,
Christian Schoenebeck, QEMU Developers, virtio-fs-list,
Stefan Hajnoczi
On Fri, Dec 11, 2020 at 06:29:56PM +0000, Dr. David Alan Gilbert wrote:
[..]
> > >
> > > Could we measure at what point does a large window size actually make
> > > performance worse?
> >
> > Will do. Will run tests with varying window sizes (small to large)
> > and see how does it impact performance for same workload with
> > same guest memory.
>
> I wonder how realistic it is though; it makes some sense if you have a
> scenario like a fairly small root filesystem - something tractable; but
> if you have a large FS you're not realistically going to be able to set
> the cache size to match it - that's why it's a cache!
I think its more about active dataset size and not necessarily total
FS size. FS might be big but if application is not accessing all of
the in reasonabl small time window, then it does not matter.
What worries me most is that cost of reclaim of a dax range seems
too high (or keeps the process blocked for long enogh), that it
kills the performance. I will need to revisit the reclaim path
and see if I can optimize something.
Vivek
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Virtio-fs] Some performance numbers for virtiofs, DAX and virtio-9p
@ 2020-12-11 19:25 ` Vivek Goyal
0 siblings, 0 replies; 20+ messages in thread
From: Vivek Goyal @ 2020-12-11 19:25 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Venegas Munoz, Jose Carlos, Christian Schoenebeck,
QEMU Developers, virtio-fs-list
On Fri, Dec 11, 2020 at 06:29:56PM +0000, Dr. David Alan Gilbert wrote:
[..]
> > >
> > > Could we measure at what point does a large window size actually make
> > > performance worse?
> >
> > Will do. Will run tests with varying window sizes (small to large)
> > and see how does it impact performance for same workload with
> > same guest memory.
>
> I wonder how realistic it is though; it makes some sense if you have a
> scenario like a fairly small root filesystem - something tractable; but
> if you have a large FS you're not realistically going to be able to set
> the cache size to match it - that's why it's a cache!
I think its more about active dataset size and not necessarily total
FS size. FS might be big but if application is not accessing all of
the in reasonabl small time window, then it does not matter.
What worries me most is that cost of reclaim of a dax range seems
too high (or keeps the process blocked for long enogh), that it
kills the performance. I will need to revisit the reclaim path
and see if I can optimize something.
Vivek
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Some performance numbers for virtiofs, DAX and virtio-9p
2020-12-11 19:25 ` [Virtio-fs] " Vivek Goyal
@ 2020-12-11 20:01 ` Vivek Goyal
-1 siblings, 0 replies; 20+ messages in thread
From: Vivek Goyal @ 2020-12-11 20:01 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Miklos Szeredi, Venegas Munoz, Jose Carlos,
Christian Schoenebeck, QEMU Developers, virtio-fs-list,
Stefan Hajnoczi
On Fri, Dec 11, 2020 at 02:25:17PM -0500, Vivek Goyal wrote:
> On Fri, Dec 11, 2020 at 06:29:56PM +0000, Dr. David Alan Gilbert wrote:
>
> [..]
> > > >
> > > > Could we measure at what point does a large window size actually make
> > > > performance worse?
> > >
> > > Will do. Will run tests with varying window sizes (small to large)
> > > and see how does it impact performance for same workload with
> > > same guest memory.
> >
> > I wonder how realistic it is though; it makes some sense if you have a
> > scenario like a fairly small root filesystem - something tractable; but
> > if you have a large FS you're not realistically going to be able to set
> > the cache size to match it - that's why it's a cache!
>
> I think its more about active dataset size and not necessarily total
> FS size. FS might be big but if application is not accessing all of
> the in reasonabl small time window, then it does not matter.
>
> What worries me most is that cost of reclaim of a dax range seems
> too high (or keeps the process blocked for long enogh), that it
> kills the performance. I will need to revisit the reclaim path
> and see if I can optimize something.
I see that while reclaiming a range, we are sending a remvemapping
command to virtiofsd. We are holding locks so that no new mappings
can be added to that inode.
We had decided that removemapping is not necessary. It helps in
the sense that if guest is not using a mapping, qemu will unmap
it too.
If we stop sending remove mapping, then it might improve reclaim
performance significantly. With the downside that host will
have something mapped (despite the fact that guest is not using
it anymore). And these will be cleaned up only when guest shuts
down.
Vivek
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Virtio-fs] Some performance numbers for virtiofs, DAX and virtio-9p
@ 2020-12-11 20:01 ` Vivek Goyal
0 siblings, 0 replies; 20+ messages in thread
From: Vivek Goyal @ 2020-12-11 20:01 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Venegas Munoz, Jose Carlos, Christian Schoenebeck,
QEMU Developers, virtio-fs-list
On Fri, Dec 11, 2020 at 02:25:17PM -0500, Vivek Goyal wrote:
> On Fri, Dec 11, 2020 at 06:29:56PM +0000, Dr. David Alan Gilbert wrote:
>
> [..]
> > > >
> > > > Could we measure at what point does a large window size actually make
> > > > performance worse?
> > >
> > > Will do. Will run tests with varying window sizes (small to large)
> > > and see how does it impact performance for same workload with
> > > same guest memory.
> >
> > I wonder how realistic it is though; it makes some sense if you have a
> > scenario like a fairly small root filesystem - something tractable; but
> > if you have a large FS you're not realistically going to be able to set
> > the cache size to match it - that's why it's a cache!
>
> I think its more about active dataset size and not necessarily total
> FS size. FS might be big but if application is not accessing all of
> the in reasonabl small time window, then it does not matter.
>
> What worries me most is that cost of reclaim of a dax range seems
> too high (or keeps the process blocked for long enogh), that it
> kills the performance. I will need to revisit the reclaim path
> and see if I can optimize something.
I see that while reclaiming a range, we are sending a remvemapping
command to virtiofsd. We are holding locks so that no new mappings
can be added to that inode.
We had decided that removemapping is not necessary. It helps in
the sense that if guest is not using a mapping, qemu will unmap
it too.
If we stop sending remove mapping, then it might improve reclaim
performance significantly. With the downside that host will
have something mapped (despite the fact that guest is not using
it anymore). And these will be cleaned up only when guest shuts
down.
Vivek
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Some performance numbers for virtiofs, DAX and virtio-9p
2020-12-11 20:01 ` [Virtio-fs] " Vivek Goyal
@ 2020-12-11 20:06 ` Dr. David Alan Gilbert
-1 siblings, 0 replies; 20+ messages in thread
From: Dr. David Alan Gilbert @ 2020-12-11 20:06 UTC (permalink / raw)
To: Vivek Goyal
Cc: Miklos Szeredi, Venegas Munoz, Jose Carlos,
Christian Schoenebeck, QEMU Developers, virtio-fs-list,
Stefan Hajnoczi
* Vivek Goyal (vgoyal@redhat.com) wrote:
> On Fri, Dec 11, 2020 at 02:25:17PM -0500, Vivek Goyal wrote:
> > On Fri, Dec 11, 2020 at 06:29:56PM +0000, Dr. David Alan Gilbert wrote:
> >
> > [..]
> > > > >
> > > > > Could we measure at what point does a large window size actually make
> > > > > performance worse?
> > > >
> > > > Will do. Will run tests with varying window sizes (small to large)
> > > > and see how does it impact performance for same workload with
> > > > same guest memory.
> > >
> > > I wonder how realistic it is though; it makes some sense if you have a
> > > scenario like a fairly small root filesystem - something tractable; but
> > > if you have a large FS you're not realistically going to be able to set
> > > the cache size to match it - that's why it's a cache!
> >
> > I think its more about active dataset size and not necessarily total
> > FS size. FS might be big but if application is not accessing all of
> > the in reasonabl small time window, then it does not matter.
> >
> > What worries me most is that cost of reclaim of a dax range seems
> > too high (or keeps the process blocked for long enogh), that it
> > kills the performance. I will need to revisit the reclaim path
> > and see if I can optimize something.
>
> I see that while reclaiming a range, we are sending a remvemapping
> command to virtiofsd. We are holding locks so that no new mappings
> can be added to that inode.
Are you doing this when mapping something new in as well?
If you're replacing an existing mapping then there's no downside to just
skipping the remove.
> We had decided that removemapping is not necessary. It helps in
> the sense that if guest is not using a mapping, qemu will unmap
> it too.
>
> If we stop sending remove mapping, then it might improve reclaim
> performance significantly. With the downside that host will
> have something mapped (despite the fact that guest is not using
> it anymore). And these will be cleaned up only when guest shuts
> down.
There's probably some scope for improving this on the QEMU side as well;
although there's always going to be latency of going through the path
to do the request.
Dave
>
> Vivek
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Virtio-fs] Some performance numbers for virtiofs, DAX and virtio-9p
@ 2020-12-11 20:06 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 20+ messages in thread
From: Dr. David Alan Gilbert @ 2020-12-11 20:06 UTC (permalink / raw)
To: Vivek Goyal
Cc: Venegas Munoz, Jose Carlos, Christian Schoenebeck,
QEMU Developers, virtio-fs-list
* Vivek Goyal (vgoyal@redhat.com) wrote:
> On Fri, Dec 11, 2020 at 02:25:17PM -0500, Vivek Goyal wrote:
> > On Fri, Dec 11, 2020 at 06:29:56PM +0000, Dr. David Alan Gilbert wrote:
> >
> > [..]
> > > > >
> > > > > Could we measure at what point does a large window size actually make
> > > > > performance worse?
> > > >
> > > > Will do. Will run tests with varying window sizes (small to large)
> > > > and see how does it impact performance for same workload with
> > > > same guest memory.
> > >
> > > I wonder how realistic it is though; it makes some sense if you have a
> > > scenario like a fairly small root filesystem - something tractable; but
> > > if you have a large FS you're not realistically going to be able to set
> > > the cache size to match it - that's why it's a cache!
> >
> > I think its more about active dataset size and not necessarily total
> > FS size. FS might be big but if application is not accessing all of
> > the in reasonabl small time window, then it does not matter.
> >
> > What worries me most is that cost of reclaim of a dax range seems
> > too high (or keeps the process blocked for long enogh), that it
> > kills the performance. I will need to revisit the reclaim path
> > and see if I can optimize something.
>
> I see that while reclaiming a range, we are sending a remvemapping
> command to virtiofsd. We are holding locks so that no new mappings
> can be added to that inode.
Are you doing this when mapping something new in as well?
If you're replacing an existing mapping then there's no downside to just
skipping the remove.
> We had decided that removemapping is not necessary. It helps in
> the sense that if guest is not using a mapping, qemu will unmap
> it too.
>
> If we stop sending remove mapping, then it might improve reclaim
> performance significantly. With the downside that host will
> have something mapped (despite the fact that guest is not using
> it anymore). And these will be cleaned up only when guest shuts
> down.
There's probably some scope for improving this on the QEMU side as well;
although there's always going to be latency of going through the path
to do the request.
Dave
>
> Vivek
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Some performance numbers for virtiofs, DAX and virtio-9p
2020-12-11 19:25 ` [Virtio-fs] " Vivek Goyal
@ 2021-01-05 15:08 ` Miklos Szeredi
-1 siblings, 0 replies; 20+ messages in thread
From: Miklos Szeredi @ 2021-01-05 15:08 UTC (permalink / raw)
To: Vivek Goyal
Cc: Venegas Munoz, Jose Carlos, Christian Schoenebeck,
Dr. David Alan Gilbert, QEMU Developers, virtio-fs-list,
Stefan Hajnoczi
On Fri, Dec 11, 2020 at 8:25 PM Vivek Goyal <vgoyal@redhat.com> wrote:
>
> On Fri, Dec 11, 2020 at 06:29:56PM +0000, Dr. David Alan Gilbert wrote:
>
> [..]
> > > >
> > > > Could we measure at what point does a large window size actually make
> > > > performance worse?
> > >
> > > Will do. Will run tests with varying window sizes (small to large)
> > > and see how does it impact performance for same workload with
> > > same guest memory.
> >
> > I wonder how realistic it is though; it makes some sense if you have a
> > scenario like a fairly small root filesystem - something tractable; but
> > if you have a large FS you're not realistically going to be able to set
> > the cache size to match it - that's why it's a cache!
>
> I think its more about active dataset size and not necessarily total
> FS size. FS might be big but if application is not accessing all of
> the in reasonabl small time window, then it does not matter.
>
> What worries me most is that cost of reclaim of a dax range seems
> too high (or keeps the process blocked for long enogh), that it
> kills the performance. I will need to revisit the reclaim path
> and see if I can optimize something.
Can we dynamically increase the dax window size? Or hot plug
additional dax ranges on demand?
That would solve the problem of trying to find the active set size in advance.
We'd still need reclaim in order to prevent the window from growing
indefinitely, but that would always be background reclaim and be done
after some time of inactivity.
Thanks,
Miklos
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Virtio-fs] Some performance numbers for virtiofs, DAX and virtio-9p
@ 2021-01-05 15:08 ` Miklos Szeredi
0 siblings, 0 replies; 20+ messages in thread
From: Miklos Szeredi @ 2021-01-05 15:08 UTC (permalink / raw)
To: Vivek Goyal
Cc: Venegas Munoz, Jose Carlos, Christian Schoenebeck,
QEMU Developers, virtio-fs-list
On Fri, Dec 11, 2020 at 8:25 PM Vivek Goyal <vgoyal@redhat.com> wrote:
>
> On Fri, Dec 11, 2020 at 06:29:56PM +0000, Dr. David Alan Gilbert wrote:
>
> [..]
> > > >
> > > > Could we measure at what point does a large window size actually make
> > > > performance worse?
> > >
> > > Will do. Will run tests with varying window sizes (small to large)
> > > and see how does it impact performance for same workload with
> > > same guest memory.
> >
> > I wonder how realistic it is though; it makes some sense if you have a
> > scenario like a fairly small root filesystem - something tractable; but
> > if you have a large FS you're not realistically going to be able to set
> > the cache size to match it - that's why it's a cache!
>
> I think its more about active dataset size and not necessarily total
> FS size. FS might be big but if application is not accessing all of
> the in reasonabl small time window, then it does not matter.
>
> What worries me most is that cost of reclaim of a dax range seems
> too high (or keeps the process blocked for long enogh), that it
> kills the performance. I will need to revisit the reclaim path
> and see if I can optimize something.
Can we dynamically increase the dax window size? Or hot plug
additional dax ranges on demand?
That would solve the problem of trying to find the active set size in advance.
We'd still need reclaim in order to prevent the window from growing
indefinitely, but that would always be background reclaim and be done
after some time of inactivity.
Thanks,
Miklos
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Some performance numbers for virtiofs, DAX and virtio-9p
2021-01-05 15:08 ` [Virtio-fs] " Miklos Szeredi
@ 2021-01-05 15:45 ` Vivek Goyal
-1 siblings, 0 replies; 20+ messages in thread
From: Vivek Goyal @ 2021-01-05 15:45 UTC (permalink / raw)
To: Miklos Szeredi
Cc: Venegas Munoz, Jose Carlos, Christian Schoenebeck,
Dr. David Alan Gilbert, QEMU Developers, virtio-fs-list,
Stefan Hajnoczi
On Tue, Jan 05, 2021 at 04:08:34PM +0100, Miklos Szeredi wrote:
> On Fri, Dec 11, 2020 at 8:25 PM Vivek Goyal <vgoyal@redhat.com> wrote:
> >
> > On Fri, Dec 11, 2020 at 06:29:56PM +0000, Dr. David Alan Gilbert wrote:
> >
> > [..]
> > > > >
> > > > > Could we measure at what point does a large window size actually make
> > > > > performance worse?
> > > >
> > > > Will do. Will run tests with varying window sizes (small to large)
> > > > and see how does it impact performance for same workload with
> > > > same guest memory.
> > >
> > > I wonder how realistic it is though; it makes some sense if you have a
> > > scenario like a fairly small root filesystem - something tractable; but
> > > if you have a large FS you're not realistically going to be able to set
> > > the cache size to match it - that's why it's a cache!
> >
> > I think its more about active dataset size and not necessarily total
> > FS size. FS might be big but if application is not accessing all of
> > the in reasonabl small time window, then it does not matter.
> >
> > What worries me most is that cost of reclaim of a dax range seems
> > too high (or keeps the process blocked for long enogh), that it
> > kills the performance. I will need to revisit the reclaim path
> > and see if I can optimize something.
>
> Can we dynamically increase the dax window size? Or hot plug
> additional dax ranges on demand?
I don't think we can do that yet. In fact I am skeptical that currently
hotplug of whole virtio-fs device itself might not work.
>
> That would solve the problem of trying to find the active set size in advance.
Agreed. Being able to grow dax window dynamically or be able to add
a new dax window to existing device dynamically will help.
David, Stefan, WDYT. Is it possible given current code of qemu and
virtio device. Do other virtio device allow similar operations.
Thanks
Vivek
>
> We'd still need reclaim in order to prevent the window from growing
> indefinitely, but that would always be background reclaim and be done
> after some time of inactivity.
>
> Thanks,
> Miklos
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Virtio-fs] Some performance numbers for virtiofs, DAX and virtio-9p
@ 2021-01-05 15:45 ` Vivek Goyal
0 siblings, 0 replies; 20+ messages in thread
From: Vivek Goyal @ 2021-01-05 15:45 UTC (permalink / raw)
To: Miklos Szeredi
Cc: Venegas Munoz, Jose Carlos, Christian Schoenebeck,
QEMU Developers, virtio-fs-list
On Tue, Jan 05, 2021 at 04:08:34PM +0100, Miklos Szeredi wrote:
> On Fri, Dec 11, 2020 at 8:25 PM Vivek Goyal <vgoyal@redhat.com> wrote:
> >
> > On Fri, Dec 11, 2020 at 06:29:56PM +0000, Dr. David Alan Gilbert wrote:
> >
> > [..]
> > > > >
> > > > > Could we measure at what point does a large window size actually make
> > > > > performance worse?
> > > >
> > > > Will do. Will run tests with varying window sizes (small to large)
> > > > and see how does it impact performance for same workload with
> > > > same guest memory.
> > >
> > > I wonder how realistic it is though; it makes some sense if you have a
> > > scenario like a fairly small root filesystem - something tractable; but
> > > if you have a large FS you're not realistically going to be able to set
> > > the cache size to match it - that's why it's a cache!
> >
> > I think its more about active dataset size and not necessarily total
> > FS size. FS might be big but if application is not accessing all of
> > the in reasonabl small time window, then it does not matter.
> >
> > What worries me most is that cost of reclaim of a dax range seems
> > too high (or keeps the process blocked for long enogh), that it
> > kills the performance. I will need to revisit the reclaim path
> > and see if I can optimize something.
>
> Can we dynamically increase the dax window size? Or hot plug
> additional dax ranges on demand?
I don't think we can do that yet. In fact I am skeptical that currently
hotplug of whole virtio-fs device itself might not work.
>
> That would solve the problem of trying to find the active set size in advance.
Agreed. Being able to grow dax window dynamically or be able to add
a new dax window to existing device dynamically will help.
David, Stefan, WDYT. Is it possible given current code of qemu and
virtio device. Do other virtio device allow similar operations.
Thanks
Vivek
>
> We'd still need reclaim in order to prevent the window from growing
> indefinitely, but that would always be background reclaim and be done
> after some time of inactivity.
>
> Thanks,
> Miklos
>
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2021-01-05 15:47 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-10 16:11 Some performance numbers for virtiofs, DAX and virtio-9p Vivek Goyal
2020-12-10 16:11 ` [Virtio-fs] " Vivek Goyal
2020-12-10 19:29 ` Miklos Szeredi
2020-12-10 19:29 ` [Virtio-fs] " Miklos Szeredi
2020-12-11 16:06 ` Vivek Goyal
2020-12-11 16:06 ` [Virtio-fs] " Vivek Goyal
2020-12-11 18:29 ` Dr. David Alan Gilbert
2020-12-11 18:29 ` [Virtio-fs] " Dr. David Alan Gilbert
2020-12-11 19:25 ` Vivek Goyal
2020-12-11 19:25 ` [Virtio-fs] " Vivek Goyal
2020-12-11 20:01 ` Vivek Goyal
2020-12-11 20:01 ` [Virtio-fs] " Vivek Goyal
2020-12-11 20:06 ` Dr. David Alan Gilbert
2020-12-11 20:06 ` [Virtio-fs] " Dr. David Alan Gilbert
2021-01-05 15:08 ` Miklos Szeredi
2021-01-05 15:08 ` [Virtio-fs] " Miklos Szeredi
2021-01-05 15:45 ` Vivek Goyal
2021-01-05 15:45 ` [Virtio-fs] " Vivek Goyal
2020-12-11 18:47 ` Vivek Goyal
2020-12-11 18:47 ` [Virtio-fs] " Vivek Goyal
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.