Hello Ted, On 1/15/20 10:18 PM, Theodore Y. Ts'o wrote: > On Wed, Jan 15, 2020 at 05:00:53AM +0530, Ritesh Harjani wrote: >> >> I too collected some performance numbers on my x86 box with >> --direct=1, bs=4K/1M & ioengine=libaio, with default opt v/s dioread_nolock >> opt on latest ext4 git tree. >> >> I found the delta to be within +/- 6% in all of the runs which includes, seq >> read, mixed rw & mixed randrw. > > Thanks for taking the performance measurements! > > Are you able to release more detail of what the deltas were for those > tests? And how stable were those results across repeated runs? > I have collected these numbers using fio. All of the column values shown in the attachment & pasted below are average of 3 runs. I checked all those individual runs too and saw that even in the run-to- run variations the delta was within +/- 5% only. The naming of those individual runs *.json files are a bit weird and will take sometime if I have to publish the individual runs. But do let me know if that as well is required. I can make some changes in the scripts to print individual run's numbers too. I have pasted the individual fio files too which I used for my testing. I have used libaio as ioengine (with iodepth=16) in the SeqRead case and psync in others. Performance numbers: =================== (seq) read (4K) - libaio ============ threads default_opt_read(MB/s) dioread_nolock_opt_read(MB/s) ---------- --------------------- ---------------------------- 1 138.928059895833 138.824869791667 8 129.345052083333 124.472005208333 24 71.6555989583333 72.275390625 (Not sure why 24 thread case has lower perf numbers, but on increasing the iodepth=32, I could see an increase in bw with 24 threads case too.) (seq) read (1M) - libaio =========== threads default_opt_read(MB/s) dioread_nolock_opt_read(MB/s) ---------- --------------------- --------------- 1 138.905598958333 138.832682291667 8 111.263997395833 109.301106770833 24 121.895182291667 127.75390625 randrw(4K) - read (psync) ========================== threads default_opt_read [KB/s] dioread_opt_read [KB/s] ---------- ---------------- ---------------- 1 414.666666666667 410.0 8 780.0 792.333333333333 24 967.333333333333 991.666666666667 randrw(4K) - write (psync) =========================== threads default_opt_write [KB/s] dioread_opt_write [KB/s] ---------- ----------------- ----------------- 1 418.0 413.666666666667 8 796.666666666667 809.666666666667 24 981.333333333333 1007.66666666667 randrw(1M) - read (psync) ================= threads default_opt_read(MB/s) dioread_nolock_opt_read(MB/s) ---------- ----------------- ----------------------- 1 39.5693359375 39.7288411458333 8 44.5179036458333 47.9098307291667 24 50.2861328125 51.720703125 randrw(1M) - write (psync) ================== threads default_opt_write(MB/s) dioread_nolock_opt_write(MB/s) ---------- ---------------------- ----------------------- 1 41.4583333333333 41.5068359375 8 46.0768229166667 49.568359375 24 49.5947265625 50.7083333333333 rw(1M) - read (psync) ============= threads default_opt_read(MB/s) dioread_nolock_opt_read(MB/s) ---------- ---------------------- ------------------- 1 43.6458333333333 43.6770833333333 8 48.1178385416667 49.2718098958333 24 50.5703125 53.7890625 rw(1M) - write (psync) ============== threads default_opt_write(MB/s) dioread_nolock_opt_write(MB/s) ---------- ----------------------- ------------------ 1 45.5065104166667 45.5654296875 8 49.7431640625 51.0690104166667 24 50.2493489583333 53.3463541666667 FIO FILES ========= dio_read.fio ============ [global] ioengine=libaio rw=read runtime=60 iodepth=16 direct=1 size=10G filename=./testfile group_reporting=1 thread=1 ;bs=$bs ;numjobs=24 [fio-run] dio_randrw.fio ============== [global] ioengine=psync rw=randrw runtime=60 iodepth=1 direct=1 size=10G filename=./testfile group_reporting=1 thread=1 ;bs=$bs ;numjobs=24 [fio-run] dio_rw.fio ========== [global] ioengine=psync rw=rw runtime=60 iodepth=1 direct=1 size=10G filename=./testfile group_reporting=1 thread=1 ;bs=$bs ;numjobs=24 [fio-run] Thanks -ritesh