From: Konstantin Kharlamov <hi-angel@yandex.ru>
To: linux-ext4@vger.kernel.org
Subject: Changing a workload results in performance drop
Date: Fri, 24 Apr 2020 17:56:09 +0300 [thread overview]
Message-ID: <73a3416a-67d2-c494-1f3f-7d7789bdf61d@yandex.ru> (raw)
* SSDs are used in testing, so random access is not a concern. But I tried the
"steps to reproduce" with raw block device, and IOPS always holds 9k for me.
* "Direct" IO is used to bypass file-system cache.
* The issue is way less visible on XFS, so it looks specific to file systems.
* The biggest difference I've seen is on 70% reads/30% writes workload. But for
simplicity in "steps to reproduce" I'm using 100% write.
* it seems over time (perhaps a day) performance gets improved, so for best
results when testing that you need to re-create ext4 anew.
* in "steps to reproduce" I grep fio stdout. That suppresses interactive
output. Interactive output may be interesting though, I've often seen workload
drops to 600-700 IOPS while average was 5-6k
* Original problem I worked with https://github.com/openzfs/zfs/issues/10231
# Steps to reproduce (in terms of terminal commands)
$ cat fio_jobfile
[job-section]
name=temp-fio
bs=8k
ioengine=libaio
rw=randrw
rwmixread=0
rwmixwrite=100
filename=/mnt/test/file1
iodepth=1
numjobs=1
group_reporting
time_based
runtime=1m
direct=1
filesize=4G
$ mkfs.ext4 /dev/sdw1
$ mount /dev/sdw1 /mnt/test
$ truncate -s 100G /mnt/test/file1
$ fio fio_jobfile | grep -i IOPS
write: IOPS=12.5k, BW=97.0MiB/s (103MB/s)(5879MiB/60001msec)
iops : min=10966, max=14730, avg=12524.20, stdev=1240.27, samples=119
$ sed -i 's/4G/100G/' fio_jobfile
$ fio fio_jobfile | grep -i IOPS
write: IOPS=5880, BW=45.9MiB/s (48.2MB/s)(2756MiB/60001msec)
iops : min= 4084, max= 6976, avg=5879.31, stdev=567.58, samples=119
## Expected
Performance should be more or less the same
## Actual
The second test is twice as slow
# Versions
* Kernel version: 5.6.2-050602-generic
It seems however that the problem is present at least in 4.19 and 5.4. as well, so not a regression.
next reply other threads:[~2020-04-24 15:01 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-04-24 14:56 Konstantin Kharlamov [this message]
2020-06-02 14:22 ` Changing a workload results in performance drop Konstantin Kharlamov
2020-07-29 17:04 ` Jan Kara
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=73a3416a-67d2-c494-1f3f-7d7789bdf61d@yandex.ru \
--to=hi-angel@yandex.ru \
--cc=linux-ext4@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).