All of lore.kernel.org
 help / color / mirror / Atom feed
From: Nick Neumann <nick@pcpartpicker.com>
To: fio@vger.kernel.org
Subject: 
Date: Sun, 28 Aug 2022 16:01:30 -0500	[thread overview]
Message-ID: <CADqNVTrsKpVM6xUV_oTaoJcLc1=4mRXagAQA7sr40KCBKYEidA@mail.gmail.com> (raw)

I've filed the issue on github, but just thought I'd mention here too.
In real-world use it appears to be intermittent. I"m not yet sure how
intermittent, but I could see it being used in production and not
caught right away. I got lucky and stumbled on it when looking at
graphs of runs and noticed 15 seconds of no activity.

https://github.com/axboe/fio/issues/1457

With the null ioengine, I can make it reproduce very reliably, which
is encouraging as I move to debug.

I had just moved to using log compression as it is really powerful,
and the only way to store per I/O logs for a long run without pushing
up against the amount of physical memory in a system.

(Without compression, a GB of sequential writes at 128K block size is
on the order of 245KB of memory per log, so a TB is 245MB per log. Now
run a job to fill a 20TB drive and you're at 4.9GB for one log file.
If you record all 3 latency numbers too, you're talking close to
20GB.)

             reply	other threads:[~2022-08-28 21:01 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-08-28 21:01 Nick Neumann [this message]
2022-09-01 17:44 ` Nick Neumann

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CADqNVTrsKpVM6xUV_oTaoJcLc1=4mRXagAQA7sr40KCBKYEidA@mail.gmail.com' \
    --to=nick@pcpartpicker.com \
    --cc=fio@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.