From: Brad Fitzpatrick <brad@danga.com>
To: Nathan Scott <nathans@sgi.com>
Cc: Andrew Morton <akpm@osdl.org>,
linux-kernel@vger.kernel.org, Lisa Phillips <lisa@danga.com>,
Mark Smith <marksmith@danga.com>,
linux-xfs@oss.sgi.com
Subject: Re: 2.6.9: unkillable processes during heavy IO
Date: Tue, 16 Nov 2004 23:00:41 -0800 (PST) [thread overview]
Message-ID: <Pine.LNX.4.58.0411162251170.7904@danga.com> (raw)
In-Reply-To: <20041117045506.GA1802@frodo>
Nathan,
On Wed, 17 Nov 2004, Nathan Scott wrote:
> > > On Sun, 14 Nov 2004, Brad Fitzpatrick wrote:
> > >
> > > > We have two database servers which freeze up during heavy IO load. The
>
> Brad, could you send me details on how you've setup mysqld
> and how to generate a load similar to yours, so that I can
> reproduce the hang locally?
The MySQL people made a tool to reproduce MySQL-like load for the specific
purpose of not putting you through the database setup pain:
http://sysbench.sourceforge.net/
To simulate our InnoDB:
-- use the tool to do "seqwr" and write out a 50GB file or so
-- use the tool to make another file, about 30GB.
-- use the tool to do "rndrw" to do random I/O over that 50 GB
file, with the O_DIRECT flag on, with a bunch of threads doing
16k reads/writes.
-- use the tool to do random IO w/o O_DIRECT on the smaller file
at the same time as the O_DIRECT run, also with a number of
threads, doing smaller reads/writes.
This is happening on a machine with LVM2, as well as directly on
/dev/sdb2 (an IBM ServeRAID), so device-mapper shouldn't be an issue one
way or another.
The filesystem size is 270 GB on one machine (on /dev/sdb2) and 120 GB on
LVM2.
> > > > The hardware/software stack is:
> > > >
> > > > - Dual Opteron 246, SMP kernel, w/ NUMA
> > > > - 9 GB of memory (4GB in one zone, 5GB in the other)
> > > > - MySQL, running mostly InnoDB, but some MyISAM
>
> ( I don't even know what those two things are, so you can
> probably guess at the level of assistance I'll need here. :)
Well, sysbench should help you find the problem.
Let me know if I can help more. Thanks for looking into this.
- Brad
>
> thanks!
>
> --
> Nathan
>
>
next prev parent reply other threads:[~2004-11-17 7:01 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2004-11-14 22:15 2.6.9: unkillable processes during heavy IO Brad Fitzpatrick
2004-11-14 21:41 ` Alan Cox
2004-11-14 22:30 ` Andrew Morton
2004-11-14 22:45 ` Andries Brouwer
2004-11-15 7:39 ` Lenar Lõhmus
2004-11-16 13:56 ` Brad Fitzpatrick
2004-11-17 4:01 ` Andrew Morton
2004-11-17 4:55 ` Nathan Scott
2004-11-17 7:00 ` Brad Fitzpatrick [this message]
2004-11-18 5:05 ` Nathan Scott
2004-11-19 4:59 ` Nathan Scott
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Pine.LNX.4.58.0411162251170.7904@danga.com \
--to=brad@danga.com \
--cc=akpm@osdl.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-xfs@oss.sgi.com \
--cc=lisa@danga.com \
--cc=marksmith@danga.com \
--cc=nathans@sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).