All of lore.kernel.org
 help / color / mirror / Atom feed
From: Dave Chinner <david@fromorbit.com>
To: Stan Hoeppner <stan@hardwarefreak.com>
Cc: xfs@oss.sgi.com
Subject: Re: A little RAID experiment
Date: Tue, 17 Jul 2012 15:26:21 +1000	[thread overview]
Message-ID: <20120717052621.GB23387@dastard> (raw)
In-Reply-To: <5004C243.6040404@hardwarefreak.com>

On Mon, Jul 16, 2012 at 08:39:15PM -0500, Stan Hoeppner wrote:
> It depends on the one, and what the one expects.  Most people on this
> list would never expect parity RAID to perform well with the workloads
> you're throwing at it.  Your expectations are clearly different than
> most on this list.

Rule of thumb: don't use RAID5/6 for small random write workloads.

> The kicker here is that most of the data you presented shows almost all
> writes being acked by cache, in which case RAID level should be
> irrelevant, but at the same time showing abysmal throughput.  When all
> write hit cache, throughput should be through the roof.

I bet it's single threaded, which means it is:

	sysbench		kernel
	write(2)
				issue io
				wait for completion
	write(2)
				issue io
				wait for completion
	write(2)
	.....

Which means throughput is limited by IO latency, not bandwidth.
If it takes 10us to do the write(2), issue and process the IO
completion, and it takes 10us for the hardware to do the IO, you're
limited to 50,000 IOPS, or 200MB/s. Given that the best being seen
is around 35MB/s, you're looking at around 10,000 IOPS of 100us
round trip time. At 5MB/s, it's 1200 IOPS or around 800us round
trip.

That's why you get different performance from the different raid
controllers - some process cache hits a lot faster than others.

As to the one that stalled - when the cache hits a certain level of
dirtiness (say 50%), it will start flushing cached writes and
depending on the algorithm may start behaving like a FIFO to new
requests. i.e. each new request coming in needs to wait for one to
drain. At that point, the write rate will tank to maybe 50 IOPS,
which will barely register on the benchmark throughput. (just look
at what happens to the IO latency that is measured...)

IOWs, welcome to Understanding RAID Controller Caching Behaviours
101 :)

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

  reply	other threads:[~2012-07-17  5:26 UTC|newest]

Thread overview: 41+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2012-04-25  8:07 A little RAID experiment Stefan Ring
2012-04-25 14:17 ` Roger Willcocks
2012-04-25 16:23   ` Stefan Ring
2012-04-27 14:03     ` Stan Hoeppner
2012-04-26  8:53   ` Stefan Ring
2012-04-27 15:10     ` Stan Hoeppner
2012-04-27 15:28     ` Joe Landman
2012-04-28  4:42       ` Stan Hoeppner
2012-04-27 13:50 ` Stan Hoeppner
2012-05-01 10:46   ` Stefan Ring
2012-05-30 11:07     ` Stefan Ring
2012-05-31  1:30       ` Stan Hoeppner
2012-05-31  6:44         ` Stefan Ring
2012-07-16 19:57 ` Stefan Ring
2012-07-16 20:03   ` Stefan Ring
2012-07-16 20:05     ` Stefan Ring
2012-07-16 21:27   ` Stan Hoeppner
2012-07-16 21:58     ` Stefan Ring
2012-07-17  1:39       ` Stan Hoeppner
2012-07-17  5:26         ` Dave Chinner [this message]
2012-07-18  2:18           ` Stan Hoeppner
2012-07-18  6:44             ` Stefan Ring
2012-07-18  7:09               ` Stan Hoeppner
2012-07-18  7:22                 ` Stefan Ring
2012-07-18 10:24                   ` Stan Hoeppner
2012-07-18 12:32                     ` Stefan Ring
2012-07-18 12:37                       ` Stefan Ring
2012-07-19  3:08                         ` Stan Hoeppner
2012-07-25  9:29                           ` Stefan Ring
2012-07-25 10:00                             ` Stan Hoeppner
2012-07-25 10:08                               ` Stefan Ring
2012-07-25 11:00                                 ` Stan Hoeppner
2012-07-26  8:32                             ` Dave Chinner
2012-09-11 16:37                               ` Stefan Ring
2012-07-16 22:16     ` Stefan Ring
2012-10-10 14:57 ` Stefan Ring
2012-10-10 21:27   ` Dave Chinner
2012-10-10 22:01     ` Stefan Ring
2012-04-26 22:33 Richard Scobie
2012-04-27 21:30 ` Emmanuel Florac
2012-04-28  4:15   ` Richard Scobie

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20120717052621.GB23387@dastard \
    --to=david@fromorbit.com \
    --cc=stan@hardwarefreak.com \
    --cc=xfs@oss.sgi.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.