Raid 6--best practices

Raid 6--best practices

[Shain Miley]

Hello all,
I have been doing some research into possible alternatives to our
OpenSolaris/ZFS/Gluster file server.  The main reason behind this is,
due to RedHat's recent purchase of Gluster, our current configuration
will no longer be supported and even before the acquisition, the upgrade
path for the OpenSolaris/ZFS stack was murky at best.

The current servers in question consist of a total of 48, 2TB drives.
My thought was that I would setup a total of 6 RAID-6 arrays (each
containing 7 drives + a spare or a flat 8 drive RAID-6 config) and place
LVM + XFS on top of that.

My questions really are:

a)    What is the maximum number of drives typically seen in a RAID-6
setup like this?  I noticed when looking at the Backblaze blog, that
they are using RAID-6 with 15 disks (13 + 2 for parity).  That number seemed 
kind of high to me....but I was wondering what others on the list thought.

b)    Would you recommend using any specific Linux distro over any other?  
Right now I am trying to decide between Debian and Ubuntu....but I would be open to 
any others...if there was a legitimate reason to do so (performance, stability, etc) in terms of the Raid codebase.

Thanks in advance,

Shain

Re: Raid 6--best practices

[Mathias Burén]

On 20 January 2012 02:54, Shain Miley <SMiley@npr.org> wrote:
> Hello all,
> I have been doing some research into possible alternatives to our
> OpenSolaris/ZFS/Gluster file server.  The main reason behind this is,
> due to RedHat's recent purchase of Gluster, our current configuration
> will no longer be supported and even before the acquisition, the upgrade
> path for the OpenSolaris/ZFS stack was murky at best.
>
> The current servers in question consist of a total of 48, 2TB drives.
> My thought was that I would setup a total of 6 RAID-6 arrays (each
> containing 7 drives + a spare or a flat 8 drive RAID-6 config) and place
> LVM + XFS on top of that.
>
> My questions really are:
>
> a)    What is the maximum number of drives typically seen in a RAID-6
> setup like this?  I noticed when looking at the Backblaze blog, that
> they are using RAID-6 with 15 disks (13 + 2 for parity).  That number seemed
> kind of high to me....but I was wondering what others on the list thought.
>
> b)    Would you recommend using any specific Linux distro over any other?
> Right now I am trying to decide between Debian and Ubuntu....but I would be open to
> any others...if there was a legitimate reason to do so (performance, stability, etc) in terms of the Raid codebase.
>
> Thanks in advance,
>
> Shain
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


Hi,

Personally I wouldn't use more than 10 HDDs in a RAID6 setup (+ spare
perhaps). Regarding the choice of distribution, it doesn't matter
really. Pick the one you're comfortable with.

Good luck!

/M
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Raid 6--best practices

[Shain Miley]

Mathias,
Thank you very much for the helpful advice.

Shain

________________________________________
From: Mathias Burén [mathias.buren@gmail.com]
Sent: Thursday, January 19, 2012 10:06 PM
To: Shain Miley
Cc: linux-raid@vger.kernel.org
Subject: Re: Raid 6--best practices

On 20 January 2012 02:54, Shain Miley <SMiley@npr.org> wrote:
> Hello all,
> I have been doing some research into possible alternatives to our
> OpenSolaris/ZFS/Gluster file server.  The main reason behind this is,
> due to RedHat's recent purchase of Gluster, our current configuration
> will no longer be supported and even before the acquisition, the upgrade
> path for the OpenSolaris/ZFS stack was murky at best.
>
> The current servers in question consist of a total of 48, 2TB drives.
> My thought was that I would setup a total of 6 RAID-6 arrays (each
> containing 7 drives + a spare or a flat 8 drive RAID-6 config) and place
> LVM + XFS on top of that.
>
> My questions really are:
>
> a)    What is the maximum number of drives typically seen in a RAID-6
> setup like this?  I noticed when looking at the Backblaze blog, that
> they are using RAID-6 with 15 disks (13 + 2 for parity).  That number seemed
> kind of high to me....but I was wondering what others on the list thought.
>
> b)    Would you recommend using any specific Linux distro over any other?
> Right now I am trying to decide between Debian and Ubuntu....but I would be open to
> any others...if there was a legitimate reason to do so (performance, stability, etc) in terms of the Raid codebase.
>
> Thanks in advance,
>
> Shain
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


Hi,

Personally I wouldn't use more than 10 HDDs in a RAID6 setup (+ spare
perhaps). Regarding the choice of distribution, it doesn't matter
really. Pick the one you're comfortable with.

Good luck!

/M


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Raid 6--best practices

[Wiliam Colls]

On 01/19/2012 09:54 PM, Shain Miley wrote:
> Hello all,
> I have been doing some research into possible alternatives to our
> OpenSolaris/ZFS/Gluster file server.  The main reason behind this is,
> due to RedHat's recent purchase of Gluster, our current configuration
> will no longer be supported and even before the acquisition, the upgrade
> path for the OpenSolaris/ZFS stack was murky at best.
>
> The current servers in question consist of a total of 48, 2TB drives.
> My thought was that I would setup a total of 6 RAID-6 arrays (each
> containing 7 drives + a spare or a flat 8 drive RAID-6 config) and place
> LVM + XFS on top of that.
>
> My questions really are:
>
> a)    What is the maximum number of drives typically seen in a RAID-6
> setup like this?  I noticed when looking at the Backblaze blog, that
> they are using RAID-6 with 15 disks (13 + 2 for parity).  That number seemed
> kind of high to me....but I was wondering what others on the list thought.
>
> b)    Would you recommend using any specific Linux distro over any other?
> Right now I am trying to decide between Debian and Ubuntu....but I would be open to
> any others...if there was a legitimate reason to do so (performance, stability, etc) in terms of the Raid codebase.
>
> Thanks in advance,
>
> Shain
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

No comment on the Raid6 question.

I have been using Ubuntu for the last 4 or 5 years. For my money, the 
best feature is the defined release intervals, and the Long Term Support 
(LTS) commitment of 5 years for specified releases, and easy upgrade 
path when moving to a new release.

Just my $0.02 Cdn.

-- 
I know that you believe that you understand what you think I said, but I 
am not sure that you realise that what you heard was not what I meant.

Re: Raid 6--best practices

[Stefan /*St0fF*/ Hübner]

Hi Shain,

Am 20.01.2012 03:54, schrieb Shain Miley:
> Hello all,
> I have been doing some research into possible alternatives to our
> OpenSolaris/ZFS/Gluster file server.  The main reason behind this is,
> due to RedHat's recent purchase of Gluster, our current configuration
> will no longer be supported and even before the acquisition, the upgrade
> path for the OpenSolaris/ZFS stack was murky at best.
> 
> The current servers in question consist of a total of 48, 2TB drives.
> My thought was that I would setup a total of 6 RAID-6 arrays (each
> containing 7 drives + a spare or a flat 8 drive RAID-6 config) and place
> LVM + XFS on top of that.
> 
> My questions really are:
> 
> a)    What is the maximum number of drives typically seen in a RAID-6
> setup like this?  I noticed when looking at the Backblaze blog, that
> they are using RAID-6 with 15 disks (13 + 2 for parity).  That number seemed 
> kind of high to me....but I was wondering what others on the list thought.
> 
Typically RAID6 Arrays are most optimized with 2^n+2 drives, as in that
case the stripe-size * data-disks adds up to a "good" big-block-size of
a power of 2.  A rule of thumb is to have one redundancy for any 4
disks.  That makes a 6-drive RAID6 over-redundant and a 10-drive RAID6
"best to do".  Next would be an 18-drive RAID6 - which compromizes the
redundancy-rule big time.  So I'd go for a 40-drive RAID60 and see what
to do with the rest of the drives...

> b)    Would you recommend using any specific Linux distro over any other?  
> Right now I am trying to decide between Debian and Ubuntu....but I would be open to 
> any others...if there was a legitimate reason to do so (performance, stability, etc) in terms of the Raid codebase.

This question isn't easy to answer - it depends on your needs.  For
storage systems I typically recommend open-e (www.open-e.com), because
that's what we sell most (see www.exomium.com).  It's also most stable
as open-e does heavy QA on kernel and features.
> 
> Thanks in advance,
> 
> Shain
> 
Hope this was helpful,

Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Raid 6--best practices

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 903 bytes --]

On Thu, 19 Jan 2012 21:54:40 -0500
Shain Miley <SMiley@npr.org> wrote:

> b)    Would you recommend using any specific Linux distro over any other?  
> Right now I am trying to decide between Debian and Ubuntu....

In my experience Ubuntu doesn't really care about packages outside of a narrow
set of "desktop" or "regular user" ones, and therefore is not a good choice for
a such use. E.g. they have been shipping an ancient version of mdadm (2.6.x)
long after it was no longer sane to even try using it. :)

You can compare 
http://packages.debian.org/search?keywords=mdadm
with
http://packages.ubuntu.com/search?keywords=mdadm&searchon=names&suite=all&section=all
and notice that Debian generally has a newer mdadm.

-- 
With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Raid 6--best practices

[David Brown]

On 20/01/2012 03:54, Shain Miley wrote:
> Hello all,
> I have been doing some research into possible alternatives to our
> OpenSolaris/ZFS/Gluster file server.  The main reason behind this is,
> due to RedHat's recent purchase of Gluster, our current configuration
> will no longer be supported and even before the acquisition, the upgrade
> path for the OpenSolaris/ZFS stack was murky at best.
>
> The current servers in question consist of a total of 48, 2TB drives.
> My thought was that I would setup a total of 6 RAID-6 arrays (each
> containing 7 drives + a spare or a flat 8 drive RAID-6 config) and place
> LVM + XFS on top of that.
>

I wouldn't bother dedicating a spare to each RAID-6 - I would rather 
have the spares in a pool that can be used by any of the low-level raids.

Before it is possible to give concrete suggestions, it is vital to know 
the usage of the system.  Are you storing mostly big files, mostly small 
ones, or a mixture?  What are the read/write ratios?  Do you have lots 
of concurrent users, or only a few - and are they accessing wildly 
different files or the same ones?  How important is uptime?  How 
important are fast rebuilds/resyncs?  How important is array speed 
during rebuilds?  What sort of space efficiencies do you need?  What 
redundancies do you really need?  What topologies do you have that 
influence speed, failure risks, and redundancies (such as multiple 
controllers/backplanes/disk racks)?  Are you using hardware raid 
controllers in this mix, or just software raid?  Are you planning to be 
able to expand the system in the future with more disks or bigger disks?

There are lots of questions here, and no immediate answers.  I certainly 
wouldn't fixate on a concatenation of RAID-6 arrays before knowing a bit 
more - it's not the only way to tie together 48 disks, and it may not be 
the best balance.

mvh.,

David




> My questions really are:
>
> a)    What is the maximum number of drives typically seen in a RAID-6
> setup like this?  I noticed when looking at the Backblaze blog, that
> they are using RAID-6 with 15 disks (13 + 2 for parity).  That number seemed
> kind of high to me....but I was wondering what others on the list thought.
>
> b)    Would you recommend using any specific Linux distro over any other?
> Right now I am trying to decide between Debian and Ubuntu....but I would be open to
> any others...if there was a legitimate reason to do so (performance, stability, etc) in terms of the Raid codebase.
>
> Thanks in advance,
>
> Shain
>

Re: Raid 6--best practices

[Shain Miley]

David,

Thanks....let me see if I provide a few more details about what we are 
trying to achieve.


At this point we are storing mostly larger files such as audio (.wav, 
.mp3, etc) and video files in various formats.  The initial purpose of 
this particular file server was meant to be a long term media storage 
'archive'.  The current setup was constructed to minimize data loss and 
maximize uptime, and other considerations such as speed were secondary.  
The main reason for not worrying too much about overall speed, is that 
reading and writing to the Gluster mount points are very heavily 
dependant on the speed of the interconnects between nodes (for writing), 
and since we are using this in a WAN setup, as expected Gluster becomes 
the limiting factor in this configuration.

The initial specification called for relatively low reads and writes, 
since we are basically placing the files there once(via CIFS or NFS), 
and they are rarely if ever going to get updated or re-written.  In 
terms of reads, we now have several web apps that basically act as a 
front end for downloading/previewing, etc these media files, however 
these are mainly internal apps (at this point) and so overall read/write 
ratio's still remain low.

Uptime is relatively important, although given that we are using 
Gluster, we should have access to our data if we have a node failure, 
the issue then becomes having to sync up the data which is always a 
little pain...but should not involve any downtime.  In terms of array 
rebuilding times, I think I would like to minimize them to the extent 
possible, but I understand they will be a reality given this setup.

We have two 3ware 9650SE-24M8 in each node, but I was planning on trying 
to just export the disks as JBODs, and try not to use the cards for 
anything other then exporting the disks to the OS.  ZFS recommends doing 
this, I didn't at the time, I just made a bunch of single disk RAID-1 
units, but I ran into some problems later on and ended up wishing I had 
just gone the JBOD route.


Anyway...I hope this helps shed a little bit more light on what we are 
trying to do.

Thanks everyone for your help.

Shain

On 01/20/2012 05:24 AM, David Brown wrote:
> On 20/01/2012 03:54, Shain Miley wrote:
>> Hello all,
>> I have been doing some research into possible alternatives to our
>> OpenSolaris/ZFS/Gluster file server.  The main reason behind this is,
>> due to RedHat's recent purchase of Gluster, our current configuration
>> will no longer be supported and even before the acquisition, the upgrade
>> path for the OpenSolaris/ZFS stack was murky at best.
>>
>> The current servers in question consist of a total of 48, 2TB drives.
>> My thought was that I would setup a total of 6 RAID-6 arrays (each
>> containing 7 drives + a spare or a flat 8 drive RAID-6 config) and place
>> LVM + XFS on top of that.
>>
> I wouldn't bother dedicating a spare to each RAID-6 - I would rather
> have the spares in a pool that can be used by any of the low-level raids.
>
> Before it is possible to give concrete suggestions, it is vital to know
> the usage of the system.  Are you storing mostly big files, mostly small
> ones, or a mixture?  What are the read/write ratios?  Do you have lots
> of concurrent users, or only a few - and are they accessing wildly
> different files or the same ones?  How important is uptime?  How
> important are fast rebuilds/resyncs?  How important is array speed
> during rebuilds?  What sort of space efficiencies do you need?  What
> redundancies do you really need?  What topologies do you have that
> influence speed, failure risks, and redundancies (such as multiple
> controllers/backplanes/disk racks)?  Are you using hardware raid
> controllers in this mix, or just software raid?  Are you planning to be
> able to expand the system in the future with more disks or bigger disks?
>
> There are lots of questions here, and no immediate answers.  I certainly
> wouldn't fixate on a concatenation of RAID-6 arrays before knowing a bit
> more - it's not the only way to tie together 48 disks, and it may not be
> the best balance.
>
> mvh.,
>
> David
>
>
>
>
>> My questions really are:
>>
>> a)    What is the maximum number of drives typically seen in a RAID-6
>> setup like this?  I noticed when looking at the Backblaze blog, that
>> they are using RAID-6 with 15 disks (13 + 2 for parity).  That number seemed
>> kind of high to me....but I was wondering what others on the list thought.
>>
>> b)    Would you recommend using any specific Linux distro over any other?
>> Right now I am trying to decide between Debian and Ubuntu....but I would be open to
>> any others...if there was a legitimate reason to do so (performance, stability, etc) in terms of the Raid codebase.
>>
>> Thanks in advance,
>>
>> Shain
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Raid 6--best practices

[linbloke]

On 20/01/12 2:16 PM, Shain Miley wrote:
> Mathias,
> Thank you very much for the helpful advice.
>
> Shain
>
> ________________________________________
> From: Mathias Burén [mathias.buren@gmail.com]
> Sent: Thursday, January 19, 2012 10:06 PM
> To: Shain Miley
> Cc: linux-raid@vger.kernel.org
> Subject: Re: Raid 6--best practices
>
> On 20 January 2012 02:54, Shain Miley<SMiley@npr.org>  wrote:
>> Hello all,
>> I have been doing some research into possible alternatives to our
>> OpenSolaris/ZFS/Gluster file server.  The main reason behind this is,
>> due to RedHat's recent purchase of Gluster, our current configuration
>> will no longer be supported and even before the acquisition, the upgrade
>> path for the OpenSolaris/ZFS stack was murky at best.
>>
>> The current servers in question consist of a total of 48, 2TB drives.
>> My thought was that I would setup a total of 6 RAID-6 arrays (each
>> containing 7 drives + a spare or a flat 8 drive RAID-6 config) and place
>> LVM + XFS on top of that.
>>
>> My questions really are:
>>
>> a)    What is the maximum number of drives typically seen in a RAID-6
>> setup like this?  I noticed when looking at the Backblaze blog, that
>> they are using RAID-6 with 15 disks (13 + 2 for parity).  That number seemed
>> kind of high to me....but I was wondering what others on the list thought.
>>
>> b)    Would you recommend using any specific Linux distro over any other?
>> Right now I am trying to decide between Debian and Ubuntu....but I would be open to
>> any others...if there was a legitimate reason to do so (performance, stability, etc) in terms of the Raid codebase.
>>
>> Thanks in advance,
>>
>> Shain
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> Hi,
>
> Personally I wouldn't use more than 10 HDDs in a RAID6 setup (+ spare
> perhaps). Regarding the choice of distribution, it doesn't matter
> really. Pick the one you're comfortable with.
>
> Good luck!
>
> /M

Whilst I love Debian, I find the release schedule a bit frustrating. If 
you can hold off until May (a big stretch for projects wanting to go 
now, I know), then it may be worth looking at Ubuntu 12.04 LTS and get 5 
years of supported OS.

https://wiki.ubuntu.com/PrecisePangolin/ReleaseSchedule

Just a thought. Get Debian stability and Ubuntu long term support.

Cheers,

Josh

>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Raid 6--best practices

[Stan Hoeppner]

On 1/20/2012 4:24 AM, David Brown wrote:

> I wouldn't bother dedicating a spare to each RAID-6 - I would rather
> have the spares in a pool that can be used by any of the low-level raids.
> 
> Before it is possible to give concrete suggestions, it is vital to know
> the usage of the system.  Are you storing mostly big files, mostly small
> ones, or a mixture?  What are the read/write ratios?  Do you have lots
> of concurrent users, or only a few - and are they accessing wildly
> different files or the same ones?  How important is uptime?  How
> important are fast rebuilds/resyncs?  How important is array speed
> during rebuilds?  What sort of space efficiencies do you need?  What
> redundancies do you really need?  What topologies do you have that
> influence speed, failure risks, and redundancies (such as multiple
> controllers/backplanes/disk racks)?  Are you using hardware raid
> controllers in this mix, or just software raid?  Are you planning to be
> able to expand the system in the future with more disks or bigger disks?
> 
> There are lots of questions here, and no immediate answers.  I certainly
> wouldn't fixate on a concatenation of RAID-6 arrays before knowing a bit
> more - it's not the only way to tie together 48 disks, and it may not be
> the best balance.

Glad I read your post before typing my response David.  You hit almost
every important point here.  You've identified the OP's one big mistake
which is:

"I have these 48 disks and I'm trying to figure out how best to use them."

What the OP should be asking himself is:

"I have a mix of applications and administration needs.  How best can I
utilize this set of 48 disks to serve both?"

Do you plan to make use of VFS snapshots?  Do you actually need any of
the functionality that LVM provides?  If not consider directly
formatting your RAID devices with XFS.  Also, do NOT use partitions, and
make sure you align XFS to the underlying RAID stripe, or your
performance will drop by a factor of 2 instantly with parity RAID.  If
you need help with mkfs.xfs stripe alignment ask on the XFS list.  If
you don't format the RAID device correctly performance will really suck
(technical term).

NOTE:  You _need_ to have already decided what RAID level and
configuration you plan to use before asking about stripe alignment, as
it is different for every single configuration.  I.e. changing from a 16
disk array to 14, or changing from RAID6 to RAID10, totally changes
filesystem alignment parameters.  You only make the filesystem once, so
the parms must be right, or performance suffers horribly.

-- 
Stan

Re: Raid 6--best practices

[Peter Grandi]

>> I have been doing some research into possible alternatives to
>> our OpenSolaris/ZFS/Gluster file server. The main reason
>> behind this is, due to RedHat's recent purchase of Gluster,
>> our current configuration will no longer be supported and
>> even before the acquisition, the upgrade path for the
>> OpenSolaris/ZFS stack was murky at best.

You could be using FreeBSD/ZFS, or just keep using Gluster
indeed as you seem going to do, which is quite good overall, and
the sale to RH just means it will have *much* better chance of
being maintained for the foreseeable future, but not OpenSolaris.

>> The current servers in question consist of a total of 48, 2TB
>> drives. My thought was that I would setup a total of 6 RAID-6
>> arrays (each containing 7 drives + a spare or a flat 8 drive
>> RAID-6 config) and place LVM + XFS on top of that.

That's the usual (euphemism alert) imaginative setup that
follows what I call a "syntactic" logic (it is syntactically
valid!).

Note: You could have 1-2 spares and share them among all sets.

Also, the 2TB drives are likely to be consumer-grade ones with
ERC disabled, unless you chose carefully or got lucky.

>> My questions really are: a) What is the maximum number of
>> drives typically seen in a RAID-6 setup like this?

Any number up to 48. Really, because "typically seen" is a naive
question, because what is "typically seen" could be pretty bad.

>> I noticed when looking at the Backblaze blog, that they are
>> using RAID-6 with 15 disks (13 + 2 for parity).

Backblaze have a very special application. A wide RAID6 _might_
make sense for them.

>> That number seemed kind of high to me....

That's good you seem to be a bit less (euphemism alert)
audacious than most sysadms, who just love very wide RAID6,
because of an assumption that I find (euphemism alert)
fascinating:

     http://WWW.sabi.co.UK/blog/1103Mar.html#110331

What matters to me is the percentage of redundancy adjusted by
disk set geometry and implications for rebuild.

In general, unless someone really knows better, RAID10 or RAID1
should be the only choices. Of course everybody knows better :-).

>> but I was wondering what others on the list thought.

I personally think that the best practice with both RAID6 and
LVM2 is never to use them (with minuscule exceptions), and in
particular never to use 'concat'.

>> b) Would you recommend using any specific Linux distro over
>> any other?  Right now I am trying to decide between Debian and
>> Ubuntu....but I would be open to any others...if there was a
>> legitimate reason to do so (performance, stability, etc) in
>> terms of the Raid codebase.

Does not matter that much, but you might want a distro that
comes with some kind of "enterprise support", like RHEL or SLES
or derivatives, or Ubuntu LTS. Of course these at most points in
time are relatively old.

> At this point we are storing mostly larger files such as audio
> (.wav, .mp3, etc) and video files in various formats.  The
> initial purpose of this particular file server was meant to be
> a long term media storage 'archive'. The current setup was
> constructed to minimize data loss and maximize uptime, and
> other considerations such as speed were secondary.  [ ... ]
> The initial specification called for relatively low reads and
> writes, since we are basically placing the files there
> once(via CIFS or NFS), and they are rarely if ever going to
> get updated or re-written.

> Uptime is relatively important, although given that we are
> using Gluster, we should have access to our data if we have a
> node failure, the issue then becomes having to sync up the
> data which is always a little pain...but should not involve
> any downtime.

Fortunately you are storing relatively large files, so a
filetree is not a totally inappropriate container for that.
Still I would use a database for "blobs" of that size, for many
reasons.

Since your application is essentially append/read only, you can
just fill one filetree, remount it RO, and start filling another
one, and so on, so you don't really need to have a shared free
space pool, or you could use Gluster over each single
independent filetree.

If you have a layer of redundancy anyhow (e.g. DRBD or Gluster
replicated volumes) as you seem to have I would use a number of
narrow RAID5 sets, something like 2+1 or 4+1 (at most), as the
independent filetrees. Because your application is like that it
seems one of the few suited to RAID5:

  http://www.sabi.co.uk/blog/1104Apr.html#110401

As a completely different alternative, if you really really need
a single free space pool, you could consider a complete change
to Lustre over DRBD, but I think that Gluster over XFS over
RAID10 or RAID5 would be good.

> In terms of array rebuilding times, I think I would like to
> minimize them to the extent possible, but I understand they
> will be a reality given this setup.

Also consider 'fsck' time and space. A nice set of 2+1 RAID5
could be reasonable here.

> We have two 3ware 9650SE-24M8 in each node, but I was planning
> on trying to just export the disks as JBODs, and try not to
> use the cards for anything other then exporting the disks to
> the OS.

3ware firmware has been known to have horrifying firmware
issues:

  http://makarevitch.org/rant/3ware/
  http://www.mattheaton.com/?p=160

Note that the really noticeable bugs are behavioural ones, as in
poor request scheduling under load, and they happen even in
single drive mode. This is sad, because up to series 7000 I had
a good impression of 3ware HAs. But many nights spent trying to
compensate for the many issues of series 9000 have changed my
opinion.

Most other RAID HAs are also buggy, consider for example:

  http://www.gridpp.rl.ac.uk/blog/2011/01/12/sata-raid-controller-experiences-at-the-tier1/

In general using MD is rather more reliable. My usual list of
things should be defaults unless one knows a lot better: MD,
RAID10, SCT ERC, JFS or XFS, GPT partitioning; of things to
avoid unless there are special cases: firmware based RAID HAs,
any parity RAID level or 'concat', drives without ERC, ext3 (and
ext4), LVM2 or MBR partitioning.