Re: XFS: Abysmal write performance because of excessive seeking (allocation groups to blame?)

[Geoffrey Wehrman <gwehrman@sgi.com>]

On Tue, Apr 10, 2012 at 02:30:39PM -0500, Stan Hoeppner wrote:
| On 4/9/2012 9:21 AM, Geoffrey Wehrman wrote:
| > On Fri, Apr 06, 2012 at 06:28:37PM -0500, Stan Hoeppner wrote:
| > | So while the XFS AG architecture may not be perfectly suited to your
| > | single 6 drive RAID6 array, it still gives rather remarkable performance
| > | given that the same architecture can scale pretty linearly to the
| > | heights above, and far beyond.  Something EXTx and others could never
| > | dream of.  Some of the SGI guys might be able to confirm deployed single
| > | XFS filesystems spanning 1000+ drives in the past.  Today we'd probably
| > | only see that scale with CXFS.
| 
| Good to hear from you Geoffrey.
| 
| > With an SGI IS16000 array which supports up to 1,200 drives, filesystems
| > with large numbers of drives isn't difficult.  Most configurations
| > using the IS16000 have 8+2 RAID6 luns.  
| 
| Is the concatenation of all these RAID6 LUNs performed within the
| IS16000, or with md/lvm, or?

The LUNs were concatenated with XVM which is SGI's md/lvm equivalent.
The filesystem was then constructed so that the LUN boundaries matched
AG boundaries in the filesystem.  The filesystem was mounted with the
inode64 mount option.  inode64 rotors directories across AGs, and then
attempts to allocate space for files created in the AG containing the
directory.  Utilizing this behavior allowed the generated load to be
spread across the entire set of LUNs.

| > I've seen sustained 15 GB/s to
| > a single filesystem on one of the arrays with a 600 drive configuration.
| 
| To be clear, this is a single Linux XFS filesystem on a single host, not
| multiple CXFS clients, correct?  If so, out of curiosity, is the host in
| this case an old Itanium Altix or the newer Xeon based Altix UV?  And
| finally, is this example system using FC or Infiniband connectivity?
| How many ports?

This was a single Linux XFS filesystem, but with two CXFS client hosts.
They were both rather ordinary dual socket x86_64 Xeon systems using
FC connectivity.  I fully expect that the same results could be obtained
from a single host with enough I/O bandwidth.


-- 
Geoffrey Wehrman

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs