Recommendation on XFS Filesystem Creation, Lefthand P4500, iSCSI ?

Recommendation on XFS Filesystem Creation, Lefthand P4500, iSCSI ?

[Svavar Örn Eysteinsson]

Hello List.

We recently purchased HP Lefthand 4500 G2 24TB iSCSI SAN Box.
The Box is configured in RAID6. (and yes, we are going to expand that with
another boxes in near future).

I'm running Centos 6.2 with the latest updates on the iSCSI initiator 
machine.
HP DL380 G7 with 24GB in RAM

I need to create a 7TB volume at the start, and grow it in near future.

Are there any recommendation on creating that XFS volume ? Particularly 
regarding
iSCSI and that big ?

This server will act as a fileserver for 50+ machines serving request 
with netatalk
and samba.

Surely I will need to use inode64 option. Right ?


Any recommendation would be much appreciated.

Thanks allot people.
Best regards,

Svavar
Reykjavik - Iceland






_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Recommendation on XFS Filesystem Creation, Lefthand P4500, iSCSI ?

[Mike Grant]

Hi Svavar,

On 06/19/2012 11:13 AM, Svavar Örn Eysteinsson wrote:
> We recently purchased HP Lefthand 4500 G2 24TB iSCSI SAN Box.
> The Box is configured in RAID6. (and yes, we are going to expand that with
> another boxes in near future).
> I need to create a 7TB volume at the start, and grow it in near future.

We have a couple of NAS servers (just direct disk connection, not iSCSI) 
with XFS filesystems up to ~30TB, but didn't do anything special for 
those (just defaults on everything, iirc).

The only problem we had was with filesystems bigger than that, where we 
get kernel crashes due to XFS overflowing the stack.  Apparently this 
may be worse if you NFS-share XFS layered over LVM (more stuff on the 
stack, I guess), but I imagine you'd be fine with 7TB.

Cheers,

Mike.

<br />
<hr />
<p><font face="Arial" size="1">
Plymouth Marine Laboratory<br />
Registered Office: <br />
Prospect Place<br />
The Hoe<br />
Plymouth  PL1 3DH
</font></p>

<p><font face="Arial" size="1">Website: <a href="http://www.pml.ac.uk">www.pml.ac.uk</a>
<br />
<a href="http://www.pml.ac.uk/pdf/PML%20Annual%20Review%202011_2.pdf">Click here for the latest PML Annual Review</a>
<br />
Registered Charity No. 1091222<br />
PML is a company limited by guarantee<br />
registered in England & Wales<br />
company number 4178503</font></p>

<p><font face="Arial" size="1" color="green">Please think before you print.</font></p>

<hr />

<p><font face="Arial" size="1">This e-mail, its content and any file attachments are confidential.</font></p>

<p><font face="Arial" size="1">If you have received this e-mail in error please do not copy, disclose it to any third party or use the contents or attachments in any way. Please notify the sender by replying to this e-mail or e-mail forinfo@pml.ac.uk and then delete the email without making any copies or using it in any other way.</font></p>

<p><font face="Arial" size="1">The content of this message may contain personal views which are not the views of Plymouth Marine Laboratory unless specifically stated.</font></p>

<p><font face="Arial" size="1">You are reminded that e-mail communications are not secure and may contain viruses. Plymouth Marine Laboratory accepts no liability for any loss or damage which may be caused by viruses.</font></p>

<hr />
<br />
<br />

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Recommendation on XFS Filesystem Creation, Lefthand P4500, iSCSI ?

[Peter Grandi]

> We recently purchased HP Lefthand 4500 G2 24TB iSCSI SAN Box.
> [ ... ] HP DL380 G7 with 24GB in RAM [ ... ]

> I need to create a 7TB volume at the start, and grow it in
> near future.

Why start small and grow? What's the point? Usually "grow" has
no problems, but any restructuring operation (on RAID or
filetrees) is a dangerous moment. Begging for trouble.

> Are there any recommendation on creating that XFS volume ?

Ideally create multiple small volumes over time and assign
different sets of users to them. For better 'fsck' and safety if
anything.

> Particularly regarding iSCSI and that big ?

Hopefully you won't be creating a single 24TB RAID6 set, but
that kind of (euphemism) strategy seems to be standard.

I would not configure the volumes exported by the iSCSI server
as sparse/thin, but in that case using 'discard' should help.

As an obvious note, but just to be sure, you will be running a
64b OS. But still be careful about stack overflows. You are
using 'netatalk' and Samba which are user-level services, so
those won't impact the kernel stack, but DM/LVM and NFS would.
A pity as the only good use of DM would be to create a snapshot
volume for "frozen" state backups. But it should still work,
after all 64b kernels have a larger kernel stack frame.

Then the major issue: backups. How are you going to create at
least 3-4 backups copies of your filetree(s)? Interesting issue.

> This server will act as a fileserver for 50+ machines serving
> request with netatalk and samba.

If you have performance requirements ideally the server will be
dual homed on an iSCSI-only *physical* (not VLAN) network and a
distinct user network (with different NICs). Tuning the Linux
IP/TCP parameters might help a fair bit too, and a 10Gb network
would be a big improvement over a 1Gb one, and short range 10Gb
cards are quite affordable (I particularly like Myri cards, but
the Dell Broadcom ones are also good).

> Surely I will need to use inode64 option. Right ?

Yes (IIRC it will soon become the default), as recommended in
http://www.spinics.net/lists/xfs/msg11455.html; as it says,
'inode64' has a secondary effect on allocation, which usually
helps, and in your case I think it does; anyhow with filetrees
much above 1TiB (with 512B sectors) it is hard to avoid
'inode64'.

But perhaps other most important point is to ensure that 'su=' and 'sw='
are set appropriately as you are using parity RAID. Usually the
'mkfs.xfs' asks the Linux kernel for geometry details, which
usually works when the RAID set is provided by MD or DM, but in
your case probably the RAID geometry will not be available from
the iSCSI layer, so probably you need to specify it manually.

Consider whether to enable barriers, and the answer is probably
yes (if supported), unless you have a really reliable storage
chains with battery backups and a very reliable kernel too.

Dramatically reducing the number of "dirty" pages in the page
cache usually is rather important too. It might be of benefit to
set the elevator to "noop".

Probably a sector size of 4096 is also a good idea, and on
recent storage system this is always true where possible.
Having jumbo frames enabled on your network (the iSCSI facing
one) would probably help a fair bit, regardless of sector size.

If you use lots of ACLs (common for fileservers) consider having
a larger-than-usual inode size, which also helps if you expect
to have large extent maps (very huge files or very fragmented
ones, for example mail archives). I personally tend to have
2048B inodes, but that is an arguable choice.

If you have a very reliable storage chain and high rates of file
creation and deletion you may want an external journal, but
probably your iSCSI host adapter has enough BBU backed cache
that it is not necessary. You may want to disable delayed
logging to reduce the vulnerability window otherwise.

PS I have adeptad here a number of hints about setting up XFS
   (and other types) of filetrees taken from:
   http://www.sabi.co.uk/Notes/linuxFS.html#fsHintsXFS

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs