3ware hardware raid with battery backup and the impact on barrier and no write cache options.

3ware hardware raid with battery backup and the impact on barrier and no write cache options.

[William Lewis]

[-- Attachment #1.1: Type: text/plain, Size: 1316 bytes --]

Hi,

I am in the process of setting up an XFS file system on underlying hardware
consisting of a 3ware 9550SXU (+ battery backup module) and 4 x Seagate
ST31500341AS 1.5TB hard drives in Raid 5 configuration.

Reading your FAQ at http://xfs.org/index.php/XFS_FAQ I understand that it is
advisable to mount the file system with nobarrier to improve performance.
However going on to read about recommended settings for write cache, the
advice for 3ware hardware doesn't seem to account for the fact that there
are 2 levels of write cache in play, that in the 3ware card itself protected
by the battery and the write cache of the disks themselves, which as far as
I can understand is also protected by the battery backup (in the correct
storage modes - balanced/protection) because the 3ware card uses write
journaling to keep track of pending write operations in the disks' cache.
Therefore unless I am misunderstanding something the most benefit is to be
gained by mounting with nobarrier and having the write cache turned on?

One thing I am not clear about is if nobarrier interacts with the page cache
at all and if the lack of barrier leaves you with a situation in which
pending writes can be lost from main memory on power failure?

Thanks in advance for any clarification you can provide.

Regards

Will Lewis

[-- Attachment #1.2: Type: text/html, Size: 1418 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 3ware hardware raid with battery backup and the impact on barrier and no write cache options.

[Justin Piszcz]

On Mon, 2 Nov 2009, William Lewis wrote:

> Hi,
>
> I am in the process of setting up an XFS file system on underlying hardware
> consisting of a 3ware 9550SXU (+ battery backup module) and 4 x Seagate
> ST31500341AS 1.5TB hard drives in Raid 5 configuration.
>
> Reading your FAQ at http://xfs.org/index.php/XFS_FAQ I understand that it is
> advisable to mount the file system with nobarrier to improve performance.
> However going on to read about recommended settings for write cache, the
> advice for 3ware hardware doesn't seem to account for the fact that there
> are 2 levels of write cache in play, that in the 3ware card itself protected
> by the battery and the write cache of the disks themselves, which as far as
> I can understand is also protected by the battery backup (in the correct
> storage modes - balanced/protection) because the 3ware card uses write
> journaling to keep track of pending write operations in the disks' cache.
> Therefore unless I am misunderstanding something the most benefit is to be
> gained by mounting with nobarrier and having the write cache turned on?
>
> One thing I am not clear about is if nobarrier interacts with the page cache
> at all and if the lack of barrier leaves you with a situation in which
> pending writes can be lost from main memory on power failure?
>
> Thanks in advance for any clarification you can provide.
>
> Regards
>
> Will Lewis
>

I am also curious, I have a 16 drive RAID-6 configuration on a 
9650SE-16ML and using -o nobarrier or mounting normal the speed/benchmarks 
seemed to be the same.  Either barriers are not enabled by default for 
3ware RAID arrays or they make no difference in performance?

Justin.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 3ware hardware raid with battery backup and the impact on barrier and no write cache options.

[Michael Monnerie]

On Montag 02 November 2009 Justin Piszcz wrote:
> I am also curious, I have a 16 drive RAID-6 configuration on a
> 9650SE-16ML and using -o nobarrier or mounting normal the
> speed/benchmarks seemed to be the same.  Either barriers are not
> enabled by default for 3ware RAID arrays or they make no difference
> in performance?

I'd say a RAID controller with it's own write cache enabled + BBM will 
effectively turn barriers off, even if you can use them as a mount 
options. What happens with barriers, is that it writes 
1,2,3,barrier,4,5,barrier,6 etc. so, that 123 are sure on disk before 45 
happen etc.
The RAID controller will happily tell you it did flush everything, 
because as soon as data is in it's cache, it's claimed sure that the 
data gets written, and therefore it will tell that the barrier is 
already done. And that's why it's a *must* to turn off disk cache 
writes, because the filesystem got it's barrier ACK and believes it, the 
controller has it's cache written to disk, powerfail.... all gone. That 
will do a bigger damage than any single disk could have done.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 3ware hardware raid with battery backup and the impact on barrier and no write cache options.

[Eric Sandeen]

William Lewis wrote:
> Hi,
> 
> I am in the process of setting up an XFS file system on underlying 
> hardware consisting of a 3ware 9550SXU (+ battery backup module) and 4 x 
> Seagate ST31500341AS 1.5TB hard drives in Raid 5 configuration.
> 
> Reading your FAQ at http://xfs.org/index.php/XFS_FAQ I understand that 
> it is advisable to mount the file system with nobarrier to improve 
> performance. However going on to read about recommended settings for 
> write cache, the advice for 3ware hardware doesn't seem to account for 
> the fact that there are 2 levels of write cache in play, that in the 
> 3ware card itself protected by the battery and the write cache of the 
> disks themselves, which as far as I can understand is also protected by 
> the battery backup (in the correct storage modes - balanced/protection) 
> because the 3ware card uses write journaling to keep track of pending 
> write operations in the disks' cache. Therefore unless I am 
> misunderstanding something the most benefit is to be gained by mounting 
> with nobarrier and having the write cache turned on?

If the write caches won't go away - or will be fully/gracefully destaged 
before they do, then nobarrier should be safe.

> One thing I am not clear about is if nobarrier interacts with the page 
> cache at all and if the lack of barrier leaves you with a situation in 
> which pending writes can be lost from main memory on power failure?

nobarrier has no interaction with the OS page cache; all the "barrier" 
option (the default) does is enforce ordering for journal IO*, and in 
practice it does this by flushing the cache at points in time.

-Eric

*well, I think it flushes the drive cache on fsync, too, for data 
integrity (vs. the metadata integrity for the journal IO ordering)

> Thanks in advance for any clarification you can provide.
> 
> Regards
> 
> Will Lewis
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 3ware hardware raid with battery backup and the impact on barrier and no write cache options.

[Michael Monnerie]

On Montag 02 November 2009 William Lewis wrote:
> Reading your FAQ at http://xfs.org/index.php/XFS_FAQ I understand
> that it is advisable to mount the file system with nobarrier to
> improve performance. 

I edited that part, so I'll answer.

> However going on to read about recommended
> settings for write cache, the advice for 3ware hardware doesn't seem
> to account for the fact that there are 2 levels of write cache in
> play, that in the 3ware card itself protected by the battery and the
> write cache of the disks themselves,

There are 2 different caches. One in the controller, on on the disks.

> which as far as I can understand
> is also protected by the battery backup (in the correct storage modes
> - balanced/protection) because the 3ware card uses write journaling
> to keep track of pending write operations in the disks' cache.

I can't believe it's possible for the controller to know when a disk 
write is actually on disk while the disk write cache is on. There are 
lots of disk who plainly *lie* about that fact. I've seen a web page 
once with a listing of disks who lie - quiet long. I don't find the link 
ATM, maybe googling helps.

> Therefore unless I am misunderstanding something the most benefit is
> to be gained by mounting with nobarrier and having the write cache
> turned on?

If you care about your data, don't turn on the disk write cache. I 
wonder why there's not a single disk vendor building a disk with a small 
battery buffer that is able to keep the disk spinning until it can 
savely empty disk cache to platters. Would be a real performance benefit 
in server environments.

> One thing I am not clear about is if nobarrier interacts with the
> page cache at all and if the lack of barrier leaves you with a
> situation in which pending writes can be lost from main memory on
> power failure?

Writes in main RAM will always be lost on power fail. If you need 
protection for that, use fsync(). The thing about barries is to ensure 
that metadata is kept consistent on powerfail. It's in fact nothing to 
protect your data, only the filesystem metadata. Well, in the end your 
data also, as the filesystem survives.

> Thanks in advance for any clarification you can provide.

I hope I could help.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs