* Linux NFS vs NetApp @ 2005-01-11 2:54 ` Phy Prabab 2005-01-11 3:58 ` Anton Blanchard 2005-01-11 9:54 ` Neil Brown 0 siblings, 2 replies; 15+ messages in thread From: Phy Prabab @ 2005-01-11 2:54 UTC (permalink / raw) To: linux-kernel Hello! I am trying to understand how NetApp can be so much better at NFS servicing than my quad Opteron 250 SAN attached machine. So I need some help and some pointers to understand how I can make my opteron machine come on par (or within 70% NFS performance range) as that of my NetApp R200. I have run through the NFS-how-to's and have heard "that is why they cost so much more", but I really have to consider that probably most of the ideas that are in the NetApp are common knowldge (just not in my head). Can anyone shed some light on this? TIA! Phy __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux NFS vs NetApp 2005-01-11 2:54 ` Linux NFS vs NetApp Phy Prabab @ 2005-01-11 3:58 ` Anton Blanchard 2005-01-11 7:42 ` Joel Jaeggli 2005-01-11 9:54 ` Neil Brown 1 sibling, 1 reply; 15+ messages in thread From: Anton Blanchard @ 2005-01-11 3:58 UTC (permalink / raw) To: Phy Prabab; +Cc: linux-kernel > I am trying to understand how NetApp can be so much > better at NFS servicing than my quad Opteron 250 SAN > attached machine. So I need some help and some > pointers to understand how I can make my opteron > machine come on par (or within 70% NFS performance > range) as that of my NetApp R200. I have run through > the NFS-how-to's and have heard "that is why they cost > so much more", but I really have to consider that > probably most of the ideas that are in the NetApp are > common knowldge (just not in my head). > > Can anyone shed some light on this? Definitely sounds like something is wrong. You can do your own comparisons of Linux 2.6 vs Netapp here (the OpenPower 720 is a ppc64 Linux box): http://www.spec.org/sfs97r1/results/sfs97r1.html Anton ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux NFS vs NetApp 2005-01-11 3:58 ` Anton Blanchard @ 2005-01-11 7:42 ` Joel Jaeggli 2005-01-11 9:19 ` Valdis.Kletnieks 0 siblings, 1 reply; 15+ messages in thread From: Joel Jaeggli @ 2005-01-11 7:42 UTC (permalink / raw) To: Anton Blanchard; +Cc: Phy Prabab, linux-kernel On Tue, 11 Jan 2005, Anton Blanchard wrote: > >> I am trying to understand how NetApp can be so much >> better at NFS servicing than my quad Opteron 250 SAN >> attached machine. So I need some help and some >> pointers to understand how I can make my opteron >> machine come on par (or within 70% NFS performance >> range) as that of my NetApp R200. I have run through >> the NFS-how-to's and have heard "that is why they cost >> so much more", but I really have to consider that >> probably most of the ideas that are in the NetApp are >> common knowldge (just not in my head). >> >> Can anyone shed some light on this? you have to quantify what sort of hardware you're benchmarking in either case and how its configured before you can reasonably conclude to much... I spent quite a bit of time benchmarking filers and linux configurations recently and while I can say with some certainty that while netapp makes some very fast and well balanced filers they don't by any means have a lock on building a high-performance nfs box. > Definitely sounds like something is wrong. You can do your own > comparisons of Linux 2.6 vs Netapp here (the OpenPower 720 is a ppc64 > Linux box): > > http://www.spec.org/sfs97r1/results/sfs97r1.html In actually using sfs97r1 published benchmarks to compare to hardware I was benchmarking (from emc, netapp and several roll-your own linux boxes) I found the published benchmark information alsmost entirely useless given that vendors tend to provide wildly silly hardware configurations. In the case of the openpower 720 (to use that for an example) the benchmarked machine has 70 15k rpm disks spread across 12 fibre channel controllers, 64GB of ram, 12GB of nvram and 7 network interfaces... > Anton > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > -- -------------------------------------------------------------------------- Joel Jaeggli Unix Consulting joelja@darkwing.uoregon.edu GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux NFS vs NetApp 2005-01-11 7:42 ` Joel Jaeggli @ 2005-01-11 9:19 ` Valdis.Kletnieks 2005-01-11 10:01 ` Jakob Oestergaard 0 siblings, 1 reply; 15+ messages in thread From: Valdis.Kletnieks @ 2005-01-11 9:19 UTC (permalink / raw) To: Joel Jaeggli; +Cc: Anton Blanchard, Phy Prabab, linux-kernel [-- Attachment #1: Type: text/plain, Size: 902 bytes --] On Mon, 10 Jan 2005 23:42:30 PST, Joel Jaeggli said: > In actually using sfs97r1 published benchmarks to compare to hardware I > was benchmarking (from emc, netapp and several roll-your own linux boxes) > I found the published benchmark information alsmost entirely useless given > that vendors tend to provide wildly silly hardware configurations. In the > case of the openpower 720 (to use that for an example) the benchmarked > machine has 70 15k rpm disks spread across 12 fibre channel controllers, > 64GB of ram, 12GB of nvram and 7 network interfaces... If you threw that much hardware at a Linux system, and then tuned it so that it didn't really care about userspace performance (oh.. say.. by giving the knfsd thread a RT priority ;), and tuned things like the filesystem, the slab allocator and the networking stack to NFS requirements, it probably would be screaming fast too.. ;) [-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux NFS vs NetApp 2005-01-11 9:19 ` Valdis.Kletnieks @ 2005-01-11 10:01 ` Jakob Oestergaard 2005-01-11 14:43 ` J. Bruce Fields 2005-01-11 18:55 ` Valdis.Kletnieks 0 siblings, 2 replies; 15+ messages in thread From: Jakob Oestergaard @ 2005-01-11 10:01 UTC (permalink / raw) To: Valdis.Kletnieks; +Cc: Joel Jaeggli, Anton Blanchard, Phy Prabab, linux-kernel On Tue, Jan 11, 2005 at 04:19:57AM -0500, Valdis.Kletnieks@vt.edu wrote: > On Mon, 10 Jan 2005 23:42:30 PST, Joel Jaeggli said: > > > In actually using sfs97r1 published benchmarks to compare to hardware I > > was benchmarking (from emc, netapp and several roll-your own linux boxes) > > I found the published benchmark information alsmost entirely useless given > > that vendors tend to provide wildly silly hardware configurations. In the > > case of the openpower 720 (to use that for an example) the benchmarked > > machine has 70 15k rpm disks spread across 12 fibre channel controllers, > > 64GB of ram, 12GB of nvram and 7 network interfaces... > > If you threw that much hardware at a Linux system, ... theory ... or have you actually tried? > and then tuned it so that it > didn't really care about userspace performance (oh.. say.. by giving the knfsd > thread a RT priority ;), and tuned things like the filesystem, the slab > allocator and the networking stack to NFS requirements, it probably would be > screaming fast too.. ;) You'd need to run a 2.4 kernel. Current problems with 2.6: 1 ext3 causes kjournald oops on load 2 xfs has bad NFS/SMP/dcache interactions (you end up with undeletable directories) 3 knfsd will give you stale handles (can be worked around by stat'ing all your directories constantly on the server side) The SGI XFS kernel from CVS actually almost solved (2) above, but not entirely - I was going to report on that again to LKML. The other problems are still, as far as I know, unsolved. Not trying to flame anyone here, just trying to be realistic ;) -- / jakob ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux NFS vs NetApp 2005-01-11 10:01 ` Jakob Oestergaard @ 2005-01-11 14:43 ` J. Bruce Fields 2005-01-12 11:32 ` Jakob Oestergaard 2005-01-11 18:55 ` Valdis.Kletnieks 1 sibling, 1 reply; 15+ messages in thread From: J. Bruce Fields @ 2005-01-11 14:43 UTC (permalink / raw) To: Jakob Oestergaard, Valdis.Kletnieks, Joel Jaeggli, Anton Blanchard, Phy Prabab, linux-kernel On Tue, Jan 11, 2005 at 11:01:10AM +0100, Jakob Oestergaard wrote: > 3 knfsd will give you stale handles (can be worked around by stat'ing > all your directories constantly on the server side) This should be fixed now. Bug reports to the contrary welcomed. --Bruce Fields ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux NFS vs NetApp 2005-01-11 14:43 ` J. Bruce Fields @ 2005-01-12 11:32 ` Jakob Oestergaard 0 siblings, 0 replies; 15+ messages in thread From: Jakob Oestergaard @ 2005-01-12 11:32 UTC (permalink / raw) To: J. Bruce Fields Cc: Valdis.Kletnieks, Joel Jaeggli, Anton Blanchard, Phy Prabab, linux-kernel On Tue, Jan 11, 2005 at 09:43:17AM -0500, J. Bruce Fields wrote: > On Tue, Jan 11, 2005 at 11:01:10AM +0100, Jakob Oestergaard wrote: > > 3 knfsd will give you stale handles (can be worked around by stat'ing > > all your directories constantly on the server side) > > This should be fixed now. Bug reports to the contrary welcomed. Excellent! It seems SGI has merged their XFS kernel up to 2.6.10 - I'll give that a try and see what happens. Thanks, -- / jakob ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux NFS vs NetApp 2005-01-11 10:01 ` Jakob Oestergaard 2005-01-11 14:43 ` J. Bruce Fields @ 2005-01-11 18:55 ` Valdis.Kletnieks 1 sibling, 0 replies; 15+ messages in thread From: Valdis.Kletnieks @ 2005-01-11 18:55 UTC (permalink / raw) To: Jakob Oestergaard; +Cc: Joel Jaeggli, Anton Blanchard, Phy Prabab, linux-kernel [-- Attachment #1: Type: text/plain, Size: 545 bytes --] On Tue, 11 Jan 2005 11:01:10 +0100, Jakob Oestergaard said: > > If you threw that much hardware at a Linux system, > > ... theory ... or have you actually tried? Merely indicating a method to approach the problem space... > Current problems with 2.6: > 1 ext3 causes kjournald oops on load > 2 xfs has bad NFS/SMP/dcache interactions (you end up with undeletable > directories) > 3 knfsd will give you stale handles (can be worked around by stat'ing > all your directories constantly on the server side) A mere matter of debugging. ;) [-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux NFS vs NetApp 2005-01-11 2:54 ` Linux NFS vs NetApp Phy Prabab 2005-01-11 3:58 ` Anton Blanchard @ 2005-01-11 9:54 ` Neil Brown 2005-01-11 11:38 ` Lincoln Dale 1 sibling, 1 reply; 15+ messages in thread From: Neil Brown @ 2005-01-11 9:54 UTC (permalink / raw) To: Phy Prabab; +Cc: linux-kernel On Monday January 10, phyprabab@yahoo.com wrote: > Hello! > > I am trying to understand how NetApp can be so much > better at NFS servicing than my quad Opteron 250 SAN > attached machine. So I need some help and some > pointers to understand how I can make my opteron > machine come on par (or within 70% NFS performance > range) as that of my NetApp R200. I have run through > the NFS-how-to's and have heard "that is why they cost > so much more", but I really have to consider that > probably most of the ideas that are in the NetApp are > common knowldge (just not in my head). > > Can anyone shed some light on this? If you want to come anything close to comparable with a Netapp, get a few hundred Megabytes of NVRAM (e.g. www.umem.com), and configure it as an external journal for your filesystem (I know this can be done for ext3, I don't know about other filesystems). Then make sure your filesystem journals all data, not just metadata (data=journal option to ext3). If you use a dedicated drive (or mirrored pair) in place of the NVRAM, you will come reasonably close. NeilBrown ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux NFS vs NetApp 2005-01-11 9:54 ` Neil Brown @ 2005-01-11 11:38 ` Lincoln Dale 2005-01-11 22:02 ` Neil Brown 0 siblings, 1 reply; 15+ messages in thread From: Lincoln Dale @ 2005-01-11 11:38 UTC (permalink / raw) To: Neil Brown; +Cc: Phy Prabab, linux-kernel At 08:54 PM 11/01/2005, Neil Brown wrote: >On Monday January 10, phyprabab@yahoo.com wrote: > > Hello! > > > > I am trying to understand how NetApp can be so much > > better at NFS servicing than my quad Opteron 250 SAN > > attached machine. So I need some help and some > > pointers to understand how I can make my opteron > > machine come on par (or within 70% NFS performance > > range) as that of my NetApp R200. I have run through > > the NFS-how-to's and have heard "that is why they cost > > so much more", but I really have to consider that > > probably most of the ideas that are in the NetApp are > > common knowldge (just not in my head). > > > > Can anyone shed some light on this? > >If you want to come anything close to comparable with a Netapp, get a >few hundred Megabytes of NVRAM (e.g. www.umem.com), and configure it >as an external journal for your filesystem (I know this can be done >for ext3, I don't know about other filesystems). Then make sure your >filesystem journals all data, not just metadata (data=journal option >to ext3). NetApp's WAFL only journals metadata in NVRAM ... (one of the primary reasons its called WAFL is that the data-write only happens once..). cheers, lincoln. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux NFS vs NetApp 2005-01-11 11:38 ` Lincoln Dale @ 2005-01-11 22:02 ` Neil Brown 2005-01-11 23:36 ` Bernd Eckenfels 0 siblings, 1 reply; 15+ messages in thread From: Neil Brown @ 2005-01-11 22:02 UTC (permalink / raw) To: Lincoln Dale; +Cc: Phy Prabab, linux-kernel On Tuesday January 11, ltd@cisco.com wrote: > At 08:54 PM 11/01/2005, Neil Brown wrote: > >If you want to come anything close to comparable with a Netapp, get a > >few hundred Megabytes of NVRAM (e.g. www.umem.com), and configure it > >as an external journal for your filesystem (I know this can be done > >for ext3, I don't know about other filesystems). Then make sure your > >filesystem journals all data, not just metadata (data=journal option > >to ext3). > > NetApp's WAFL only journals metadata in NVRAM ... > (one of the primary reasons its called WAFL is that the data-write only > happens once..). > That may be, though it doesn't fit with my (admittedly limitted) understanding of WAFL. However Linux NFS definitely runs faster over ext3 if data=journal is selected. NeilBrown ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux NFS vs NetApp 2005-01-11 22:02 ` Neil Brown @ 2005-01-11 23:36 ` Bernd Eckenfels 2005-01-12 0:41 ` Trond Myklebust 0 siblings, 1 reply; 15+ messages in thread From: Bernd Eckenfels @ 2005-01-11 23:36 UTC (permalink / raw) To: linux-kernel In article <16868.19707.857446.864762@cse.unsw.edu.au> you wrote: >> NetApp's WAFL only journals metadata in NVRAM ... >> (one of the primary reasons its called WAFL is that the data-write only >> happens once..). > That may be, though it doesn't fit with my (admittedly limitted) > understanding of WAFL. Yes, AFAIK the NVRAM is used for the RAID-4, independend of WAFL and as a write-back cache. However since also the read performance of Linux NFS is bad (at least not very well selftuning) the Hardware is not really the reason for the fast NFS implementation. Greetings Bernd ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux NFS vs NetApp 2005-01-11 23:36 ` Bernd Eckenfels @ 2005-01-12 0:41 ` Trond Myklebust 0 siblings, 0 replies; 15+ messages in thread From: Trond Myklebust @ 2005-01-12 0:41 UTC (permalink / raw) To: Bernd Eckenfels; +Cc: linux-kernel on den 12.01.2005 Klokka 00:36 (+0100) skreiv Bernd Eckenfels: > However since also the read performance of Linux NFS is bad (at least not > very well selftuning) the Hardware is not really the reason for the fast NFS > implementation. Indeed: NFS readahead requests are often processed out of order by the server (due to the basic unordered nature of RPC calls, the lack of ordering between nfsd server threads, use of UDP, etc) and so I suspect the generic readahead algorithm will tend to default to the random access mode in many cases where it should really be doing sequential access. Cheers, Trond -- Trond Myklebust <trond.myklebust@fys.uio.no> ^ permalink raw reply [flat|nested] 15+ messages in thread
[parent not found: <200501111711.50218.as@cohaesio.com>]
* Re: Linux NFS vs NetApp [not found] <200501111711.50218.as@cohaesio.com> @ 2005-01-11 16:21 ` J. Bruce Fields 2005-01-11 17:53 ` Anders Saaby 0 siblings, 1 reply; 15+ messages in thread From: J. Bruce Fields @ 2005-01-11 16:21 UTC (permalink / raw) To: Anders Saaby; +Cc: linux-kernel On Tue, Jan 11, 2005 at 05:11:50PM +0100, Anders Saaby wrote: > In which kernel version should this have been fixed? 2.6.10, I believe; see http://marc.theaimsgroup.com/?l=linux-nfs&m=110021733807921&w=2 --Bruce Fields ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux NFS vs NetApp 2005-01-11 16:21 ` J. Bruce Fields @ 2005-01-11 17:53 ` Anders Saaby 0 siblings, 0 replies; 15+ messages in thread From: Anders Saaby @ 2005-01-11 17:53 UTC (permalink / raw) To: J. Bruce Fields; +Cc: linux-kernel Hi, Thanks for the tip - it actually seems that it is fixed in 2.6.10. I am now subscribed to linux-nfs :) On Tuesday 11 January 2005 17:21, you wrote: > On Tue, Jan 11, 2005 at 05:11:50PM +0100, Anders Saaby wrote: > > In which kernel version should this have been fixed? > > 2.6.10, I believe; see > > http://marc.theaimsgroup.com/?l=linux-nfs&m=110021733807921&w=2 > > --Bruce Fields -- Med venlig hilsen - Best regards - Meilleures salutations Anders Saaby Systems Engineer ------------------------------------------------ Cohaesio A/S - Maglebjergvej 5D - DK-2800 Lyngby Phone: +45 45 880 888 - Fax: +45 45 880 777 Mail: as@cohaesio.com - http://www.cohaesio.com ------------------------------------------------ ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2005-01-12 11:32 UTC | newest] Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <message from Phy Prabab on Monday January 10> 2005-01-11 2:54 ` Linux NFS vs NetApp Phy Prabab 2005-01-11 3:58 ` Anton Blanchard 2005-01-11 7:42 ` Joel Jaeggli 2005-01-11 9:19 ` Valdis.Kletnieks 2005-01-11 10:01 ` Jakob Oestergaard 2005-01-11 14:43 ` J. Bruce Fields 2005-01-12 11:32 ` Jakob Oestergaard 2005-01-11 18:55 ` Valdis.Kletnieks 2005-01-11 9:54 ` Neil Brown 2005-01-11 11:38 ` Lincoln Dale 2005-01-11 22:02 ` Neil Brown 2005-01-11 23:36 ` Bernd Eckenfels 2005-01-12 0:41 ` Trond Myklebust [not found] <200501111711.50218.as@cohaesio.com> 2005-01-11 16:21 ` J. Bruce Fields 2005-01-11 17:53 ` Anders Saaby
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).