From: Justin Piszcz <jpiszcz@lucidpixels.com>
To: Stan Hoeppner <stan@hardwarefreak.com>
Cc: "Dr. David Alan Gilbert" <linux@treblig.org>,
Emmanuel Florac <eflorac@intellique.com>,
linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org,
linux-net@vger.kernel.org, Alan Piszcz <ap@solarrain.com>
Subject: Re: Supermicro X8DTH-6: Only ~250MiB/s from RAID<->RAID over 10GbE?
Date: Sun, 6 Feb 2011 08:46:08 -0500 (EST) [thread overview]
Message-ID: <alpine.DEB.2.00.1102060817590.28109@p34.internal.lan> (raw)
In-Reply-To: <alpine.DEB.2.00.1102060515230.28109@p34.internal.lan>
On Sun, 6 Feb 2011, Justin Piszcz wrote:
>
>
> On Sat, 5 Feb 2011, Stan Hoeppner wrote:
>
>> Justin Piszcz put forth on 2/5/2011 7:08 PM:
>>
Hi,
1. Defaults below:
sysctl -w net.core.wmem_max=131071
sysctl -w net.core.rmem_max=131071
sysctl -w net.core.wmem_default=118784
sysctl -w net.core.rmem_default=118784
sysctl -w net.core.optmem_max=20480
sysctl -w net.ipv4.igmp_max_memberships=20
sysctl -w net.ipv4.tcp_mem="379104 505472 758208"
sysctl -w net.ipv4.tcp_wmem="4096 16384 4194304"
sysctl -w net.ipv4.tcp_rmem="4096 87380 4194304"
sysctl -w net.ipv4.udp_mem="379104 505472 758208"
sysctl -w net.ipv4.udp_rmem_min=4096
sysctl -w net.ipv4.udp_wmem_min=4096
sysctl -w net.core.netdev_max_backlog=1024
2. Optimized settings, for > 800MiB/:
# for 3ware raid, use 16384 readahead, > 16384 readahead, no improvement
blockdev --setra 16384 /dev/sda
# not sure if this helps much
ethtool -K eth0 lro on
# seems to get performance > 600-700MiB/s faster
sysctl -w net.core.wmem_max=4194304
sysctl -w net.core.rmem_max=4194304
sysctl -w net.core.wmem_default=4194304
sysctl -w net.core.rmem_default=4194304
sysctl -w net.core.optmem_max=20480
sysctl -w net.ipv4.igmp_max_memberships=20
sysctl -w net.ipv4.tcp_mem="4194304 4194304 4194304"
sysctl -w net.ipv4.tcp_wmem="4194304 4194304 4194304"
sysctl -w net.ipv4.tcp_rmem="4194304 4194304 4194304"
sysctl -w net.ipv4.udp_mem="4194304 4194304 4194304"
sysctl -w net.ipv4.udp_rmem_min=4096
sysctl -w net.ipv4.udp_wmem_min=4096
sysctl -w net.core.netdev_max_backlog=1048576
# the main option that makes all of the difference, the golden option
# is the rszie and wsize of 1megabyte below:
10.0.1.4:/r1 /nfs/box2/r1 nfs tcp,bg,rw,hard,intr,nolock,nfsvers=3,rsize=1048576,wsize=1048576 0 0
CPU utilization:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2069 root 20 0 18640 1304 688 R 91 0.0 0:15.50 cp
703 root 20 0 0 0 0 S 25 0.0 2:46.95 kswapd0
With a single copy I get roughly 700-800MiB/s:
Device eth0 [10.0.1.3] (1/1):
================================================================================
Incoming:
###################### #################### ####
###################### #################### ####
###################### #################### ####
###################### #################### ####
###################### #################### ####
###################### #################### #### Curr: 808.71 MByte/s
###################### #################### #### Avg: 706.11 MByte/s
###################### #################### #### Min: 0.00 MByte/s
###################### #################### #### Max: 860.17 MByte/s
###################### #################### #### Ttl: 344.70 GByte
With two copies I get up to 830-850MiB/s:
Device eth0 [10.0.1.3] (1/1):
================================================================================
Incoming:
############################################ ####
############################################ ####
############################################ ####
############################################ ####
############################################ ####
############################################ #### Curr: 846.61 MByte/s
############################################ #### Avg: 683.14 MByte/s
############################################ #### Min: 0.00 MByte/s
############################################ #### Max: 860.17 MByte/s
############################################ #### Ttl: 305.71 GByte
Using a 4MiB r/w size with NFS improves performance to sustain > 750MiB/s
a little better I think:
10.0.1.4:/r1 /nfs/box2/r1 nfs tcp,bg,rw,hard,intr,nolock,nfsvers=3,rsize=4194304,wsize=4194304 0
Anyhow, roughly 750-850MiB/s it would be nice to get 1Gbyte/sec but I guess
the kerrnel (or my HW, CPU not fast enough) is not there yet.
Also found a good doc from RedHat:
http://www.redhat.com/promo/summit/2008/downloads/pdf/Thursday/Mark_Wagner.pdf
Justin.
next prev parent reply other threads:[~2011-02-06 13:46 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-02-05 19:35 Supermicro X8DTH-6: Only ~250MiB/s from RAID<->RAID over 10GbE? Justin Piszcz
2011-02-05 19:42 ` Jean Gobin
2011-02-05 19:54 ` Justin Piszcz
2011-02-05 20:18 ` Justin Piszcz
2011-02-05 20:39 ` Stan Hoeppner
2011-02-05 20:45 ` Emmanuel Florac
2011-02-05 21:05 ` Justin Piszcz
2011-02-05 22:06 ` Dr. David Alan Gilbert
2011-02-05 22:06 ` Dr. David Alan Gilbert
2011-02-05 22:30 ` Justin Piszcz
2011-02-05 22:56 ` Justin Piszcz
2011-02-05 23:54 ` Stan Hoeppner
2011-02-06 1:08 ` Justin Piszcz
2011-02-06 3:16 ` Stan Hoeppner
2011-02-06 10:16 ` Justin Piszcz
2011-02-06 13:46 ` Justin Piszcz [this message]
2011-02-06 16:44 ` Emmanuel Florac
2011-02-06 16:55 ` Zdenek Kaspar
2011-02-06 17:52 ` Justin Piszcz
2011-02-06 22:01 ` Stan Hoeppner
2011-02-07 3:59 ` Julian Calaby
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=alpine.DEB.2.00.1102060817590.28109@p34.internal.lan \
--to=jpiszcz@lucidpixels.com \
--cc=ap@solarrain.com \
--cc=eflorac@intellique.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-net@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=linux@treblig.org \
--cc=stan@hardwarefreak.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.