* stable 10GBE under 3.4.2 or 3.5.0-rc2
@ 2012-06-18 7:40 Stefan Priebe - Profihost AG
2012-06-18 8:11 ` Alexandre DERUMIER
2012-06-18 13:56 ` Mark Nelson
0 siblings, 2 replies; 8+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-06-18 7:40 UTC (permalink / raw)
To: ceph-devel
Hi list,
i've still problems with stable network speed under recent kernels. With
3.0.32 i get stable 9.90 Gbit/s in both directions. With 3.4.2 or
3.5.0-rc2 it drops sometimes down to around 1 Gbit/s. Sadly i've no idea
when this happen? It works sometimes for minutes at 9.9Gbit/s and then
suddently only with 3-4Gbit/s or even 1Gbit/s using recent kernels.
I'm using various tunings recommanded by intel (Improving Performance)
=> http://downloadmirror.intel.com/5874/eng/README.txt
Has anybody a hint for me or additional settings intel does not mention?
Stefan
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: stable 10GBE under 3.4.2 or 3.5.0-rc2
2012-06-18 7:40 stable 10GBE under 3.4.2 or 3.5.0-rc2 Stefan Priebe - Profihost AG
@ 2012-06-18 8:11 ` Alexandre DERUMIER
2012-06-18 9:11 ` Stefan Priebe - Profihost AG
2012-06-18 13:56 ` Mark Nelson
1 sibling, 1 reply; 8+ messages in thread
From: Alexandre DERUMIER @ 2012-06-18 8:11 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: ceph-devel
maybe try to play with net.ipv4.tcp_congestion_control ?
The available algorithms are:
• BIC - The default on Gentoo
• Reno - The classic TCP protocol. Most OSes use this.
• highspeed - HighSpeed TCP: Sally Floyd's suggested algorithm
• htcp - Hamilton TCP
• hybla - For Satellite Links
• scalable - Scalable TCP
• vegas - Vegas TCP
• westwood - Optimized for lossy networks
http://lwn.net/Articles/128681/
The High-speed TCP algorithm is optimized for very fat pipes - 10G Ethernet and such. When things are congested, it behaves much like the Reno algorithm. When the congestion window is being increased, however, the high-speed algorithm makes use of a table to pick large increment values. This approach lets the congestion window get very large (i.e. tens of thousands of segments) quickly, and to stay large, without requiring that the network function for long periods of time without a single dropped packet.
----- Mail original -----
De: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag>
À: ceph-devel@vger.kernel.org
Envoyé: Lundi 18 Juin 2012 09:40:47
Objet: stable 10GBE under 3.4.2 or 3.5.0-rc2
Hi list,
i've still problems with stable network speed under recent kernels. With
3.0.32 i get stable 9.90 Gbit/s in both directions. With 3.4.2 or
3.5.0-rc2 it drops sometimes down to around 1 Gbit/s. Sadly i've no idea
when this happen? It works sometimes for minutes at 9.9Gbit/s and then
suddently only with 3-4Gbit/s or even 1Gbit/s using recent kernels.
I'm using various tunings recommanded by intel (Improving Performance)
=> http://downloadmirror.intel.com/5874/eng/README.txt
Has anybody a hint for me or additional settings intel does not mention?
Stefan
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
--
Alexandre D erumier
Ingénieur Système
Fixe : 03 20 68 88 90
Fax : 03 20 68 90 81
45 Bvd du Général Leclerc 59100 Roubaix - France
12 rue Marivaux 75002 Paris - France
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: stable 10GBE under 3.4.2 or 3.5.0-rc2
2012-06-18 8:11 ` Alexandre DERUMIER
@ 2012-06-18 9:11 ` Stefan Priebe - Profihost AG
2012-06-18 9:21 ` Alexandre DERUMIER
0 siblings, 1 reply; 8+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-06-18 9:11 UTC (permalink / raw)
To: Alexandre DERUMIER; +Cc: ceph-devel
Am 18.06.2012 10:11, schrieb Alexandre DERUMIER:
> maybe try to play with net.ipv4.tcp_congestion_control ?
The default on my machines and even under RHEL6 is cubic. But i've now
alos tried, reno, bic and highspeed. But it doesn't change anything.
Everything is fine under 3.0.32 and pretty bad under 3.4.X or 3.5rc2.
Stefan
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: stable 10GBE under 3.4.2 or 3.5.0-rc2
2012-06-18 9:11 ` Stefan Priebe - Profihost AG
@ 2012-06-18 9:21 ` Alexandre DERUMIER
2012-06-18 9:25 ` Stefan Priebe - Profihost AG
2012-06-18 9:27 ` Alexandre DERUMIER
0 siblings, 2 replies; 8+ messages in thread
From: Alexandre DERUMIER @ 2012-06-18 9:21 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: ceph-devel
Did you use same driver version with the differents kernel version ?
Maybe try to desactive tso,gso,... with ethtool ?
----- Mail original -----
De: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag>
À: "Alexandre DERUMIER" <aderumier@odiso.com>
Cc: ceph-devel@vger.kernel.org
Envoyé: Lundi 18 Juin 2012 11:11:16
Objet: Re: stable 10GBE under 3.4.2 or 3.5.0-rc2
Am 18.06.2012 10:11, schrieb Alexandre DERUMIER:
> maybe try to play with net.ipv4.tcp_congestion_control ?
The default on my machines and even under RHEL6 is cubic. But i've now
alos tried, reno, bic and highspeed. But it doesn't change anything.
Everything is fine under 3.0.32 and pretty bad under 3.4.X or 3.5rc2.
Stefan
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: stable 10GBE under 3.4.2 or 3.5.0-rc2
2012-06-18 9:21 ` Alexandre DERUMIER
@ 2012-06-18 9:25 ` Stefan Priebe - Profihost AG
2012-06-18 9:27 ` Alexandre DERUMIER
1 sibling, 0 replies; 8+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-06-18 9:25 UTC (permalink / raw)
To: Alexandre DERUMIER; +Cc: ceph-devel
Am 18.06.2012 11:21, schrieb Alexandre DERUMIER:
> Did you use same driver version with the differents kernel version ?
Yes driver: ixgbe version: 3.9.17-NAPI.
> Maybe try to desactive tso,gso,... with ethtool ?
Already tried that no change and remember it works fine with 3.0.32.
Stefan
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: stable 10GBE under 3.4.2 or 3.5.0-rc2
2012-06-18 9:21 ` Alexandre DERUMIER
2012-06-18 9:25 ` Stefan Priebe - Profihost AG
@ 2012-06-18 9:27 ` Alexandre DERUMIER
1 sibling, 0 replies; 8+ messages in thread
From: Alexandre DERUMIER @ 2012-06-18 9:27 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: ceph-devel
also , I see a new feature in 3.3
byte_queue_limits
1.5. Bufferbloat fighting: Byte queue limits
https://lwn.net/Articles/454390/
"Bufferbloat" is a term used to describe the latency and throughput problems caused by excessive buffering trough the several elements of a network connection. Some tools are being developed to help to alleviate these problems, and this feature is one of them.
Byte queue limits are a configurable limit of packet data that can be put in the transmission queue of a network device. As a result one can tune things such that high priority packets get serviced with a reasonable amount of latency whilst not subjecting the hardware queue to emptying when data is available to send. Configuration of the queue limits is in the tx-<n> sysfs directory for the queue under the byte_queue_limits directory.
I see some bug report about spike:
http://www.mail-archive.com/e1000-devel@lists.sourceforge.net/msg05538.html
maybe can you try to play with values in
/sys/class/net/eth0/queues/tx-0/byte_queue_limits/*
----- Mail original -----
De: "Alexandre DERUMIER" <aderumier@odiso.com>
À: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag>
Cc: ceph-devel@vger.kernel.org
Envoyé: Lundi 18 Juin 2012 11:21:09
Objet: Re: stable 10GBE under 3.4.2 or 3.5.0-rc2
Did you use same driver version with the differents kernel version ?
Maybe try to desactive tso,gso,... with ethtool ?
----- Mail original -----
De: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag>
À: "Alexandre DERUMIER" <aderumier@odiso.com>
Cc: ceph-devel@vger.kernel.org
Envoyé: Lundi 18 Juin 2012 11:11:16
Objet: Re: stable 10GBE under 3.4.2 or 3.5.0-rc2
Am 18.06.2012 10:11, schrieb Alexandre DERUMIER:
> maybe try to play with net.ipv4.tcp_congestion_control ?
The default on my machines and even under RHEL6 is cubic. But i've now
alos tried, reno, bic and highspeed. But it doesn't change anything.
Everything is fine under 3.0.32 and pretty bad under 3.4.X or 3.5rc2.
Stefan
--
--
Alexandre D erumier
Ingénieur Système
Fixe : 03 20 68 88 90
Fax : 03 20 68 90 81
45 Bvd du Général Leclerc 59100 Roubaix - France
12 rue Marivaux 75002 Paris - France
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: stable 10GBE under 3.4.2 or 3.5.0-rc2
2012-06-18 7:40 stable 10GBE under 3.4.2 or 3.5.0-rc2 Stefan Priebe - Profihost AG
2012-06-18 8:11 ` Alexandre DERUMIER
@ 2012-06-18 13:56 ` Mark Nelson
2012-06-18 14:03 ` Stefan Priebe - Profihost AG
1 sibling, 1 reply; 8+ messages in thread
From: Mark Nelson @ 2012-06-18 13:56 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: ceph-devel
On 6/18/12 2:40 AM, Stefan Priebe - Profihost AG wrote:
> Hi list,
>
> i've still problems with stable network speed under recent kernels. With
> 3.0.32 i get stable 9.90 Gbit/s in both directions. With 3.4.2 or
> 3.5.0-rc2 it drops sometimes down to around 1 Gbit/s. Sadly i've no idea
> when this happen? It works sometimes for minutes at 9.9Gbit/s and then
> suddently only with 3-4Gbit/s or even 1Gbit/s using recent kernels.
>
> I'm using various tunings recommanded by intel (Improving Performance)
> => http://downloadmirror.intel.com/5874/eng/README.txt
>
Hi Stefan,
Did you ever get a chance to talk to Jason Wang about the commit that
was causing the problems? It might be a good idea to report all of this
upstream and see what they have to say.
Mark
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: stable 10GBE under 3.4.2 or 3.5.0-rc2
2012-06-18 13:56 ` Mark Nelson
@ 2012-06-18 14:03 ` Stefan Priebe - Profihost AG
0 siblings, 0 replies; 8+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-06-18 14:03 UTC (permalink / raw)
To: Mark Nelson; +Cc: ceph-devel
Am 18.06.2012 15:56, schrieb Mark Nelson:
> On 6/18/12 2:40 AM, Stefan Priebe - Profihost AG wrote:
>> Hi list,
>>
>> i've still problems with stable network speed under recent kernels. With
>> 3.0.32 i get stable 9.90 Gbit/s in both directions. With 3.4.2 or
>> 3.5.0-rc2 it drops sometimes down to around 1 Gbit/s. Sadly i've no idea
>> when this happen? It works sometimes for minutes at 9.9Gbit/s and then
>> suddently only with 3-4Gbit/s or even 1Gbit/s using recent kernels.
>>
>> I'm using various tunings recommanded by intel (Improving Performance)
>> => http://downloadmirror.intel.com/5874/eng/README.txt
>>
>
> Hi Stefan,
>
> Did you ever get a chance to talk to Jason Wang about the commit that
> was causing the problems? It might be a good idea to report all of this
> upstream and see what they have to say.
Hi Mark,
yes i talked to him and he told me that his change was partically
removed in more recent kernel version.
Right now i'm in discussion with eric dumazet at the netdev kernel
mailinglist. And he gives me some good advices which seem to work well.
I'm still testing and will share the results here.
Stefan
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2012-06-18 14:03 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-06-18 7:40 stable 10GBE under 3.4.2 or 3.5.0-rc2 Stefan Priebe - Profihost AG
2012-06-18 8:11 ` Alexandre DERUMIER
2012-06-18 9:11 ` Stefan Priebe - Profihost AG
2012-06-18 9:21 ` Alexandre DERUMIER
2012-06-18 9:25 ` Stefan Priebe - Profihost AG
2012-06-18 9:27 ` Alexandre DERUMIER
2012-06-18 13:56 ` Mark Nelson
2012-06-18 14:03 ` Stefan Priebe - Profihost AG
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.