From: "Hefty, Sean" <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
To: Gandalf Corvotempesta
<gandalf.corvotempesta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>,
"linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org"
<linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
Subject: RE: Slow performance with librspreload.so
Date: Fri, 30 Aug 2013 17:38:25 +0000 [thread overview]
Message-ID: <1828884A29C6694DAF28B7E6B8A8237388CAA1B9@ORSMSX109.amr.corp.intel.com> (raw)
In-Reply-To: <CAJH6TXidOVJDTokOwxjCT9RRTOO6E_xbyG_K++YVkFx8NbhMTA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
> Another strange issue:
>
> $ sudo LD_PRELOAD=/usr/local/lib/rsocket/librspreload.so iperf -c
> 172.17.0.2
> ------------------------------------------------------------
> Client connecting to 172.17.0.2, TCP port 5001
> TCP window size: 128 KByte (default)
Increasing the window size may improve the results. E.g. on my systems I go from 17.7 Gbps at 128 KB to 24.3 Gbps for 512 KB.
> ------------------------------------------------------------
> [ 3] local 172.17.0.1 port 57926 connected with 172.17.0.2 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0-10.0 sec 12.2 GBytes 10.4 Gbits/sec
>
> $ iperf -c 172.17.0.2
> ------------------------------------------------------------
> Client connecting to 172.17.0.2, TCP port 5001
> TCP window size: 648 KByte (default)
> ------------------------------------------------------------
> [ 3] local 172.17.0.1 port 58113 connected with 172.17.0.2 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0-10.0 sec 14.5 GBytes 12.5 Gbits/sec
>
> rsocket slower than IPoIB ?
This is surprising to me - just getting 12.5 Gbps out of ipoib is surprising. Does iperf use sendfile()?
My results with iperf (version 2.0.5) over ipoib (default configurations) vary considerably based on the TCP window size. (Note that this is a 40 Gbps link.) Results summarized:
TCP window size: 27.9 KByte (default)
[ 3] 0.0-10.0 sec 12.8 GBytes 11.0 Gbits/sec
TCP window size: 416 KByte (WARNING: requested 500 KByte)
[ 3] 0.0-10.0 sec 8.19 GBytes 7.03 Gbits/sec
TCP window size: 250 KByte (WARNING: requested 125 KByte)
[ 3] 0.0-10.0 sec 4.99 GBytes 4.29 Gbits/sec
I'm guessing that there are some settings I can change to increase the ipoib performance on my systems. Using rspreload, I get:
LD_PRELOAD=/usr/local/lib/rsocket/librspreload.so iperf -c 192.168.0.103
TCP window size: 512 KByte (default)
[ 3] 0.0-10.0 sec 28.3 GBytes 24.3 Gbits/sec
It seems that ipoib bandwidth should be close to rsockets, similar to what you see. I also don't understand the effect that the TCP window size is having on the results. The smallest window gives the best bandwidth for ipoib?!
- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2013-08-30 17:38 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-08-28 15:20 Slow performance with librspreload.so Gandalf Corvotempesta
[not found] ` <CAJH6TXgf2LeMH+1L290w_KZ5tTN7NWpQxntF58Z506G3h_qKVw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-28 15:50 ` Hefty, Sean
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA937E-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2013-08-28 16:19 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXjEx+41G_7wvQybMXzb60tu-ha2d2Bu_J_erNDPJRbQFw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-28 17:16 ` Hefty, Sean
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA96AD-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2013-08-28 18:24 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXhAuSDytS5O1cJMg3iatq+STkwhPUG2zmexJ5tmt3Foqg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-28 19:20 ` Hefty, Sean
2013-08-28 20:05 ` Hefty, Sean
[not found] ` <CAJH6TXgA0ghKX1P8UUAMFKY9o0xBJ0j4-kFa_M4a4ecdzoD0HA@mail.gmail.com>
[not found] ` <CAJH6TXgA0ghKX1P8UUAMFKY9o0xBJ0j4-kFa_M4a4ecdzoD0HA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-29 13:03 ` Fwd: " Gandalf Corvotempesta
2013-08-29 13:03 ` Gandalf Corvotempesta
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA9C6B@ORSMSX109.amr.corp.intel.com>
[not found] ` <CAJH6TXiYLKt3b1UFsZt7uFwDbWcDFnHNnS8CTO24Gt-2zn+Qiw@mail.gmail.com>
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA9D8A@ORSMSX109.amr.corp.intel.com>
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA9D8A-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2013-08-29 18:57 ` Gandalf Corvotempesta
2013-08-30 8:19 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXhWWMBbopDLZY2+rrNOm2m5gcmObj7Sr16u2qrNW_NHgw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-30 8:23 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXgR=wVGyyHrpTKkBw-5M=A9-tGzjpYcV6NWpk8uKvFn8Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-30 10:29 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXidOVJDTokOwxjCT9RRTOO6E_xbyG_K++YVkFx8NbhMTA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-30 17:38 ` Hefty, Sean [this message]
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CAA1B9-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2013-08-30 18:08 ` Atchley, Scott
2013-08-30 15:51 ` Hefty, Sean
[not found] ` <CAJH6TXgu4L8gnqQX1fKZ=ioZDxUMnj=s3h0qkYh1_35VWpMJ1g@mail.gmai! l.com>
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CAA11B-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2013-08-30 16:26 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXgvViraH4SEcYydCerGyM6kK61eoiaENCy6PSf_1ocSVA@mail.gmai! l.com>
[not found] ` <CAJH6TXgvViraH4SEcYydCerGyM6kK61eoiaENCy6PSf_1ocSVA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-30 18:19 ` Rupert Dance
2013-08-31 9:20 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXgu4L8gnqQX1fKZ=ioZDxUMnj=s3h0qkYh1_35VWpMJ1g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-31 11:34 ` Rupert Dance
2013-08-31 19:51 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXiOToa2-EOj6Hz-rkVHt3tCSh4jnLbKZs69GhT7dFeH0A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-09-01 13:28 ` Rupert Dance
2013-09-01 17:41 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXj53RZmzqA94CfPJjXBetK7us5v=cFyoMmjVUNQ-oeqpQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-09-03 9:17 ` Gandalf Corvotempesta
2013-09-03 12:06 ` Hal Rosenstock
[not found] ` <5225D0DD.7060803-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2013-09-03 12:21 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXhSoxFshtuE0YqdVShQKSgX_wNwxekjy1+ZcGHR0vjC9w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-09-16 16:18 ` Gandalf Corvotempesta
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1828884A29C6694DAF28B7E6B8A8237388CAA1B9@ORSMSX109.amr.corp.intel.com \
--to=sean.hefty-ral2jqcrhueavxtiumwx3w@public.gmane.org \
--cc=gandalf.corvotempesta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org \
--cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.