From: "Atchley, Scott" <atchleyes-1Heg1YXhbW8@public.gmane.org>
To: "Hefty, Sean" <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Cc: Gandalf Corvotempesta
<gandalf.corvotempesta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>,
"linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org"
<linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
Subject: Re: Slow performance with librspreload.so
Date: Fri, 30 Aug 2013 14:08:18 -0400 [thread overview]
Message-ID: <4AD98A71-E24B-4F99-B0D0-9F492FB932D1@ornl.gov> (raw)
In-Reply-To: <1828884A29C6694DAF28B7E6B8A8237388CAA1B9-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
On Aug 30, 2013, at 1:38 PM, "Hefty, Sean" <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>> Another strange issue:
>>
>> $ sudo LD_PRELOAD=/usr/local/lib/rsocket/librspreload.so iperf -c
>> 172.17.0.2
>> ------------------------------------------------------------
>> Client connecting to 172.17.0.2, TCP port 5001
>> TCP window size: 128 KByte (default)
>
> Increasing the window size may improve the results. E.g. on my systems I go from 17.7 Gbps at 128 KB to 24.3 Gbps for 512 KB.
>
>> ------------------------------------------------------------
>> [ 3] local 172.17.0.1 port 57926 connected with 172.17.0.2 port 5001
>> [ ID] Interval Transfer Bandwidth
>> [ 3] 0.0-10.0 sec 12.2 GBytes 10.4 Gbits/sec
>>
>> $ iperf -c 172.17.0.2
>> ------------------------------------------------------------
>> Client connecting to 172.17.0.2, TCP port 5001
>> TCP window size: 648 KByte (default)
>> ------------------------------------------------------------
>> [ 3] local 172.17.0.1 port 58113 connected with 172.17.0.2 port 5001
>> [ ID] Interval Transfer Bandwidth
>> [ 3] 0.0-10.0 sec 14.5 GBytes 12.5 Gbits/sec
>>
>> rsocket slower than IPoIB ?
>
> This is surprising to me - just getting 12.5 Gbps out of ipoib is surprising. Does iperf use sendfile()?
I have a pair of nodes connected by QDR via a switch. Using normal IPoIB, a single Netperf can reach 18.4 Gb/s if I bind to the same core that the IRQ handler is bound to. With four concurrent Netperfs, I can reach 23 Gb/s. This is in datagram mode. Connected mode is slower.
I have not tried rsockets on these nodes.
Scott
>
> My results with iperf (version 2.0.5) over ipoib (default configurations) vary considerably based on the TCP window size. (Note that this is a 40 Gbps link.) Results summarized:
>
> TCP window size: 27.9 KByte (default)
> [ 3] 0.0-10.0 sec 12.8 GBytes 11.0 Gbits/sec
>
> TCP window size: 416 KByte (WARNING: requested 500 KByte)
> [ 3] 0.0-10.0 sec 8.19 GBytes 7.03 Gbits/sec
>
> TCP window size: 250 KByte (WARNING: requested 125 KByte)
> [ 3] 0.0-10.0 sec 4.99 GBytes 4.29 Gbits/sec
>
> I'm guessing that there are some settings I can change to increase the ipoib performance on my systems. Using rspreload, I get:
>
> LD_PRELOAD=/usr/local/lib/rsocket/librspreload.so iperf -c 192.168.0.103
> TCP window size: 512 KByte (default)
> [ 3] 0.0-10.0 sec 28.3 GBytes 24.3 Gbits/sec
>
> It seems that ipoib bandwidth should be close to rsockets, similar to what you see. I also don't understand the effect that the TCP window size is having on the results. The smallest window gives the best bandwidth for ipoib?!
>
> - Sean
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2013-08-30 18:08 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-08-28 15:20 Slow performance with librspreload.so Gandalf Corvotempesta
[not found] ` <CAJH6TXgf2LeMH+1L290w_KZ5tTN7NWpQxntF58Z506G3h_qKVw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-28 15:50 ` Hefty, Sean
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA937E-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2013-08-28 16:19 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXjEx+41G_7wvQybMXzb60tu-ha2d2Bu_J_erNDPJRbQFw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-28 17:16 ` Hefty, Sean
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA96AD-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2013-08-28 18:24 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXhAuSDytS5O1cJMg3iatq+STkwhPUG2zmexJ5tmt3Foqg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-28 19:20 ` Hefty, Sean
2013-08-28 20:05 ` Hefty, Sean
[not found] ` <CAJH6TXgA0ghKX1P8UUAMFKY9o0xBJ0j4-kFa_M4a4ecdzoD0HA@mail.gmail.com>
[not found] ` <CAJH6TXgA0ghKX1P8UUAMFKY9o0xBJ0j4-kFa_M4a4ecdzoD0HA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-29 13:03 ` Fwd: " Gandalf Corvotempesta
2013-08-29 13:03 ` Gandalf Corvotempesta
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA9C6B@ORSMSX109.amr.corp.intel.com>
[not found] ` <CAJH6TXiYLKt3b1UFsZt7uFwDbWcDFnHNnS8CTO24Gt-2zn+Qiw@mail.gmail.com>
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA9D8A@ORSMSX109.amr.corp.intel.com>
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA9D8A-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2013-08-29 18:57 ` Gandalf Corvotempesta
2013-08-30 8:19 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXhWWMBbopDLZY2+rrNOm2m5gcmObj7Sr16u2qrNW_NHgw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-30 8:23 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXgR=wVGyyHrpTKkBw-5M=A9-tGzjpYcV6NWpk8uKvFn8Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-30 10:29 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXidOVJDTokOwxjCT9RRTOO6E_xbyG_K++YVkFx8NbhMTA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-30 17:38 ` Hefty, Sean
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CAA1B9-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2013-08-30 18:08 ` Atchley, Scott [this message]
2013-08-30 15:51 ` Hefty, Sean
[not found] ` <CAJH6TXgu4L8gnqQX1fKZ=ioZDxUMnj=s3h0qkYh1_35VWpMJ1g@mail.gmai! l.com>
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CAA11B-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2013-08-30 16:26 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXgvViraH4SEcYydCerGyM6kK61eoiaENCy6PSf_1ocSVA@mail.gmai! l.com>
[not found] ` <CAJH6TXgvViraH4SEcYydCerGyM6kK61eoiaENCy6PSf_1ocSVA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-30 18:19 ` Rupert Dance
2013-08-31 9:20 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXgu4L8gnqQX1fKZ=ioZDxUMnj=s3h0qkYh1_35VWpMJ1g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-31 11:34 ` Rupert Dance
2013-08-31 19:51 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXiOToa2-EOj6Hz-rkVHt3tCSh4jnLbKZs69GhT7dFeH0A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-09-01 13:28 ` Rupert Dance
2013-09-01 17:41 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXj53RZmzqA94CfPJjXBetK7us5v=cFyoMmjVUNQ-oeqpQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-09-03 9:17 ` Gandalf Corvotempesta
2013-09-03 12:06 ` Hal Rosenstock
[not found] ` <5225D0DD.7060803-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2013-09-03 12:21 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXhSoxFshtuE0YqdVShQKSgX_wNwxekjy1+ZcGHR0vjC9w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-09-16 16:18 ` Gandalf Corvotempesta
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4AD98A71-E24B-4F99-B0D0-9F492FB932D1@ornl.gov \
--to=atchleyes-1heg1yxhbw8@public.gmane.org \
--cc=gandalf.corvotempesta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org \
--cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
--cc=sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.