* Slow performance with librspreload.so
@ 2013-08-28 15:20 Gandalf Corvotempesta
[not found] ` <CAJH6TXgf2LeMH+1L290w_KZ5tTN7NWpQxntF58Z506G3h_qKVw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 27+ messages in thread
From: Gandalf Corvotempesta @ 2013-08-28 15:20 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
Hi
i'm trying the preloader librspreload.so on two directly connected hosts:
host1:$ sudo ibstatus
Infiniband device 'mlx4_0' port 1 status:
default gid: fe80:0000:0000:0000:0002:c903:004d:dd45
base lid: 0x1
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 20 Gb/sec (4X DDR)
link_layer: InfiniBand
Infiniband device 'mlx4_0' port 2 status:
default gid: fe80:0000:0000:0000:0002:c903:004d:dd46
base lid: 0x0
sm lid: 0x0
state: 1: DOWN
phys state: 2: Polling
rate: 10 Gb/sec (4X)
link_layer: InfiniBand
host2:$ sudo ibstatus
Infiniband device 'mthca0' port 1 status:
default gid: fe80:0000:0000:0000:0008:f104:0398:14cd
base lid: 0x2
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 20 Gb/sec (4X DDR)
link_layer: InfiniBand
Infiniband device 'mthca0' port 2 status:
default gid: fe80:0000:0000:0000:0008:f104:0398:14ce
base lid: 0x0
sm lid: 0x0
state: 1: DOWN
phys state: 2: Polling
rate: 10 Gb/sec (4X)
link_layer: InfiniBand
i've connected just one port between two hosts.
Ports is detected properly as 20Gb/s (4x DDR) but i'm unable to reach
speed over 5Gbit/s:
host1:$ sudo LD_PRELOAD=/usr/lib/x86_64-linux-gnu/rsocket/librspreload.so
NPtcp -h 172.17.0.2
Send and receive buffers are 131072 and 131072 bytes
(A bug in Linux doubles the requested buffer sizes)
Now starting the main loop
0: 1 bytes 17008 times --> 1.24 Mbps in 6.13 usec
1: 2 bytes 16306 times --> 2.02 Mbps in 7.56 usec
2: 3 bytes 13223 times --> 3.10 Mbps in 7.38 usec
3: 4 bytes 9037 times --> 4.21 Mbps in 7.25 usec
4: 6 bytes 10345 times --> 6.49 Mbps in 7.05 usec
5: 8 bytes 7093 times --> 7.77 Mbps in 7.85 usec
6: 12 bytes 7957 times --> 17.08 Mbps in 5.36 usec
7: 13 bytes 7772 times --> 14.75 Mbps in 6.73 usec
8: 16 bytes 6861 times --> 16.11 Mbps in 7.58 usec
9: 19 bytes 7424 times --> 18.91 Mbps in 7.67 usec
10: 21 bytes 8237 times --> 17.69 Mbps in 9.06 usec
11: 24 bytes 7361 times --> 19.72 Mbps in 9.28 usec
12: 27 bytes 7628 times --> 24.14 Mbps in 8.53 usec
13: 29 bytes 5207 times --> 29.81 Mbps in 7.42 usec
14: 32 bytes 6504 times --> 29.42 Mbps in 8.30 usec
15: 35 bytes 6401 times --> 39.08 Mbps in 6.83 usec
16: 45 bytes 8362 times --> 45.19 Mbps in 7.60 usec
17: 48 bytes 8774 times --> 46.10 Mbps in 7.94 usec
18: 51 bytes 8654 times --> 55.19 Mbps in 7.05 usec
19: 61 bytes 5562 times --> 57.42 Mbps in 8.10 usec
20: 64 bytes 6068 times --> 72.31 Mbps in 6.75 usec
21: 67 bytes 7636 times --> 42.93 Mbps in 11.91 usec
22: 93 bytes 4512 times --> 55.84 Mbps in 12.71 usec
23: 96 bytes 5246 times --> 60.13 Mbps in 12.18 usec
24: 99 bytes 5558 times --> 59.49 Mbps in 12.70 usec
25: 125 bytes 2864 times --> 75.25 Mbps in 12.67 usec
26: 128 bytes 3913 times --> 75.78 Mbps in 12.89 usec
27: 131 bytes 3940 times --> 74.77 Mbps in 13.37 usec
28: 189 bytes 3883 times --> 113.42 Mbps in 12.71 usec
29: 192 bytes 5243 times --> 109.85 Mbps in 13.33 usec
30: 195 bytes 5038 times --> 115.66 Mbps in 12.86 usec
31: 253 bytes 2710 times --> 146.61 Mbps in 13.17 usec
32: 256 bytes 3782 times --> 142.77 Mbps in 13.68 usec
33: 259 bytes 3683 times --> 144.75 Mbps in 13.65 usec
34: 381 bytes 3733 times --> 201.64 Mbps in 14.42 usec
35: 384 bytes 4624 times --> 204.22 Mbps in 14.35 usec
36: 387 bytes 4665 times --> 204.65 Mbps in 14.43 usec
37: 509 bytes 2364 times --> 265.12 Mbps in 14.65 usec
38: 512 bytes 3406 times --> 267.89 Mbps in 14.58 usec
39: 515 bytes 3442 times --> 266.90 Mbps in 14.72 usec
40: 765 bytes 3429 times --> 381.51 Mbps in 15.30 usec
41: 768 bytes 4357 times --> 384.85 Mbps in 15.23 usec
42: 771 bytes 4387 times --> 386.35 Mbps in 15.23 usec
43: 1021 bytes 2214 times --> 495.38 Mbps in 15.72 usec
44: 1024 bytes 3176 times --> 499.56 Mbps in 15.64 usec
45: 1027 bytes 3203 times --> 497.19 Mbps in 15.76 usec
46: 1533 bytes 3188 times --> 692.19 Mbps in 16.90 usec
47: 1536 bytes 3945 times --> 688.52 Mbps in 17.02 usec
48: 1539 bytes 3920 times --> 693.85 Mbps in 16.92 usec
49: 2045 bytes 1981 times --> 858.05 Mbps in 18.18 usec
50: 2048 bytes 2748 times --> 862.22 Mbps in 18.12 usec
51: 2051 bytes 2761 times --> 832.50 Mbps in 18.80 usec
52: 3069 bytes 2666 times --> 1174.72 Mbps in 19.93 usec
53: 3072 bytes 3344 times --> 1183.58 Mbps in 19.80 usec
54: 3075 bytes 3368 times --> 1177.98 Mbps in 19.92 usec
55: 4093 bytes 1678 times --> 1495.79 Mbps in 20.88 usec
56: 4096 bytes 2394 times --> 1486.91 Mbps in 21.02 usec
57: 4099 bytes 2380 times --> 1490.11 Mbps in 20.99 usec
58: 6141 bytes 2385 times --> 2417.56 Mbps in 19.38 usec
59: 6144 bytes 3439 times --> 2491.24 Mbps in 18.82 usec
60: 6147 bytes 3543 times --> 2393.71 Mbps in 19.59 usec
61: 8189 bytes 1703 times --> 2486.93 Mbps in 25.12 usec
62: 8192 bytes 1990 times --> 2501.61 Mbps in 24.98 usec
63: 8195 bytes 2001 times --> 2470.25 Mbps in 25.31 usec
64: 12285 bytes 1976 times --> 3335.91 Mbps in 28.10 usec
65: 12288 bytes 2372 times --> 3346.71 Mbps in 28.01 usec
66: 12291 bytes 2380 times --> 3325.57 Mbps in 28.20 usec
67: 16381 bytes 1183 times --> 3404.87 Mbps in 36.71 usec
68: 16384 bytes 1362 times --> 3396.27 Mbps in 36.81 usec
69: 16387 bytes 1358 times --> 3338.60 Mbps in 37.45 usec
70: 24573 bytes 1335 times --> 3952.93 Mbps in 47.43 usec
71: 24576 bytes 1405 times --> 3870.35 Mbps in 48.45 usec
72: 24579 bytes 1376 times --> 3947.46 Mbps in 47.50 usec
73: 32765 bytes 701 times --> 3708.77 Mbps in 67.40 usec
74: 32768 bytes 741 times --> 3670.93 Mbps in 68.10 usec
75: 32771 bytes 734 times --> 3713.07 Mbps in 67.34 usec
76: 49149 bytes 742 times --> 4269.21 Mbps in 87.83 usec
77: 49152 bytes 759 times --> 4213.58 Mbps in 89.00 usec
78: 49155 bytes 749 times --> 4261.68 Mbps in 88.00 usec
79: 65533 bytes 378 times --> 4397.40 Mbps in 113.70 usec
80: 65536 bytes 439 times --> 4495.83 Mbps in 111.21 usec
81: 65539 bytes 449 times --> 4373.61 Mbps in 114.33 usec
82: 98301 bytes 437 times --> 4581.69 Mbps in 163.69 usec
83: 98304 bytes 407 times --> 4643.01 Mbps in 161.53 usec
84: 98307 bytes 412 times --> 4574.63 Mbps in 163.95 usec
85: 131069 bytes 203 times --> 4663.35 Mbps in 214.43 usec
86: 131072 bytes 233 times --> 4643.97 Mbps in 215.33 usec
87: 131075 bytes 232 times --> 4663.00 Mbps in 214.46 usec
88: 196605 bytes 233 times --> 4820.71 Mbps in 311.15 usec
89: 196608 bytes 214 times --> 4838.05 Mbps in 310.04 usec
90: 196611 bytes 215 times --> 4833.56 Mbps in 310.34 usec
91: 262141 bytes 107 times --> 4946.10 Mbps in 404.35 usec
92: 262144 bytes 123 times --> 4955.13 Mbps in 403.62 usec
93: 262147 bytes 123 times --> 4940.46 Mbps in 404.83 usec
94: 393213 bytes 123 times --> 5061.76 Mbps in 592.67 usec
95: 393216 bytes 112 times --> 5053.05 Mbps in 593.70 usec
96: 393219 bytes 112 times --> 5022.92 Mbps in 597.27 usec
97: 524285 bytes 55 times --> 5125.96 Mbps in 780.34 usec
98: 524288 bytes 64 times --> 5117.60 Mbps in 781.62 usec
99: 524291 bytes 63 times --> 5122.30 Mbps in 780.90 usec
100: 786429 bytes 64 times --> 5189.59 Mbps in 1156.16 usec
101: 786432 bytes 57 times --> 5186.53 Mbps in 1156.84 usec
102: 786435 bytes 57 times --> 5183.52 Mbps in 1157.52 usec
103: 1048573 bytes 28 times --> 5217.00 Mbps in 1533.44 usec
104: 1048576 bytes 32 times --> 5198.91 Mbps in 1538.78 usec
105: 1048579 bytes 32 times --> 5218.60 Mbps in 1532.98 usec
106: 1572861 bytes 32 times --> 5242.06 Mbps in 2289.17 usec
107: 1572864 bytes 29 times --> 5242.86 Mbps in 2288.83 usec
108: 1572867 bytes 29 times --> 5249.47 Mbps in 2285.95 usec
109: 2097149 bytes 14 times --> 5252.47 Mbps in 3046.18 usec
110: 2097152 bytes 16 times --> 5260.67 Mbps in 3041.44 usec
111: 2097155 bytes 16 times --> 5255.55 Mbps in 3044.40 usec
112: 3145725 bytes 16 times --> 5255.34 Mbps in 4566.78 usec
113: 3145728 bytes 14 times --> 5259.21 Mbps in 4563.43 usec
114: 3145731 bytes 14 times --> 5263.82 Mbps in 4559.43 usec
115: 4194301 bytes 7 times --> 5256.99 Mbps in 6087.13 usec
116: 4194304 bytes 8 times --> 5265.97 Mbps in 6076.75 usec
117: 4194307 bytes 8 times --> 5257.70 Mbps in 6086.32 usec
118: 6291453 bytes 8 times --> 5242.18 Mbps in 9156.50 usec
119: 6291456 bytes 7 times --> 5238.10 Mbps in 9163.64 usec
120: 6291459 bytes 7 times --> 5223.28 Mbps in 9189.64 usec
121: 8388605 bytes 3 times --> 5192.27 Mbps in 12326.00 usec
122: 8388608 bytes 4 times --> 5206.80 Mbps in 12291.61 usec
123: 8388611 bytes 4 times --> 5197.97 Mbps in 12312.50 usec
host1:$ sudo LD_PRELOAD=/usr/lib/x86_64-linux-gnu/rsocket/librspreload.so
iperf -c 172.17.0.2
------------------------------------------------------------
Client connecting to 172.17.0.2, TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 3] local 172.17.0.1 port 36085 connected with 172.17.0.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 7.82 GBytes 6.72 Gbits/sec
I'm also trying to set connected as IB mode but without success:
host1:$ sudo echo connected > /sys/class/net/ib0/mode
host1:$ sudo cat /sys/class/net/ib0/mode
datagram
Any advice ?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 27+ messages in thread[parent not found: <CAJH6TXgf2LeMH+1L290w_KZ5tTN7NWpQxntF58Z506G3h_qKVw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* RE: Slow performance with librspreload.so [not found] ` <CAJH6TXgf2LeMH+1L290w_KZ5tTN7NWpQxntF58Z506G3h_qKVw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-08-28 15:50 ` Hefty, Sean [not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA937E-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org> 0 siblings, 1 reply; 27+ messages in thread From: Hefty, Sean @ 2013-08-28 15:50 UTC (permalink / raw) To: 'Gandalf Corvotempesta', 'linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org' > i've connected just one port between two hosts. > Ports is detected properly as 20Gb/s (4x DDR) but i'm unable to reach > speed over 5Gbit/s: It's possible that this is falling back to using normal TCP sockets. Can you run the rstream test program to verify that you can get faster than 5 Gbps? rstream without any options will use rsockets directly. If you use the -T s option, it will use standard TCP sockets. You can use LD_PRELOAD with -T s to verify that the preload brings your per performance to the same level as using rsockets directly. - Sean -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
[parent not found: <1828884A29C6694DAF28B7E6B8A8237388CA937E-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>]
* Re: Slow performance with librspreload.so [not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA937E-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org> @ 2013-08-28 16:19 ` Gandalf Corvotempesta [not found] ` <CAJH6TXjEx+41G_7wvQybMXzb60tu-ha2d2Bu_J_erNDPJRbQFw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 27+ messages in thread From: Gandalf Corvotempesta @ 2013-08-28 16:19 UTC (permalink / raw) To: Hefty, Sean; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA 2013/8/28 Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>: > Can you run the rstream test program to verify that you can get faster than 5 Gbps? > > rstream without any options will use rsockets directly. If you use the -T s option, it will use standard TCP sockets. You can use LD_PRELOAD with -T s to verify that the preload brings your per performance to the same level as using rsockets directly. 5Gb/s with rstream: $ sudo ./rstream -s 172.17.0.2 name bytes xfers iters total time Gb/sec usec/xfer 64_lat 64 1 100k 12m 0.70s 0.15 3.52 4k_lat 4k 1 10k 78m 0.29s 2.23 14.69 64k_lat 64k 1 1k 125m 0.21s 4.94 106.07 1m_lat 1m 1 100 200m 0.30s 5.61 1495.89 64_bw 64 100k 1 12m 0.25s 0.42 1.23 4k_bw 4k 10k 1 78m 0.13s 5.17 6.34 64k_bw 64k 1k 1 125m 0.19s 5.58 94.03 1m_bw 1m 100 1 200m 0.30s 5.64 1486.53 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
[parent not found: <CAJH6TXjEx+41G_7wvQybMXzb60tu-ha2d2Bu_J_erNDPJRbQFw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* RE: Slow performance with librspreload.so [not found] ` <CAJH6TXjEx+41G_7wvQybMXzb60tu-ha2d2Bu_J_erNDPJRbQFw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-08-28 17:16 ` Hefty, Sean [not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA96AD-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org> 0 siblings, 1 reply; 27+ messages in thread From: Hefty, Sean @ 2013-08-28 17:16 UTC (permalink / raw) To: 'Gandalf Corvotempesta' Cc: 'linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org' > > Can you run the rstream test program to verify that you can get faster > than 5 Gbps? > > > > rstream without any options will use rsockets directly. If you use the - > T s option, it will use standard TCP sockets. You can use LD_PRELOAD with > -T s to verify that the preload brings your per performance to the same > level as using rsockets directly. > > > 5Gb/s with rstream: Can you explain your environment more? The performance seems low. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
[parent not found: <1828884A29C6694DAF28B7E6B8A8237388CA96AD-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>]
* Re: Slow performance with librspreload.so [not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA96AD-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org> @ 2013-08-28 18:24 ` Gandalf Corvotempesta [not found] ` <CAJH6TXhAuSDytS5O1cJMg3iatq+STkwhPUG2zmexJ5tmt3Foqg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 27+ messages in thread From: Gandalf Corvotempesta @ 2013-08-28 18:24 UTC (permalink / raw) To: Hefty, Sean; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA 2013/8/28 Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>: > Can you explain your environment more? The performance seems low. Ubuntu 13.04 Server on both nodes. node1: $ cat /proc/cpuinfo | grep 'model name' model name : Intel(R) Xeon(R) CPU E5-2603 0 @ 1.80GHz model name : Intel(R) Xeon(R) CPU E5-2603 0 @ 1.80GHz model name : Intel(R) Xeon(R) CPU E5-2603 0 @ 1.80GHz model name : Intel(R) Xeon(R) CPU E5-2603 0 @ 1.80GHz $ free -m total used free shared buffers cached Mem: 16022 966 15056 0 95 534 -/+ buffers/cache: 336 15686 Swap: 16353 0 16353 node2: $ cat /proc/cpuinfo | grep 'model name' model name : Intel(R) Xeon(R) CPU 3065 @ 2.33GHz model name : Intel(R) Xeon(R) CPU 3065 @ 2.33GHz $ free -m total used free shared buffers cached Mem: 2001 718 1282 0 53 516 -/+ buffers/cache: 148 1853 Swap: 2044 0 2044 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
[parent not found: <CAJH6TXhAuSDytS5O1cJMg3iatq+STkwhPUG2zmexJ5tmt3Foqg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* RE: Slow performance with librspreload.so [not found] ` <CAJH6TXhAuSDytS5O1cJMg3iatq+STkwhPUG2zmexJ5tmt3Foqg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-08-28 19:20 ` Hefty, Sean 2013-08-28 20:05 ` Hefty, Sean 1 sibling, 0 replies; 27+ messages in thread From: Hefty, Sean @ 2013-08-28 19:20 UTC (permalink / raw) To: 'Gandalf Corvotempesta' Cc: 'linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org' > 2013/8/28 Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>: > > Can you explain your environment more? The performance seems low. > > Ubuntu 13.04 Server on both nodes. > > node1: > > $ cat /proc/cpuinfo | grep 'model name' > model name : Intel(R) Xeon(R) CPU E5-2603 0 @ 1.80GHz > $ cat /proc/cpuinfo | grep 'model name' > model name : Intel(R) Xeon(R) CPU 3065 @ 2.33GHz Can you run rstream using the loopback address? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
* RE: Slow performance with librspreload.so [not found] ` <CAJH6TXhAuSDytS5O1cJMg3iatq+STkwhPUG2zmexJ5tmt3Foqg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2013-08-28 19:20 ` Hefty, Sean @ 2013-08-28 20:05 ` Hefty, Sean [not found] ` <CAJH6TXgA0ghKX1P8UUAMFKY9o0xBJ0j4-kFa_M4a4ecdzoD0HA@mail.gmail.com> 1 sibling, 1 reply; 27+ messages in thread From: Hefty, Sean @ 2013-08-28 20:05 UTC (permalink / raw) To: 'Gandalf Corvotempesta' Cc: 'linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org' > Ubuntu 13.04 Server on both nodes. > > node1: > > $ cat /proc/cpuinfo | grep 'model name' > model name : Intel(R) Xeon(R) CPU E5-2603 0 @ 1.80GHz If you can provide your PCIe information and the results from running the perftest tools (rdma_bw), that could help as well. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
[parent not found: <CAJH6TXgA0ghKX1P8UUAMFKY9o0xBJ0j4-kFa_M4a4ecdzoD0HA@mail.gmail.com>]
[parent not found: <CAJH6TXgA0ghKX1P8UUAMFKY9o0xBJ0j4-kFa_M4a4ecdzoD0HA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Fwd: Slow performance with librspreload.so [not found] ` <CAJH6TXgA0ghKX1P8UUAMFKY9o0xBJ0j4-kFa_M4a4ecdzoD0HA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-08-29 13:03 ` Gandalf Corvotempesta 2013-08-29 13:03 ` Gandalf Corvotempesta 1 sibling, 0 replies; 27+ messages in thread From: Gandalf Corvotempesta @ 2013-08-29 13:03 UTC (permalink / raw) To: linux-rdma-u79uwXL29TY76Z2rM5mHXA ---------- Forwarded message ---------- From: Gandalf Corvotempesta <gandalf.corvotempesta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> Date: 2013/8/29 Subject: Re: Slow performance with librspreload.so To: "Hefty, Sean" <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> 2013/8/28 Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>: > If you can provide your PCIe information and the results from running the perftest tools (rdma_bw), that could help as well. node1 (172.17.0.1 is ip configured on ib0): $ sudo ./rstream -s 172.17.0.1 name bytes xfers iters total time Gb/sec usec/xfer 64_lat 64 1 100k 12m 0.26s 0.40 1.28 4k_lat 4k 1 10k 78m 0.17s 3.96 8.28 64k_lat 64k 1 1k 125m 0.11s 9.86 53.19 1m_lat 1m 1 100 200m 0.14s 12.34 679.73 64_bw 64 100k 1 12m 0.06s 1.75 0.29 4k_bw 4k 10k 1 78m 0.06s 11.79 2.78 64k_bw 64k 1k 1 125m 0.09s 12.20 42.97 1m_bw 1m 100 1 200m 0.13s 12.78 656.55 $ lspci | grep -i infiniband 04:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX VPI PCIe 2.0 2.5GT/s - IB DDR / 10GigE] (rev a0) node2 (172.17.0.2 is ip configured on ib0): $ sudo ./rstream -s 172.17.0.2 name bytes xfers iters total time Gb/sec usec/xfer 64_lat 64 1 100k 12m 1.10s 0.09 5.49 4k_lat 4k 1 10k 78m 0.43s 1.53 21.49 64k_lat 64k 1 1k 125m 0.29s 3.64 143.99 1m_lat 1m 1 100 200m 0.37s 4.53 1852.70 64_bw 64 100k 1 12m 0.42s 0.24 2.12 4k_bw 4k 10k 1 78m 0.16s 4.16 7.87 64k_bw 64k 1k 1 125m 0.23s 4.49 116.69 1m_bw 1m 100 1 200m 0.36s 4.63 1813.52 $ lspci | grep -i infiniband 02:00.0 InfiniBand: Mellanox Technologies MT25208 InfiniHost III Ex (Tavor compatibility mode) (rev 20) (this is a Voltaire 400Ex-D card) Same result by using 127.0.0.1 on both hosts, obviously. I'm unable to run rdma_bw due to different CPU speed any my versions doesn't have the ignore flag. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Slow performance with librspreload.so [not found] ` <CAJH6TXgA0ghKX1P8UUAMFKY9o0xBJ0j4-kFa_M4a4ecdzoD0HA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2013-08-29 13:03 ` Fwd: " Gandalf Corvotempesta @ 2013-08-29 13:03 ` Gandalf Corvotempesta 1 sibling, 0 replies; 27+ messages in thread From: Gandalf Corvotempesta @ 2013-08-29 13:03 UTC (permalink / raw) To: Hefty, Sean, linux-rdma-u79uwXL29TY76Z2rM5mHXA 2013/8/29 Gandalf Corvotempesta <gandalf.corvotempesta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>: > node1 (172.17.0.1 is ip configured on ib0): > > $ sudo ./rstream -s 172.17.0.1 > name bytes xfers iters total time Gb/sec usec/xfer > 64_lat 64 1 100k 12m 0.26s 0.40 1.28 > 4k_lat 4k 1 10k 78m 0.17s 3.96 8.28 > 64k_lat 64k 1 1k 125m 0.11s 9.86 53.19 > 1m_lat 1m 1 100 200m 0.14s 12.34 679.73 > 64_bw 64 100k 1 12m 0.06s 1.75 0.29 > 4k_bw 4k 10k 1 78m 0.06s 11.79 2.78 > 64k_bw 64k 1k 1 125m 0.09s 12.20 42.97 > 1m_bw 1m 100 1 200m 0.13s 12.78 656.55 With standard sockets: $ sudo ./rstream -s 172.17.0.1 -T s name bytes xfers iters total time Gb/sec usec/xfer 64_lat 64 1 100k 12m 1.07s 0.10 5.36 4k_lat 4k 1 10k 78m 0.13s 4.89 6.70 64k_lat 64k 1 1k 125m 0.06s 18.38 28.52 1m_lat 1m 1 100 200m 0.06s 25.90 323.89 64_bw 64 100k 1 12m 0.98s 0.10 4.91 4k_bw 4k 10k 1 78m 0.12s 5.29 6.20 64k_bw 64k 1k 1 125m 0.04s 27.04 19.39 1m_bw 1m 100 1 200m 0.05s 31.52 266.14 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
[parent not found: <1828884A29C6694DAF28B7E6B8A8237388CA9C6B@ORSMSX109.amr.corp.intel.com>]
[parent not found: <CAJH6TXiYLKt3b1UFsZt7uFwDbWcDFnHNnS8CTO24Gt-2zn+Qiw@mail.gmail.com>]
[parent not found: <1828884A29C6694DAF28B7E6B8A8237388CA9D8A@ORSMSX109.amr.corp.intel.com>]
[parent not found: <1828884A29C6694DAF28B7E6B8A8237388CA9D8A-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>]
* Re: Slow performance with librspreload.so [not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA9D8A-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org> @ 2013-08-29 18:57 ` Gandalf Corvotempesta 2013-08-30 8:19 ` Gandalf Corvotempesta 1 sibling, 0 replies; 27+ messages in thread From: Gandalf Corvotempesta @ 2013-08-29 18:57 UTC (permalink / raw) To: Hefty, Sean, linux-rdma-u79uwXL29TY76Z2rM5mHXA 2013/8/29 Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>: > 12 Gbps on a 20 Gb link actually seems reasonable to me. I only see around 25 Gbps on a 40 Gb link, with raw perftest performance coming in at about 26 Gbps. Ok. I think that i've connected the HBA to the wrong PCI-Express slot. I have a DELL R200 that has 3 PCI-Express slot but one of them is just x4. probably i've connected the card to this. Tomorrow i'll try to connect the HBA to the x8 slot. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Slow performance with librspreload.so [not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA9D8A-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org> 2013-08-29 18:57 ` Gandalf Corvotempesta @ 2013-08-30 8:19 ` Gandalf Corvotempesta [not found] ` <CAJH6TXhWWMBbopDLZY2+rrNOm2m5gcmObj7Sr16u2qrNW_NHgw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 1 sibling, 1 reply; 27+ messages in thread From: Gandalf Corvotempesta @ 2013-08-30 8:19 UTC (permalink / raw) To: Hefty, Sean, linux-rdma-u79uwXL29TY76Z2rM5mHXA 2013/8/29 Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>: > 12 Gbps on a 20 Gb link actually seems reasonable to me. I only see around 25 Gbps on a 40 Gb link, with raw perftest performance coming in at about 26 Gbps. Is this a rstream limits or an IB limit? I've read somewhere that DDR should transfer at 16Gbps By the way, moving the HBA on the second slot, brought me to 12Gbps on both hosts. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
[parent not found: <CAJH6TXhWWMBbopDLZY2+rrNOm2m5gcmObj7Sr16u2qrNW_NHgw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Slow performance with librspreload.so [not found] ` <CAJH6TXhWWMBbopDLZY2+rrNOm2m5gcmObj7Sr16u2qrNW_NHgw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-08-30 8:23 ` Gandalf Corvotempesta [not found] ` <CAJH6TXgR=wVGyyHrpTKkBw-5M=A9-tGzjpYcV6NWpk8uKvFn8Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 27+ messages in thread From: Gandalf Corvotempesta @ 2013-08-30 8:23 UTC (permalink / raw) To: Hefty, Sean, linux-rdma-u79uwXL29TY76Z2rM5mHXA 2013/8/30 Gandalf Corvotempesta <gandalf.corvotempesta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>: > By the way, moving the HBA on the second slot, brought me to 12Gbps on > both hosts. This is great: $ sudo LD_PRELOAD=/usr/local/lib/rsocket/librspreload.so iperf -c 172.17.0.2 ------------------------------------------------------------ Client connecting to 172.17.0.2, TCP port 5001 TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 3] local 172.17.0.1 port 34108 connected with 172.17.0.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 12.2 GBytes 10.5 Gbits/sec $ sudo LD_PRELOAD=/usr/local/lib/rsocket/librspreload.so iperf -c 172.17.0.2 -P 2 ------------------------------------------------------------ Client connecting to 172.17.0.2, TCP port 5001 TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 4] local 172.17.0.1 port 55323 connected with 172.17.0.2 port 5001 [ 3] local 172.17.0.1 port 36579 connected with 172.17.0.2 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 7.46 GBytes 6.41 Gbits/sec [ 3] 0.0-10.0 sec 7.46 GBytes 6.41 Gbits/sec [SUM] 0.0-10.0 sec 14.9 GBytes 12.8 Gbits/sec with 2 parallel connection i'm able to reach "rate" speed with iperf, the same speed archived with rstream. Is iperf affected by IPoIB MTU size when used with librspreload.so ? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
[parent not found: <CAJH6TXgR=wVGyyHrpTKkBw-5M=A9-tGzjpYcV6NWpk8uKvFn8Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Slow performance with librspreload.so [not found] ` <CAJH6TXgR=wVGyyHrpTKkBw-5M=A9-tGzjpYcV6NWpk8uKvFn8Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-08-30 10:29 ` Gandalf Corvotempesta [not found] ` <CAJH6TXidOVJDTokOwxjCT9RRTOO6E_xbyG_K++YVkFx8NbhMTA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2013-08-30 15:51 ` Hefty, Sean 1 sibling, 1 reply; 27+ messages in thread From: Gandalf Corvotempesta @ 2013-08-30 10:29 UTC (permalink / raw) To: Hefty, Sean, linux-rdma-u79uwXL29TY76Z2rM5mHXA 2013/8/30 Gandalf Corvotempesta <gandalf.corvotempesta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>: > Is iperf affected by IPoIB MTU size when used with librspreload.so ? Another strange issue: $ sudo LD_PRELOAD=/usr/local/lib/rsocket/librspreload.so iperf -c 172.17.0.2 ------------------------------------------------------------ Client connecting to 172.17.0.2, TCP port 5001 TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 3] local 172.17.0.1 port 57926 connected with 172.17.0.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 12.2 GBytes 10.4 Gbits/sec $ iperf -c 172.17.0.2 ------------------------------------------------------------ Client connecting to 172.17.0.2, TCP port 5001 TCP window size: 648 KByte (default) ------------------------------------------------------------ [ 3] local 172.17.0.1 port 58113 connected with 172.17.0.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 14.5 GBytes 12.5 Gbits/sec rsocket slower than IPoIB ? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
[parent not found: <CAJH6TXidOVJDTokOwxjCT9RRTOO6E_xbyG_K++YVkFx8NbhMTA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* RE: Slow performance with librspreload.so [not found] ` <CAJH6TXidOVJDTokOwxjCT9RRTOO6E_xbyG_K++YVkFx8NbhMTA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-08-30 17:38 ` Hefty, Sean [not found] ` <1828884A29C6694DAF28B7E6B8A8237388CAA1B9-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org> 0 siblings, 1 reply; 27+ messages in thread From: Hefty, Sean @ 2013-08-30 17:38 UTC (permalink / raw) To: Gandalf Corvotempesta, linux-rdma-u79uwXL29TY76Z2rM5mHXA > Another strange issue: > > $ sudo LD_PRELOAD=/usr/local/lib/rsocket/librspreload.so iperf -c > 172.17.0.2 > ------------------------------------------------------------ > Client connecting to 172.17.0.2, TCP port 5001 > TCP window size: 128 KByte (default) Increasing the window size may improve the results. E.g. on my systems I go from 17.7 Gbps at 128 KB to 24.3 Gbps for 512 KB. > ------------------------------------------------------------ > [ 3] local 172.17.0.1 port 57926 connected with 172.17.0.2 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 12.2 GBytes 10.4 Gbits/sec > > $ iperf -c 172.17.0.2 > ------------------------------------------------------------ > Client connecting to 172.17.0.2, TCP port 5001 > TCP window size: 648 KByte (default) > ------------------------------------------------------------ > [ 3] local 172.17.0.1 port 58113 connected with 172.17.0.2 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 14.5 GBytes 12.5 Gbits/sec > > rsocket slower than IPoIB ? This is surprising to me - just getting 12.5 Gbps out of ipoib is surprising. Does iperf use sendfile()? My results with iperf (version 2.0.5) over ipoib (default configurations) vary considerably based on the TCP window size. (Note that this is a 40 Gbps link.) Results summarized: TCP window size: 27.9 KByte (default) [ 3] 0.0-10.0 sec 12.8 GBytes 11.0 Gbits/sec TCP window size: 416 KByte (WARNING: requested 500 KByte) [ 3] 0.0-10.0 sec 8.19 GBytes 7.03 Gbits/sec TCP window size: 250 KByte (WARNING: requested 125 KByte) [ 3] 0.0-10.0 sec 4.99 GBytes 4.29 Gbits/sec I'm guessing that there are some settings I can change to increase the ipoib performance on my systems. Using rspreload, I get: LD_PRELOAD=/usr/local/lib/rsocket/librspreload.so iperf -c 192.168.0.103 TCP window size: 512 KByte (default) [ 3] 0.0-10.0 sec 28.3 GBytes 24.3 Gbits/sec It seems that ipoib bandwidth should be close to rsockets, similar to what you see. I also don't understand the effect that the TCP window size is having on the results. The smallest window gives the best bandwidth for ipoib?! - Sean -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
[parent not found: <1828884A29C6694DAF28B7E6B8A8237388CAA1B9-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>]
* Re: Slow performance with librspreload.so [not found] ` <1828884A29C6694DAF28B7E6B8A8237388CAA1B9-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org> @ 2013-08-30 18:08 ` Atchley, Scott 0 siblings, 0 replies; 27+ messages in thread From: Atchley, Scott @ 2013-08-30 18:08 UTC (permalink / raw) To: Hefty, Sean; +Cc: Gandalf Corvotempesta, linux-rdma-u79uwXL29TY76Z2rM5mHXA On Aug 30, 2013, at 1:38 PM, "Hefty, Sean" <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote: >> Another strange issue: >> >> $ sudo LD_PRELOAD=/usr/local/lib/rsocket/librspreload.so iperf -c >> 172.17.0.2 >> ------------------------------------------------------------ >> Client connecting to 172.17.0.2, TCP port 5001 >> TCP window size: 128 KByte (default) > > Increasing the window size may improve the results. E.g. on my systems I go from 17.7 Gbps at 128 KB to 24.3 Gbps for 512 KB. > >> ------------------------------------------------------------ >> [ 3] local 172.17.0.1 port 57926 connected with 172.17.0.2 port 5001 >> [ ID] Interval Transfer Bandwidth >> [ 3] 0.0-10.0 sec 12.2 GBytes 10.4 Gbits/sec >> >> $ iperf -c 172.17.0.2 >> ------------------------------------------------------------ >> Client connecting to 172.17.0.2, TCP port 5001 >> TCP window size: 648 KByte (default) >> ------------------------------------------------------------ >> [ 3] local 172.17.0.1 port 58113 connected with 172.17.0.2 port 5001 >> [ ID] Interval Transfer Bandwidth >> [ 3] 0.0-10.0 sec 14.5 GBytes 12.5 Gbits/sec >> >> rsocket slower than IPoIB ? > > This is surprising to me - just getting 12.5 Gbps out of ipoib is surprising. Does iperf use sendfile()? I have a pair of nodes connected by QDR via a switch. Using normal IPoIB, a single Netperf can reach 18.4 Gb/s if I bind to the same core that the IRQ handler is bound to. With four concurrent Netperfs, I can reach 23 Gb/s. This is in datagram mode. Connected mode is slower. I have not tried rsockets on these nodes. Scott > > My results with iperf (version 2.0.5) over ipoib (default configurations) vary considerably based on the TCP window size. (Note that this is a 40 Gbps link.) Results summarized: > > TCP window size: 27.9 KByte (default) > [ 3] 0.0-10.0 sec 12.8 GBytes 11.0 Gbits/sec > > TCP window size: 416 KByte (WARNING: requested 500 KByte) > [ 3] 0.0-10.0 sec 8.19 GBytes 7.03 Gbits/sec > > TCP window size: 250 KByte (WARNING: requested 125 KByte) > [ 3] 0.0-10.0 sec 4.99 GBytes 4.29 Gbits/sec > > I'm guessing that there are some settings I can change to increase the ipoib performance on my systems. Using rspreload, I get: > > LD_PRELOAD=/usr/local/lib/rsocket/librspreload.so iperf -c 192.168.0.103 > TCP window size: 512 KByte (default) > [ 3] 0.0-10.0 sec 28.3 GBytes 24.3 Gbits/sec > > It seems that ipoib bandwidth should be close to rsockets, similar to what you see. I also don't understand the effect that the TCP window size is having on the results. The smallest window gives the best bandwidth for ipoib?! > > - Sean > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
* RE: Slow performance with librspreload.so [not found] ` <CAJH6TXgR=wVGyyHrpTKkBw-5M=A9-tGzjpYcV6NWpk8uKvFn8Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2013-08-30 10:29 ` Gandalf Corvotempesta @ 2013-08-30 15:51 ` Hefty, Sean [not found] ` <CAJH6TXgu4L8gnqQX1fKZ=ioZDxUMnj=s3h0qkYh1_35VWpMJ1g@mail.gmai! l.com> ` (2 more replies) 1 sibling, 3 replies; 27+ messages in thread From: Hefty, Sean @ 2013-08-30 15:51 UTC (permalink / raw) To: Gandalf Corvotempesta, linux-rdma-u79uwXL29TY76Z2rM5mHXA > with 2 parallel connection i'm able to reach "rate" speed with iperf, > the same speed archived with rstream. > Is iperf affected by IPoIB MTU size when used with librspreload.so ? Not directly. The ipoib mtu is usually set based on the mtu of the IB link. The latter does affect rsocket performance. However if the ipoib mtu is changed separately from the IB link mtu, it will not affect rsockets. - Sean -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
[parent not found: <CAJH6TXgu4L8gnqQX1fKZ=ioZDxUMnj=s3h0qkYh1_35VWpMJ1g@mail.gmai! l.com>]
[parent not found: <1828884A29C6694DAF28B7E6B8A8237388CAA11B-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>]
* Re: Slow performance with librspreload.so [not found] ` <1828884A29C6694DAF28B7E6B8A8237388CAA11B-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org> @ 2013-08-30 16:26 ` Gandalf Corvotempesta 0 siblings, 0 replies; 27+ messages in thread From: Gandalf Corvotempesta @ 2013-08-30 16:26 UTC (permalink / raw) To: Hefty, Sean; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA 2013/8/30 Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>: > Not directly. The ipoib mtu is usually set based on the mtu of the IB link. The latter does affect rsocket performance. However if the ipoib mtu is changed separately from the IB link mtu, it will not affect rsockets. Actually i'm going faster with IPoIB than rsockets. How can I change the MTU for IB link ? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
[parent not found: <CAJH6TXgvViraH4SEcYydCerGyM6kK61eoiaENCy6PSf_1ocSVA@mail.gmai! l.com>]
[parent not found: <CAJH6TXgvViraH4SEcYydCerGyM6kK61eoiaENCy6PSf_1ocSVA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* RE: Slow performance with librspreload.so [not found] ` <CAJH6TXgvViraH4SEcYydCerGyM6kK61eoiaENCy6PSf_1ocSVA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-08-30 18:19 ` Rupert Dance 2013-08-31 9:20 ` Gandalf Corvotempesta 0 siblings, 1 reply; 27+ messages in thread From: Rupert Dance @ 2013-08-30 18:19 UTC (permalink / raw) To: 'Gandalf Corvotempesta', 'Hefty, Sean' Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA One way to set or check mtu is with the ibportstate utility: Usage: ibportstate [options] <dest dr_path|lid|guid> <portnum> [<op>] Supported ops: enable, disable, reset, speed, width, query, down, arm, active, vls, mtu, lid, smlid, lmc -----Original Message----- From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Gandalf Corvotempesta Sent: Friday, August 30, 2013 12:27 PM To: Hefty, Sean Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Subject: Re: Slow performance with librspreload.so 2013/8/30 Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>: > Not directly. The ipoib mtu is usually set based on the mtu of the IB link. The latter does affect rsocket performance. However if the ipoib mtu is changed separately from the IB link mtu, it will not affect rsockets. Actually i'm going faster with IPoIB than rsockets. How can I change the MTU for IB link ? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Slow performance with librspreload.so 2013-08-30 18:19 ` Rupert Dance @ 2013-08-31 9:20 ` Gandalf Corvotempesta [not found] ` <CAJH6TXgu4L8gnqQX1fKZ=ioZDxUMnj=s3h0qkYh1_35VWpMJ1g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 27+ messages in thread From: Gandalf Corvotempesta @ 2013-08-31 9:20 UTC (permalink / raw) To: Rupert Dance; +Cc: Hefty, Sean, linux-rdma-u79uwXL29TY76Z2rM5mHXA 2013/8/30 Rupert Dance <rsdance-rzwDkyJvnYokmLvzuZlaBw@public.gmane.org>: > One way to set or check mtu is with the ibportstate utility: > > Usage: ibportstate [options] <dest dr_path|lid|guid> <portnum> [<op>] > Supported ops: enable, disable, reset, speed, width, query, down, arm, > active, vls, mtu, lid, smlid, lmc I've tried but max MTU is 2048 on one device: $ sudo ibv_devinfo hca_id: mthca0 transport: InfiniBand (0) fw_ver: 4.7.600 node_guid: 0008:f104:0398:14cc sys_image_guid: 0008:f104:0398:14cf vendor_id: 0x08f1 vendor_part_id: 25208 hw_ver: 0xA0 board_id: VLT0040010001 phys_port_cnt: 2 port: 1 state: PORT_ACTIVE (4) max_mtu: 2048 (4) active_mtu: 2048 (4) sm_lid: 1 port_lid: 2 port_lmc: 0x00 link_layer: InfiniBand any workaround? Maybe a firmware update ? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
[parent not found: <CAJH6TXgu4L8gnqQX1fKZ=ioZDxUMnj=s3h0qkYh1_35VWpMJ1g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* RE: Slow performance with librspreload.so [not found] ` <CAJH6TXgu4L8gnqQX1fKZ=ioZDxUMnj=s3h0qkYh1_35VWpMJ1g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-08-31 11:34 ` Rupert Dance 2013-08-31 19:51 ` Gandalf Corvotempesta 0 siblings, 1 reply; 27+ messages in thread From: Rupert Dance @ 2013-08-31 11:34 UTC (permalink / raw) To: 'Gandalf Corvotempesta' Cc: 'Hefty, Sean', linux-rdma-u79uwXL29TY76Z2rM5mHXA The Vendor ID indicates that this is a Voltaire card which probably means it is an older card. Some of the early Mellanox based cards did not support anything bigger than 2048. 00-08-F1 (hex) Voltaire 0008F1 (base 16) Voltaire 9 Hamenofim st. Herzelia 46725 ISRAEL Checking for FW updates cannot hurt but you may well be restricted to 2048 -----Original Message----- From: Gandalf Corvotempesta [mailto:gandalf.corvotempesta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org] Sent: Saturday, August 31, 2013 5:21 AM To: Rupert Dance Cc: Hefty, Sean; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Subject: Re: Slow performance with librspreload.so 2013/8/30 Rupert Dance <rsdance-rzwDkyJvnYokmLvzuZlaBw@public.gmane.org>: > One way to set or check mtu is with the ibportstate utility: > > Usage: ibportstate [options] <dest dr_path|lid|guid> <portnum> [<op>] > Supported ops: enable, disable, reset, speed, width, query, down, arm, > active, vls, mtu, lid, smlid, lmc I've tried but max MTU is 2048 on one device: $ sudo ibv_devinfo hca_id: mthca0 transport: InfiniBand (0) fw_ver: 4.7.600 node_guid: 0008:f104:0398:14cc sys_image_guid: 0008:f104:0398:14cf vendor_id: 0x08f1 vendor_part_id: 25208 hw_ver: 0xA0 board_id: VLT0040010001 phys_port_cnt: 2 port: 1 state: PORT_ACTIVE (4) max_mtu: 2048 (4) active_mtu: 2048 (4) sm_lid: 1 port_lid: 2 port_lmc: 0x00 link_layer: InfiniBand any workaround? Maybe a firmware update ? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Slow performance with librspreload.so 2013-08-31 11:34 ` Rupert Dance @ 2013-08-31 19:51 ` Gandalf Corvotempesta [not found] ` <CAJH6TXiOToa2-EOj6Hz-rkVHt3tCSh4jnLbKZs69GhT7dFeH0A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 27+ messages in thread From: Gandalf Corvotempesta @ 2013-08-31 19:51 UTC (permalink / raw) To: Rupert Dance; +Cc: Hefty, Sean, linux-rdma-u79uwXL29TY76Z2rM5mHXA 2013/8/31 Rupert Dance <rsdance-rzwDkyJvnYokmLvzuZlaBw@public.gmane.org>: > The Vendor ID indicates that this is a Voltaire card which probably means it > is an older card. Some of the early Mellanox based cards did not support > anything bigger than 2048. Yes, it's an older card used just for this test. By the way, increasing MTU to 4096 will give me more performance? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
[parent not found: <CAJH6TXiOToa2-EOj6Hz-rkVHt3tCSh4jnLbKZs69GhT7dFeH0A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* RE: Slow performance with librspreload.so [not found] ` <CAJH6TXiOToa2-EOj6Hz-rkVHt3tCSh4jnLbKZs69GhT7dFeH0A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-09-01 13:28 ` Rupert Dance 2013-09-01 17:41 ` Gandalf Corvotempesta 2013-09-03 12:06 ` Hal Rosenstock 1 sibling, 1 reply; 27+ messages in thread From: Rupert Dance @ 2013-09-01 13:28 UTC (permalink / raw) To: 'Gandalf Corvotempesta' Cc: 'Hefty, Sean', linux-rdma-u79uwXL29TY76Z2rM5mHXA My guess is that it will not make a huge difference and that the solution lies elsewhere. -----Original Message----- From: Gandalf Corvotempesta [mailto:gandalf.corvotempesta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org] Sent: Saturday, August 31, 2013 3:51 PM To: Rupert Dance Cc: Hefty, Sean; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Subject: Re: Slow performance with librspreload.so 2013/8/31 Rupert Dance <rsdance-rzwDkyJvnYokmLvzuZlaBw@public.gmane.org>: > The Vendor ID indicates that this is a Voltaire card which probably > means it is an older card. Some of the early Mellanox based cards did > not support anything bigger than 2048. Yes, it's an older card used just for this test. By the way, increasing MTU to 4096 will give me more performance? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Slow performance with librspreload.so 2013-09-01 13:28 ` Rupert Dance @ 2013-09-01 17:41 ` Gandalf Corvotempesta [not found] ` <CAJH6TXj53RZmzqA94CfPJjXBetK7us5v=cFyoMmjVUNQ-oeqpQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 27+ messages in thread From: Gandalf Corvotempesta @ 2013-09-01 17:41 UTC (permalink / raw) To: Rupert Dance; +Cc: Hefty, Sean, linux-rdma-u79uwXL29TY76Z2rM5mHXA 2013/9/1 Rupert Dance <rsdance-rzwDkyJvnYokmLvzuZlaBw@public.gmane.org>: > My guess is that it will not make a huge difference and that the solution > lies elsewhere. What is strange to me is that rsocket is slower than IPoIB and limited to 10Gbit more or less. With IPoIB i'm able to reach 12.5 Gbit -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
[parent not found: <CAJH6TXj53RZmzqA94CfPJjXBetK7us5v=cFyoMmjVUNQ-oeqpQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Slow performance with librspreload.so [not found] ` <CAJH6TXj53RZmzqA94CfPJjXBetK7us5v=cFyoMmjVUNQ-oeqpQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-09-03 9:17 ` Gandalf Corvotempesta 0 siblings, 0 replies; 27+ messages in thread From: Gandalf Corvotempesta @ 2013-09-03 9:17 UTC (permalink / raw) To: Rupert Dance; +Cc: Hefty, Sean, linux-rdma-u79uwXL29TY76Z2rM5mHXA 2013/9/1 Gandalf Corvotempesta <gandalf.corvotempesta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>: > What is strange to me is that rsocket is slower than IPoIB and limited > to 10Gbit more or less. With IPoIB i'm able to reach 12.5 Gbit qperf is giving the same strange speed: FROM NODE1 to NODE2: $ sudo qperf -ub 77.95.175.106 ud_lat ud_bw ud_lat: latency = 12.5 us ud_bw: send_bw = 12.5 Gb/sec recv_bw = 12.5 Gb/sec FROM NODE1 TO NODE2, slower and with more latency than remote host! $ sudo qperf -ub 172.17.0.1 ud_lat ud_bw ud_lat: latency = 13.8 us ud_bw: send_bw = 11.9 Gb/sec recv_bw = 11.9 Gb/sec how can I check if this is due to an hardware bottleneck ? CPU and RAM are good. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Slow performance with librspreload.so [not found] ` <CAJH6TXiOToa2-EOj6Hz-rkVHt3tCSh4jnLbKZs69GhT7dFeH0A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2013-09-01 13:28 ` Rupert Dance @ 2013-09-03 12:06 ` Hal Rosenstock [not found] ` <5225D0DD.7060803-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> 1 sibling, 1 reply; 27+ messages in thread From: Hal Rosenstock @ 2013-09-03 12:06 UTC (permalink / raw) To: Gandalf Corvotempesta Cc: Rupert Dance, Hefty, Sean, linux-rdma-u79uwXL29TY76Z2rM5mHXA On 8/31/2013 3:51 PM, Gandalf Corvotempesta wrote: > By the way, increasing MTU to 4096 will give me more performance? With mthca, due to quirk, optimal performance is achieved at 1K MTU. OpenSM can reduce the MTU in returned PathRecords to 1K when one end of the path is mthca and actual path MTU is > 1K. This is controlled by enable_quirks config parameter which defaults to FALSE (don't do this). -- Hal -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
[parent not found: <5225D0DD.7060803-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>]
* Re: Slow performance with librspreload.so [not found] ` <5225D0DD.7060803-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> @ 2013-09-03 12:21 ` Gandalf Corvotempesta [not found] ` <CAJH6TXhSoxFshtuE0YqdVShQKSgX_wNwxekjy1+ZcGHR0vjC9w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 27+ messages in thread From: Gandalf Corvotempesta @ 2013-09-03 12:21 UTC (permalink / raw) To: Hal Rosenstock Cc: Rupert Dance, Hefty, Sean, linux-rdma-u79uwXL29TY76Z2rM5mHXA 2013/9/3 Hal Rosenstock <hal-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>: > With mthca, due to quirk, optimal performance is achieved at 1K MTU. > OpenSM can reduce the MTU in returned PathRecords to 1K when one end of > the path is mthca and actual path MTU is > 1K. This is controlled by > enable_quirks config parameter which defaults to FALSE (don't do this). I'll try. Actually these are my results, from node1 to node2 $ sudo qperf -ub 172.17.0.2 rc_bi_bw rc_lat rc_bw rc_rdma_read_lat rc_rdma_read_bw rc_rdma_write_lat rc_rdma_write_bw tcp_lat tcp_bw rc_bi_bw: bw = 20.5 Gb/sec rc_lat: latency = 15.4 us rc_bw: bw = 13.7 Gb/sec rc_rdma_read_lat: latency = 12.9 us rc_rdma_read_bw: bw = 11.5 Gb/sec rc_rdma_write_lat: latency = 15.2 us rc_rdma_write_bw: bw = 13.7 Gb/sec tcp_lat: latency = 48.8 us tcp_bw: bw = 12.5 Gb/sec I don't know if they are good for a DDR fabric. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
[parent not found: <CAJH6TXhSoxFshtuE0YqdVShQKSgX_wNwxekjy1+ZcGHR0vjC9w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Slow performance with librspreload.so [not found] ` <CAJH6TXhSoxFshtuE0YqdVShQKSgX_wNwxekjy1+ZcGHR0vjC9w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-09-16 16:18 ` Gandalf Corvotempesta 0 siblings, 0 replies; 27+ messages in thread From: Gandalf Corvotempesta @ 2013-09-16 16:18 UTC (permalink / raw) To: Hal Rosenstock Cc: Rupert Dance, Hefty, Sean, linux-rdma-u79uwXL29TY76Z2rM5mHXA 2013/9/3 Gandalf Corvotempesta <gandalf.corvotempesta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>: > $ sudo qperf -ub 172.17.0.2 rc_bi_bw rc_lat rc_bw rc_rdma_read_lat > rc_rdma_read_bw rc_rdma_write_lat rc_rdma_write_bw tcp_lat tcp_bw > rc_bi_bw: > bw = 20.5 Gb/sec > rc_lat: > latency = 15.4 us > rc_bw: > bw = 13.7 Gb/sec > rc_rdma_read_lat: > latency = 12.9 us > rc_rdma_read_bw: > bw = 11.5 Gb/sec > rc_rdma_write_lat: > latency = 15.2 us > rc_rdma_write_bw: > bw = 13.7 Gb/sec > tcp_lat: > latency = 48.8 us > tcp_bw: > bw = 12.5 Gb/sec > > I don't know if they are good for a DDR fabric. Just to clarify, why I'm getting the same bandwidth with librspreload.so and with plain use of IPoIB ? Should I check something ? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 27+ messages in thread
end of thread, other threads:[~2013-09-16 16:18 UTC | newest]
Thread overview: 27+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-08-28 15:20 Slow performance with librspreload.so Gandalf Corvotempesta
[not found] ` <CAJH6TXgf2LeMH+1L290w_KZ5tTN7NWpQxntF58Z506G3h_qKVw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-28 15:50 ` Hefty, Sean
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA937E-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2013-08-28 16:19 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXjEx+41G_7wvQybMXzb60tu-ha2d2Bu_J_erNDPJRbQFw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-28 17:16 ` Hefty, Sean
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA96AD-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2013-08-28 18:24 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXhAuSDytS5O1cJMg3iatq+STkwhPUG2zmexJ5tmt3Foqg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-28 19:20 ` Hefty, Sean
2013-08-28 20:05 ` Hefty, Sean
[not found] ` <CAJH6TXgA0ghKX1P8UUAMFKY9o0xBJ0j4-kFa_M4a4ecdzoD0HA@mail.gmail.com>
[not found] ` <CAJH6TXgA0ghKX1P8UUAMFKY9o0xBJ0j4-kFa_M4a4ecdzoD0HA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-29 13:03 ` Fwd: " Gandalf Corvotempesta
2013-08-29 13:03 ` Gandalf Corvotempesta
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA9C6B@ORSMSX109.amr.corp.intel.com>
[not found] ` <CAJH6TXiYLKt3b1UFsZt7uFwDbWcDFnHNnS8CTO24Gt-2zn+Qiw@mail.gmail.com>
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA9D8A@ORSMSX109.amr.corp.intel.com>
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA9D8A-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2013-08-29 18:57 ` Gandalf Corvotempesta
2013-08-30 8:19 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXhWWMBbopDLZY2+rrNOm2m5gcmObj7Sr16u2qrNW_NHgw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-30 8:23 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXgR=wVGyyHrpTKkBw-5M=A9-tGzjpYcV6NWpk8uKvFn8Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-30 10:29 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXidOVJDTokOwxjCT9RRTOO6E_xbyG_K++YVkFx8NbhMTA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-30 17:38 ` Hefty, Sean
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CAA1B9-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2013-08-30 18:08 ` Atchley, Scott
2013-08-30 15:51 ` Hefty, Sean
[not found] ` <CAJH6TXgu4L8gnqQX1fKZ=ioZDxUMnj=s3h0qkYh1_35VWpMJ1g@mail.gmai! l.com>
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CAA11B-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2013-08-30 16:26 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXgvViraH4SEcYydCerGyM6kK61eoiaENCy6PSf_1ocSVA@mail.gmai! l.com>
[not found] ` <CAJH6TXgvViraH4SEcYydCerGyM6kK61eoiaENCy6PSf_1ocSVA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-30 18:19 ` Rupert Dance
2013-08-31 9:20 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXgu4L8gnqQX1fKZ=ioZDxUMnj=s3h0qkYh1_35VWpMJ1g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-31 11:34 ` Rupert Dance
2013-08-31 19:51 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXiOToa2-EOj6Hz-rkVHt3tCSh4jnLbKZs69GhT7dFeH0A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-09-01 13:28 ` Rupert Dance
2013-09-01 17:41 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXj53RZmzqA94CfPJjXBetK7us5v=cFyoMmjVUNQ-oeqpQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-09-03 9:17 ` Gandalf Corvotempesta
2013-09-03 12:06 ` Hal Rosenstock
[not found] ` <5225D0DD.7060803-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
2013-09-03 12:21 ` Gandalf Corvotempesta
[not found] ` <CAJH6TXhSoxFshtuE0YqdVShQKSgX_wNwxekjy1+ZcGHR0vjC9w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-09-16 16:18 ` Gandalf Corvotempesta
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.