Soft-RoCE performance

Soft-RoCE performance

[Christian Blume]

Hello!

I am seeing that Soft-RoCE has much lower throughput than TCP. Is that
expected? If not, are there typical config parameters I can fiddle
with?

When running iperf I am getting around 300MB/s whereas it's only
around 100MB/s using ib_write_bw from perftests.

This is between two machines running Ubuntu20.04 with the 5.11 kernel.

Cheers,
Christian

RE: Soft-RoCE performance

[Pearson, Robert B]

Christian,

There are two key differences between TCP and soft RoCE. Most importantly TCP can use a 64KiB MTU which is fragmented by TSO or GSO if your NIC doesn't support TSO while soft RoCE is limited by the protocol to a 4KiB payload. With overhead for headers you need a link MTU of about 4096+256. If your application is going between soft RoCE and hard RoCE you have to live with this limit and compute ICRC on each packet. Checking is optional since RoCE packets have a crc32 checksum from ethernet. If you are using soft RoCE to soft RoCE you can ignore both ICRC calculations and with a patch increase the MTU above 4KiB. I have measured write performance up to around 35 GB/s in local loopback on a single 12 core box (AMD 3900x) using 12 IO threads, 16KB MTU, and ICRC disabled for 1MB messages. This is on head of tree with some patches not yet upstream.

Bob Pearson
rpearsonhpe@gmail.com
rpearson@hpe.com


-----Original Message-----
From: Christian Blume <chr.blume@gmail.com> 
Sent: Wednesday, February 9, 2022 9:34 PM
To: RDMA mailing list <linux-rdma@vger.kernel.org>
Subject: Soft-RoCE performance

Hello!

I am seeing that Soft-RoCE has much lower throughput than TCP. Is that expected? If not, are there typical config parameters I can fiddle with?

When running iperf I am getting around 300MB/s whereas it's only around 100MB/s using ib_write_bw from perftests.

This is between two machines running Ubuntu20.04 with the 5.11 kernel.

Cheers,
Christian

RE: Soft-RoCE performance

[Pearson, Robert B]

Sorry got distracted and forgot to mention the other thing. Ib_write_bw is limited to a single threaded process. If you run more IO threads you can improve performance. Ib_write_bw will let you have more than one QP but it has to run on a single core so it doesn't scale all that well.

Bob

-----Original Message-----
From: Pearson, Robert B <robert.pearson2@hpe.com> 
Sent: Wednesday, February 9, 2022 11:13 PM
To: Christian Blume <chr.blume@gmail.com>; RDMA mailing list <linux-rdma@vger.kernel.org>
Subject: RE: Soft-RoCE performance

Christian,

There are two key differences between TCP and soft RoCE. Most importantly TCP can use a 64KiB MTU which is fragmented by TSO or GSO if your NIC doesn't support TSO while soft RoCE is limited by the protocol to a 4KiB payload. With overhead for headers you need a link MTU of about 4096+256. If your application is going between soft RoCE and hard RoCE you have to live with this limit and compute ICRC on each packet. Checking is optional since RoCE packets have a crc32 checksum from ethernet. If you are using soft RoCE to soft RoCE you can ignore both ICRC calculations and with a patch increase the MTU above 4KiB. I have measured write performance up to around 35 GB/s in local loopback on a single 12 core box (AMD 3900x) using 12 IO threads, 16KB MTU, and ICRC disabled for 1MB messages. This is on head of tree with some patches not yet upstream.

Bob Pearson
rpearsonhpe@gmail.com
rpearson@hpe.com


-----Original Message-----
From: Christian Blume <chr.blume@gmail.com> 
Sent: Wednesday, February 9, 2022 9:34 PM
To: RDMA mailing list <linux-rdma@vger.kernel.org>
Subject: Soft-RoCE performance

Hello!

I am seeing that Soft-RoCE has much lower throughput than TCP. Is that expected? If not, are there typical config parameters I can fiddle with?

When running iperf I am getting around 300MB/s whereas it's only around 100MB/s using ib_write_bw from perftests.

This is between two machines running Ubuntu20.04 with the 5.11 kernel.

Cheers,
Christian

Re: Soft-RoCE performance

[Yanjun Zhu]

在 2022/2/10 13:13, Pearson, Robert B 写道:
> Christian,
> 
> There are two key differences between TCP and soft RoCE. Most importantly TCP can use a 64KiB MTU which is fragmented by TSO or GSO if your NIC doesn't support TSO while soft RoCE is limited by the protocol to a 4KiB payload. With overhead for headers you need a link MTU of about 4096+256. If your application is going between soft RoCE and hard RoCE you have to live with this limit and compute ICRC on each packet. Checking is optional since RoCE packets have a crc32 checksum from ethernet. If you are using soft RoCE to soft RoCE you can ignore both ICRC calculations and with a patch increase the MTU above 4KiB. I have measured write performance up to around 35 GB/s 

Thanks, I have also reached the big bandwidth with the same methods.
How about latency of soft roce?

Zhu Yanjun


in local loopback on a single 12 core box (AMD 3900x) using 12 IO 
threads, 16KB MTU, and ICRC disabled for 1MB messages. This is on head 
of tree with some patches not yet upstream.
> 
> Bob Pearson
> rpearsonhpe@gmail.com
> rpearson@hpe.com
> 
> 
> -----Original Message-----
> From: Christian Blume <chr.blume@gmail.com>
> Sent: Wednesday, February 9, 2022 9:34 PM
> To: RDMA mailing list <linux-rdma@vger.kernel.org>
> Subject: Soft-RoCE performance
> 
> Hello!
> 
> I am seeing that Soft-RoCE has much lower throughput than TCP. Is that expected? If not, are there typical config parameters I can fiddle with?
> 
> When running iperf I am getting around 300MB/s whereas it's only around 100MB/s using ib_write_bw from perftests.
> 
> This is between two machines running Ubuntu20.04 with the 5.11 kernel.
> 
> Cheers,
> Christian

Re: Soft-RoCE performance

[Christian Blume]

Hi Bob,

Thanks for clarifying! I am looking forward to testing larger MTUs
with the new functionality!

Cheers,
Christian

On Fri, Feb 11, 2022 at 3:04 AM Yanjun Zhu <yanjun.zhu@linux.dev> wrote:
>
> 在 2022/2/10 13:13, Pearson, Robert B 写道:
> > Christian,
> >
> > There are two key differences between TCP and soft RoCE. Most importantly TCP can use a 64KiB MTU which is fragmented by TSO or GSO if your NIC doesn't support TSO while soft RoCE is limited by the protocol to a 4KiB payload. With overhead for headers you need a link MTU of about 4096+256. If your application is going between soft RoCE and hard RoCE you have to live with this limit and compute ICRC on each packet. Checking is optional since RoCE packets have a crc32 checksum from ethernet. If you are using soft RoCE to soft RoCE you can ignore both ICRC calculations and with a patch increase the MTU above 4KiB. I have measured write performance up to around 35 GB/s
>
> Thanks, I have also reached the big bandwidth with the same methods.
> How about latency of soft roce?
>
> Zhu Yanjun
>
>
> in local loopback on a single 12 core box (AMD 3900x) using 12 IO
> threads, 16KB MTU, and ICRC disabled for 1MB messages. This is on head
> of tree with some patches not yet upstream.
> >
> > Bob Pearson
> > rpearsonhpe@gmail.com
> > rpearson@hpe.com
> >
> >
> > -----Original Message-----
> > From: Christian Blume <chr.blume@gmail.com>
> > Sent: Wednesday, February 9, 2022 9:34 PM
> > To: RDMA mailing list <linux-rdma@vger.kernel.org>
> > Subject: Soft-RoCE performance
> >
> > Hello!
> >
> > I am seeing that Soft-RoCE has much lower throughput than TCP. Is that expected? If not, are there typical config parameters I can fiddle with?
> >
> > When running iperf I am getting around 300MB/s whereas it's only around 100MB/s using ib_write_bw from perftests.
> >
> > This is between two machines running Ubuntu20.04 with the 5.11 kernel.
> >
> > Cheers,
> > Christian
>

RE: Soft-RoCE performance

[Bernard Metzler]

> -----Original Message-----
> From: Pearson, Robert B <robert.pearson2@hpe.com>
> Sent: Thursday, 10 February 2022 06:13
> To: Christian Blume <chr.blume@gmail.com>; RDMA mailing list <linux-
> rdma@vger.kernel.org>
> Subject: [EXTERNAL] RE: Soft-RoCE performance
> 
> Christian,
> 
> There are two key differences between TCP and soft RoCE. Most importantly
> TCP can use a 64KiB MTU which is fragmented by TSO or GSO if your NIC
> doesn't support TSO while soft RoCE is limited by the protocol to a 4KiB
> payload. With overhead for headers you need a link MTU of about 4096+256.
> If your application is going between soft RoCE and hard RoCE you have to
> live with this limit and compute ICRC on each packet. Checking is optional
> since RoCE packets have a crc32 checksum from ethernet. If you are using
> soft RoCE to soft RoCE you can ignore both ICRC calculations and with a
> patch increase the MTU above 4KiB. I have measured write performance up to
> around 35 GB/s in local loopback on a single 12 core box (AMD 3900x) using
> 12 IO threads, 16KB MTU, and ICRC disabled for 1MB messages. This is on
> head of tree with some patches not yet upstream.
> 
> Bob Pearson
> rpearsonhpe@gmail.com
> rpearson@hpe.com
> 
> 
> -----Original Message-----
> From: Christian Blume <chr.blume@gmail.com>
> Sent: Wednesday, February 9, 2022 9:34 PM
> To: RDMA mailing list <linux-rdma@vger.kernel.org>
> Subject: Soft-RoCE performance
> 
> Hello!
> 
> I am seeing that Soft-RoCE has much lower throughput than TCP. Is that
> expected? If not, are there typical config parameters I can fiddle with?
> 
> When running iperf I am getting around 300MB/s whereas it's only around
> 100MB/s using ib_write_bw from perftests.
> 
> This is between two machines running Ubuntu20.04 with the 5.11 kernel.
> 
> Cheers,
> Christian

It reminds me of a discussion we had a while ago - see https://patchwork.kernel.org/project/linux-rdma/patch/20200414144822.2365-1-bmt@zurich.ibm.com/

Running on TCP and implementing iWarp, siw suffers the same problem. Maybe
it makes sense looking into a generic solution to the problem for software
based RDMA implementations, potentially using the given RDMA core 
infrastructure?

Back then, we proposed using a spare protocol bit to do GSO signaling.
Krishna extended that idea to an MTU size negotiation using multiple spare
bits.

Another idea was to use the rdma netlink protocol for doing those settings.
That may also cover toggling CRC calculation. iWarp allows for that
negotiation, but there is no API. Control could be provided per interface,
or per QP ID, or both (I'd prefer). With the rxe driver coming up with a
similar thing, I tend to prefer such a generic solution, even if it further
complicates common man's RDMA usage.

What do other think?

Thanks,
Bernard.

Re: Soft-RoCE performance

[Bob Pearson]

On 2/10/22 08:04, Yanjun Zhu wrote:

> 
> Thanks, I have also reached the big bandwidth with the same methods.
> How about latency of soft roce?
> 
> Zhu Yanjun

In loopback on my system with ib_write_lat I see the following. It isn't very exciting. The use case I am interested in doesn't require super low latency though.


 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 

 2       1000          1.57           9.65         1.84     	       2.04        	0.69   		5.02    		9.65   

 4       1000          1.58           7.89         1.82     	       1.99        	0.54   		4.68    		7.89   

 8       1000          1.60           6.86         1.79     	       1.95        	0.51   		4.08    		6.86   

 16      1000          1.59           5.28         1.83     	       1.95        	0.42   		3.72    		5.28   

 32      1000          1.64           8.18         1.84     	       1.99        	0.52   		4.07    		8.18   

 64      1000          1.65           8.58         1.84     	       1.99        	0.48   		3.92    		8.58   

 128     1000          1.80           10.58        1.96     	       2.10        	0.47   		3.94    		10.58  

 256     1000          1.76           12.54        2.00     	       2.17        	0.52   		4.00    		12.54  

 512     1000          1.69           11.09        1.95     	       2.21        	0.66   		3.97    		11.09  

 1024    1000          1.80           12.91        2.00     	       2.22        	0.56   		3.89    		12.91  

 2048    1000          2.04           9.29         2.20     	       2.37        	0.50   		4.03    		9.29   

 4096    1000          2.39           11.66        2.58     	       2.76        	0.56   		4.67    		11.66