From: "Pearson, Robert B" <robert.pearson2@hpe.com>
To: "Pearson, Robert B" <robert.pearson2@hpe.com>,
Christian Blume <chr.blume@gmail.com>,
RDMA mailing list <linux-rdma@vger.kernel.org>
Subject: RE: Soft-RoCE performance
Date: Thu, 10 Feb 2022 05:28:24 +0000 [thread overview]
Message-ID: <PH7PR84MB148879A6AEDC6BFAB2FD44F1BC2F9@PH7PR84MB1488.NAMPRD84.PROD.OUTLOOK.COM> (raw)
In-Reply-To: <PH7PR84MB148872A4BB08EAFECCFF6B01BC2F9@PH7PR84MB1488.NAMPRD84.PROD.OUTLOOK.COM>
Sorry got distracted and forgot to mention the other thing. Ib_write_bw is limited to a single threaded process. If you run more IO threads you can improve performance. Ib_write_bw will let you have more than one QP but it has to run on a single core so it doesn't scale all that well.
Bob
-----Original Message-----
From: Pearson, Robert B <robert.pearson2@hpe.com>
Sent: Wednesday, February 9, 2022 11:13 PM
To: Christian Blume <chr.blume@gmail.com>; RDMA mailing list <linux-rdma@vger.kernel.org>
Subject: RE: Soft-RoCE performance
Christian,
There are two key differences between TCP and soft RoCE. Most importantly TCP can use a 64KiB MTU which is fragmented by TSO or GSO if your NIC doesn't support TSO while soft RoCE is limited by the protocol to a 4KiB payload. With overhead for headers you need a link MTU of about 4096+256. If your application is going between soft RoCE and hard RoCE you have to live with this limit and compute ICRC on each packet. Checking is optional since RoCE packets have a crc32 checksum from ethernet. If you are using soft RoCE to soft RoCE you can ignore both ICRC calculations and with a patch increase the MTU above 4KiB. I have measured write performance up to around 35 GB/s in local loopback on a single 12 core box (AMD 3900x) using 12 IO threads, 16KB MTU, and ICRC disabled for 1MB messages. This is on head of tree with some patches not yet upstream.
Bob Pearson
rpearsonhpe@gmail.com
rpearson@hpe.com
-----Original Message-----
From: Christian Blume <chr.blume@gmail.com>
Sent: Wednesday, February 9, 2022 9:34 PM
To: RDMA mailing list <linux-rdma@vger.kernel.org>
Subject: Soft-RoCE performance
Hello!
I am seeing that Soft-RoCE has much lower throughput than TCP. Is that expected? If not, are there typical config parameters I can fiddle with?
When running iperf I am getting around 300MB/s whereas it's only around 100MB/s using ib_write_bw from perftests.
This is between two machines running Ubuntu20.04 with the 5.11 kernel.
Cheers,
Christian
next prev parent reply other threads:[~2022-02-10 5:28 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-02-10 3:33 Soft-RoCE performance Christian Blume
2022-02-10 5:13 ` Pearson, Robert B
2022-02-10 5:28 ` Pearson, Robert B [this message]
2022-02-10 14:04 ` Yanjun Zhu
2022-02-10 22:23 ` Christian Blume
2022-02-11 17:48 ` Bob Pearson
2022-02-11 10:53 ` Bernard Metzler
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=PH7PR84MB148879A6AEDC6BFAB2FD44F1BC2F9@PH7PR84MB1488.NAMPRD84.PROD.OUTLOOK.COM \
--to=robert.pearson2@hpe.com \
--cc=chr.blume@gmail.com \
--cc=linux-rdma@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.