From: Bart Van Assche <bvanassche@acm.org>
To: Jinpu Wang <jinpu.wang@cloud.ionos.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>, Jack Wang <jinpuwang@gmail.com>,
linux-block@vger.kernel.org, linux-rdma@vger.kernel.org,
Jens Axboe <axboe@kernel.dk>,
Christoph Hellwig <hch@infradead.org>,
Sagi Grimberg <sagi@grimberg.me>,
Leon Romanovsky <leon@kernel.org>,
Doug Ledford <dledford@redhat.com>,
Danil Kipnis <danil.kipnis@cloud.ionos.com>,
rpenyaev@suse.de
Subject: Re: [PATCH v5 00/25] RTRS (former IBTRS) rdma transport library and the corresponding RNBD (former IBNBD) rdma network block device
Date: Thu, 16 Jan 2020 08:41:04 -0800 [thread overview]
Message-ID: <c0c05209-1f43-c242-96d6-1944e43d207c@acm.org> (raw)
In-Reply-To: <CAMGffEm=1ai=vkxLV7Qp0z3M-98BDej4mpc4WnsOpSHUJWuPeA@mail.gmail.com>
On 1/7/20 2:56 AM, Jinpu Wang wrote:
> On Mon, Jan 6, 2020 at 6:07 PM Jinpu Wang <jinpu.wang@cloud.ionos.com> wrote:
>>
>> On Fri, Jan 3, 2020 at 5:29 PM Bart Van Assche <bvanassche@acm.org> wrote:
>>>
>>> On 1/3/20 4:39 AM, Jinpu Wang wrote:
>>>> Performance results for the v5.5-rc1 kernel are here:
>>>> link: https://github.com/ionos-enterprise/ibnbd/tree/develop/performance/v5-v5.5-rc1
>>>>
>>>> Some workloads RNBD are faster, some workloads NVMeoF are faster.
>>>
>>> Thank you for having shared these graphs.
>>>
>>> Do the graphs in RNBD-SinglePath.pdf show that NVMeOF achieves similar
>>> or higher IOPS, higher bandwidth and lower latency than RNBD for
>>> workloads with a block size of 4 KB and also for mixed workloads with
>>> less than 20 disks, whether or not invalidation is enabled for RNBD?
>> Hi Bart,
>>
>> Yes, that's the result on one pair of Server with Connect X4 HCA, I
>> did another test on another
>> 2 servers with Connect X5 HCA, results are quite different, we will
>> double-check the
>> performance results also on old machines, will share new results later.
>>
> here are the results with ConnectX5 HCA MT4119 + Intel(R) Xeon(R) Gold
> 6130 CPU @ 2.10GHz, sorry no graph for now,
> will prepare the next round.
>
> disks 4k nvme dual 4k nvme single 4k rnbd dual 4k rnbd single
> 4k rnbd-inv dual 4k rnbd-inv single
> x1 251637.436256 254312.068793 270886.311369 260934.306569
> 218632.336766 190800.519948
> x2 460894.610539 463925.907409 496318.068193 466374.862514
> 418960.30397 372848.815118
> x3 603263.673633 605004.49955 675073.892611 614552.144786
> 586223.077692 524221.977802
> x4 731648.935106 733174.482552 850245.575442 743062.493751
> 744380.361964 656861.813819
> x5 827732.326767 827444.855514 1026939.306069 840515.548445
> 897801.719828 762707.329267
> x6 876705.329467 873963.0037 1142399.960004 876974.70253
> 1037773.522648 834073.892611
> x7 893808.719128 893268.073193 1239282.471753 892728.027197
> 1135570.742926 871336.966303
> x8 906589.741026 905938.006199 1287178.964207 906189.381062
> 1225040.9959 895292.070793
> x9 912048.09519 912400.259974 1386211.878812 913885.311469
> 1302472.964884 910176.282372
> x10 915566.243376 915602.739726 1442959.70403 916288.871113
> 1350296.325879 914966.40336
> x11 917116.188381 916905.809419 1418574.942506 916264.373563
> 1370438.698083 915255.874413
> x12 915852.814719 917710.128987 1423534.546545 916348.386452
> 1352357.364264 914966.656684
> x13 919042.69573 918819.536093 1429697.830217 917631.036896
> 1378083.824558 916519.161677
> x14 920000.49995 920031.59684 1443317.268273 917562.843716
> 1395023.56936 918935.706429
> x15 920160.883912 920367.363264 1445306.425863 918278.472153
> 1440776.944611 916352.265921
> x16 920652.869426 920673.832617 1454705.229477 917902.948526
> 1455708.2501 918198.001998
> x17 916892.310769 916883.623275 1491071.841948 918936.706329
> 1436507.428457 917182.934132
> x18 917247.775222 917762.523748 1612129.058036 918546.835949
> 1488716.583417 919521.095781
> x19 920084.791521 920349.930014 1615690.87821 915371.496958
> 1406747.32954 918347.248577
> x20 922339.232154 922208.058388 1591415.958404 917922.631526
> 1343806.744488 918903.393214
> x21 923141.771646 923297.040592 1581547.169811 919371.025795
> 1342568.406347 919157.752944
> x22 923063.787243 924072.385523 1574972.846162 919173.713143
> 1340318.639673 920577.995403
> x23 923549.490102 924643.471306 1573597.440256 918705.060385
> 1333047.771337 917469.027431
> x24 925584.483103 925224.955009 1578143.485651 921744.15117
> 1321494.708466 920001.498352
> x25 926165.366927 926842.031594 1579288.271173 921845.392764
> 1319902.568202 920448.830702
> x26 926852.729454 927382.123575 1585351.318945 922669.59912
> 1325670.791338 919137.796847
> x27 928196.260748 928093.981204 1581427.557244 921379.155436
> 1325009.972078 919017.550858
> x28 929330.433913 929606.778644 1581726.527347 924325.834833
> 1331721.074174 919557.761373
> x29 929885.522895 929876.924615 1578317.436513 922977.240966
> 1333612.86783 921094.386736
> x30 930635.972805 930599.520144 1583537.946205 922746.107784
> 1333446.651362 922821.171531
Hi Jack,
What does "dual" mean? What explains the big difference between the NVMe
and RNBD results for the "dual" columns?
Thanks,
Bart.
next prev parent reply other threads:[~2020-01-16 16:41 UTC|newest]
Thread overview: 45+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-12-20 15:50 [PATCH v5 00/25] RTRS (former IBTRS) rdma transport library and the corresponding RNBD (former IBNBD) rdma network block device Jack Wang
2019-12-20 15:50 ` [PATCH v5 01/25] sysfs: export sysfs_remove_file_self() Jack Wang
2019-12-20 16:26 ` Jinpu Wang
2019-12-20 15:50 ` [PATCH v5 02/25] rtrs: public interface header to establish RDMA connections Jack Wang
2019-12-21 10:15 ` Leon Romanovsky
2019-12-21 14:27 ` Danil Kipnis
2019-12-22 7:36 ` Leon Romanovsky
2019-12-23 7:38 ` Jinpu Wang
2019-12-23 8:04 ` Leon Romanovsky
2019-12-23 10:31 ` Jinpu Wang
2019-12-20 15:50 ` [PATCH v5 03/25] rtrs: private headers with rtrs protocol structs and helpers Jack Wang
2019-12-20 15:50 ` [PATCH v5 04/25] rtrs: core: lib functions shared between client and server modules Jack Wang
2019-12-20 15:50 ` [PATCH v5 05/25] rtrs: client: private header with client structs and functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 06/25] rtrs: client: main functionality Jack Wang
2019-12-20 15:50 ` [PATCH v5 07/25] rtrs: client: statistics functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 08/25] rtrs: client: sysfs interface functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 09/25] rtrs: server: private header with server structs and functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 10/25] rtrs: server: main functionality Jack Wang
2019-12-20 15:50 ` [PATCH v5 11/25] rtrs: server: statistics functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 12/25] rtrs: server: sysfs interface functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 13/25] rtrs: include client and server modules into kernel compilation Jack Wang
2019-12-20 15:50 ` [PATCH v5 14/25] rtrs: a bit of documentation Jack Wang
2019-12-20 15:50 ` [PATCH v5 15/25] rnbd: private headers with rnbd protocol structs and helpers Jack Wang
2019-12-20 15:51 ` [PATCH v5 16/25] rnbd: client: private header with client structs and functions Jack Wang
2019-12-20 15:51 ` [PATCH v5 17/25] rnbd: client: main functionality Jack Wang
2019-12-20 15:51 ` [PATCH v5 18/25] rnbd: client: sysfs interface functions Jack Wang
2019-12-20 15:51 ` [PATCH v5 19/25] rnbd: server: private header with server structs and functions Jack Wang
2019-12-20 15:51 ` [PATCH v5 20/25] rnbd: server: main functionality Jack Wang
2019-12-20 15:51 ` [PATCH v5 21/25] rnbd: server: functionality for IO submission to file or block dev Jack Wang
2019-12-20 15:51 ` [PATCH v5 22/25] rnbd: server: sysfs interface functions Jack Wang
2019-12-23 8:14 ` Leon Romanovsky
2019-12-23 8:33 ` Jinpu Wang
2019-12-20 15:51 ` [PATCH v5 23/25] rnbd: include client and server modules into kernel compilation Jack Wang
2019-12-20 15:51 ` [PATCH v5 24/25] rnbd: a bit of documentation Jack Wang
2019-12-20 15:51 ` [PATCH v5 25/25] MAINTAINERS: Add maintainers for RNBD/RTRS modules Jack Wang
2019-12-22 9:55 ` Gal Pressman
2019-12-23 7:20 ` Jinpu Wang
2019-12-21 10:17 ` [PATCH v5 00/25] RTRS (former IBTRS) rdma transport library and the corresponding RNBD (former IBNBD) rdma network block device Leon Romanovsky
2020-01-02 18:18 ` Jason Gunthorpe
2020-01-03 12:39 ` Jinpu Wang
2020-01-03 16:28 ` Bart Van Assche
2020-01-06 17:07 ` Jinpu Wang
2020-01-07 10:56 ` Jinpu Wang
2020-01-16 16:41 ` Bart Van Assche [this message]
2020-01-16 16:46 ` Jinpu Wang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c0c05209-1f43-c242-96d6-1944e43d207c@acm.org \
--to=bvanassche@acm.org \
--cc=axboe@kernel.dk \
--cc=danil.kipnis@cloud.ionos.com \
--cc=dledford@redhat.com \
--cc=hch@infradead.org \
--cc=jgg@ziepe.ca \
--cc=jinpu.wang@cloud.ionos.com \
--cc=jinpuwang@gmail.com \
--cc=leon@kernel.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=rpenyaev@suse.de \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).