From: Jinpu Wang <jinpu.wang@cloud.ionos.com>
To: Bart Van Assche <bvanassche@acm.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>, Jack Wang <jinpuwang@gmail.com>,
linux-block@vger.kernel.org, linux-rdma@vger.kernel.org,
Jens Axboe <axboe@kernel.dk>,
Christoph Hellwig <hch@infradead.org>,
Sagi Grimberg <sagi@grimberg.me>,
Leon Romanovsky <leon@kernel.org>,
Doug Ledford <dledford@redhat.com>,
Danil Kipnis <danil.kipnis@cloud.ionos.com>,
Roman Penyaev <rpenyaev@suse.de>
Subject: Re: [PATCH v5 00/25] RTRS (former IBTRS) rdma transport library and the corresponding RNBD (former IBNBD) rdma network block device
Date: Thu, 16 Jan 2020 17:46:19 +0100 [thread overview]
Message-ID: <CAMGffEkXfwMU3AdNGhX9Z202UWJ-YhiCwaaQ=5NwfRmvyhOajg@mail.gmail.com> (raw)
In-Reply-To: <c0c05209-1f43-c242-96d6-1944e43d207c@acm.org>
On Thu, Jan 16, 2020 at 5:41 PM Bart Van Assche <bvanassche@acm.org> wrote:
>
> On 1/7/20 2:56 AM, Jinpu Wang wrote:
> > On Mon, Jan 6, 2020 at 6:07 PM Jinpu Wang <jinpu.wang@cloud.ionos.com> wrote:
> >>
> >> On Fri, Jan 3, 2020 at 5:29 PM Bart Van Assche <bvanassche@acm.org> wrote:
> >>>
> >>> On 1/3/20 4:39 AM, Jinpu Wang wrote:
> >>>> Performance results for the v5.5-rc1 kernel are here:
> >>>> link: https://github.com/ionos-enterprise/ibnbd/tree/develop/performance/v5-v5.5-rc1
> >>>>
> >>>> Some workloads RNBD are faster, some workloads NVMeoF are faster.
> >>>
> >>> Thank you for having shared these graphs.
> >>>
> >>> Do the graphs in RNBD-SinglePath.pdf show that NVMeOF achieves similar
> >>> or higher IOPS, higher bandwidth and lower latency than RNBD for
> >>> workloads with a block size of 4 KB and also for mixed workloads with
> >>> less than 20 disks, whether or not invalidation is enabled for RNBD?
> >> Hi Bart,
> >>
> >> Yes, that's the result on one pair of Server with Connect X4 HCA, I
> >> did another test on another
> >> 2 servers with Connect X5 HCA, results are quite different, we will
> >> double-check the
> >> performance results also on old machines, will share new results later.
> >>
> > here are the results with ConnectX5 HCA MT4119 + Intel(R) Xeon(R) Gold
> > 6130 CPU @ 2.10GHz, sorry no graph for now,
> > will prepare the next round.
> >
> > disks 4k nvme dual 4k nvme single 4k rnbd dual 4k rnbd single
> > 4k rnbd-inv dual 4k rnbd-inv single
> > x1 251637.436256 254312.068793 270886.311369 260934.306569
> > 218632.336766 190800.519948
> > x2 460894.610539 463925.907409 496318.068193 466374.862514
> > 418960.30397 372848.815118
> > x3 603263.673633 605004.49955 675073.892611 614552.144786
> > 586223.077692 524221.977802
> > x4 731648.935106 733174.482552 850245.575442 743062.493751
> > 744380.361964 656861.813819
> > x5 827732.326767 827444.855514 1026939.306069 840515.548445
> > 897801.719828 762707.329267
> > x6 876705.329467 873963.0037 1142399.960004 876974.70253
> > 1037773.522648 834073.892611
> > x7 893808.719128 893268.073193 1239282.471753 892728.027197
> > 1135570.742926 871336.966303
> > x8 906589.741026 905938.006199 1287178.964207 906189.381062
> > 1225040.9959 895292.070793
> > x9 912048.09519 912400.259974 1386211.878812 913885.311469
> > 1302472.964884 910176.282372
> > x10 915566.243376 915602.739726 1442959.70403 916288.871113
> > 1350296.325879 914966.40336
> > x11 917116.188381 916905.809419 1418574.942506 916264.373563
> > 1370438.698083 915255.874413
> > x12 915852.814719 917710.128987 1423534.546545 916348.386452
> > 1352357.364264 914966.656684
> > x13 919042.69573 918819.536093 1429697.830217 917631.036896
> > 1378083.824558 916519.161677
> > x14 920000.49995 920031.59684 1443317.268273 917562.843716
> > 1395023.56936 918935.706429
> > x15 920160.883912 920367.363264 1445306.425863 918278.472153
> > 1440776.944611 916352.265921
> > x16 920652.869426 920673.832617 1454705.229477 917902.948526
> > 1455708.2501 918198.001998
> > x17 916892.310769 916883.623275 1491071.841948 918936.706329
> > 1436507.428457 917182.934132
> > x18 917247.775222 917762.523748 1612129.058036 918546.835949
> > 1488716.583417 919521.095781
> > x19 920084.791521 920349.930014 1615690.87821 915371.496958
> > 1406747.32954 918347.248577
> > x20 922339.232154 922208.058388 1591415.958404 917922.631526
> > 1343806.744488 918903.393214
> > x21 923141.771646 923297.040592 1581547.169811 919371.025795
> > 1342568.406347 919157.752944
> > x22 923063.787243 924072.385523 1574972.846162 919173.713143
> > 1340318.639673 920577.995403
> > x23 923549.490102 924643.471306 1573597.440256 918705.060385
> > 1333047.771337 917469.027431
> > x24 925584.483103 925224.955009 1578143.485651 921744.15117
> > 1321494.708466 920001.498352
> > x25 926165.366927 926842.031594 1579288.271173 921845.392764
> > 1319902.568202 920448.830702
> > x26 926852.729454 927382.123575 1585351.318945 922669.59912
> > 1325670.791338 919137.796847
> > x27 928196.260748 928093.981204 1581427.557244 921379.155436
> > 1325009.972078 919017.550858
> > x28 929330.433913 929606.778644 1581726.527347 924325.834833
> > 1331721.074174 919557.761373
> > x29 929885.522895 929876.924615 1578317.436513 922977.240966
> > 1333612.86783 921094.386736
> > x30 930635.972805 930599.520144 1583537.946205 922746.107784
> > 1333446.651362 922821.171531
>
> Hi Jack,
>
> What does "dual" mean? What explains the big difference between the NVMe
> and RNBD results for the "dual" columns?
I just sent out v7 with also graphs, it's here:
https://github.com/ionos-enterprise/ibnbd/tree/develop/performance/v7-v5.5-rc2
Measurements are performed in single path (-single) and
multipath(-dual) setting. For multipath measurement two different
policies have been tested:
- RNBD - min-inflight (choose path with minimum inflights)
- NVMEoF - numa iopolicy
>
> Thanks,
>
> Bart.
>
>
Thanks
prev parent reply other threads:[~2020-01-16 16:46 UTC|newest]
Thread overview: 45+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-12-20 15:50 [PATCH v5 00/25] RTRS (former IBTRS) rdma transport library and the corresponding RNBD (former IBNBD) rdma network block device Jack Wang
2019-12-20 15:50 ` [PATCH v5 01/25] sysfs: export sysfs_remove_file_self() Jack Wang
2019-12-20 16:26 ` Jinpu Wang
2019-12-20 15:50 ` [PATCH v5 02/25] rtrs: public interface header to establish RDMA connections Jack Wang
2019-12-21 10:15 ` Leon Romanovsky
2019-12-21 14:27 ` Danil Kipnis
2019-12-22 7:36 ` Leon Romanovsky
2019-12-23 7:38 ` Jinpu Wang
2019-12-23 8:04 ` Leon Romanovsky
2019-12-23 10:31 ` Jinpu Wang
2019-12-20 15:50 ` [PATCH v5 03/25] rtrs: private headers with rtrs protocol structs and helpers Jack Wang
2019-12-20 15:50 ` [PATCH v5 04/25] rtrs: core: lib functions shared between client and server modules Jack Wang
2019-12-20 15:50 ` [PATCH v5 05/25] rtrs: client: private header with client structs and functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 06/25] rtrs: client: main functionality Jack Wang
2019-12-20 15:50 ` [PATCH v5 07/25] rtrs: client: statistics functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 08/25] rtrs: client: sysfs interface functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 09/25] rtrs: server: private header with server structs and functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 10/25] rtrs: server: main functionality Jack Wang
2019-12-20 15:50 ` [PATCH v5 11/25] rtrs: server: statistics functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 12/25] rtrs: server: sysfs interface functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 13/25] rtrs: include client and server modules into kernel compilation Jack Wang
2019-12-20 15:50 ` [PATCH v5 14/25] rtrs: a bit of documentation Jack Wang
2019-12-20 15:50 ` [PATCH v5 15/25] rnbd: private headers with rnbd protocol structs and helpers Jack Wang
2019-12-20 15:51 ` [PATCH v5 16/25] rnbd: client: private header with client structs and functions Jack Wang
2019-12-20 15:51 ` [PATCH v5 17/25] rnbd: client: main functionality Jack Wang
2019-12-20 15:51 ` [PATCH v5 18/25] rnbd: client: sysfs interface functions Jack Wang
2019-12-20 15:51 ` [PATCH v5 19/25] rnbd: server: private header with server structs and functions Jack Wang
2019-12-20 15:51 ` [PATCH v5 20/25] rnbd: server: main functionality Jack Wang
2019-12-20 15:51 ` [PATCH v5 21/25] rnbd: server: functionality for IO submission to file or block dev Jack Wang
2019-12-20 15:51 ` [PATCH v5 22/25] rnbd: server: sysfs interface functions Jack Wang
2019-12-23 8:14 ` Leon Romanovsky
2019-12-23 8:33 ` Jinpu Wang
2019-12-20 15:51 ` [PATCH v5 23/25] rnbd: include client and server modules into kernel compilation Jack Wang
2019-12-20 15:51 ` [PATCH v5 24/25] rnbd: a bit of documentation Jack Wang
2019-12-20 15:51 ` [PATCH v5 25/25] MAINTAINERS: Add maintainers for RNBD/RTRS modules Jack Wang
2019-12-22 9:55 ` Gal Pressman
2019-12-23 7:20 ` Jinpu Wang
2019-12-21 10:17 ` [PATCH v5 00/25] RTRS (former IBTRS) rdma transport library and the corresponding RNBD (former IBNBD) rdma network block device Leon Romanovsky
2020-01-02 18:18 ` Jason Gunthorpe
2020-01-03 12:39 ` Jinpu Wang
2020-01-03 16:28 ` Bart Van Assche
2020-01-06 17:07 ` Jinpu Wang
2020-01-07 10:56 ` Jinpu Wang
2020-01-16 16:41 ` Bart Van Assche
2020-01-16 16:46 ` Jinpu Wang [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAMGffEkXfwMU3AdNGhX9Z202UWJ-YhiCwaaQ=5NwfRmvyhOajg@mail.gmail.com' \
--to=jinpu.wang@cloud.ionos.com \
--cc=axboe@kernel.dk \
--cc=bvanassche@acm.org \
--cc=danil.kipnis@cloud.ionos.com \
--cc=dledford@redhat.com \
--cc=hch@infradead.org \
--cc=jgg@ziepe.ca \
--cc=jinpuwang@gmail.com \
--cc=leon@kernel.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=rpenyaev@suse.de \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).