Linux-Block Archive on lore.kernel.org
 help / color / Atom feed
From: Jinpu Wang <jinpu.wang@cloud.ionos.com>
To: Bart Van Assche <bvanassche@acm.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>, Jack Wang <jinpuwang@gmail.com>,
	linux-block@vger.kernel.org, linux-rdma@vger.kernel.org,
	Jens Axboe <axboe@kernel.dk>,
	Christoph Hellwig <hch@infradead.org>,
	Sagi Grimberg <sagi@grimberg.me>,
	Leon Romanovsky <leon@kernel.org>,
	Doug Ledford <dledford@redhat.com>,
	Danil Kipnis <danil.kipnis@cloud.ionos.com>,
	Roman Penyaev <rpenyaev@suse.de>
Subject: Re: [PATCH v5 00/25] RTRS (former IBTRS) rdma transport library and the corresponding RNBD (former IBNBD) rdma network block device
Date: Thu, 16 Jan 2020 17:46:19 +0100
Message-ID: <CAMGffEkXfwMU3AdNGhX9Z202UWJ-YhiCwaaQ=5NwfRmvyhOajg@mail.gmail.com> (raw)
In-Reply-To: <c0c05209-1f43-c242-96d6-1944e43d207c@acm.org>

On Thu, Jan 16, 2020 at 5:41 PM Bart Van Assche <bvanassche@acm.org> wrote:
>
> On 1/7/20 2:56 AM, Jinpu Wang wrote:
> > On Mon, Jan 6, 2020 at 6:07 PM Jinpu Wang <jinpu.wang@cloud.ionos.com> wrote:
> >>
> >> On Fri, Jan 3, 2020 at 5:29 PM Bart Van Assche <bvanassche@acm.org> wrote:
> >>>
> >>> On 1/3/20 4:39 AM, Jinpu Wang wrote:
> >>>> Performance results for the v5.5-rc1 kernel are here:
> >>>>     link: https://github.com/ionos-enterprise/ibnbd/tree/develop/performance/v5-v5.5-rc1
> >>>>
> >>>> Some workloads RNBD are faster, some workloads NVMeoF are faster.
> >>>
> >>> Thank you for having shared these graphs.
> >>>
> >>> Do the graphs in RNBD-SinglePath.pdf show that NVMeOF achieves similar
> >>> or higher IOPS, higher bandwidth and lower latency than RNBD for
> >>> workloads with a block size of 4 KB and also for mixed workloads with
> >>> less than 20 disks, whether or not invalidation is enabled for RNBD?
> >> Hi Bart,
> >>
> >> Yes, that's the result on one pair of Server with Connect X4 HCA, I
> >> did another test on another
> >> 2 servers with Connect X5 HCA, results are quite different, we will
> >> double-check the
> >> performance results also on old machines, will share new results later.
> >>
> > here are the results with ConnectX5 HCA MT4119 + Intel(R) Xeon(R) Gold
> > 6130 CPU @ 2.10GHz, sorry no graph for now,
> > will prepare the next round.
> >
> >   disks   4k nvme dual  4k nvme single    4k rnbd dual  4k rnbd single
> > 4k rnbd-inv dual  4k rnbd-inv single
> >     x1  251637.436256   254312.068793   270886.311369   260934.306569
> >    218632.336766       190800.519948
> >     x2  460894.610539   463925.907409   496318.068193   466374.862514
> >     418960.30397       372848.815118
> >     x3  603263.673633    605004.49955   675073.892611   614552.144786
> >    586223.077692       524221.977802
> >     x4  731648.935106   733174.482552   850245.575442   743062.493751
> >    744380.361964       656861.813819
> >     x5  827732.326767   827444.855514  1026939.306069   840515.548445
> >    897801.719828       762707.329267
> >     x6  876705.329467     873963.0037  1142399.960004    876974.70253
> >   1037773.522648       834073.892611
> >     x7  893808.719128   893268.073193  1239282.471753   892728.027197
> >   1135570.742926       871336.966303
> >     x8  906589.741026   905938.006199  1287178.964207   906189.381062
> >     1225040.9959       895292.070793
> >     x9   912048.09519   912400.259974  1386211.878812   913885.311469
> >   1302472.964884       910176.282372
> >    x10  915566.243376   915602.739726   1442959.70403   916288.871113
> >   1350296.325879        914966.40336
> >    x11  917116.188381   916905.809419  1418574.942506   916264.373563
> >   1370438.698083       915255.874413
> >    x12  915852.814719   917710.128987  1423534.546545   916348.386452
> >   1352357.364264       914966.656684
> >    x13   919042.69573   918819.536093  1429697.830217   917631.036896
> >   1378083.824558       916519.161677
> >    x14   920000.49995    920031.59684  1443317.268273   917562.843716
> >    1395023.56936       918935.706429
> >    x15  920160.883912   920367.363264  1445306.425863   918278.472153
> >   1440776.944611       916352.265921
> >    x16  920652.869426   920673.832617  1454705.229477   917902.948526
> >     1455708.2501       918198.001998
> >    x17  916892.310769   916883.623275  1491071.841948   918936.706329
> >   1436507.428457       917182.934132
> >    x18  917247.775222   917762.523748  1612129.058036   918546.835949
> >   1488716.583417       919521.095781
> >    x19  920084.791521   920349.930014   1615690.87821   915371.496958
> >    1406747.32954       918347.248577
> >    x20  922339.232154   922208.058388  1591415.958404   917922.631526
> >   1343806.744488       918903.393214
> >    x21  923141.771646   923297.040592  1581547.169811   919371.025795
> >   1342568.406347       919157.752944
> >    x22  923063.787243   924072.385523  1574972.846162   919173.713143
> >   1340318.639673       920577.995403
> >    x23  923549.490102   924643.471306  1573597.440256   918705.060385
> >   1333047.771337       917469.027431
> >    x24  925584.483103   925224.955009  1578143.485651    921744.15117
> >   1321494.708466       920001.498352
> >    x25  926165.366927   926842.031594  1579288.271173   921845.392764
> >   1319902.568202       920448.830702
> >    x26  926852.729454   927382.123575  1585351.318945    922669.59912
> >   1325670.791338       919137.796847
> >    x27  928196.260748   928093.981204  1581427.557244   921379.155436
> >   1325009.972078       919017.550858
> >    x28  929330.433913   929606.778644  1581726.527347   924325.834833
> >   1331721.074174       919557.761373
> >    x29  929885.522895   929876.924615  1578317.436513   922977.240966
> >    1333612.86783       921094.386736
> >    x30  930635.972805   930599.520144  1583537.946205   922746.107784
> >   1333446.651362       922821.171531
>
> Hi Jack,
>
> What does "dual" mean? What explains the big difference between the NVMe
> and RNBD results for the "dual" columns?
I just sent out v7 with also graphs,  it's here:
https://github.com/ionos-enterprise/ibnbd/tree/develop/performance/v7-v5.5-rc2

Measurements are performed in single path (-single) and
multipath(-dual) setting. For multipath measurement two different
policies have been tested:
- RNBD - min-inflight (choose path with minimum inflights)
- NVMEoF - numa iopolicy


>
> Thanks,
>
> Bart.
>
>
Thanks

      reply index

Thread overview: 45+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-20 15:50 Jack Wang
2019-12-20 15:50 ` [PATCH v5 01/25] sysfs: export sysfs_remove_file_self() Jack Wang
2019-12-20 16:26   ` Jinpu Wang
2019-12-20 15:50 ` [PATCH v5 02/25] rtrs: public interface header to establish RDMA connections Jack Wang
2019-12-21 10:15   ` Leon Romanovsky
2019-12-21 14:27     ` Danil Kipnis
2019-12-22  7:36       ` Leon Romanovsky
2019-12-23  7:38         ` Jinpu Wang
2019-12-23  8:04           ` Leon Romanovsky
2019-12-23 10:31             ` Jinpu Wang
2019-12-20 15:50 ` [PATCH v5 03/25] rtrs: private headers with rtrs protocol structs and helpers Jack Wang
2019-12-20 15:50 ` [PATCH v5 04/25] rtrs: core: lib functions shared between client and server modules Jack Wang
2019-12-20 15:50 ` [PATCH v5 05/25] rtrs: client: private header with client structs and functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 06/25] rtrs: client: main functionality Jack Wang
2019-12-20 15:50 ` [PATCH v5 07/25] rtrs: client: statistics functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 08/25] rtrs: client: sysfs interface functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 09/25] rtrs: server: private header with server structs and functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 10/25] rtrs: server: main functionality Jack Wang
2019-12-20 15:50 ` [PATCH v5 11/25] rtrs: server: statistics functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 12/25] rtrs: server: sysfs interface functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 13/25] rtrs: include client and server modules into kernel compilation Jack Wang
2019-12-20 15:50 ` [PATCH v5 14/25] rtrs: a bit of documentation Jack Wang
2019-12-20 15:50 ` [PATCH v5 15/25] rnbd: private headers with rnbd protocol structs and helpers Jack Wang
2019-12-20 15:51 ` [PATCH v5 16/25] rnbd: client: private header with client structs and functions Jack Wang
2019-12-20 15:51 ` [PATCH v5 17/25] rnbd: client: main functionality Jack Wang
2019-12-20 15:51 ` [PATCH v5 18/25] rnbd: client: sysfs interface functions Jack Wang
2019-12-20 15:51 ` [PATCH v5 19/25] rnbd: server: private header with server structs and functions Jack Wang
2019-12-20 15:51 ` [PATCH v5 20/25] rnbd: server: main functionality Jack Wang
2019-12-20 15:51 ` [PATCH v5 21/25] rnbd: server: functionality for IO submission to file or block dev Jack Wang
2019-12-20 15:51 ` [PATCH v5 22/25] rnbd: server: sysfs interface functions Jack Wang
2019-12-23  8:14   ` Leon Romanovsky
2019-12-23  8:33     ` Jinpu Wang
2019-12-20 15:51 ` [PATCH v5 23/25] rnbd: include client and server modules into kernel compilation Jack Wang
2019-12-20 15:51 ` [PATCH v5 24/25] rnbd: a bit of documentation Jack Wang
2019-12-20 15:51 ` [PATCH v5 25/25] MAINTAINERS: Add maintainers for RNBD/RTRS modules Jack Wang
2019-12-22  9:55   ` Gal Pressman
2019-12-23  7:20     ` Jinpu Wang
2019-12-21 10:17 ` [PATCH v5 00/25] RTRS (former IBTRS) rdma transport library and the corresponding RNBD (former IBNBD) rdma network block device Leon Romanovsky
2020-01-02 18:18 ` Jason Gunthorpe
2020-01-03 12:39   ` Jinpu Wang
2020-01-03 16:28     ` Bart Van Assche
2020-01-06 17:07       ` Jinpu Wang
2020-01-07 10:56         ` Jinpu Wang
2020-01-16 16:41           ` Bart Van Assche
2020-01-16 16:46             ` Jinpu Wang [this message]

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAMGffEkXfwMU3AdNGhX9Z202UWJ-YhiCwaaQ=5NwfRmvyhOajg@mail.gmail.com' \
    --to=jinpu.wang@cloud.ionos.com \
    --cc=axboe@kernel.dk \
    --cc=bvanassche@acm.org \
    --cc=danil.kipnis@cloud.ionos.com \
    --cc=dledford@redhat.com \
    --cc=hch@infradead.org \
    --cc=jgg@ziepe.ca \
    --cc=jinpuwang@gmail.com \
    --cc=leon@kernel.org \
    --cc=linux-block@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=rpenyaev@suse.de \
    --cc=sagi@grimberg.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Block Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-block/0 linux-block/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-block linux-block/ https://lore.kernel.org/linux-block \
		linux-block@vger.kernel.org
	public-inbox-index linux-block

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-block


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git