From: Jack Wang <jinpuwang@gmail.com>
To: linux-block@vger.kernel.org, linux-rdma@vger.kernel.org
Cc: axboe@kernel.dk, hch@infradead.org, sagi@grimberg.me,
bvanassche@acm.org, leon@kernel.org, dledford@redhat.com,
danil.kipnis@cloud.ionos.com, jinpu.wang@cloud.ionos.com,
rpenyaev@suse.de
Subject: [PATCH v5 00/25] RTRS (former IBTRS) rdma transport library and the corresponding RNBD (former IBNBD) rdma network block device
Date: Fri, 20 Dec 2019 16:50:44 +0100 [thread overview]
Message-ID: <20191220155109.8959-1-jinpuwang@gmail.com> (raw)
Hi all,
here is V5 of the RTRS (former IBTRS) rdma transport library and the
corresponding RNBD (former IBNBD) rdma network block device.
Main changes are the following:
1. Fix the security problem pointed out by Jason
2. Implement code-style/readability/API/etc suggestions by Bart van Assche
3. Rename IBTRS and IBNBD to RTRS and RNBD accordingly
4. Fileio mode support in rnbd-srv has been removed.
The main functional change is a fix for the security problem pointed out by
Jason and discussed both on the mailing list and during the last LPC RDMA MC 2019.
On the server side we now invalidate in RTRS each rdma buffer before we hand it
over to RNBD server and in turn to the block layer. A new rkey is generated and
registered for the buffer after it returns back from the block layer and RNBD
server. The new rkey is sent back to the client along with the IO result.
The procedure is the default behaviour of the driver. This invalidation and
registration on each IO causes performance drop of up to 20%. A user of the
driver may choose to load the modules with this mechanism switched off
(always_invalidate=N), if he understands and can take the risk of a malicious
client being able to corrupt memory of a server it is connected to. This might
be a reasonable option in a scenario where all the clients and all the servers
are located within a secure datacenter.
Huge thanks to Bart van Assche for the very detailed review of both RNBD and
RTRS. These included suggestions for style fixes, better readability and
documentation, code simplifications, eliminating usage of deprecated APIs,
too many to name.
The transport library and the network block device using it have been renamed to
RTRS and RNBD accordingly in order to reflect the fact that they are based on
the rdma subsystem and not bound to InfiniBand only.
Fileio mode support in rnbd-server is not so efficent as pointed out by Bart,
and we can use loop device in between if there is need, hence we just
removed the fileio mode support.
Introduction
-------------
RTRS (RDMA Transport) is a reliable high speed transport library
which allows for establishing connection between client and server
machines via RDMA. It is based on RDMA-CM, so expect also to support RoCE
and iWARP, but we mainly tested in IB environment. It is optimized to
transfer (read/write) IO blocks in the sense that it follows the BIO
semantics of providing the possibility to either write data from a
scatter-gather list to the remote side or to request ("read") data
transfer from the remote side into a given set of buffers.
RTRS is multipath capable and provides I/O fail-over and load-balancing
functionality, i.e. in RTRS terminology, an RTRS path is a set of RDMA
connections and particular path is selected according to the load-balancing policy.
It can be used for other components not bind to RNBD.
RNBD (InfiniBand Network Block Device) is a pair of kernel modules
(client and server) that allow for remote access of a block device on
the server over RTRS protocol. After being mapped, the remote block
devices can be accessed on the client side as local block devices.
Internally RNBD uses RTRS as an RDMA transport library.
Commits for kernel can be found here:
https://github.com/ionos-enterprise/ibnbd/commits/linux-5.5-rc2-ibnbd-v5
The out-of-tree modules are here:
https://github.com/ionos-enterprise/ibnbd
LPC 2019 presentation:
https://linuxplumbersconf.org/event/4/contributions/367/attachments/331/555/LPC_2019_RMDA_MC_IBNBD_IBTRS_Upstreaming.pdf
Performance results for the v5.5-rc1 kernel
link: https://github.com/ionos-enterprise/ibnbd/tree/develop/performance/v5-v5.5-rc1
Jack Wang (25):
sysfs: export sysfs_remove_file_self()
rtrs: public interface header to establish RDMA connections
rtrs: private headers with rtrs protocol structs and helpers
rtrs: core: lib functions shared between client and server modules
rtrs: client: private header with client structs and functions
rtrs: client: main functionality
rtrs: client: statistics functions
rtrs: client: sysfs interface functions
rtrs: server: private header with server structs and functions
rtrs: server: main functionality
rtrs: server: statistics functions
rtrs: server: sysfs interface functions
rtrs: include client and server modules into kernel compilation
rtrs: a bit of documentation
rnbd: private headers with rnbd protocol structs and helpers
rnbd: client: private header with client structs and functions
rnbd: client: main functionality
rnbd: client: sysfs interface functions
rnbd: server: private header with server structs and functions
rnbd: server: main functionality
rnbd: server: functionality for IO submission to file or block dev
rnbd: server: sysfs interface functions
rnbd: include client and server modules into kernel compilation
rnbd: a bit of documentation
MAINTAINERS: Add maintainers for RNBD/RTRS modules
Documentation/ABI/testing/sysfs-block-rnbd | 51 +
.../ABI/testing/sysfs-class-rnbd-client | 117 +
.../ABI/testing/sysfs-class-rnbd-server | 57 +
.../ABI/testing/sysfs-class-rtrs-client | 190 ++
.../ABI/testing/sysfs-class-rtrs-server | 81 +
MAINTAINERS | 14 +
drivers/block/Kconfig | 2 +
drivers/block/Makefile | 1 +
drivers/block/rnbd/Kconfig | 28 +
drivers/block/rnbd/Makefile | 17 +
drivers/block/rnbd/README | 80 +
drivers/block/rnbd/rnbd-clt-sysfs.c | 659 ++++
drivers/block/rnbd/rnbd-clt.c | 1761 ++++++++++
drivers/block/rnbd/rnbd-clt.h | 169 +
drivers/block/rnbd/rnbd-common.c | 43 +
drivers/block/rnbd/rnbd-log.h | 61 +
drivers/block/rnbd/rnbd-proto.h | 325 ++
drivers/block/rnbd/rnbd-srv-dev.c | 162 +
drivers/block/rnbd/rnbd-srv-dev.h | 130 +
drivers/block/rnbd/rnbd-srv-sysfs.c | 234 ++
drivers/block/rnbd/rnbd-srv.c | 882 +++++
drivers/block/rnbd/rnbd-srv.h | 99 +
drivers/infiniband/Kconfig | 1 +
drivers/infiniband/ulp/Makefile | 1 +
drivers/infiniband/ulp/rtrs/Kconfig | 27 +
drivers/infiniband/ulp/rtrs/Makefile | 17 +
drivers/infiniband/ulp/rtrs/README | 137 +
drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c | 453 +++
drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c | 519 +++
drivers/infiniband/ulp/rtrs/rtrs-clt.c | 2952 +++++++++++++++++
drivers/infiniband/ulp/rtrs/rtrs-clt.h | 314 ++
drivers/infiniband/ulp/rtrs/rtrs-log.h | 50 +
drivers/infiniband/ulp/rtrs/rtrs-pri.h | 426 +++
drivers/infiniband/ulp/rtrs/rtrs-srv-stats.c | 109 +
drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c | 315 ++
drivers/infiniband/ulp/rtrs/rtrs-srv.c | 2187 ++++++++++++
drivers/infiniband/ulp/rtrs/rtrs-srv.h | 159 +
drivers/infiniband/ulp/rtrs/rtrs.c | 646 ++++
drivers/infiniband/ulp/rtrs/rtrs.h | 334 ++
fs/sysfs/file.c | 1 +
40 files changed, 13811 insertions(+)
create mode 100644 Documentation/ABI/testing/sysfs-block-rnbd
create mode 100644 Documentation/ABI/testing/sysfs-class-rnbd-client
create mode 100644 Documentation/ABI/testing/sysfs-class-rnbd-server
create mode 100644 Documentation/ABI/testing/sysfs-class-rtrs-client
create mode 100644 Documentation/ABI/testing/sysfs-class-rtrs-server
create mode 100644 drivers/block/rnbd/Kconfig
create mode 100644 drivers/block/rnbd/Makefile
create mode 100644 drivers/block/rnbd/README
create mode 100644 drivers/block/rnbd/rnbd-clt-sysfs.c
create mode 100644 drivers/block/rnbd/rnbd-clt.c
create mode 100644 drivers/block/rnbd/rnbd-clt.h
create mode 100644 drivers/block/rnbd/rnbd-common.c
create mode 100644 drivers/block/rnbd/rnbd-log.h
create mode 100644 drivers/block/rnbd/rnbd-proto.h
create mode 100644 drivers/block/rnbd/rnbd-srv-dev.c
create mode 100644 drivers/block/rnbd/rnbd-srv-dev.h
create mode 100644 drivers/block/rnbd/rnbd-srv-sysfs.c
create mode 100644 drivers/block/rnbd/rnbd-srv.c
create mode 100644 drivers/block/rnbd/rnbd-srv.h
create mode 100644 drivers/infiniband/ulp/rtrs/Kconfig
create mode 100644 drivers/infiniband/ulp/rtrs/Makefile
create mode 100644 drivers/infiniband/ulp/rtrs/README
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-clt-sysfs.c
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-clt.c
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-clt.h
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-log.h
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-pri.h
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-srv-stats.c
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-srv-sysfs.c
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-srv.c
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs-srv.h
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs.c
create mode 100644 drivers/infiniband/ulp/rtrs/rtrs.h
--
2.17.1
next reply other threads:[~2019-12-20 15:51 UTC|newest]
Thread overview: 45+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-12-20 15:50 Jack Wang [this message]
2019-12-20 15:50 ` [PATCH v5 01/25] sysfs: export sysfs_remove_file_self() Jack Wang
2019-12-20 16:26 ` Jinpu Wang
2019-12-20 15:50 ` [PATCH v5 02/25] rtrs: public interface header to establish RDMA connections Jack Wang
2019-12-21 10:15 ` Leon Romanovsky
2019-12-21 14:27 ` Danil Kipnis
2019-12-22 7:36 ` Leon Romanovsky
2019-12-23 7:38 ` Jinpu Wang
2019-12-23 8:04 ` Leon Romanovsky
2019-12-23 10:31 ` Jinpu Wang
2019-12-20 15:50 ` [PATCH v5 03/25] rtrs: private headers with rtrs protocol structs and helpers Jack Wang
2019-12-20 15:50 ` [PATCH v5 04/25] rtrs: core: lib functions shared between client and server modules Jack Wang
2019-12-20 15:50 ` [PATCH v5 05/25] rtrs: client: private header with client structs and functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 06/25] rtrs: client: main functionality Jack Wang
2019-12-20 15:50 ` [PATCH v5 07/25] rtrs: client: statistics functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 08/25] rtrs: client: sysfs interface functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 09/25] rtrs: server: private header with server structs and functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 10/25] rtrs: server: main functionality Jack Wang
2019-12-20 15:50 ` [PATCH v5 11/25] rtrs: server: statistics functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 12/25] rtrs: server: sysfs interface functions Jack Wang
2019-12-20 15:50 ` [PATCH v5 13/25] rtrs: include client and server modules into kernel compilation Jack Wang
2019-12-20 15:50 ` [PATCH v5 14/25] rtrs: a bit of documentation Jack Wang
2019-12-20 15:50 ` [PATCH v5 15/25] rnbd: private headers with rnbd protocol structs and helpers Jack Wang
2019-12-20 15:51 ` [PATCH v5 16/25] rnbd: client: private header with client structs and functions Jack Wang
2019-12-20 15:51 ` [PATCH v5 17/25] rnbd: client: main functionality Jack Wang
2019-12-20 15:51 ` [PATCH v5 18/25] rnbd: client: sysfs interface functions Jack Wang
2019-12-20 15:51 ` [PATCH v5 19/25] rnbd: server: private header with server structs and functions Jack Wang
2019-12-20 15:51 ` [PATCH v5 20/25] rnbd: server: main functionality Jack Wang
2019-12-20 15:51 ` [PATCH v5 21/25] rnbd: server: functionality for IO submission to file or block dev Jack Wang
2019-12-20 15:51 ` [PATCH v5 22/25] rnbd: server: sysfs interface functions Jack Wang
2019-12-23 8:14 ` Leon Romanovsky
2019-12-23 8:33 ` Jinpu Wang
2019-12-20 15:51 ` [PATCH v5 23/25] rnbd: include client and server modules into kernel compilation Jack Wang
2019-12-20 15:51 ` [PATCH v5 24/25] rnbd: a bit of documentation Jack Wang
2019-12-20 15:51 ` [PATCH v5 25/25] MAINTAINERS: Add maintainers for RNBD/RTRS modules Jack Wang
2019-12-22 9:55 ` Gal Pressman
2019-12-23 7:20 ` Jinpu Wang
2019-12-21 10:17 ` [PATCH v5 00/25] RTRS (former IBTRS) rdma transport library and the corresponding RNBD (former IBNBD) rdma network block device Leon Romanovsky
2020-01-02 18:18 ` Jason Gunthorpe
2020-01-03 12:39 ` Jinpu Wang
2020-01-03 16:28 ` Bart Van Assche
2020-01-06 17:07 ` Jinpu Wang
2020-01-07 10:56 ` Jinpu Wang
2020-01-16 16:41 ` Bart Van Assche
2020-01-16 16:46 ` Jinpu Wang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191220155109.8959-1-jinpuwang@gmail.com \
--to=jinpuwang@gmail.com \
--cc=axboe@kernel.dk \
--cc=bvanassche@acm.org \
--cc=danil.kipnis@cloud.ionos.com \
--cc=dledford@redhat.com \
--cc=hch@infradead.org \
--cc=jinpu.wang@cloud.ionos.com \
--cc=leon@kernel.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=rpenyaev@suse.de \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).