From: Roman Pen <roman.penyaev@profitbricks.com>
To: linux-block@vger.kernel.org, linux-rdma@vger.kernel.org
Cc: Jens Axboe <axboe@kernel.dk>,
Christoph Hellwig <hch@infradead.org>,
Sagi Grimberg <sagi@grimberg.me>,
Bart Van Assche <bart.vanassche@sandisk.com>,
Or Gerlitz <ogerlitz@mellanox.com>,
Roman Pen <roman.penyaev@profitbricks.com>,
Danil Kipnis <danil.kipnis@profitbricks.com>,
Jack Wang <jinpu.wang@profitbricks.com>
Subject: [PATCH 23/24] ibnbd: a bit of documentation
Date: Fri, 2 Feb 2018 15:09:03 +0100 [thread overview]
Message-ID: <20180202140904.2017-24-roman.penyaev@profitbricks.com> (raw)
In-Reply-To: <20180202140904.2017-1-roman.penyaev@profitbricks.com>
README with description of major sysfs entries.
Signed-off-by: Roman Pen <roman.penyaev@profitbricks.com>
Signed-off-by: Danil Kipnis <danil.kipnis@profitbricks.com>
Cc: Jack Wang <jinpu.wang@profitbricks.com>
---
drivers/block/ibnbd/README | 272 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 272 insertions(+)
diff --git a/drivers/block/ibnbd/README b/drivers/block/ibnbd/README
new file mode 100644
index 000000000000..e0feb39fad14
--- /dev/null
+++ b/drivers/block/ibnbd/README
@@ -0,0 +1,272 @@
+***************************************
+Infiniband Network Block Device (IBNBD)
+***************************************
+
+Introduction
+------------
+
+IBNBD (InfiniBand Network Block Device) is a pair of kernel modules
+(client and server) that allow for remote access of a block device on
+the server over IBTRS protocol using the RDMA (InfiniBand, RoCE, iWarp)
+transport. After being mapped, the remote block devices can be accessed
+on the client side as local block devices.
+
+I/O is transfered between client and server by the IBTRS transport
+modules. The administration of IBNBD and IBTRS modules is done via
+sysfs entries.
+
+Requirements
+------------
+
+ IBTRS kernel modules
+
+Quick Start
+-----------
+
+Server side:
+ # modprobe ibnbd_server
+
+Client side:
+ # modprobe ibnbd_client
+ # echo "sessname=blya path=ip:10.50.100.66 device_path=/dev/ram0" > \
+ /sys/kernel/ibnbd_client/map_device
+
+ Where "sessname=" is a session name, a string to identify the session
+ on client and on server sides; "path=" is a destination IP address or
+ a pair of a source and a destination IPs, separated by comma. Multiple
+ "path=" options can be specified in order to use multipath (see IBTRS
+ description for details); "device_path=" is the block device to be
+ mapped from the server side. After the session to the server machine is
+ established, the mapped device will appear on the client side under
+ /dev/ibnbd<N>.
+
+
+======================
+Client Sysfs Interface
+======================
+
+All sysfs files that are not read-only provide the usage information on read:
+
+Example:
+ # cat /sys/kernel/ibnbd_client/map_device
+
+ > Usage: echo "sessname=<name of the ibtrs session> path=<[srcaddr,]dstaddr>
+ > [path=<[srcaddr,]dstaddr>] device_path=<full path on remote side>
+ > [access_mode=<ro|rw|migration>] [input_mode=<mq|rq>]
+ > [io_mode=<fileio|blockio>]" > map_device
+ >
+ > addr ::= [ ip:<ipv4> | ip:<ipv6> | gid:<gid> ]
+
+Entries under /sys/kernel/ibnbd_client/
+=======================================
+
+map_device (RW)
+---------------
+
+Expected format is the following:
+
+ sessname=<name of the ibtrs session>
+ path=<[srcaddr,]dstaddr> [path=<[srcaddr,]dstaddr> ...]
+ device_path=<full path on remote side>
+ [access_mode=<ro|rw|migration>]
+ [input_mode=<mq|rq>]
+ [io_mode=<fileio|blockio>]
+
+Where:
+
+sessname: accepts a string not bigger than 256 chars, which identifies
+ a given session on the client and on the server.
+ I.e. "clt_hostname-srv_hostname" could be a natural choice.
+
+path: describes a connection between the client and the server by
+ specifying destination and, when required, the source address.
+ The addresses are to be provided in the following format:
+
+ ip:<IPv6>
+ ip:<IPv4>
+ gid:<GID>
+
+ for example:
+
+ path=ip:10.0.0.66
+ The single addr is treated as the destination.
+ The connection will be established to this
+ server from any client IP address.
+
+ path=ip:10.0.0.66,ip:10.0.1.66
+ First addr is the source address and the second
+ is the destination.
+
+ If multiple "path=" options are specified multiple connection
+ will be established and data will be sent according to
+ the selected multipath policy (see IBTRS mp_policy sysfs entry
+ description).
+
+device_path: Path to the block device on the server side. Path is specified
+ relative to the directory on server side configured in the
+ 'dev_search_path' module parameter of the ibnbd_server.
+ The ibnbd_server prepends the <device_path> received from client
+ with <dev_search_path> and tries to open the
+ <dev_search_path>/<device_path> block device. On success,
+ a /dev/ibnbd<N> device file, a /sys/block/ibnbd_client/ibnbd<N>/
+ directory and an entry in /sys/kernel/ibnbd_client/devices will be
+ created.
+
+access_mode: the access_mode parameter specifies if the device is to be
+ mapped as "ro" read-only or "rw" read-write. The server allows
+ a device to be exported in rw mode only once. The "migration"
+ access mode has to be specified if a second mapping in read-write
+ mode is desired.
+
+ By default "rw" is used.
+
+input_mode: the input_mode parameter specifies the internal I/O
+ processing mode of the block device on the client. Accepts
+ "mq" and "rq".
+
+ By default "mq" mode is used.
+
+io_mode: the io_mode parameter specifies if the device on the server
+ will be opened as block device "blockio" or as file "fileio".
+ When the device is opened as file, the VFS page cache is used
+ for read I/O operations, write I/O operations bypass the page
+ cache and go directly to disk (except meta updates, like file
+ access time).
+
+ By default "blockio" mode is used.
+
+Exit Codes:
+
+If the device is already mapped it will fail with EEXIST. If the input
+has an invalid format it will return EINVAL. If the device path cannot
+be found on the server, it will fail with ENOENT.
+
+Finding device file after mapping
+---------------------------------
+
+After mapping, the device file can be found by:
+ o The symlink /sys/kernel/ibnbd_client/devices/<device_id> points to
+ /sys/block/<dev-name>. The last part of the symlink destination is
+ the same as the device name. By extracting the last part of the
+ path the path to the device /dev/<dev-name> can be build.
+
+ o /dev/block/$(cat /sys/kernel/ibnbd_client/devices/<device_id>/dev)
+
+How to find the <device_id> of the device is described on the next
+section.
+
+Entries under /sys/kernel/ibnbd_client/devices/
+===============================================
+
+For each device mapped on the client a new symbolic link is created as
+/sys/kernel/ibnbd_client/devices/<device_id>, which points to the block
+device created by ibnbd (/sys/block/ibnbd<N>/). The <device_id> of each
+device is created as follows:
+
+- If the 'device_path' provided during mapping contains slashes ("/"),
+ they are replaced by exclamation mark ("!") and used as as the
+ <device_id>. Otherwise, the <device_id> will be the same as the
+ "device_path" provided.
+
+Entries under /sys/block/ibnbd<N>/ibnbd_client/
+===============================================
+
+unmap_device (RW)
+-----------------
+
+To unmap a volume, "normal" or "force" has to be written to:
+ /sys/block/ibnbd<N>/ibnbd_client/unmap_device
+
+When "normal" is used, the operation will fail with EBUSY if any process
+is using the device. When "force" is used, the device is also unmapped
+when device is in use. All I/Os that are in progress will fail.
+
+Example:
+
+ # echo "normal" > /sys/block/ibnbd0/ibnbd/unmap_device
+
+state (RO)
+----------
+
+The file contains the current state of the block device. The state file
+returns "open" when the device is successfully mapped from the server
+and accepting I/O requests. When the connection to the server gets
+disconnected in case of an error (e.g. link failure), the state file
+returns "closed" and all I/O requests submitted to it will fail with -EIO.
+
+session (RO)
+------------
+
+IBNBD uses IBTRS session to transport the data between client and
+server. The entry "session" contains the name of the session, that
+was used to establish the IBTRS session. It's the same name that
+was passed as server parameter to the map_device entry.
+
+mapping_path (RO)
+-----------------
+
+Contains the path that was passed as "device_path" to the map_device
+operation.
+
+======================
+Server Sysfs Interface
+======================
+
+Entries under /sys/kernel/ibnbd_server/
+=======================================
+
+When a client maps a device, a directory entry with the name of the
+block device is created under /sys/kernel/ibnbd_server/devices/.
+
+Entries under /sys/kernel/ibnbd_server/devices/<device_name>/
+=============================================================
+
+block_dev (link)
+---------------
+
+Is a symlink to the sysfs entry of the exported device.
+
+Example:
+
+ block_dev -> ../../../../devices/virtual/block/ram0
+
+Entries under /sys/kernel/ibnbd_server/devices/<device_name>/sessions/
+======================================================================
+
+For each client a particular device is exported to, following directory will be
+created:
+
+/sys/kernel/ibnbd_server/devices/<device_name>/sessions/<session-name>/
+
+When the device is unmapped by that client, the directory will be removed.
+
+Entries under /sys/kernel/ibnbd_server/devices/<device_name>/sessions/<session-name>
+====================================================================================
+
+read_only (RO)
+--------------
+
+Contains '1' if device is mapped read-only, otherwise '0'.
+
+mapping_path (RO)
+-----------------
+
+Contains the relative device path provided by the user during mapping.
+
+==============================
+IBNBD-Server Module Parameters
+==============================
+
+dev_search_path
+---------------
+
+When a device is mapped from the client, the server generates the path
+to the block device on the server side by concatenating dev_search_path
+and the "device_path" that was specified in the map_device operation.
+
+The default dev_search_path is: "/".
+
+Contact
+-------
+
+Mailing list: "IBNBD/IBTRS Storage Team" <ibnbd@profitbricks.com>
--
2.13.1
next prev parent reply other threads:[~2018-02-02 14:09 UTC|newest]
Thread overview: 79+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-02-02 14:08 [PATCH 00/24] InfiniBand Transport (IBTRS) and Network Block Device (IBNBD) Roman Pen
2018-02-02 14:08 ` [PATCH 01/24] ibtrs: public interface header to establish RDMA connections Roman Pen
2018-02-02 14:08 ` [PATCH 02/24] ibtrs: private headers with IBTRS protocol structs and helpers Roman Pen
2018-02-02 14:08 ` [PATCH 03/24] ibtrs: core: lib functions shared between client and server modules Roman Pen
2018-02-05 10:52 ` Sagi Grimberg
2018-02-06 12:01 ` Roman Penyaev
2018-02-06 16:10 ` Jason Gunthorpe
2018-02-07 10:34 ` Roman Penyaev
2018-02-02 14:08 ` [PATCH 04/24] ibtrs: client: private header with client structs and functions Roman Pen
2018-02-05 10:59 ` Sagi Grimberg
2018-02-06 12:23 ` Roman Penyaev
2018-02-02 14:08 ` [PATCH 05/24] ibtrs: client: main functionality Roman Pen
2018-02-02 16:54 ` Bart Van Assche
2018-02-05 13:27 ` Roman Penyaev
2018-02-05 14:14 ` Sagi Grimberg
2018-02-05 17:05 ` Roman Penyaev
2018-02-05 11:19 ` Sagi Grimberg
2018-02-05 14:19 ` Roman Penyaev
2018-02-05 16:24 ` Bart Van Assche
2018-02-02 14:08 ` [PATCH 06/24] ibtrs: client: statistics functions Roman Pen
2018-02-02 14:08 ` [PATCH 07/24] ibtrs: client: sysfs interface functions Roman Pen
2018-02-05 11:20 ` Sagi Grimberg
2018-02-06 12:28 ` Roman Penyaev
2018-02-02 14:08 ` [PATCH 08/24] ibtrs: server: private header with server structs and functions Roman Pen
2018-02-02 14:08 ` [PATCH 09/24] ibtrs: server: main functionality Roman Pen
2018-02-05 11:29 ` Sagi Grimberg
2018-02-06 12:46 ` Roman Penyaev
2018-02-02 14:08 ` [PATCH 10/24] ibtrs: server: statistics functions Roman Pen
2018-02-02 14:08 ` [PATCH 11/24] ibtrs: server: sysfs interface functions Roman Pen
2018-02-02 14:08 ` [PATCH 12/24] ibtrs: include client and server modules into kernel compilation Roman Pen
2018-02-02 14:08 ` [PATCH 13/24] ibtrs: a bit of documentation Roman Pen
2018-02-02 14:08 ` [PATCH 14/24] ibnbd: private headers with IBNBD protocol structs and helpers Roman Pen
2018-02-02 14:08 ` [PATCH 15/24] ibnbd: client: private header with client structs and functions Roman Pen
2018-02-02 14:08 ` [PATCH 16/24] ibnbd: client: main functionality Roman Pen
2018-02-02 15:11 ` Jens Axboe
2018-02-05 12:54 ` Roman Penyaev
2018-02-02 14:08 ` [PATCH 17/24] ibnbd: client: sysfs interface functions Roman Pen
2018-02-02 14:08 ` [PATCH 18/24] ibnbd: server: private header with server structs and functions Roman Pen
2018-02-02 14:08 ` [PATCH 19/24] ibnbd: server: main functionality Roman Pen
2018-02-02 14:09 ` [PATCH 20/24] ibnbd: server: functionality for IO submission to file or block dev Roman Pen
2018-02-02 14:09 ` [PATCH 21/24] ibnbd: server: sysfs interface functions Roman Pen
2018-02-02 14:09 ` [PATCH 22/24] ibnbd: include client and server modules into kernel compilation Roman Pen
2018-02-02 14:09 ` Roman Pen [this message]
2018-02-02 15:55 ` [PATCH 23/24] ibnbd: a bit of documentation Bart Van Assche
2018-02-05 13:03 ` Roman Penyaev
2018-02-05 14:16 ` Sagi Grimberg
2018-02-02 14:09 ` [PATCH 24/24] MAINTAINERS: Add maintainer for IBNBD/IBTRS modules Roman Pen
2018-02-02 16:07 ` [PATCH 00/24] InfiniBand Transport (IBTRS) and Network Block Device (IBNBD) Bart Van Assche
2018-02-02 16:40 ` Doug Ledford
2018-02-05 8:45 ` Jinpu Wang
2018-06-04 12:14 ` Danil Kipnis
2018-02-02 17:05 ` Bart Van Assche
2018-02-05 8:56 ` Jinpu Wang
2018-02-05 11:36 ` Sagi Grimberg
2018-02-05 13:38 ` Danil Kipnis
2018-02-05 14:17 ` Sagi Grimberg
2018-02-05 16:40 ` Danil Kipnis
2018-02-05 18:38 ` Bart Van Assche
2018-02-06 9:44 ` Danil Kipnis
2018-02-06 15:35 ` Bart Van Assche
2018-02-05 16:16 ` Bart Van Assche
2018-02-05 16:36 ` Jinpu Wang
2018-02-07 16:35 ` Christopher Lameter
2018-02-07 17:18 ` Roman Penyaev
2018-02-07 17:32 ` Bart Van Assche
2018-02-08 17:38 ` Danil Kipnis
2018-02-08 18:09 ` Bart Van Assche
2018-06-04 12:27 ` Danil Kipnis
2018-02-05 12:16 ` Sagi Grimberg
2018-02-05 12:30 ` Sagi Grimberg
2018-02-07 13:06 ` Roman Penyaev
2018-02-05 16:58 ` Bart Van Assche
2018-02-05 17:16 ` Roman Penyaev
2018-02-05 17:20 ` Bart Van Assche
2018-02-06 11:47 ` Roman Penyaev
2018-02-06 13:12 ` Roman Penyaev
2018-02-06 16:01 ` Bart Van Assche
2018-02-07 12:57 ` Roman Penyaev
2018-02-07 16:35 ` Bart Van Assche
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180202140904.2017-24-roman.penyaev@profitbricks.com \
--to=roman.penyaev@profitbricks.com \
--cc=axboe@kernel.dk \
--cc=bart.vanassche@sandisk.com \
--cc=danil.kipnis@profitbricks.com \
--cc=hch@infradead.org \
--cc=jinpu.wang@profitbricks.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=ogerlitz@mellanox.com \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).