[PATCH] Docs: ublk: add ublk document

[PATCH] Docs: ublk: add ublk document

[Ming Lei]

ublk document is missed when merging ublk driver, so add it now.

Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Richard W.M. Jones <rjones@redhat.com>
Cc: ZiyangZhang <ZiyangZhang@linux.alibaba.com>
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Cc: Xiaoguang Wang <xiaoguang.wang@linux.alibaba.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
 Documentation/block/ublk.rst | 203 +++++++++++++++++++++++++++++++++++
 1 file changed, 203 insertions(+)
 create mode 100644 Documentation/block/ublk.rst

diff --git a/Documentation/block/ublk.rst b/Documentation/block/ublk.rst
new file mode 100644
index 000000000000..9e8f7ba518a3
--- /dev/null
+++ b/Documentation/block/ublk.rst
@@ -0,0 +1,203 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==========================================
+Userspace block device driver(ublk driver)
+==========================================
+
+Overview
+========
+
+ublk is one generic framework for implementing block device logic from
+userspace. It is very helpful to move virtual block drivers into userspace,
+such as loop, nbd and similar block drivers. It can help to implement new
+virtual block device, such as ublk-qcow2, and there was several attempts
+of implementing qcow2 driver in kernel.
+
+ublk block device(``/dev/ublkb*``) is added by ublk driver. Any IO request
+submitted to ublk device will be forwarded to ublk's userspace part(
+ublksrv [1]), and after the IO is handled by ublksrv, the result is
+committed back to ublk driver, then ublk IO request can be completed. With
+this way, any specific IO handling logic is totally done inside ublksrv,
+and ublk driver doe _not_ handle any device specific IO logic, such as
+loop's IO handling, NBD's IO communication, or qcow2's IO mapping, ...
+
+/dev/ublkbN is driven by blk-mq request based driver, each request is
+assigned by one queue wide unique tag. ublksrv assigns unique tag to each
+IO too, which is 1:1 mapped with IO of /dev/ublkb*.
+
+Both the IO request forward and IO handling result committing are done via
+io_uring passthrough command, that is why ublk is also one io_uring based
+block driver. It has been observed that io_uring passthrough command can get
+better IOPS than block IO. So ublk is one high performance implementation
+of userspace block device. Not only IO request communication is done by
+io_uring, but also the preferred IO handling in ublksrv is io_uring based
+approach too.
+
+ublk provides control interface to set/get ublk block device parameters, and
+the interface is extendable and kabi compatible, so basically any ublk request
+queue's parameter or ublk generic feature parameters can be set/get via this
+extendable interface. So ublk is generic userspace block device framework, such
+as, it is easy to setup one ublk device with specified block parameters from
+userspace.
+
+How to use ublk
+===============
+
+After building ublksrv[1], ublk block device(``/dev/ublkb*``) can be added
+and deleted by the utility, then existed block IO applications can talk with
+it.
+
+See usage details in README[2] of ublksrv, for example of ublk-loop:
+
+- add ublk device:
+  ublk add -t loop -f ublk-loop.img
+
+- use it:
+  mkfs.xfs /dev/ublkb0
+  mount /dev/ublkb0 /mnt
+  ....                     # all IOs are handled by io_uring!!!
+  umount /mnt
+
+- get ublk dev info:
+  ublk list
+
+- delete ublk device
+  ublk del -a
+  ublk del -n $ublk_dev_id
+
+Design
+======
+
+Control plane
+-------------
+
+ublk driver provides global misc device node(``/dev/ublk-control``) for
+managing and controlling ublk devices with help of several control commands:
+
+- UBLK_CMD_ADD_DEV
+  Add one ublk char device(``/dev/ublkc*``) which is talked with ublksrv wrt.
+  IO command communication. Basic device info is sent together with this
+  command, see UAPI structure of ublksrv_ctrl_dev_info, such as nr_hw_queues,
+  queue_depth, and max IO request buffer size, which info is negotiated with
+  ublk driver and sent back to ublksrv. After this command is completed, the
+  basic device info can't be changed any more.
+
+- UBLK_CMD_SET_PARAMS / UBLK_CMD_GET_PARAMS
+  Set or get ublk device's parameters, which can be generic feature related,
+  or request queue limit related, but can't be IO logic specific, cause ublk
+  driver does not handle any IO logic. This command has to be sent before
+  sending UBLK_CMD_START_DEV.
+
+- UBLK_CMD_START_DEV
+  After ublksrv prepares userspace resource such as, creating per-queue
+  pthread & io_ruing for handling ublk IO, this command is set for ublk
+  driver to allocate & expose /dev/ublkb*. Parameters set via
+  UBLK_CMD_SET_PARAMS are applied for creating /dev/ublkb*.
+
+- UBLK_CMD_STOP_DEV
+  Quiesce IO on /dev/ublkb* and delete the disk. After this command returns,
+  ublksrv can release resource, such as destroy per-queue pthread & io_uring
+  for handling io command.
+
+- UBLK_CMD_DEL_DEV
+  Delete /dev/ublkc*. After this command returns, the allocated ublk device
+  number can be reused.
+
+- UBLK_CMD_GET_QUEUE_AFFINITY
+  After /dev/ublkc is added, ublk driver creates block layer tagset, so each
+  queue's affinity info is available, ublksrv sends UBLK_CMD_GET_QUEUE_AFFINITY
+  to retrieve queue affinity info, so ublksrv can setup the per-queue context
+  efficiently, such as bind affine CPUs with IO pthread, and try to allocate
+  buffers in IO thread context.
+
+- UBLK_CMD_GET_DEV_INFO
+  For retrieve device info of ublksrv_ctrl_dev_info. And it is ublksrv's
+  responsibility to save IO target specific info in userspace.
+
+Data plane
+----------
+
+ublksrv needs to create per-queue IO pthread & io_uring for handling IO
+command (io_uring passthrough command), and the per-queue IO pthread
+focuses on IO handling and shouldn't handle any control & management
+task.
+
+ublksrv's IO is assigned by one unique tag, which is 1:1 mapping with IO
+request of /dev/ublkb*.
+
+UAPI structure of ublksrv_io_desc is defined for describing each IO from
+ublk driver. One fixed mmaped area(array) on /dev/ublkc* is provided for
+exporting IO info to ublksrv, such as IO offset, length, OP/flags and
+buffer address. Each ublksrv_io_desc instance can be indexed via queue id
+and IO tag directly.
+
+Following IO commands are communicated via io_uring passthrough command,
+and each command is only for forwarding ublk IO and committing IO result
+with specified IO tag in the command data:
+
+- UBLK_IO_FETCH_REQ
+  Sent from ublksrv IO pthread for fetching future coming IO request
+  issued to /dev/ublkb*. This command is just sent once from ublksrv IO
+  pthread for ublk driver to setup IO forward environment.
+
+- UBLK_IO_COMMIT_AND_FETCH_REQ
+  After one IO request is issued to /dev/ublkb*, ublk driver stores this
+  IO's ublksrv_io_desc to the specified mapped area, then the previous
+  received IO command of this IO tag, either UBLK_IO_FETCH_REQ or
+  UBLK_IO_COMMIT_AND_FETCH_REQ, is completed, so ulksrv gets the IO
+  notification via io_uring.
+
+  After ublksrv handles this IO, this IO's result is committed back to ublk
+  driver by sending UBLK_IO_COMMIT_AND_FETCH_REQ back. Once ublkdrv received
+  this command, it parses the IO result and complete the IO request to
+  /dev/ublkb*. Meantime setup environment for fetching future IO request
+  with this IO tag. So UBLK_IO_COMMIT_AND_FETCH_REQ is reused for both
+  fetching request and committing back IO result.
+
+- UBLK_IO_NEED_GET_DATA
+  ublksrv pre-allocates IO buffer for each IO at default, any new project
+  should use this IO buffer to communicate with ublk driver. But existed
+  project may not work or be changed to in this way, so add this command
+  to provide chance for userspace to use its existed buffer for handling
+  IO.
+
+- data copy between ublkserv IO buffer and ublk block IO request
+  ublk driver needs to copy ublk block IO request pages into ublksrv buffer
+  (pages) first for WRITE before notifying ublksrv of the coming IO, so
+  ublksrv can hanldle WRITE request.
+
+  After ublksrv handles READ request and sends UBLK_IO_COMMIT_AND_FETCH_REQ
+  to ublksrv, ublkdrv needs to copy read ublksrv buffer(pages) to the ublk
+  IO request pages.
+
+Future development
+==================
+
+Container-ware ublk deivice
+---------------------------
+
+ublk driver doesn't handle any IO logic, and its function is well defined
+so far, and very limited userspace interfaces are needed, and each one is
+well defined too, then it is very likely to make ublk device one
+container-ware block device in future, as Stefan Hajnoczi suggested[3], by
+removing ADMIN privilege.
+
+Zero copy
+---------
+
+Wrt. zero copy support, it is one generic requirement for nbd, fuse or
+similar drivers, one problem Xiaoguang mentioned is that pages mapped to
+userspace can't be remapped any more in kernel with existed mm interfaces,
+and it can be involved when submitting direct IO to /dev/ublkb*. Also
+Xiaoguang reported that big request may benefit from zero copy a lot,
+such as >= 256KB IO.
+
+
+References
+==========
+
+[1] https://github.com/ming1/ubdsrv
+
+[2] https://github.com/ming1/ubdsrv/blob/master/README
+
+[3] https://lore.kernel.org/linux-block/YoOr6jBfgVm8GvWg@stefanha-x1.localdomain/
-- 
2.31.1

Re: [PATCH] Docs: ublk: add ublk document

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 25928 bytes --]

On Sun, Aug 28, 2022 at 12:50:03PM +0800, Ming Lei wrote:
> ublk document is missed when merging ublk driver, so add it now.
> 

Better say "Add documentation for ublk subsystem. It was supposed to be
documented when merging the driver, but missing at that time."

> diff --git a/Documentation/block/ublk.rst b/Documentation/block/ublk.rst
> new file mode 100644
> index 000000000000..9e8f7ba518a3
> --- /dev/null
> +++ b/Documentation/block/ublk.rst
> @@ -0,0 +1,203 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +==========================================
> +Userspace block device driver(ublk driver)
> +==========================================
> +
> +Overview
> +========
> +
> +ublk is one generic framework for implementing block device logic from
> +userspace. It is very helpful to move virtual block drivers into userspace,
> +such as loop, nbd and similar block drivers. It can help to implement new
> +virtual block device, such as ublk-qcow2, and there was several attempts
> +of implementing qcow2 driver in kernel.
> +
> +ublk block device(``/dev/ublkb*``) is added by ublk driver. Any IO request
> +submitted to ublk device will be forwarded to ublk's userspace part(
> +ublksrv [1]), and after the IO is handled by ublksrv, the result is
> +committed back to ublk driver, then ublk IO request can be completed. With
> +this way, any specific IO handling logic is totally done inside ublksrv,
> +and ublk driver doe _not_ handle any device specific IO logic, such as
> +loop's IO handling, NBD's IO communication, or qcow2's IO mapping, ...
> +
> +/dev/ublkbN is driven by blk-mq request based driver, each request is
> +assigned by one queue wide unique tag. ublksrv assigns unique tag to each
> +IO too, which is 1:1 mapped with IO of /dev/ublkb*.
> +
> +Both the IO request forward and IO handling result committing are done via
> +io_uring passthrough command, that is why ublk is also one io_uring based
> +block driver. It has been observed that io_uring passthrough command can get
> +better IOPS than block IO. So ublk is one high performance implementation
> +of userspace block device. Not only IO request communication is done by
> +io_uring, but also the preferred IO handling in ublksrv is io_uring based
> +approach too.
> +
> +ublk provides control interface to set/get ublk block device parameters, and
> +the interface is extendable and kabi compatible, so basically any ublk request
> +queue's parameter or ublk generic feature parameters can be set/get via this
> +extendable interface. So ublk is generic userspace block device framework, such
> +as, it is easy to setup one ublk device with specified block parameters from
> +userspace.
> +
> +How to use ublk
> +===============
> +
> +After building ublksrv[1], ublk block device(``/dev/ublkb*``) can be added
> +and deleted by the utility, then existed block IO applications can talk with
> +it.
> +
> +See usage details in README[2] of ublksrv, for example of ublk-loop:
> +
> +- add ublk device:
> +  ublk add -t loop -f ublk-loop.img
> +
> +- use it:
> +  mkfs.xfs /dev/ublkb0
> +  mount /dev/ublkb0 /mnt
> +  ....                     # all IOs are handled by io_uring!!!
> +  umount /mnt
> +
> +- get ublk dev info:
> +  ublk list
> +
> +- delete ublk device
> +  ublk del -a
> +  ublk del -n $ublk_dev_id
> +
> +Design
> +======
> +
> +Control plane
> +-------------
> +
> +ublk driver provides global misc device node(``/dev/ublk-control``) for
> +managing and controlling ublk devices with help of several control commands:
> +
> +- UBLK_CMD_ADD_DEV
> +  Add one ublk char device(``/dev/ublkc*``) which is talked with ublksrv wrt.
> +  IO command communication. Basic device info is sent together with this
> +  command, see UAPI structure of ublksrv_ctrl_dev_info, such as nr_hw_queues,
> +  queue_depth, and max IO request buffer size, which info is negotiated with
> +  ublk driver and sent back to ublksrv. After this command is completed, the
> +  basic device info can't be changed any more.
> +
> +- UBLK_CMD_SET_PARAMS / UBLK_CMD_GET_PARAMS
> +  Set or get ublk device's parameters, which can be generic feature related,
> +  or request queue limit related, but can't be IO logic specific, cause ublk
> +  driver does not handle any IO logic. This command has to be sent before
> +  sending UBLK_CMD_START_DEV.
> +
> +- UBLK_CMD_START_DEV
> +  After ublksrv prepares userspace resource such as, creating per-queue
> +  pthread & io_ruing for handling ublk IO, this command is set for ublk
> +  driver to allocate & expose /dev/ublkb*. Parameters set via
> +  UBLK_CMD_SET_PARAMS are applied for creating /dev/ublkb*.
> +
> +- UBLK_CMD_STOP_DEV
> +  Quiesce IO on /dev/ublkb* and delete the disk. After this command returns,
> +  ublksrv can release resource, such as destroy per-queue pthread & io_uring
> +  for handling io command.
> +
> +- UBLK_CMD_DEL_DEV
> +  Delete /dev/ublkc*. After this command returns, the allocated ublk device
> +  number can be reused.
> +
> +- UBLK_CMD_GET_QUEUE_AFFINITY
> +  After /dev/ublkc is added, ublk driver creates block layer tagset, so each
> +  queue's affinity info is available, ublksrv sends UBLK_CMD_GET_QUEUE_AFFINITY
> +  to retrieve queue affinity info, so ublksrv can setup the per-queue context
> +  efficiently, such as bind affine CPUs with IO pthread, and try to allocate
> +  buffers in IO thread context.
> +
> +- UBLK_CMD_GET_DEV_INFO
> +  For retrieve device info of ublksrv_ctrl_dev_info. And it is ublksrv's
> +  responsibility to save IO target specific info in userspace.
> +
> +Data plane
> +----------
> +
> +ublksrv needs to create per-queue IO pthread & io_uring for handling IO
> +command (io_uring passthrough command), and the per-queue IO pthread
> +focuses on IO handling and shouldn't handle any control & management
> +task.
> +
> +ublksrv's IO is assigned by one unique tag, which is 1:1 mapping with IO
> +request of /dev/ublkb*.
> +
> +UAPI structure of ublksrv_io_desc is defined for describing each IO from
> +ublk driver. One fixed mmaped area(array) on /dev/ublkc* is provided for
> +exporting IO info to ublksrv, such as IO offset, length, OP/flags and
> +buffer address. Each ublksrv_io_desc instance can be indexed via queue id
> +and IO tag directly.
> +
> +Following IO commands are communicated via io_uring passthrough command,
> +and each command is only for forwarding ublk IO and committing IO result
> +with specified IO tag in the command data:
> +
> +- UBLK_IO_FETCH_REQ
> +  Sent from ublksrv IO pthread for fetching future coming IO request
> +  issued to /dev/ublkb*. This command is just sent once from ublksrv IO
> +  pthread for ublk driver to setup IO forward environment.
> +
> +- UBLK_IO_COMMIT_AND_FETCH_REQ
> +  After one IO request is issued to /dev/ublkb*, ublk driver stores this
> +  IO's ublksrv_io_desc to the specified mapped area, then the previous
> +  received IO command of this IO tag, either UBLK_IO_FETCH_REQ or
> +  UBLK_IO_COMMIT_AND_FETCH_REQ, is completed, so ulksrv gets the IO
> +  notification via io_uring.
> +
> +  After ublksrv handles this IO, this IO's result is committed back to ublk
> +  driver by sending UBLK_IO_COMMIT_AND_FETCH_REQ back. Once ublkdrv received
> +  this command, it parses the IO result and complete the IO request to
> +  /dev/ublkb*. Meantime setup environment for fetching future IO request
> +  with this IO tag. So UBLK_IO_COMMIT_AND_FETCH_REQ is reused for both
> +  fetching request and committing back IO result.
> +
> +- UBLK_IO_NEED_GET_DATA
> +  ublksrv pre-allocates IO buffer for each IO at default, any new project
> +  should use this IO buffer to communicate with ublk driver. But existed
> +  project may not work or be changed to in this way, so add this command
> +  to provide chance for userspace to use its existed buffer for handling
> +  IO.
> +
> +- data copy between ublkserv IO buffer and ublk block IO request
> +  ublk driver needs to copy ublk block IO request pages into ublksrv buffer
> +  (pages) first for WRITE before notifying ublksrv of the coming IO, so
> +  ublksrv can hanldle WRITE request.
> +
> +  After ublksrv handles READ request and sends UBLK_IO_COMMIT_AND_FETCH_REQ
> +  to ublksrv, ublkdrv needs to copy read ublksrv buffer(pages) to the ublk
> +  IO request pages.
> +
> +Future development
> +==================
> +
> +Container-ware ublk deivice
> +---------------------------
> +
> +ublk driver doesn't handle any IO logic, and its function is well defined
> +so far, and very limited userspace interfaces are needed, and each one is
> +well defined too, then it is very likely to make ublk device one
> +container-ware block device in future, as Stefan Hajnoczi suggested[3], by
> +removing ADMIN privilege.
> +
> +Zero copy
> +---------
> +
> +Wrt. zero copy support, it is one generic requirement for nbd, fuse or
> +similar drivers, one problem Xiaoguang mentioned is that pages mapped to
> +userspace can't be remapped any more in kernel with existed mm interfaces,
> +and it can be involved when submitting direct IO to /dev/ublkb*. Also
> +Xiaoguang reported that big request may benefit from zero copy a lot,
> +such as >= 256KB IO.
> +

At what thread on lore the problem is mentioned?

> +
> +References
> +==========
> +
> +[1] https://github.com/ming1/ubdsrv
> +
> +[2] https://github.com/ming1/ubdsrv/blob/master/README
> +
> +[3] https://lore.kernel.org/linux-block/YoOr6jBfgVm8GvWg@stefanha-x1.localdomain/

The documentation can be improved:

  - Its slug should be added to table of contents (index.rst)
  - Use footnote syntax for external references
  - The wordings are weird, almost to the point that I can't comprehend
    it (barely understand the meaning).

Here's the improv:

---- >8 ----

diff --git a/Documentation/block/index.rst b/Documentation/block/index.rst
index 68f115f2b1c6bf..c4c73db748a81f 100644
--- a/Documentation/block/index.rst
+++ b/Documentation/block/index.rst
@@ -23,3 +23,4 @@ Block
    stat
    switching-sched
    writeback_cache_control
+   ublk
diff --git a/Documentation/block/ublk.rst b/Documentation/block/ublk.rst
index 9e8f7ba518a3bb..7262c8b198074b 100644
--- a/Documentation/block/ublk.rst
+++ b/Documentation/block/ublk.rst
@@ -14,56 +14,63 @@ virtual block device, such as ublk-qcow2, and there was several attempts
 of implementing qcow2 driver in kernel.
 
 ublk block device(``/dev/ublkb*``) is added by ublk driver. Any IO request
-submitted to ublk device will be forwarded to ublk's userspace part(
-ublksrv [1]), and after the IO is handled by ublksrv, the result is
-committed back to ublk driver, then ublk IO request can be completed. With
-this way, any specific IO handling logic is totally done inside ublksrv,
-and ublk driver doe _not_ handle any device specific IO logic, such as
-loop's IO handling, NBD's IO communication, or qcow2's IO mapping, ...
+on the device will be forwarded to the userspace program ``ublksrv``
+[#userspace]_.
+After the IO is handled by userspace, the result is committed back to the
+driver, thus completing the request cycle. This way, any specific IO handling
+logic is totally done by userspace, such as
+loop's IO handling, NBD's IO communication, or qcow2's IO mapping.
 
-/dev/ublkbN is driven by blk-mq request based driver, each request is
+``/dev/ublkbN`` is driven by blk-mq request-based driver. Each request is
 assigned by one queue wide unique tag. ublksrv assigns unique tag to each
-IO too, which is 1:1 mapped with IO of /dev/ublkb*.
+IO too, which is 1:1 mapped with IO of ``/dev/ublkb*``.
 
 Both the IO request forward and IO handling result committing are done via
-io_uring passthrough command, that is why ublk is also one io_uring based
-block driver. It has been observed that io_uring passthrough command can get
-better IOPS than block IO. So ublk is one high performance implementation
-of userspace block device. Not only IO request communication is done by
-io_uring, but also the preferred IO handling in ublksrv is io_uring based
-approach too.
+``io_uring`` passthrough command; that is why ublk is also one io_uring based
+block driver. It has been observed that using io_uring passthrough command can
+give better IOPS than block IO; which is why ublk is one of high performance
+implementation of userspace block device: not only IO request communication is
+done by io_uring, but also the preferred IO handling in ublksrv is io_uring
+based approach too.
 
-ublk provides control interface to set/get ublk block device parameters, and
-the interface is extendable and kabi compatible, so basically any ublk request
+ublk provides control interface to set/get ublk block device parameters.
+The interface is extendable and kabi compatible: basically any ublk request
 queue's parameter or ublk generic feature parameters can be set/get via this
-extendable interface. So ublk is generic userspace block device framework, such
-as, it is easy to setup one ublk device with specified block parameters from
-userspace.
+extendable interface. Thus ublk is generic userspace block device framework.
+For example, it is easy to setup one ublk device with specified block
+parameters from userspace.
 
-How to use ublk
-===============
+Using ublk
+==========
 
-After building ublksrv[1], ublk block device(``/dev/ublkb*``) can be added
-and deleted by the utility, then existed block IO applications can talk with
-it.
+ublk requires ublksrv [#userspace]_ to be installed. The userspace program manages ublk
+block devices (``/dev/ublkb*``), while block IO applications can talk with
+them.
 
-See usage details in README[2] of ublksrv, for example of ublk-loop:
+See usage details in README of ublksrv [#userspace_readme]_.
 
-- add ublk device:
-  ublk add -t loop -f ublk-loop.img
+Below is example of using ublk as loop device.
 
-- use it:
-  mkfs.xfs /dev/ublkb0
-  mount /dev/ublkb0 /mnt
-  ....                     # all IOs are handled by io_uring!!!
-  umount /mnt
+- add ublk device::
 
-- get ublk dev info:
-  ublk list
+     ublk add -t loop -f ublk-loop.img
 
-- delete ublk device
-  ublk del -a
-  ublk del -n $ublk_dev_id
+- format with xfs, then use it::
+
+     mkfs.xfs /dev/ublkb0
+     mount /dev/ublkb0 /mnt
+     # do anything. all IOs are handled by io_uring
+     ...
+     umount /mnt
+
+- get ublk dev info::
+
+     ublk list
+
+- delete ublk device::
+
+     ublk del -a
+     ublk del -n $ublk_dev_id
 
 Design
 ======
@@ -74,130 +81,143 @@ Control plane
 ublk driver provides global misc device node(``/dev/ublk-control``) for
 managing and controlling ublk devices with help of several control commands:
 
-- UBLK_CMD_ADD_DEV
-  Add one ublk char device(``/dev/ublkc*``) which is talked with ublksrv wrt.
-  IO command communication. Basic device info is sent together with this
-  command, see UAPI structure of ublksrv_ctrl_dev_info, such as nr_hw_queues,
-  queue_depth, and max IO request buffer size, which info is negotiated with
-  ublk driver and sent back to ublksrv. After this command is completed, the
-  basic device info can't be changed any more.
+- ``UBLK_CMD_ADD_DEV``
 
-- UBLK_CMD_SET_PARAMS / UBLK_CMD_GET_PARAMS
-  Set or get ublk device's parameters, which can be generic feature related,
-  or request queue limit related, but can't be IO logic specific, cause ublk
-  driver does not handle any IO logic. This command has to be sent before
-  sending UBLK_CMD_START_DEV.
+  Add one ublk char device(``/dev/ublkc*``) which is talked with ublksrv
+  WRT IO command communication. Basic device info is sent together with this
+  command. It sets UAPI structure of ``ublksrv_ctrl_dev_info``,
+  such as ``nr_hw_queues``, ``queue_depth``, and max IO request buffer size,
+  for which the info is negotiated with ublk driver and sent back to ublksrv.
+  After this command is completed, the basic device info is immutable.
 
-- UBLK_CMD_START_DEV
-  After ublksrv prepares userspace resource such as, creating per-queue
-  pthread & io_ruing for handling ublk IO, this command is set for ublk
-  driver to allocate & expose /dev/ublkb*. Parameters set via
-  UBLK_CMD_SET_PARAMS are applied for creating /dev/ublkb*.
+- ``UBLK_CMD_SET_PARAMS`` / ``UBLK_CMD_GET_PARAMS``
 
-- UBLK_CMD_STOP_DEV
-  Quiesce IO on /dev/ublkb* and delete the disk. After this command returns,
-  ublksrv can release resource, such as destroy per-queue pthread & io_uring
-  for handling io command.
+  Set or get ublk device's parameters, which can be either generic feature
+  related, or request queue limit related, but can't be IO logic specific,
+  because ublk driver does not handle any IO logic. This command has to be
+  sent before sending ``UBLK_CMD_START_DEV``.
 
-- UBLK_CMD_DEL_DEV
-  Delete /dev/ublkc*. After this command returns, the allocated ublk device
+- ``UBLK_CMD_START_DEV``
+
+  After ublksrv prepares userspace resources (such as creating per-queue
+  pthread & io_uring for handling ublk IO), this command is set for ublk
+  driver to allocate & expose ``/dev/ublkb*``. Parameters set via
+  ``UBLK_CMD_SET_PARAMS`` are applied for creating the device.
+
+- ``UBLK_CMD_STOP_DEV``
+
+  Halt IO on ``/dev/ublkb*`` and remove the device. When this command returns,
+  ublksrv will release resources (such as destroying per-queue pthread &
+  io_uring).
+
+- ``UBLK_CMD_DEL_DEV``
+
+  Remove ``/dev/ublkc*``. When this command returns, the allocated ublk device
   number can be reused.
 
-- UBLK_CMD_GET_QUEUE_AFFINITY
-  After /dev/ublkc is added, ublk driver creates block layer tagset, so each
-  queue's affinity info is available, ublksrv sends UBLK_CMD_GET_QUEUE_AFFINITY
-  to retrieve queue affinity info, so ublksrv can setup the per-queue context
-  efficiently, such as bind affine CPUs with IO pthread, and try to allocate
+- ``UBLK_CMD_GET_QUEUE_AFFINITY``
+
+  When ``/dev/ublkc`` is added, ublk driver creates block layer tagset, so
+  that each
+  queue's affinity info is available. ublksrv sends
+  ``UBLK_CMD_GET_QUEUE_AFFINITY``
+  to retrieve queue affinity info. It can setup the per-queue context
+  efficiently, such as bind affine CPUs with IO pthread and try to allocate
   buffers in IO thread context.
 
-- UBLK_CMD_GET_DEV_INFO
-  For retrieve device info of ublksrv_ctrl_dev_info. And it is ublksrv's
+- ``UBLK_CMD_GET_DEV_INFO``
+
+  For retrieving device info via ``ublksrv_ctrl_dev_info``. It is ublksrv's
   responsibility to save IO target specific info in userspace.
 
 Data plane
 ----------
 
 ublksrv needs to create per-queue IO pthread & io_uring for handling IO
-command (io_uring passthrough command), and the per-queue IO pthread
+commands via io_uring passthrough. The per-queue IO pthread
 focuses on IO handling and shouldn't handle any control & management
-task.
+tasks.
 
-ublksrv's IO is assigned by one unique tag, which is 1:1 mapping with IO
-request of /dev/ublkb*.
+ublksrv's IO is assigned by a unique tag, which is 1:1 mapping with IO
+request of ``/dev/ublkb*``.
 
-UAPI structure of ublksrv_io_desc is defined for describing each IO from
-ublk driver. One fixed mmaped area(array) on /dev/ublkc* is provided for
-exporting IO info to ublksrv, such as IO offset, length, OP/flags and
-buffer address. Each ublksrv_io_desc instance can be indexed via queue id
+UAPI structure of ``ublksrv_io_desc`` is defined for describing each IO from
+ublk driver. A fixed mmaped area (array) on ``/dev/ublkc*`` is provided for
+exporting IO info to ublksrv; such as IO offset, length, OP/flags and
+buffer address. Each ``ublksrv_io_desc`` instance can be indexed via queue id
 and IO tag directly.
 
-Following IO commands are communicated via io_uring passthrough command,
+The following IO commands are communicated via io_uring passthrough command,
 and each command is only for forwarding ublk IO and committing IO result
 with specified IO tag in the command data:
 
-- UBLK_IO_FETCH_REQ
-  Sent from ublksrv IO pthread for fetching future coming IO request
-  issued to /dev/ublkb*. This command is just sent once from ublksrv IO
+- ``UBLK_IO_FETCH_REQ``
+
+  Sent from ublksrv IO pthread for fetching future incoming IO requests
+  destined to ``/dev/ublkb*``. This command is sent only once from ublksrv IO
   pthread for ublk driver to setup IO forward environment.
 
-- UBLK_IO_COMMIT_AND_FETCH_REQ
-  After one IO request is issued to /dev/ublkb*, ublk driver stores this
-  IO's ublksrv_io_desc to the specified mapped area, then the previous
-  received IO command of this IO tag, either UBLK_IO_FETCH_REQ or
-  UBLK_IO_COMMIT_AND_FETCH_REQ, is completed, so ulksrv gets the IO
+- ``UBLK_IO_COMMIT_AND_FETCH_REQ``
+
+  When an IO request is destined to ``/dev/ublkb*``, ublk driver stores
+  the IO's ``ublksrv_io_desc`` to the specified mapped area; then the
+  previous received IO command of this IO tag (either UBLK_IO_FETCH_REQ or
+  UBLK_IO_COMMIT_AND_FETCH_REQ) is completed, so ublksrv gets the IO
   notification via io_uring.
 
-  After ublksrv handles this IO, this IO's result is committed back to ublk
-  driver by sending UBLK_IO_COMMIT_AND_FETCH_REQ back. Once ublkdrv received
-  this command, it parses the IO result and complete the IO request to
-  /dev/ublkb*. Meantime setup environment for fetching future IO request
-  with this IO tag. So UBLK_IO_COMMIT_AND_FETCH_REQ is reused for both
-  fetching request and committing back IO result.
+  After ublksrv handles the IO, its result is committed back to ublk
+  driver by sending ``UBLK_IO_COMMIT_AND_FETCH_REQ`` back. Once ublkdrv
+  received this command, it parses the result and complete the request to
+  ``/dev/ublkb*``. In the meantime setup environment for fetching future
+  requests with the same IO tag. That is, ``UBLK_IO_COMMIT_AND_FETCH_REQ``
+  is reused for both fetching request and committing back IO result.
 
-- UBLK_IO_NEED_GET_DATA
-  ublksrv pre-allocates IO buffer for each IO at default, any new project
-  should use this IO buffer to communicate with ublk driver. But existed
-  project may not work or be changed to in this way, so add this command
-  to provide chance for userspace to use its existed buffer for handling
-  IO.
+- ``UBLK_IO_NEED_GET_DATA``
+
+  ublksrv pre-allocates IO buffer for each IO by default. Any new projects
+  should use this buffer to communicate with ublk driver. However,
+  existing
+  projects may break or not able to consume the new buffer interface; that's
+  why this command is added for backwards compatibility so that existing
+  projects can still consume existing buffers.
 
 - data copy between ublkserv IO buffer and ublk block IO request
-  ublk driver needs to copy ublk block IO request pages into ublksrv buffer
-  (pages) first for WRITE before notifying ublksrv of the coming IO, so
-  ublksrv can hanldle WRITE request.
 
-  After ublksrv handles READ request and sends UBLK_IO_COMMIT_AND_FETCH_REQ
-  to ublksrv, ublkdrv needs to copy read ublksrv buffer(pages) to the ublk
-  IO request pages.
+  ublk driver needs to copy the block IO request pages into ublksrv buffer
+  (pages) first for WRITE before notifying ublksrv of the coming IO, so
+  that ublksrv can hanldle WRITE request.
+
+  When ublksrv handles READ request and sends ``UBLK_IO_COMMIT_AND_FETCH_REQ``
+  to ublksrv, ublkdrv needs to copy read ublksrv buffer (pages) to the IO
+  request pages.
 
 Future development
 ==================
 
-Container-ware ublk deivice
----------------------------
+Container-aware ublk deivice
+----------------------------
 
-ublk driver doesn't handle any IO logic, and its function is well defined
-so far, and very limited userspace interfaces are needed, and each one is
-well defined too, then it is very likely to make ublk device one
-container-ware block device in future, as Stefan Hajnoczi suggested[3], by
-removing ADMIN privilege.
+ublk driver doesn't handle any IO logic. Its function is well defined
+for now, and very limited userspace interfaces are needed, which is also
+well defined too. It is possible to make ublk devices container-aware block
+devices in future as Stefan Hajnoczi suggested [#stefan]_, by removing
+ADMIN privilege.
 
 Zero copy
 ---------
 
-Wrt. zero copy support, it is one generic requirement for nbd, fuse or
-similar drivers, one problem Xiaoguang mentioned is that pages mapped to
-userspace can't be remapped any more in kernel with existed mm interfaces,
-and it can be involved when submitting direct IO to /dev/ublkb*. Also
-Xiaoguang reported that big request may benefit from zero copy a lot,
-such as >= 256KB IO.
+Wrt. zero copy support, which is a generic requirement for nbd, fuse or
+similar drivers, a problem Xiaoguang mentioned is that pages mapped to
+userspace can't be remapped any more in kernel with existing mm interfaces.
+This can occurs when destining direct IO to ``/dev/ublkb*``. Also
+he reported that big requests (>= 256 KB IO) may benefit a lot from zero copy.
 
 
 References
 ==========
 
-[1] https://github.com/ming1/ubdsrv
+.. [#userspace] https://github.com/ming1/ubdsrv
 
-[2] https://github.com/ming1/ubdsrv/blob/master/README
+.. [#userspace_readme] https://github.com/ming1/ubdsrv/blob/master/README
 
-[3] https://lore.kernel.org/linux-block/YoOr6jBfgVm8GvWg@stefanha-x1.localdomain/
+.. [#stefan] https://lore.kernel.org/linux-block/YoOr6jBfgVm8GvWg@stefanha-x1.localdomain/

Thanks.

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH] Docs: ublk: add ublk document

[Richard W.M. Jones]

On Sun, Aug 28, 2022 at 12:50:03PM +0800, Ming Lei wrote:
> ublk document is missed when merging ublk driver, so add it now.
> 
> Cc: Jonathan Corbet <corbet@lwn.net>
> Cc: Richard W.M. Jones <rjones@redhat.com>
> Cc: ZiyangZhang <ZiyangZhang@linux.alibaba.com>
> Cc: Stefan Hajnoczi <stefanha@redhat.com>
> Cc: Xiaoguang Wang <xiaoguang.wang@linux.alibaba.com>
> Signed-off-by: Ming Lei <ming.lei@redhat.com>
> ---
>  Documentation/block/ublk.rst | 203 +++++++++++++++++++++++++++++++++++
>  1 file changed, 203 insertions(+)
>  create mode 100644 Documentation/block/ublk.rst

Thanks for preparing this.  As you know I had a lot of trouble writing
the NBD ublk daemon and this would have helped.  TBH I would suggest
anyone trying to write a ublk daemon just looks at this source:
https://gitlab.com/rwmjones/libnbd/-/tree/nbdublk/ublk

> diff --git a/Documentation/block/ublk.rst b/Documentation/block/ublk.rst
> new file mode 100644
> index 000000000000..9e8f7ba518a3
> --- /dev/null
> +++ b/Documentation/block/ublk.rst
> @@ -0,0 +1,203 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +==========================================
> +Userspace block device driver(ublk driver)
> +==========================================
> +
> +Overview
> +========
> +
> +ublk is one generic framework for implementing block device logic from

"one generic framework" - probably better to say "a generic framework ..."

> +userspace. It is very helpful to move virtual block drivers into userspace,
> +such as loop, nbd and similar block drivers. It can help to implement new
> +virtual block device, such as ublk-qcow2, and there was several attempts
> +of implementing qcow2 driver in kernel.

On the general topic of this, the qemu developers would greatly prefer
that there are not multiple qcow2 implementations.  I believe the plan
is to modify qemu-storage-daemon (a daemon containing the qemu block
layer) to implement ublk.  I don't think you really need to mention
qcow2 though since it'll be implemented.

> +ublk block device(``/dev/ublkb*``) is added by ublk driver. Any IO request
> +submitted to ublk device will be forwarded to ublk's userspace part(

Add a space between "part" and "("?

> +ublksrv [1]), and after the IO is handled by ublksrv, the result is
> +committed back to ublk driver, then ublk IO request can be completed. With
> +this way, any specific IO handling logic is totally done inside ublksrv,
> +and ublk driver doe _not_ handle any device specific IO logic, such as

does

> +loop's IO handling, NBD's IO communication, or qcow2's IO mapping, ...
>
> +/dev/ublkbN is driven by blk-mq request based driver, each request is
> +assigned by one queue wide unique tag. ublksrv assigns unique tag to each
> +IO too, which is 1:1 mapped with IO of /dev/ublkb*.
> +
> +Both the IO request forward and IO handling result committing are done via
> +io_uring passthrough command, that is why ublk is also one io_uring based
> +block driver. It has been observed that io_uring passthrough command can get
> +better IOPS than block IO. So ublk is one high performance implementation
> +of userspace block device. Not only IO request communication is done by
> +io_uring, but also the preferred IO handling in ublksrv is io_uring based
> +approach too.
> +
> +ublk provides control interface to set/get ublk block device parameters, and
> +the interface is extendable and kabi compatible, so basically any ublk request
> +queue's parameter or ublk generic feature parameters can be set/get via this
> +extendable interface. So ublk is generic userspace block device framework, such
> +as, it is easy to setup one ublk device with specified block parameters from
> +userspace.
> +
> +How to use ublk
> +===============
> +
> +After building ublksrv[1], ublk block device(``/dev/ublkb*``) can be added
> +and deleted by the utility, then existed block IO applications can talk with

existing

> +it.
> +
> +See usage details in README[2] of ublksrv, for example of ublk-loop:
> +
> +- add ublk device:
> +  ublk add -t loop -f ublk-loop.img
> +
> +- use it:
> +  mkfs.xfs /dev/ublkb0
> +  mount /dev/ublkb0 /mnt
> +  ....                     # all IOs are handled by io_uring!!!
> +  umount /mnt
> +
> +- get ublk dev info:
> +  ublk list
> +
> +- delete ublk device
> +  ublk del -a
> +  ublk del -n $ublk_dev_id
> +
> +Design
> +======
> +
> +Control plane
> +-------------
> +
> +ublk driver provides global misc device node(``/dev/ublk-control``) for

Space between "node" and "(".  There are a few more of these below.

> +managing and controlling ublk devices with help of several control commands:
> +
> +- UBLK_CMD_ADD_DEV
> +  Add one ublk char device(``/dev/ublkc*``) which is talked with ublksrv wrt.
> +  IO command communication. Basic device info is sent together with this
> +  command, see UAPI structure of ublksrv_ctrl_dev_info, such as nr_hw_queues,
> +  queue_depth, and max IO request buffer size, which info is negotiated with
> +  ublk driver and sent back to ublksrv. After this command is completed, the
> +  basic device info can't be changed any more.
> +
> +- UBLK_CMD_SET_PARAMS / UBLK_CMD_GET_PARAMS
> +  Set or get ublk device's parameters, which can be generic feature related,
> +  or request queue limit related, but can't be IO logic specific, cause ublk
> +  driver does not handle any IO logic. This command has to be sent before
> +  sending UBLK_CMD_START_DEV.
> +
> +- UBLK_CMD_START_DEV
> +  After ublksrv prepares userspace resource such as, creating per-queue
> +  pthread & io_ruing for handling ublk IO, this command is set for ublk

set -> sent

> +  driver to allocate & expose /dev/ublkb*. Parameters set via
> +  UBLK_CMD_SET_PARAMS are applied for creating /dev/ublkb*.

Is this command synchronous?  ie. When it completes, is /dev/ublkb*
definitely present in the /dev filesystem?  (I'm going to guess this
depends on something complicated about udevd).

> +- UBLK_CMD_STOP_DEV
> +  Quiesce IO on /dev/ublkb* and delete the disk. After this command returns,
> +  ublksrv can release resource, such as destroy per-queue pthread & io_uring
> +  for handling io command.
> +
> +- UBLK_CMD_DEL_DEV
> +  Delete /dev/ublkc*. After this command returns, the allocated ublk device
> +  number can be reused.
> +
> +- UBLK_CMD_GET_QUEUE_AFFINITY
> +  After /dev/ublkc is added, ublk driver creates block layer tagset, so each
> +  queue's affinity info is available, ublksrv sends UBLK_CMD_GET_QUEUE_AFFINITY
> +  to retrieve queue affinity info, so ublksrv can setup the per-queue context
> +  efficiently, such as bind affine CPUs with IO pthread, and try to allocate
> +  buffers in IO thread context.
> +
> +- UBLK_CMD_GET_DEV_INFO
> +  For retrieve device info of ublksrv_ctrl_dev_info. And it is ublksrv's
> +  responsibility to save IO target specific info in userspace.
> +
> +Data plane
> +----------
> +
> +ublksrv needs to create per-queue IO pthread & io_uring for handling IO
> +command (io_uring passthrough command), and the per-queue IO pthread
> +focuses on IO handling and shouldn't handle any control & management
> +task.
> +
> +ublksrv's IO is assigned by one unique tag, which is 1:1 mapping with IO
> +request of /dev/ublkb*.
> +
> +UAPI structure of ublksrv_io_desc is defined for describing each IO from
> +ublk driver. One fixed mmaped area(array) on /dev/ublkc* is provided for
> +exporting IO info to ublksrv, such as IO offset, length, OP/flags and
> +buffer address. Each ublksrv_io_desc instance can be indexed via queue id
> +and IO tag directly.
> +
> +Following IO commands are communicated via io_uring passthrough command,
> +and each command is only for forwarding ublk IO and committing IO result
> +with specified IO tag in the command data:
> +
> +- UBLK_IO_FETCH_REQ
> +  Sent from ublksrv IO pthread for fetching future coming IO request
> +  issued to /dev/ublkb*. This command is just sent once from ublksrv IO
> +  pthread for ublk driver to setup IO forward environment.
> +
> +- UBLK_IO_COMMIT_AND_FETCH_REQ
> +  After one IO request is issued to /dev/ublkb*, ublk driver stores this
> +  IO's ublksrv_io_desc to the specified mapped area, then the previous
> +  received IO command of this IO tag, either UBLK_IO_FETCH_REQ or
> +  UBLK_IO_COMMIT_AND_FETCH_REQ, is completed, so ulksrv gets the IO
> +  notification via io_uring.
> +
> +  After ublksrv handles this IO, this IO's result is committed back to ublk
> +  driver by sending UBLK_IO_COMMIT_AND_FETCH_REQ back. Once ublkdrv received
> +  this command, it parses the IO result and complete the IO request to
> +  /dev/ublkb*. Meantime setup environment for fetching future IO request
> +  with this IO tag. So UBLK_IO_COMMIT_AND_FETCH_REQ is reused for both
> +  fetching request and committing back IO result.
> +
> +- UBLK_IO_NEED_GET_DATA
> +  ublksrv pre-allocates IO buffer for each IO at default, any new project

at default -> by default

> +  should use this IO buffer to communicate with ublk driver. But existed

existed -> existing

> +  project may not work or be changed to in this way, so add this command
> +  to provide chance for userspace to use its existed buffer for handling

existed -> existing

> +  IO.
> +
> +- data copy between ublkserv IO buffer and ublk block IO request
> +  ublk driver needs to copy ublk block IO request pages into ublksrv buffer
> +  (pages) first for WRITE before notifying ublksrv of the coming IO, so
> +  ublksrv can hanldle WRITE request.
> +
> +  After ublksrv handles READ request and sends UBLK_IO_COMMIT_AND_FETCH_REQ
> +  to ublksrv, ublkdrv needs to copy read ublksrv buffer(pages) to the ublk
> +  IO request pages.
> +
> +Future development
> +==================
> +
> +Container-ware ublk deivice

Container-aware ublk device

> +---------------------------
> +
> +ublk driver doesn't handle any IO logic, and its function is well defined
> +so far, and very limited userspace interfaces are needed, and each one is
> +well defined too, then it is very likely to make ublk device one
> +container-ware block device in future, as Stefan Hajnoczi suggested[3], by
> +removing ADMIN privilege.

Is it advisable for non-root to be able create arbitrary /dev devices?
It sounds like a security nightmare because you're exposing
potentially any arbitrary, malicious filesystem to the kernel to
parse.

> +Zero copy
> +---------
> +
> +Wrt. zero copy support, it is one generic requirement for nbd, fuse or
> +similar drivers, one problem Xiaoguang mentioned is that pages mapped to
> +userspace can't be remapped any more in kernel with existed mm interfaces,
> +and it can be involved when submitting direct IO to /dev/ublkb*. Also
> +Xiaoguang reported that big request may benefit from zero copy a lot,
> +such as >= 256KB IO.
> +
> +
> +References
> +==========
> +
> +[1] https://github.com/ming1/ubdsrv
> +
> +[2] https://github.com/ming1/ubdsrv/blob/master/README
> +
> +[3] https://lore.kernel.org/linux-block/YoOr6jBfgVm8GvWg@stefanha-x1.localdomain/
> -- 
> 2.31.1

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit

Re: [PATCH] Docs: ublk: add ublk document

[Randy Dunlap]

On 8/28/22 05:40, Bagas Sanjaya wrote:
>  Zero copy
>  ---------
>  
> -Wrt. zero copy support, it is one generic requirement for nbd, fuse or
> -similar drivers, one problem Xiaoguang mentioned is that pages mapped to
> -userspace can't be remapped any more in kernel with existed mm interfaces,
> -and it can be involved when submitting direct IO to /dev/ublkb*. Also
> -Xiaoguang reported that big request may benefit from zero copy a lot,
> -such as >= 256KB IO.
> +Wrt. zero copy support, which is a generic requirement for nbd, fuse or
> +similar drivers, a problem Xiaoguang mentioned is that pages mapped to
> +userspace can't be remapped any more in kernel with existing mm interfaces.
> +This can occurs when destining direct IO to ``/dev/ublkb*``. Also
> +he reported that big requests (>= 256 KB IO) may benefit a lot from zero copy.

I suggest eliminating "Wrt." also.

-- 
~Randy

Re: [PATCH] Docs: ublk: add ublk document

[kernel test robot]

Hi Ming,

I love your patch! Perhaps something to improve:

[auto build test WARNING on axboe-block/for-next]
[also build test WARNING on lwn-2.6/docs-next linus/master v6.0-rc2 next-20220826]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Ming-Lei/Docs-ublk-add-ublk-document/20220828-125214
base:   https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git for-next
reproduce:
        # https://github.com/intel-lab-lkp/linux/commit/4cd40d237bac08efb23668766ba39b36a0b1632a
        git remote add linux-review https://github.com/intel-lab-lkp/linux
        git fetch --no-tags linux-review Ming-Lei/Docs-ublk-add-ublk-document/20220828-125214
        git checkout 4cd40d237bac08efb23668766ba39b36a0b1632a
        make menuconfig
        # enable CONFIG_COMPILE_TEST, CONFIG_WARN_MISSING_DOCUMENTS, CONFIG_WARN_ABI_ERRORS
        make htmldocs

If you fix the issue, kindly add following tag where applicable
Reported-by: kernel test robot <lkp@intel.com>

All warnings (new ones prefixed by >>):

>> Documentation/block/ublk.rst: WARNING: document isn't included in any toctree

-- 
0-DAY CI Kernel Test Service
https://01.org/lkp

Re: [PATCH] Docs: ublk: add ublk document

[Gao Xiang]

On Sun, Aug 28, 2022 at 04:09:11PM +0100, Richard W.M. Jones wrote:

...

> > +
> > +ublk driver doesn't handle any IO logic, and its function is well defined
> > +so far, and very limited userspace interfaces are needed, and each one is
> > +well defined too, then it is very likely to make ublk device one
> > +container-ware block device in future, as Stefan Hajnoczi suggested[3], by
> > +removing ADMIN privilege.
> 
> Is it advisable for non-root to be able create arbitrary /dev devices?
> It sounds like a security nightmare because you're exposing
> potentially any arbitrary, malicious filesystem to the kernel to
> parse.

+1, such malicious daemons can also dynamically update/attack fs metadata
runtimely, I think most current fs corruption tests are for pre-built fs
images but not for runtime attack via daemon itself or network,
unprivileged daemon makes all local fses life harder.

Also for swap device use cases, malicious unprivileged daemons enlarge
the possibility of corrupting/attacking any anonymous memory (maybe
belong to privileged processes) on purpose regardless of other concerns.

Thanks,
Gao Xiang

Re: [PATCH] Docs: ublk: add ublk document

[Ming Lei]

On Sun, Aug 28, 2022 at 07:40:46PM +0700, Bagas Sanjaya wrote:
> On Sun, Aug 28, 2022 at 12:50:03PM +0800, Ming Lei wrote:
> > ublk document is missed when merging ublk driver, so add it now.
> > 
> 
> Better say "Add documentation for ublk subsystem. It was supposed to be
> documented when merging the driver, but missing at that time."

OK.

> 
> > diff --git a/Documentation/block/ublk.rst b/Documentation/block/ublk.rst
> > new file mode 100644
> > index 000000000000..9e8f7ba518a3
> > --- /dev/null
> > +++ b/Documentation/block/ublk.rst
> > @@ -0,0 +1,203 @@
> > +.. SPDX-License-Identifier: GPL-2.0
> > +
> > +==========================================
> > +Userspace block device driver(ublk driver)
> > +==========================================
> > +
> > +Overview
> > +========
> > +
> > +ublk is one generic framework for implementing block device logic from
> > +userspace. It is very helpful to move virtual block drivers into userspace,
> > +such as loop, nbd and similar block drivers. It can help to implement new
> > +virtual block device, such as ublk-qcow2, and there was several attempts
> > +of implementing qcow2 driver in kernel.
> > +
> > +ublk block device(``/dev/ublkb*``) is added by ublk driver. Any IO request
> > +submitted to ublk device will be forwarded to ublk's userspace part(
> > +ublksrv [1]), and after the IO is handled by ublksrv, the result is
> > +committed back to ublk driver, then ublk IO request can be completed. With
> > +this way, any specific IO handling logic is totally done inside ublksrv,
> > +and ublk driver doe _not_ handle any device specific IO logic, such as
> > +loop's IO handling, NBD's IO communication, or qcow2's IO mapping, ...
> > +
> > +/dev/ublkbN is driven by blk-mq request based driver, each request is
> > +assigned by one queue wide unique tag. ublksrv assigns unique tag to each
> > +IO too, which is 1:1 mapped with IO of /dev/ublkb*.
> > +
> > +Both the IO request forward and IO handling result committing are done via
> > +io_uring passthrough command, that is why ublk is also one io_uring based
> > +block driver. It has been observed that io_uring passthrough command can get
> > +better IOPS than block IO. So ublk is one high performance implementation
> > +of userspace block device. Not only IO request communication is done by
> > +io_uring, but also the preferred IO handling in ublksrv is io_uring based
> > +approach too.
> > +
> > +ublk provides control interface to set/get ublk block device parameters, and
> > +the interface is extendable and kabi compatible, so basically any ublk request
> > +queue's parameter or ublk generic feature parameters can be set/get via this
> > +extendable interface. So ublk is generic userspace block device framework, such
> > +as, it is easy to setup one ublk device with specified block parameters from
> > +userspace.
> > +
> > +How to use ublk
> > +===============
> > +
> > +After building ublksrv[1], ublk block device(``/dev/ublkb*``) can be added
> > +and deleted by the utility, then existed block IO applications can talk with
> > +it.
> > +
> > +See usage details in README[2] of ublksrv, for example of ublk-loop:
> > +
> > +- add ublk device:
> > +  ublk add -t loop -f ublk-loop.img
> > +
> > +- use it:
> > +  mkfs.xfs /dev/ublkb0
> > +  mount /dev/ublkb0 /mnt
> > +  ....                     # all IOs are handled by io_uring!!!
> > +  umount /mnt
> > +
> > +- get ublk dev info:
> > +  ublk list
> > +
> > +- delete ublk device
> > +  ublk del -a
> > +  ublk del -n $ublk_dev_id
> > +
> > +Design
> > +======
> > +
> > +Control plane
> > +-------------
> > +
> > +ublk driver provides global misc device node(``/dev/ublk-control``) for
> > +managing and controlling ublk devices with help of several control commands:
> > +
> > +- UBLK_CMD_ADD_DEV
> > +  Add one ublk char device(``/dev/ublkc*``) which is talked with ublksrv wrt.
> > +  IO command communication. Basic device info is sent together with this
> > +  command, see UAPI structure of ublksrv_ctrl_dev_info, such as nr_hw_queues,
> > +  queue_depth, and max IO request buffer size, which info is negotiated with
> > +  ublk driver and sent back to ublksrv. After this command is completed, the
> > +  basic device info can't be changed any more.
> > +
> > +- UBLK_CMD_SET_PARAMS / UBLK_CMD_GET_PARAMS
> > +  Set or get ublk device's parameters, which can be generic feature related,
> > +  or request queue limit related, but can't be IO logic specific, cause ublk
> > +  driver does not handle any IO logic. This command has to be sent before
> > +  sending UBLK_CMD_START_DEV.
> > +
> > +- UBLK_CMD_START_DEV
> > +  After ublksrv prepares userspace resource such as, creating per-queue
> > +  pthread & io_ruing for handling ublk IO, this command is set for ublk
> > +  driver to allocate & expose /dev/ublkb*. Parameters set via
> > +  UBLK_CMD_SET_PARAMS are applied for creating /dev/ublkb*.
> > +
> > +- UBLK_CMD_STOP_DEV
> > +  Quiesce IO on /dev/ublkb* and delete the disk. After this command returns,
> > +  ublksrv can release resource, such as destroy per-queue pthread & io_uring
> > +  for handling io command.
> > +
> > +- UBLK_CMD_DEL_DEV
> > +  Delete /dev/ublkc*. After this command returns, the allocated ublk device
> > +  number can be reused.
> > +
> > +- UBLK_CMD_GET_QUEUE_AFFINITY
> > +  After /dev/ublkc is added, ublk driver creates block layer tagset, so each
> > +  queue's affinity info is available, ublksrv sends UBLK_CMD_GET_QUEUE_AFFINITY
> > +  to retrieve queue affinity info, so ublksrv can setup the per-queue context
> > +  efficiently, such as bind affine CPUs with IO pthread, and try to allocate
> > +  buffers in IO thread context.
> > +
> > +- UBLK_CMD_GET_DEV_INFO
> > +  For retrieve device info of ublksrv_ctrl_dev_info. And it is ublksrv's
> > +  responsibility to save IO target specific info in userspace.
> > +
> > +Data plane
> > +----------
> > +
> > +ublksrv needs to create per-queue IO pthread & io_uring for handling IO
> > +command (io_uring passthrough command), and the per-queue IO pthread
> > +focuses on IO handling and shouldn't handle any control & management
> > +task.
> > +
> > +ublksrv's IO is assigned by one unique tag, which is 1:1 mapping with IO
> > +request of /dev/ublkb*.
> > +
> > +UAPI structure of ublksrv_io_desc is defined for describing each IO from
> > +ublk driver. One fixed mmaped area(array) on /dev/ublkc* is provided for
> > +exporting IO info to ublksrv, such as IO offset, length, OP/flags and
> > +buffer address. Each ublksrv_io_desc instance can be indexed via queue id
> > +and IO tag directly.
> > +
> > +Following IO commands are communicated via io_uring passthrough command,
> > +and each command is only for forwarding ublk IO and committing IO result
> > +with specified IO tag in the command data:
> > +
> > +- UBLK_IO_FETCH_REQ
> > +  Sent from ublksrv IO pthread for fetching future coming IO request
> > +  issued to /dev/ublkb*. This command is just sent once from ublksrv IO
> > +  pthread for ublk driver to setup IO forward environment.
> > +
> > +- UBLK_IO_COMMIT_AND_FETCH_REQ
> > +  After one IO request is issued to /dev/ublkb*, ublk driver stores this
> > +  IO's ublksrv_io_desc to the specified mapped area, then the previous
> > +  received IO command of this IO tag, either UBLK_IO_FETCH_REQ or
> > +  UBLK_IO_COMMIT_AND_FETCH_REQ, is completed, so ulksrv gets the IO
> > +  notification via io_uring.
> > +
> > +  After ublksrv handles this IO, this IO's result is committed back to ublk
> > +  driver by sending UBLK_IO_COMMIT_AND_FETCH_REQ back. Once ublkdrv received
> > +  this command, it parses the IO result and complete the IO request to
> > +  /dev/ublkb*. Meantime setup environment for fetching future IO request
> > +  with this IO tag. So UBLK_IO_COMMIT_AND_FETCH_REQ is reused for both
> > +  fetching request and committing back IO result.
> > +
> > +- UBLK_IO_NEED_GET_DATA
> > +  ublksrv pre-allocates IO buffer for each IO at default, any new project
> > +  should use this IO buffer to communicate with ublk driver. But existed
> > +  project may not work or be changed to in this way, so add this command
> > +  to provide chance for userspace to use its existed buffer for handling
> > +  IO.
> > +
> > +- data copy between ublkserv IO buffer and ublk block IO request
> > +  ublk driver needs to copy ublk block IO request pages into ublksrv buffer
> > +  (pages) first for WRITE before notifying ublksrv of the coming IO, so
> > +  ublksrv can hanldle WRITE request.
> > +
> > +  After ublksrv handles READ request and sends UBLK_IO_COMMIT_AND_FETCH_REQ
> > +  to ublksrv, ublkdrv needs to copy read ublksrv buffer(pages) to the ublk
> > +  IO request pages.
> > +
> > +Future development
> > +==================
> > +
> > +Container-ware ublk deivice
> > +---------------------------
> > +
> > +ublk driver doesn't handle any IO logic, and its function is well defined
> > +so far, and very limited userspace interfaces are needed, and each one is
> > +well defined too, then it is very likely to make ublk device one
> > +container-ware block device in future, as Stefan Hajnoczi suggested[3], by
> > +removing ADMIN privilege.
> > +
> > +Zero copy
> > +---------
> > +
> > +Wrt. zero copy support, it is one generic requirement for nbd, fuse or
> > +similar drivers, one problem Xiaoguang mentioned is that pages mapped to
> > +userspace can't be remapped any more in kernel with existed mm interfaces,
> > +and it can be involved when submitting direct IO to /dev/ublkb*. Also
> > +Xiaoguang reported that big request may benefit from zero copy a lot,
> > +such as >= 256KB IO.
> > +
> 
> At what thread on lore the problem is mentioned?

https://lore.kernel.org/linux-mm/20220318095531.15479-1-xiaoguang.wang@linux.alibaba.com/

> 
> > +
> > +References
> > +==========
> > +
> > +[1] https://github.com/ming1/ubdsrv
> > +
> > +[2] https://github.com/ming1/ubdsrv/blob/master/README
> > +
> > +[3] https://lore.kernel.org/linux-block/YoOr6jBfgVm8GvWg@stefanha-x1.localdomain/
> 
> The documentation can be improved:
> 
>   - Its slug should be added to table of contents (index.rst)
>   - Use footnote syntax for external references
>   - The wordings are weird, almost to the point that I can't comprehend
>     it (barely understand the meaning).
> 
> Here's the improv:

OK, I will integrate it in V2.

thanks, 
Ming

Re: [PATCH] Docs: ublk: add ublk document

[Ming Lei]

On Sun, Aug 28, 2022 at 04:09:11PM +0100, Richard W.M. Jones wrote:
> On Sun, Aug 28, 2022 at 12:50:03PM +0800, Ming Lei wrote:
> > ublk document is missed when merging ublk driver, so add it now.
> > 
> > Cc: Jonathan Corbet <corbet@lwn.net>
> > Cc: Richard W.M. Jones <rjones@redhat.com>
> > Cc: ZiyangZhang <ZiyangZhang@linux.alibaba.com>
> > Cc: Stefan Hajnoczi <stefanha@redhat.com>
> > Cc: Xiaoguang Wang <xiaoguang.wang@linux.alibaba.com>
> > Signed-off-by: Ming Lei <ming.lei@redhat.com>
> > ---
> >  Documentation/block/ublk.rst | 203 +++++++++++++++++++++++++++++++++++
> >  1 file changed, 203 insertions(+)
> >  create mode 100644 Documentation/block/ublk.rst
> 
> Thanks for preparing this.  As you know I had a lot of trouble writing
> the NBD ublk daemon and this would have helped.  TBH I would suggest
> anyone trying to write a ublk daemon just looks at this source:
> https://gitlab.com/rwmjones/libnbd/-/tree/nbdublk/ublk

Yeah, now we have the 3rd ublk target example: nbdublk.

Maybe the network IO can be converted into io_uring, and see if perf
can be improved much.

> 
> > diff --git a/Documentation/block/ublk.rst b/Documentation/block/ublk.rst
> > new file mode 100644
> > index 000000000000..9e8f7ba518a3
> > --- /dev/null
> > +++ b/Documentation/block/ublk.rst
> > @@ -0,0 +1,203 @@
> > +.. SPDX-License-Identifier: GPL-2.0
> > +
> > +==========================================
> > +Userspace block device driver(ublk driver)
> > +==========================================
> > +
> > +Overview
> > +========
> > +
> > +ublk is one generic framework for implementing block device logic from
> 
> "one generic framework" - probably better to say "a generic framework ..."

OK.

> 
> > +userspace. It is very helpful to move virtual block drivers into userspace,
> > +such as loop, nbd and similar block drivers. It can help to implement new
> > +virtual block device, such as ublk-qcow2, and there was several attempts
> > +of implementing qcow2 driver in kernel.
> 
> On the general topic of this, the qemu developers would greatly prefer
> that there are not multiple qcow2 implementations.  I believe the plan
> is to modify qemu-storage-daemon (a daemon containing the qemu block
> layer) to implement ublk.  I don't think you really need to mention
> qcow2 though since it'll be implemented.

It is just one real example.

> 
> > +ublk block device(``/dev/ublkb*``) is added by ublk driver. Any IO request
> > +submitted to ublk device will be forwarded to ublk's userspace part(
> 
> Add a space between "part" and "("?

OK

> 
> > +ublksrv [1]), and after the IO is handled by ublksrv, the result is
> > +committed back to ublk driver, then ublk IO request can be completed. With
> > +this way, any specific IO handling logic is totally done inside ublksrv,
> > +and ublk driver doe _not_ handle any device specific IO logic, such as
> 
> does

OK.

> 
> > +loop's IO handling, NBD's IO communication, or qcow2's IO mapping, ...
> >
> > +/dev/ublkbN is driven by blk-mq request based driver, each request is
> > +assigned by one queue wide unique tag. ublksrv assigns unique tag to each
> > +IO too, which is 1:1 mapped with IO of /dev/ublkb*.
> > +
> > +Both the IO request forward and IO handling result committing are done via
> > +io_uring passthrough command, that is why ublk is also one io_uring based
> > +block driver. It has been observed that io_uring passthrough command can get
> > +better IOPS than block IO. So ublk is one high performance implementation
> > +of userspace block device. Not only IO request communication is done by
> > +io_uring, but also the preferred IO handling in ublksrv is io_uring based
> > +approach too.
> > +
> > +ublk provides control interface to set/get ublk block device parameters, and
> > +the interface is extendable and kabi compatible, so basically any ublk request
> > +queue's parameter or ublk generic feature parameters can be set/get via this
> > +extendable interface. So ublk is generic userspace block device framework, such
> > +as, it is easy to setup one ublk device with specified block parameters from
> > +userspace.
> > +
> > +How to use ublk
> > +===============
> > +
> > +After building ublksrv[1], ublk block device(``/dev/ublkb*``) can be added
> > +and deleted by the utility, then existed block IO applications can talk with
> 
> existing

OK

> 
> > +it.
> > +
> > +See usage details in README[2] of ublksrv, for example of ublk-loop:
> > +
> > +- add ublk device:
> > +  ublk add -t loop -f ublk-loop.img
> > +
> > +- use it:
> > +  mkfs.xfs /dev/ublkb0
> > +  mount /dev/ublkb0 /mnt
> > +  ....                     # all IOs are handled by io_uring!!!
> > +  umount /mnt
> > +
> > +- get ublk dev info:
> > +  ublk list
> > +
> > +- delete ublk device
> > +  ublk del -a
> > +  ublk del -n $ublk_dev_id
> > +
> > +Design
> > +======
> > +
> > +Control plane
> > +-------------
> > +
> > +ublk driver provides global misc device node(``/dev/ublk-control``) for
> 
> Space between "node" and "(".  There are a few more of these below.
> 
> > +managing and controlling ublk devices with help of several control commands:
> > +
> > +- UBLK_CMD_ADD_DEV
> > +  Add one ublk char device(``/dev/ublkc*``) which is talked with ublksrv wrt.
> > +  IO command communication. Basic device info is sent together with this
> > +  command, see UAPI structure of ublksrv_ctrl_dev_info, such as nr_hw_queues,
> > +  queue_depth, and max IO request buffer size, which info is negotiated with
> > +  ublk driver and sent back to ublksrv. After this command is completed, the
> > +  basic device info can't be changed any more.
> > +
> > +- UBLK_CMD_SET_PARAMS / UBLK_CMD_GET_PARAMS
> > +  Set or get ublk device's parameters, which can be generic feature related,
> > +  or request queue limit related, but can't be IO logic specific, cause ublk
> > +  driver does not handle any IO logic. This command has to be sent before
> > +  sending UBLK_CMD_START_DEV.
> > +
> > +- UBLK_CMD_START_DEV
> > +  After ublksrv prepares userspace resource such as, creating per-queue
> > +  pthread & io_ruing for handling ublk IO, this command is set for ublk
> 
> set -> sent

OK.

> 
> > +  driver to allocate & expose /dev/ublkb*. Parameters set via
> > +  UBLK_CMD_SET_PARAMS are applied for creating /dev/ublkb*.
> 
> Is this command synchronous?

All control commands are synchronous.

> ie. When it completes, is /dev/ublkb*
> definitely present in the /dev filesystem?  (I'm going to guess this
> depends on something complicated about udevd).

/dev/ublkb* is made when handling START_DEV command.

> 
> > +- UBLK_CMD_STOP_DEV
> > +  Quiesce IO on /dev/ublkb* and delete the disk. After this command returns,
> > +  ublksrv can release resource, such as destroy per-queue pthread & io_uring
> > +  for handling io command.
> > +
> > +- UBLK_CMD_DEL_DEV
> > +  Delete /dev/ublkc*. After this command returns, the allocated ublk device
> > +  number can be reused.
> > +
> > +- UBLK_CMD_GET_QUEUE_AFFINITY
> > +  After /dev/ublkc is added, ublk driver creates block layer tagset, so each
> > +  queue's affinity info is available, ublksrv sends UBLK_CMD_GET_QUEUE_AFFINITY
> > +  to retrieve queue affinity info, so ublksrv can setup the per-queue context
> > +  efficiently, such as bind affine CPUs with IO pthread, and try to allocate
> > +  buffers in IO thread context.
> > +
> > +- UBLK_CMD_GET_DEV_INFO
> > +  For retrieve device info of ublksrv_ctrl_dev_info. And it is ublksrv's
> > +  responsibility to save IO target specific info in userspace.
> > +
> > +Data plane
> > +----------
> > +
> > +ublksrv needs to create per-queue IO pthread & io_uring for handling IO
> > +command (io_uring passthrough command), and the per-queue IO pthread
> > +focuses on IO handling and shouldn't handle any control & management
> > +task.
> > +
> > +ublksrv's IO is assigned by one unique tag, which is 1:1 mapping with IO
> > +request of /dev/ublkb*.
> > +
> > +UAPI structure of ublksrv_io_desc is defined for describing each IO from
> > +ublk driver. One fixed mmaped area(array) on /dev/ublkc* is provided for
> > +exporting IO info to ublksrv, such as IO offset, length, OP/flags and
> > +buffer address. Each ublksrv_io_desc instance can be indexed via queue id
> > +and IO tag directly.
> > +
> > +Following IO commands are communicated via io_uring passthrough command,
> > +and each command is only for forwarding ublk IO and committing IO result
> > +with specified IO tag in the command data:
> > +
> > +- UBLK_IO_FETCH_REQ
> > +  Sent from ublksrv IO pthread for fetching future coming IO request
> > +  issued to /dev/ublkb*. This command is just sent once from ublksrv IO
> > +  pthread for ublk driver to setup IO forward environment.
> > +
> > +- UBLK_IO_COMMIT_AND_FETCH_REQ
> > +  After one IO request is issued to /dev/ublkb*, ublk driver stores this
> > +  IO's ublksrv_io_desc to the specified mapped area, then the previous
> > +  received IO command of this IO tag, either UBLK_IO_FETCH_REQ or
> > +  UBLK_IO_COMMIT_AND_FETCH_REQ, is completed, so ulksrv gets the IO
> > +  notification via io_uring.
> > +
> > +  After ublksrv handles this IO, this IO's result is committed back to ublk
> > +  driver by sending UBLK_IO_COMMIT_AND_FETCH_REQ back. Once ublkdrv received
> > +  this command, it parses the IO result and complete the IO request to
> > +  /dev/ublkb*. Meantime setup environment for fetching future IO request
> > +  with this IO tag. So UBLK_IO_COMMIT_AND_FETCH_REQ is reused for both
> > +  fetching request and committing back IO result.
> > +
> > +- UBLK_IO_NEED_GET_DATA
> > +  ublksrv pre-allocates IO buffer for each IO at default, any new project
> 
> at default -> by default
> 
> > +  should use this IO buffer to communicate with ublk driver. But existed
> 
> existed -> existing
> 
> > +  project may not work or be changed to in this way, so add this command
> > +  to provide chance for userspace to use its existed buffer for handling
> 
> existed -> existing

OK.

> 
> > +  IO.
> > +
> > +- data copy between ublkserv IO buffer and ublk block IO request
> > +  ublk driver needs to copy ublk block IO request pages into ublksrv buffer
> > +  (pages) first for WRITE before notifying ublksrv of the coming IO, so
> > +  ublksrv can hanldle WRITE request.
> > +
> > +  After ublksrv handles READ request and sends UBLK_IO_COMMIT_AND_FETCH_REQ
> > +  to ublksrv, ublkdrv needs to copy read ublksrv buffer(pages) to the ublk
> > +  IO request pages.
> > +
> > +Future development
> > +==================
> > +
> > +Container-ware ublk deivice
> 
> Container-aware ublk device
> 
> > +---------------------------
> > +
> > +ublk driver doesn't handle any IO logic, and its function is well defined
> > +so far, and very limited userspace interfaces are needed, and each one is
> > +well defined too, then it is very likely to make ublk device one
> > +container-ware block device in future, as Stefan Hajnoczi suggested[3], by
> > +removing ADMIN privilege.
> 
> Is it advisable for non-root to be able create arbitrary /dev devices?
> It sounds like a security nightmare because you're exposing
> potentially any arbitrary, malicious filesystem to the kernel to
> parse.

FUSE supports unprivileged mounts too, and maybe ublk can refer to FUSE's
model.


thanks,
Ming

Re: [PATCH] Docs: ublk: add ublk document

[Bagas Sanjaya]

On 8/28/22 11:50, Ming Lei wrote:
> +- UBLK_CMD_ADD_DEV
> +  Add one ublk char device(``/dev/ublkc*``) which is talked with ublksrv wrt.
> +  IO command communication. Basic device info is sent together with this
> +  command, see UAPI structure of ublksrv_ctrl_dev_info, such as nr_hw_queues,
> +  queue_depth, and max IO request buffer size, which info is negotiated with
> +  ublk driver and sent back to ublksrv. After this command is completed, the
> +  basic device info can't be changed any more.
> +

Is "see UAPI structure" means set the structure?

-- 
An old man doll... just what I always wanted! - Clara

Re: [PATCH] Docs: ublk: add ublk document

[Ming Lei]

On Tue, Aug 30, 2022 at 07:14:51PM +0700, Bagas Sanjaya wrote:
> On 8/28/22 11:50, Ming Lei wrote:
> > +- UBLK_CMD_ADD_DEV
> > +  Add one ublk char device(``/dev/ublkc*``) which is talked with ublksrv wrt.
> > +  IO command communication. Basic device info is sent together with this
> > +  command, see UAPI structure of ublksrv_ctrl_dev_info, such as nr_hw_queues,
> > +  queue_depth, and max IO request buffer size, which info is negotiated with
> > +  ublk driver and sent back to ublksrv. After this command is completed, the
> > +  basic device info can't be changed any more.
> > +
> 
> Is "see UAPI structure" means set the structure?

Yes, but some fields may be changed by driver and sent back.

thanks,
Ming

Re: [PATCH] Docs: ublk: add ublk document

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 12211 bytes --]

On Sun, Aug 28, 2022 at 12:50:03PM +0800, Ming Lei wrote:
> ublk document is missed when merging ublk driver, so add it now.
> 
> Cc: Jonathan Corbet <corbet@lwn.net>
> Cc: Richard W.M. Jones <rjones@redhat.com>
> Cc: ZiyangZhang <ZiyangZhang@linux.alibaba.com>
> Cc: Stefan Hajnoczi <stefanha@redhat.com>
> Cc: Xiaoguang Wang <xiaoguang.wang@linux.alibaba.com>
> Signed-off-by: Ming Lei <ming.lei@redhat.com>
> ---
>  Documentation/block/ublk.rst | 203 +++++++++++++++++++++++++++++++++++
>  1 file changed, 203 insertions(+)
>  create mode 100644 Documentation/block/ublk.rst
> 
> diff --git a/Documentation/block/ublk.rst b/Documentation/block/ublk.rst
> new file mode 100644
> index 000000000000..9e8f7ba518a3
> --- /dev/null
> +++ b/Documentation/block/ublk.rst
> @@ -0,0 +1,203 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +==========================================
> +Userspace block device driver(ublk driver)

Usually there is a space before '('. There are more instances throughout
the document.

> +==========================================
> +
> +Overview
> +========
> +
> +ublk is one generic framework for implementing block device logic from
> +userspace. It is very helpful to move virtual block drivers into userspace,
> +such as loop, nbd and similar block drivers. It can help to implement new
> +virtual block device, such as ublk-qcow2, and there was several attempts
> +of implementing qcow2 driver in kernel.

This paragraph doesn't give specific reasons why userspace block devices
are "very helpful".

Userspace block devices are attractive because:
- They can be written many programming languages.
- They can use libraries that are not available in the kernel.
- They can be debugged with tools familiar to application developers.
- Crashes do not kernel panic the machine.
- Bugs are likely to have a lower security impact than bugs in kernel
  code.
- They can be installed and updated independently of the kernel.

I think including this list makes it clear why userspace block devices
are appropriate in certain cases.

> +
> +ublk block device(``/dev/ublkb*``) is added by ublk driver. Any IO request
> +submitted to ublk device will be forwarded to ublk's userspace part(
> +ublksrv [1]), and after the IO is handled by ublksrv, the result is
> +committed back to ublk driver, then ublk IO request can be completed. With
> +this way, any specific IO handling logic is totally done inside ublksrv,
> +and ublk driver doe _not_ handle any device specific IO logic, such as

s/doe/does/

> +loop's IO handling, NBD's IO communication, or qcow2's IO mapping, ...
> +
> +/dev/ublkbN is driven by blk-mq request based driver, each request is
> +assigned by one queue wide unique tag. ublksrv assigns unique tag to each
> +IO too, which is 1:1 mapped with IO of /dev/ublkb*.

Is "/dev/ublkbN" the same as "/dev/ublkb*"? Please use one term consistently.

> +
> +Both the IO request forward and IO handling result committing are done via
> +io_uring passthrough command, that is why ublk is also one io_uring based

s/also one/a/

> +block driver. It has been observed that io_uring passthrough command can get
> +better IOPS than block IO. So ublk is one high performance implementation
> +of userspace block device.

I think this sentence means "Therefore ublk uses io_uring passthrough
commands to implement userspace block devices in a performant way".

> Not only IO request communication is done by
> +io_uring, but also the preferred IO handling in ublksrv is io_uring based
> +approach too.
> +
> +ublk provides control interface to set/get ublk block device parameters, and
> +the interface is extendable and kabi compatible, so basically any ublk request
> +queue's parameter or ublk generic feature parameters can be set/get via this
> +extendable interface. So ublk is generic userspace block device framework, such
> +as, it is easy to setup one ublk device with specified block parameters from

"such as" -> "and as such"

> +userspace.
> +
> +How to use ublk
> +===============
> +
> +After building ublksrv[1], ublk block device(``/dev/ublkb*``) can be added

It might be worth separating two use cases:

- Quickstart and people who just want to use existing ublksrv
  functionality.

- Developers who are writing their own ublk server (e.g. rjones' nbdkit).

ublksrv isn't needed, it's possible to code directly against the ublk
driver UAPI. I think it's worth pointing this out so it's clear to
people when and why you would use ublksrv and also when to code against
the ublk driver UAPI.

The documentation currently leaves it to the reader to figure out that
ublksrv is just one possible userspace server implementation.

> +and deleted by the utility, then existed block IO applications can talk with
> +it.
> +
> +See usage details in README[2] of ublksrv, for example of ublk-loop:
> +
> +- add ublk device:
> +  ublk add -t loop -f ublk-loop.img
> +
> +- use it:
> +  mkfs.xfs /dev/ublkb0
> +  mount /dev/ublkb0 /mnt
> +  ....                     # all IOs are handled by io_uring!!!
> +  umount /mnt
> +
> +- get ublk dev info:
> +  ublk list
> +
> +- delete ublk device
> +  ublk del -a
> +  ublk del -n $ublk_dev_id
> +
> +Design
> +======
> +
> +Control plane
> +-------------
> +
> +ublk driver provides global misc device node(``/dev/ublk-control``) for
> +managing and controlling ublk devices with help of several control commands:
> +
> +- UBLK_CMD_ADD_DEV
> +  Add one ublk char device(``/dev/ublkc*``) which is talked with ublksrv wrt.

It may be clearer to say "the server" or "userspace" instead of
"ublksrv", since that is the name of a specific server implementation
and people may develop other servers.

> +  IO command communication. Basic device info is sent together with this
> +  command, see UAPI structure of ublksrv_ctrl_dev_info, such as nr_hw_queues,
> +  queue_depth, and max IO request buffer size, which info is negotiated with
> +  ublk driver and sent back to ublksrv. After this command is completed, the
> +  basic device info can't be changed any more.
> +
> +- UBLK_CMD_SET_PARAMS / UBLK_CMD_GET_PARAMS
> +  Set or get ublk device's parameters, which can be generic feature related,
> +  or request queue limit related, but can't be IO logic specific, cause ublk
> +  driver does not handle any IO logic. This command has to be sent before
> +  sending UBLK_CMD_START_DEV.
> +
> +- UBLK_CMD_START_DEV
> +  After ublksrv prepares userspace resource such as, creating per-queue
> +  pthread & io_ruing for handling ublk IO, this command is set for ublk

s/io_ruing/io_uring/

> +  driver to allocate & expose /dev/ublkb*. Parameters set via
> +  UBLK_CMD_SET_PARAMS are applied for creating /dev/ublkb*.
> +
> +- UBLK_CMD_STOP_DEV
> +  Quiesce IO on /dev/ublkb* and delete the disk. After this command returns,
> +  ublksrv can release resource, such as destroy per-queue pthread & io_uring
> +  for handling io command.
> +
> +- UBLK_CMD_DEL_DEV
> +  Delete /dev/ublkc*. After this command returns, the allocated ublk device
> +  number can be reused.
> +
> +- UBLK_CMD_GET_QUEUE_AFFINITY
> +  After /dev/ublkc is added, ublk driver creates block layer tagset, so each
> +  queue's affinity info is available, ublksrv sends UBLK_CMD_GET_QUEUE_AFFINITY
> +  to retrieve queue affinity info, so ublksrv can setup the per-queue context
> +  efficiently, such as bind affine CPUs with IO pthread, and try to allocate
> +  buffers in IO thread context.
> +
> +- UBLK_CMD_GET_DEV_INFO
> +  For retrieve device info of ublksrv_ctrl_dev_info. And it is ublksrv's
> +  responsibility to save IO target specific info in userspace.
> +
> +Data plane
> +----------
> +
> +ublksrv needs to create per-queue IO pthread & io_uring for handling IO
> +command (io_uring passthrough command), and the per-queue IO pthread
> +focuses on IO handling and shouldn't handle any control & management
> +task.
> +
> +ublksrv's IO is assigned by one unique tag, which is 1:1 mapping with IO
> +request of /dev/ublkb*.
> +
> +UAPI structure of ublksrv_io_desc is defined for describing each IO from
> +ublk driver. One fixed mmaped area(array) on /dev/ublkc* is provided for
> +exporting IO info to ublksrv, such as IO offset, length, OP/flags and
> +buffer address. Each ublksrv_io_desc instance can be indexed via queue id
> +and IO tag directly.
> +
> +Following IO commands are communicated via io_uring passthrough command,
> +and each command is only for forwarding ublk IO and committing IO result
> +with specified IO tag in the command data:
> +
> +- UBLK_IO_FETCH_REQ
> +  Sent from ublksrv IO pthread for fetching future coming IO request
> +  issued to /dev/ublkb*. This command is just sent once from ublksrv IO
> +  pthread for ublk driver to setup IO forward environment.
> +
> +- UBLK_IO_COMMIT_AND_FETCH_REQ
> +  After one IO request is issued to /dev/ublkb*, ublk driver stores this
> +  IO's ublksrv_io_desc to the specified mapped area, then the previous
> +  received IO command of this IO tag, either UBLK_IO_FETCH_REQ or
> +  UBLK_IO_COMMIT_AND_FETCH_REQ, is completed, so ulksrv gets the IO
> +  notification via io_uring.
> +
> +  After ublksrv handles this IO, this IO's result is committed back to ublk
> +  driver by sending UBLK_IO_COMMIT_AND_FETCH_REQ back. Once ublkdrv received
> +  this command, it parses the IO result and complete the IO request to
> +  /dev/ublkb*. Meantime setup environment for fetching future IO request
> +  with this IO tag. So UBLK_IO_COMMIT_AND_FETCH_REQ is reused for both
> +  fetching request and committing back IO result.
> +
> +- UBLK_IO_NEED_GET_DATA
> +  ublksrv pre-allocates IO buffer for each IO at default, any new project
> +  should use this IO buffer to communicate with ublk driver. But existed
> +  project may not work or be changed to in this way, so add this command
> +  to provide chance for userspace to use its existed buffer for handling
> +  IO.

I find it hard to understand this paragraph. It seems the
UBLK_IO_NEED_GET_DATA command allows userspace to set up something
related to IO buffers. What exactly does this command do?

> +
> +- data copy between ublkserv IO buffer and ublk block IO request

s/ublkserv/ublksrv/

> +  ublk driver needs to copy ublk block IO request pages into ublksrv buffer
> +  (pages) first for WRITE before notifying ublksrv of the coming IO, so
> +  ublksrv can hanldle WRITE request.

s/hanldle/handle/

> +
> +  After ublksrv handles READ request and sends UBLK_IO_COMMIT_AND_FETCH_REQ
> +  to ublksrv, ublkdrv needs to copy read ublksrv buffer(pages) to the ublk
> +  IO request pages.

In the READ case userspace doesn't know exactly when the ublk driver is
done copying in the buffer. I guess this isn't a problem because the
userspace buffer will be reused when the next request is fetched?

> +
> +Future development
> +==================
> +
> +Container-ware ublk deivice
> +---------------------------
> +
> +ublk driver doesn't handle any IO logic, and its function is well defined
> +so far, and very limited userspace interfaces are needed, and each one is
> +well defined too, then it is very likely to make ublk device one
> +container-ware block device in future, as Stefan Hajnoczi suggested[3], by
> +removing ADMIN privilege.
> +
> +Zero copy
> +---------
> +
> +Wrt. zero copy support, it is one generic requirement for nbd, fuse or
> +similar drivers, one problem Xiaoguang mentioned is that pages mapped to
> +userspace can't be remapped any more in kernel with existed mm interfaces,
> +and it can be involved when submitting direct IO to /dev/ublkb*. Also
> +Xiaoguang reported that big request may benefit from zero copy a lot,
> +such as >= 256KB IO.
> +
> +
> +References
> +==========
> +
> +[1] https://github.com/ming1/ubdsrv
> +
> +[2] https://github.com/ming1/ubdsrv/blob/master/README
> +
> +[3] https://lore.kernel.org/linux-block/YoOr6jBfgVm8GvWg@stefanha-x1.localdomain/
> -- 
> 2.31.1
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH] Docs: ublk: add ublk document

[Ziyang Zhang]

On 2022/8/30 23:23, Stefan Hajnoczi wrote:
> On Sun, Aug 28, 2022 at 12:50:03PM +0800, Ming Lei wrote:
>> ublk document is missed when merging ublk driver, so add it now.
>>
>> Cc: Jonathan Corbet <corbet@lwn.net>
>> Cc: Richard W.M. Jones <rjones@redhat.com>
>> Cc: ZiyangZhang <ZiyangZhang@linux.alibaba.com>
>> Cc: Stefan Hajnoczi <stefanha@redhat.com>
>> Cc: Xiaoguang Wang <xiaoguang.wang@linux.alibaba.com>
>> Signed-off-by: Ming Lei <ming.lei@redhat.com>
>> ---
>>  Documentation/block/ublk.rst | 203 +++++++++++++++++++++++++++++++++++
>>  1 file changed, 203 insertions(+)
>>  create mode 100644 Documentation/block/ublk.rst
>>
>> diff --git a/Documentation/block/ublk.rst b/Documentation/block/ublk.rst
>> new file mode 100644
>> index 000000000000..9e8f7ba518a3
>> --- /dev/null
>> +++ b/Documentation/block/ublk.rst
>> @@ -0,0 +1,203 @@
>> +.. SPDX-License-Identifier: GPL-2.0
>> +
>> +==========================================
>> +Userspace block device driver(ublk driver)
> 
> Usually there is a space before '('. There are more instances throughout
> the document.
> 
>> +==========================================
>> +
>> +Overview
>> +========
>> +
>> +ublk is one generic framework for implementing block device logic from
>> +userspace. It is very helpful to move virtual block drivers into userspace,
>> +such as loop, nbd and similar block drivers. It can help to implement new
>> +virtual block device, such as ublk-qcow2, and there was several attempts
>> +of implementing qcow2 driver in kernel.
> 
> This paragraph doesn't give specific reasons why userspace block devices
> are "very helpful".
> 
> Userspace block devices are attractive because:
> - They can be written many programming languages.
> - They can use libraries that are not available in the kernel.
> - They can be debugged with tools familiar to application developers.
> - Crashes do not kernel panic the machine.
> - Bugs are likely to have a lower security impact than bugs in kernel
>   code.
> - They can be installed and updated independently of the kernel.
> 
> I think including this list makes it clear why userspace block devices
> are appropriate in certain cases.

+1, this list is really attractive to ublk developers/users.

> 
>> +
>> +ublk block device(``/dev/ublkb*``) is added by ublk driver. Any IO request
>> +submitted to ublk device will be forwarded to ublk's userspace part(
>> +ublksrv [1]), and after the IO is handled by ublksrv, the result is
>> +committed back to ublk driver, then ublk IO request can be completed. With
>> +this way, any specific IO handling logic is totally done inside ublksrv,
>> +and ublk driver doe _not_ handle any device specific IO logic, such as
> 
> s/doe/does/
> 
>> +loop's IO handling, NBD's IO communication, or qcow2's IO mapping, ...
>> +
>> +/dev/ublkbN is driven by blk-mq request based driver, each request is
>> +assigned by one queue wide unique tag. ublksrv assigns unique tag to each
>> +IO too, which is 1:1 mapped with IO of /dev/ublkb*.
> 
> Is "/dev/ublkbN" the same as "/dev/ublkb*"? Please use one term consistently.
> 
>> +
>> +Both the IO request forward and IO handling result committing are done via
>> +io_uring passthrough command, that is why ublk is also one io_uring based
> 
> s/also one/a/
> 
>> +block driver. It has been observed that io_uring passthrough command can get
>> +better IOPS than block IO. So ublk is one high performance implementation
>> +of userspace block device.
> 
> I think this sentence means "Therefore ublk uses io_uring passthrough
> commands to implement userspace block devices in a performant way".
> 
>> Not only IO request communication is done by
>> +io_uring, but also the preferred IO handling in ublksrv is io_uring based
>> +approach too.
>> +
>> +ublk provides control interface to set/get ublk block device parameters, and
>> +the interface is extendable and kabi compatible, so basically any ublk request
>> +queue's parameter or ublk generic feature parameters can be set/get via this
>> +extendable interface. So ublk is generic userspace block device framework, such
>> +as, it is easy to setup one ublk device with specified block parameters from
> 
> "such as" -> "and as such"
> 
>> +userspace.
>> +
>> +How to use ublk
>> +===============
>> +
>> +After building ublksrv[1], ublk block device(``/dev/ublkb*``) can be added
> 
> It might be worth separating two use cases:
> 
> - Quickstart and people who just want to use existing ublksrv
>   functionality.
> 
> - Developers who are writing their own ublk server (e.g. rjones' nbdkit).
> 
> ublksrv isn't needed, it's possible to code directly against the ublk
> driver UAPI. I think it's worth pointing this out so it's clear to
> people when and why you would use ublksrv and also when to code against
> the ublk driver UAPI.
> 
> The documentation currently leaves it to the reader to figure out that
> ublksrv is just one possible userspace server implementation.

+1. Although now ublksrv seems to be the only one userspace implementation, I
expect that developers can directly use ublk_drv's UAPI. Maybe I will do
so while embedding ublk into our own backend.

> 
>> +and deleted by the utility, then existed block IO applications can talk with
>> +it.
>> +
>> +See usage details in README[2] of ublksrv, for example of ublk-loop:
>> +
>> +- add ublk device:
>> +  ublk add -t loop -f ublk-loop.img
>> +
>> +- use it:
>> +  mkfs.xfs /dev/ublkb0
>> +  mount /dev/ublkb0 /mnt
>> +  ....                     # all IOs are handled by io_uring!!!
>> +  umount /mnt
>> +
>> +- get ublk dev info:
>> +  ublk list
>> +
>> +- delete ublk device
>> +  ublk del -a
>> +  ublk del -n $ublk_dev_id
>> +
>> +Design
>> +======
>> +
>> +Control plane
>> +-------------
>> +
>> +ublk driver provides global misc device node(``/dev/ublk-control``) for
>> +managing and controlling ublk devices with help of several control commands:
>> +
>> +- UBLK_CMD_ADD_DEV
>> +  Add one ublk char device(``/dev/ublkc*``) which is talked with ublksrv wrt.
> 
> It may be clearer to say "the server" or "userspace" instead of
> "ublksrv", since that is the name of a specific server implementation
> and people may develop other servers.
> 
>> +  IO command communication. Basic device info is sent together with this
>> +  command, see UAPI structure of ublksrv_ctrl_dev_info, such as nr_hw_queues,
>> +  queue_depth, and max IO request buffer size, which info is negotiated with
>> +  ublk driver and sent back to ublksrv. After this command is completed, the
>> +  basic device info can't be changed any more.
>> +
>> +- UBLK_CMD_SET_PARAMS / UBLK_CMD_GET_PARAMS
>> +  Set or get ublk device's parameters, which can be generic feature related,
>> +  or request queue limit related, but can't be IO logic specific, cause ublk
>> +  driver does not handle any IO logic. This command has to be sent before
>> +  sending UBLK_CMD_START_DEV.
>> +
>> +- UBLK_CMD_START_DEV
>> +  After ublksrv prepares userspace resource such as, creating per-queue
>> +  pthread & io_ruing for handling ublk IO, this command is set for ublk
> 
> s/io_ruing/io_uring/
> 
>> +  driver to allocate & expose /dev/ublkb*. Parameters set via
>> +  UBLK_CMD_SET_PARAMS are applied for creating /dev/ublkb*.
>> +
>> +- UBLK_CMD_STOP_DEV
>> +  Quiesce IO on /dev/ublkb* and delete the disk. After this command returns,
>> +  ublksrv can release resource, such as destroy per-queue pthread & io_uring
>> +  for handling io command.
>> +
>> +- UBLK_CMD_DEL_DEV
>> +  Delete /dev/ublkc*. After this command returns, the allocated ublk device
>> +  number can be reused.
>> +
>> +- UBLK_CMD_GET_QUEUE_AFFINITY
>> +  After /dev/ublkc is added, ublk driver creates block layer tagset, so each
>> +  queue's affinity info is available, ublksrv sends UBLK_CMD_GET_QUEUE_AFFINITY
>> +  to retrieve queue affinity info, so ublksrv can setup the per-queue context
>> +  efficiently, such as bind affine CPUs with IO pthread, and try to allocate
>> +  buffers in IO thread context.
>> +
>> +- UBLK_CMD_GET_DEV_INFO
>> +  For retrieve device info of ublksrv_ctrl_dev_info. And it is ublksrv's
>> +  responsibility to save IO target specific info in userspace.
>> +
>> +Data plane
>> +----------
>> +
>> +ublksrv needs to create per-queue IO pthread & io_uring for handling IO
>> +command (io_uring passthrough command), and the per-queue IO pthread
>> +focuses on IO handling and shouldn't handle any control & management
>> +task.
>> +
>> +ublksrv's IO is assigned by one unique tag, which is 1:1 mapping with IO
>> +request of /dev/ublkb*.
>> +
>> +UAPI structure of ublksrv_io_desc is defined for describing each IO from
>> +ublk driver. One fixed mmaped area(array) on /dev/ublkc* is provided for
>> +exporting IO info to ublksrv, such as IO offset, length, OP/flags and
>> +buffer address. Each ublksrv_io_desc instance can be indexed via queue id
>> +and IO tag directly.
>> +
>> +Following IO commands are communicated via io_uring passthrough command,
>> +and each command is only for forwarding ublk IO and committing IO result
>> +with specified IO tag in the command data:
>> +
>> +- UBLK_IO_FETCH_REQ
>> +  Sent from ublksrv IO pthread for fetching future coming IO request
>> +  issued to /dev/ublkb*. This command is just sent once from ublksrv IO
>> +  pthread for ublk driver to setup IO forward environment.
>> +
>> +- UBLK_IO_COMMIT_AND_FETCH_REQ
>> +  After one IO request is issued to /dev/ublkb*, ublk driver stores this
>> +  IO's ublksrv_io_desc to the specified mapped area, then the previous
>> +  received IO command of this IO tag, either UBLK_IO_FETCH_REQ or
>> +  UBLK_IO_COMMIT_AND_FETCH_REQ, is completed, so ulksrv gets the IO
>> +  notification via io_uring.
>> +
>> +  After ublksrv handles this IO, this IO's result is committed back to ublk
>> +  driver by sending UBLK_IO_COMMIT_AND_FETCH_REQ back. Once ublkdrv received
>> +  this command, it parses the IO result and complete the IO request to
>> +  /dev/ublkb*. Meantime setup environment for fetching future IO request
>> +  with this IO tag. So UBLK_IO_COMMIT_AND_FETCH_REQ is reused for both
>> +  fetching request and committing back IO result.
>> +
>> +- UBLK_IO_NEED_GET_DATA
>> +  ublksrv pre-allocates IO buffer for each IO at default, any new project
>> +  should use this IO buffer to communicate with ublk driver. But existed
>> +  project may not work or be changed to in this way, so add this command
>> +  to provide chance for userspace to use its existed buffer for handling
>> +  IO.
> 
> I find it hard to understand this paragraph. It seems the
> UBLK_IO_NEED_GET_DATA command allows userspace to set up something
> related to IO buffers. What exactly does this command do?

Let me explain UBLK_IO_NEED_GET_DATA since it is designed by myself.

Without UBLK_IO_NEED_GET_DATA, ublk_drv will copy data from biovecs
into a pre-allocated buffer(addr is passed with the last COMMIT_AMD_FETCH ioucmd)
while processing a WRITE request. Please consider two cases:

(1)  if the backend(such as a dist-storage system using RPC) provides the data
     buffer, it has to provide the buffer IN ADVANCE(before sending the last
     COMMIT_AMD_FETCH) without knowing any knowledge of this incoming request.
     This makes existing backend very hard to adapt to ublk because they may
     want to know the data length or other attributes of the new request.

(2) If the backend does not provide the data buffer IN ADVANCE, ublksrv must
    pre-allocates data buffer. So a additional data copy from ublksrv to
    the backend(such as a RPC mempool) is unavoidable.

With UBLK_IO_NEED_GET_DATA, the WRITE request will be firstly issued to ublksrv
without data copy. Then, backend gets the request and it can allocate data
buffer and embed its addr inside a new ioucmd. After the kernel driver gets the
ioucmd, the data copy happens(from biovecs to backend's buffer). Finally,
the backend gets the request again with data to be written and it can truly
handle the request.

> 
>> +
>> +- data copy between ublkserv IO buffer and ublk block IO request
> 
> s/ublkserv/ublksrv/
> 
>> +  ublk driver needs to copy ublk block IO request pages into ublksrv buffer
>> +  (pages) first for WRITE before notifying ublksrv of the coming IO, so
>> +  ublksrv can hanldle WRITE request.
> 
> s/hanldle/handle/
> 
>> +
>> +  After ublksrv handles READ request and sends UBLK_IO_COMMIT_AND_FETCH_REQ
>> +  to ublksrv, ublkdrv needs to copy read ublksrv buffer(pages) to the ublk
>> +  IO request pages.

ublksrv sends UBLK_IO_COMMIT_AND_FETCH_REQ to "ublkdrv", not "ublksrv", I think.

BTW, "ublkdrv" is very similar to "ublksrv"... Maybe we should use "ublk_drv"?

> 
> In the READ case userspace doesn't know exactly when the ublk driver is
> done copying in the buffer. I guess this isn't a problem because the
> userspace buffer will be reused when the next request is fetched?

In the READ case, userspace does not know exactly when the data copy happens.

The data buffer(add is passed with current UBLK_IO_COMMIT_AND_FETCH_REQ) is
reused for a future request(the same tag).

More specifically, whthout UBLK_IO_NEED_GET_DATA, if the future request is a
write request, this buffer is filled with biovecs' data before ublk_drv passes
the request to userspace. BTW, that's why UBLK_IO_NEED_GET_DATA is introduced.
Otherwise the read buffer is actually reused for a future write buffer...

Regards,
Zhang.

Re: [PATCH] Docs: ublk: add ublk document

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2511 bytes --]

On Wed, Aug 31, 2022 at 02:31:12PM +0800, Ziyang Zhang wrote:
> On 2022/8/30 23:23, Stefan Hajnoczi wrote:
> > On Sun, Aug 28, 2022 at 12:50:03PM +0800, Ming Lei wrote:
> >> +- UBLK_IO_NEED_GET_DATA
> >> +  ublksrv pre-allocates IO buffer for each IO at default, any new project
> >> +  should use this IO buffer to communicate with ublk driver. But existed
> >> +  project may not work or be changed to in this way, so add this command
> >> +  to provide chance for userspace to use its existed buffer for handling
> >> +  IO.
> > 
> > I find it hard to understand this paragraph. It seems the
> > UBLK_IO_NEED_GET_DATA command allows userspace to set up something
> > related to IO buffers. What exactly does this command do?
> 
> Let me explain UBLK_IO_NEED_GET_DATA since it is designed by myself.
> 
> Without UBLK_IO_NEED_GET_DATA, ublk_drv will copy data from biovecs
> into a pre-allocated buffer(addr is passed with the last COMMIT_AMD_FETCH ioucmd)
> while processing a WRITE request. Please consider two cases:
> 
> (1)  if the backend(such as a dist-storage system using RPC) provides the data
>      buffer, it has to provide the buffer IN ADVANCE(before sending the last
>      COMMIT_AMD_FETCH) without knowing any knowledge of this incoming request.
>      This makes existing backend very hard to adapt to ublk because they may
>      want to know the data length or other attributes of the new request.
> 
> (2) If the backend does not provide the data buffer IN ADVANCE, ublksrv must
>     pre-allocates data buffer. So a additional data copy from ublksrv to
>     the backend(such as a RPC mempool) is unavoidable.
> 
> With UBLK_IO_NEED_GET_DATA, the WRITE request will be firstly issued to ublksrv
> without data copy. Then, backend gets the request and it can allocate data
> buffer and embed its addr inside a new ioucmd. After the kernel driver gets the
> ioucmd, the data copy happens(from biovecs to backend's buffer). Finally,
> the backend gets the request again with data to be written and it can truly
> handle the request.

Thanks for the explanation. Maybe it can be included in the
documentation.

This reminds me of io_uring's IOSQE_BUFFER_SELECT where userspace
provides the kernel with a buffer pool and the kernel selects buffers.
It doesn't require an extra io_uring command roundtrip
(UBLK_IO_NEED_GET_DATA).

Did you already look at IOSQE_BUFFER_SELECT and decide a similar
approach won't work for your use case?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH] Docs: ublk: add ublk document

[Ming Lei]

On Tue, Aug 30, 2022 at 11:23:31AM -0400, Stefan Hajnoczi wrote:
> On Sun, Aug 28, 2022 at 12:50:03PM +0800, Ming Lei wrote:
> > ublk document is missed when merging ublk driver, so add it now.
> > 
> > Cc: Jonathan Corbet <corbet@lwn.net>
> > Cc: Richard W.M. Jones <rjones@redhat.com>
> > Cc: ZiyangZhang <ZiyangZhang@linux.alibaba.com>
> > Cc: Stefan Hajnoczi <stefanha@redhat.com>
> > Cc: Xiaoguang Wang <xiaoguang.wang@linux.alibaba.com>
> > Signed-off-by: Ming Lei <ming.lei@redhat.com>
> > ---
> >  Documentation/block/ublk.rst | 203 +++++++++++++++++++++++++++++++++++
> >  1 file changed, 203 insertions(+)
> >  create mode 100644 Documentation/block/ublk.rst
> > 
> > diff --git a/Documentation/block/ublk.rst b/Documentation/block/ublk.rst
> > new file mode 100644
> > index 000000000000..9e8f7ba518a3
> > --- /dev/null
> > +++ b/Documentation/block/ublk.rst
> > @@ -0,0 +1,203 @@
> > +.. SPDX-License-Identifier: GPL-2.0
> > +
> > +==========================================
> > +Userspace block device driver(ublk driver)
> 
> Usually there is a space before '('. There are more instances throughout
> the document.
> 
> > +==========================================
> > +
> > +Overview
> > +========
> > +
> > +ublk is one generic framework for implementing block device logic from
> > +userspace. It is very helpful to move virtual block drivers into userspace,
> > +such as loop, nbd and similar block drivers. It can help to implement new
> > +virtual block device, such as ublk-qcow2, and there was several attempts
> > +of implementing qcow2 driver in kernel.
> 
> This paragraph doesn't give specific reasons why userspace block devices
> are "very helpful".
> 
> Userspace block devices are attractive because:
> - They can be written many programming languages.
> - They can use libraries that are not available in the kernel.
> - They can be debugged with tools familiar to application developers.
> - Crashes do not kernel panic the machine.
> - Bugs are likely to have a lower security impact than bugs in kernel
>   code.
> - They can be installed and updated independently of the kernel.
> 
> I think including this list makes it clear why userspace block devices
> are appropriate in certain cases.

Good point!

Another use case: - can simulate generic block device with different type
with different parameters easily

> 
> > +
> > +ublk block device(``/dev/ublkb*``) is added by ublk driver. Any IO request
> > +submitted to ublk device will be forwarded to ublk's userspace part(
> > +ublksrv [1]), and after the IO is handled by ublksrv, the result is
> > +committed back to ublk driver, then ublk IO request can be completed. With
> > +this way, any specific IO handling logic is totally done inside ublksrv,
> > +and ublk driver doe _not_ handle any device specific IO logic, such as
> 
> s/doe/does/
> 
> > +loop's IO handling, NBD's IO communication, or qcow2's IO mapping, ...
> > +
> > +/dev/ublkbN is driven by blk-mq request based driver, each request is
> > +assigned by one queue wide unique tag. ublksrv assigns unique tag to each
> > +IO too, which is 1:1 mapped with IO of /dev/ublkb*.
> 
> Is "/dev/ublkbN" the same as "/dev/ublkb*"? Please use one term consistently.
> 
> > +
> > +Both the IO request forward and IO handling result committing are done via
> > +io_uring passthrough command, that is why ublk is also one io_uring based
> 
> s/also one/a/
> 
> > +block driver. It has been observed that io_uring passthrough command can get
> > +better IOPS than block IO. So ublk is one high performance implementation
> > +of userspace block device.
> 
> I think this sentence means "Therefore ublk uses io_uring passthrough
> commands to implement userspace block devices in a performant way".

Yeah, because io_uring passthrough is proved to be very efficient.

> 
> > Not only IO request communication is done by
> > +io_uring, but also the preferred IO handling in ublksrv is io_uring based
> > +approach too.
> > +
> > +ublk provides control interface to set/get ublk block device parameters, and
> > +the interface is extendable and kabi compatible, so basically any ublk request
> > +queue's parameter or ublk generic feature parameters can be set/get via this
> > +extendable interface. So ublk is generic userspace block device framework, such
> > +as, it is easy to setup one ublk device with specified block parameters from
> 
> "such as" -> "and as such"
> 
> > +userspace.
> > +
> > +How to use ublk
> > +===============
> > +
> > +After building ublksrv[1], ublk block device(``/dev/ublkb*``) can be added
> 
> It might be worth separating two use cases:
> 
> - Quickstart and people who just want to use existing ublksrv
>   functionality.
> 
> - Developers who are writing their own ublk server (e.g. rjones' nbdkit).
> 
> ublksrv isn't needed, it's possible to code directly against the ublk
> driver UAPI. I think it's worth pointing this out so it's clear to
> people when and why you would use ublksrv and also when to code against
> the ublk driver UAPI.
> 
> The documentation currently leaves it to the reader to figure out that
> ublksrv is just one possible userspace server implementation.

OK, we can add the words, but it is usually one common sense since
userspace is independent with kernel.

> 
> > +and deleted by the utility, then existed block IO applications can talk with
> > +it.
> > +
> > +See usage details in README[2] of ublksrv, for example of ublk-loop:
> > +
> > +- add ublk device:
> > +  ublk add -t loop -f ublk-loop.img
> > +
> > +- use it:
> > +  mkfs.xfs /dev/ublkb0
> > +  mount /dev/ublkb0 /mnt
> > +  ....                     # all IOs are handled by io_uring!!!
> > +  umount /mnt
> > +
> > +- get ublk dev info:
> > +  ublk list
> > +
> > +- delete ublk device
> > +  ublk del -a
> > +  ublk del -n $ublk_dev_id
> > +
> > +Design
> > +======
> > +
> > +Control plane
> > +-------------
> > +
> > +ublk driver provides global misc device node(``/dev/ublk-control``) for
> > +managing and controlling ublk devices with help of several control commands:
> > +
> > +- UBLK_CMD_ADD_DEV
> > +  Add one ublk char device(``/dev/ublkc*``) which is talked with ublksrv wrt.
> 
> It may be clearer to say "the server" or "userspace" instead of
> "ublksrv", since that is the name of a specific server implementation
> and people may develop other servers.

For the sake of simplicity, I will use ublksrv or ublk server to represent any
userspace application of ublk.

> 
> > +  IO command communication. Basic device info is sent together with this
> > +  command, see UAPI structure of ublksrv_ctrl_dev_info, such as nr_hw_queues,
> > +  queue_depth, and max IO request buffer size, which info is negotiated with
> > +  ublk driver and sent back to ublksrv. After this command is completed, the
> > +  basic device info can't be changed any more.
> > +
> > +- UBLK_CMD_SET_PARAMS / UBLK_CMD_GET_PARAMS
> > +  Set or get ublk device's parameters, which can be generic feature related,
> > +  or request queue limit related, but can't be IO logic specific, cause ublk
> > +  driver does not handle any IO logic. This command has to be sent before
> > +  sending UBLK_CMD_START_DEV.
> > +
> > +- UBLK_CMD_START_DEV
> > +  After ublksrv prepares userspace resource such as, creating per-queue
> > +  pthread & io_ruing for handling ublk IO, this command is set for ublk
> 
> s/io_ruing/io_uring/
> 
> > +  driver to allocate & expose /dev/ublkb*. Parameters set via
> > +  UBLK_CMD_SET_PARAMS are applied for creating /dev/ublkb*.
> > +
> > +- UBLK_CMD_STOP_DEV
> > +  Quiesce IO on /dev/ublkb* and delete the disk. After this command returns,
> > +  ublksrv can release resource, such as destroy per-queue pthread & io_uring
> > +  for handling io command.
> > +
> > +- UBLK_CMD_DEL_DEV
> > +  Delete /dev/ublkc*. After this command returns, the allocated ublk device
> > +  number can be reused.
> > +
> > +- UBLK_CMD_GET_QUEUE_AFFINITY
> > +  After /dev/ublkc is added, ublk driver creates block layer tagset, so each
> > +  queue's affinity info is available, ublksrv sends UBLK_CMD_GET_QUEUE_AFFINITY
> > +  to retrieve queue affinity info, so ublksrv can setup the per-queue context
> > +  efficiently, such as bind affine CPUs with IO pthread, and try to allocate
> > +  buffers in IO thread context.
> > +
> > +- UBLK_CMD_GET_DEV_INFO
> > +  For retrieve device info of ublksrv_ctrl_dev_info. And it is ublksrv's
> > +  responsibility to save IO target specific info in userspace.
> > +
> > +Data plane
> > +----------
> > +
> > +ublksrv needs to create per-queue IO pthread & io_uring for handling IO
> > +command (io_uring passthrough command), and the per-queue IO pthread
> > +focuses on IO handling and shouldn't handle any control & management
> > +task.
> > +
> > +ublksrv's IO is assigned by one unique tag, which is 1:1 mapping with IO
> > +request of /dev/ublkb*.
> > +
> > +UAPI structure of ublksrv_io_desc is defined for describing each IO from
> > +ublk driver. One fixed mmaped area(array) on /dev/ublkc* is provided for
> > +exporting IO info to ublksrv, such as IO offset, length, OP/flags and
> > +buffer address. Each ublksrv_io_desc instance can be indexed via queue id
> > +and IO tag directly.
> > +
> > +Following IO commands are communicated via io_uring passthrough command,
> > +and each command is only for forwarding ublk IO and committing IO result
> > +with specified IO tag in the command data:
> > +
> > +- UBLK_IO_FETCH_REQ
> > +  Sent from ublksrv IO pthread for fetching future coming IO request
> > +  issued to /dev/ublkb*. This command is just sent once from ublksrv IO
> > +  pthread for ublk driver to setup IO forward environment.
> > +
> > +- UBLK_IO_COMMIT_AND_FETCH_REQ
> > +  After one IO request is issued to /dev/ublkb*, ublk driver stores this
> > +  IO's ublksrv_io_desc to the specified mapped area, then the previous
> > +  received IO command of this IO tag, either UBLK_IO_FETCH_REQ or
> > +  UBLK_IO_COMMIT_AND_FETCH_REQ, is completed, so ulksrv gets the IO
> > +  notification via io_uring.
> > +
> > +  After ublksrv handles this IO, this IO's result is committed back to ublk
> > +  driver by sending UBLK_IO_COMMIT_AND_FETCH_REQ back. Once ublkdrv received
> > +  this command, it parses the IO result and complete the IO request to
> > +  /dev/ublkb*. Meantime setup environment for fetching future IO request
> > +  with this IO tag. So UBLK_IO_COMMIT_AND_FETCH_REQ is reused for both
> > +  fetching request and committing back IO result.
> > +
> > +- UBLK_IO_NEED_GET_DATA
> > +  ublksrv pre-allocates IO buffer for each IO at default, any new project
> > +  should use this IO buffer to communicate with ublk driver. But existed
> > +  project may not work or be changed to in this way, so add this command
> > +  to provide chance for userspace to use its existed buffer for handling
> > +  IO.
> 
> I find it hard to understand this paragraph. It seems the
> UBLK_IO_NEED_GET_DATA command allows userspace to set up something
> related to IO buffers. What exactly does this command do?

Looks Ziyang has replied on this question.

> 
> > +
> > +- data copy between ublkserv IO buffer and ublk block IO request
> 
> s/ublkserv/ublksrv/
> 
> > +  ublk driver needs to copy ublk block IO request pages into ublksrv buffer
> > +  (pages) first for WRITE before notifying ublksrv of the coming IO, so
> > +  ublksrv can hanldle WRITE request.
> 
> s/hanldle/handle/
> 
> > +
> > +  After ublksrv handles READ request and sends UBLK_IO_COMMIT_AND_FETCH_REQ
> > +  to ublksrv, ublkdrv needs to copy read ublksrv buffer(pages) to the ublk
> > +  IO request pages.
> 
> In the READ case userspace doesn't know exactly when the ublk driver is
> done copying in the buffer. I guess this isn't a problem because the
> userspace buffer will be reused when the next request is fetched?

In short, one buffer is dedicated for one single tag/slot, either in driver
side or userspace side. In kernel side, the buffer is used/prepared for
accepting new io request. In userspace side, it is used for handling
the io request. And that is how UBLK_IO_COMMIT_AND_FETCH_REQ is
defined.

Yeah, it is pre-allocation of virtual memory, but pages are just allocated
when the buffer/IO is used. If the buffer isn't used for long enough
period(so far it is set as 20sec), ublksrv will discard these pages via
madvise(DONT_NEED), so swap out isn't involved usually.



Thanks, 
Ming

Re: [PATCH] Docs: ublk: add ublk document

[Jens Axboe]

On 8/27/22 10:50 PM, Ming Lei wrote:
> ublk document is missed when merging ublk driver, so add it now.

Ming, and you send a v2 of this so we can get it queued up for
6.0?

-- 
Jens Axboe

Re: [PATCH] Docs: ublk: add ublk document

[Ming Lei]

On Wed, Aug 31, 2022 at 02:31:12PM +0800, Ziyang Zhang wrote:
> On 2022/8/30 23:23, Stefan Hajnoczi wrote:
> > On Sun, Aug 28, 2022 at 12:50:03PM +0800, Ming Lei wrote:
> >> ublk document is missed when merging ublk driver, so add it now.
> >>
> >> Cc: Jonathan Corbet <corbet@lwn.net>
> >> Cc: Richard W.M. Jones <rjones@redhat.com>
> >> Cc: ZiyangZhang <ZiyangZhang@linux.alibaba.com>
> >> Cc: Stefan Hajnoczi <stefanha@redhat.com>
> >> Cc: Xiaoguang Wang <xiaoguang.wang@linux.alibaba.com>
> >> Signed-off-by: Ming Lei <ming.lei@redhat.com>
> >> ---
> >>  Documentation/block/ublk.rst | 203 +++++++++++++++++++++++++++++++++++
> >>  1 file changed, 203 insertions(+)
> >>  create mode 100644 Documentation/block/ublk.rst
> >>
> >> diff --git a/Documentation/block/ublk.rst b/Documentation/block/ublk.rst
> >> new file mode 100644
> >> index 000000000000..9e8f7ba518a3
> >> --- /dev/null
> >> +++ b/Documentation/block/ublk.rst
> >> @@ -0,0 +1,203 @@
> >> +.. SPDX-License-Identifier: GPL-2.0
> >> +
> >> +==========================================
> >> +Userspace block device driver(ublk driver)
> > 
> > Usually there is a space before '('. There are more instances throughout
> > the document.
> > 
> >> +==========================================
> >> +
> >> +Overview
> >> +========
> >> +
> >> +ublk is one generic framework for implementing block device logic from
> >> +userspace. It is very helpful to move virtual block drivers into userspace,
> >> +such as loop, nbd and similar block drivers. It can help to implement new
> >> +virtual block device, such as ublk-qcow2, and there was several attempts
> >> +of implementing qcow2 driver in kernel.
> > 
> > This paragraph doesn't give specific reasons why userspace block devices
> > are "very helpful".
> > 
> > Userspace block devices are attractive because:
> > - They can be written many programming languages.
> > - They can use libraries that are not available in the kernel.
> > - They can be debugged with tools familiar to application developers.
> > - Crashes do not kernel panic the machine.
> > - Bugs are likely to have a lower security impact than bugs in kernel
> >   code.
> > - They can be installed and updated independently of the kernel.
> > 
> > I think including this list makes it clear why userspace block devices
> > are appropriate in certain cases.
> 
> +1, this list is really attractive to ublk developers/users.
> 
> > 
> >> +
> >> +ublk block device(``/dev/ublkb*``) is added by ublk driver. Any IO request
> >> +submitted to ublk device will be forwarded to ublk's userspace part(
> >> +ublksrv [1]), and after the IO is handled by ublksrv, the result is
> >> +committed back to ublk driver, then ublk IO request can be completed. With
> >> +this way, any specific IO handling logic is totally done inside ublksrv,
> >> +and ublk driver doe _not_ handle any device specific IO logic, such as
> > 
> > s/doe/does/
> > 
> >> +loop's IO handling, NBD's IO communication, or qcow2's IO mapping, ...
> >> +
> >> +/dev/ublkbN is driven by blk-mq request based driver, each request is
> >> +assigned by one queue wide unique tag. ublksrv assigns unique tag to each
> >> +IO too, which is 1:1 mapped with IO of /dev/ublkb*.
> > 
> > Is "/dev/ublkbN" the same as "/dev/ublkb*"? Please use one term consistently.
> > 
> >> +
> >> +Both the IO request forward and IO handling result committing are done via
> >> +io_uring passthrough command, that is why ublk is also one io_uring based
> > 
> > s/also one/a/
> > 
> >> +block driver. It has been observed that io_uring passthrough command can get
> >> +better IOPS than block IO. So ublk is one high performance implementation
> >> +of userspace block device.
> > 
> > I think this sentence means "Therefore ublk uses io_uring passthrough
> > commands to implement userspace block devices in a performant way".
> > 
> >> Not only IO request communication is done by
> >> +io_uring, but also the preferred IO handling in ublksrv is io_uring based
> >> +approach too.
> >> +
> >> +ublk provides control interface to set/get ublk block device parameters, and
> >> +the interface is extendable and kabi compatible, so basically any ublk request
> >> +queue's parameter or ublk generic feature parameters can be set/get via this
> >> +extendable interface. So ublk is generic userspace block device framework, such
> >> +as, it is easy to setup one ublk device with specified block parameters from
> > 
> > "such as" -> "and as such"
> > 
> >> +userspace.
> >> +
> >> +How to use ublk
> >> +===============
> >> +
> >> +After building ublksrv[1], ublk block device(``/dev/ublkb*``) can be added
> > 
> > It might be worth separating two use cases:
> > 
> > - Quickstart and people who just want to use existing ublksrv
> >   functionality.
> > 
> > - Developers who are writing their own ublk server (e.g. rjones' nbdkit).
> > 
> > ublksrv isn't needed, it's possible to code directly against the ublk
> > driver UAPI. I think it's worth pointing this out so it's clear to
> > people when and why you would use ublksrv and also when to code against
> > the ublk driver UAPI.
> > 
> > The documentation currently leaves it to the reader to figure out that
> > ublksrv is just one possible userspace server implementation.
> 
> +1. Although now ublksrv seems to be the only one userspace implementation, I
> expect that developers can directly use ublk_drv's UAPI. Maybe I will do
> so while embedding ublk into our own backend.

Yeah, anyone can do that.

But when there is any bug you want to ask for help from community, can
you share your use case? Or is it easy for you to do so? At that time,
you still need to figure out how to trigger it in one open source
implementation, then share the use case. Otherwise, not sure if
community can help you easily.

There will be more functions or features added to ublksrv/ublk driver, it
really needs one open source project to verify/test. You may say the
feature is verified in your private/closed source environment, but
usually not good, cause no one can reproduce it.

> 
> > 
> >> +and deleted by the utility, then existed block IO applications can talk with
> >> +it.
> >> +
> >> +See usage details in README[2] of ublksrv, for example of ublk-loop:
> >> +
> >> +- add ublk device:
> >> +  ublk add -t loop -f ublk-loop.img
> >> +
> >> +- use it:
> >> +  mkfs.xfs /dev/ublkb0
> >> +  mount /dev/ublkb0 /mnt
> >> +  ....                     # all IOs are handled by io_uring!!!
> >> +  umount /mnt
> >> +
> >> +- get ublk dev info:
> >> +  ublk list
> >> +
> >> +- delete ublk device
> >> +  ublk del -a
> >> +  ublk del -n $ublk_dev_id
> >> +
> >> +Design
> >> +======
> >> +
> >> +Control plane
> >> +-------------
> >> +
> >> +ublk driver provides global misc device node(``/dev/ublk-control``) for
> >> +managing and controlling ublk devices with help of several control commands:
> >> +
> >> +- UBLK_CMD_ADD_DEV
> >> +  Add one ublk char device(``/dev/ublkc*``) which is talked with ublksrv wrt.
> > 
> > It may be clearer to say "the server" or "userspace" instead of
> > "ublksrv", since that is the name of a specific server implementation
> > and people may develop other servers.
> > 
> >> +  IO command communication. Basic device info is sent together with this
> >> +  command, see UAPI structure of ublksrv_ctrl_dev_info, such as nr_hw_queues,
> >> +  queue_depth, and max IO request buffer size, which info is negotiated with
> >> +  ublk driver and sent back to ublksrv. After this command is completed, the
> >> +  basic device info can't be changed any more.
> >> +
> >> +- UBLK_CMD_SET_PARAMS / UBLK_CMD_GET_PARAMS
> >> +  Set or get ublk device's parameters, which can be generic feature related,
> >> +  or request queue limit related, but can't be IO logic specific, cause ublk
> >> +  driver does not handle any IO logic. This command has to be sent before
> >> +  sending UBLK_CMD_START_DEV.
> >> +
> >> +- UBLK_CMD_START_DEV
> >> +  After ublksrv prepares userspace resource such as, creating per-queue
> >> +  pthread & io_ruing for handling ublk IO, this command is set for ublk
> > 
> > s/io_ruing/io_uring/
> > 
> >> +  driver to allocate & expose /dev/ublkb*. Parameters set via
> >> +  UBLK_CMD_SET_PARAMS are applied for creating /dev/ublkb*.
> >> +
> >> +- UBLK_CMD_STOP_DEV
> >> +  Quiesce IO on /dev/ublkb* and delete the disk. After this command returns,
> >> +  ublksrv can release resource, such as destroy per-queue pthread & io_uring
> >> +  for handling io command.
> >> +
> >> +- UBLK_CMD_DEL_DEV
> >> +  Delete /dev/ublkc*. After this command returns, the allocated ublk device
> >> +  number can be reused.
> >> +
> >> +- UBLK_CMD_GET_QUEUE_AFFINITY
> >> +  After /dev/ublkc is added, ublk driver creates block layer tagset, so each
> >> +  queue's affinity info is available, ublksrv sends UBLK_CMD_GET_QUEUE_AFFINITY
> >> +  to retrieve queue affinity info, so ublksrv can setup the per-queue context
> >> +  efficiently, such as bind affine CPUs with IO pthread, and try to allocate
> >> +  buffers in IO thread context.
> >> +
> >> +- UBLK_CMD_GET_DEV_INFO
> >> +  For retrieve device info of ublksrv_ctrl_dev_info. And it is ublksrv's
> >> +  responsibility to save IO target specific info in userspace.
> >> +
> >> +Data plane
> >> +----------
> >> +
> >> +ublksrv needs to create per-queue IO pthread & io_uring for handling IO
> >> +command (io_uring passthrough command), and the per-queue IO pthread
> >> +focuses on IO handling and shouldn't handle any control & management
> >> +task.
> >> +
> >> +ublksrv's IO is assigned by one unique tag, which is 1:1 mapping with IO
> >> +request of /dev/ublkb*.
> >> +
> >> +UAPI structure of ublksrv_io_desc is defined for describing each IO from
> >> +ublk driver. One fixed mmaped area(array) on /dev/ublkc* is provided for
> >> +exporting IO info to ublksrv, such as IO offset, length, OP/flags and
> >> +buffer address. Each ublksrv_io_desc instance can be indexed via queue id
> >> +and IO tag directly.
> >> +
> >> +Following IO commands are communicated via io_uring passthrough command,
> >> +and each command is only for forwarding ublk IO and committing IO result
> >> +with specified IO tag in the command data:
> >> +
> >> +- UBLK_IO_FETCH_REQ
> >> +  Sent from ublksrv IO pthread for fetching future coming IO request
> >> +  issued to /dev/ublkb*. This command is just sent once from ublksrv IO
> >> +  pthread for ublk driver to setup IO forward environment.
> >> +
> >> +- UBLK_IO_COMMIT_AND_FETCH_REQ
> >> +  After one IO request is issued to /dev/ublkb*, ublk driver stores this
> >> +  IO's ublksrv_io_desc to the specified mapped area, then the previous
> >> +  received IO command of this IO tag, either UBLK_IO_FETCH_REQ or
> >> +  UBLK_IO_COMMIT_AND_FETCH_REQ, is completed, so ulksrv gets the IO
> >> +  notification via io_uring.
> >> +
> >> +  After ublksrv handles this IO, this IO's result is committed back to ublk
> >> +  driver by sending UBLK_IO_COMMIT_AND_FETCH_REQ back. Once ublkdrv received
> >> +  this command, it parses the IO result and complete the IO request to
> >> +  /dev/ublkb*. Meantime setup environment for fetching future IO request
> >> +  with this IO tag. So UBLK_IO_COMMIT_AND_FETCH_REQ is reused for both
> >> +  fetching request and committing back IO result.
> >> +
> >> +- UBLK_IO_NEED_GET_DATA
> >> +  ublksrv pre-allocates IO buffer for each IO at default, any new project
> >> +  should use this IO buffer to communicate with ublk driver. But existed
> >> +  project may not work or be changed to in this way, so add this command
> >> +  to provide chance for userspace to use its existed buffer for handling
> >> +  IO.
> > 
> > I find it hard to understand this paragraph. It seems the
> > UBLK_IO_NEED_GET_DATA command allows userspace to set up something
> > related to IO buffers. What exactly does this command do?
> 
> Let me explain UBLK_IO_NEED_GET_DATA since it is designed by myself.
> 
> Without UBLK_IO_NEED_GET_DATA, ublk_drv will copy data from biovecs
> into a pre-allocated buffer(addr is passed with the last COMMIT_AMD_FETCH ioucmd)
> while processing a WRITE request. Please consider two cases:
> 
> (1)  if the backend(such as a dist-storage system using RPC) provides the data
>      buffer, it has to provide the buffer IN ADVANCE(before sending the last
>      COMMIT_AMD_FETCH) without knowing any knowledge of this incoming request.

The max buffer bytes is defined in ublksrv_ctrl_dev_info, and that is
max length of any incoming io request.

I explained it in last email, this way won't waste memory, and needn't extra
swap out too.

>      This makes existing backend very hard to adapt to ublk because they may
>      want to know the data length or other attributes of the new request.

It is just for existing project.

Any new project can read the data from the pre-allocated buffer
directly. That is exactly the handling flow: ublksrv gets one request from
ublk driver, then let backend handle the request.

> 
> (2) If the backend does not provide the data buffer IN ADVANCE, ublksrv must
>     pre-allocates data buffer. So a additional data copy from ublksrv to
>     the backend(such as a RPC mempool) is unavoidable.

Can you explain why backend can't use the pre-allocated buffer directly? Before
backend completes the io request, the io request and buffer won't be reused, that
is owned by this tag/slot.

> 
> With UBLK_IO_NEED_GET_DATA, the WRITE request will be firstly issued to ublksrv
> without data copy. Then, backend gets the request and it can allocate data
> buffer and embed its addr inside a new ioucmd. After the kernel driver gets the
> ioucmd, the data copy happens(from biovecs to backend's buffer). Finally,
> the backend gets the request again with data to be written and it can truly
> handle the request.

That is definitely inefficient, and I won't encourage any new project to
use this command.

> 
> > 
> >> +
> >> +- data copy between ublkserv IO buffer and ublk block IO request
> > 
> > s/ublkserv/ublksrv/
> > 
> >> +  ublk driver needs to copy ublk block IO request pages into ublksrv buffer
> >> +  (pages) first for WRITE before notifying ublksrv of the coming IO, so
> >> +  ublksrv can hanldle WRITE request.
> > 
> > s/hanldle/handle/
> > 
> >> +
> >> +  After ublksrv handles READ request and sends UBLK_IO_COMMIT_AND_FETCH_REQ
> >> +  to ublksrv, ublkdrv needs to copy read ublksrv buffer(pages) to the ublk
> >> +  IO request pages.
> 
> ublksrv sends UBLK_IO_COMMIT_AND_FETCH_REQ to "ublkdrv", not "ublksrv", I think.
> 
> BTW, "ublkdrv" is very similar to "ublksrv"... Maybe we should use "ublk_drv"?
> 
> > 
> > In the READ case userspace doesn't know exactly when the ublk driver is
> > done copying in the buffer. I guess this isn't a problem because the
> > userspace buffer will be reused when the next request is fetched?
> 
> In the READ case, userspace does not know exactly when the data copy happens.

ublksrv just needs to write data to the buffer, then send the buffer
with UBLK_IO_COMMIT_AND_FETCH_REQ to driver, and driver will copy the
data to request's pages.

Why does the userspace need to know when the copy happens?

> 
> The data buffer(add is passed with current UBLK_IO_COMMIT_AND_FETCH_REQ) is
> reused for a future request(the same tag).

Yeah, that is why it is completely safe since the future request can
only come _after_ ublk driver has copied the previous buffer to request
pages.

> 
> More specifically, whthout UBLK_IO_NEED_GET_DATA, if the future request is a
> write request, this buffer is filled with biovecs' data before ublk_drv passes
> the request to userspace. BTW, that's why UBLK_IO_NEED_GET_DATA is introduced.
> Otherwise the read buffer is actually reused for a future write buffer...

Not sure I get your point, but in short UBLK_IO_NEED_GET_DATA is only
for legacy uses.


Thanks,
Ming

Re: [PATCH] Docs: ublk: add ublk document

[Ming Lei]

On Wed, Aug 31, 2022 at 07:04:33PM -0600, Jens Axboe wrote:
> On 8/27/22 10:50 PM, Ming Lei wrote:
> > ublk document is missed when merging ublk driver, so add it now.
> 
> Ming, and you send a v2 of this so we can get it queued up for
> 6.0?

OK, I will send v2 soon.


Thanks,
Ming

Re: [PATCH] Docs: ublk: add ublk document

[Ziyang Zhang]

On 2022/9/1 09:34, Ming Lei wrote:
> 
>>      This makes existing backend very hard to adapt to ublk because they may
>>      want to know the data length or other attributes of the new request.
> 
> It is just for existing project.

Existing project is very important. I believe that embedding libublksrv/UAPI
into existing projects(products) makes ublk more popular and useful.

> 
> Any new project can read the data from the pre-allocated buffer
> directly. That is exactly the handling flow: ublksrv gets one request from
> ublk driver, then let backend handle the request.

Your are correct, Ming. ublksrv tgts does not need UBLK_IO_NEED_GET_DATA.

> 
>>
>> (2) If the backend does not provide the data buffer IN ADVANCE, ublksrv must
>>     pre-allocates data buffer. So a additional data copy from ublksrv to
>>     the backend(such as a RPC mempool) is unavoidable.
> 
> Can you explain why backend can't use the pre-allocated buffer directly? Before
> backend completes the io request, the io request and buffer won't be reused, that
> is owned by this tag/slot.

For existing projects using ublksrv, why it must use ublksrv's pre-allocated buffer?
The backend has its own buffer management.

Besides, existing projects may directly embed libublksrv/UAPI into it.
UBLK_IO_NEED_GET_DATA is just an option for them.

Ming, UBLK_IO_NEED_GET_DATA usecases has been proved useful and we have discussed
it when I introduced it into kernel driver. Really (1)users use ublksrv directly
or (2)developers implement new ublksrv targets do not have to care about it.

> 
>>
>> With UBLK_IO_NEED_GET_DATA, the WRITE request will be firstly issued to ublksrv
>> without data copy. Then, backend gets the request and it can allocate data
>> buffer and embed its addr inside a new ioucmd. After the kernel driver gets the
>> ioucmd, the data copy happens(from biovecs to backend's buffer). Finally,
>> the backend gets the request again with data to be written and it can truly
>> handle the request.
> 
> That is definitely inefficient, and I won't encourage any new project to
> use this command.

UBLK_IO_NEED_GET_DATA is an option. Any user thinks that it may lower performance
should not use it.

BTW, our tests shows that UBLK_IO_NEED_GET_DATA add one additional
round-trip in ublk_drv and one io_uring_enter() syscall.

UBLK_IO_NEED_GET_DATA does not lower the IOPS too much if:
(1) iodepth is bigger. This is because io_uring batches sqes(ioucmds) so the
    syscall overhead is not significant.
(2) the backend is slow. For example, with a network(RPC) backend, we really
    do not care this round-trip since the backend IO handling
    is far slower than ublk_drv's data path.

In conclusion, UBLK_IO_NEED_GET_DATA is designed for existing projects, not for
ublksrv(though it supports this feature) targets. UBLK_IO_NEED_GET_DATA is COMPLETELY
motivated by our real practice in developing userspace storage products.

Regards,
Zhang