Re: [LSF/MM/BPF TOPIC] block drivers in user space

From: Mike Christie <michael.christie@oracle.com>
To: Gabriel Krisman Bertazi <krisman@collabora.com>,
	lsf-pc@lists.linux-foundation.org
Cc: linux-block@vger.kernel.org
Subject: Re: [LSF/MM/BPF TOPIC] block drivers in user space
Date: Wed, 2 Mar 2022 10:52:23 -0600	[thread overview]
Message-ID: <526a8360-d7d3-1211-087b-c86d5a68380b@oracle.com> (raw)
In-Reply-To: <87tucsf0sr.fsf@collabora.com>

On 2/21/22 1:59 PM, Gabriel Krisman Bertazi wrote:
> I'd like to discuss an interface to implement user space block devices,
> while avoiding local network NBD solutions.  There has been reiterated

Besides the tcmu approach, I've also worked on the local nbd based
solution like here:

https://github.com/gluster/nbd-runner

Have you looked into a modern take that uses io_uring's socket features
with the zero copy work that's being worked on for it? If so, what are
the issues you have hit with that? Was it mostly issues with the zero
copy part of it?

> interest in the topic, both from researchers [1] and from the community,
> including a proposed session in LSFMM2018 [2] (though I don't think it
> happened).
> 
> I've been working on top of the Google iblock implementation to find
> something upstreamable and would like to present my design and gather
> feedback on some points, in particular zero-copy and overall user space
> interface.
> 
> The design I'm pending towards uses special fds opened by the driver to
> transfer data to/from the block driver, preferably through direct
> splicing as much as possible, to keep data only in kernel space.  This
> is because, in my use case, the driver usually only manipulates
> metadata, while data is forwarded directly through the network, or
> similar. It would be neat if we can leverage the existing
> splice/copy_file_range syscalls such that we don't ever need to bring
> disk data to user space, if we can avoid it.  I've also experimented
> with regular pipes, But I found no way around keeping a lot of pipes
> opened, one for each possible command 'slot'.