Re: [LSF/MM/BPF TOPIC] block drivers in user space

From: Hannes Reinecke <hare@suse.de>
To: Bart Van Assche <bvanassche@acm.org>,
	Sagi Grimberg <sagi@grimberg.me>,
	Mike Christie <michael.christie@oracle.com>,
	Gabriel Krisman Bertazi <krisman@collabora.com>
Cc: lsf-pc@lists.linux-foundation.org, linux-block@vger.kernel.org
Subject: Re: [LSF/MM/BPF TOPIC] block drivers in user space
Date: Tue, 15 Mar 2022 07:52:37 +0100	[thread overview]
Message-ID: <c618c809-4ec0-69f9-0cab-87149ad6b45a@suse.de> (raw)
In-Reply-To: <0b85385b-e8cf-2ab3-ce22-c63d4346cc16@acm.org>

On 3/14/22 20:21, Bart Van Assche wrote:
> On 3/13/22 14:15, Sagi Grimberg wrote:
>>
>>> We don't want to re-use tcmu's interface.
>>>
>>> Bodo has been looking into on a new interface to avoid issues tcmu has
>>> and to improve performance. If it's allowed to add a tcmu like 
>>> backend to
>>> nvmet then it would be great because lio was not really made with mq and
>>> perf in mind so it already starts with issues. I just started doing the
>>> basics like removing locks from the main lio IO path but it seems like
>>> there is just so much work.
>>
>> Good to know...
>>
>> So I hear there is a desire to do this. So I think we should list the
>> use-cases for this first because that would lead to different design
>> choices.. For example one use-case is just to send read/write/flush
>> to userspace, another may want to passthru nvme commands to userspace
>> and there may be others...
> 
> (resending my reply without truncating the Cc-list)
> 
> Hi Sagi,
> 
> Haven't these use cases already been mentioned in the email at the start 
> of this thread? The use cases I am aware of are implementing 
> cloud-specific block storage functionality and also block storage in 
> user space for Android. Having to parse NVMe commands and PRP or SGL 
> lists would be an unnecessary source of complexity and overhead for 
> these use cases. My understanding is that what is needed for these use 
> cases is something that is close to the block layer request interface 
> (REQ_OP_* + request flags + data buffer).
> 

Curiously, the former was exactly my idea. I was thinking about having a 
simple nvmet userspace driver where all the transport 'magic' was 
handled in the nvmet driver, and just the NVMe SQEs passed on to the 
userland driver. The userland driver would then send the CQEs back to 
the driver.
With that the kernel driver becomes extremely simple, and would allow 
userspace to do all the magic it wants. More to the point, one could 
implement all sorts of fancy features which are out of scope for the 
current nvmet implementation.
Which is why I've been talking about 'inverse' io_uring; the userland 
driver will have to wait for SQEs, and write CQEs back to the driver.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer