Re: [LSF/MM/BPF TOPIC] block drivers in user space

From: Sagi Grimberg <sagi@grimberg.me>
To: Hannes Reinecke <hare@suse.de>,
	Bart Van Assche <bvanassche@acm.org>,
	Mike Christie <michael.christie@oracle.com>,
	Gabriel Krisman Bertazi <krisman@collabora.com>
Cc: lsf-pc@lists.linux-foundation.org, linux-block@vger.kernel.org
Subject: Re: [LSF/MM/BPF TOPIC] block drivers in user space
Date: Tue, 15 Mar 2022 10:08:55 +0200	[thread overview]
Message-ID: <d2950977-9930-1e80-a46d-8311935e8da4@grimberg.me> (raw)
In-Reply-To: <c618c809-4ec0-69f9-0cab-87149ad6b45a@suse.de>

>> Hi Sagi,
>>
>> Haven't these use cases already been mentioned in the email at the 
>> start of this thread? The use cases I am aware of are implementing 
>> cloud-specific block storage functionality and also block storage in 
>> user space for Android. Having to parse NVMe commands and PRP or SGL 
>> lists would be an unnecessary source of complexity and overhead for 
>> these use cases. My understanding is that what is needed for these use 
>> cases is something that is close to the block layer request interface 
>> (REQ_OP_* + request flags + data buffer).
>>
> 
> Curiously, the former was exactly my idea. I was thinking about having a 
> simple nvmet userspace driver where all the transport 'magic' was 
> handled in the nvmet driver, and just the NVMe SQEs passed on to the 
> userland driver. The userland driver would then send the CQEs back to 
> the driver.
> With that the kernel driver becomes extremely simple, and would allow 
> userspace to do all the magic it wants. More to the point, one could 
> implement all sorts of fancy features which are out of scope for the 
> current nvmet implementation.

My thinking is that this simplification can be done in a userland
core library with a simpler interface for backends to plug into (or
a richer interface if that is what the use-case warrants).

> Which is why I've been talking about 'inverse' io_uring; the userland 
> driver will have to wait for SQEs, and write CQEs back to the driver.

"inverse" io_uring is just a ring interface, tcmu has it as well, I'm
assuming you are talking about the scalability attributes of it...