Re: [LSF/MM/BPF TOPIC] block drivers in user space

From: Mike Christie <michael.christie@oracle.com>
To: Sagi Grimberg <sagi@grimberg.me>,
	Gabriel Krisman Bertazi <krisman@collabora.com>
Cc: Hannes Reinecke <hare@suse.de>,
	lsf-pc@lists.linux-foundation.org, linux-block@vger.kernel.org
Subject: Re: [LSF/MM/BPF TOPIC] block drivers in user space
Date: Wed, 2 Mar 2022 10:16:25 -0600	[thread overview]
Message-ID: <45caea9d-53d0-6f06-bb98-9174a08972d4@oracle.com> (raw)
In-Reply-To: <e0a6ca51-8202-0b61-dd50-349e6f27761b@grimberg.me>

On 2/24/22 4:12 AM, Sagi Grimberg wrote:
> 
>>>> Actually, I'd rather have something like an 'inverse io_uring', where
>>>> an application creates a memory region separated into several 'ring'
>>>> for submission and completion.
>>>> Then the kernel could write/map the incoming data onto the rings, and
>>>> application can read from there.
>>>> Maybe it'll be worthwhile to look at virtio here.
>>>
>>> There is lio loopback backed by tcmu... I'm assuming that nvmet can
>>> hook into the same/similar interface. nvmet is pretty lean, and we
>>> can probably help tcmu/equivalent scale better if that is a concern...
>>
>> Sagi,
>>
>> I looked at tcmu prior to starting this work.  Other than the tcmu
>> overhead, one concern was the complexity of a scsi device interface
>> versus sending block requests to userspace.
> 
> The complexity is understandable, though it can be viewed as a
> capability as well. Note I do not have any desire to promote tcmu here,
> just trying to understand if we need a brand new interface rather than
> making the existing one better.

Ccing tcmu maintainer Bodo.

We don't want to re-use tcmu's interface.

Bodo has been looking into on a new interface to avoid issues tcmu has
and to improve performance. If it's allowed to add a tcmu like backend to
nvmet then it would be great because lio was not really made with mq and
perf in mind so it already starts with issues. I just started doing the
basics like removing locks from the main lio IO path but it seems like
there is just so much work.

> 
>> What would be the advantage of doing it as a nvme target over delivering
>> directly to userspace as a block driver?
> 
> Well, for starters you gain the features and tools that are extensively
> used with nvme. Plus you get the ecosystem support (development,
> features, capabilities and testing). There are clear advantages of
> plugging into an established ecosystem.

Yeah, tcmu has been really nice to export storage that people for whatever
reason didn't want in the kernel. We got the benefits you described where
distros have the required tools and their support teams have experience, etc.