Re: [LSF/MM/BPF TOPIC] block drivers in user space

[Xiaoguang Wang <xiaoguang.wang@linux.alibaba.com>]

hi,

> Sagi Grimberg <sagi@grimberg.me> writes:
>
>>> Actually, I'd rather have something like an 'inverse io_uring', where
>>> an application creates a memory region separated into several 'ring'
>>> for submission and completion.
>>> Then the kernel could write/map the incoming data onto the rings, and
>>> application can read from there.
>>> Maybe it'll be worthwhile to look at virtio here.
>> There is lio loopback backed by tcmu... I'm assuming that nvmet can
>> hook into the same/similar interface. nvmet is pretty lean, and we
>> can probably help tcmu/equivalent scale better if that is a concern...
> Sagi,
>
> I looked at tcmu prior to starting this work.  Other than the tcmu
> overhead, one concern was the complexity of a scsi device interface
> versus sending block requests to userspace.
Yeah, Some of our costumers have tried to use tcmu and found obvious 
overhead, which
impact io throughput tremendously, especially it lacks zero-copy and 
multi-queue support.
Previously I have sent a report to tcmu community:
     https://www.spinics.net/lists/target-devel/msg21121.html

And currently I have implemented a zero-copy prototype for tcmu(not sent 
out yet), which
increases user's io throughput from 3.6GB to 11.5GB/s, fio 4 jobs, 8 
iodepth, io size 256kb.
This prototype uses remap_pfn_range() to map io requests' sg pages to 
user space, but
remap_pfn_range() have obvious overhead while intel pat is enabled. I 
also sent a mail to
mm community:
https://lore.kernel.org/linux-mm/c5526629-5ce4-1f99-e9af-36da2876b258@linux.alibaba.com/T/#u
About how to map sg pages to use space correctly, but there's no 
response yet.
If anybody is familiar with my question, may kindly give help, thanks.

Regards,
Xiaoguang Wang
>
> What would be the advantage of doing it as a nvme target over delivering
> directly to userspace as a block driver?
>
> Also, when considering the case where userspace wants to just look at the IO
> descriptor, without actually sending data to userspace, I'm not sure
> that would be doable with tcmu?
>
> Another attempt to do the same thing here, now with device-mapper:
>
> https://patchwork.kernel.org/project/dm-devel/patch/20201203215859.2719888-4-palmer@dabbelt.com/
>