Re: [LSF/MM/BPF TOPIC] block drivers in user space

[Damien Le Moal <damien.lemoal@opensource.wdc.com>]

On 2/23/22 17:11, Gao Xiang wrote:
> On Wed, Feb 23, 2022 at 04:46:41PM +0900, Damien Le Moal wrote:
>> On 2/23/22 14:57, Gao Xiang wrote:
>>> On Mon, Feb 21, 2022 at 02:59:48PM -0500, Gabriel Krisman Bertazi wrote:
>>>> I'd like to discuss an interface to implement user space block devices,
>>>> while avoiding local network NBD solutions.  There has been reiterated
>>>> interest in the topic, both from researchers [1] and from the community,
>>>> including a proposed session in LSFMM2018 [2] (though I don't think it
>>>> happened).
>>>>
>>>> I've been working on top of the Google iblock implementation to find
>>>> something upstreamable and would like to present my design and gather
>>>> feedback on some points, in particular zero-copy and overall user space
>>>> interface.
>>>>
>>>> The design I'm pending towards uses special fds opened by the driver to
>>>> transfer data to/from the block driver, preferably through direct
>>>> splicing as much as possible, to keep data only in kernel space.  This
>>>> is because, in my use case, the driver usually only manipulates
>>>> metadata, while data is forwarded directly through the network, or
>>>> similar. It would be neat if we can leverage the existing
>>>> splice/copy_file_range syscalls such that we don't ever need to bring
>>>> disk data to user space, if we can avoid it.  I've also experimented
>>>> with regular pipes, But I found no way around keeping a lot of pipes
>>>> opened, one for each possible command 'slot'.
>>>>
>>>> [1] https://dl.acm.org/doi/10.1145/3456727.3463768
>>>> [2] https://www.spinics.net/lists/linux-fsdevel/msg120674.html
>>>
>>> I'm interested in this general topic too. One of our use cases is
>>> that we need to process network data in some degree since many
>>> protocols are application layer protocols so it seems more reasonable
>>> to process such protocols in userspace. And another difference is that
>>> we may have thousands of devices in a machine since we'd better to run
>>> containers as many as possible so the block device solution seems
>>> suboptimal to us. Yet I'm still interested in this topic to get more
>>> ideas.
>>>
>>> Btw, As for general userspace block device solutions, IMHO, there could
>>> be some deadlock issues out of direct reclaim, writeback, and userspace
>>> implementation due to writeback user requests can be tripped back to
>>> the kernel side (even the dependency crosses threads). I think they are
>>> somewhat hard to fix with user block device solutions. For example,
>>> https://lore.kernel.org/r/CAM1OiDPxh0B1sXkyGCSTEpdgDd196-ftzLE-ocnM8Jd2F9w7AA@mail.gmail.com
>>
>> This is already fixed with prctl() support. See:
>>
>> https://lore.kernel.org/linux-fsdevel/20191112001900.9206-1-mchristi@redhat.com/
> 
> As I mentioned above, IMHO, we could add some per-task state to avoid
> the majority of such deadlock cases (also what I mentioned above), but
> there may still some potential dependency could happen between threads,
> such as using another kernel workqueue and waiting on it (in principle
> at least) since userspace program can call any syscall in principle (
> which doesn't like in-kernel drivers). So I think it can cause some
> risk due to generic userspace block device restriction, please kindly
> correct me if I'm wrong.

Not sure what you mean with all this. prctl() works per process/thread
and a context that has PR_SET_IO_FLUSHER set will have PF_MEMALLOC_NOIO
set. So for the case of a user block device driver, setting this means
that it cannot reenter itself during a memory allocation, regardless of
the system call it executes (FS etc): all memory allocations in any
syscall executed by the context will have GFP_NOIO.

If the kernel-side driver for the user block device driver does any
allocation that does not have GFP_NOIO, or cause any such allocation
(e.g. within a workqueue it is waiting for), then that is a kernel bug.
Block device drivers are not supposed to ever do a memory allocation in
the IO hot path without GFP_NOIO.

> 
> Thanks,
> Gao Xiang
> 
>>
>>
>> -- 
>> Damien Le Moal
>> Western Digital Research


-- 
Damien Le Moal
Western Digital Research