Re: [SPDK] NBD with SPDK

[Mittal, Rishabh <rimittal at ebay.com>]

[-- Attachment #1: Type: text/plain, Size: 3282 bytes --]

That’s right.  I am thinking of using function remap_page_range in kernel only for the buffers which are in use currently. I don't think there will be much cost of mapping the physical address to virtual address.

Xiaoxi,

What data size are you using in your testing.


Thanks
Rishabh Mittal

On 8/14/19, 9:54 AM, "Harris, James R" <james.r.harris(a)intel.com> wrote:

    
    
    On 8/14/19, 9:18 AM, "Walker, Benjamin" <benjamin.walker(a)intel.com> wrote:
    
    <trim>
        
        When an I/O is performed in the process initiating the I/O to a file, the data
        goes into the OS page cache buffers at a layer far above the bio stack
        (somewhere up in VFS). If SPDK were to reserve some memory and hand it off to
        your kernel driver, your kernel driver would still need to copy it to that
        location out of the page cache buffers. We can't safely share the page cache
        buffers with a user space process.
       
    I think Rishabh was suggesting the SPDK reserve the virtual address space only.
    Then the kernel could map the page cache buffers into that virtual address space.
    That would not require a data copy, but would require the mapping operations.
    
    I think the profiling data would be really helpful - to quantify how much of the 50us
    Is due to copying the 4KB of data.  That can help drive next steps on how to optimize
    the SPDK NBD module.
    
    Thanks,
    
    -Jim
    
    
        As Paul said, I'm skeptical that the memcpy is significant in the overall
        performance you're measuring. I encourage you to go look at some profiling data
        and confirm that the memcpy is really showing up. I suspect the overhead is
        instead primarily in these spots:
        
        1) Dynamic buffer allocation in the SPDK NBD backend.
        
        As Paul indicated, the NBD target is dynamically allocating memory for each I/O.
        The NBD backend wasn't designed to be fast - it was designed to be simple.
        Pooling would be a lot faster and is something fairly easy to implement.
        
        2) The way SPDK does the syscalls when it implements the NBD backend.
        
        Again, the code was designed to be simple, not high performance. It simply calls
        read() and write() on the socket for each command. There are much higher
        performance ways of doing this, they're just more complex to implement.
        
        3) The lack of multi-queue support in NBD
        
        Every I/O is funneled through a single sockpair up to user space. That means
        there is locking going on. I believe this is just a limitation of NBD today - it
        doesn't plug into the block-mq stuff in the kernel and expose multiple
        sockpairs. But someone more knowledgeable on the kernel stack would need to take
        a look.
        
        Thanks,
        Ben
        
        > 
        > Couple of things that I am not really sure in this flow is :- 1. How memory
        > registration is going to work with RDMA driver.
        > 2. What changes are required in spdk memory management
        > 
        > Thanks
        > Rishabh Mittal