[SPDK] Re: bdev architecture

[SPDK] Re: bdev architecture

[Rodriguez, Edwin]

[-- Attachment #1: Type: text/plain, Size: 2925 bytes --]

We're actually implementing a fabric attached KV store using a bdev for the storage.  Eventually, we'd like use a KV SSD on the back end and envision using a pass through nvme bdev to access it.  But for now, we have a couple of types of bdevs using disk based kv stores.  It would have been cool to use the blobstore enabled rocksdb used by SPDK but I haven't yet figured out how to configure one bdev while in the midst of configuring a different bdev.  And it also introduces a circular dependency between rocksdb and spdk.
________________________________
From: Harris, James R <james.r.harris(a)intel.com>
Sent: Tuesday, March 9, 2021 9:24 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: bdev architecture

NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.




Hi Edwin,

Another possibility would be to use spdk_bdev_nvme_io_passthru and avoid trying to create a KV abstraction at all in the bdev layer.  Then the bdev nvme module can just pass those operations straight through to a KV-capable NVMe SSD.  Or a bdev module can implement a handler to emulate NVMe KV opcodes.

-Jim



On 3/9/21, 6:31 AM, "Rodriguez, Edwin" <Ed.Rodriguez(a)netapp.com> wrote:

    I've been implementing the NVMeOf KV protocol and have a question about implementing the KV list operation.  Currently, I've implemented it as returning the same array of packed keys expected by the KV protocol, but it strikes me as tying bdev too closely to the NVME protocol.

    A simple solution is to return one key for each list operation, then iteratively increment the key and pass it back to bdev to search for the next key.  Downside is that the bdev implementation could be expensive to search for a key and that would have to be done for each and every key.

    Another solution is to return a list or array which the controller iterates over to fill in the NVME list of keys, however this requires the controller to estimate how many keys will be needed to fill in the results buffer.  Since keys are variable length and packed together that will pose a challenge.

    And another solution would be to create bdev operations to iterate over the keys in the bdev, but that requires at least 3 operations - open, next and close and requires maintaining state somewhere, probably in the bdev_io itself.

    What would be in keeping with the design philosphy of bdev?  Keep it as is, returning an nvme structure of packed keys or something more abstract?
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org

_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: bdev architecture

[Rodriguez, Edwin]

[-- Attachment #1: Type: text/plain, Size: 2666 bytes --]

Maybe another approach is for the caller to pass a function pointer to a callback function that gets called for each element in the list by the bdev.  That might be the most flexible approach from the bdev perspective, though it requires adding more state to bdev_io.
________________________________
From: Harris, James R <james.r.harris(a)intel.com>
Sent: Tuesday, March 9, 2021 9:24 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: bdev architecture

NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.




Hi Edwin,

Another possibility would be to use spdk_bdev_nvme_io_passthru and avoid trying to create a KV abstraction at all in the bdev layer.  Then the bdev nvme module can just pass those operations straight through to a KV-capable NVMe SSD.  Or a bdev module can implement a handler to emulate NVMe KV opcodes.

-Jim



On 3/9/21, 6:31 AM, "Rodriguez, Edwin" <Ed.Rodriguez(a)netapp.com> wrote:

    I've been implementing the NVMeOf KV protocol and have a question about implementing the KV list operation.  Currently, I've implemented it as returning the same array of packed keys expected by the KV protocol, but it strikes me as tying bdev too closely to the NVME protocol.

    A simple solution is to return one key for each list operation, then iteratively increment the key and pass it back to bdev to search for the next key.  Downside is that the bdev implementation could be expensive to search for a key and that would have to be done for each and every key.

    Another solution is to return a list or array which the controller iterates over to fill in the NVME list of keys, however this requires the controller to estimate how many keys will be needed to fill in the results buffer.  Since keys are variable length and packed together that will pose a challenge.

    And another solution would be to create bdev operations to iterate over the keys in the bdev, but that requires at least 3 operations - open, next and close and requires maintaining state somewhere, probably in the bdev_io itself.

    What would be in keeping with the design philosphy of bdev?  Keep it as is, returning an nvme structure of packed keys or something more abstract?
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org

_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: bdev architecture

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 1857 bytes --]

Hi Edwin,

Another possibility would be to use spdk_bdev_nvme_io_passthru and avoid trying to create a KV abstraction at all in the bdev layer.  Then the bdev nvme module can just pass those operations straight through to a KV-capable NVMe SSD.  Or a bdev module can implement a handler to emulate NVMe KV opcodes.

-Jim



On 3/9/21, 6:31 AM, "Rodriguez, Edwin" <Ed.Rodriguez(a)netapp.com> wrote:

    I've been implementing the NVMeOf KV protocol and have a question about implementing the KV list operation.  Currently, I've implemented it as returning the same array of packed keys expected by the KV protocol, but it strikes me as tying bdev too closely to the NVME protocol.

    A simple solution is to return one key for each list operation, then iteratively increment the key and pass it back to bdev to search for the next key.  Downside is that the bdev implementation could be expensive to search for a key and that would have to be done for each and every key.

    Another solution is to return a list or array which the controller iterates over to fill in the NVME list of keys, however this requires the controller to estimate how many keys will be needed to fill in the results buffer.  Since keys are variable length and packed together that will pose a challenge.

    And another solution would be to create bdev operations to iterate over the keys in the bdev, but that requires at least 3 operations - open, next and close and requires maintaining state somewhere, probably in the bdev_io itself.

    What would be in keeping with the design philosphy of bdev?  Keep it as is, returning an nvme structure of packed keys or something more abstract?
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org