[Qemu-devel] If Qemu support NVMe over Fabrics ?

[Qemu-devel] If Qemu support NVMe over Fabrics ?

[Yang Zhong]

Hello all,

Do you know if Qemu support NVMe over Fabrics(NVMe-oF)?
https://nvmexpress.org/wp-content/uploads/NVMe_Over_Fabrics.pdf

The Qemu has enabled RDMA in last year, and i am not sure if Qemu 
should support NVME-oF. If Qemu support it, would you please share
me the qemu related command or guides? thanks a lot!

Regards,

Yang

Re: [Qemu-devel] If Qemu support NVMe over Fabrics ?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 741 bytes --]

On Thu, Jan 10, 2019 at 04:37:53PM +0800, Yang Zhong wrote:
> Do you know if Qemu support NVMe over Fabrics(NVMe-oF)?
> https://nvmexpress.org/wp-content/uploads/NVMe_Over_Fabrics.pdf
> 
> The Qemu has enabled RDMA in last year, and i am not sure if Qemu 
> should support NVME-oF. If Qemu support it, would you please share
> me the qemu related command or guides? thanks a lot!

QEMU supports many different storage configurations.  Can you be more
specific?

For example, if your host has NVMe-oF set up then you can give the NVMe
block devices to QEMU just like any other host block device (-drive
file=/dev/sdc,...).

But maybe you are thinking about other configurations, like exposing
NVMe-oF to the guest?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] If Qemu support NVMe over Fabrics ?y

[Yang Zhong]

On Thu, Jan 10, 2019 at 10:36:37AM +0000, Stefan Hajnoczi wrote:
> On Thu, Jan 10, 2019 at 04:37:53PM +0800, Yang Zhong wrote:
> > Do you know if Qemu support NVMe over Fabrics(NVMe-oF)?
> > https://nvmexpress.org/wp-content/uploads/NVMe_Over_Fabrics.pdf
> > 
> > The Qemu has enabled RDMA in last year, and i am not sure if Qemu 
> > should support NVME-oF. If Qemu support it, would you please share
> > me the qemu related command or guides? thanks a lot!
> 
> QEMU supports many different storage configurations.  Can you be more
> specific?
> 
> For example, if your host has NVMe-oF set up then you can give the NVMe
> block devices to QEMU just like any other host block device (-drive
> file=/dev/sdc,...).
> 
> But maybe you are thinking about other configurations, like exposing
> NVMe-oF to the guest?
> 
  Thanks Stefan's comments. We only want Qemu as NVMe-oF initiator to
  access remote target's resource.

  I checked the block/nvme.c and hw/block/nvme.c code, which seems do
  not support NVMe-oF . If i am wrong please correct me.

  If Qemu support NVMe-oF initiator, please share me how to use it. 
  If Qemu does not support it, please tell me if community has plan 
  to implement it. thanks a lot!
 
  Regards,

  Yang
 
> Stefan

Re: [Qemu-devel] If Qemu support NVMe over Fabrics ?y

[Paolo Bonzini]

On 11/01/19 06:46, Yang Zhong wrote:
> On Thu, Jan 10, 2019 at 10:36:37AM +0000, Stefan Hajnoczi wrote:
>> On Thu, Jan 10, 2019 at 04:37:53PM +0800, Yang Zhong wrote:
>>> Do you know if Qemu support NVMe over Fabrics(NVMe-oF)?
>>> https://nvmexpress.org/wp-content/uploads/NVMe_Over_Fabrics.pdf
>>>
>>> The Qemu has enabled RDMA in last year, and i am not sure if Qemu 
>>> should support NVME-oF. If Qemu support it, would you please share
>>> me the qemu related command or guides? thanks a lot!
>>
>> QEMU supports many different storage configurations.  Can you be more
>> specific?
>>
>> For example, if your host has NVMe-oF set up then you can give the NVMe
>> block devices to QEMU just like any other host block device (-drive
>> file=/dev/sdc,...).
>>
>> But maybe you are thinking about other configurations, like exposing
>> NVMe-oF to the guest?
>>
>   Thanks Stefan's comments. We only want Qemu as NVMe-oF initiator to
>   access remote target's resource.
> 
>   I checked the block/nvme.c and hw/block/nvme.c code, which seems do
>   not support NVMe-oF . If i am wrong please correct me.
> 
>   If Qemu support NVMe-oF initiator, please share me how to use it. 
>   If Qemu does not support it, please tell me if community has plan 
>   to implement it. thanks a lot!

QEMU's native NVMe driver only supports NVMe over PCI, but it should be
possible to extract common code if you want to add a native NVMe over
RDMA driver to QEMU.  There are currently no plans to add such a driver,
but it would certainly be a welcome addition.

Paolo

Re: [Qemu-devel] If Qemu support NVMe over Fabrics ?y

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2063 bytes --]

On Fri, Jan 11, 2019 at 11:48:52AM +0100, Paolo Bonzini wrote:
> On 11/01/19 06:46, Yang Zhong wrote:
> > On Thu, Jan 10, 2019 at 10:36:37AM +0000, Stefan Hajnoczi wrote:
> >> On Thu, Jan 10, 2019 at 04:37:53PM +0800, Yang Zhong wrote:
> >>> Do you know if Qemu support NVMe over Fabrics(NVMe-oF)?
> >>> https://nvmexpress.org/wp-content/uploads/NVMe_Over_Fabrics.pdf
> >>>
> >>> The Qemu has enabled RDMA in last year, and i am not sure if Qemu 
> >>> should support NVME-oF. If Qemu support it, would you please share
> >>> me the qemu related command or guides? thanks a lot!
> >>
> >> QEMU supports many different storage configurations.  Can you be more
> >> specific?
> >>
> >> For example, if your host has NVMe-oF set up then you can give the NVMe
> >> block devices to QEMU just like any other host block device (-drive
> >> file=/dev/sdc,...).
> >>
> >> But maybe you are thinking about other configurations, like exposing
> >> NVMe-oF to the guest?
> >>
> >   Thanks Stefan's comments. We only want Qemu as NVMe-oF initiator to
> >   access remote target's resource.
> > 
> >   I checked the block/nvme.c and hw/block/nvme.c code, which seems do
> >   not support NVMe-oF . If i am wrong please correct me.
> > 
> >   If Qemu support NVMe-oF initiator, please share me how to use it. 
> >   If Qemu does not support it, please tell me if community has plan 
> >   to implement it. thanks a lot!
> 
> QEMU's native NVMe driver only supports NVMe over PCI, but it should be
> possible to extract common code if you want to add a native NVMe over
> RDMA driver to QEMU.  There are currently no plans to add such a driver,
> but it would certainly be a welcome addition.

Before investing time in doing that, what is the goal?

Is this for test and bring-up of NVMe-oF?  Or why does the guest need to
know that the storage is NVMe-oF?

As I mentioned before, if your host supports NVMe-oF you can simply give
the block device to QEMU and let the guest access it via virtio-blk,
virtio-scsi, NVMe, etc.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] If Qemu support NVMe over Fabrics ?y

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 2466 bytes --]

On 11/01/19 16:58, Stefan Hajnoczi wrote:
> On Fri, Jan 11, 2019 at 11:48:52AM +0100, Paolo Bonzini wrote:
>> On 11/01/19 06:46, Yang Zhong wrote:
>>> On Thu, Jan 10, 2019 at 10:36:37AM +0000, Stefan Hajnoczi wrote:
>>>> On Thu, Jan 10, 2019 at 04:37:53PM +0800, Yang Zhong wrote:
>>>>> Do you know if Qemu support NVMe over Fabrics(NVMe-oF)?
>>>>> https://nvmexpress.org/wp-content/uploads/NVMe_Over_Fabrics.pdf
>>>>>
>>>>> The Qemu has enabled RDMA in last year, and i am not sure if Qemu 
>>>>> should support NVME-oF. If Qemu support it, would you please share
>>>>> me the qemu related command or guides? thanks a lot!
>>>>
>>>> QEMU supports many different storage configurations.  Can you be more
>>>> specific?
>>>>
>>>> For example, if your host has NVMe-oF set up then you can give the NVMe
>>>> block devices to QEMU just like any other host block device (-drive
>>>> file=/dev/sdc,...).
>>>>
>>>> But maybe you are thinking about other configurations, like exposing
>>>> NVMe-oF to the guest?
>>>>
>>>   Thanks Stefan's comments. We only want Qemu as NVMe-oF initiator to
>>>   access remote target's resource.
>>>
>>>   I checked the block/nvme.c and hw/block/nvme.c code, which seems do
>>>   not support NVMe-oF . If i am wrong please correct me.
>>>
>>>   If Qemu support NVMe-oF initiator, please share me how to use it. 
>>>   If Qemu does not support it, please tell me if community has plan 
>>>   to implement it. thanks a lot!
>>
>> QEMU's native NVMe driver only supports NVMe over PCI, but it should be
>> possible to extract common code if you want to add a native NVMe over
>> RDMA driver to QEMU.  There are currently no plans to add such a driver,
>> but it would certainly be a welcome addition.
> 
> Before investing time in doing that, what is the goal?
> 
> Is this for test and bring-up of NVMe-oF?  Or why does the guest need to
> know that the storage is NVMe-oF?
> 
> As I mentioned before, if your host supports NVMe-oF you can simply give
> the block device to QEMU and let the guest access it via virtio-blk,
> virtio-scsi, NVMe, etc.

NVMe-OF is an RDMA protocol, basically a different transport for the
NVMe command set and queue abstraction.  It would allow QEMU to access
the device directly (similar to what block/nvme.c does for PCI using
VFIO) without going through the host kernel.  The guest would see the
device as virtio-blk/scsi, NVMe or anything else.

Paolo


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] If Qemu support NVMe over Fabrics ?y

[Keith Busch]

On Fri, Jan 11, 2019 at 05:07:26PM +0100, Paolo Bonzini wrote:
> On 11/01/19 16:58, Stefan Hajnoczi wrote:
> > Before investing time in doing that, what is the goal?
> > 
> > Is this for test and bring-up of NVMe-oF?  Or why does the guest need to
> > know that the storage is NVMe-oF?
> > 
> > As I mentioned before, if your host supports NVMe-oF you can simply give
> > the block device to QEMU and let the guest access it via virtio-blk,
> > virtio-scsi, NVMe, etc.
> 
> NVMe-OF is an RDMA protocol, basically a different transport for the
> NVMe command set and queue abstraction.  It would allow QEMU to access
> the device directly (similar to what block/nvme.c does for PCI using
> VFIO) without going through the host kernel.  The guest would see the
> device as virtio-blk/scsi, NVMe or anything else.
> 
> Paolo

I think there's multiple ways to interpret the original request, and I'm
not sure which is the right one.

If he just wants a guest's backing storage to be an nvmeof target,
then Stefan's response sounds right with qemu abstracted from the
underlying protocol as long as the host supports it.

The other way I interpreted this is he wants a guest to connect to an
nvmeof target. If so, then the guest just needs to be provisioned with an
RDMA NIC or fibre channel, or even a virtio-nic may work if the fabric
target is NVMe-over-TCP. You'd just need a capable nvmeof driver stack
in the guest.

If the request is really to implement a vfio based solution to bypass
the host kernel ... that sounds like an interesting project. :)