[SPDK] Running spdk in a virtualized env

[SPDK] Running spdk in a virtualized env

[Andrey Kuzmin]

[-- Attachment #1: Type: text/plain, Size: 227 bytes --]

Are there any special considerations to running spdk in a virtualized
environment such as ESX (cpus, polling control, memory reservation, nvme
passthrough)? Any suggestions/pointers will be appreciated.


Thanks,
Andrey

Re: [SPDK] Running spdk in a virtualized env

[Andrey Kuzmin]

[-- Attachment #1: Type: text/plain, Size: 3622 bytes --]

Thanks Seth.

On Wed, Jul 10, 2019, 07:57 Howell, Seth <seth.howell(a)intel.com> wrote:

> Hi Andrey,
>
> As Changpeng said, we typically run spdk inside of virtual machines for
> testing. What is the use case for your virtualized environment?
>
> If you are doing testing, I would recommend bootstrapping using the
> vagrant scripts in the spdk directory. It's (usually) pretty quick and
> painless. There is also a readme for it on spdk.io
> https://spdk.io/doc/vagrant.html. I think there are just a couple of
> prerequisites described in the readme, then there is a pretty simple top
> level script in scripts/vagrant/create_vbox.sh which you can call to set up
> a test machine.
>
> Please see below if you are looking for a more custom solution.
>
> A few top level things:
>
> You will obviously need vt-x enabled in the host bios and intel_iommu
> turned on in the host's grub.
> When we test SPDK using virtual machines, we typically use qemu's built in
> NVMe drive emulation. We don't typically use passthru NVMe SSDs when
> running SPDK in a virtualized environment, so I don't have any specific
> advice about passing a physical drive through to the VM directly.
> We typically use hugepages as the backing memory when starting a VM just
> to be safe, but I don't know that that is actually required.
>
> This is what my typical command looks like when starting a VM:
>
> Sudo qemu-system-x86_64 -drive format=qcow2,file=fedora_30_canonical.qcow2
> -drive format=raw,file=nvme.img,if=none,id=nvmedrive -device
> nvme,drive=nvmedrive,serial=1234 -m 8192 -cpu host -smp 8 -enable-kvm (this
> command doesn't have the hugepage backing portion).
>
> A note on security:
> Virtual machines don't have an iommu (I don't even think there is an
> emulated one) so you won't be able to bind any drives to vfio-pci, instead
> you will have to use uio_pci_generic which doesn't have the same
> protections as vfio. This just means that from a security perspective your
> threat model changes. Essentially you run a greater risk of DMAing to some
> bad memory (within the scope of the VM) because of using uio_pci_generic.
>
> I hope this answers some of your questions.
>
> Thanks,
>
> Seth
>
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Liu, Changpeng
> Sent: Wednesday, July 10, 2019 8:38 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] Running spdk in a virtualized env
>
> Actually most of SPDK CI  are running in a QEMU VM image, I see a
> requirement with huge pages for vhost/virtio cases.
>
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey
> > Kuzmin
> > Sent: Tuesday, July 9, 2019 10:58 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: [SPDK] Running spdk in a virtualized env
> >
> > Are there any special considerations to running spdk in a virtualized
> > environment such as ESX (cpus, polling control, memory reservation,
> > nvme passthrough)? Any suggestions/pointers will be appreciated.
> >
> >
> > Thanks,
> > Andrey
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

Re: [SPDK] Running spdk in a virtualized env

[Howell, Seth]

[-- Attachment #1: Type: text/plain, Size: 3229 bytes --]

Hi Andrey,

As Changpeng said, we typically run spdk inside of virtual machines for testing. What is the use case for your virtualized environment?

If you are doing testing, I would recommend bootstrapping using the vagrant scripts in the spdk directory. It's (usually) pretty quick and painless. There is also a readme for it on spdk.io https://spdk.io/doc/vagrant.html. I think there are just a couple of prerequisites described in the readme, then there is a pretty simple top level script in scripts/vagrant/create_vbox.sh which you can call to set up a test machine.

Please see below if you are looking for a more custom solution.

A few top level things:

You will obviously need vt-x enabled in the host bios and intel_iommu turned on in the host's grub.
When we test SPDK using virtual machines, we typically use qemu's built in NVMe drive emulation. We don't typically use passthru NVMe SSDs when running SPDK in a virtualized environment, so I don't have any specific advice about passing a physical drive through to the VM directly.
We typically use hugepages as the backing memory when starting a VM just to be safe, but I don't know that that is actually required.

This is what my typical command looks like when starting a VM:

Sudo qemu-system-x86_64 -drive format=qcow2,file=fedora_30_canonical.qcow2 -drive format=raw,file=nvme.img,if=none,id=nvmedrive -device nvme,drive=nvmedrive,serial=1234 -m 8192 -cpu host -smp 8 -enable-kvm (this command doesn't have the hugepage backing portion).

A note on security:
Virtual machines don't have an iommu (I don't even think there is an emulated one) so you won't be able to bind any drives to vfio-pci, instead you will have to use uio_pci_generic which doesn't have the same protections as vfio. This just means that from a security perspective your threat model changes. Essentially you run a greater risk of DMAing to some bad memory (within the scope of the VM) because of using uio_pci_generic.

I hope this answers some of your questions. 

Thanks,

Seth

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Liu, Changpeng
Sent: Wednesday, July 10, 2019 8:38 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Running spdk in a virtualized env

Actually most of SPDK CI  are running in a QEMU VM image, I see a requirement with huge pages for vhost/virtio cases.

> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey 
> Kuzmin
> Sent: Tuesday, July 9, 2019 10:58 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Running spdk in a virtualized env
> 
> Are there any special considerations to running spdk in a virtualized 
> environment such as ESX (cpus, polling control, memory reservation, 
> nvme passthrough)? Any suggestions/pointers will be appreciated.
> 
> 
> Thanks,
> Andrey
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Running spdk in a virtualized env

[Liu, Changpeng]

[-- Attachment #1: Type: text/plain, Size: 773 bytes --]

Actually most of SPDK CI  are running in a QEMU VM image, I see a requirement with huge pages for vhost/virtio cases.

> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
> Sent: Tuesday, July 9, 2019 10:58 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Running spdk in a virtualized env
> 
> Are there any special considerations to running spdk in a virtualized
> environment such as ESX (cpus, polling control, memory reservation, nvme
> passthrough)? Any suggestions/pointers will be appreciated.
> 
> 
> Thanks,
> Andrey
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Running spdk in a virtualized env

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 757 bytes --]

I personally don't have anything for ya, but would love to hear about your experiences as you try and get things working there...

-Paul

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
Sent: Tuesday, July 9, 2019 7:58 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Running spdk in a virtualized env

Are there any special considerations to running spdk in a virtualized environment such as ESX (cpus, polling control, memory reservation, nvme passthrough)? Any suggestions/pointers will be appreciated.


Thanks,
Andrey
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk