* [Qemu-devel] Qemu migration with vhost-user-blk on top of local storage
@ 2019-01-09 10:23 wuzhouhui
2019-01-09 12:42 ` Stefan Hajnoczi
0 siblings, 1 reply; 9+ messages in thread
From: wuzhouhui @ 2019-01-09 10:23 UTC (permalink / raw)
To: qemu-devel; +Cc: lilin24, xieyongji
Hi everyone,
I'm working qemu with vhost target (e.g. spdk), and I attempt to migrate VM with
2 local storages. One local storage is a regular file, e.g. /tmp/c74.qcow2, and
the other is a malloc bdev that spdk created. This malloc bdev will exported to
VM via vhost-user-blk. When I execute following command:
virsh migrate --live --persistent --unsafe --undefinesource --copy-storage-all \
--p2p --auto-converge --verbose --desturi qemu+tcp://<uri>/system vm0
The libvirt reports:
qemu-2.12.1: error: internal error: unable to execute QEMU command \
'nbd-server-add': Cannot find device=drive-virtio-disk1 nor \
node_name=drive-virtio-disk1
Does it means that qemu with spdk on top of local storage don't support migration?
QEMU: 2.12.1
SPDK: 18.10
Thanks.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] Qemu migration with vhost-user-blk on top of local storage
2019-01-09 10:23 [Qemu-devel] Qemu migration with vhost-user-blk on top of local storage wuzhouhui
@ 2019-01-09 12:42 ` Stefan Hajnoczi
2019-09-28 2:06 ` [SPDK] " wuzhouhui
0 siblings, 1 reply; 9+ messages in thread
From: Stefan Hajnoczi @ 2019-01-09 12:42 UTC (permalink / raw)
To: wuzhouhui; +Cc: qemu-devel, xieyongji, lilin24, libvir-list, spdk
[-- Attachment #1: Type: text/plain, Size: 1078 bytes --]
On Wed, Jan 09, 2019 at 06:23:42PM +0800, wuzhouhui wrote:
> Hi everyone,
>
> I'm working qemu with vhost target (e.g. spdk), and I attempt to migrate VM with
> 2 local storages. One local storage is a regular file, e.g. /tmp/c74.qcow2, and
> the other is a malloc bdev that spdk created. This malloc bdev will exported to
> VM via vhost-user-blk. When I execute following command:
>
> virsh migrate --live --persistent --unsafe --undefinesource --copy-storage-all \
> --p2p --auto-converge --verbose --desturi qemu+tcp://<uri>/system vm0
>
> The libvirt reports:
>
> qemu-2.12.1: error: internal error: unable to execute QEMU command \
> 'nbd-server-add': Cannot find device=drive-virtio-disk1 nor \
> node_name=drive-virtio-disk1
Please post your libvirt domain XML.
> Does it means that qemu with spdk on top of local storage don't support migration?
>
> QEMU: 2.12.1
> SPDK: 18.10
vhost-user-blk bypasses the QEMU block layer, so NBD storage migration
at the QEMU level will not work for the vhost-user-blk disk.
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] Qemu migration with vhost-user-blk on top of local storage
@ 2019-09-28 2:06 ` wuzhouhui
0 siblings, 0 replies; 9+ messages in thread
From: wuzhouhui @ 2019-01-09 13:23 UTC (permalink / raw)
To: stefan hajnoczi; +Cc: qemu-devel, xieyongji, lilin24, libvir-list, spdk
> -----Original Messages-----
> From: "Stefan Hajnoczi" <stefanha@gmail.com>
> Sent Time: 2019-01-09 20:42:58 (Wednesday)
> To: wuzhouhui <wuzhouhui14@mails.ucas.ac.cn>
> Cc: qemu-devel@nongnu.org, xieyongji@baidu.com, lilin24@baidu.com, libvir-list@redhat.com, spdk@lists.01.org
> Subject: Re: [Qemu-devel] Qemu migration with vhost-user-blk on top of local storage
>
> On Wed, Jan 09, 2019 at 06:23:42PM +0800, wuzhouhui wrote:
> > Hi everyone,
> >
> > I'm working qemu with vhost target (e.g. spdk), and I attempt to migrate VM with
> > 2 local storages. One local storage is a regular file, e.g. /tmp/c74.qcow2, and
> > the other is a malloc bdev that spdk created. This malloc bdev will exported to
> > VM via vhost-user-blk. When I execute following command:
> >
> > virsh migrate --live --persistent --unsafe --undefinesource --copy-storage-all \
> > --p2p --auto-converge --verbose --desturi qemu+tcp://<uri>/system vm0
> >
> > The libvirt reports:
> >
> > qemu-2.12.1: error: internal error: unable to execute QEMU command \
> > 'nbd-server-add': Cannot find device=drive-virtio-disk1 nor \
> > node_name=drive-virtio-disk1
>
> Please post your libvirt domain XML.
My libvirt is based on libvirt-1.1.1-29.el7, and add many patches to satisfy our
own needs, e.g. add support for vhost-user-blk. Post domain xml may not useful.
Anyway, following is full contents of XML:
<domain type='kvm'>
<name>wzh</name>
<uuid>a84e96e6-2c53-408d-986b-c709bc6a0e51</uuid>
<memory unit='MiB'>4096</memory>
<memoryBacking>
<hugepages/>
</memoryBacking>
<currentMemory unit='MiB'>4096</currentMemory>
<vcpu placement='static' cpuset='16-31'>2</vcpu>
<os>
<type arch='x86_64' machine='pc'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
</features>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/data/wzh/x86_64-softmmu/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/data/wzh/c74.qcow2'/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
</disk>
<disk type='vhost-user-blk' device='disk'>
<source type='unix' path='/var/tmp/lv0' mode='client'>
</source>
<target dev='vdb' bus='virtio'/>
<driver queues='4'/>
</disk>
<controller type='usb' index='0'>
<alias name='usb0'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<serial type='pty'>
<target port='0'/>
<alias name='serial0'/>
</serial>
<serial type='pty'>
<target port='1'/>
<alias name='serial1'/>
</serial>
<input type='tablet' bus='usb'>
<alias name='input0'/>
</input>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' autoport='yes' listen='0.0.0.0' keymap='en-us'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<alias name='video0'/>
</video>
</devices>
<seclabel type='none'/>
</domain>
>
> > Does it means that qemu with spdk on top of local storage don't support migration?
> >
> > QEMU: 2.12.1
> > SPDK: 18.10
>
> vhost-user-blk bypasses the QEMU block layer, so NBD storage migration
> at the QEMU level will not work for the vhost-user-blk disk.
>
> Stefan
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [libvirt] Qemu migration with vhost-user-blk on top of local storage
2019-09-28 2:06 ` [SPDK] " wuzhouhui
(?)
@ 2019-01-09 14:13 ` Peter Krempa
-1 siblings, 0 replies; 9+ messages in thread
From: Peter Krempa @ 2019-01-09 14:13 UTC (permalink / raw)
To: wuzhouhui
Cc: stefan hajnoczi, libvir-list, xieyongji, spdk, qemu-devel, lilin24
[-- Attachment #1: Type: text/plain, Size: 1742 bytes --]
On Wed, Jan 09, 2019 at 21:23:25 +0800, wuzhouhui wrote:
> > -----Original Messages-----
> > From: "Stefan Hajnoczi" <stefanha@gmail.com>
> > Sent Time: 2019-01-09 20:42:58 (Wednesday)
> > To: wuzhouhui <wuzhouhui14@mails.ucas.ac.cn>
> > Cc: qemu-devel@nongnu.org, xieyongji@baidu.com, lilin24@baidu.com, libvir-list@redhat.com, spdk@lists.01.org
> > Subject: Re: [Qemu-devel] Qemu migration with vhost-user-blk on top of local storage
> >
> > On Wed, Jan 09, 2019 at 06:23:42PM +0800, wuzhouhui wrote:
> > > Hi everyone,
> > >
> > > I'm working qemu with vhost target (e.g. spdk), and I attempt to migrate VM with
> > > 2 local storages. One local storage is a regular file, e.g. /tmp/c74.qcow2, and
> > > the other is a malloc bdev that spdk created. This malloc bdev will exported to
> > > VM via vhost-user-blk. When I execute following command:
> > >
> > > virsh migrate --live --persistent --unsafe --undefinesource --copy-storage-all \
> > > --p2p --auto-converge --verbose --desturi qemu+tcp://<uri>/system vm0
> > >
> > > The libvirt reports:
> > >
> > > qemu-2.12.1: error: internal error: unable to execute QEMU command \
> > > 'nbd-server-add': Cannot find device=drive-virtio-disk1 nor \
> > > node_name=drive-virtio-disk1
> >
> > Please post your libvirt domain XML.
>
> My libvirt is based on libvirt-1.1.1-29.el7, and add many patches to satisfy our
> own needs, e.g. add support for vhost-user-blk. Post domain xml may not useful.
So you added support for vhost-user-blk, but did not fix the
NBD migration code. You are on your own, sorry. We can't support
arbitrary downstream changes.
Please either upstream your patches or make sure to skip it for
vhost-user.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [SPDK] Qemu migration with vhost-user-blk on top of local storage
@ 2019-01-09 16:19 ` Wodkowski, PawelX
0 siblings, 0 replies; 9+ messages in thread
From: Wodkowski, PawelX @ 2019-01-09 16:19 UTC (permalink / raw)
To: stefanha, spdk; +Cc: libvir-list, xieyongji, qemu-devel, lilin24
On Wed, 2019-01-09 at 21:23 +0800, wuzhouhui wrote:
> > -----Original Messages-----
> > From: "Stefan Hajnoczi" <stefanha@gmail.com>
> > Sent Time: 2019-01-09 20:42:58 (Wednesday)
> > To: wuzhouhui <wuzhouhui14@mails.ucas.ac.cn>
> > Cc: qemu-devel@nongnu.org, xieyongji@baidu.com, lilin24@baidu.com,
> > libvir-list@redhat.com, spdk@lists.01.org
> > Subject: Re: [Qemu-devel] Qemu migration with vhost-user-blk on top
> > of local storage
> >
> > On Wed, Jan 09, 2019 at 06:23:42PM +0800, wuzhouhui wrote:
> > > Hi everyone,
> > >
> > > I'm working qemu with vhost target (e.g. spdk), and I attempt to
> > > migrate VM with
> > > 2 local storages. One local storage is a regular file, e.g.
> > > /tmp/c74.qcow2, and
> > > the other is a malloc bdev that spdk created. This malloc bdev
> > > will exported to
> > > VM via vhost-user-blk. When I execute following command:
> > >
> > > virsh migrate --live --persistent --unsafe --undefinesource --
> > > copy-storage-all \
> > > --p2p --auto-converge --verbose --desturi
> > > qemu+tcp://<uri>/system vm0
> > >
> > > The libvirt reports:
> > >
> > > qemu-2.12.1: error: internal error: unable to execute QEMU
> > > command \
> > > 'nbd-server-add': Cannot find device=drive-virtio-disk1 nor \
> > > node_name=drive-virtio-disk1
> >
> > Please post your libvirt domain XML.
>
> My libvirt is based on libvirt-1.1.1-29.el7, and add many patches to
> satisfy our
> own needs, e.g. add support for vhost-user-blk. Post domain xml may
> not useful.
> Anyway, following is full contents of XML:
>
> <domain type='kvm'>
> <name>wzh</name>
> <uuid>a84e96e6-2c53-408d-986b-c709bc6a0e51</uuid>
> <memory unit='MiB'>4096</memory>
> <memoryBacking>
> <hugepages/>
> </memoryBacking>
> <currentMemory unit='MiB'>4096</currentMemory>
> <vcpu placement='static' cpuset='16-31'>2</vcpu>
> <os>
> <type arch='x86_64' machine='pc'>hvm</type>
> <boot dev='hd'/>
> </os>
> <features>
> <acpi/>
> </features>
> <clock offset='utc'>
> <timer name='rtc' tickpolicy='catchup'/>
> </clock>
> <on_poweroff>destroy</on_poweroff>
> <on_reboot>restart</on_reboot>
> <on_crash>destroy</on_crash>
> <devices>
> <emulator>/data/wzh/x86_64-softmmu/qemu-system-
> x86_64</emulator>
> <disk type='file' device='disk'>
> <driver name='qemu' type='qcow2' cache='none'/>
> <source file='/data/wzh/c74.qcow2'/>
> <target dev='vda' bus='virtio'/>
> <alias name='virtio-disk0'/>
> </disk>
>
> <disk type='vhost-user-blk' device='disk'>
> <source type='unix' path='/var/tmp/lv0' mode='client'>
> </source>
> <target dev='vdb' bus='virtio'/>
> <driver queues='4'/>
> </disk>
>
> <controller type='usb' index='0'>
> <alias name='usb0'/>
> </controller>
> <controller type='pci' index='0' model='pci-root'>
> <alias name='pci.0'/>
> </controller>
> <serial type='pty'>
> <target port='0'/>
> <alias name='serial0'/>
> </serial>
> <serial type='pty'>
> <target port='1'/>
> <alias name='serial1'/>
> </serial>
> <input type='tablet' bus='usb'>
> <alias name='input0'/>
> </input>
> <input type='mouse' bus='ps2'/>
> <graphics type='vnc' autoport='yes' listen='0.0.0.0'
> keymap='en-us'>
> <listen type='address' address='0.0.0.0'/>
> </graphics>
> <video>
> <model type='cirrus' vram='9216' heads='1'/>
> <alias name='video0'/>
> </video>
> </devices>
> <seclabel type='none'/>
> </domain>
>
> >
> > > Does it means that qemu with spdk on top of local storage don't
> > > support migration?
> > >
> > > QEMU: 2.12.1
> > > SPDK: 18.10
> >
> > vhost-user-blk bypasses the QEMU block layer, so NBD storage
> > migration
> > at the QEMU level will not work for the vhost-user-blk disk.
> >
> > Stefan
Don't know if this is the case that wuzhouhui is using but generally
migration should work if all storages are accesible from both machines.
The QEMU image and Malloc bdev should be exposed using some kind of
network sharing. SPDK on migration target should be configured the same
way as on source machine.
We are testing this issuing QEMU monitor commands so I can't help with
libvirt here.
Example setup/test case we are using in one of our tests:
Machine1:
Expose VM image over SSHF
Expose some block device using SPDK nvmf target
SPDK nvmf initiator:
- connects to Machine1 SPDK nvmf target to access block device
SPDK vhost-scsi:
- presents block device to QEMU from SPDK nvmf initiator:
QEMU VM uses
- shared VM image
- block device from SPDK nvmf initiator over vhost-user-scsi
Machine2:
SPDK nvmf initiator:
- connects to Machine1 SPDK nvmf target to access block device
SPDK vhost-scsi:
- presents block device to QEMU from nvmf
initiator:
QEMU instance - waiting for incoming migration
- shared VM image over SSHFS
- block device from SPDK nvmf initiator over vhost-user-scsi
Some traffic is generated using FIO and then we just migrate VM from
Machine1 to Machine2.
Pawel
>
> _______________________________________________
> SPDK mailing list
> SPDK@lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [SPDK] [Qemu-devel] Qemu migration with vhost-user-blk on top of local storage
@ 2019-01-09 16:19 ` Wodkowski, PawelX
0 siblings, 0 replies; 9+ messages in thread
From: Wodkowski, PawelX @ 2019-01-09 16:19 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 5446 bytes --]
On Wed, 2019-01-09 at 21:23 +0800, wuzhouhui wrote:
> > -----Original Messages-----
> > From: "Stefan Hajnoczi" <stefanha(a)gmail.com>
> > Sent Time: 2019-01-09 20:42:58 (Wednesday)
> > To: wuzhouhui <wuzhouhui14(a)mails.ucas.ac.cn>
> > Cc: qemu-devel(a)nongnu.org, xieyongji(a)baidu.com, lilin24(a)baidu.com,
> > libvir-list(a)redhat.com, spdk(a)lists.01.org
> > Subject: Re: [Qemu-devel] Qemu migration with vhost-user-blk on top
> > of local storage
> >
> > On Wed, Jan 09, 2019 at 06:23:42PM +0800, wuzhouhui wrote:
> > > Hi everyone,
> > >
> > > I'm working qemu with vhost target (e.g. spdk), and I attempt to
> > > migrate VM with
> > > 2 local storages. One local storage is a regular file, e.g.
> > > /tmp/c74.qcow2, and
> > > the other is a malloc bdev that spdk created. This malloc bdev
> > > will exported to
> > > VM via vhost-user-blk. When I execute following command:
> > >
> > > virsh migrate --live --persistent --unsafe --undefinesource --
> > > copy-storage-all \
> > > --p2p --auto-converge --verbose --desturi
> > > qemu+tcp://<uri>/system vm0
> > >
> > > The libvirt reports:
> > >
> > > qemu-2.12.1: error: internal error: unable to execute QEMU
> > > command \
> > > 'nbd-server-add': Cannot find device=drive-virtio-disk1 nor \
> > > node_name=drive-virtio-disk1
> >
> > Please post your libvirt domain XML.
>
> My libvirt is based on libvirt-1.1.1-29.el7, and add many patches to
> satisfy our
> own needs, e.g. add support for vhost-user-blk. Post domain xml may
> not useful.
> Anyway, following is full contents of XML:
>
> <domain type='kvm'>
> <name>wzh</name>
> <uuid>a84e96e6-2c53-408d-986b-c709bc6a0e51</uuid>
> <memory unit='MiB'>4096</memory>
> <memoryBacking>
> <hugepages/>
> </memoryBacking>
> <currentMemory unit='MiB'>4096</currentMemory>
> <vcpu placement='static' cpuset='16-31'>2</vcpu>
> <os>
> <type arch='x86_64' machine='pc'>hvm</type>
> <boot dev='hd'/>
> </os>
> <features>
> <acpi/>
> </features>
> <clock offset='utc'>
> <timer name='rtc' tickpolicy='catchup'/>
> </clock>
> <on_poweroff>destroy</on_poweroff>
> <on_reboot>restart</on_reboot>
> <on_crash>destroy</on_crash>
> <devices>
> <emulator>/data/wzh/x86_64-softmmu/qemu-system-
> x86_64</emulator>
> <disk type='file' device='disk'>
> <driver name='qemu' type='qcow2' cache='none'/>
> <source file='/data/wzh/c74.qcow2'/>
> <target dev='vda' bus='virtio'/>
> <alias name='virtio-disk0'/>
> </disk>
>
> <disk type='vhost-user-blk' device='disk'>
> <source type='unix' path='/var/tmp/lv0' mode='client'>
> </source>
> <target dev='vdb' bus='virtio'/>
> <driver queues='4'/>
> </disk>
>
> <controller type='usb' index='0'>
> <alias name='usb0'/>
> </controller>
> <controller type='pci' index='0' model='pci-root'>
> <alias name='pci.0'/>
> </controller>
> <serial type='pty'>
> <target port='0'/>
> <alias name='serial0'/>
> </serial>
> <serial type='pty'>
> <target port='1'/>
> <alias name='serial1'/>
> </serial>
> <input type='tablet' bus='usb'>
> <alias name='input0'/>
> </input>
> <input type='mouse' bus='ps2'/>
> <graphics type='vnc' autoport='yes' listen='0.0.0.0'
> keymap='en-us'>
> <listen type='address' address='0.0.0.0'/>
> </graphics>
> <video>
> <model type='cirrus' vram='9216' heads='1'/>
> <alias name='video0'/>
> </video>
> </devices>
> <seclabel type='none'/>
> </domain>
>
> >
> > > Does it means that qemu with spdk on top of local storage don't
> > > support migration?
> > >
> > > QEMU: 2.12.1
> > > SPDK: 18.10
> >
> > vhost-user-blk bypasses the QEMU block layer, so NBD storage
> > migration
> > at the QEMU level will not work for the vhost-user-blk disk.
> >
> > Stefan
Don't know if this is the case that wuzhouhui is using but generally
migration should work if all storages are accesible from both machines.
The QEMU image and Malloc bdev should be exposed using some kind of
network sharing. SPDK on migration target should be configured the same
way as on source machine.
We are testing this issuing QEMU monitor commands so I can't help with
libvirt here.
Example setup/test case we are using in one of our tests:
Machine1:
Expose VM image over SSHF
Expose some block device using SPDK nvmf target
SPDK nvmf initiator:
- connects to Machine1 SPDK nvmf target to access block device
SPDK vhost-scsi:
- presents block device to QEMU from SPDK nvmf initiator:
QEMU VM uses
- shared VM image
- block device from SPDK nvmf initiator over vhost-user-scsi
Machine2:
SPDK nvmf initiator:
- connects to Machine1 SPDK nvmf target to access block device
SPDK vhost-scsi:
- presents block device to QEMU from nvmf
initiator:
QEMU instance - waiting for incoming migration
- shared VM image over SSHFS
- block device from SPDK nvmf initiator over vhost-user-scsi
Some traffic is generated using FIO and then we just migrate VM from
Machine1 to Machine2.
Pawel
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [SPDK] Qemu migration with vhost-user-blk on top of local storage
@ 2019-09-28 2:06 ` wuzhouhui
0 siblings, 0 replies; 9+ messages in thread
From: wuzhouhui @ 2019-01-10 1:04 UTC (permalink / raw)
To: storage performance development kit
Cc: stefanha, libvir-list, xieyongji, qemu-devel, lilin24
> -----Original Messages-----
> From: "Wodkowski, PawelX" <pawelx.wodkowski@intel.com>
> Sent Time: 2019-01-10 00:19:48 (Thursday)
> To: "stefanha@gmail.com" <stefanha@gmail.com>, "spdk@lists.01.org" <spdk@lists.01.org>
> Cc: "libvir-list@redhat.com" <libvir-list@redhat.com>, "xieyongji@baidu.com" <xieyongji@baidu.com>, "qemu-devel@nongnu.org" <qemu-devel@nongnu.org>, "lilin24@baidu.com" <lilin24@baidu.com>
> Subject: Re: [SPDK] [Qemu-devel] Qemu migration with vhost-user-blk on top of local storage
>
> On Wed, 2019-01-09 at 21:23 +0800, wuzhouhui wrote:
> > > -----Original Messages-----
> > > From: "Stefan Hajnoczi" <stefanha@gmail.com>
> > > Sent Time: 2019-01-09 20:42:58 (Wednesday)
> > > To: wuzhouhui <wuzhouhui14@mails.ucas.ac.cn>
> > > Cc: qemu-devel@nongnu.org, xieyongji@baidu.com, lilin24@baidu.com,
> > > libvir-list@redhat.com, spdk@lists.01.org
> > > Subject: Re: [Qemu-devel] Qemu migration with vhost-user-blk on top
> > > of local storage
> > >
> > > On Wed, Jan 09, 2019 at 06:23:42PM +0800, wuzhouhui wrote:
> > > > Hi everyone,
> > > >
> > > > I'm working qemu with vhost target (e.g. spdk), and I attempt to
> > > > migrate VM with
> > > > 2 local storages. One local storage is a regular file, e.g.
> > > > /tmp/c74.qcow2, and
> > > > the other is a malloc bdev that spdk created. This malloc bdev
> > > > will exported to
> > > > VM via vhost-user-blk. When I execute following command:
> > > >
> > > > virsh migrate --live --persistent --unsafe --undefinesource --
> > > > copy-storage-all \
> > > > --p2p --auto-converge --verbose --desturi
> > > > qemu+tcp://<uri>/system vm0
> > > >
> > > > The libvirt reports:
> > > >
> > > > qemu-2.12.1: error: internal error: unable to execute QEMU
> > > > command \
> > > > 'nbd-server-add': Cannot find device=drive-virtio-disk1 nor \
> > > > node_name=drive-virtio-disk1
> > >
> > > Please post your libvirt domain XML.
> >
> > My libvirt is based on libvirt-1.1.1-29.el7, and add many patches to
> > satisfy our
> > own needs, e.g. add support for vhost-user-blk. Post domain xml may
> > not useful.
> > Anyway, following is full contents of XML:
> >
> > <domain type='kvm'>
> > <name>wzh</name>
> > <uuid>a84e96e6-2c53-408d-986b-c709bc6a0e51</uuid>
> > <memory unit='MiB'>4096</memory>
> > <memoryBacking>
> > <hugepages/>
> > </memoryBacking>
> > <currentMemory unit='MiB'>4096</currentMemory>
> > <vcpu placement='static' cpuset='16-31'>2</vcpu>
> > <os>
> > <type arch='x86_64' machine='pc'>hvm</type>
> > <boot dev='hd'/>
> > </os>
> > <features>
> > <acpi/>
> > </features>
> > <clock offset='utc'>
> > <timer name='rtc' tickpolicy='catchup'/>
> > </clock>
> > <on_poweroff>destroy</on_poweroff>
> > <on_reboot>restart</on_reboot>
> > <on_crash>destroy</on_crash>
> > <devices>
> > <emulator>/data/wzh/x86_64-softmmu/qemu-system-
> > x86_64</emulator>
> > <disk type='file' device='disk'>
> > <driver name='qemu' type='qcow2' cache='none'/>
> > <source file='/data/wzh/c74.qcow2'/>
> > <target dev='vda' bus='virtio'/>
> > <alias name='virtio-disk0'/>
> > </disk>
> >
> > <disk type='vhost-user-blk' device='disk'>
> > <source type='unix' path='/var/tmp/lv0' mode='client'>
> > </source>
> > <target dev='vdb' bus='virtio'/>
> > <driver queues='4'/>
> > </disk>
> >
> > <controller type='usb' index='0'>
> > <alias name='usb0'/>
> > </controller>
> > <controller type='pci' index='0' model='pci-root'>
> > <alias name='pci.0'/>
> > </controller>
> > <serial type='pty'>
> > <target port='0'/>
> > <alias name='serial0'/>
> > </serial>
> > <serial type='pty'>
> > <target port='1'/>
> > <alias name='serial1'/>
> > </serial>
> > <input type='tablet' bus='usb'>
> > <alias name='input0'/>
> > </input>
> > <input type='mouse' bus='ps2'/>
> > <graphics type='vnc' autoport='yes' listen='0.0.0.0'
> > keymap='en-us'>
> > <listen type='address' address='0.0.0.0'/>
> > </graphics>
> > <video>
> > <model type='cirrus' vram='9216' heads='1'/>
> > <alias name='video0'/>
> > </video>
> > </devices>
> > <seclabel type='none'/>
> > </domain>
> >
> > >
> > > > Does it means that qemu with spdk on top of local storage don't
> > > > support migration?
> > > >
> > > > QEMU: 2.12.1
> > > > SPDK: 18.10
> > >
> > > vhost-user-blk bypasses the QEMU block layer, so NBD storage
> > > migration
> > > at the QEMU level will not work for the vhost-user-blk disk.
> > >
> > > Stefan
>
>
> Don't know if this is the case that wuzhouhui is using but generally
My case is different from yours. All storages (including qemu image and malloc
bdev) are local, and only accessible from one of hosts. Anyway, thanks for
everyone's response.
wuzhouhui
> migration should work if all storages are accesible from both machines.
> The QEMU image and Malloc bdev should be exposed using some kind of
> network sharing. SPDK on migration target should be configured the same
> way as on source machine.
>
> We are testing this issuing QEMU monitor commands so I can't help with
> libvirt here.
>
> Example setup/test case we are using in one of our tests:
>
> Machine1:
> Expose VM image over SSHF
> Expose some block device using SPDK nvmf target
> SPDK nvmf initiator:
> - connects to Machine1 SPDK nvmf target to access block device
> SPDK vhost-scsi:
> - presents block device to QEMU from SPDK nvmf initiator:
> QEMU VM uses
> - shared VM image
> - block device from SPDK nvmf initiator over vhost-user-scsi
>
> Machine2:
> SPDK nvmf initiator:
> - connects to Machine1 SPDK nvmf target to access block device
> SPDK vhost-scsi:
> - presents block device to QEMU from nvmf
> initiator:
> QEMU instance - waiting for incoming migration
> - shared VM image over SSHFS
> - block device from SPDK nvmf initiator over vhost-user-scsi
>
> Some traffic is generated using FIO and then we just migrate VM from
> Machine1 to Machine2.
>
> Pawel
>
> >
> > _______________________________________________
> > SPDK mailing list
> > SPDK@lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK@lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [SPDK] [Qemu-devel] Qemu migration with vhost-user-blk on top of local storage
@ 2019-09-28 2:06 ` wuzhouhui
0 siblings, 0 replies; 9+ messages in thread
From: wuzhouhui @ 2019-09-28 2:06 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 3670 bytes --]
> -----Original Messages-----
> From: "Stefan Hajnoczi" <stefanha(a)gmail.com>
> Sent Time: 2019-01-09 20:42:58 (Wednesday)
> To: wuzhouhui <wuzhouhui14(a)mails.ucas.ac.cn>
> Cc: qemu-devel(a)nongnu.org, xieyongji(a)baidu.com, lilin24(a)baidu.com, libvir-list(a)redhat.com, spdk(a)lists.01.org
> Subject: Re: [Qemu-devel] Qemu migration with vhost-user-blk on top of local storage
>
> On Wed, Jan 09, 2019 at 06:23:42PM +0800, wuzhouhui wrote:
> > Hi everyone,
> >
> > I'm working qemu with vhost target (e.g. spdk), and I attempt to migrate VM with
> > 2 local storages. One local storage is a regular file, e.g. /tmp/c74.qcow2, and
> > the other is a malloc bdev that spdk created. This malloc bdev will exported to
> > VM via vhost-user-blk. When I execute following command:
> >
> > virsh migrate --live --persistent --unsafe --undefinesource --copy-storage-all \
> > --p2p --auto-converge --verbose --desturi qemu+tcp://<uri>/system vm0
> >
> > The libvirt reports:
> >
> > qemu-2.12.1: error: internal error: unable to execute QEMU command \
> > 'nbd-server-add': Cannot find device=drive-virtio-disk1 nor \
> > node_name=drive-virtio-disk1
>
> Please post your libvirt domain XML.
My libvirt is based on libvirt-1.1.1-29.el7, and add many patches to satisfy our
own needs, e.g. add support for vhost-user-blk. Post domain xml may not useful.
Anyway, following is full contents of XML:
<domain type='kvm'>
<name>wzh</name>
<uuid>a84e96e6-2c53-408d-986b-c709bc6a0e51</uuid>
<memory unit='MiB'>4096</memory>
<memoryBacking>
<hugepages/>
</memoryBacking>
<currentMemory unit='MiB'>4096</currentMemory>
<vcpu placement='static' cpuset='16-31'>2</vcpu>
<os>
<type arch='x86_64' machine='pc'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
</features>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/data/wzh/x86_64-softmmu/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/data/wzh/c74.qcow2'/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
</disk>
<disk type='vhost-user-blk' device='disk'>
<source type='unix' path='/var/tmp/lv0' mode='client'>
</source>
<target dev='vdb' bus='virtio'/>
<driver queues='4'/>
</disk>
<controller type='usb' index='0'>
<alias name='usb0'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<serial type='pty'>
<target port='0'/>
<alias name='serial0'/>
</serial>
<serial type='pty'>
<target port='1'/>
<alias name='serial1'/>
</serial>
<input type='tablet' bus='usb'>
<alias name='input0'/>
</input>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' autoport='yes' listen='0.0.0.0' keymap='en-us'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<alias name='video0'/>
</video>
</devices>
<seclabel type='none'/>
</domain>
>
> > Does it means that qemu with spdk on top of local storage don't support migration?
> >
> > QEMU: 2.12.1
> > SPDK: 18.10
>
> vhost-user-blk bypasses the QEMU block layer, so NBD storage migration
> at the QEMU level will not work for the vhost-user-blk disk.
>
> Stefan
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [SPDK] [Qemu-devel] Qemu migration with vhost-user-blk on top of local storage
@ 2019-09-28 2:06 ` wuzhouhui
0 siblings, 0 replies; 9+ messages in thread
From: wuzhouhui @ 2019-09-28 2:06 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 6655 bytes --]
> -----Original Messages-----
> From: "Wodkowski, PawelX" <pawelx.wodkowski(a)intel.com>
> Sent Time: 2019-01-10 00:19:48 (Thursday)
> To: "stefanha(a)gmail.com" <stefanha(a)gmail.com>, "spdk(a)lists.01.org" <spdk(a)lists.01.org>
> Cc: "libvir-list(a)redhat.com" <libvir-list(a)redhat.com>, "xieyongji(a)baidu.com" <xieyongji(a)baidu.com>, "qemu-devel(a)nongnu.org" <qemu-devel(a)nongnu.org>, "lilin24(a)baidu.com" <lilin24(a)baidu.com>
> Subject: Re: [SPDK] [Qemu-devel] Qemu migration with vhost-user-blk on top of local storage
>
> On Wed, 2019-01-09 at 21:23 +0800, wuzhouhui wrote:
> > > -----Original Messages-----
> > > From: "Stefan Hajnoczi" <stefanha(a)gmail.com>
> > > Sent Time: 2019-01-09 20:42:58 (Wednesday)
> > > To: wuzhouhui <wuzhouhui14(a)mails.ucas.ac.cn>
> > > Cc: qemu-devel(a)nongnu.org, xieyongji(a)baidu.com, lilin24(a)baidu.com,
> > > libvir-list(a)redhat.com, spdk(a)lists.01.org
> > > Subject: Re: [Qemu-devel] Qemu migration with vhost-user-blk on top
> > > of local storage
> > >
> > > On Wed, Jan 09, 2019 at 06:23:42PM +0800, wuzhouhui wrote:
> > > > Hi everyone,
> > > >
> > > > I'm working qemu with vhost target (e.g. spdk), and I attempt to
> > > > migrate VM with
> > > > 2 local storages. One local storage is a regular file, e.g.
> > > > /tmp/c74.qcow2, and
> > > > the other is a malloc bdev that spdk created. This malloc bdev
> > > > will exported to
> > > > VM via vhost-user-blk. When I execute following command:
> > > >
> > > > virsh migrate --live --persistent --unsafe --undefinesource --
> > > > copy-storage-all \
> > > > --p2p --auto-converge --verbose --desturi
> > > > qemu+tcp://<uri>/system vm0
> > > >
> > > > The libvirt reports:
> > > >
> > > > qemu-2.12.1: error: internal error: unable to execute QEMU
> > > > command \
> > > > 'nbd-server-add': Cannot find device=drive-virtio-disk1 nor \
> > > > node_name=drive-virtio-disk1
> > >
> > > Please post your libvirt domain XML.
> >
> > My libvirt is based on libvirt-1.1.1-29.el7, and add many patches to
> > satisfy our
> > own needs, e.g. add support for vhost-user-blk. Post domain xml may
> > not useful.
> > Anyway, following is full contents of XML:
> >
> > <domain type='kvm'>
> > <name>wzh</name>
> > <uuid>a84e96e6-2c53-408d-986b-c709bc6a0e51</uuid>
> > <memory unit='MiB'>4096</memory>
> > <memoryBacking>
> > <hugepages/>
> > </memoryBacking>
> > <currentMemory unit='MiB'>4096</currentMemory>
> > <vcpu placement='static' cpuset='16-31'>2</vcpu>
> > <os>
> > <type arch='x86_64' machine='pc'>hvm</type>
> > <boot dev='hd'/>
> > </os>
> > <features>
> > <acpi/>
> > </features>
> > <clock offset='utc'>
> > <timer name='rtc' tickpolicy='catchup'/>
> > </clock>
> > <on_poweroff>destroy</on_poweroff>
> > <on_reboot>restart</on_reboot>
> > <on_crash>destroy</on_crash>
> > <devices>
> > <emulator>/data/wzh/x86_64-softmmu/qemu-system-
> > x86_64</emulator>
> > <disk type='file' device='disk'>
> > <driver name='qemu' type='qcow2' cache='none'/>
> > <source file='/data/wzh/c74.qcow2'/>
> > <target dev='vda' bus='virtio'/>
> > <alias name='virtio-disk0'/>
> > </disk>
> >
> > <disk type='vhost-user-blk' device='disk'>
> > <source type='unix' path='/var/tmp/lv0' mode='client'>
> > </source>
> > <target dev='vdb' bus='virtio'/>
> > <driver queues='4'/>
> > </disk>
> >
> > <controller type='usb' index='0'>
> > <alias name='usb0'/>
> > </controller>
> > <controller type='pci' index='0' model='pci-root'>
> > <alias name='pci.0'/>
> > </controller>
> > <serial type='pty'>
> > <target port='0'/>
> > <alias name='serial0'/>
> > </serial>
> > <serial type='pty'>
> > <target port='1'/>
> > <alias name='serial1'/>
> > </serial>
> > <input type='tablet' bus='usb'>
> > <alias name='input0'/>
> > </input>
> > <input type='mouse' bus='ps2'/>
> > <graphics type='vnc' autoport='yes' listen='0.0.0.0'
> > keymap='en-us'>
> > <listen type='address' address='0.0.0.0'/>
> > </graphics>
> > <video>
> > <model type='cirrus' vram='9216' heads='1'/>
> > <alias name='video0'/>
> > </video>
> > </devices>
> > <seclabel type='none'/>
> > </domain>
> >
> > >
> > > > Does it means that qemu with spdk on top of local storage don't
> > > > support migration?
> > > >
> > > > QEMU: 2.12.1
> > > > SPDK: 18.10
> > >
> > > vhost-user-blk bypasses the QEMU block layer, so NBD storage
> > > migration
> > > at the QEMU level will not work for the vhost-user-blk disk.
> > >
> > > Stefan
>
>
> Don't know if this is the case that wuzhouhui is using but generally
My case is different from yours. All storages (including qemu image and malloc
bdev) are local, and only accessible from one of hosts. Anyway, thanks for
everyone's response.
wuzhouhui
> migration should work if all storages are accesible from both machines.
> The QEMU image and Malloc bdev should be exposed using some kind of
> network sharing. SPDK on migration target should be configured the same
> way as on source machine.
>
> We are testing this issuing QEMU monitor commands so I can't help with
> libvirt here.
>
> Example setup/test case we are using in one of our tests:
>
> Machine1:
> Expose VM image over SSHF
> Expose some block device using SPDK nvmf target
> SPDK nvmf initiator:
> - connects to Machine1 SPDK nvmf target to access block device
> SPDK vhost-scsi:
> - presents block device to QEMU from SPDK nvmf initiator:
> QEMU VM uses
> - shared VM image
> - block device from SPDK nvmf initiator over vhost-user-scsi
>
> Machine2:
> SPDK nvmf initiator:
> - connects to Machine1 SPDK nvmf target to access block device
> SPDK vhost-scsi:
> - presents block device to QEMU from nvmf
> initiator:
> QEMU instance - waiting for incoming migration
> - shared VM image over SSHFS
> - block device from SPDK nvmf initiator over vhost-user-scsi
>
> Some traffic is generated using FIO and then we just migrate VM from
> Machine1 to Machine2.
>
> Pawel
>
> >
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2019-09-28 2:06 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-01-09 10:23 [Qemu-devel] Qemu migration with vhost-user-blk on top of local storage wuzhouhui
2019-01-09 12:42 ` Stefan Hajnoczi
2019-01-09 13:23 ` wuzhouhui
2019-09-28 2:06 ` [SPDK] " wuzhouhui
2019-01-09 14:13 ` [Qemu-devel] [libvirt] " Peter Krempa
2019-01-09 16:19 ` [Qemu-devel] [SPDK] " Wodkowski, PawelX
2019-01-09 16:19 ` [SPDK] [Qemu-devel] " Wodkowski, PawelX
2019-01-10 1:04 ` [Qemu-devel] [SPDK] " wuzhouhui
2019-09-28 2:06 ` [SPDK] [Qemu-devel] " wuzhouhui
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.