* [Qemu-devel] How live migration work for vhost-user @ 2019-04-15 8:50 fengyd 2019-04-15 14:53 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 9+ messages in thread From: fengyd @ 2019-04-15 8:50 UTC (permalink / raw) To: qemu-devel Hi, During live migration, the folloing log can see in nova-compute.log in my environment: ERROR nova.virt.libvirt.driver [req-039a85e1-e7a1-4a63-bc6d-c4b9a044aab6 0cdab20dc79f4bc6ae5790e7b4a898ac 3363c319773549178acc67f32c78310e - default default] [instance: 5ec719f4-1865-4afe-a207-3d9fae22c410] Live Migration failure: internal error: qemu unexpectedly closed the monitor: 2019-04-15T02:58:22.213897Z qemu-kvm: VQ 0 size 0x100 < last_avail_idx 0x1e - used_idx 0x23 It's OK for standard Linux VM, but not OK for our VM where virtio is implemented by ourself. KVM version as follow: qemu-kvm-common-ev-2.12.0-18.el7_6.3.1.x86_64 qemu-kvm-ev-2.12.0-18.el7_6.3.1.x86_64 libvirt-daemon-kvm-3.9.0-14.2.el7.centos.ncir.8.x86_64 Do you know what's the difference between virtio and vhost-user during migration? The function virtio_load in Qemu is called for virtio and vhost-user during migration. For virtio, last_avail_idx and used_idx are stored in Qemu, Qemu is responsible for updating their values accordingly For vhost-user, last_avail_idx and used_idx are stored in vhost-user app, eg. DPDK, not in Qemu? How does migration work for vhost-user? Thanks in advance Yafeng ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] How live migration work for vhost-user 2019-04-15 8:50 [Qemu-devel] How live migration work for vhost-user fengyd @ 2019-04-15 14:53 ` Dr. David Alan Gilbert [not found] ` <CACnvu8a5DjAUhfAGmi6GtpD1V2hfoNLTkKEgbsFuL_6GrRc+pg@mail.gmail.com> 0 siblings, 1 reply; 9+ messages in thread From: Dr. David Alan Gilbert @ 2019-04-15 14:53 UTC (permalink / raw) To: fengyd; +Cc: qemu-devel * fengyd (fengyd81@gmail.com) wrote: > Hi, > > During live migration, the folloing log can see in nova-compute.log in my > environment: > ERROR nova.virt.libvirt.driver [req-039a85e1-e7a1-4a63-bc6d-c4b9a044aab6 > 0cdab20dc79f4bc6ae5790e7b4a898ac 3363c319773549178acc67f32c78310e - default > default] [instance: 5ec719f4-1865-4afe-a207-3d9fae22c410] Live Migration > failure: internal error: qemu unexpectedly closed the monitor: > 2019-04-15T02:58:22.213897Z qemu-kvm: VQ 0 > size 0x100 < last_avail_idx 0x1e - used_idx 0x23 > > It's OK for standard Linux VM, but not OK for our VM where virtio is > implemented by ourself. > KVM version as follow: > qemu-kvm-common-ev-2.12.0-18.el7_6.3.1.x86_64 > qemu-kvm-ev-2.12.0-18.el7_6.3.1.x86_64 > libvirt-daemon-kvm-3.9.0-14.2.el7.centos.ncir.8.x86_64 > > Do you know what's the difference between virtio and vhost-user during > migration? > The function virtio_load in Qemu is called for virtio and vhost-user during > migration. > For virtio, last_avail_idx and used_idx are stored in Qemu, Qemu is > responsible for updating their values accordingly > For vhost-user, last_avail_idx and used_idx are stored in vhost-user app, > eg. DPDK, not in Qemu? > How does migration work for vhost-user? I don't know the details, but my understanding is that vhost-user tells the vhost-user client about an area of 'log' memory, where the vhost-user client must mark pages as dirty. In the qemu source, see docs/interop/vhost-user.txt and see the VHOST_SET_LOG_BASE and VHOST_USER_SET_LOG_FD calls. If the client correctly marks the areas as dirty, then qemu should resend those pages across. Dave > Thanks in advance > Yafeng -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 9+ messages in thread
[parent not found: <CACnvu8a5DjAUhfAGmi6GtpD1V2hfoNLTkKEgbsFuL_6GrRc+pg@mail.gmail.com>]
* [Qemu-devel] Fwd: How live migration work for vhost-user [not found] ` <CACnvu8a5DjAUhfAGmi6GtpD1V2hfoNLTkKEgbsFuL_6GrRc+pg@mail.gmail.com> @ 2019-04-16 7:59 ` fengyd 2019-04-16 8:47 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 9+ messages in thread From: fengyd @ 2019-04-16 7:59 UTC (permalink / raw) To: qemu-devel ---------- Forwarded message --------- From: fengyd <fengyd81@gmail.com> Date: Tue, 16 Apr 2019 at 09:17 Subject: Re: [Qemu-devel] How live migration work for vhost-user To: Dr. David Alan Gilbert <dgilbert@redhat.com> Hi, Any special feature needs to be supported on guest driver? Because it's OK for standard Linux VM, but not OK for our VM where virtio is implemented by ourself. And with qemu-kvm-ev-2.6, live migration can work with our VM where virtio is implemented by ourself. Thanks Yafeng On Mon, 15 Apr 2019 at 22:54, Dr. David Alan Gilbert <dgilbert@redhat.com> wrote: > * fengyd (fengyd81@gmail.com) wrote: > > Hi, > > > > During live migration, the folloing log can see in nova-compute.log in > my > > environment: > > ERROR nova.virt.libvirt.driver [req-039a85e1-e7a1-4a63-bc6d-c4b9a044aab6 > > 0cdab20dc79f4bc6ae5790e7b4a898ac 3363c319773549178acc67f32c78310e - > default > > default] [instance: 5ec719f4-1865-4afe-a207-3d9fae22c410] Live Migration > > failure: internal error: qemu unexpectedly closed the monitor: > > 2019-04-15T02:58:22.213897Z qemu-kvm: VQ 0 > > size 0x100 < last_avail_idx 0x1e - used_idx 0x23 > > > > It's OK for standard Linux VM, but not OK for our VM where virtio is > > implemented by ourself. > > KVM version as follow: > > qemu-kvm-common-ev-2.12.0-18.el7_6.3.1.x86_64 > > qemu-kvm-ev-2.12.0-18.el7_6.3.1.x86_64 > > libvirt-daemon-kvm-3.9.0-14.2.el7.centos.ncir.8.x86_64 > > > > Do you know what's the difference between virtio and vhost-user during > > migration? > > The function virtio_load in Qemu is called for virtio and vhost-user > during > > migration. > > For virtio, last_avail_idx and used_idx are stored in Qemu, Qemu is > > responsible for updating their values accordingly > > For vhost-user, last_avail_idx and used_idx are stored in vhost-user > app, > > eg. DPDK, not in Qemu? > > How does migration work for vhost-user? > > I don't know the details, but my understanding is that vhost-user > tells the vhost-user client about an area of 'log' memory, where the > vhost-user client must mark pages as dirty. > > In the qemu source, see docs/interop/vhost-user.txt and see > the VHOST_SET_LOG_BASE and VHOST_USER_SET_LOG_FD calls. > > If the client correctly marks the areas as dirty, then qemu > should resend those pages across. > > > Dave > > > Thanks in advance > > Yafeng > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] Fwd: How live migration work for vhost-user 2019-04-16 7:59 ` [Qemu-devel] Fwd: " fengyd @ 2019-04-16 8:47 ` Dr. David Alan Gilbert 2019-04-23 6:20 ` fengyd 0 siblings, 1 reply; 9+ messages in thread From: Dr. David Alan Gilbert @ 2019-04-16 8:47 UTC (permalink / raw) To: fengyd; +Cc: qemu-devel * fengyd (fengyd81@gmail.com) wrote: > ---------- Forwarded message --------- > From: fengyd <fengyd81@gmail.com> > Date: Tue, 16 Apr 2019 at 09:17 > Subject: Re: [Qemu-devel] How live migration work for vhost-user > To: Dr. David Alan Gilbert <dgilbert@redhat.com> > > > Hi, > > Any special feature needs to be supported on guest driver? > Because it's OK for standard Linux VM, but not OK for our VM where virtio > is implemented by ourself. I'm not sure; you do have to support that 'log' mechanism but I don't know what else is needed. > And with qemu-kvm-ev-2.6, live migration can work with our VM where virtio > is implemented by ourself. 2.6 is pretty old, so there's a lot of changes - not sure what's relevant. Dave > Thanks > Yafeng > > On Mon, 15 Apr 2019 at 22:54, Dr. David Alan Gilbert <dgilbert@redhat.com> > wrote: > > > * fengyd (fengyd81@gmail.com) wrote: > > > Hi, > > > > > > During live migration, the folloing log can see in nova-compute.log in > > my > > > environment: > > > ERROR nova.virt.libvirt.driver [req-039a85e1-e7a1-4a63-bc6d-c4b9a044aab6 > > > 0cdab20dc79f4bc6ae5790e7b4a898ac 3363c319773549178acc67f32c78310e - > > default > > > default] [instance: 5ec719f4-1865-4afe-a207-3d9fae22c410] Live Migration > > > failure: internal error: qemu unexpectedly closed the monitor: > > > 2019-04-15T02:58:22.213897Z qemu-kvm: VQ 0 > > > size 0x100 < last_avail_idx 0x1e - used_idx 0x23 > > > > > > It's OK for standard Linux VM, but not OK for our VM where virtio is > > > implemented by ourself. > > > KVM version as follow: > > > qemu-kvm-common-ev-2.12.0-18.el7_6.3.1.x86_64 > > > qemu-kvm-ev-2.12.0-18.el7_6.3.1.x86_64 > > > libvirt-daemon-kvm-3.9.0-14.2.el7.centos.ncir.8.x86_64 > > > > > > Do you know what's the difference between virtio and vhost-user during > > > migration? > > > The function virtio_load in Qemu is called for virtio and vhost-user > > during > > > migration. > > > For virtio, last_avail_idx and used_idx are stored in Qemu, Qemu is > > > responsible for updating their values accordingly > > > For vhost-user, last_avail_idx and used_idx are stored in vhost-user > > app, > > > eg. DPDK, not in Qemu? > > > How does migration work for vhost-user? > > > > I don't know the details, but my understanding is that vhost-user > > tells the vhost-user client about an area of 'log' memory, where the > > vhost-user client must mark pages as dirty. > > > > In the qemu source, see docs/interop/vhost-user.txt and see > > the VHOST_SET_LOG_BASE and VHOST_USER_SET_LOG_FD calls. > > > > If the client correctly marks the areas as dirty, then qemu > > should resend those pages across. > > > > > > Dave > > > > > Thanks in advance > > > Yafeng > > -- > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] Fwd: How live migration work for vhost-user 2019-04-16 8:47 ` Dr. David Alan Gilbert @ 2019-04-23 6:20 ` fengyd 2019-04-24 3:14 ` fengyd 2019-04-29 16:15 ` [Qemu-devel] " Dr. David Alan Gilbert 0 siblings, 2 replies; 9+ messages in thread From: fengyd @ 2019-04-23 6:20 UTC (permalink / raw) To: Dr. David Alan Gilbert; +Cc: qemu-devel Hi, I want to add some log to qemu-kvm-ev. Do you know how to compile qemu-kvm-ev from source code? Thanks Yafeng On Tue, 16 Apr 2019 at 16:47, Dr. David Alan Gilbert <dgilbert@redhat.com> wrote: > * fengyd (fengyd81@gmail.com) wrote: > > ---------- Forwarded message --------- > > From: fengyd <fengyd81@gmail.com> > > Date: Tue, 16 Apr 2019 at 09:17 > > Subject: Re: [Qemu-devel] How live migration work for vhost-user > > To: Dr. David Alan Gilbert <dgilbert@redhat.com> > > > > > > Hi, > > > > Any special feature needs to be supported on guest driver? > > Because it's OK for standard Linux VM, but not OK for our VM where virtio > > is implemented by ourself. > > I'm not sure; you do have to support that 'log' mechanism but I don't > know what else is needed. > > > And with qemu-kvm-ev-2.6, live migration can work with our VM where > virtio > > is implemented by ourself. > > 2.6 is pretty old, so there's a lot of changes - not sure what's > relevant. > > Dave > > > Thanks > > Yafeng > > > > On Mon, 15 Apr 2019 at 22:54, Dr. David Alan Gilbert < > dgilbert@redhat.com> > > wrote: > > > > > * fengyd (fengyd81@gmail.com) wrote: > > > > Hi, > > > > > > > > During live migration, the folloing log can see in nova-compute.log > in > > > my > > > > environment: > > > > ERROR nova.virt.libvirt.driver > [req-039a85e1-e7a1-4a63-bc6d-c4b9a044aab6 > > > > 0cdab20dc79f4bc6ae5790e7b4a898ac 3363c319773549178acc67f32c78310e - > > > default > > > > default] [instance: 5ec719f4-1865-4afe-a207-3d9fae22c410] Live > Migration > > > > failure: internal error: qemu unexpectedly closed the monitor: > > > > 2019-04-15T02:58:22.213897Z qemu-kvm: VQ 0 > > > > size 0x100 < last_avail_idx 0x1e - used_idx 0x23 > > > > > > > > It's OK for standard Linux VM, but not OK for our VM where virtio is > > > > implemented by ourself. > > > > KVM version as follow: > > > > qemu-kvm-common-ev-2.12.0-18.el7_6.3.1.x86_64 > > > > qemu-kvm-ev-2.12.0-18.el7_6.3.1.x86_64 > > > > libvirt-daemon-kvm-3.9.0-14.2.el7.centos.ncir.8.x86_64 > > > > > > > > Do you know what's the difference between virtio and vhost-user > during > > > > migration? > > > > The function virtio_load in Qemu is called for virtio and vhost-user > > > during > > > > migration. > > > > For virtio, last_avail_idx and used_idx are stored in Qemu, Qemu is > > > > responsible for updating their values accordingly > > > > For vhost-user, last_avail_idx and used_idx are stored in vhost-user > > > app, > > > > eg. DPDK, not in Qemu? > > > > How does migration work for vhost-user? > > > > > > I don't know the details, but my understanding is that vhost-user > > > tells the vhost-user client about an area of 'log' memory, where the > > > vhost-user client must mark pages as dirty. > > > > > > In the qemu source, see docs/interop/vhost-user.txt and see > > > the VHOST_SET_LOG_BASE and VHOST_USER_SET_LOG_FD calls. > > > > > > If the client correctly marks the areas as dirty, then qemu > > > should resend those pages across. > > > > > > > > > Dave > > > > > > > Thanks in advance > > > > Yafeng > > > -- > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > > > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] Fwd: How live migration work for vhost-user 2019-04-23 6:20 ` fengyd @ 2019-04-24 3:14 ` fengyd 2019-04-29 16:22 ` Dr. David Alan Gilbert 2019-04-29 16:15 ` [Qemu-devel] " Dr. David Alan Gilbert 1 sibling, 1 reply; 9+ messages in thread From: fengyd @ 2019-04-24 3:14 UTC (permalink / raw) To: Dr. David Alan Gilbert; +Cc: qemu-devel Hi, For vhost, *last_avail_idx* is maintained in vhost_virtqueue but during live migration, *last_avail_idx* is fetched from VirtQueue. Do you know how these two *last_avail_idx *are synchronized? virtio_load related code which is called during live migration: * vdev->vq[i].inuse = (uint16_t)(vdev->vq[i].last_avail_idx -* * vdev->vq[i].used_idx);* * if (vdev->vq[i].inuse > vdev->vq[i].vring.num) {* * error_report("VQ %d size 0x%x < last_avail_idx 0x%x - "* * "used_idx 0x%x",* * i, vdev->vq[i].vring.num,* * vdev->vq[i].last_avail_idx,* Thanks On Tue, 23 Apr 2019 at 14:20, fengyd <fengyd81@gmail.com> wrote: > Hi, > > I want to add some log to qemu-kvm-ev. > Do you know how to compile qemu-kvm-ev from source code? > > Thanks > > Yafeng > > On Tue, 16 Apr 2019 at 16:47, Dr. David Alan Gilbert <dgilbert@redhat.com> > wrote: > >> * fengyd (fengyd81@gmail.com) wrote: >> > ---------- Forwarded message --------- >> > From: fengyd <fengyd81@gmail.com> >> > Date: Tue, 16 Apr 2019 at 09:17 >> > Subject: Re: [Qemu-devel] How live migration work for vhost-user >> > To: Dr. David Alan Gilbert <dgilbert@redhat.com> >> > >> > >> > Hi, >> > >> > Any special feature needs to be supported on guest driver? >> > Because it's OK for standard Linux VM, but not OK for our VM where >> virtio >> > is implemented by ourself. >> >> I'm not sure; you do have to support that 'log' mechanism but I don't >> know what else is needed. >> >> > And with qemu-kvm-ev-2.6, live migration can work with our VM where >> virtio >> > is implemented by ourself. >> >> 2.6 is pretty old, so there's a lot of changes - not sure what's >> relevant. >> >> Dave >> >> > Thanks >> > Yafeng >> > >> > On Mon, 15 Apr 2019 at 22:54, Dr. David Alan Gilbert < >> dgilbert@redhat.com> >> > wrote: >> > >> > > * fengyd (fengyd81@gmail.com) wrote: >> > > > Hi, >> > > > >> > > > During live migration, the folloing log can see in >> nova-compute.log in >> > > my >> > > > environment: >> > > > ERROR nova.virt.libvirt.driver >> [req-039a85e1-e7a1-4a63-bc6d-c4b9a044aab6 >> > > > 0cdab20dc79f4bc6ae5790e7b4a898ac 3363c319773549178acc67f32c78310e - >> > > default >> > > > default] [instance: 5ec719f4-1865-4afe-a207-3d9fae22c410] Live >> Migration >> > > > failure: internal error: qemu unexpectedly closed the monitor: >> > > > 2019-04-15T02:58:22.213897Z qemu-kvm: VQ 0 >> > > > size 0x100 < last_avail_idx 0x1e - used_idx 0x23 >> > > > >> > > > It's OK for standard Linux VM, but not OK for our VM where virtio is >> > > > implemented by ourself. >> > > > KVM version as follow: >> > > > qemu-kvm-common-ev-2.12.0-18.el7_6.3.1.x86_64 >> > > > qemu-kvm-ev-2.12.0-18.el7_6.3.1.x86_64 >> > > > libvirt-daemon-kvm-3.9.0-14.2.el7.centos.ncir.8.x86_64 >> > > > >> > > > Do you know what's the difference between virtio and vhost-user >> during >> > > > migration? >> > > > The function virtio_load in Qemu is called for virtio and vhost-user >> > > during >> > > > migration. >> > > > For virtio, last_avail_idx and used_idx are stored in Qemu, Qemu >> is >> > > > responsible for updating their values accordingly >> > > > For vhost-user, last_avail_idx and used_idx are stored in >> vhost-user >> > > app, >> > > > eg. DPDK, not in Qemu? >> > > > How does migration work for vhost-user? >> > > >> > > I don't know the details, but my understanding is that vhost-user >> > > tells the vhost-user client about an area of 'log' memory, where the >> > > vhost-user client must mark pages as dirty. >> > > >> > > In the qemu source, see docs/interop/vhost-user.txt and see >> > > the VHOST_SET_LOG_BASE and VHOST_USER_SET_LOG_FD calls. >> > > >> > > If the client correctly marks the areas as dirty, then qemu >> > > should resend those pages across. >> > > >> > > >> > > Dave >> > > >> > > > Thanks in advance >> > > > Yafeng >> > > -- >> > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >> > > >> -- >> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >> > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] Fwd: How live migration work for vhost-user 2019-04-24 3:14 ` fengyd @ 2019-04-29 16:22 ` Dr. David Alan Gilbert [not found] ` <CACnvu8bBykwoSHOAwTa0L_FD3kF=i+KW+H1MZk-x3xu1tV9gkw@mail.gmail.com> 0 siblings, 1 reply; 9+ messages in thread From: Dr. David Alan Gilbert @ 2019-04-29 16:22 UTC (permalink / raw) To: fengyd; +Cc: qemu-devel * fengyd (fengyd81@gmail.com) wrote: > Hi, > > For vhost, *last_avail_idx* is maintained in vhost_virtqueue > but during live migration, *last_avail_idx* is fetched from VirtQueue. > Do you know how these two *last_avail_idx *are synchronized? > > virtio_load related code which is called during live migration: > > * vdev->vq[i].inuse = (uint16_t)(vdev->vq[i].last_avail_idx -* > * vdev->vq[i].used_idx);* > * if (vdev->vq[i].inuse > vdev->vq[i].vring.num) {* > * error_report("VQ %d size 0x%x < last_avail_idx 0x%x - "* > * "used_idx 0x%x",* > * i, vdev->vq[i].vring.num,* > * vdev->vq[i].last_avail_idx,* I don't know that code well; but I think the answer is that since the queues themselves are in guest memory, the guest memory is migrated by the normal migration code and so the queues version of last_avail_idx should be correct. The 'log' mechanism I previously mentioned will need to make sure the queue pages are marked dirty to make sure these are updated correctly. Dave > > > Thanks > > On Tue, 23 Apr 2019 at 14:20, fengyd <fengyd81@gmail.com> wrote: > > > Hi, > > > > I want to add some log to qemu-kvm-ev. > > Do you know how to compile qemu-kvm-ev from source code? > > > > Thanks > > > > Yafeng > > > > On Tue, 16 Apr 2019 at 16:47, Dr. David Alan Gilbert <dgilbert@redhat.com> > > wrote: > > > >> * fengyd (fengyd81@gmail.com) wrote: > >> > ---------- Forwarded message --------- > >> > From: fengyd <fengyd81@gmail.com> > >> > Date: Tue, 16 Apr 2019 at 09:17 > >> > Subject: Re: [Qemu-devel] How live migration work for vhost-user > >> > To: Dr. David Alan Gilbert <dgilbert@redhat.com> > >> > > >> > > >> > Hi, > >> > > >> > Any special feature needs to be supported on guest driver? > >> > Because it's OK for standard Linux VM, but not OK for our VM where > >> virtio > >> > is implemented by ourself. > >> > >> I'm not sure; you do have to support that 'log' mechanism but I don't > >> know what else is needed. > >> > >> > And with qemu-kvm-ev-2.6, live migration can work with our VM where > >> virtio > >> > is implemented by ourself. > >> > >> 2.6 is pretty old, so there's a lot of changes - not sure what's > >> relevant. > >> > >> Dave > >> > >> > Thanks > >> > Yafeng > >> > > >> > On Mon, 15 Apr 2019 at 22:54, Dr. David Alan Gilbert < > >> dgilbert@redhat.com> > >> > wrote: > >> > > >> > > * fengyd (fengyd81@gmail.com) wrote: > >> > > > Hi, > >> > > > > >> > > > During live migration, the folloing log can see in > >> nova-compute.log in > >> > > my > >> > > > environment: > >> > > > ERROR nova.virt.libvirt.driver > >> [req-039a85e1-e7a1-4a63-bc6d-c4b9a044aab6 > >> > > > 0cdab20dc79f4bc6ae5790e7b4a898ac 3363c319773549178acc67f32c78310e - > >> > > default > >> > > > default] [instance: 5ec719f4-1865-4afe-a207-3d9fae22c410] Live > >> Migration > >> > > > failure: internal error: qemu unexpectedly closed the monitor: > >> > > > 2019-04-15T02:58:22.213897Z qemu-kvm: VQ 0 > >> > > > size 0x100 < last_avail_idx 0x1e - used_idx 0x23 > >> > > > > >> > > > It's OK for standard Linux VM, but not OK for our VM where virtio is > >> > > > implemented by ourself. > >> > > > KVM version as follow: > >> > > > qemu-kvm-common-ev-2.12.0-18.el7_6.3.1.x86_64 > >> > > > qemu-kvm-ev-2.12.0-18.el7_6.3.1.x86_64 > >> > > > libvirt-daemon-kvm-3.9.0-14.2.el7.centos.ncir.8.x86_64 > >> > > > > >> > > > Do you know what's the difference between virtio and vhost-user > >> during > >> > > > migration? > >> > > > The function virtio_load in Qemu is called for virtio and vhost-user > >> > > during > >> > > > migration. > >> > > > For virtio, last_avail_idx and used_idx are stored in Qemu, Qemu > >> is > >> > > > responsible for updating their values accordingly > >> > > > For vhost-user, last_avail_idx and used_idx are stored in > >> vhost-user > >> > > app, > >> > > > eg. DPDK, not in Qemu? > >> > > > How does migration work for vhost-user? > >> > > > >> > > I don't know the details, but my understanding is that vhost-user > >> > > tells the vhost-user client about an area of 'log' memory, where the > >> > > vhost-user client must mark pages as dirty. > >> > > > >> > > In the qemu source, see docs/interop/vhost-user.txt and see > >> > > the VHOST_SET_LOG_BASE and VHOST_USER_SET_LOG_FD calls. > >> > > > >> > > If the client correctly marks the areas as dirty, then qemu > >> > > should resend those pages across. > >> > > > >> > > > >> > > Dave > >> > > > >> > > > Thanks in advance > >> > > > Yafeng > >> > > -- > >> > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > >> > > > >> -- > >> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > >> > > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 9+ messages in thread
[parent not found: <CACnvu8bBykwoSHOAwTa0L_FD3kF=i+KW+H1MZk-x3xu1tV9gkw@mail.gmail.com>]
* [Qemu-devel] Fwd: Fwd: How live migration work for vhost-user [not found] ` <CACnvu8bBykwoSHOAwTa0L_FD3kF=i+KW+H1MZk-x3xu1tV9gkw@mail.gmail.com> @ 2019-05-08 4:50 ` fengyd 0 siblings, 0 replies; 9+ messages in thread From: fengyd @ 2019-05-08 4:50 UTC (permalink / raw) To: qemu-devel ---------- Forwarded message --------- From: fengyd <fengyd81@gmail.com> Date: Wed, 8 May 2019 at 12:50 Subject: Re: [Qemu-devel] Fwd: How live migration work for vhost-user To: Dr. David Alan Gilbert <dgilbert@redhat.com> Hi, I checked the DPDK code and found that the function rte_vhost_enqueue_burst in virtio_net.c is used to receive message. uint16_t rte_vhost_enqueue_burst(int vid, uint16_t queue_id, struct rte_mbuf **pkts, uint16_t count) { struct virtio_net *dev = get_device(vid); if (!dev) return 0; if (dev->features & (1 << VIRTIO_NET_F_MRG_RXBUF)) return virtio_dev_merge_rx(dev, queue_id, pkts, count); else return virtio_dev_rx(dev, queue_id, pkts, count); } If VIRTIO_NET_F_MRG_RXBUF is supported by guest, virtio_dev_merge_rx is called, otherwise virtio_dev_rx is called. And in virtio_dev_merge_rx, last_avail_idx will be updated, and in virtio_dev_rx, last_avail_idx will not be touched, so it is kept as the initial value 0. Our guest virito driver doesn't support VIRTIO_NET_F_MRG_RXBUF, so last_avail_idx QEMU read from DPDK is always 0? And The strange thing is the function vhost_user_get_vring_base in vhost_user.c which returns last_avail_idx to QEMU. I checked the code in v18.05 <https://git.dpdk.org/dpdk/tree/lib/librte_vhost/vhost_user.c?h=v18.05>, last_avail_idx is assigned to the return message. msg->payload.state.num = vq->last_avail_idx; And in DPDK v17.11 <https://git.dpdk.org/dpdk/tree/lib/librte_vhost/vhost_user.c?h=v17.11>, last_used_idx is assigned to the return message. msg->payload.state.num = vq->last_used_idx; On Tue, 30 Apr 2019 at 00:22, Dr. David Alan Gilbert <dgilbert@redhat.com> wrote: > * fengyd (fengyd81@gmail.com) wrote: > > Hi, > > > > For vhost, *last_avail_idx* is maintained in vhost_virtqueue > > but during live migration, *last_avail_idx* is fetched from VirtQueue. > > Do you know how these two *last_avail_idx *are synchronized? > > > > virtio_load related code which is called during live migration: > > > > * vdev->vq[i].inuse = (uint16_t)(vdev->vq[i].last_avail_idx -* > > * vdev->vq[i].used_idx);* > > * if (vdev->vq[i].inuse > vdev->vq[i].vring.num) {* > > * error_report("VQ %d size 0x%x < last_avail_idx 0x%x - "* > > * "used_idx 0x%x",* > > * i, vdev->vq[i].vring.num,* > > * vdev->vq[i].last_avail_idx,* > > I don't know that code well; but I think the answer is that since the > queues themselves are in guest memory, the guest memory is migrated > by the normal migration code and so the queues version of last_avail_idx > should be correct. The 'log' mechanism I previously mentioned will > need to make sure the queue pages are marked dirty to make sure these > are updated correctly. > > Dave > > > > > > > Thanks > > > > On Tue, 23 Apr 2019 at 14:20, fengyd <fengyd81@gmail.com> wrote: > > > > > Hi, > > > > > > I want to add some log to qemu-kvm-ev. > > > Do you know how to compile qemu-kvm-ev from source code? > > > > > > Thanks > > > > > > Yafeng > > > > > > On Tue, 16 Apr 2019 at 16:47, Dr. David Alan Gilbert < > dgilbert@redhat.com> > > > wrote: > > > > > >> * fengyd (fengyd81@gmail.com) wrote: > > >> > ---------- Forwarded message --------- > > >> > From: fengyd <fengyd81@gmail.com> > > >> > Date: Tue, 16 Apr 2019 at 09:17 > > >> > Subject: Re: [Qemu-devel] How live migration work for vhost-user > > >> > To: Dr. David Alan Gilbert <dgilbert@redhat.com> > > >> > > > >> > > > >> > Hi, > > >> > > > >> > Any special feature needs to be supported on guest driver? > > >> > Because it's OK for standard Linux VM, but not OK for our VM where > > >> virtio > > >> > is implemented by ourself. > > >> > > >> I'm not sure; you do have to support that 'log' mechanism but I don't > > >> know what else is needed. > > >> > > >> > And with qemu-kvm-ev-2.6, live migration can work with our VM where > > >> virtio > > >> > is implemented by ourself. > > >> > > >> 2.6 is pretty old, so there's a lot of changes - not sure what's > > >> relevant. > > >> > > >> Dave > > >> > > >> > Thanks > > >> > Yafeng > > >> > > > >> > On Mon, 15 Apr 2019 at 22:54, Dr. David Alan Gilbert < > > >> dgilbert@redhat.com> > > >> > wrote: > > >> > > > >> > > * fengyd (fengyd81@gmail.com) wrote: > > >> > > > Hi, > > >> > > > > > >> > > > During live migration, the folloing log can see in > > >> nova-compute.log in > > >> > > my > > >> > > > environment: > > >> > > > ERROR nova.virt.libvirt.driver > > >> [req-039a85e1-e7a1-4a63-bc6d-c4b9a044aab6 > > >> > > > 0cdab20dc79f4bc6ae5790e7b4a898ac > 3363c319773549178acc67f32c78310e - > > >> > > default > > >> > > > default] [instance: 5ec719f4-1865-4afe-a207-3d9fae22c410] Live > > >> Migration > > >> > > > failure: internal error: qemu unexpectedly closed the monitor: > > >> > > > 2019-04-15T02:58:22.213897Z qemu-kvm: VQ 0 > > >> > > > size 0x100 < last_avail_idx 0x1e - used_idx 0x23 > > >> > > > > > >> > > > It's OK for standard Linux VM, but not OK for our VM where > virtio is > > >> > > > implemented by ourself. > > >> > > > KVM version as follow: > > >> > > > qemu-kvm-common-ev-2.12.0-18.el7_6.3.1.x86_64 > > >> > > > qemu-kvm-ev-2.12.0-18.el7_6.3.1.x86_64 > > >> > > > libvirt-daemon-kvm-3.9.0-14.2.el7.centos.ncir.8.x86_64 > > >> > > > > > >> > > > Do you know what's the difference between virtio and vhost-user > > >> during > > >> > > > migration? > > >> > > > The function virtio_load in Qemu is called for virtio and > vhost-user > > >> > > during > > >> > > > migration. > > >> > > > For virtio, last_avail_idx and used_idx are stored in Qemu, > Qemu > > >> is > > >> > > > responsible for updating their values accordingly > > >> > > > For vhost-user, last_avail_idx and used_idx are stored in > > >> vhost-user > > >> > > app, > > >> > > > eg. DPDK, not in Qemu? > > >> > > > How does migration work for vhost-user? > > >> > > > > >> > > I don't know the details, but my understanding is that vhost-user > > >> > > tells the vhost-user client about an area of 'log' memory, where > the > > >> > > vhost-user client must mark pages as dirty. > > >> > > > > >> > > In the qemu source, see docs/interop/vhost-user.txt and see > > >> > > the VHOST_SET_LOG_BASE and VHOST_USER_SET_LOG_FD calls. > > >> > > > > >> > > If the client correctly marks the areas as dirty, then qemu > > >> > > should resend those pages across. > > >> > > > > >> > > > > >> > > Dave > > >> > > > > >> > > > Thanks in advance > > >> > > > Yafeng > > >> > > -- > > >> > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > >> > > > > >> -- > > >> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > >> > > > > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] Fwd: How live migration work for vhost-user 2019-04-23 6:20 ` fengyd 2019-04-24 3:14 ` fengyd @ 2019-04-29 16:15 ` Dr. David Alan Gilbert 1 sibling, 0 replies; 9+ messages in thread From: Dr. David Alan Gilbert @ 2019-04-29 16:15 UTC (permalink / raw) To: fengyd; +Cc: qemu-devel * fengyd (fengyd81@gmail.com) wrote: > Hi, > > I want to add some log to qemu-kvm-ev. > Do you know how to compile qemu-kvm-ev from source code? That's our downstream package build; it should build fine with rpmbuild, or you can configure and build it like any other qemu. Dave > Thanks > > Yafeng > > On Tue, 16 Apr 2019 at 16:47, Dr. David Alan Gilbert <dgilbert@redhat.com> > wrote: > > > * fengyd (fengyd81@gmail.com) wrote: > > > ---------- Forwarded message --------- > > > From: fengyd <fengyd81@gmail.com> > > > Date: Tue, 16 Apr 2019 at 09:17 > > > Subject: Re: [Qemu-devel] How live migration work for vhost-user > > > To: Dr. David Alan Gilbert <dgilbert@redhat.com> > > > > > > > > > Hi, > > > > > > Any special feature needs to be supported on guest driver? > > > Because it's OK for standard Linux VM, but not OK for our VM where virtio > > > is implemented by ourself. > > > > I'm not sure; you do have to support that 'log' mechanism but I don't > > know what else is needed. > > > > > And with qemu-kvm-ev-2.6, live migration can work with our VM where > > virtio > > > is implemented by ourself. > > > > 2.6 is pretty old, so there's a lot of changes - not sure what's > > relevant. > > > > Dave > > > > > Thanks > > > Yafeng > > > > > > On Mon, 15 Apr 2019 at 22:54, Dr. David Alan Gilbert < > > dgilbert@redhat.com> > > > wrote: > > > > > > > * fengyd (fengyd81@gmail.com) wrote: > > > > > Hi, > > > > > > > > > > During live migration, the folloing log can see in nova-compute.log > > in > > > > my > > > > > environment: > > > > > ERROR nova.virt.libvirt.driver > > [req-039a85e1-e7a1-4a63-bc6d-c4b9a044aab6 > > > > > 0cdab20dc79f4bc6ae5790e7b4a898ac 3363c319773549178acc67f32c78310e - > > > > default > > > > > default] [instance: 5ec719f4-1865-4afe-a207-3d9fae22c410] Live > > Migration > > > > > failure: internal error: qemu unexpectedly closed the monitor: > > > > > 2019-04-15T02:58:22.213897Z qemu-kvm: VQ 0 > > > > > size 0x100 < last_avail_idx 0x1e - used_idx 0x23 > > > > > > > > > > It's OK for standard Linux VM, but not OK for our VM where virtio is > > > > > implemented by ourself. > > > > > KVM version as follow: > > > > > qemu-kvm-common-ev-2.12.0-18.el7_6.3.1.x86_64 > > > > > qemu-kvm-ev-2.12.0-18.el7_6.3.1.x86_64 > > > > > libvirt-daemon-kvm-3.9.0-14.2.el7.centos.ncir.8.x86_64 > > > > > > > > > > Do you know what's the difference between virtio and vhost-user > > during > > > > > migration? > > > > > The function virtio_load in Qemu is called for virtio and vhost-user > > > > during > > > > > migration. > > > > > For virtio, last_avail_idx and used_idx are stored in Qemu, Qemu is > > > > > responsible for updating their values accordingly > > > > > For vhost-user, last_avail_idx and used_idx are stored in vhost-user > > > > app, > > > > > eg. DPDK, not in Qemu? > > > > > How does migration work for vhost-user? > > > > > > > > I don't know the details, but my understanding is that vhost-user > > > > tells the vhost-user client about an area of 'log' memory, where the > > > > vhost-user client must mark pages as dirty. > > > > > > > > In the qemu source, see docs/interop/vhost-user.txt and see > > > > the VHOST_SET_LOG_BASE and VHOST_USER_SET_LOG_FD calls. > > > > > > > > If the client correctly marks the areas as dirty, then qemu > > > > should resend those pages across. > > > > > > > > > > > > Dave > > > > > > > > > Thanks in advance > > > > > Yafeng > > > > -- > > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > > > > > -- > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2019-05-08 4:51 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-04-15 8:50 [Qemu-devel] How live migration work for vhost-user fengyd 2019-04-15 14:53 ` Dr. David Alan Gilbert [not found] ` <CACnvu8a5DjAUhfAGmi6GtpD1V2hfoNLTkKEgbsFuL_6GrRc+pg@mail.gmail.com> 2019-04-16 7:59 ` [Qemu-devel] Fwd: " fengyd 2019-04-16 8:47 ` Dr. David Alan Gilbert 2019-04-23 6:20 ` fengyd 2019-04-24 3:14 ` fengyd 2019-04-29 16:22 ` Dr. David Alan Gilbert [not found] ` <CACnvu8bBykwoSHOAwTa0L_FD3kF=i+KW+H1MZk-x3xu1tV9gkw@mail.gmail.com> 2019-05-08 4:50 ` [Qemu-devel] Fwd: " fengyd 2019-04-29 16:15 ` [Qemu-devel] " Dr. David Alan Gilbert
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.