* [Bug 1761798] Re: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" [not found] <152303245198.3233.1033096993665779324.malonedeb@wampee.canonical.com> @ 2020-11-23 15:17 ` Balazs Gibizer 2020-11-23 15:30 ` sean mooney ` (11 subsequent siblings) 12 siblings, 0 replies; 13+ messages in thread From: Balazs Gibizer @ 2020-11-23 15:17 UTC (permalink / raw) To: qemu-devel Source node https://zuul.opendev.org/t/openstack/build/d50877ae15db4022b82f4bb1d1d52cea/log/logs/subnode-2/libvirt/qemu/instance-0000001a.txt 2020-11-20 14:25:24.887+0000: starting up libvirt version: 6.0.0, package: 0ubuntu8.4~cloud0 (Openstack Ubuntu Testing Bot <openstack- testing-bot@ubuntu.com> Tue, 15 Sep 2020 20:36:28 +0000), qemu version: 4.2.1Debian 1:4.2-3ubuntu6.7~cloud0, kernel: 4.15.0-124-generic, hostname: ubuntu-bionic-ovh-bhs1-0021872195 LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ HOME=/var/lib/libvirt/qemu/domain-19-instance-0000001a \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-19-instance- 0000001a/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-19-instance-0000001a/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-19-instance- 0000001a/.config \ QEMU_AUDIO_DRV=none \ /usr/bin/qemu-system-x86_64 \ -name guest=instance-0000001a,debug-threads=on \ -S \ -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-19 -instance-0000001a/master-key.aes \ -machine pc-i440fx-4.2,accel=tcg,usb=off,dump-guest-core=off \ -cpu qemu64,hypervisor=on,lahf-lm=on \ -m 128 \ -overcommit mem-lock=off \ -smp 1,sockets=1,cores=1,threads=1 \ -uuid 2c468d92-4b19-426a-8c25-16b4624c21a4 \ -smbios 'type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=22.1.0,serial=2c468d92-4b19-426a- 8c25-16b4624c21a4,uuid=2c468d92-4b19-426a- 8c25-16b4624c21a4,family=Virtual Machine' \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=35,server,nowait \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc \ -no-shutdown \ -boot strict=on \ -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \ -blockdev '{"driver":"file","filename":"/opt/stack/data/nova/instances/_base/61bd5e531ab4c82456aa5300ede7266b3610be79 ","node-name":"libvirt-2-storage","cache":{"direct":true,"no- flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read- only":true,"cache":{"direct":true,"no- flush":false},"driver":"raw","file":"libvirt-2-storage"}' \ -blockdev '{"driver":"file","filename":"/opt/stack/data/nova/instances/2c468d92-4b19 -426a-8c25-16b4624c21a4/disk","node- name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false },"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read- only":false,"cache":{"direct":true,"no- flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}' \ -device virtio-blk- pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio- disk0,bootindex=1,write-cache=on \ -netdev tap,fd=37,id=hostnet0 \ -device virtio-net- pci,host_mtu=1400,netdev=hostnet0,id=net0,mac=fa:16:3e:43:11:f4,bus=pci.0,addr=0x3 \ -add-fd set=2,fd=39 \ -chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on \ -device isa-serial,chardev=charserial0,id=serial0 \ -vnc 127.0.0.1:0 \ -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \ -incoming defer \ -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 \ -object rng-random,id=objrng0,filename=/dev/urandom \ -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on char device redirected to /dev/pts/2 (label charserial0) virtio: bogus descriptor or out of resources 2020-11-20 14:25:39.911+0000: initiating migration 2020-11-20T14:26:21.409517Z qemu-system-x86_64: terminating on signal 15 from pid 17395 (/usr/sbin/libvirtd) 2020-11-20 14:26:21.610+0000: shutting down, reason=destroyed Target node https://zuul.opendev.org/t/openstack/build/d50877ae15db4022b82f4bb1d1d52cea/log/logs/libvir t/qemu/instance-0000001a.txt 2020-11-20 14:25:11.589+0000: starting up libvirt version: 6.0.0, package: 0ubuntu8.4~cloud0 (Openstack Ubuntu Testing Bot <openstack- testing-bot@ubuntu.com> Tue, 15 Sep 2020 20:36:28 +0000), qemu version: 4.2.1Debian 1:4.2-3ubuntu6.7~cloud0, kernel: 4.15.0-124-generic, hostname: ubuntu-bionic-ovh-bhs1-0021872194 LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ HOME=/var/lib/libvirt/qemu/domain-10-instance-0000001a \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-10-instance- 0000001a/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-10-instance-0000001a/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-10-instance- 0000001a/.config \ QEMU_AUDIO_DRV=none \ /usr/bin/qemu-system-x86_64 \ -name guest=instance-0000001a,debug-threads=on \ -S \ -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-10 -instance-0000001a/master-key.aes \ -machine pc-i440fx-4.2,accel=tcg,usb=off,dump-guest-core=off \ -cpu qemu64 \ -m 128 \ -overcommit mem-lock=off \ -smp 1,sockets=1,cores=1,threads=1 \ -uuid 2c468d92-4b19-426a-8c25-16b4624c21a4 \ -smbios 'type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=22.1.0,serial=2c468d92-4b19-426a- 8c25-16b4624c21a4,uuid=2c468d92-4b19-426a- 8c25-16b4624c21a4,family=Virtual Machine' \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=32,server,nowait \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc \ -no-shutdown \ -boot strict=on \ -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \ -blockdev '{"driver":"file","filename":"/opt/stack/data/nova/instances/_base/61bd5e531ab4c82456aa5300ede7266b3610be79 ","node-name":"libvirt-2-storage","cache":{"direct":true,"no- flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read- only":true,"cache":{"direct":true,"no- flush":false},"driver":"raw","file":"libvirt-2-storage"}' \ -blockdev '{"driver":"file","filename":"/opt/stack/data/nova/instances/2c468d92-4b19 -426a-8c25-16b4624c21a4/disk","node- name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false },"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read- only":false,"cache":{"direct":true,"no- flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}' \ -device virtio-blk- pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio- disk0,bootindex=1,write-cache=on \ -netdev tap,fd=34,id=hostnet0 \ -device virtio-net- pci,host_mtu=1400,netdev=hostnet0,id=net0,mac=fa:16:3e:43:11:f4,bus=pci.0,addr=0x3 \ -add-fd set=2,fd=36 \ -chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on \ -device isa-serial,chardev=charserial0,id=serial0 \ -vnc 0.0.0.0:0 \ -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \ -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 \ -object rng-random,id=objrng0,filename=/dev/urandom \ -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on char device redirected to /dev/pts/0 (label charserial0) 2020-11-20 14:25:25.637+0000: initiating migration 2020-11-20 14:25:26.776+0000: shutting down, reason=migrated 2020-11-20T14:25:26.777394Z qemu-system-x86_64: terminating on signal 15 from pid 31113 (/usr/sbin/libvirtd) 2020-11-20 14:25:38.909+0000: starting up libvirt version: 6.0.0, package: 0ubuntu8.4~cloud0 (Openstack Ubuntu Testing Bot <openstack- testing-bot@ubuntu.com> Tue, 15 Sep 2020 20:36:28 +0000), qemu version: 4.2.1Debian 1:4.2-3ubuntu6.7~cloud0, kernel: 4.15.0-124-generic, hostname: ubuntu-bionic-ovh-bhs1-0021872194 LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ HOME=/var/lib/libvirt/qemu/domain-13-instance-0000001a \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-13-instance- 0000001a/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-13-instance-0000001a/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-13-instance- 0000001a/.config \ QEMU_AUDIO_DRV=none \ /usr/bin/qemu-system-x86_64 \ -name guest=instance-0000001a,debug-threads=on \ -S \ -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-13 -instance-0000001a/master-key.aes \ -machine pc-i440fx-4.2,accel=tcg,usb=off,dump-guest-core=off \ -cpu qemu64,hypervisor=on,lahf-lm=on \ -m 128 \ -overcommit mem-lock=off \ -smp 1,sockets=1,cores=1,threads=1 \ -uuid 2c468d92-4b19-426a-8c25-16b4624c21a4 \ -smbios 'type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=22.1.0,serial=2c468d92-4b19-426a- 8c25-16b4624c21a4,uuid=2c468d92-4b19-426a- 8c25-16b4624c21a4,family=Virtual Machine' \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=34,server,nowait \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc \ -no-shutdown \ -boot strict=on \ -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \ -blockdev '{"driver":"file","filename":"/opt/stack/data/nova/instances/_base/61bd5e531ab4c82456aa5300ede7266b3610be79 ","node-name":"libvirt-2-storage","cache":{"direct":true,"no- flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read- only":true,"cache":{"direct":true,"no- flush":false},"driver":"raw","file":"libvirt-2-storage"}' \ -blockdev '{"driver":"file","filename":"/opt/stack/data/nova/instances/2c468d92-4b19 -426a-8c25-16b4624c21a4/disk","node- name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false },"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read- only":false,"cache":{"direct":true,"no- flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}' \ -device virtio-blk- pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio- disk0,bootindex=1,write-cache=on \ -netdev tap,fd=36,id=hostnet0 \ -device virtio-net- pci,host_mtu=1400,netdev=hostnet0,id=net0,mac=fa:16:3e:43:11:f4,bus=pci.0,addr=0x3 \ -add-fd set=2,fd=38 \ -chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on \ -device isa-serial,chardev=charserial0,id=serial0 \ -vnc 0.0.0.0:0 \ -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \ -incoming defer \ -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 \ -object rng-random,id=objrng0,filename=/dev/urandom \ -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on char device redirected to /dev/pts/1 (label charserial0) 2020-11-20T14:25:40.720757Z qemu-system-x86_64: VQ 0 size 0x80 Guest index 0xb8 inconsistent with Host index 0xe0: delta 0xffd8 2020-11-20T14:25:40.720785Z qemu-system-x86_64: Failed to load virtio- blk:virtio 2020-11-20T14:25:40.720790Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:04.0/virtio-blk' 2020-11-20T14:25:40.720824Z qemu-system-x86_64: load of migration failed: Operation not permitted 2020-11-20 14:25:40.778+0000: shutting down, reason=failed ** Also affects: qemu Importance: Undecided Status: New -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1761798 Title: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" Status in OpenStack Compute (nova): Confirmed Status in QEMU: New Bug description: Seen here: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz 2018-04-05T21:48:38.205752Z qemu-system-x86_64: -chardev pty,id=charserial0,logfile=/dev/fdset/1,logappend=on: char device redirected to /dev/pts/0 (label charserial0) warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] 2018-04-05T21:48:43.153268Z qemu-system-x86_64: VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8 2018-04-05T21:48:43.153288Z qemu-system-x86_64: Failed to load virtio-blk:virtio 2018-04-05T21:48:43.153292Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:04.0/virtio-blk' 2018-04-05T21:48:43.153347Z qemu-system-x86_64: load of migration failed: Operation not permitted 2018-04-05 21:48:43.198+0000: shutting down, reason=crashed And in the n-cpu logs on the other host: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/screen-n-cpu.txt.gz#_Apr_05_21_48_43_257541 There is a related Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1450524 The CI job failures are at present using the Pike UCA: ii libvirt-bin 3.6.0-1ubuntu6.2~cloud0 ii qemu-system-x86 1:2.10+dfsg-0ubuntu3.5~cloud0 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1761798/+subscriptions ^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug 1761798] Re: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" [not found] <152303245198.3233.1033096993665779324.malonedeb@wampee.canonical.com> 2020-11-23 15:17 ` [Bug 1761798] Re: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" Balazs Gibizer @ 2020-11-23 15:30 ` sean mooney 2020-11-23 16:56 ` Kashyap Chamarthy ` (10 subsequent siblings) 12 siblings, 0 replies; 13+ messages in thread From: sean mooney @ 2020-11-23 15:30 UTC (permalink / raw) To: qemu-devel in the new case we have the same qemu/libvirt version in both nodes 4.2.1Debian 1:4.2-3ubuntu6.7~cloud0 and in this cae its failng for the virtio-blk device not the memroy ballon before the migration starts we see -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on char device redirected to /dev/pts/2 (label charserial0) virtio: bogus descriptor or out of resources 2020-11-20 14:25:39.911+0000: initiating migration on the destination https://zuul.opendev.org/t/openstack/build/d50877ae15db4022b82f4bb1d1d52cea/log/logs/subnode-2/libvirt/qemu/instance-0000001a.txt then on the source we see char device redirected to /dev/pts/1 (label charserial0) 2020-11-20T14:25:40.720757Z qemu-system-x86_64: VQ 0 size 0x80 Guest index 0xb8 inconsistent with Host index 0xe0: delta 0xffd8 2020-11-20T14:25:40.720785Z qemu-system-x86_64: Failed to load virtio-blk:virtio 2020-11-20T14:25:40.720790Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:04.0/virtio-blk' 2020-11-20T14:25:40.720824Z qemu-system-x86_64: load of migration failed: Operation not permitted when it fails https://zuul.opendev.org/t/openstack/build/d50877ae15db4022b82f4bb1d1d52cea/log/logs/libvirt/qemu/instance-0000001a.txt -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1761798 Title: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" Status in OpenStack Compute (nova): Confirmed Status in QEMU: New Bug description: Seen here: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz 2018-04-05T21:48:38.205752Z qemu-system-x86_64: -chardev pty,id=charserial0,logfile=/dev/fdset/1,logappend=on: char device redirected to /dev/pts/0 (label charserial0) warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] 2018-04-05T21:48:43.153268Z qemu-system-x86_64: VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8 2018-04-05T21:48:43.153288Z qemu-system-x86_64: Failed to load virtio-blk:virtio 2018-04-05T21:48:43.153292Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:04.0/virtio-blk' 2018-04-05T21:48:43.153347Z qemu-system-x86_64: load of migration failed: Operation not permitted 2018-04-05 21:48:43.198+0000: shutting down, reason=crashed And in the n-cpu logs on the other host: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/screen-n-cpu.txt.gz#_Apr_05_21_48_43_257541 There is a related Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1450524 The CI job failures are at present using the Pike UCA: ii libvirt-bin 3.6.0-1ubuntu6.2~cloud0 ii qemu-system-x86 1:2.10+dfsg-0ubuntu3.5~cloud0 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1761798/+subscriptions ^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug 1761798] Re: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" [not found] <152303245198.3233.1033096993665779324.malonedeb@wampee.canonical.com> 2020-11-23 15:17 ` [Bug 1761798] Re: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" Balazs Gibizer 2020-11-23 15:30 ` sean mooney @ 2020-11-23 16:56 ` Kashyap Chamarthy 2020-11-24 12:17 ` Dr. David Alan Gilbert ` (9 subsequent siblings) 12 siblings, 0 replies; 13+ messages in thread From: Kashyap Chamarthy @ 2020-11-23 16:56 UTC (permalink / raw) To: qemu-devel [Copy/pasting my comment from here: https://bugs.launchpad.net/nova/+bug/1737625/comments/4] I just talked to Dave Gilbert from upstream QEMU. Overall, as I implied in comment#2, this gnarly issue requires specialized debugging, digging deep into the bowels of QEMU, 'virtio-blk' and 'virtio. That said, Dave notes that we get this "guest index inconsistent" error when the migrated RAM is inconsistent with the migrated 'virtio' device state. And a common case is where a 'virtio' device does an operation after the vCPU is stopped and after RAM has been transmitted. Dave makes some guesswork of a potential scenario where this can occur: - Guest is running - ... live migration starts - ... a "block read" request gets submitteed - ... live migration stops the vCPUs, finishes transmitting RAM - ... the "block read" completes, 'virtio-blk' updates pointers - ... live migration "serializes" the 'virito-blk' state So the "guest index inconsistent" state would only happen if you got unlucky with the timing of that read. Another possibility, Dave points out, is that the guest has screwed up the device state somehow; the migration code in 'virtio' checks the state a lot. We have ruled this possibility out becausethe guest is just a garden-variety CirrOS instance idling; nothing special about it. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1761798 Title: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" Status in OpenStack Compute (nova): Confirmed Status in QEMU: New Bug description: Seen here: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz 2018-04-05T21:48:38.205752Z qemu-system-x86_64: -chardev pty,id=charserial0,logfile=/dev/fdset/1,logappend=on: char device redirected to /dev/pts/0 (label charserial0) warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] 2018-04-05T21:48:43.153268Z qemu-system-x86_64: VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8 2018-04-05T21:48:43.153288Z qemu-system-x86_64: Failed to load virtio-blk:virtio 2018-04-05T21:48:43.153292Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:04.0/virtio-blk' 2018-04-05T21:48:43.153347Z qemu-system-x86_64: load of migration failed: Operation not permitted 2018-04-05 21:48:43.198+0000: shutting down, reason=crashed And in the n-cpu logs on the other host: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/screen-n-cpu.txt.gz#_Apr_05_21_48_43_257541 There is a related Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1450524 The CI job failures are at present using the Pike UCA: ii libvirt-bin 3.6.0-1ubuntu6.2~cloud0 ii qemu-system-x86 1:2.10+dfsg-0ubuntu3.5~cloud0 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1761798/+subscriptions ^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug 1761798] Re: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" [not found] <152303245198.3233.1033096993665779324.malonedeb@wampee.canonical.com> ` (2 preceding siblings ...) 2020-11-23 16:56 ` Kashyap Chamarthy @ 2020-11-24 12:17 ` Dr. David Alan Gilbert 2020-11-25 13:44 ` Balazs Gibizer ` (8 subsequent siblings) 12 siblings, 0 replies; 13+ messages in thread From: Dr. David Alan Gilbert @ 2020-11-24 12:17 UTC (permalink / raw) To: qemu-devel hang on, I've just noticed the : char device redirected to /dev/pts/2 (label charserial0) virtio: bogus descriptor or out of resources 2020-11-20 14:25:39.911+0000: initiating migration I've never seen that 'bogus descriptor or out of resources' one before; the fact that's happening before the migration starts is suspicious that something has already gone wrong before migration started. Is that warning present in all these failures? -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1761798 Title: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" Status in OpenStack Compute (nova): Confirmed Status in QEMU: New Bug description: Seen here: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz 2018-04-05T21:48:38.205752Z qemu-system-x86_64: -chardev pty,id=charserial0,logfile=/dev/fdset/1,logappend=on: char device redirected to /dev/pts/0 (label charserial0) warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] 2018-04-05T21:48:43.153268Z qemu-system-x86_64: VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8 2018-04-05T21:48:43.153288Z qemu-system-x86_64: Failed to load virtio-blk:virtio 2018-04-05T21:48:43.153292Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:04.0/virtio-blk' 2018-04-05T21:48:43.153347Z qemu-system-x86_64: load of migration failed: Operation not permitted 2018-04-05 21:48:43.198+0000: shutting down, reason=crashed And in the n-cpu logs on the other host: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/screen-n-cpu.txt.gz#_Apr_05_21_48_43_257541 There is a related Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1450524 The CI job failures are at present using the Pike UCA: ii libvirt-bin 3.6.0-1ubuntu6.2~cloud0 ii qemu-system-x86 1:2.10+dfsg-0ubuntu3.5~cloud0 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1761798/+subscriptions ^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug 1761798] Re: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" [not found] <152303245198.3233.1033096993665779324.malonedeb@wampee.canonical.com> ` (3 preceding siblings ...) 2020-11-24 12:17 ` Dr. David Alan Gilbert @ 2020-11-25 13:44 ` Balazs Gibizer 2020-11-25 14:03 ` Dr. David Alan Gilbert ` (7 subsequent siblings) 12 siblings, 0 replies; 13+ messages in thread From: Balazs Gibizer @ 2020-11-25 13:44 UTC (permalink / raw) To: qemu-devel I see two recent hits we have still logs for. The one described in comment #10 And another but there the error message is a bit different: on the migration source host: : https://6f0be18d925d64906a23-689ad0b9b6f06bc0c51bfb99bf86ea04.ssl.cf5.rackcdn.com/698706/4/check/nova-grenade-multinode/ee2dbea/logs/libvirt/qemu/instance-0000001b.txt char device redirected to /dev/pts/0 (label charserial0) virtio: zero sized buffers are not allowed 2020-11-23 22:20:54.297+0000: initiating migration on the migration destination https://6f0be18d925d64906a23-689ad0b9b6f06bc0c51bfb99bf86ea04.ssl.cf5.rackcdn.com/698706/4/check/nova-grenade-multinode/ee2dbea/logs/subnode-2/libvirt/qemu/instance-0000001b.txt char device redirected to /dev/pts/0 (label charserial0) 2020-11-23T22:20:55.129189Z qemu-system-x86_64: VQ 0 size 0x80 Guest index 0x62 inconsistent with Host index 0xa1: delta 0xffc1 2020-11-23T22:20:55.129230Z qemu-system-x86_64: Failed to load virtio-blk:virtio 2020-11-23T22:20:55.129241Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-blk' 2020-11-23T22:20:55.129259Z qemu-system-x86_64: load of migration failed: Operation not permitted -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1761798 Title: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" Status in OpenStack Compute (nova): Confirmed Status in QEMU: New Bug description: Seen here: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz 2018-04-05T21:48:38.205752Z qemu-system-x86_64: -chardev pty,id=charserial0,logfile=/dev/fdset/1,logappend=on: char device redirected to /dev/pts/0 (label charserial0) warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] 2018-04-05T21:48:43.153268Z qemu-system-x86_64: VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8 2018-04-05T21:48:43.153288Z qemu-system-x86_64: Failed to load virtio-blk:virtio 2018-04-05T21:48:43.153292Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:04.0/virtio-blk' 2018-04-05T21:48:43.153347Z qemu-system-x86_64: load of migration failed: Operation not permitted 2018-04-05 21:48:43.198+0000: shutting down, reason=crashed And in the n-cpu logs on the other host: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/screen-n-cpu.txt.gz#_Apr_05_21_48_43_257541 There is a related Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1450524 The CI job failures are at present using the Pike UCA: ii libvirt-bin 3.6.0-1ubuntu6.2~cloud0 ii qemu-system-x86 1:2.10+dfsg-0ubuntu3.5~cloud0 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1761798/+subscriptions ^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug 1761798] Re: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" [not found] <152303245198.3233.1033096993665779324.malonedeb@wampee.canonical.com> ` (4 preceding siblings ...) 2020-11-25 13:44 ` Balazs Gibizer @ 2020-11-25 14:03 ` Dr. David Alan Gilbert 2021-05-10 4:32 ` Thomas Huth ` (6 subsequent siblings) 12 siblings, 0 replies; 13+ messages in thread From: Dr. David Alan Gilbert @ 2020-11-25 14:03 UTC (permalink / raw) To: qemu-devel OK, but that still says in both cases here we've got a virtio error telling us that the queues are broken before migration even starts; so we should try and figure out why that's happening first. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1761798 Title: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" Status in OpenStack Compute (nova): Confirmed Status in QEMU: New Bug description: Seen here: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz 2018-04-05T21:48:38.205752Z qemu-system-x86_64: -chardev pty,id=charserial0,logfile=/dev/fdset/1,logappend=on: char device redirected to /dev/pts/0 (label charserial0) warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] 2018-04-05T21:48:43.153268Z qemu-system-x86_64: VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8 2018-04-05T21:48:43.153288Z qemu-system-x86_64: Failed to load virtio-blk:virtio 2018-04-05T21:48:43.153292Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:04.0/virtio-blk' 2018-04-05T21:48:43.153347Z qemu-system-x86_64: load of migration failed: Operation not permitted 2018-04-05 21:48:43.198+0000: shutting down, reason=crashed And in the n-cpu logs on the other host: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/screen-n-cpu.txt.gz#_Apr_05_21_48_43_257541 There is a related Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1450524 The CI job failures are at present using the Pike UCA: ii libvirt-bin 3.6.0-1ubuntu6.2~cloud0 ii qemu-system-x86 1:2.10+dfsg-0ubuntu3.5~cloud0 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1761798/+subscriptions ^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug 1761798] Re: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" [not found] <152303245198.3233.1033096993665779324.malonedeb@wampee.canonical.com> ` (5 preceding siblings ...) 2020-11-25 14:03 ` Dr. David Alan Gilbert @ 2021-05-10 4:32 ` Thomas Huth 2021-05-19 9:53 ` ignazio ` (5 subsequent siblings) 12 siblings, 0 replies; 13+ messages in thread From: Thomas Huth @ 2021-05-10 4:32 UTC (permalink / raw) To: qemu-devel Is this still an issue with the latest release of QEMU (v6.0)? ** Changed in: qemu Status: New => Incomplete -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1761798 Title: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" Status in OpenStack Compute (nova): Confirmed Status in QEMU: Incomplete Bug description: Seen here: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz 2018-04-05T21:48:38.205752Z qemu-system-x86_64: -chardev pty,id=charserial0,logfile=/dev/fdset/1,logappend=on: char device redirected to /dev/pts/0 (label charserial0) warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] 2018-04-05T21:48:43.153268Z qemu-system-x86_64: VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8 2018-04-05T21:48:43.153288Z qemu-system-x86_64: Failed to load virtio-blk:virtio 2018-04-05T21:48:43.153292Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:04.0/virtio-blk' 2018-04-05T21:48:43.153347Z qemu-system-x86_64: load of migration failed: Operation not permitted 2018-04-05 21:48:43.198+0000: shutting down, reason=crashed And in the n-cpu logs on the other host: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/screen-n-cpu.txt.gz#_Apr_05_21_48_43_257541 There is a related Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1450524 The CI job failures are at present using the Pike UCA: ii libvirt-bin 3.6.0-1ubuntu6.2~cloud0 ii qemu-system-x86 1:2.10+dfsg-0ubuntu3.5~cloud0 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1761798/+subscriptions ^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug 1761798] Re: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" [not found] <152303245198.3233.1033096993665779324.malonedeb@wampee.canonical.com> ` (6 preceding siblings ...) 2021-05-10 4:32 ` Thomas Huth @ 2021-05-19 9:53 ` ignazio 2021-05-19 15:18 ` ignazio ` (4 subsequent siblings) 12 siblings, 0 replies; 13+ messages in thread From: ignazio @ 2021-05-19 9:53 UTC (permalink / raw) To: qemu-devel I got same issue on centos 7 stein -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1761798 Title: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" Status in OpenStack Compute (nova): Confirmed Status in QEMU: Incomplete Bug description: Seen here: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz 2018-04-05T21:48:38.205752Z qemu-system-x86_64: -chardev pty,id=charserial0,logfile=/dev/fdset/1,logappend=on: char device redirected to /dev/pts/0 (label charserial0) warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] 2018-04-05T21:48:43.153268Z qemu-system-x86_64: VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8 2018-04-05T21:48:43.153288Z qemu-system-x86_64: Failed to load virtio-blk:virtio 2018-04-05T21:48:43.153292Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:04.0/virtio-blk' 2018-04-05T21:48:43.153347Z qemu-system-x86_64: load of migration failed: Operation not permitted 2018-04-05 21:48:43.198+0000: shutting down, reason=crashed And in the n-cpu logs on the other host: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/screen-n-cpu.txt.gz#_Apr_05_21_48_43_257541 There is a related Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1450524 The CI job failures are at present using the Pike UCA: ii libvirt-bin 3.6.0-1ubuntu6.2~cloud0 ii qemu-system-x86 1:2.10+dfsg-0ubuntu3.5~cloud0 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1761798/+subscriptions ^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug 1761798] Re: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" [not found] <152303245198.3233.1033096993665779324.malonedeb@wampee.canonical.com> ` (7 preceding siblings ...) 2021-05-19 9:53 ` ignazio @ 2021-05-19 15:18 ` ignazio 2021-05-19 15:50 ` Dr. David Alan Gilbert ` (3 subsequent siblings) 12 siblings, 0 replies; 13+ messages in thread From: ignazio @ 2021-05-19 15:18 UTC (permalink / raw) To: qemu-devel Hello, some news ....I wonder if they can help: I am testing with some virtual machine again. If I follows this steps it works (but I lost network connection): 1) Detach network interface from instance 2) Attach network interface to instance 3) Migrate instance 4) Loggin into instance using console and restart networking while if I restart networking before live migration it does not work. So, when someone mentioned ######################## we get this "guest index inconsistent" error when the migrated RAM is inconsistent with the migrated 'virtio' device state. And a common case is where a 'virtio' device does an operation after the vCPU is stopped and after RAM has been transmitted. #############################à the network traffic could be the problem ? Ignazio -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1761798 Title: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" Status in OpenStack Compute (nova): Confirmed Status in QEMU: Incomplete Bug description: Seen here: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz 2018-04-05T21:48:38.205752Z qemu-system-x86_64: -chardev pty,id=charserial0,logfile=/dev/fdset/1,logappend=on: char device redirected to /dev/pts/0 (label charserial0) warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] 2018-04-05T21:48:43.153268Z qemu-system-x86_64: VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8 2018-04-05T21:48:43.153288Z qemu-system-x86_64: Failed to load virtio-blk:virtio 2018-04-05T21:48:43.153292Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:04.0/virtio-blk' 2018-04-05T21:48:43.153347Z qemu-system-x86_64: load of migration failed: Operation not permitted 2018-04-05 21:48:43.198+0000: shutting down, reason=crashed And in the n-cpu logs on the other host: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/screen-n-cpu.txt.gz#_Apr_05_21_48_43_257541 There is a related Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1450524 The CI job failures are at present using the Pike UCA: ii libvirt-bin 3.6.0-1ubuntu6.2~cloud0 ii qemu-system-x86 1:2.10+dfsg-0ubuntu3.5~cloud0 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1761798/+subscriptions ^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug 1761798] Re: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" [not found] <152303245198.3233.1033096993665779324.malonedeb@wampee.canonical.com> ` (8 preceding siblings ...) 2021-05-19 15:18 ` ignazio @ 2021-05-19 15:50 ` Dr. David Alan Gilbert 2021-08-25 7:28 ` Thomas Huth ` (2 subsequent siblings) 12 siblings, 0 replies; 13+ messages in thread From: Dr. David Alan Gilbert @ 2021-05-19 15:50 UTC (permalink / raw) To: qemu-devel Be careful, it might not be the same bug. Yes, it *shouldn't* be a problem, but if the virtio code in qemu is broken then it will keep accepting incoming packets even when the guest is stopped in the final part of the migration and you get the contents of the RAM taken before the reception ofthe packet, but hte virtio state that's in the migration stream after the reception of the packet, and it's inconsistent. But the case the other reporter mentioned is on a virtio-blk device; the same thing can happen if the storage device stalls/is slow during the migration code - i.e. a block read takes ages to complete and happens to complete after the point it should have stopped for migration. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1761798 Title: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" Status in OpenStack Compute (nova): Confirmed Status in QEMU: Incomplete Bug description: Seen here: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz 2018-04-05T21:48:38.205752Z qemu-system-x86_64: -chardev pty,id=charserial0,logfile=/dev/fdset/1,logappend=on: char device redirected to /dev/pts/0 (label charserial0) warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] 2018-04-05T21:48:43.153268Z qemu-system-x86_64: VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8 2018-04-05T21:48:43.153288Z qemu-system-x86_64: Failed to load virtio-blk:virtio 2018-04-05T21:48:43.153292Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:04.0/virtio-blk' 2018-04-05T21:48:43.153347Z qemu-system-x86_64: load of migration failed: Operation not permitted 2018-04-05 21:48:43.198+0000: shutting down, reason=crashed And in the n-cpu logs on the other host: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/screen-n-cpu.txt.gz#_Apr_05_21_48_43_257541 There is a related Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1450524 The CI job failures are at present using the Pike UCA: ii libvirt-bin 3.6.0-1ubuntu6.2~cloud0 ii qemu-system-x86 1:2.10+dfsg-0ubuntu3.5~cloud0 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1761798/+subscriptions ^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug 1761798] Re: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" [not found] <152303245198.3233.1033096993665779324.malonedeb@wampee.canonical.com> ` (9 preceding siblings ...) 2021-05-19 15:50 ` Dr. David Alan Gilbert @ 2021-08-25 7:28 ` Thomas Huth 2021-10-25 4:17 ` Launchpad Bug Tracker 2021-10-25 4:17 ` Launchpad Bug Tracker 12 siblings, 0 replies; 13+ messages in thread From: Thomas Huth @ 2021-08-25 7:28 UTC (permalink / raw) To: qemu-devel Is this still happening with the latest release? ** Changed in: nova Status: Confirmed => Incomplete -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1761798 Title: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" Status in OpenStack Compute (nova): Incomplete Status in QEMU: Incomplete Bug description: Seen here: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz 2018-04-05T21:48:38.205752Z qemu-system-x86_64: -chardev pty,id=charserial0,logfile=/dev/fdset/1,logappend=on: char device redirected to /dev/pts/0 (label charserial0) warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] 2018-04-05T21:48:43.153268Z qemu-system-x86_64: VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8 2018-04-05T21:48:43.153288Z qemu-system-x86_64: Failed to load virtio-blk:virtio 2018-04-05T21:48:43.153292Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:04.0/virtio-blk' 2018-04-05T21:48:43.153347Z qemu-system-x86_64: load of migration failed: Operation not permitted 2018-04-05 21:48:43.198+0000: shutting down, reason=crashed And in the n-cpu logs on the other host: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live-migration/8de6e74/logs/screen-n- cpu.txt.gz#_Apr_05_21_48_43_257541 There is a related Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1450524 The CI job failures are at present using the Pike UCA: ii libvirt-bin 3.6.0-1ubuntu6.2~cloud0 ii qemu-system-x86 1:2.10+dfsg-0ubuntu3.5~cloud0 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1761798/+subscriptions ^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug 1761798] Re: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" [not found] <152303245198.3233.1033096993665779324.malonedeb@wampee.canonical.com> ` (10 preceding siblings ...) 2021-08-25 7:28 ` Thomas Huth @ 2021-10-25 4:17 ` Launchpad Bug Tracker 2021-10-25 4:17 ` Launchpad Bug Tracker 12 siblings, 0 replies; 13+ messages in thread From: Launchpad Bug Tracker @ 2021-10-25 4:17 UTC (permalink / raw) To: qemu-devel [Expired for OpenStack Compute (nova) because there has been no activity for 60 days.] ** Changed in: nova Status: Incomplete => Expired -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1761798 Title: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" Status in OpenStack Compute (nova): Expired Status in QEMU: Expired Bug description: Seen here: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz 2018-04-05T21:48:38.205752Z qemu-system-x86_64: -chardev pty,id=charserial0,logfile=/dev/fdset/1,logappend=on: char device redirected to /dev/pts/0 (label charserial0) warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] 2018-04-05T21:48:43.153268Z qemu-system-x86_64: VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8 2018-04-05T21:48:43.153288Z qemu-system-x86_64: Failed to load virtio-blk:virtio 2018-04-05T21:48:43.153292Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:04.0/virtio-blk' 2018-04-05T21:48:43.153347Z qemu-system-x86_64: load of migration failed: Operation not permitted 2018-04-05 21:48:43.198+0000: shutting down, reason=crashed And in the n-cpu logs on the other host: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live-migration/8de6e74/logs/screen-n- cpu.txt.gz#_Apr_05_21_48_43_257541 There is a related Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1450524 The CI job failures are at present using the Pike UCA: ii libvirt-bin 3.6.0-1ubuntu6.2~cloud0 ii qemu-system-x86 1:2.10+dfsg-0ubuntu3.5~cloud0 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1761798/+subscriptions ^ permalink raw reply [flat|nested] 13+ messages in thread
* [Bug 1761798] Re: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" [not found] <152303245198.3233.1033096993665779324.malonedeb@wampee.canonical.com> ` (11 preceding siblings ...) 2021-10-25 4:17 ` Launchpad Bug Tracker @ 2021-10-25 4:17 ` Launchpad Bug Tracker 12 siblings, 0 replies; 13+ messages in thread From: Launchpad Bug Tracker @ 2021-10-25 4:17 UTC (permalink / raw) To: qemu-devel [Expired for QEMU because there has been no activity for 60 days.] ** Changed in: qemu Status: Incomplete => Expired -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1761798 Title: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" Status in OpenStack Compute (nova): Expired Status in QEMU: Expired Bug description: Seen here: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live- migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz 2018-04-05T21:48:38.205752Z qemu-system-x86_64: -chardev pty,id=charserial0,logfile=/dev/fdset/1,logappend=on: char device redirected to /dev/pts/0 (label charserial0) warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] 2018-04-05T21:48:43.153268Z qemu-system-x86_64: VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8 2018-04-05T21:48:43.153288Z qemu-system-x86_64: Failed to load virtio-blk:virtio 2018-04-05T21:48:43.153292Z qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:04.0/virtio-blk' 2018-04-05T21:48:43.153347Z qemu-system-x86_64: load of migration failed: Operation not permitted 2018-04-05 21:48:43.198+0000: shutting down, reason=crashed And in the n-cpu logs on the other host: http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm- multinode-live-migration/8de6e74/logs/screen-n- cpu.txt.gz#_Apr_05_21_48_43_257541 There is a related Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1450524 The CI job failures are at present using the Pike UCA: ii libvirt-bin 3.6.0-1ubuntu6.2~cloud0 ii qemu-system-x86 1:2.10+dfsg-0ubuntu3.5~cloud0 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1761798/+subscriptions ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2021-10-25 4:32 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <152303245198.3233.1033096993665779324.malonedeb@wampee.canonical.com>
2020-11-23 15:17 ` [Bug 1761798] Re: live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8" Balazs Gibizer
2020-11-23 15:30 ` sean mooney
2020-11-23 16:56 ` Kashyap Chamarthy
2020-11-24 12:17 ` Dr. David Alan Gilbert
2020-11-25 13:44 ` Balazs Gibizer
2020-11-25 14:03 ` Dr. David Alan Gilbert
2021-05-10 4:32 ` Thomas Huth
2021-05-19 9:53 ` ignazio
2021-05-19 15:18 ` ignazio
2021-05-19 15:50 ` Dr. David Alan Gilbert
2021-08-25 7:28 ` Thomas Huth
2021-10-25 4:17 ` Launchpad Bug Tracker
2021-10-25 4:17 ` Launchpad Bug Tracker
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.