* [Qemu-devel] qemu process crash: Assertion failed: QLIST_EMPTY(&bs->tracked_requests) @ 2017-12-07 10:18 Fernando Casas Schössow 2017-12-11 11:56 ` Stefan Hajnoczi 0 siblings, 1 reply; 10+ messages in thread From: Fernando Casas Schössow @ 2017-12-07 10:18 UTC (permalink / raw) To: qemu-devel Hi there, Last night while doing a backup of a guest using the live snapshot mechanism the qemu process for the guest seem to had crashed. The snapshot succeeded then the backup of the VM disk had place and also succeeded but the commit to the original disk after the backup seem to have failed. The command I use in the script to take the snapshot is: virsh snapshot-create-as --domain $VM backup-job.qcow2 --disk-only --atomic --quiesce --no-metadata And then to commit back is: virsh blockcommit $VM $TARGETDISK --base $DISKFILE --top $SNAPFILE --active --pivot In the qemu log for the guest I found the following while the commit back was having place: Assertion failed: QLIST_EMPTY(&bs->tracked_requests) (/home/buildozer/aports/main/qemu/src/qemu-2.10.1/block/mirror.c: mirror_run: 884) I'm running qemu 2.10.1 with libvirt 3.9.0 and kernel 4.9.65 on Alpine Linux 3.7. This is the complete guest info from the logs: LC_ALL=C PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin HOME=/root USER=root QEMU_AUDIO_DRV=spice /usr/bin/qemu-system-x86_64 -name guest=DOCKER01,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-6-DOCKER01/master-key.aes -machine pc-i440fx-2.8,accel=kvm,usb=off,dump-guest-core=off -cpu IvyBridge,ss=on,vmx=on,pcid=on,hypervisor=on,arat=on,tsc_adjust=on,xsaveopt=on -drive file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/DOCKER01_VARS.fd,if=pflash,format=raw,unit=1 -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 4705b146-3b14-4c20-923c-42105d47e7fc -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-6-DOCKER01/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x4.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x4 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x4.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x4.0x2 -device ahci,id=sata0,bus=pci.0,addr=0x9 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/storage/storage-ssd-vms/virtual_machines_ssd/docker01.qcow2,format=qcow2,if=none,id=drive-sata0-0-0,cache=none,aio=threads -device ide-hd,bus=sata0.0,drive=drive-sata0-0-0,id=sata0-0-0,bootindex=1 -netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=35 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:1c:af:ce,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-6-DOCKER01/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -spice port=5905,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x8 -msg timestamp=on I was running on qemu 2.8.1 for months and didn't have any problems with the backups but yesterday I updated to qemu 2.10.1 and I hit this problem last night. Is this a bug? Any ideas will be appreciated. Thanks. Kind regards, Fer ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Qemu-devel] qemu process crash: Assertion failed: QLIST_EMPTY(&bs->tracked_requests) 2017-12-07 10:18 [Qemu-devel] qemu process crash: Assertion failed: QLIST_EMPTY(&bs->tracked_requests) Fernando Casas Schössow @ 2017-12-11 11:56 ` Stefan Hajnoczi 2017-12-11 13:42 ` Fernando Casas Schössow 0 siblings, 1 reply; 10+ messages in thread From: Stefan Hajnoczi @ 2017-12-11 11:56 UTC (permalink / raw) To: Fernando Casas Schössow; +Cc: qemu-devel, qemu-block, Jeff Cody [-- Attachment #1: Type: text/plain, Size: 4328 bytes --] On Thu, Dec 07, 2017 at 10:18:52AM +0000, Fernando Casas Schössow wrote: > Hi there, > > > Last night while doing a backup of a guest using the live snapshot mechanism the qemu process for the guest seem to had crashed. > > The snapshot succeeded then the backup of the VM disk had place and also succeeded but the commit to the original disk after the backup seem to have failed. > > The command I use in the script to take the snapshot is: > > > virsh snapshot-create-as --domain $VM backup-job.qcow2 --disk-only --atomic --quiesce --no-metadata > > > And then to commit back is: > > > virsh blockcommit $VM $TARGETDISK --base $DISKFILE --top $SNAPFILE --active --pivot > > > In the qemu log for the guest I found the following while the commit back was having place: > > > Assertion failed: QLIST_EMPTY(&bs->tracked_requests) (/home/buildozer/aports/main/qemu/src/qemu-2.10.1/block/mirror.c: mirror_run: 884) > > I'm running qemu 2.10.1 with libvirt 3.9.0 and kernel 4.9.65 on Alpine Linux 3.7. > > This is the complete guest info from the logs: > > > LC_ALL=C PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin HOME=/root USER=root QEMU_AUDIO_DRV=spice /usr/bin/qemu-system-x86_64 -name guest=DOCKER01,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-6-DOCKER01/master-key.aes -machine pc-i440fx-2.8,accel=kvm,usb=off,dump-guest-core=off -cpu IvyBridge,ss=on,vmx=on,pcid=on,hypervisor=on,arat=on,tsc_adjust=on,xsaveopt=on -drive file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/DOCKER01_VARS.fd,if=pflash,format=raw,unit=1 -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 4705b146-3b14-4c20-923c-42105d47e7fc -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-6-DOCKER01/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x4.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x4 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x4.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x4.0x2 -device ahci,id=sata0,bus=pci.0,addr=0x9 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/storage/storage-ssd-vms/virtual_machines_ssd/docker01.qcow2,format=qcow2,if=none,id=drive-sata0-0-0,cache=none,aio=threads -device ide-hd,bus=sata0.0,drive=drive-sata0-0-0,id=sata0-0-0,bootindex=1 -netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=35 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:1c:af:ce,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-6-DOCKER01/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -spice port=5905,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x8 -msg timestamp=on > > > > I was running on qemu 2.8.1 for months and didn't have any problems with the backups but yesterday I updated to qemu 2.10.1 and I hit this problem last night. > > > Is this a bug? Any ideas will be appreciated. Thanks for reporting this bug. Can you reproduce it reliably? Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 455 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Qemu-devel] qemu process crash: Assertion failed: QLIST_EMPTY(&bs->tracked_requests) 2017-12-11 11:56 ` Stefan Hajnoczi @ 2017-12-11 13:42 ` Fernando Casas Schössow 2017-12-13 11:23 ` Stefan Hajnoczi 0 siblings, 1 reply; 10+ messages in thread From: Fernando Casas Schössow @ 2017-12-11 13:42 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: qemu-devel, qemu-block, Jeff Cody Hello Stefan, Thanks for your reply. Fortunately I didn’t have the problem again and it’s not clear how it can be consistently reproduced. Daily backups are running as usual at the moment. If there is anything I can do from my side or if you have any ideas to try to reproduce it let me know. Thanks. Fer Sent from my iPhone On 11 Dec 2017, at 12:56, Stefan Hajnoczi <stefanha@gmail.com<mailto:stefanha@gmail.com>> wrote: On Thu, Dec 07, 2017 at 10:18:52AM +0000, Fernando Casas Schössow wrote: Hi there, Last night while doing a backup of a guest using the live snapshot mechanism the qemu process for the guest seem to had crashed. The snapshot succeeded then the backup of the VM disk had place and also succeeded but the commit to the original disk after the backup seem to have failed. The command I use in the script to take the snapshot is: virsh snapshot-create-as --domain $VM backup-job.qcow2 --disk-only --atomic --quiesce --no-metadata And then to commit back is: virsh blockcommit $VM $TARGETDISK --base $DISKFILE --top $SNAPFILE --active --pivot In the qemu log for the guest I found the following while the commit back was having place: Assertion failed: QLIST_EMPTY(&bs->tracked_requests) (/home/buildozer/aports/main/qemu/src/qemu-2.10.1/block/mirror.c: mirror_run: 884) I'm running qemu 2.10.1 with libvirt 3.9.0 and kernel 4.9.65 on Alpine Linux 3.7. This is the complete guest info from the logs: LC_ALL=C PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin HOME=/root USER=root QEMU_AUDIO_DRV=spice /usr/bin/qemu-system-x86_64 -name guest=DOCKER01,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-6-DOCKER01/master-key.aes -machine pc-i440fx-2.8,accel=kvm,usb=off,dump-guest-core=off -cpu IvyBridge,ss=on,vmx=on,pcid=on,hypervisor=on,arat=on,tsc_adjust=on,xsaveopt=on -drive file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/DOCKER01_VARS.fd,if=pflash,format=raw,unit=1 -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 4705b146-3b14-4c20-923c-42105d47e7fc -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-6-DOCKER01/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x4.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x4 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x4.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x4.0x2 -device ahci,id=sata0,bus=pci.0,addr=0x9 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/storage/storage-ssd-vms/virtual_machines_ssd/docker01.qcow2,format=qcow2,if=none,id=drive-sata0-0-0,cache=none,aio=threads -device ide-hd,bus=sata0.0,drive=drive-sata0-0-0,id=sata0-0-0,bootindex=1 -netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=35 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:1c:af:ce,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-6-DOCKER01/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -spice port=5905,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x8 -msg timestamp=on I was running on qemu 2.8.1 for months and didn't have any problems with the backups but yesterday I updated to qemu 2.10.1 and I hit this problem last night. Is this a bug? Any ideas will be appreciated. Thanks for reporting this bug. Can you reproduce it reliably? Stefan ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Qemu-devel] qemu process crash: Assertion failed: QLIST_EMPTY(&bs->tracked_requests) 2017-12-11 13:42 ` Fernando Casas Schössow @ 2017-12-13 11:23 ` Stefan Hajnoczi 2017-12-13 15:33 ` Fernando Casas Schössow 0 siblings, 1 reply; 10+ messages in thread From: Stefan Hajnoczi @ 2017-12-13 11:23 UTC (permalink / raw) To: Fernando Casas Schössow; +Cc: qemu-devel, qemu-block, Jeff Cody [-- Attachment #1: Type: text/plain, Size: 2052 bytes --] On Mon, Dec 11, 2017 at 01:42:38PM +0000, Fernando Casas Schössow wrote: > Hello Stefan, > > Thanks for your reply. > Fortunately I didn’t have the problem again and it’s not clear how it can be consistently reproduced. Daily backups are running as usual at the moment. > > If there is anything I can do from my side or if you have any ideas to try to reproduce it let me know. Okay, thanks. In case it happens again, here is some information about the assertion failure. It will probably be necessary to use GDB to inspect the failed process (core dump). The starting point for debugging is mirror_run() in block/mirror.c: if (cnt == 0 && should_complete) { /* The dirty bitmap is not updated while operations are pending. * If we're about to exit, wait for pending operations before * calling bdrv_get_dirty_count(bs), or we may exit while the * source has dirty data to copy! * * Note that I/O can be submitted by the guest while * mirror_populate runs, so pause it now. Before deciding * whether to switch to target check one last time if I/O has * come in the meanwhile, and if not flush the data to disk. */ trace_mirror_before_drain(s, cnt); bdrv_drained_begin(bs); cnt = bdrv_get_dirty_count(s->dirty_bitmap); if (cnt > 0 || mirror_flush(s) < 0) { bdrv_drained_end(bs); continue; } /* The two disks are in sync. Exit and report successful * completion. */ assert(QLIST_EMPTY(&bs->tracked_requests)); The reason why this assertion makes sense is because bdrv_drained_begin(bs) starts a region where I/O is quiesced (all requests should have been completed). Either there is a bug with blockjobs vs bdrv_drained_debug(), or maybe mirror_flush() submitted new I/O requests. It would be interesting to print the bs->tracked_requests linked list in the GDB debugger. Those requests might give a clue about the root cause. Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 455 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Qemu-devel] qemu process crash: Assertion failed: QLIST_EMPTY(&bs->tracked_requests) 2017-12-13 11:23 ` Stefan Hajnoczi @ 2017-12-13 15:33 ` Fernando Casas Schössow 2018-01-29 16:01 ` Stefan Hajnoczi 0 siblings, 1 reply; 10+ messages in thread From: Fernando Casas Schössow @ 2017-12-13 15:33 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: qemu-devel, qemu-block, Jeff Cody Thanks for the explanation Stefan. Maybe I’m missing something here but, if I recall correctly, the qemu process for the guest is terminated when this problem happens. So how a debugger will be attached to a process that is gone? Also I don’t think the Alpine packages for qemu include the debug info. :( But I’m not 100% sure at the moment. Sent from my iPhone On 13 Dec 2017, at 12:23, Stefan Hajnoczi <stefanha@gmail.com<mailto:stefanha@gmail.com>> wrote: On Mon, Dec 11, 2017 at 01:42:38PM +0000, Fernando Casas Schössow wrote: Hello Stefan, Thanks for your reply. Fortunately I didn’t have the problem again and it’s not clear how it can be consistently reproduced. Daily backups are running as usual at the moment. If there is anything I can do from my side or if you have any ideas to try to reproduce it let me know. Okay, thanks. In case it happens again, here is some information about the assertion failure. It will probably be necessary to use GDB to inspect the failed process (core dump). The starting point for debugging is mirror_run() in block/mirror.c: if (cnt == 0 && should_complete) { /* The dirty bitmap is not updated while operations are pending. * If we're about to exit, wait for pending operations before * calling bdrv_get_dirty_count(bs), or we may exit while the * source has dirty data to copy! * * Note that I/O can be submitted by the guest while * mirror_populate runs, so pause it now. Before deciding * whether to switch to target check one last time if I/O has * come in the meanwhile, and if not flush the data to disk. */ trace_mirror_before_drain(s, cnt); bdrv_drained_begin(bs); cnt = bdrv_get_dirty_count(s->dirty_bitmap); if (cnt > 0 || mirror_flush(s) < 0) { bdrv_drained_end(bs); continue; } /* The two disks are in sync. Exit and report successful * completion. */ assert(QLIST_EMPTY(&bs->tracked_requests)); The reason why this assertion makes sense is because bdrv_drained_begin(bs) starts a region where I/O is quiesced (all requests should have been completed). Either there is a bug with blockjobs vs bdrv_drained_debug(), or maybe mirror_flush() submitted new I/O requests. It would be interesting to print the bs->tracked_requests linked list in the GDB debugger. Those requests might give a clue about the root cause. Stefan ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Qemu-devel] qemu process crash: Assertion failed: QLIST_EMPTY(&bs->tracked_requests) 2017-12-13 15:33 ` Fernando Casas Schössow @ 2018-01-29 16:01 ` Stefan Hajnoczi 2018-10-18 12:25 ` Fernando Casas Schössow 0 siblings, 1 reply; 10+ messages in thread From: Stefan Hajnoczi @ 2018-01-29 16:01 UTC (permalink / raw) To: Fernando Casas Schössow; +Cc: qemu-devel, qemu-block, Jeff Cody [-- Attachment #1: Type: text/plain, Size: 697 bytes --] On Wed, Dec 13, 2017 at 03:33:01PM +0000, Fernando Casas Schössow wrote: > Maybe I’m missing something here but, if I recall correctly, the qemu process for the guest is terminated when this problem happens. So how a debugger will be attached to a process that is gone? Sorry, this got lost in my inbox. assert(false) sends SIGABRT to the process. The default behavior is to dump a core file that can be analyzed later with GDB. Your system must have core dumps enabled (i.e. ulimit -c unlimited). Also, various pieces of software like systemd's coredumpctl can influence where and how core dumps are stored. But in short, an assertion failure produces a core dump. Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 455 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Qemu-devel] qemu process crash: Assertion failed: QLIST_EMPTY(&bs->tracked_requests) 2018-01-29 16:01 ` Stefan Hajnoczi @ 2018-10-18 12:25 ` Fernando Casas Schössow 2018-10-18 12:38 ` [Qemu-devel] [Qemu-block] " Kevin Wolf 0 siblings, 1 reply; 10+ messages in thread From: Fernando Casas Schössow @ 2018-10-18 12:25 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: qemu-devel, qemu-block, Jeff Cody Hi Stefan, I hope this email finds you well and I apologize in advance for resurrecting this thread. I'm currently running on qemu 2.12.1 and I'm still having this issue every few days but now I managed to get a core dump generated (without including the guest memory). Would you take a look at the dump? Please let me know how do you prefer me to share it. The file is around 290MB as is but I can try to compress it. Looking forward your reply. Thanks Kind regards, Fernando ________________________________ From: Stefan Hajnoczi <stefanha@gmail.com> Sent: Monday, January 29, 2018 5:01 PM To: Fernando Casas Schössow Cc: qemu-devel; qemu-block@nongnu.org; Jeff Cody Subject: Re: [Qemu-devel] qemu process crash: Assertion failed: QLIST_EMPTY(&bs->tracked_requests) On Wed, Dec 13, 2017 at 03:33:01PM +0000, Fernando Casas Schössow wrote: > Maybe I’m missing something here but, if I recall correctly, the qemu process for the guest is terminated when this problem happens. So how a debugger will be attached to a process that is gone? Sorry, this got lost in my inbox. assert(false) sends SIGABRT to the process. The default behavior is to dump a core file that can be analyzed later with GDB. Your system must have core dumps enabled (i.e. ulimit -c unlimited). Also, various pieces of software like systemd's coredumpctl can influence where and how core dumps are stored. But in short, an assertion failure produces a core dump. Stefan ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Qemu-devel] [Qemu-block] qemu process crash: Assertion failed: QLIST_EMPTY(&bs->tracked_requests) 2018-10-18 12:25 ` Fernando Casas Schössow @ 2018-10-18 12:38 ` Kevin Wolf 2018-10-18 13:40 ` Fernando Casas Schössow 0 siblings, 1 reply; 10+ messages in thread From: Kevin Wolf @ 2018-10-18 12:38 UTC (permalink / raw) To: Fernando Casas Schössow Cc: Stefan Hajnoczi, Jeff Cody, qemu-devel, qemu-block Hi Fernando, Am 18.10.2018 um 14:25 hat Fernando Casas Schössow geschrieben: > I hope this email finds you well and I apologize in advance for resurrecting this thread. > I'm currently running on qemu 2.12.1 and I'm still having this issue every few days but now I managed to get a core dump generated (without including the guest memory). > Would you take a look at the dump? Please let me know how do you prefer me to share it. The file is around 290MB as is but I can try to compress it. would it be possible for you to test whether the bug still exists in QEMU 3.0? There were a few fixes related to request draining in that release, so maybe it's already solved now. Kevin ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Qemu-devel] [Qemu-block] qemu process crash: Assertion failed: QLIST_EMPTY(&bs->tracked_requests) 2018-10-18 12:38 ` [Qemu-devel] [Qemu-block] " Kevin Wolf @ 2018-10-18 13:40 ` Fernando Casas Schössow [not found] ` <AM6PR10MB2247D63F532AE33155BA57EFB7F80@AM6PR10MB2247.EURPRD10.PROD.OUTLOOK.C OM> 0 siblings, 1 reply; 10+ messages in thread From: Fernando Casas Schössow @ 2018-10-18 13:40 UTC (permalink / raw) To: Kevin Wolf; +Cc: Stefan Hajnoczi, Jeff Cody, qemu-devel, qemu-block Hi Kevin, Not at the moment. This is a production system and pretty much up to date but can't upgrade to 3.0 yet. If the dump can be of any use, I can upload it somewhere for analysis. BR, Fernando On jue, oct 18, 2018 at 2:38 PM, Kevin Wolf <kwolf@redhat.com> wrote: Hi Fernando, Am 18.10.2018 um 14:25 hat Fernando Casas Schössow geschrieben: I hope this email finds you well and I apologize in advance for resurrecting this thread. I'm currently running on qemu 2.12.1 and I'm still having this issue every few days but now I managed to get a core dump generated (without including the guest memory). Would you take a look at the dump? Please let me know how do you prefer me to share it. The file is around 290MB as is but I can try to compress it. would it be possible for you to test whether the bug still exists in QEMU 3.0? There were a few fixes related to request draining in that release, so maybe it's already solved now. Kevin ^ permalink raw reply [flat|nested] 10+ messages in thread
[parent not found: <AM6PR10MB2247D63F532AE33155BA57EFB7F80@AM6PR10MB2247.EURPRD10.PROD.OUTLOOK.C OM>]
* Re: [Qemu-devel] [Qemu-block] qemu process crash: Assertion failed: QLIST_EMPTY(&bs->tracked_requests) [not found] ` <AM6PR10MB2247D63F532AE33155BA57EFB7F80@AM6PR10MB2247.EURPRD10.PROD.OUTLOOK.C OM> @ 2018-11-16 11:04 ` Fernando Casas Schössow 0 siblings, 0 replies; 10+ messages in thread From: Fernando Casas Schössow @ 2018-11-16 11:04 UTC (permalink / raw) To: Fernando Casas Schössow Cc: Kevin Wolf, Stefan Hajnoczi, Jeff Cody, qemu-devel, qemu-block Hi again, I hit another occurrence of this issue and collected the qemy dump as well. So at the moment I have two qemu dumps for this issue happening on two different guests. I wish I know how to analyze them myself but is not the case so if anyone in the list is willing to take a look I'm more than willing to share them. Thanks in advance. Fernando On jue, oct 18, 2018 at 3:40 PM, Fernando Casas Schössow <casasfernando@hotmail.com> wrote: Hi Kevin, Not at the moment. This is a production system and pretty much up to date but can't upgrade to 3.0 yet. If the dump can be of any use, I can upload it somewhere for analysis. BR, Fernando On jue, oct 18, 2018 at 2:38 PM, Kevin Wolf <kwolf@redhat.com> wrote: Hi Fernando, Am 18.10.2018 um 14:25 hat Fernando Casas Schössow geschrieben: I hope this email finds you well and I apologize in advance for resurrecting this thread. I'm currently running on qemu 2.12.1 and I'm still having this issue every few days but now I managed to get a core dump generated (without including the guest memory). Would you take a look at the dump? Please let me know how do you prefer me to share it. The file is around 290MB as is but I can try to compress it. would it be possible for you to test whether the bug still exists in QEMU 3.0? There were a few fixes related to request draining in that release, so maybe it's already solved now. Kevin ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2018-11-16 11:05 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-12-07 10:18 [Qemu-devel] qemu process crash: Assertion failed: QLIST_EMPTY(&bs->tracked_requests) Fernando Casas Schössow
2017-12-11 11:56 ` Stefan Hajnoczi
2017-12-11 13:42 ` Fernando Casas Schössow
2017-12-13 11:23 ` Stefan Hajnoczi
2017-12-13 15:33 ` Fernando Casas Schössow
2018-01-29 16:01 ` Stefan Hajnoczi
2018-10-18 12:25 ` Fernando Casas Schössow
2018-10-18 12:38 ` [Qemu-devel] [Qemu-block] " Kevin Wolf
2018-10-18 13:40 ` Fernando Casas Schössow
[not found] ` <AM6PR10MB2247D63F532AE33155BA57EFB7F80@AM6PR10MB2247.EURPRD10.PROD.OUTLOOK.C OM>
2018-11-16 11:04 ` Fernando Casas Schössow
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.