From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:60393) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zw6fC-00073d-5b for qemu-devel@nongnu.org; Tue, 10 Nov 2015 06:00:40 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Zw6f7-0003Zl-Ap for qemu-devel@nongnu.org; Tue, 10 Nov 2015 06:00:34 -0500 Received: from mx1.redhat.com ([209.132.183.28]:45558) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zw6f6-0003Zc-RB for qemu-devel@nongnu.org; Tue, 10 Nov 2015 06:00:29 -0500 Date: Tue, 10 Nov 2015 11:00:25 +0000 From: Stefan Hajnoczi Message-ID: <20151110110025.GB19772@stefanha-x1.localdomain> References: <563E16AF.8070209@openvz.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="JYK4vJDZwFMowpUq" Content-Disposition: inline In-Reply-To: Subject: Re: [Qemu-devel] assert during internal snapshot List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Li, Liang Z" Cc: Amit Shah , "Denis V. Lunev" , Juan Quintela , QEMU , Paolo Bonzini --JYK4vJDZwFMowpUq Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Nov 09, 2015 at 03:29:13AM +0000, Li, Liang Z wrote: > > -----Original Message----- > > From: Denis V. Lunev [mailto:den@openvz.org] > > Sent: Saturday, November 07, 2015 11:20 PM > > To: Li, Liang Z; Paolo Bonzini; Juan Quintela; Amit Shah > > Cc: QEMU > > Subject: assert during internal snapshot > >=20 > > Hello, All! > >=20 > > This commit > >=20 > > commit 94f5a43704129ca4995aa3385303c5ae225bde42 > > Author: Liang Li > > Date: Mon Nov 2 15:37:00 2015 +0800 > >=20 > > migration: defer migration_end & blk_mig_cleanup > >=20 > > Because of the patch 3ea3b7fa9af067982f34b of kvm, which introduce= s a > > lazy collapsing of small sptes into large sptes mechanism, now > > migration_end() is a time consuming operation because it calls > > memroy_global_dirty_log_stop(), which will trigger the dropping of= small > > sptes operation and takes about dozens of milliseconds, so call > > migration_end() before all the vmsate data has already been transf= erred > > to the destination will prolong VM downtime. This operation should= be > > deferred after all the data has been transferred to the destinatio= n. > >=20 > > blk_mig_cleanup() can be deferred too. > >=20 > > For a VM with 8G RAM, this patch can reduce the VM downtime about > > 30 ms. > >=20 > > Signed-off-by: Liang Li > > Reviewed-by: Paolo Bonzini > > Reviewed-by: Juan Quintela al3 > > Reviewed-by: Amit Shah al3 > > Signed-off-by: Juan Quintela al3 > >=20 > > introduces the following regression > >=20 > > (gdb) bt > > #0 0x00007fd5d314a267 in __GI_raise (sig=3Dsig@entry=3D6) > > at ../sysdeps/unix/sysv/linux/raise.c:55 > > #1 0x00007fd5d314beca in __GI_abort () at abort.c:89 > > #2 0x00007fd5d314303d in __assert_fail_base ( > > fmt=3D0x7fd5d32a5028 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", > > assertion=3Dassertion@entry=3D0x557288ed5b69 "i !=3D mr->ioeventfd= _nb", > > file=3Dfile@entry=3D0x557288ed5a36 "/home/den/src/qemu/memory.c", > > line=3Dline@entry=3D1731, > > function=3Dfunction@entry=3D0x557288ed5fb0 <__PRETTY_FUNCTION__.32= 545> > > "memory_region_del_eventfd") at assert.c:92 > > #3 0x00007fd5d31430f2 in __GI___assert_fail ( > > assertion=3D0x557288ed5b69 "i !=3D mr->ioeventfd_nb", > > file=3D0x557288ed5a36 "/home/den/src/qemu/memory.c", line=3D1731, > > function=3D0x557288ed5fb0 <__PRETTY_FUNCTION__.32545> > > "memory_region_del_eventfd") at assert.c:101 > > #4 0x0000557288b108fa in memory_region_del_eventfd > > (mr=3D0x55728ad83700, > > addr=3D16, size=3D2, match_data=3Dtrue, data=3D0, e=3D0x55728b21ff= 40) > > at /home/den/src/qemu/memory.c:1731 > > #5 0x0000557288d9fc18 in virtio_pci_set_host_notifier_internal ( > > proxy=3D0x55728ad82e80, n=3D0, assign=3Dfalse, set_handler=3Dfalse) > > at hw/virtio/virtio-pci.c:178 > > #6 0x0000557288da19a9 in virtio_pci_set_host_notifier (d=3D0x55728ad82= e80, > > n=3D0, > > assign=3Dfalse) at hw/virtio/virtio-pci.c:984 > > #7 0x0000557288b523df in virtio_scsi_dataplane_start (s=3D0x55728ad8af= a0) > > at /home/den/src/qemu/hw/scsi/virtio-scsi-dataplane.c:268 > > #8 0x0000557288b50210 in virtio_scsi_handle_cmd (vdev=3D0x55728ad8afa0, > > vq=3D0x55728b21ffc0) at /home/den/src/qemu/hw/scsi/virtio-scsi.c:5= 74 > > #9 0x0000557288b65cb7 in virtio_queue_notify_vq (vq=3D0x55728b21ffc0) > > at /home/den/src/qemu/hw/virtio/virtio.c:966 > > #10 0x0000557288b67bbf in virtio_queue_host_notifier_read > > (n=3D0x55728b220010) > > at /home/den/src/qemu/hw/virtio/virtio.c:1643 > > #11 0x0000557288e12a2b in aio_dispatch (ctx=3D0x55728acaeab0) at > > aio-posix.c:160 > > #12 0x0000557288e03194 in aio_ctx_dispatch (source=3D0x55728acaeab0, > > callback=3D0x0, user_data=3D0x0) at async.c:226 > > #13 0x00007fd5d409fff7 in g_main_context_dispatch () > > from /lib/x86_64-linux-gnu/libglib-2.0.so.0 > > ---Type to continue, or q to quit--- > > #14 0x0000557288e1110d in glib_pollfds_poll () at main-loop.c:211 > > #15 0x0000557288e111e8 in os_host_main_loop_wait (timeout=3D0) at > > main-loop.c:256 > > #16 0x0000557288e11295 in main_loop_wait (nonblocking=3D0) at main- > > loop.c:504 > > #17 0x0000557288c1c31c in main_loop () at vl.c:1890 > > #18 0x0000557288c23dec in main (argc=3D105, argv=3D0x7ffca9a6fa08, > > envp=3D0x7ffca9a6fd58) at vl.c:4644 > > (gdb) > >=20 > > during 'virsh create-snapshot' operation over alive VM. > > It happens 100% of time when VM is run using the following command line: > >=20 > > 7498 ? tl 0:37 qemu-system-x86_64 -enable-kvm -name rhel7 > > -S -machine pc-i440fx-2.2,accel=3Dkvm,usb=3Doff -cpu SandyBridge -m 102= 4 - > > realtime mlock=3Doff -smp 1,sockets=3D1,cores=3D1,threads=3D1 -object > > iothread,id=3Diothread1 -uuid 456af4d3-5d67-41c6-a229-c55ded6098e9 > > -no-user-config -nodefaults -chardev > > socket,id=3Dcharmonitor,path=3D/var/lib/libvirt/qemu/rhel7.monitor,serv= er,nowait > > -mon chardev=3Dcharmonitor,id=3Dmonitor,mode=3Dcontrol -rtc > > base=3Dutc,driftfix=3Dslew -global kvm-pit.lost_tick_policy=3Ddiscard -= no-hpet -no- > > shutdown -global PIIX4_PM.disable_s3=3D1 -global PIIX4_PM.disable_s4=3D= 1 -boot > > strict=3Don -device ich9-usb-ehci1,id=3Dusb,bus=3Dpci.0,addr=3D0x6.0x7 > > -device > > ich9-usb- > > uhci1,masterbus=3Dusb.0,firstport=3D0,bus=3Dpci.0,multifunction=3Don,ad= dr=3D0x6 > > -device > > ich9-usb-uhci2,masterbus=3Dusb.0,firstport=3D2,bus=3Dpci.0,addr=3D0x6.0= x1 > > -device > > ich9-usb-uhci3,masterbus=3Dusb.0,firstport=3D4,bus=3Dpci.0,addr=3D0x6.0= x2 > > -device virtio-scsi-pci,id=3Dscsi0,bus=3Dpci.0,addr=3D0x5 -device > > virtio-serial-pci,id=3Dvirtio-serial0,bus=3Dpci.0,addr=3D0x7 -drive > > file=3D/var/lib/libvirt/images/rhel7.qcow2,if=3Dnone,id=3Ddrive-scsi0-0= -0- > > 0,format=3Dqcow2,cache=3Dnone,aio=3Dnative > > -device > > scsi-hd,bus=3Dscsi0.0,channel=3D0,scsi-id=3D0,lun=3D0,drive=3Ddrive-scs= i0-0-0-0,id=3Dscsi0-0- > > 0-0,bootindex=3D1 > > -drive > > file=3D/home/den/tmp/CentOS-7.0-1406-x86_64-DVD.iso,if=3Dnone,id=3Ddriv= e-ide0-0- > > 0,readonly=3Don,format=3Draw > > -device ide-cd,bus=3Dide.0,unit=3D0,drive=3Ddrive-ide0-0-0,id=3Dide0-0-= 0 -netdev > > tap,fd=3D24,id=3Dhostnet0,vhost=3Don,vhostfd=3D25 -device > > virtio-net- > > pci,netdev=3Dhostnet0,id=3Dnet0,mac=3D52:54:00:97:2f:1d,bus=3Dpci.0,add= r=3D0x3 > > -chardev > > socket,id=3Dcharchannel0,path=3D/var/lib/libvirt/qemu/channel/target/rh= el7.org.qe > > mu.guest_agent.0,server,nowait > > -device > > virtserialport,bus=3Dvirtio- > > serial0.0,nr=3D1,chardev=3Dcharchannel0,id=3Dchannel0,name=3Dorg.qemu.g= uest_agen > > t.0 > > -chardev spicevmc,id=3Dcharchannel1,name=3Dvdagent -device > > virtserialport,bus=3Dvirtio- > > serial0.0,nr=3D2,chardev=3Dcharchannel1,id=3Dchannel1,name=3Dcom.redhat= =2Espice.0 > > -chardev pty,id=3Dcharconsole0 -device > > virtconsole,chardev=3Dcharconsole0,id=3Dconsole0 -device > > usb-tablet,id=3Dinput0 -spice > > port=3D5900,addr=3D127.0.0.1,disable-ticketing,seamless-migration=3Don = -device > > qxl- > > vga,id=3Dvideo0,ram_size=3D67108864,vram_size=3D67108864,vgamem_mb=3D16= ,bus=3D > > pci.0,addr=3D0x2 > > -device intel-hda,id=3Dsound0,bus=3Dpci.0,addr=3D0x4 -device > > hda-duplex,id=3Dsound0-codec0,bus=3Dsound0.0,cad=3D0 -chardev > > spicevmc,id=3Dcharredir0,name=3Dusbredir -device > > usb-redir,chardev=3Dcharredir0,id=3Dredir0 -chardev > > spicevmc,id=3Dcharredir1,name=3Dusbredir -device > > usb-redir,chardev=3Dcharredir1,id=3Dredir1 -chardev > > spicevmc,id=3Dcharredir2,name=3Dusbredir -device > > usb-redir,chardev=3Dcharredir2,id=3Dredir2 -chardev > > spicevmc,id=3Dcharredir3,name=3Dusbredir -device > > usb-redir,chardev=3Dcharredir3,id=3Dredir3 -device > > virtio-balloon-pci,id=3Dballoon0,bus=3Dpci.0,addr=3D0x8 -set > > device.scsi0.iothread=3Diothread1 -msg timestamp=3Don > >=20 > > Minimal dumb (!!!) fix is the following: > >=20 > > hades ~/src/qemu $ cat 1.diff > > diff --git a/migration/ram.c b/migration/ram.c index 25e9eeb..25ebf0d 1= 00644 > > --- a/migration/ram.c > > +++ b/migration/ram.c > > @@ -1100,7 +1100,8 @@ static void migration_bitmap_free(struct BitmapRcu > > *bmap) > > g_free(bmap); > > } > >=20 > > -static void migration_end(void) > > +extern void migration_end(void); > > +void migration_end(void) > > { > > /* caller have hold iothread lock or is in a bh, so there is > > * no writing race against this migration_bitmap diff --git > > a/migration/savevm.c b/migration/savevm.c index dbcc39a..01da865 100644 > > --- a/migration/savevm.c > > +++ b/migration/savevm.c > > @@ -914,6 +914,7 @@ void qemu_savevm_state_cancel(void) > > } > > } > >=20 > > +extern void migration_end(void); > > static int qemu_savevm_state(QEMUFile *f, Error **errp) > > { > > int ret; > > @@ -942,6 +943,8 @@ static int qemu_savevm_state(QEMUFile *f, Error > > **errp) > > qemu_savevm_state_complete(f); > > ret =3D qemu_file_get_error(f); > > } > > + migration_end(); > > + > > if (ret !=3D 0) { > > qemu_savevm_state_cancel(); > > error_setg_errno(errp, -ret, "Error while writing VM state");= hades > > ~/src/qemu $ > >=20 > > (patch attached to start a discussion). > >=20 > > For me it looks like commit is wrong and should be reverted and reworke= d. > >=20 > > Den >=20 > Cc to : Stefan >=20 > It seems the 'bdrv_drain_all()' in ' blk_mig_cleanup ()' should not be = deferred. Stefan, is that right? Denis, do you still experience this crash with your own savevm patch series applied? Since that series does AioContext acquire/release where missing in savevm, it might fix this bug. Stefan --JYK4vJDZwFMowpUq Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJWQc5IAAoJEJykq7OBq3PIvAEH/32t94fk+v836XHiPXpkkmj8 /6sgb7JRMYfzm0Q/UXlMPGeJ9B77fFyq3hgpmho+MzTEsuI5B+8DxUk3H3r9uqd3 BmHblzksvaHaENSfRzuVFqFubwxwuP3jTgd1tvnOoKeoSSVwQ+mYl0IpGtdeDFF3 AoOppL2fzRZZ8cs2vHAIc9yqalTIXqylFAFltfNcZ24CCz9T68hhNJtVZDjYeWBV CZSQzFUTyZpouqpEmMyBLBnclKjq8O0KR2PN9Zt6P/YIleXvwCWdzOAKRa9SrRae xep1e8Jao1ahJYWI6x/TxGYhIVz/fg+2IaroL6uCy3OCPaJUkjJc2iOOWesPJRg= =7C6A -----END PGP SIGNATURE----- --JYK4vJDZwFMowpUq--