From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34400) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1boXGB-00047L-M3 for qemu-devel@nongnu.org; Mon, 26 Sep 2016 10:52:01 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1boXG5-0007rB-Kl for qemu-devel@nongnu.org; Mon, 26 Sep 2016 10:51:58 -0400 Received: from [202.114.0.240] (port=55165 helo=mail.hust.edu.cn) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1boXG4-0007nt-Q3 for qemu-devel@nongnu.org; Mon, 26 Sep 2016 10:51:53 -0400 Date: Mon, 26 Sep 2016 22:55:01 +0800 (GMT+08:00) From: "Chunguang Li" Message-ID: <13289d.86da.15766fdf27c.Coremail.lichunguang@hust.edu.cn> In-Reply-To: <20160926112349.GF2029@work-vm> References: <5feb15.7e53.1576070ae2d.Coremail.lichunguang@hust.edu.cn> <20160926112349.GF2029@work-vm> MIME-Version: 1.0 Content-Type: text/plain; charset=GBK Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Dr. David Alan Gilbert" Cc: qemu-devel@nongnu.org, amit.shah@redhat.com, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com > -----=D4=AD=CA=BC=D3=CA=BC=FE----- > =B7=A2=BC=FE=C8=CB: "Dr. David Alan Gilbert" > =B7=A2=CB=CD=CA=B1=BC=E4: 2016=C4=EA9=D4=C226=C8=D5 =D0=C7=C6=DA=D2=BB > =CA=D5=BC=FE=C8=CB: "Chunguang Li" > =B3=AD=CB=CD: qemu-devel@nongnu.org, amit.shah@redhat.com, pbonzini@redha= t.com, stefanha@redhat.com, quintela@redhat.com > =D6=F7=CC=E2: Re: [Qemu-devel] Migration dirty bitmap: should only mark p= ages as dirty after they have been sent >=20 > * Chunguang Li (lichunguang@hust.edu.cn) wrote: > > Hi all! > > I have some confusion about the dirty bitmap during migration. I have d= igged into the code. I figure out that every now and then during migration,= the dirty bitmap will be grabbed from the kernel space through ioctl(KVM_G= ET_DIRTY_LOG), and then be used to update qemu's dirty bitmap. However I th= ink this mechanism leads to resendness of some NON-dirty pages. > >=20 > > Take the first iteration of precopy for instance, during which all the = pages will be sent. Before that during the migration setup, the ioctl(KVM_G= ET_DIRTY_LOG) is called once, so the kernel begins to produce the dirty bit= map from this moment. When the pages "that haven't been sent" are written, = the kernel space marks them as dirty. However I don't think this is correct= , because these pages will be sent during this and the next iterations with= the same content (if they are not written again after they are sent). It o= nly makes sense to mark the pages which have already been sent during one i= teration as dirty when they are written. > >=20 > >=20 > > Am I right about this consideration? If I am right, is there some advic= e to improve this? >=20 > I think you're right that this can happen; to clarify I think the > case you're talking about is: >=20 > Iteration 1 > sync bitmap > start sending pages > page 'n' is modified - but hasn't been sent yet > page 'n' gets sent > Iteration 2 > sync bitmap > 'page n is shown as modified' > send page 'n' again > Yes=A3=ACthis is right the case I am talking about. =20 > So you're right that is wasteful; I guess it's more wasteful > on big VMs with slow networks where the length of each iteration > is large. I think this is "very" wasteful. Assume the workload writes the pages dirty= randomly within the guest address space, and the transfer speed is constan= t. Intuitively, I think nearly half of the dirty pages produced in Iteratio= n 1 is not really dirty. This means the time of Iteration 2 is double of th= at to send only really dirty pages. Thanks, Chunguang >=20 > Fixing it is not easy, because you have to be really careful > never to miss a page modification, even if the page is sent > about the same time it's dirtied. >=20 > One way would be to sync the dirty log from the kernel > in smaller chunks. >=20 > Dave >=20 >=20 > >=20 > > Thanks, > > Chunguang Li > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- Chunguang Li, Ph.D. Candidate Wuhan National Laboratory for Optoelectronics (WNLO) Huazhong University of Science & Technology (HUST) Wuhan, Hubei Prov., China