From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:39745) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bob1G-0001Nh-2G for qemu-devel@nongnu.org; Mon, 26 Sep 2016 14:52:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bob1A-0001S8-Gq for qemu-devel@nongnu.org; Mon, 26 Sep 2016 14:52:49 -0400 Received: from mx1.redhat.com ([209.132.183.28]:56962) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bob1A-0001Rj-82 for qemu-devel@nongnu.org; Mon, 26 Sep 2016 14:52:44 -0400 Date: Mon, 26 Sep 2016 19:52:36 +0100 From: "Dr. David Alan Gilbert" Message-ID: <20160926185235.GJ2029@work-vm> References: <5feb15.7e53.1576070ae2d.Coremail.lichunguang@hust.edu.cn> <20160926112349.GF2029@work-vm> <13289d.86da.15766fdf27c.Coremail.lichunguang@hust.edu.cn> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <13289d.86da.15766fdf27c.Coremail.lichunguang@hust.edu.cn> Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Chunguang Li Cc: qemu-devel@nongnu.org, amit.shah@redhat.com, pbonzini@redhat.com, stefanha@redhat.com, quintela@redhat.com * Chunguang Li (lichunguang@hust.edu.cn) wrote: >=20 >=20 >=20 > > -----=E5=8E=9F=E5=A7=8B=E9=82=AE=E4=BB=B6----- > > =E5=8F=91=E4=BB=B6=E4=BA=BA: "Dr. David Alan Gilbert" > > =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2016=E5=B9=B49=E6=9C=8826=E6=97= =A5 =E6=98=9F=E6=9C=9F=E4=B8=80 > > =E6=94=B6=E4=BB=B6=E4=BA=BA: "Chunguang Li" > > =E6=8A=84=E9=80=81: qemu-devel@nongnu.org, amit.shah@redhat.com, pbon= zini@redhat.com, stefanha@redhat.com, quintela@redhat.com > > =E4=B8=BB=E9=A2=98: Re: [Qemu-devel] Migration dirty bitmap: should o= nly mark pages as dirty after they have been sent > >=20 > > * Chunguang Li (lichunguang@hust.edu.cn) wrote: > > > Hi all! > > > I have some confusion about the dirty bitmap during migration. I ha= ve digged into the code. I figure out that every now and then during migr= ation, the dirty bitmap will be grabbed from the kernel space through ioc= tl(KVM_GET_DIRTY_LOG), and then be used to update qemu's dirty bitmap. Ho= wever I think this mechanism leads to resendness of some NON-dirty pages. > > >=20 > > > Take the first iteration of precopy for instance, during which all = the pages will be sent. Before that during the migration setup, the ioctl= (KVM_GET_DIRTY_LOG) is called once, so the kernel begins to produce the d= irty bitmap from this moment. When the pages "that haven't been sent" are= written, the kernel space marks them as dirty. However I don't think thi= s is correct, because these pages will be sent during this and the next i= terations with the same content (if they are not written again after they= are sent). It only makes sense to mark the pages which have already been= sent during one iteration as dirty when they are written. > > >=20 > > >=20 > > > Am I right about this consideration? If I am right, is there some a= dvice to improve this? > >=20 > > I think you're right that this can happen; to clarify I think the > > case you're talking about is: > >=20 > > Iteration 1 > > sync bitmap > > start sending pages > > page 'n' is modified - but hasn't been sent yet > > page 'n' gets sent > > Iteration 2 > > sync bitmap > > 'page n is shown as modified' > > send page 'n' again > > >=20 > Yes=EF=BC=8Cthis is right the case I am talking about. > =20 > > So you're right that is wasteful; I guess it's more wasteful > > on big VMs with slow networks where the length of each iteration > > is large. >=20 > I think this is "very" wasteful. Assume the workload writes the pages d= irty randomly within the guest address space, and the transfer speed is c= onstant. Intuitively, I think nearly half of the dirty pages produced in = Iteration 1 is not really dirty. This means the time of Iteration 2 is do= uble of that to send only really dirty pages. Yes, it's probably pretty bad; and we really need to do something like split the sync into smaller chunks; there are other suggestions for how to improve it (e.g. there's the page-modification-logging changes). However, I don't think you usually get really random writes, if you do precopy rarely converges at all, because even without your observation it changes lots and lots of pages. Dave > Thanks, >=20 > Chunguang >=20 > >=20 > > Fixing it is not easy, because you have to be really careful > > never to miss a page modification, even if the page is sent > > about the same time it's dirtied. > >=20 > > One way would be to sync the dirty log from the kernel > > in smaller chunks. > >=20 > > Dave > >=20 > >=20 > > >=20 > > > Thanks, > > > Chunguang Li > > -- > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >=20 >=20 > -- > Chunguang Li, Ph.D. Candidate > Wuhan National Laboratory for Optoelectronics (WNLO) > Huazhong University of Science & Technology (HUST) > Wuhan, Hubei Prov., China >=20 >=20 >=20 -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK