From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:39954) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Ula1s-0002FN-3C for qemu-devel@nongnu.org; Sun, 09 Jun 2013 03:27:11 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Ula1p-0006Xm-5R for qemu-devel@nongnu.org; Sun, 09 Jun 2013 03:27:08 -0400 References: <51A7036A.3050407@ozlabs.ru> <51A7049F.6040207@redhat.com> <51A70B3D.90609@ozlabs.ru> <51A71705.6060009@kamp.de> <51A74D79.7040204@redhat.com> <2765FDFA-8050-4AA3-8621-7E9EA2C89F9C@kamp.de> <51A764FC.7080705@redhat.com> <51ADF122.70307@kamp.de> <51ADF637.7060804@redhat.com> <51ADFBCE.3080200@kamp.de> <51ADFC7A.7030009@redhat.com> <51AE035A.5070301@kamp.de> <51B2EB0A.7000704@linux.vnet.ibm.com> <51B2EBA2.5060401@ozlabs.ru> <51B3E58C.50301@linux.vnet.ibm.com> <51B3E9A8.5010705@ozlabs.ru> <51B3EFFA.4040608@linux.vnet.ibm.com> <51B3F1FD.1090401@ozlabs.ru> Mime-Version: 1.0 (1.0) In-Reply-To: <51B3F1FD.1090401@ozlabs.ru> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-Id: From: Peter Lieven Date: Sun, 9 Jun 2013 09:27:01 +0200 Subject: Re: [Qemu-devel] broken incoming migration List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alexey Kardashevskiy Cc: Paolo Bonzini , David Gibson , "qemu-ppc@nongnu.org" , Wenchao Xia , "qemu-devel@nongnu.org" Am 09.06.2013 um 05:09 schrieb Alexey Kardashevskiy : > On 06/09/2013 01:01 PM, Wenchao Xia wrote: >> =E4=BA=8E 2013-6-9 10:34, Alexey Kardashevskiy =E5=86=99=E9=81=93: >>> On 06/09/2013 12:16 PM, Wenchao Xia wrote: >>>> =E4=BA=8E 2013-6-8 16:30, Alexey Kardashevskiy =E5=86=99=E9=81=93: >>>>> On 06/08/2013 06:27 PM, Wenchao Xia wrote: >>>>>>> On 04.06.2013 16:40, Paolo Bonzini wrote: >>>>>>>> Il 04/06/2013 16:38, Peter Lieven ha scritto: >>>>>>>>> On 04.06.2013 16:14, Paolo Bonzini wrote: >>>>>>>>>> Il 04/06/2013 15:52, Peter Lieven ha scritto: >>>>>>>>>>> On 30.05.2013 16:41, Paolo Bonzini wrote: >>>>>>>>>>>> Il 30/05/2013 16:38, Peter Lieven ha scritto: >>>>>>>>>>>>>>> You could also scan the page for nonzero values before >>>>>>>>>>>>>>> writing it. >>>>>>>>>>>>> i had this in mind, but then choosed the other approach.... tu= rned >>>>>>>>>>>>> out to be a bad idea. >>>>>>>>>>>>>=20 >>>>>>>>>>>>> alexey: i will prepare a patch later today, could you then ple= ase >>>>>>>>>>>>> verify it fixes your problem. >>>>>>>>>>>>>=20 >>>>>>>>>>>>> paolo: would we still need the madvise or is it enough to not >>>>>>>>>>>>> write >>>>>>>>>>>>> the zeroes? >>>>>>>>>>>> It should be enough to not write them. >>>>>>>>>>> Problem: checking the pages for zero allocates them. even at the= >>>>>>>>>>> source. >>>>>>>>>> It doesn't look like. I tried this program and top doesn't show a= n >>>>>>>>>> increasing amount of reserved memory: >>>>>>>>>>=20 >>>>>>>>>> #include >>>>>>>>>> #include >>>>>>>>>> int main() >>>>>>>>>> { >>>>>>>>>> char *x =3D malloc(500 << 20); >>>>>>>>>> int i, j; >>>>>>>>>> for (i =3D 0; i < 500; i +=3D 10) { >>>>>>>>>> for (j =3D 0; j < 10 << 20; j +=3D 4096) { >>>>>>>>>> *(volatile char*) (x + (i << 20) + j); >>>>>>>>>> } >>>>>>>>>> getchar(); >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>> strange. we are talking about RSS size, right? >>>>>>>> None of the three top values change, and only VIRT is >500 MB. >>>>>>>>=20 >>>>>>>>> is the malloc above using mmapped memory? >>>>>>>> Yes. >>>>>>>>=20 >>>>>>>>> which kernel version do you use? >>>>>>>> 3.9. >>>>>>>>=20 >>>>>>>>> what avoids allocating the memory for me is the following (with >>>>>>>>> whatever side effects it has ;-)) >>>>>>>> This would also fail to migrate any page that is swapped out, break= ing >>>>>>>> overcommit in a more subtle way. :) >>>>>>>>=20 >>>>>>>> Paolo >>>>>>> the following does also not allocate memory, but qemu does... >>>>>> Hi, Peter >>>>>> As the patch writes >>>>>>=20 >>>>>> "not sending zero pages breaks migration if a page is zero >>>>>> at the source but not at the destination." >>>>>>=20 >>>>>> I don't understand why it would be trouble, shouldn't all page >>>>>> not received in dest be treated as zero pages? >>>>>=20 >>>>>=20 >>>>> How would the destination guest know if some page must be cleared? The= >>>>> previous patch (which Peter reverted) did not send anything for the pa= ges >>>>> which were zero on the source side. >>>> If an page was not received and destination knows that page should >>>> exist according to total size, fill it with zero at destination, would >>>> it solve the problem? >>>=20 >>> It is _live_ migration, the source sends changes, same pages can change a= nd >>> be sent several times. So we would need to turn tracking on on the >>> destination to know if some page was received from the source or changed= by >>> the destination itself (by writing there bios/firmware images, etc) and >>> then clear pages which were touched by the destination and were not sent= by >>> the source. >> OK, I can understand the problem is, for example: >> Destination boots up with 0x0000-0xFFFF filled with bios image. >> Source forgot to send zero pages in 0x0000-0xFFFF. >=20 >=20 > The source did not forget, instead it zeroed these pages during its life > and thought that they must be zeroed at the destination already (as the > destination did not start and did not have a chance to write something the= re). >=20 >=20 >> After migration destination got 0x0000-0xFFFF dirty(different with >> source) >=20 > Yep. And those pages were empty on the source what made debugging very eas= y :) >=20 >=20 >> Thanks for explain. >>=20 >> This seems refer to the migration protocol: how should the guest treat >> unsent pages. The patch causing the problem, actually treat zero pages >> as "not to sent" at source, but another half is missing: treat "not >> received" as zero pages at destination. I guess if second half is added, >> problem is gone: >> after page transfer completed, before destination resume, >> fill zero in "not received" pages. >=20 >=20 >=20 > Make a working patch, we'll discuss it :) I do not see much acceleration > coming from there. I would also not spent much time with this. I would either look to find an e= asy way to fix the initialization code to not unneccessarily load data into R= AM or i will sent a v2 of my patch following Eric's concerns. Peter >=20 >=20 > --=20 > Alexey