* [Qemu-devel] QEMU LIve Migration - swap_free: Bad swap file entry @ 2014-02-05 17:51 Stefan Priebe 2014-02-05 20:15 ` Dr. David Alan Gilbert 2014-02-06 11:14 ` [Qemu-devel] [pve-devel] " Alexandre DERUMIER 0 siblings, 2 replies; 50+ messages in thread From: Stefan Priebe @ 2014-02-05 17:51 UTC (permalink / raw) To: pve-devel, qemu-devel Hello, after live migrating machines with a lot of memory (32GB, 48GB, ...) i see pretty often crashing services after migration and the guest kernel prints: [1707620.031806] swap_free: Bad swap file entry 00377410 [1707620.031806] swap_free: Bad swap file entry 00593c48 [1707620.031807] swap_free: Bad swap file entry 03201430 [1707620.031807] swap_free: Bad swap file entry 01bc5900 [1707620.031807] swap_free: Bad swap file entry 0173ce40 [1707620.031808] swap_free: Bad swap file entry 011c0270 [1707620.031808] swap_free: Bad swap file entry 03c58ae8 [1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:1 val:1536 [1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:2 val:-1536 Qemu is 1.7 Does anybody know a fix? Greets, Stefan ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-05 17:51 [Qemu-devel] QEMU LIve Migration - swap_free: Bad swap file entry Stefan Priebe @ 2014-02-05 20:15 ` Dr. David Alan Gilbert 2014-02-06 7:20 ` Stefan Priebe - Profihost AG 2014-02-06 11:14 ` [Qemu-devel] [pve-devel] " Alexandre DERUMIER 1 sibling, 1 reply; 50+ messages in thread From: Dr. David Alan Gilbert @ 2014-02-05 20:15 UTC (permalink / raw) To: Stefan Priebe; +Cc: qemu-devel, pve-devel * Stefan Priebe (s.priebe@profihost.ag) wrote: > Hello, > > after live migrating machines with a lot of memory (32GB, 48GB, ...) > i see pretty often crashing services after migration and the guest > kernel prints: > > [1707620.031806] swap_free: Bad swap file entry 00377410 > [1707620.031806] swap_free: Bad swap file entry 00593c48 > [1707620.031807] swap_free: Bad swap file entry 03201430 > [1707620.031807] swap_free: Bad swap file entry 01bc5900 > [1707620.031807] swap_free: Bad swap file entry 0173ce40 > [1707620.031808] swap_free: Bad swap file entry 011c0270 > [1707620.031808] swap_free: Bad swap file entry 03c58ae8 > [1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380 > idx:1 val:1536 > [1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380 > idx:2 val:-1536 > > Qemu is 1.7 > > Does anybody know a fix? I don't, but some more information about: 1) What guest you're running 2) The configuration of your hosts 3) The command line (or XML if you're running libvirt) for your qemu so we can see what devices you're running. would probably help people. Do you get any messages on either the source or destination qemu during the migrate? Dave -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-05 20:15 ` Dr. David Alan Gilbert @ 2014-02-06 7:20 ` Stefan Priebe - Profihost AG 2014-02-06 10:22 ` Orit Wasserman 0 siblings, 1 reply; 50+ messages in thread From: Stefan Priebe - Profihost AG @ 2014-02-06 7:20 UTC (permalink / raw) To: Dr. David Alan Gilbert; +Cc: qemu-devel, pve-devel Am 05.02.2014 21:15, schrieb Dr. David Alan Gilbert: > * Stefan Priebe (s.priebe@profihost.ag) wrote: >> Hello, >> >> after live migrating machines with a lot of memory (32GB, 48GB, ...) >> i see pretty often crashing services after migration and the guest >> kernel prints: >> >> [1707620.031806] swap_free: Bad swap file entry 00377410 >> [1707620.031806] swap_free: Bad swap file entry 00593c48 >> [1707620.031807] swap_free: Bad swap file entry 03201430 >> [1707620.031807] swap_free: Bad swap file entry 01bc5900 >> [1707620.031807] swap_free: Bad swap file entry 0173ce40 >> [1707620.031808] swap_free: Bad swap file entry 011c0270 >> [1707620.031808] swap_free: Bad swap file entry 03c58ae8 >> [1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380 >> idx:1 val:1536 >> [1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380 >> idx:2 val:-1536 >> >> Qemu is 1.7 >> >> Does anybody know a fix? > > I don't, but some more information about: > 1) What guest you're running Linux guest the output is also from the guest. Kernel 3.10.26 > 2) The configuration of your hosts What do you mean by that? > 3) The command line (or XML if you're running libvirt) for > your qemu so we can see what devices you're running. qemu -chardev socket,id=qmp,path=/var/run/qemu-server/179.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/179.vnc,x509,password -pidfile /var/run/qemu-server/179.pid -daemonize -name K31953 -smp sockets=1,cores=16 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000 -vga cirrus -cpu kvm64,+lahf_lm,+x2apic,+sep -k de -m 32768 -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -drive if=none,id=drive-ide2,media=cdrom,aio=native -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive file=rbd:...,if=none,id=drive-scsi0,iops_rd=1000,iops_wr=500,bps_rd=314572800,bps_wr=209715200,aio=native,discard=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100 -netdev type=tap,id=net0,ifname=tap179i0,script=/var/lib/qemu-server/pve-bridge,vhost=on -device virtio-net-pci,mac=CA:CA:23:AC:2D:C5,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -rtc base=localtime -machine type=pc-i440fx-1.7 > Do you get any messages on either the source or destination > qemu during the migrate? no Stefan ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-06 7:20 ` Stefan Priebe - Profihost AG @ 2014-02-06 10:22 ` Orit Wasserman 2014-02-06 10:49 ` Stefan Priebe - Profihost AG 0 siblings, 1 reply; 50+ messages in thread From: Orit Wasserman @ 2014-02-06 10:22 UTC (permalink / raw) To: Stefan Priebe - Profihost AG, Dr. David Alan Gilbert Cc: qemu-devel, pve-devel On 02/06/2014 09:20 AM, Stefan Priebe - Profihost AG wrote: > Am 05.02.2014 21:15, schrieb Dr. David Alan Gilbert: >> * Stefan Priebe (s.priebe@profihost.ag) wrote: >>> Hello, >>> >>> after live migrating machines with a lot of memory (32GB, 48GB, ...) >>> i see pretty often crashing services after migration and the guest >>> kernel prints: >>> >>> [1707620.031806] swap_free: Bad swap file entry 00377410 >>> [1707620.031806] swap_free: Bad swap file entry 00593c48 >>> [1707620.031807] swap_free: Bad swap file entry 03201430 >>> [1707620.031807] swap_free: Bad swap file entry 01bc5900 >>> [1707620.031807] swap_free: Bad swap file entry 0173ce40 >>> [1707620.031808] swap_free: Bad swap file entry 011c0270 >>> [1707620.031808] swap_free: Bad swap file entry 03c58ae8 >>> [1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380 >>> idx:1 val:1536 >>> [1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380 >>> idx:2 val:-1536 >>> Is this live migration with shared storage? what kind of shared storage? Does this happens with smaller guests? >>> Qemu is 1.7 >>> >>> Does anybody know a fix? >> >> I don't, but some more information about: >> 1) What guest you're running > > Linux guest the output is also from the guest. Kernel 3.10.26 > >> 2) The configuration of your hosts > > What do you mean by that? > >> 3) The command line (or XML if you're running libvirt) for >> your qemu so we can see what devices you're running. > > qemu -chardev > socket,id=qmp,path=/var/run/qemu-server/179.qmp,server,nowait -mon > chardev=qmp,mode=control -vnc > unix:/var/run/qemu-server/179.vnc,x509,password -pidfile > /var/run/qemu-server/179.pid -daemonize -name K31953 -smp > sockets=1,cores=16 -nodefaults -boot > menu=on,strict=on,reboot-timeout=1000 -vga cirrus -cpu > kvm64,+lahf_lm,+x2apic,+sep -k de -m 32768 -device > piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device > usb-tablet,id=tablet,bus=uhci.0,port=1 -device > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -drive > if=none,id=drive-ide2,media=cdrom,aio=native -device > ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -device > virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive > file=rbd:...,if=none,id=drive-scsi0,iops_rd=1000,iops_wr=500,bps_rd=314572800,bps_wr=209715200,aio=native,discard=on > -device > scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100 > -netdev > type=tap,id=net0,ifname=tap179i0,script=/var/lib/qemu-server/pve-bridge,vhost=on > -device > virtio-net-pci,mac=CA:CA:23:AC:2D:C5,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 > -rtc base=localtime -machine type=pc-i440fx-1.7 > >> Do you get any messages on either the source or destination >> qemu during the migrate? > > no > > Stefan > ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-06 10:22 ` Orit Wasserman @ 2014-02-06 10:49 ` Stefan Priebe - Profihost AG 0 siblings, 0 replies; 50+ messages in thread From: Stefan Priebe - Profihost AG @ 2014-02-06 10:49 UTC (permalink / raw) To: Orit Wasserman, Dr. David Alan Gilbert; +Cc: qemu-devel, pve-devel Am 06.02.2014 11:22, schrieb Orit Wasserman: > On 02/06/2014 09:20 AM, Stefan Priebe - Profihost AG wrote: >> Am 05.02.2014 21:15, schrieb Dr. David Alan Gilbert: >>> * Stefan Priebe (s.priebe@profihost.ag) wrote: >>>> Hello, >>>> >>>> after live migrating machines with a lot of memory (32GB, 48GB, ...) >>>> i see pretty often crashing services after migration and the guest >>>> kernel prints: >>>> >>>> [1707620.031806] swap_free: Bad swap file entry 00377410 >>>> [1707620.031806] swap_free: Bad swap file entry 00593c48 >>>> [1707620.031807] swap_free: Bad swap file entry 03201430 >>>> [1707620.031807] swap_free: Bad swap file entry 01bc5900 >>>> [1707620.031807] swap_free: Bad swap file entry 0173ce40 >>>> [1707620.031808] swap_free: Bad swap file entry 011c0270 >>>> [1707620.031808] swap_free: Bad swap file entry 03c58ae8 >>>> [1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380 >>>> idx:1 val:1536 >>>> [1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380 >>>> idx:2 val:-1536 >>>> > > Is this live migration with shared storage? what kind of shared storage? Yes - i'm using ceph / rbd. > Does this happens with smaller guests? Never seen that. Always with guest having memory > 16GB. >>>> Qemu is 1.7 >>>> >>>> Does anybody know a fix? >>> >>> I don't, but some more information about: >>> 1) What guest you're running >> >> Linux guest the output is also from the guest. Kernel 3.10.26 >> >>> 2) The configuration of your hosts >> >> What do you mean by that? >> >>> 3) The command line (or XML if you're running libvirt) for >>> your qemu so we can see what devices you're running. >> >> qemu -chardev >> socket,id=qmp,path=/var/run/qemu-server/179.qmp,server,nowait -mon >> chardev=qmp,mode=control -vnc >> unix:/var/run/qemu-server/179.vnc,x509,password -pidfile >> /var/run/qemu-server/179.pid -daemonize -name K31953 -smp >> sockets=1,cores=16 -nodefaults -boot >> menu=on,strict=on,reboot-timeout=1000 -vga cirrus -cpu >> kvm64,+lahf_lm,+x2apic,+sep -k de -m 32768 -device >> piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device >> usb-tablet,id=tablet,bus=uhci.0,port=1 -device >> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -drive >> if=none,id=drive-ide2,media=cdrom,aio=native -device >> ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -device >> virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive >> file=rbd:...,if=none,id=drive-scsi0,iops_rd=1000,iops_wr=500,bps_rd=314572800,bps_wr=209715200,aio=native,discard=on >> >> -device >> scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100 >> >> -netdev >> type=tap,id=net0,ifname=tap179i0,script=/var/lib/qemu-server/pve-bridge,vhost=on >> >> -device >> virtio-net-pci,mac=CA:CA:23:AC:2D:C5,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 >> >> -rtc base=localtime -machine type=pc-i440fx-1.7 >> >>> Do you get any messages on either the source or destination >>> qemu during the migrate? no Stefan ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-05 17:51 [Qemu-devel] QEMU LIve Migration - swap_free: Bad swap file entry Stefan Priebe 2014-02-05 20:15 ` Dr. David Alan Gilbert @ 2014-02-06 11:14 ` Alexandre DERUMIER 2014-02-06 11:19 ` Stefan Priebe - Profihost AG 1 sibling, 1 reply; 50+ messages in thread From: Alexandre DERUMIER @ 2014-02-06 11:14 UTC (permalink / raw) To: Stefan Priebe; +Cc: qemu-devel, pve-devel Do you force rbd_cache=true in ceph.conf? if yes, do you use cache=writeback ? according to ceph doc: http://ceph.com/docs/next/rbd/qemu-rbd/ "Important If you set rbd_cache=true, you must set cache=writeback or risk data loss. Without cache=writeback, QEMU will not send flush requests to librbd. If QEMU exits uncleanly in this configuration, filesystems on top of rbd can be corrupted." ----- Mail original ----- De: "Stefan Priebe" <s.priebe@profihost.ag> À: pve-devel@pve.proxmox.com, "qemu-devel" <qemu-devel@nongnu.org> Envoyé: Mercredi 5 Février 2014 18:51:15 Objet: [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry Hello, after live migrating machines with a lot of memory (32GB, 48GB, ...) i see pretty often crashing services after migration and the guest kernel prints: [1707620.031806] swap_free: Bad swap file entry 00377410 [1707620.031806] swap_free: Bad swap file entry 00593c48 [1707620.031807] swap_free: Bad swap file entry 03201430 [1707620.031807] swap_free: Bad swap file entry 01bc5900 [1707620.031807] swap_free: Bad swap file entry 0173ce40 [1707620.031808] swap_free: Bad swap file entry 011c0270 [1707620.031808] swap_free: Bad swap file entry 03c58ae8 [1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:1 val:1536 [1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:2 val:-1536 Qemu is 1.7 Does anybody know a fix? Greets, Stefan _______________________________________________ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-06 11:14 ` [Qemu-devel] [pve-devel] " Alexandre DERUMIER @ 2014-02-06 11:19 ` Stefan Priebe - Profihost AG 2014-02-06 11:40 ` Alexandre DERUMIER 0 siblings, 1 reply; 50+ messages in thread From: Stefan Priebe - Profihost AG @ 2014-02-06 11:19 UTC (permalink / raw) To: Alexandre DERUMIER; +Cc: qemu-devel, pve-devel Am 06.02.2014 12:14, schrieb Alexandre DERUMIER: > Do you force rbd_cache=true in ceph.conf? no > if yes, do you use cache=writeback ? yes So this should be safe. PS: all my guests do not even have !!SWAP!! # free|grep Swap Swap: 0 0 0 Stefan > according to ceph doc: > http://ceph.com/docs/next/rbd/qemu-rbd/ > > "Important If you set rbd_cache=true, you must set cache=writeback or risk data loss. Without cache=writeback, QEMU will not send flush requests to librbd. If QEMU exits uncleanly in this configuration, filesystems on top of rbd can be corrupted." > > > > ----- Mail original ----- > > De: "Stefan Priebe" <s.priebe@profihost.ag> > À: pve-devel@pve.proxmox.com, "qemu-devel" <qemu-devel@nongnu.org> > Envoyé: Mercredi 5 Février 2014 18:51:15 > Objet: [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry > > Hello, > > after live migrating machines with a lot of memory (32GB, 48GB, ...) i > see pretty often crashing services after migration and the guest kernel > prints: > > [1707620.031806] swap_free: Bad swap file entry 00377410 > [1707620.031806] swap_free: Bad swap file entry 00593c48 > [1707620.031807] swap_free: Bad swap file entry 03201430 > [1707620.031807] swap_free: Bad swap file entry 01bc5900 > [1707620.031807] swap_free: Bad swap file entry 0173ce40 > [1707620.031808] swap_free: Bad swap file entry 011c0270 > [1707620.031808] swap_free: Bad swap file entry 03c58ae8 > [1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:1 > val:1536 > [1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:2 > val:-1536 > > Qemu is 1.7 > > Does anybody know a fix? > > Greets, > Stefan > _______________________________________________ > pve-devel mailing list > pve-devel@pve.proxmox.com > http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-06 11:19 ` Stefan Priebe - Profihost AG @ 2014-02-06 11:40 ` Alexandre DERUMIER 2014-02-06 12:10 ` Stefan Priebe - Profihost AG 0 siblings, 1 reply; 50+ messages in thread From: Alexandre DERUMIER @ 2014-02-06 11:40 UTC (permalink / raw) To: Stefan Priebe - Profihost AG; +Cc: qemu-devel, pve-devel >>PS: all my guests do not even have !!SWAP!! Not sure is related to swap file. I found an similar problem here, triggered with suspend/resume on ext4 http://lkml.indiana.edu/hypermail/linux/kernel/1106.3/01340.html Maybe is it a guest kernel bug ? ----- Mail original ----- De: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag> À: "Alexandre DERUMIER" <aderumier@odiso.com> Cc: pve-devel@pve.proxmox.com, "qemu-devel" <qemu-devel@nongnu.org> Envoyé: Jeudi 6 Février 2014 12:19:36 Objet: Re: [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry Am 06.02.2014 12:14, schrieb Alexandre DERUMIER: > Do you force rbd_cache=true in ceph.conf? no > if yes, do you use cache=writeback ? yes So this should be safe. PS: all my guests do not even have !!SWAP!! # free|grep Swap Swap: 0 0 0 Stefan > according to ceph doc: > http://ceph.com/docs/next/rbd/qemu-rbd/ > > "Important If you set rbd_cache=true, you must set cache=writeback or risk data loss. Without cache=writeback, QEMU will not send flush requests to librbd. If QEMU exits uncleanly in this configuration, filesystems on top of rbd can be corrupted." > > > > ----- Mail original ----- > > De: "Stefan Priebe" <s.priebe@profihost.ag> > À: pve-devel@pve.proxmox.com, "qemu-devel" <qemu-devel@nongnu.org> > Envoyé: Mercredi 5 Février 2014 18:51:15 > Objet: [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry > > Hello, > > after live migrating machines with a lot of memory (32GB, 48GB, ...) i > see pretty often crashing services after migration and the guest kernel > prints: > > [1707620.031806] swap_free: Bad swap file entry 00377410 > [1707620.031806] swap_free: Bad swap file entry 00593c48 > [1707620.031807] swap_free: Bad swap file entry 03201430 > [1707620.031807] swap_free: Bad swap file entry 01bc5900 > [1707620.031807] swap_free: Bad swap file entry 0173ce40 > [1707620.031808] swap_free: Bad swap file entry 011c0270 > [1707620.031808] swap_free: Bad swap file entry 03c58ae8 > [1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:1 > val:1536 > [1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:2 > val:-1536 > > Qemu is 1.7 > > Does anybody know a fix? > > Greets, > Stefan > _______________________________________________ > pve-devel mailing list > pve-devel@pve.proxmox.com > http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-06 11:40 ` Alexandre DERUMIER @ 2014-02-06 12:10 ` Stefan Priebe - Profihost AG 2014-02-06 14:03 ` Stefan Priebe - Profihost AG 0 siblings, 1 reply; 50+ messages in thread From: Stefan Priebe - Profihost AG @ 2014-02-06 12:10 UTC (permalink / raw) To: Alexandre DERUMIER; +Cc: qemu-devel, pve-devel May be, sadly i've no idea. Only using 3.10 Kernel with XFS. Stefan Am 06.02.2014 12:40, schrieb Alexandre DERUMIER: >>> PS: all my guests do not even have !!SWAP!! > > Not sure is related to swap file. > > I found an similar problem here, triggered with suspend/resume on ext4 > > http://lkml.indiana.edu/hypermail/linux/kernel/1106.3/01340.html > > > Maybe is it a guest kernel bug ? > > ----- Mail original ----- > > De: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag> > À: "Alexandre DERUMIER" <aderumier@odiso.com> > Cc: pve-devel@pve.proxmox.com, "qemu-devel" <qemu-devel@nongnu.org> > Envoyé: Jeudi 6 Février 2014 12:19:36 > Objet: Re: [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry > > > Am 06.02.2014 12:14, schrieb Alexandre DERUMIER: >> Do you force rbd_cache=true in ceph.conf? > > no > >> if yes, do you use cache=writeback ? > > yes > > So this should be safe. > > PS: all my guests do not even have !!SWAP!! > > # free|grep Swap > Swap: 0 0 0 > > Stefan > >> according to ceph doc: >> http://ceph.com/docs/next/rbd/qemu-rbd/ >> >> "Important If you set rbd_cache=true, you must set cache=writeback or risk data loss. Without cache=writeback, QEMU will not send flush requests to librbd. If QEMU exits uncleanly in this configuration, filesystems on top of rbd can be corrupted." >> >> >> >> ----- Mail original ----- >> >> De: "Stefan Priebe" <s.priebe@profihost.ag> >> À: pve-devel@pve.proxmox.com, "qemu-devel" <qemu-devel@nongnu.org> >> Envoyé: Mercredi 5 Février 2014 18:51:15 >> Objet: [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry >> >> Hello, >> >> after live migrating machines with a lot of memory (32GB, 48GB, ...) i >> see pretty often crashing services after migration and the guest kernel >> prints: >> >> [1707620.031806] swap_free: Bad swap file entry 00377410 >> [1707620.031806] swap_free: Bad swap file entry 00593c48 >> [1707620.031807] swap_free: Bad swap file entry 03201430 >> [1707620.031807] swap_free: Bad swap file entry 01bc5900 >> [1707620.031807] swap_free: Bad swap file entry 0173ce40 >> [1707620.031808] swap_free: Bad swap file entry 011c0270 >> [1707620.031808] swap_free: Bad swap file entry 03c58ae8 >> [1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:1 >> val:1536 >> [1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:2 >> val:-1536 >> >> Qemu is 1.7 >> >> Does anybody know a fix? >> >> Greets, >> Stefan >> _______________________________________________ >> pve-devel mailing list >> pve-devel@pve.proxmox.com >> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel >> ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-06 12:10 ` Stefan Priebe - Profihost AG @ 2014-02-06 14:03 ` Stefan Priebe - Profihost AG 2014-02-06 14:12 ` Marcin Gibuła 2014-02-06 19:51 ` Dr. David Alan Gilbert 0 siblings, 2 replies; 50+ messages in thread From: Stefan Priebe - Profihost AG @ 2014-02-06 14:03 UTC (permalink / raw) To: Alexandre DERUMIER; +Cc: qemu-devel, pve-devel some more things which happen during migration: php5.2[20258]: segfault at a0 ip 0000000000740656 sp 00007fff53b694a0 error 4 in php-cgi[400000+6d7000] php5.2[20249]: segfault at c ip 00007f1fb8ecb2b8 sp 00007fff642d9c20 error 4 in ZendOptimizer.so[7f1fb8e71000+147000] cron[3154]: segfault at 7f0008a70ed4 ip 00007fc890b9d440 sp 00007fff08a6f9b0 error 4 in libc-2.13.so[7fc890b67000+182000] Stefan Am 06.02.2014 13:10, schrieb Stefan Priebe - Profihost AG: > May be, > > sadly i've no idea. Only using 3.10 Kernel with XFS. > > Stefan > > Am 06.02.2014 12:40, schrieb Alexandre DERUMIER: >>>> PS: all my guests do not even have !!SWAP!! >> >> Not sure is related to swap file. >> >> I found an similar problem here, triggered with suspend/resume on ext4 >> >> http://lkml.indiana.edu/hypermail/linux/kernel/1106.3/01340.html >> >> >> Maybe is it a guest kernel bug ? >> >> ----- Mail original ----- >> >> De: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag> >> À: "Alexandre DERUMIER" <aderumier@odiso.com> >> Cc: pve-devel@pve.proxmox.com, "qemu-devel" <qemu-devel@nongnu.org> >> Envoyé: Jeudi 6 Février 2014 12:19:36 >> Objet: Re: [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry >> >> >> Am 06.02.2014 12:14, schrieb Alexandre DERUMIER: >>> Do you force rbd_cache=true in ceph.conf? >> >> no >> >>> if yes, do you use cache=writeback ? >> >> yes >> >> So this should be safe. >> >> PS: all my guests do not even have !!SWAP!! >> >> # free|grep Swap >> Swap: 0 0 0 >> >> Stefan >> >>> according to ceph doc: >>> http://ceph.com/docs/next/rbd/qemu-rbd/ >>> >>> "Important If you set rbd_cache=true, you must set cache=writeback or risk data loss. Without cache=writeback, QEMU will not send flush requests to librbd. If QEMU exits uncleanly in this configuration, filesystems on top of rbd can be corrupted." >>> >>> >>> >>> ----- Mail original ----- >>> >>> De: "Stefan Priebe" <s.priebe@profihost.ag> >>> À: pve-devel@pve.proxmox.com, "qemu-devel" <qemu-devel@nongnu.org> >>> Envoyé: Mercredi 5 Février 2014 18:51:15 >>> Objet: [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry >>> >>> Hello, >>> >>> after live migrating machines with a lot of memory (32GB, 48GB, ...) i >>> see pretty often crashing services after migration and the guest kernel >>> prints: >>> >>> [1707620.031806] swap_free: Bad swap file entry 00377410 >>> [1707620.031806] swap_free: Bad swap file entry 00593c48 >>> [1707620.031807] swap_free: Bad swap file entry 03201430 >>> [1707620.031807] swap_free: Bad swap file entry 01bc5900 >>> [1707620.031807] swap_free: Bad swap file entry 0173ce40 >>> [1707620.031808] swap_free: Bad swap file entry 011c0270 >>> [1707620.031808] swap_free: Bad swap file entry 03c58ae8 >>> [1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:1 >>> val:1536 >>> [1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:2 >>> val:-1536 >>> >>> Qemu is 1.7 >>> >>> Does anybody know a fix? >>> >>> Greets, >>> Stefan >>> _______________________________________________ >>> pve-devel mailing list >>> pve-devel@pve.proxmox.com >>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel >>> ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-06 14:03 ` Stefan Priebe - Profihost AG @ 2014-02-06 14:12 ` Marcin Gibuła 2014-02-06 19:51 ` Dr. David Alan Gilbert 1 sibling, 0 replies; 50+ messages in thread From: Marcin Gibuła @ 2014-02-06 14:12 UTC (permalink / raw) To: qemu-devel On 06.02.2014 15:03, Stefan Priebe - Profihost AG wrote: > some more things which happen during migration: > > php5.2[20258]: segfault at a0 ip 0000000000740656 sp 00007fff53b694a0 > error 4 in php-cgi[400000+6d7000] > > php5.2[20249]: segfault at c ip 00007f1fb8ecb2b8 sp 00007fff642d9c20 > error 4 in ZendOptimizer.so[7f1fb8e71000+147000] > > cron[3154]: segfault at 7f0008a70ed4 ip 00007fc890b9d440 sp > 00007fff08a6f9b0 error 4 in libc-2.13.so[7fc890b67000+182000] Hi, I've seen memory corruptions after live (and offline) migrations as well. But in our enviroment its mostly (but not only) seen as timer corruption - guest hangs or have insane date in future. But I've seen segfaults and oopses as well. Sadly it's very hard for me to reproduce it reliably but it occures on all types of linux guests - all versions of ubuntu, centos, debian, etc, so it doesn't seem to be connected to a specific guest kernel version. I've never seen windows crashing though. There was another guy here on qemu-devel who had similar issue and fixed it by running guest with no-kvmclock. I've tested qemu 1.4 - 1.6 and kernels 3.4 - 3.10. -- mg ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-06 14:03 ` Stefan Priebe - Profihost AG 2014-02-06 14:12 ` Marcin Gibuła @ 2014-02-06 19:51 ` Dr. David Alan Gilbert 2014-02-06 20:00 ` Stefan Priebe 1 sibling, 1 reply; 50+ messages in thread From: Dr. David Alan Gilbert @ 2014-02-06 19:51 UTC (permalink / raw) To: Stefan Priebe - Profihost AG; +Cc: qemu-devel, Alexandre DERUMIER * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: > some more things which happen during migration: > > php5.2[20258]: segfault at a0 ip 0000000000740656 sp 00007fff53b694a0 > error 4 in php-cgi[400000+6d7000] > > php5.2[20249]: segfault at c ip 00007f1fb8ecb2b8 sp 00007fff642d9c20 > error 4 in ZendOptimizer.so[7f1fb8e71000+147000] > > cron[3154]: segfault at 7f0008a70ed4 ip 00007fc890b9d440 sp > 00007fff08a6f9b0 error 4 in libc-2.13.so[7fc890b67000+182000] OK, so lets just assume some part of memory (or CPU state, or memory loaded off disk...) You said before that it was happening on a 32GB image - is it *only* happening on a 32GB or bigger VM, or is it just more likely? I think you also said you were using 1.7; have you tried an older version - i.e. is this a regression in 1.7 or don't we know? Dave -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-06 19:51 ` Dr. David Alan Gilbert @ 2014-02-06 20:00 ` Stefan Priebe 2014-02-07 8:15 ` Alexandre DERUMIER 0 siblings, 1 reply; 50+ messages in thread From: Stefan Priebe @ 2014-02-06 20:00 UTC (permalink / raw) To: Dr. David Alan Gilbert; +Cc: qemu-devel, Alexandre DERUMIER Hi, Am 06.02.2014 20:51, schrieb Dr. David Alan Gilbert: > * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: >> some more things which happen during migration: >> >> php5.2[20258]: segfault at a0 ip 0000000000740656 sp 00007fff53b694a0 >> error 4 in php-cgi[400000+6d7000] >> >> php5.2[20249]: segfault at c ip 00007f1fb8ecb2b8 sp 00007fff642d9c20 >> error 4 in ZendOptimizer.so[7f1fb8e71000+147000] >> >> cron[3154]: segfault at 7f0008a70ed4 ip 00007fc890b9d440 sp >> 00007fff08a6f9b0 error 4 in libc-2.13.so[7fc890b67000+182000] > > OK, so lets just assume some part of memory (or CPU state, or memory > loaded off disk...) > > You said before that it was happening on a 32GB image - is it *only* > happening on a 32GB or bigger VM, or is it just more likely? Not image, memory. I've only seen this with vms having more than 16GB or 32GB memory. But maybe this also indicates that just the migration takes longer. > I think you also said you were using 1.7; have you tried an older > version - i.e. is this a regression in 1.7 or don't we know? Don't know. Sadly i cannot reproduce this with test VMs only with production ones. Stefan > Dave > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-06 20:00 ` Stefan Priebe @ 2014-02-07 8:15 ` Alexandre DERUMIER 2014-02-07 8:17 ` Stefan Priebe - Profihost AG 0 siblings, 1 reply; 50+ messages in thread From: Alexandre DERUMIER @ 2014-02-07 8:15 UTC (permalink / raw) To: Stefan Priebe; +Cc: qemu-devel, Dr. David Alan Gilbert do you use xbzrle for live migration ? ----- Mail original ----- De: "Stefan Priebe" <s.priebe@profihost.ag> À: "Dr. David Alan Gilbert" <dgilbert@redhat.com> Cc: "Alexandre DERUMIER" <aderumier@odiso.com>, "qemu-devel" <qemu-devel@nongnu.org> Envoyé: Jeudi 6 Février 2014 21:00:27 Objet: Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry Hi, Am 06.02.2014 20:51, schrieb Dr. David Alan Gilbert: > * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: >> some more things which happen during migration: >> >> php5.2[20258]: segfault at a0 ip 0000000000740656 sp 00007fff53b694a0 >> error 4 in php-cgi[400000+6d7000] >> >> php5.2[20249]: segfault at c ip 00007f1fb8ecb2b8 sp 00007fff642d9c20 >> error 4 in ZendOptimizer.so[7f1fb8e71000+147000] >> >> cron[3154]: segfault at 7f0008a70ed4 ip 00007fc890b9d440 sp >> 00007fff08a6f9b0 error 4 in libc-2.13.so[7fc890b67000+182000] > > OK, so lets just assume some part of memory (or CPU state, or memory > loaded off disk...) > > You said before that it was happening on a 32GB image - is it *only* > happening on a 32GB or bigger VM, or is it just more likely? Not image, memory. I've only seen this with vms having more than 16GB or 32GB memory. But maybe this also indicates that just the migration takes longer. > I think you also said you were using 1.7; have you tried an older > version - i.e. is this a regression in 1.7 or don't we know? Don't know. Sadly i cannot reproduce this with test VMs only with production ones. Stefan > Dave > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 8:15 ` Alexandre DERUMIER @ 2014-02-07 8:17 ` Stefan Priebe - Profihost AG 2014-02-07 9:15 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 50+ messages in thread From: Stefan Priebe - Profihost AG @ 2014-02-07 8:17 UTC (permalink / raw) To: Alexandre DERUMIER; +Cc: qemu-devel, Dr. David Alan Gilbert Am 07.02.2014 09:15, schrieb Alexandre DERUMIER: > > do you use xbzrle for live migration ? no - i'm really stucked right now with this. Biggest problem i can't reproduce with test machines ;-( Stefan > > ----- Mail original ----- > > De: "Stefan Priebe" <s.priebe@profihost.ag> > À: "Dr. David Alan Gilbert" <dgilbert@redhat.com> > Cc: "Alexandre DERUMIER" <aderumier@odiso.com>, "qemu-devel" <qemu-devel@nongnu.org> > Envoyé: Jeudi 6 Février 2014 21:00:27 > Objet: Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry > > Hi, > Am 06.02.2014 20:51, schrieb Dr. David Alan Gilbert: >> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: >>> some more things which happen during migration: >>> >>> php5.2[20258]: segfault at a0 ip 0000000000740656 sp 00007fff53b694a0 >>> error 4 in php-cgi[400000+6d7000] >>> >>> php5.2[20249]: segfault at c ip 00007f1fb8ecb2b8 sp 00007fff642d9c20 >>> error 4 in ZendOptimizer.so[7f1fb8e71000+147000] >>> >>> cron[3154]: segfault at 7f0008a70ed4 ip 00007fc890b9d440 sp >>> 00007fff08a6f9b0 error 4 in libc-2.13.so[7fc890b67000+182000] >> >> OK, so lets just assume some part of memory (or CPU state, or memory >> loaded off disk...) >> >> You said before that it was happening on a 32GB image - is it *only* >> happening on a 32GB or bigger VM, or is it just more likely? > > Not image, memory. I've only seen this with vms having more than 16GB or > 32GB memory. But maybe this also indicates that just the migration takes > longer. > >> I think you also said you were using 1.7; have you tried an older >> version - i.e. is this a regression in 1.7 or don't we know? > Don't know. Sadly i cannot reproduce this with test VMs only with > production ones. > > Stefan > >> Dave >> -- >> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >> ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 8:17 ` Stefan Priebe - Profihost AG @ 2014-02-07 9:15 ` Dr. David Alan Gilbert 2014-02-07 9:20 ` Stefan Priebe - Profihost AG 0 siblings, 1 reply; 50+ messages in thread From: Dr. David Alan Gilbert @ 2014-02-07 9:15 UTC (permalink / raw) To: Stefan Priebe - Profihost AG; +Cc: qemu-devel, Alexandre DERUMIER * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: > Am 07.02.2014 09:15, schrieb Alexandre DERUMIER: > > > > do you use xbzrle for live migration ? > > no - i'm really stucked right now with this. Biggest problem i can't > reproduce with test machines ;-( Only being able to test on your production VMs isn't fun; is it possible or you to run an extra program on these VMs - e.g. if we came up with a simple (userland) memory test? Dave > > Stefan > > > > > > ----- Mail original ----- > > > > De: "Stefan Priebe" <s.priebe@profihost.ag> > > À: "Dr. David Alan Gilbert" <dgilbert@redhat.com> > > Cc: "Alexandre DERUMIER" <aderumier@odiso.com>, "qemu-devel" <qemu-devel@nongnu.org> > > Envoyé: Jeudi 6 Février 2014 21:00:27 > > Objet: Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry > > > > Hi, > > Am 06.02.2014 20:51, schrieb Dr. David Alan Gilbert: > >> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: > >>> some more things which happen during migration: > >>> > >>> php5.2[20258]: segfault at a0 ip 0000000000740656 sp 00007fff53b694a0 > >>> error 4 in php-cgi[400000+6d7000] > >>> > >>> php5.2[20249]: segfault at c ip 00007f1fb8ecb2b8 sp 00007fff642d9c20 > >>> error 4 in ZendOptimizer.so[7f1fb8e71000+147000] > >>> > >>> cron[3154]: segfault at 7f0008a70ed4 ip 00007fc890b9d440 sp > >>> 00007fff08a6f9b0 error 4 in libc-2.13.so[7fc890b67000+182000] > >> > >> OK, so lets just assume some part of memory (or CPU state, or memory > >> loaded off disk...) > >> > >> You said before that it was happening on a 32GB image - is it *only* > >> happening on a 32GB or bigger VM, or is it just more likely? > > > > Not image, memory. I've only seen this with vms having more than 16GB or > > 32GB memory. But maybe this also indicates that just the migration takes > > longer. > > > >> I think you also said you were using 1.7; have you tried an older > >> version - i.e. is this a regression in 1.7 or don't we know? > > Don't know. Sadly i cannot reproduce this with test VMs only with > > production ones. > > > > Stefan > > > >> Dave > >> -- > >> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > >> > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 9:15 ` Dr. David Alan Gilbert @ 2014-02-07 9:20 ` Stefan Priebe - Profihost AG 2014-02-07 9:29 ` Marcin Gibuła 2014-02-07 9:31 ` Dr. David Alan Gilbert 0 siblings, 2 replies; 50+ messages in thread From: Stefan Priebe - Profihost AG @ 2014-02-07 9:20 UTC (permalink / raw) To: Dr. David Alan Gilbert; +Cc: qemu-devel, Alexandre DERUMIER Am 07.02.2014 10:15, schrieb Dr. David Alan Gilbert: > * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: >> Am 07.02.2014 09:15, schrieb Alexandre DERUMIER: >>> >>> do you use xbzrle for live migration ? >> >> no - i'm really stucked right now with this. Biggest problem i can't >> reproduce with test machines ;-( > > Only being able to test on your production VMs isn't fun; > is it possible or you to run an extra program on these VMs - e.g. > if we came up with a simple (userland) memory test? You mean to reproduce? I already tried https://code.google.com/p/stressapptest/ while migrating on a test VM but this works fine. I also tried running mysql bench while migrating on a test vm and this works too ;-( Stefan > Dave > >> >> Stefan >> >> >>> >>> ----- Mail original ----- >>> >>> De: "Stefan Priebe" <s.priebe@profihost.ag> >>> À: "Dr. David Alan Gilbert" <dgilbert@redhat.com> >>> Cc: "Alexandre DERUMIER" <aderumier@odiso.com>, "qemu-devel" <qemu-devel@nongnu.org> >>> Envoyé: Jeudi 6 Février 2014 21:00:27 >>> Objet: Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry >>> >>> Hi, >>> Am 06.02.2014 20:51, schrieb Dr. David Alan Gilbert: >>>> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: >>>>> some more things which happen during migration: >>>>> >>>>> php5.2[20258]: segfault at a0 ip 0000000000740656 sp 00007fff53b694a0 >>>>> error 4 in php-cgi[400000+6d7000] >>>>> >>>>> php5.2[20249]: segfault at c ip 00007f1fb8ecb2b8 sp 00007fff642d9c20 >>>>> error 4 in ZendOptimizer.so[7f1fb8e71000+147000] >>>>> >>>>> cron[3154]: segfault at 7f0008a70ed4 ip 00007fc890b9d440 sp >>>>> 00007fff08a6f9b0 error 4 in libc-2.13.so[7fc890b67000+182000] >>>> >>>> OK, so lets just assume some part of memory (or CPU state, or memory >>>> loaded off disk...) >>>> >>>> You said before that it was happening on a 32GB image - is it *only* >>>> happening on a 32GB or bigger VM, or is it just more likely? >>> >>> Not image, memory. I've only seen this with vms having more than 16GB or >>> 32GB memory. But maybe this also indicates that just the migration takes >>> longer. >>> >>>> I think you also said you were using 1.7; have you tried an older >>>> version - i.e. is this a regression in 1.7 or don't we know? >>> Don't know. Sadly i cannot reproduce this with test VMs only with >>> production ones. >>> >>> Stefan >>> >>>> Dave >>>> -- >>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >>>> >> > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 9:20 ` Stefan Priebe - Profihost AG @ 2014-02-07 9:29 ` Marcin Gibuła 2014-02-07 9:30 ` Stefan Priebe - Profihost AG 2014-02-07 9:31 ` Dr. David Alan Gilbert 1 sibling, 1 reply; 50+ messages in thread From: Marcin Gibuła @ 2014-02-07 9:29 UTC (permalink / raw) To: Stefan Priebe - Profihost AG, Dr. David Alan Gilbert Cc: qemu-devel, Alexandre DERUMIER >>>> do you use xbzrle for live migration ? >>> >>> no - i'm really stucked right now with this. Biggest problem i can't >>> reproduce with test machines ;-( >> >> Only being able to test on your production VMs isn't fun; >> is it possible or you to run an extra program on these VMs - e.g. >> if we came up with a simple (userland) memory test? > > You mean to reproduce? > > I already tried https://code.google.com/p/stressapptest/ while migrating > on a test VM but this works fine. > > I also tried running mysql bench while migrating on a test vm and this > works too ;-( Have you tried to let test VM run idle for some time before migrating? (like 18-24 hours) Having the same (or very similar) problem, I had bigger luck with reproducing it by not using freshly started VMs. -- mg ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 9:29 ` Marcin Gibuła @ 2014-02-07 9:30 ` Stefan Priebe - Profihost AG 0 siblings, 0 replies; 50+ messages in thread From: Stefan Priebe - Profihost AG @ 2014-02-07 9:30 UTC (permalink / raw) To: Marcin Gibuła, Dr. David Alan Gilbert; +Cc: qemu-devel, Alexandre DERUMIER Hi, Am 07.02.2014 10:29, schrieb Marcin Gibuła: >>>>> do you use xbzrle for live migration ? >>>> >>>> no - i'm really stucked right now with this. Biggest problem i can't >>>> reproduce with test machines ;-( >>> >>> Only being able to test on your production VMs isn't fun; >>> is it possible or you to run an extra program on these VMs - e.g. >>> if we came up with a simple (userland) memory test? >> >> You mean to reproduce? >> >> I already tried https://code.google.com/p/stressapptest/ while migrating >> on a test VM but this works fine. >> >> I also tried running mysql bench while migrating on a test vm and this >> works too ;-( > > Have you tried to let test VM run idle for some time before migrating? > (like 18-24 hours) > > Having the same (or very similar) problem, I had bigger luck with > reproducing it by not using freshly started VMs. no i haven't tried this will do so soon. Stefan ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 9:20 ` Stefan Priebe - Profihost AG 2014-02-07 9:29 ` Marcin Gibuła @ 2014-02-07 9:31 ` Dr. David Alan Gilbert 2014-02-07 9:37 ` Stefan Priebe - Profihost AG 2014-02-07 9:59 ` Marcin Gibuła 1 sibling, 2 replies; 50+ messages in thread From: Dr. David Alan Gilbert @ 2014-02-07 9:31 UTC (permalink / raw) To: Stefan Priebe - Profihost AG; +Cc: qemu-devel, Alexandre DERUMIER * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: > > Am 07.02.2014 10:15, schrieb Dr. David Alan Gilbert: > > * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: > >> Am 07.02.2014 09:15, schrieb Alexandre DERUMIER: > >>> > >>> do you use xbzrle for live migration ? > >> > >> no - i'm really stucked right now with this. Biggest problem i can't > >> reproduce with test machines ;-( > > > > Only being able to test on your production VMs isn't fun; > > is it possible or you to run an extra program on these VMs - e.g. > > if we came up with a simple (userland) memory test? > > You mean to reproduce? I'm more interested in seeing what type of corruption is happening; if you've got a test VM that corrupts memory and we can run a program in that vm that writes a known pattern into memory and checks it then see what changed after migration, it might give a clue. But obviously this would only be of any use if run on the VM that actually fails. > I already tried https://code.google.com/p/stressapptest/ while migrating > on a test VM but this works fine. > > I also tried running mysql bench while migrating on a test vm and this > works too ;-( Dave -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 9:31 ` Dr. David Alan Gilbert @ 2014-02-07 9:37 ` Stefan Priebe - Profihost AG 2014-02-07 12:02 ` Stefan Priebe - Profihost AG 2014-02-07 9:59 ` Marcin Gibuła 1 sibling, 1 reply; 50+ messages in thread From: Stefan Priebe - Profihost AG @ 2014-02-07 9:37 UTC (permalink / raw) To: Dr. David Alan Gilbert; +Cc: qemu-devel, Alexandre DERUMIER Am 07.02.2014 10:31, schrieb Dr. David Alan Gilbert: > * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: >> >> Am 07.02.2014 10:15, schrieb Dr. David Alan Gilbert: >>> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: >>>> Am 07.02.2014 09:15, schrieb Alexandre DERUMIER: >>>>> >>>>> do you use xbzrle for live migration ? >>>> >>>> no - i'm really stucked right now with this. Biggest problem i can't >>>> reproduce with test machines ;-( >>> >>> Only being able to test on your production VMs isn't fun; >>> is it possible or you to run an extra program on these VMs - e.g. >>> if we came up with a simple (userland) memory test? >> >> You mean to reproduce? > > I'm more interested in seeing what type of corruption is happening; > if you've got a test VM that corrupts memory and we can run a program > in that vm that writes a known pattern into memory and checks it > then see what changed after migration, it might give a clue. > > But obviously this would only be of any use if run on the VM that actually > fails. Right that makes sense - sadly i still don't know how to reproduce? Any app ideas i can try? >> I already tried https://code.google.com/p/stressapptest/ while migrating >> on a test VM but this works fine. >> >> I also tried running mysql bench while migrating on a test vm and this >> works too ;-( > > > Dave > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 9:37 ` Stefan Priebe - Profihost AG @ 2014-02-07 12:02 ` Stefan Priebe - Profihost AG 2014-02-07 12:21 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 50+ messages in thread From: Stefan Priebe - Profihost AG @ 2014-02-07 12:02 UTC (permalink / raw) To: Dr. David Alan Gilbert; +Cc: m.gibula, qemu-devel, Alexandre DERUMIER Hi, i was able to reproduce with a longer running test VM running the google stress test. And it happens exacly when the migration finishes it does not happen while the migration is running. Google Stress Output displays Memory errors: ------------------------------------------------ Page Error: miscompare on CPU 5(0xffff) at 0x7f52431341c0(0x0:DIMM Unknown): read:0x00000040000000bf, reread:0x00000040000000bf expected:0x00000040ffffffbf Report Error: miscompare : DIMM Unknown : 1 : 571s Page Error: miscompare on CPU 5(0xffff) at 0x7f52431341c8(0x0:DIMM Unknown): read:0x00000020000000df, reread:0x00000020000000df expected:0x00000020ffffffdf Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34020(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34028(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34060(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34068(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340a0(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340a8(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340e0(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340e8(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34120(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34128(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34160(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34168(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341a0(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341a8(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341e0(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341e8(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34220(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34228(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34260(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34268(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342a0(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342a8(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342e0(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342e8(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34320(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34328(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34360(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34368(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c343a0(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 571s Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c343a8(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235030(0x0:DIMM Unknown): read:0xfffffffd000000fd, reread:0xfffffffd000000fd expected:0xfffffffdfffffffd Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235070(0x0:DIMM Unknown): read:0xfffffff7000000f7, reread:0xfffffff7000000f7 expected:0xfffffff7fffffff7 Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2350b0(0x0:DIMM Unknown): read:0xffffffdf000000df, reread:0xffffffdf000000df expected:0xffffffdfffffffdf Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2350f0(0x0:DIMM Unknown): read:0xffffff7f0000007f, reread:0xffffff7f0000007f expected:0xffffff7fffffff7f Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235130(0x0:DIMM Unknown): read:0xfffffdff0000fd00, reread:0xfffffdff0000fd00 expected:0xfffffdfffffffdff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235170(0x0:DIMM Unknown): read:0xfffff7ff0000f700, reread:0xfffff7ff0000f700 expected:0xfffff7fffffff7ff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2351b0(0x0:DIMM Unknown): read:0xffffdfff0000df00, reread:0xffffdfff0000df00 expected:0xffffdfffffffdfff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2351f0(0x0:DIMM Unknown): read:0xffff7fff00007f00, reread:0xffff7fff00007f00 expected:0xffff7fffffff7fff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235230(0x0:DIMM Unknown): read:0xfffdffff00fd0000, reread:0xfffdffff00fd0000 expected:0xfffdfffffffdffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235270(0x0:DIMM Unknown): read:0xfff7ffff00f70000, reread:0xfff7ffff00f70000 expected:0xfff7fffffff7ffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2352b0(0x0:DIMM Unknown): read:0xffdfffff00df0000, reread:0xffdfffff00df0000 expected:0xffdfffffffdfffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2352f0(0x0:DIMM Unknown): read:0xff7fffff007f0000, reread:0xff7fffff007f0000 expected:0xff7fffffff7fffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235330(0x0:DIMM Unknown): read:0xfdfffffffd000000, reread:0xfdfffffffd000000 expected:0xfdfffffffdffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235370(0x0:DIMM Unknown): read:0xf7fffffff7000000, reread:0xf7fffffff7000000 expected:0xf7fffffff7ffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2353b0(0x0:DIMM Unknown): read:0xdfffffffdf000000, reread:0xdfffffffdf000000 expected:0xdfffffffdfffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2353f0(0x0:DIMM Unknown): read:0x7fffffff7f000000, reread:0x7fffffff7f000000 expected:0x7fffffff7fffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235430(0x0:DIMM Unknown): read:0xdfffffffdf000000, reread:0xdfffffffdf000000 expected:0xdfffffffdfffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235470(0x0:DIMM Unknown): read:0xf7fffffff7000000, reread:0xf7fffffff7000000 expected:0xf7fffffff7ffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2354b0(0x0:DIMM Unknown): read:0xfdfffffffd000000, reread:0xfdfffffffd000000 expected:0xfdfffffffdffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2354f0(0x0:DIMM Unknown): read:0xff7fffff007f0000, reread:0xff7fffff007f0000 expected:0xff7fffffff7fffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235530(0x0:DIMM Unknown): read:0xffdfffff00df0000, reread:0xffdfffff00df0000 expected:0xffdfffffffdfffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235570(0x0:DIMM Unknown): read:0xfff7ffff00f70000, reread:0xfff7ffff00f70000 expected:0xfff7fffffff7ffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2355b0(0x0:DIMM Unknown): read:0xfffdffff00fd0000, reread:0xfffdffff00fd0000 expected:0xfffdfffffffdffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2355f0(0x0:DIMM Unknown): read:0xffff7fff00007f00, reread:0xffff7fff00007f00 expected:0xffff7fffffff7fff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235630(0x0:DIMM Unknown): read:0xffffdfff0000df00, reread:0xffffdfff0000df00 expected:0xffffdfffffffdfff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235670(0x0:DIMM Unknown): read:0xfffff7ff0000f700, reread:0xfffff7ff0000f700 expected:0xfffff7fffffff7ff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2356b0(0x0:DIMM Unknown): read:0xfffffdff0000fd00, reread:0xfffffdff0000fd00 expected:0xfffffdfffffffdff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2356f0(0x0:DIMM Unknown): read:0xffffff7f0000007f, reread:0xffffff7f0000007f expected:0xffffff7fffffff7f Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235730(0x0:DIMM Unknown): read:0xffffffdf000000df, reread:0xffffffdf000000df expected:0xffffffdfffffffdf Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235770(0x0:DIMM Unknown): read:0xfffffff7000000f7, reread:0xfffffff7000000f7 expected:0xfffffff7fffffff7 Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834020(0x0:DIMM Unknown): read:0x00000010000000ef, reread:0x00000010000000ef expected:0x00000010ffffffef Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834028(0x0:DIMM Unknown): read:0x00000020000000df, reread:0x00000020000000df expected:0x00000020ffffffdf Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834030(0x0:DIMM Unknown): read:0x00000040000000bf, reread:0x00000040000000bf expected:0x00000040ffffffbf Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834038(0x0:DIMM Unknown): read:0x000000800000007f, reread:0x000000800000007f expected:0x00000080ffffff7f Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834060(0x0:DIMM Unknown): read:0x000010000000efff, reread:0x000010000000efff expected:0x00001000ffffefff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834068(0x0:DIMM Unknown): read:0x000020000000dfff, reread:0x000020000000dfff expected:0x00002000ffffdfff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834070(0x0:DIMM Unknown): read:0x000040000000bfff, reread:0x000040000000bfff expected:0x00004000ffffbfff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834078(0x0:DIMM Unknown): read:0x0000800000007fff, reread:0x0000800000007fff expected:0x00008000ffff7fff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340a0(0x0:DIMM Unknown): read:0x0010000000ef00ff, reread:0x0010000000ef00ff expected:0x00100000ffefffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340a8(0x0:DIMM Unknown): read:0x0020000000df00ff, reread:0x0020000000df00ff expected:0x00200000ffdfffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340b0(0x0:DIMM Unknown): read:0x0040000000bf00ff, reread:0x0040000000bf00ff expected:0x00400000ffbfffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340b8(0x0:DIMM Unknown): read:0x00800000007f00ff, reread:0x00800000007f00ff expected:0x00800000ff7fffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340e0(0x0:DIMM Unknown): read:0x10000000ef0000ff, reread:0x10000000ef0000ff expected:0x10000000efffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340e8(0x0:DIMM Unknown): read:0x20000000df0000ff, reread:0x20000000df0000ff expected:0x20000000dfffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340f0(0x0:DIMM Unknown): read:0x40000000bf0000ff, reread:0x40000000bf0000ff expected:0x40000000bfffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340f8(0x0:DIMM Unknown): read:0x800000007f0000ff, reread:0x800000007f0000ff expected:0x800000007fffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834120(0x0:DIMM Unknown): read:0x04000000fb00ff00, reread:0x04000000fb00ff00 expected:0x04000000fbffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834128(0x0:DIMM Unknown): read:0x02000000fd00ff00, reread:0x02000000fd00ff00 expected:0x02000000fdffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834130(0x0:DIMM Unknown): read:0x01000000fe00ff00, reread:0x01000000fe00ff00 expected:0x01000000feffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834138(0x0:DIMM Unknown): read:0x00800000007fff00, reread:0x00800000007fff00 expected:0x00800000ff7fffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834160(0x0:DIMM Unknown): read:0x0004000000fbff00, reread:0x0004000000fbff00 expected:0x00040000fffbffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834168(0x0:DIMM Unknown): read:0x0002000000fdff00, reread:0x0002000000fdff00 expected:0x00020000fffdffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834170(0x0:DIMM Unknown): read:0x0001000000feff00, reread:0x0001000000feff00 expected:0x00010000fffeffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834178(0x0:DIMM Unknown): read:0x0000800000007f00, reread:0x0000800000007f00 expected:0x00008000ffff7fff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341a0(0x0:DIMM Unknown): read:0x000004000000fb00, reread:0x000004000000fb00 expected:0x00000400fffffbff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341a8(0x0:DIMM Unknown): read:0x000002000000fd00, reread:0x000002000000fd00 expected:0x00000200fffffdff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341b0(0x0:DIMM Unknown): read:0x000001000000fe00, reread:0x000001000000fe00 expected:0x00000100fffffeff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341b8(0x0:DIMM Unknown): read:0x000000800000ff7f, reread:0x000000800000ff7f expected:0x00000080ffffff7f Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341e0(0x0:DIMM Unknown): read:0x000000040000fffb, reread:0x000000040000fffb expected:0x00000004fffffffb Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341e8(0x0:DIMM Unknown): read:0x000000020000fffd, reread:0x000000020000fffd expected:0x00000002fffffffd Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00008(0x0:DIMM Unknown): read:0xfffffff7000000fb, reread:0xfffffff7000000fb expected:0xfffffff7fffffffb Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00018(0x0:DIMM Unknown): read:0xffffff7f000000bf, reread:0xffffff7f000000bf expected:0xffffff7fffffffbf Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00028(0x0:DIMM Unknown): read:0xfffff7ff0000fb00, reread:0xfffff7ff0000fb00 expected:0xfffff7fffffffbff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00038(0x0:DIMM Unknown): read:0xffff7fff0000bf00, reread:0xffff7fff0000bf00 expected:0xffff7fffffffbfff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00048(0x0:DIMM Unknown): read:0xfff7ffff00fb0000, reread:0xfff7ffff00fb0000 expected:0xfff7fffffffbffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00058(0x0:DIMM Unknown): read:0xff7fffff00bf0000, reread:0xff7fffff00bf0000 expected:0xff7fffffffbfffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00068(0x0:DIMM Unknown): read:0xf7fffffffb000000, reread:0xf7fffffffb000000 expected:0xf7fffffffbffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00078(0x0:DIMM Unknown): read:0x7fffffffbf000000, reread:0x7fffffffbf000000 expected:0x7fffffffbfffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00088(0x0:DIMM Unknown): read:0xf7ffffffef000000, reread:0xf7ffffffef000000 expected:0xf7ffffffefffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00098(0x0:DIMM Unknown): read:0xff7ffffffe000000, reread:0xff7ffffffe000000 expected:0xff7ffffffeffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000a8(0x0:DIMM Unknown): read:0xfff7ffff00ef0000, reread:0xfff7ffff00ef0000 expected:0xfff7ffffffefffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000b8(0x0:DIMM Unknown): read:0xffff7fff00fe0000, reread:0xffff7fff00fe0000 expected:0xffff7ffffffeffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000c8(0x0:DIMM Unknown): read:0xfffff7ff0000ef00, reread:0xfffff7ff0000ef00 expected:0xfffff7ffffffefff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000d8(0x0:DIMM Unknown): read:0xffffff7f0000fe00, reread:0xffffff7f0000fe00 expected:0xffffff7ffffffeff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000e8(0x0:DIMM Unknown): read:0xfffffff7000000ef, reread:0xfffffff7000000ef expected:0xfffffff7ffffffef Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000f8(0x0:DIMM Unknown): read:0xffffffff000000fe, reread:0xffffffff000000fe expected:0xfffffffffffffffe Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00108(0x0:DIMM Unknown): read:0xfffffff7000000fb, reread:0xfffffff7000000fb expected:0xfffffff7fffffffb Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00118(0x0:DIMM Unknown): read:0xffffff7f000000bf, reread:0xffffff7f000000bf expected:0xffffff7fffffffbf Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00128(0x0:DIMM Unknown): read:0xfffff7ff0000fb00, reread:0xfffff7ff0000fb00 expected:0xfffff7fffffffbff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00138(0x0:DIMM Unknown): read:0xffff7fff0000bf00, reread:0xffff7fff0000bf00 expected:0xffff7fffffffbfff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00148(0x0:DIMM Unknown): read:0xfff7ffff00fb0000, reread:0xfff7ffff00fb0000 expected:0xfff7fffffffbffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00158(0x0:DIMM Unknown): read:0xff7fffff00bf0000, reread:0xff7fffff00bf0000 expected:0xff7fffffffbfffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00168(0x0:DIMM Unknown): read:0xf7fffffffb000000, reread:0xf7fffffffb000000 expected:0xf7fffffffbffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00178(0x0:DIMM Unknown): read:0x7fffffffbf000000, reread:0x7fffffffbf000000 expected:0x7fffffffbfffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00188(0x0:DIMM Unknown): read:0xf7ffffffef000000, reread:0xf7ffffffef000000 expected:0xf7ffffffefffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00198(0x0:DIMM Unknown): read:0xff7ffffffe000000, reread:0xff7ffffffe000000 expected:0xff7ffffffeffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001a8(0x0:DIMM Unknown): read:0xfff7ffff00ef0000, reread:0xfff7ffff00ef0000 expected:0xfff7ffffffefffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001b8(0x0:DIMM Unknown): read:0xffff7fff00fe0000, reread:0xffff7fff00fe0000 expected:0xffff7ffffffeffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001c8(0x0:DIMM Unknown): read:0xfffff7ff0000ef00, reread:0xfffff7ff0000ef00 expected:0xfffff7ffffffefff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001d8(0x0:DIMM Unknown): read:0xffffff7f0000fe00, reread:0xffffff7f0000fe00 expected:0xffffff7ffffffeff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200010(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200018(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200050(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200058(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200090(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200098(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72000d0(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72000d8(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200110(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200118(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200150(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200158(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200190(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200198(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72001d0(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72001d8(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200210(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200218(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200250(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200258(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200290(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200298(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72002d0(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72002d8(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200310(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200318(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200350(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200358(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200390(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200398(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00000(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00008(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00010(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00018(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00040(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00048(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00050(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00058(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00080(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00088(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00090(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00098(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000c0(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000c8(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000d0(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000d8(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00100(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00108(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00110(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00118(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00140(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00148(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00150(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00158(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00180(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00188(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00190(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00198(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa001c0(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Report Error: miscompare : DIMM Unknown : 1 : 572s Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa001c8(0x0:DIMM Unknown): read:0x0000000000000000, reread:0x0000000000000000 expected:0xffffffffffffffff Log: Thread 61 found 85568 hardware incidents Log: Thread 62 found 169344 hardware incidents Log: Thread 63 found 44544 hardware incidents Log: Thread 64 found 149504 hardware incidents Log: Thread 65 found 131968 hardware incidents Log: Thread 66 found 150528 hardware incidents Log: Thread 67 found 144384 hardware incidents Log: Thread 68 found 149888 hardware incidents Stats: Found 1025728 hardware incidents Stats: Completed: 9176812.00M in 524.63s 17491.92MB/s, with 1025728 hardware incidents, 0 errors Stats: Memory Copy: 4890244.00M at 9402.74MB/s Stats: File Copy: 0.00M at 0.00MB/s Stats: Net Copy: 0.00M at 0.00MB/s Stats: Data Check: 0.00M at 0.00MB/s Stats: Invert Data: 4286568.00M at 8242.44MB/s Stats: Disk: 0.00M at 0.00MB/s Status: FAIL - test discovered HW problems ----------------------------------------------------------------------------------- Stefan Am 07.02.2014 10:37, schrieb Stefan Priebe - Profihost AG: > > Am 07.02.2014 10:31, schrieb Dr. David Alan Gilbert: >> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: >>> >>> Am 07.02.2014 10:15, schrieb Dr. David Alan Gilbert: >>>> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: >>>>> Am 07.02.2014 09:15, schrieb Alexandre DERUMIER: >>>>>> >>>>>> do you use xbzrle for live migration ? >>>>> >>>>> no - i'm really stucked right now with this. Biggest problem i can't >>>>> reproduce with test machines ;-( >>>> >>>> Only being able to test on your production VMs isn't fun; >>>> is it possible or you to run an extra program on these VMs - e.g. >>>> if we came up with a simple (userland) memory test? >>> >>> You mean to reproduce? >> >> I'm more interested in seeing what type of corruption is happening; >> if you've got a test VM that corrupts memory and we can run a program >> in that vm that writes a known pattern into memory and checks it >> then see what changed after migration, it might give a clue. >> >> But obviously this would only be of any use if run on the VM that actually >> fails. > > Right that makes sense - sadly i still don't know how to reproduce? Any > app ideas i can try? > > >>> I already tried https://code.google.com/p/stressapptest/ while migrating >>> on a test VM but this works fine. >>> >>> I also tried running mysql bench while migrating on a test vm and this >>> works too ;-( >> >> >> Dave >> -- >> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >> ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 12:02 ` Stefan Priebe - Profihost AG @ 2014-02-07 12:21 ` Dr. David Alan Gilbert 2014-02-07 12:30 ` Stefan Priebe - Profihost AG 0 siblings, 1 reply; 50+ messages in thread From: Dr. David Alan Gilbert @ 2014-02-07 12:21 UTC (permalink / raw) To: Stefan Priebe - Profihost AG; +Cc: m.gibula, qemu-devel, Alexandre DERUMIER * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: > Hi, > > i was able to reproduce with a longer running test VM running the google > stress test. Hmm that's quite a fun set of differences; I think I'd like to understand whether the pattern is related to the pattern of what the test is doing. Can you just give an explanation of exactly how you ran that test? What you installed, how exactly you ran it. Then Marcin and I can try and replicate it. Dave > And it happens exacly when the migration finishes it does not happen > while the migration is running. > > Google Stress Output displays Memory errors: > ------------------------------------------------ > Page Error: miscompare on CPU 5(0xffff) at 0x7f52431341c0(0x0:DIMM > Unknown): read:0x00000040000000bf, reread:0x00000040000000bf > expected:0x00000040ffffffbf > Report Error: miscompare : DIMM Unknown : 1 : 571s > Page Error: miscompare on CPU 5(0xffff) at 0x7f52431341c8(0x0:DIMM > Unknown): read:0x00000020000000df, reread:0x00000020000000df > expected:0x00000020ffffffdf > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34020(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34028(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34060(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34068(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340a0(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340a8(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340e0(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340e8(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34120(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34128(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34160(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34168(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341a0(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341a8(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341e0(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341e8(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34220(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34228(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34260(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34268(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342a0(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342a8(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342e0(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342e8(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34320(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34328(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34360(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34368(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c343a0(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 571s > Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c343a8(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235030(0x0:DIMM > Unknown): read:0xfffffffd000000fd, reread:0xfffffffd000000fd > expected:0xfffffffdfffffffd > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235070(0x0:DIMM > Unknown): read:0xfffffff7000000f7, reread:0xfffffff7000000f7 > expected:0xfffffff7fffffff7 > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2350b0(0x0:DIMM > Unknown): read:0xffffffdf000000df, reread:0xffffffdf000000df > expected:0xffffffdfffffffdf > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2350f0(0x0:DIMM > Unknown): read:0xffffff7f0000007f, reread:0xffffff7f0000007f > expected:0xffffff7fffffff7f > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235130(0x0:DIMM > Unknown): read:0xfffffdff0000fd00, reread:0xfffffdff0000fd00 > expected:0xfffffdfffffffdff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235170(0x0:DIMM > Unknown): read:0xfffff7ff0000f700, reread:0xfffff7ff0000f700 > expected:0xfffff7fffffff7ff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2351b0(0x0:DIMM > Unknown): read:0xffffdfff0000df00, reread:0xffffdfff0000df00 > expected:0xffffdfffffffdfff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2351f0(0x0:DIMM > Unknown): read:0xffff7fff00007f00, reread:0xffff7fff00007f00 > expected:0xffff7fffffff7fff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235230(0x0:DIMM > Unknown): read:0xfffdffff00fd0000, reread:0xfffdffff00fd0000 > expected:0xfffdfffffffdffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235270(0x0:DIMM > Unknown): read:0xfff7ffff00f70000, reread:0xfff7ffff00f70000 > expected:0xfff7fffffff7ffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2352b0(0x0:DIMM > Unknown): read:0xffdfffff00df0000, reread:0xffdfffff00df0000 > expected:0xffdfffffffdfffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2352f0(0x0:DIMM > Unknown): read:0xff7fffff007f0000, reread:0xff7fffff007f0000 > expected:0xff7fffffff7fffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235330(0x0:DIMM > Unknown): read:0xfdfffffffd000000, reread:0xfdfffffffd000000 > expected:0xfdfffffffdffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235370(0x0:DIMM > Unknown): read:0xf7fffffff7000000, reread:0xf7fffffff7000000 > expected:0xf7fffffff7ffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2353b0(0x0:DIMM > Unknown): read:0xdfffffffdf000000, reread:0xdfffffffdf000000 > expected:0xdfffffffdfffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2353f0(0x0:DIMM > Unknown): read:0x7fffffff7f000000, reread:0x7fffffff7f000000 > expected:0x7fffffff7fffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235430(0x0:DIMM > Unknown): read:0xdfffffffdf000000, reread:0xdfffffffdf000000 > expected:0xdfffffffdfffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235470(0x0:DIMM > Unknown): read:0xf7fffffff7000000, reread:0xf7fffffff7000000 > expected:0xf7fffffff7ffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2354b0(0x0:DIMM > Unknown): read:0xfdfffffffd000000, reread:0xfdfffffffd000000 > expected:0xfdfffffffdffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2354f0(0x0:DIMM > Unknown): read:0xff7fffff007f0000, reread:0xff7fffff007f0000 > expected:0xff7fffffff7fffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235530(0x0:DIMM > Unknown): read:0xffdfffff00df0000, reread:0xffdfffff00df0000 > expected:0xffdfffffffdfffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235570(0x0:DIMM > Unknown): read:0xfff7ffff00f70000, reread:0xfff7ffff00f70000 > expected:0xfff7fffffff7ffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2355b0(0x0:DIMM > Unknown): read:0xfffdffff00fd0000, reread:0xfffdffff00fd0000 > expected:0xfffdfffffffdffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2355f0(0x0:DIMM > Unknown): read:0xffff7fff00007f00, reread:0xffff7fff00007f00 > expected:0xffff7fffffff7fff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235630(0x0:DIMM > Unknown): read:0xffffdfff0000df00, reread:0xffffdfff0000df00 > expected:0xffffdfffffffdfff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235670(0x0:DIMM > Unknown): read:0xfffff7ff0000f700, reread:0xfffff7ff0000f700 > expected:0xfffff7fffffff7ff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2356b0(0x0:DIMM > Unknown): read:0xfffffdff0000fd00, reread:0xfffffdff0000fd00 > expected:0xfffffdfffffffdff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2356f0(0x0:DIMM > Unknown): read:0xffffff7f0000007f, reread:0xffffff7f0000007f > expected:0xffffff7fffffff7f > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235730(0x0:DIMM > Unknown): read:0xffffffdf000000df, reread:0xffffffdf000000df > expected:0xffffffdfffffffdf > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235770(0x0:DIMM > Unknown): read:0xfffffff7000000f7, reread:0xfffffff7000000f7 > expected:0xfffffff7fffffff7 > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834020(0x0:DIMM > Unknown): read:0x00000010000000ef, reread:0x00000010000000ef > expected:0x00000010ffffffef > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834028(0x0:DIMM > Unknown): read:0x00000020000000df, reread:0x00000020000000df > expected:0x00000020ffffffdf > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834030(0x0:DIMM > Unknown): read:0x00000040000000bf, reread:0x00000040000000bf > expected:0x00000040ffffffbf > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834038(0x0:DIMM > Unknown): read:0x000000800000007f, reread:0x000000800000007f > expected:0x00000080ffffff7f > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834060(0x0:DIMM > Unknown): read:0x000010000000efff, reread:0x000010000000efff > expected:0x00001000ffffefff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834068(0x0:DIMM > Unknown): read:0x000020000000dfff, reread:0x000020000000dfff > expected:0x00002000ffffdfff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834070(0x0:DIMM > Unknown): read:0x000040000000bfff, reread:0x000040000000bfff > expected:0x00004000ffffbfff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834078(0x0:DIMM > Unknown): read:0x0000800000007fff, reread:0x0000800000007fff > expected:0x00008000ffff7fff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340a0(0x0:DIMM > Unknown): read:0x0010000000ef00ff, reread:0x0010000000ef00ff > expected:0x00100000ffefffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340a8(0x0:DIMM > Unknown): read:0x0020000000df00ff, reread:0x0020000000df00ff > expected:0x00200000ffdfffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340b0(0x0:DIMM > Unknown): read:0x0040000000bf00ff, reread:0x0040000000bf00ff > expected:0x00400000ffbfffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340b8(0x0:DIMM > Unknown): read:0x00800000007f00ff, reread:0x00800000007f00ff > expected:0x00800000ff7fffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340e0(0x0:DIMM > Unknown): read:0x10000000ef0000ff, reread:0x10000000ef0000ff > expected:0x10000000efffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340e8(0x0:DIMM > Unknown): read:0x20000000df0000ff, reread:0x20000000df0000ff > expected:0x20000000dfffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340f0(0x0:DIMM > Unknown): read:0x40000000bf0000ff, reread:0x40000000bf0000ff > expected:0x40000000bfffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340f8(0x0:DIMM > Unknown): read:0x800000007f0000ff, reread:0x800000007f0000ff > expected:0x800000007fffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834120(0x0:DIMM > Unknown): read:0x04000000fb00ff00, reread:0x04000000fb00ff00 > expected:0x04000000fbffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834128(0x0:DIMM > Unknown): read:0x02000000fd00ff00, reread:0x02000000fd00ff00 > expected:0x02000000fdffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834130(0x0:DIMM > Unknown): read:0x01000000fe00ff00, reread:0x01000000fe00ff00 > expected:0x01000000feffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834138(0x0:DIMM > Unknown): read:0x00800000007fff00, reread:0x00800000007fff00 > expected:0x00800000ff7fffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834160(0x0:DIMM > Unknown): read:0x0004000000fbff00, reread:0x0004000000fbff00 > expected:0x00040000fffbffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834168(0x0:DIMM > Unknown): read:0x0002000000fdff00, reread:0x0002000000fdff00 > expected:0x00020000fffdffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834170(0x0:DIMM > Unknown): read:0x0001000000feff00, reread:0x0001000000feff00 > expected:0x00010000fffeffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834178(0x0:DIMM > Unknown): read:0x0000800000007f00, reread:0x0000800000007f00 > expected:0x00008000ffff7fff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341a0(0x0:DIMM > Unknown): read:0x000004000000fb00, reread:0x000004000000fb00 > expected:0x00000400fffffbff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341a8(0x0:DIMM > Unknown): read:0x000002000000fd00, reread:0x000002000000fd00 > expected:0x00000200fffffdff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341b0(0x0:DIMM > Unknown): read:0x000001000000fe00, reread:0x000001000000fe00 > expected:0x00000100fffffeff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341b8(0x0:DIMM > Unknown): read:0x000000800000ff7f, reread:0x000000800000ff7f > expected:0x00000080ffffff7f > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341e0(0x0:DIMM > Unknown): read:0x000000040000fffb, reread:0x000000040000fffb > expected:0x00000004fffffffb > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341e8(0x0:DIMM > Unknown): read:0x000000020000fffd, reread:0x000000020000fffd > expected:0x00000002fffffffd > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00008(0x0:DIMM > Unknown): read:0xfffffff7000000fb, reread:0xfffffff7000000fb > expected:0xfffffff7fffffffb > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00018(0x0:DIMM > Unknown): read:0xffffff7f000000bf, reread:0xffffff7f000000bf > expected:0xffffff7fffffffbf > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00028(0x0:DIMM > Unknown): read:0xfffff7ff0000fb00, reread:0xfffff7ff0000fb00 > expected:0xfffff7fffffffbff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00038(0x0:DIMM > Unknown): read:0xffff7fff0000bf00, reread:0xffff7fff0000bf00 > expected:0xffff7fffffffbfff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00048(0x0:DIMM > Unknown): read:0xfff7ffff00fb0000, reread:0xfff7ffff00fb0000 > expected:0xfff7fffffffbffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00058(0x0:DIMM > Unknown): read:0xff7fffff00bf0000, reread:0xff7fffff00bf0000 > expected:0xff7fffffffbfffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00068(0x0:DIMM > Unknown): read:0xf7fffffffb000000, reread:0xf7fffffffb000000 > expected:0xf7fffffffbffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00078(0x0:DIMM > Unknown): read:0x7fffffffbf000000, reread:0x7fffffffbf000000 > expected:0x7fffffffbfffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00088(0x0:DIMM > Unknown): read:0xf7ffffffef000000, reread:0xf7ffffffef000000 > expected:0xf7ffffffefffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00098(0x0:DIMM > Unknown): read:0xff7ffffffe000000, reread:0xff7ffffffe000000 > expected:0xff7ffffffeffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000a8(0x0:DIMM > Unknown): read:0xfff7ffff00ef0000, reread:0xfff7ffff00ef0000 > expected:0xfff7ffffffefffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000b8(0x0:DIMM > Unknown): read:0xffff7fff00fe0000, reread:0xffff7fff00fe0000 > expected:0xffff7ffffffeffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000c8(0x0:DIMM > Unknown): read:0xfffff7ff0000ef00, reread:0xfffff7ff0000ef00 > expected:0xfffff7ffffffefff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000d8(0x0:DIMM > Unknown): read:0xffffff7f0000fe00, reread:0xffffff7f0000fe00 > expected:0xffffff7ffffffeff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000e8(0x0:DIMM > Unknown): read:0xfffffff7000000ef, reread:0xfffffff7000000ef > expected:0xfffffff7ffffffef > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000f8(0x0:DIMM > Unknown): read:0xffffffff000000fe, reread:0xffffffff000000fe > expected:0xfffffffffffffffe > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00108(0x0:DIMM > Unknown): read:0xfffffff7000000fb, reread:0xfffffff7000000fb > expected:0xfffffff7fffffffb > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00118(0x0:DIMM > Unknown): read:0xffffff7f000000bf, reread:0xffffff7f000000bf > expected:0xffffff7fffffffbf > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00128(0x0:DIMM > Unknown): read:0xfffff7ff0000fb00, reread:0xfffff7ff0000fb00 > expected:0xfffff7fffffffbff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00138(0x0:DIMM > Unknown): read:0xffff7fff0000bf00, reread:0xffff7fff0000bf00 > expected:0xffff7fffffffbfff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00148(0x0:DIMM > Unknown): read:0xfff7ffff00fb0000, reread:0xfff7ffff00fb0000 > expected:0xfff7fffffffbffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00158(0x0:DIMM > Unknown): read:0xff7fffff00bf0000, reread:0xff7fffff00bf0000 > expected:0xff7fffffffbfffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00168(0x0:DIMM > Unknown): read:0xf7fffffffb000000, reread:0xf7fffffffb000000 > expected:0xf7fffffffbffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00178(0x0:DIMM > Unknown): read:0x7fffffffbf000000, reread:0x7fffffffbf000000 > expected:0x7fffffffbfffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00188(0x0:DIMM > Unknown): read:0xf7ffffffef000000, reread:0xf7ffffffef000000 > expected:0xf7ffffffefffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00198(0x0:DIMM > Unknown): read:0xff7ffffffe000000, reread:0xff7ffffffe000000 > expected:0xff7ffffffeffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001a8(0x0:DIMM > Unknown): read:0xfff7ffff00ef0000, reread:0xfff7ffff00ef0000 > expected:0xfff7ffffffefffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001b8(0x0:DIMM > Unknown): read:0xffff7fff00fe0000, reread:0xffff7fff00fe0000 > expected:0xffff7ffffffeffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001c8(0x0:DIMM > Unknown): read:0xfffff7ff0000ef00, reread:0xfffff7ff0000ef00 > expected:0xfffff7ffffffefff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001d8(0x0:DIMM > Unknown): read:0xffffff7f0000fe00, reread:0xffffff7f0000fe00 > expected:0xffffff7ffffffeff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200010(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200018(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200050(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200058(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200090(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200098(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72000d0(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72000d8(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200110(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200118(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200150(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200158(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200190(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200198(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72001d0(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72001d8(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200210(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200218(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200250(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200258(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200290(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200298(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72002d0(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72002d8(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200310(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200318(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200350(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200358(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200390(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200398(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00000(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00008(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00010(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00018(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00040(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00048(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00050(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00058(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00080(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00088(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00090(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00098(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000c0(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000c8(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000d0(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000d8(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00100(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00108(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00110(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00118(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00140(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00148(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00150(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00158(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00180(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00188(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00190(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00198(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa001c0(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Report Error: miscompare : DIMM Unknown : 1 : 572s > Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa001c8(0x0:DIMM > Unknown): read:0x0000000000000000, reread:0x0000000000000000 > expected:0xffffffffffffffff > Log: Thread 61 found 85568 hardware incidents > Log: Thread 62 found 169344 hardware incidents > Log: Thread 63 found 44544 hardware incidents > Log: Thread 64 found 149504 hardware incidents > Log: Thread 65 found 131968 hardware incidents > Log: Thread 66 found 150528 hardware incidents > Log: Thread 67 found 144384 hardware incidents > Log: Thread 68 found 149888 hardware incidents > Stats: Found 1025728 hardware incidents > Stats: Completed: 9176812.00M in 524.63s 17491.92MB/s, with 1025728 > hardware incidents, 0 errors > Stats: Memory Copy: 4890244.00M at 9402.74MB/s > Stats: File Copy: 0.00M at 0.00MB/s > Stats: Net Copy: 0.00M at 0.00MB/s > Stats: Data Check: 0.00M at 0.00MB/s > Stats: Invert Data: 4286568.00M at 8242.44MB/s > Stats: Disk: 0.00M at 0.00MB/s > > Status: FAIL - test discovered HW problems > ----------------------------------------------------------------------------------- > > Stefan > Am 07.02.2014 10:37, schrieb Stefan Priebe - Profihost AG: > > > > Am 07.02.2014 10:31, schrieb Dr. David Alan Gilbert: > >> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: > >>> > >>> Am 07.02.2014 10:15, schrieb Dr. David Alan Gilbert: > >>>> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: > >>>>> Am 07.02.2014 09:15, schrieb Alexandre DERUMIER: > >>>>>> > >>>>>> do you use xbzrle for live migration ? > >>>>> > >>>>> no - i'm really stucked right now with this. Biggest problem i can't > >>>>> reproduce with test machines ;-( > >>>> > >>>> Only being able to test on your production VMs isn't fun; > >>>> is it possible or you to run an extra program on these VMs - e.g. > >>>> if we came up with a simple (userland) memory test? > >>> > >>> You mean to reproduce? > >> > >> I'm more interested in seeing what type of corruption is happening; > >> if you've got a test VM that corrupts memory and we can run a program > >> in that vm that writes a known pattern into memory and checks it > >> then see what changed after migration, it might give a clue. > >> > >> But obviously this would only be of any use if run on the VM that actually > >> fails. > > > > Right that makes sense - sadly i still don't know how to reproduce? Any > > app ideas i can try? > > > > > >>> I already tried https://code.google.com/p/stressapptest/ while migrating > >>> on a test VM but this works fine. > >>> > >>> I also tried running mysql bench while migrating on a test vm and this > >>> works too ;-( > >> > >> > >> Dave > >> -- > >> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > >> -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 12:21 ` Dr. David Alan Gilbert @ 2014-02-07 12:30 ` Stefan Priebe - Profihost AG 2014-02-07 12:44 ` Paolo Bonzini 0 siblings, 1 reply; 50+ messages in thread From: Stefan Priebe - Profihost AG @ 2014-02-07 12:30 UTC (permalink / raw) To: Dr. David Alan Gilbert; +Cc: m.gibula, qemu-devel, Alexandre DERUMIER Hi, Am 07.02.2014 13:21, schrieb Dr. David Alan Gilbert: > * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: >> Hi, >> >> i was able to reproduce with a longer running test VM running the google >> stress test. > > Hmm that's quite a fun set of differences; I think I'd like > to understand whether the pattern is related to the pattern of what > the test is doing. > > Can you just give an explanation of exactly how you ran that test? > What you installed, how exactly you ran it. While migrating i've still no reliable way to reproduce but i'll try to. I can force the problem without migration when start with: bin/stressapptest -s 3600 -m 20 -i 20 -C 20 --force_errors => inject false errors to test error handling Stefan > Then Marcin and I can try and replicate it. > > Dave > >> And it happens exacly when the migration finishes it does not happen >> while the migration is running. >> >> Google Stress Output displays Memory errors: >> ------------------------------------------------ >> Page Error: miscompare on CPU 5(0xffff) at 0x7f52431341c0(0x0:DIMM >> Unknown): read:0x00000040000000bf, reread:0x00000040000000bf >> expected:0x00000040ffffffbf >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Page Error: miscompare on CPU 5(0xffff) at 0x7f52431341c8(0x0:DIMM >> Unknown): read:0x00000020000000df, reread:0x00000020000000df >> expected:0x00000020ffffffdf >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34020(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34028(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34060(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34068(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340a0(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340a8(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340e0(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340e8(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34120(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34128(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34160(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34168(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341a0(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341a8(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341e0(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341e8(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34220(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34228(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34260(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34268(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342a0(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342a8(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342e0(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342e8(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34320(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34328(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34360(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34368(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c343a0(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 571s >> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c343a8(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235030(0x0:DIMM >> Unknown): read:0xfffffffd000000fd, reread:0xfffffffd000000fd >> expected:0xfffffffdfffffffd >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235070(0x0:DIMM >> Unknown): read:0xfffffff7000000f7, reread:0xfffffff7000000f7 >> expected:0xfffffff7fffffff7 >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2350b0(0x0:DIMM >> Unknown): read:0xffffffdf000000df, reread:0xffffffdf000000df >> expected:0xffffffdfffffffdf >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2350f0(0x0:DIMM >> Unknown): read:0xffffff7f0000007f, reread:0xffffff7f0000007f >> expected:0xffffff7fffffff7f >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235130(0x0:DIMM >> Unknown): read:0xfffffdff0000fd00, reread:0xfffffdff0000fd00 >> expected:0xfffffdfffffffdff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235170(0x0:DIMM >> Unknown): read:0xfffff7ff0000f700, reread:0xfffff7ff0000f700 >> expected:0xfffff7fffffff7ff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2351b0(0x0:DIMM >> Unknown): read:0xffffdfff0000df00, reread:0xffffdfff0000df00 >> expected:0xffffdfffffffdfff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2351f0(0x0:DIMM >> Unknown): read:0xffff7fff00007f00, reread:0xffff7fff00007f00 >> expected:0xffff7fffffff7fff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235230(0x0:DIMM >> Unknown): read:0xfffdffff00fd0000, reread:0xfffdffff00fd0000 >> expected:0xfffdfffffffdffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235270(0x0:DIMM >> Unknown): read:0xfff7ffff00f70000, reread:0xfff7ffff00f70000 >> expected:0xfff7fffffff7ffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2352b0(0x0:DIMM >> Unknown): read:0xffdfffff00df0000, reread:0xffdfffff00df0000 >> expected:0xffdfffffffdfffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2352f0(0x0:DIMM >> Unknown): read:0xff7fffff007f0000, reread:0xff7fffff007f0000 >> expected:0xff7fffffff7fffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235330(0x0:DIMM >> Unknown): read:0xfdfffffffd000000, reread:0xfdfffffffd000000 >> expected:0xfdfffffffdffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235370(0x0:DIMM >> Unknown): read:0xf7fffffff7000000, reread:0xf7fffffff7000000 >> expected:0xf7fffffff7ffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2353b0(0x0:DIMM >> Unknown): read:0xdfffffffdf000000, reread:0xdfffffffdf000000 >> expected:0xdfffffffdfffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2353f0(0x0:DIMM >> Unknown): read:0x7fffffff7f000000, reread:0x7fffffff7f000000 >> expected:0x7fffffff7fffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235430(0x0:DIMM >> Unknown): read:0xdfffffffdf000000, reread:0xdfffffffdf000000 >> expected:0xdfffffffdfffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235470(0x0:DIMM >> Unknown): read:0xf7fffffff7000000, reread:0xf7fffffff7000000 >> expected:0xf7fffffff7ffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2354b0(0x0:DIMM >> Unknown): read:0xfdfffffffd000000, reread:0xfdfffffffd000000 >> expected:0xfdfffffffdffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2354f0(0x0:DIMM >> Unknown): read:0xff7fffff007f0000, reread:0xff7fffff007f0000 >> expected:0xff7fffffff7fffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235530(0x0:DIMM >> Unknown): read:0xffdfffff00df0000, reread:0xffdfffff00df0000 >> expected:0xffdfffffffdfffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235570(0x0:DIMM >> Unknown): read:0xfff7ffff00f70000, reread:0xfff7ffff00f70000 >> expected:0xfff7fffffff7ffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2355b0(0x0:DIMM >> Unknown): read:0xfffdffff00fd0000, reread:0xfffdffff00fd0000 >> expected:0xfffdfffffffdffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2355f0(0x0:DIMM >> Unknown): read:0xffff7fff00007f00, reread:0xffff7fff00007f00 >> expected:0xffff7fffffff7fff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235630(0x0:DIMM >> Unknown): read:0xffffdfff0000df00, reread:0xffffdfff0000df00 >> expected:0xffffdfffffffdfff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235670(0x0:DIMM >> Unknown): read:0xfffff7ff0000f700, reread:0xfffff7ff0000f700 >> expected:0xfffff7fffffff7ff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2356b0(0x0:DIMM >> Unknown): read:0xfffffdff0000fd00, reread:0xfffffdff0000fd00 >> expected:0xfffffdfffffffdff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2356f0(0x0:DIMM >> Unknown): read:0xffffff7f0000007f, reread:0xffffff7f0000007f >> expected:0xffffff7fffffff7f >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235730(0x0:DIMM >> Unknown): read:0xffffffdf000000df, reread:0xffffffdf000000df >> expected:0xffffffdfffffffdf >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235770(0x0:DIMM >> Unknown): read:0xfffffff7000000f7, reread:0xfffffff7000000f7 >> expected:0xfffffff7fffffff7 >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834020(0x0:DIMM >> Unknown): read:0x00000010000000ef, reread:0x00000010000000ef >> expected:0x00000010ffffffef >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834028(0x0:DIMM >> Unknown): read:0x00000020000000df, reread:0x00000020000000df >> expected:0x00000020ffffffdf >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834030(0x0:DIMM >> Unknown): read:0x00000040000000bf, reread:0x00000040000000bf >> expected:0x00000040ffffffbf >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834038(0x0:DIMM >> Unknown): read:0x000000800000007f, reread:0x000000800000007f >> expected:0x00000080ffffff7f >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834060(0x0:DIMM >> Unknown): read:0x000010000000efff, reread:0x000010000000efff >> expected:0x00001000ffffefff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834068(0x0:DIMM >> Unknown): read:0x000020000000dfff, reread:0x000020000000dfff >> expected:0x00002000ffffdfff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834070(0x0:DIMM >> Unknown): read:0x000040000000bfff, reread:0x000040000000bfff >> expected:0x00004000ffffbfff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834078(0x0:DIMM >> Unknown): read:0x0000800000007fff, reread:0x0000800000007fff >> expected:0x00008000ffff7fff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340a0(0x0:DIMM >> Unknown): read:0x0010000000ef00ff, reread:0x0010000000ef00ff >> expected:0x00100000ffefffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340a8(0x0:DIMM >> Unknown): read:0x0020000000df00ff, reread:0x0020000000df00ff >> expected:0x00200000ffdfffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340b0(0x0:DIMM >> Unknown): read:0x0040000000bf00ff, reread:0x0040000000bf00ff >> expected:0x00400000ffbfffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340b8(0x0:DIMM >> Unknown): read:0x00800000007f00ff, reread:0x00800000007f00ff >> expected:0x00800000ff7fffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340e0(0x0:DIMM >> Unknown): read:0x10000000ef0000ff, reread:0x10000000ef0000ff >> expected:0x10000000efffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340e8(0x0:DIMM >> Unknown): read:0x20000000df0000ff, reread:0x20000000df0000ff >> expected:0x20000000dfffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340f0(0x0:DIMM >> Unknown): read:0x40000000bf0000ff, reread:0x40000000bf0000ff >> expected:0x40000000bfffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340f8(0x0:DIMM >> Unknown): read:0x800000007f0000ff, reread:0x800000007f0000ff >> expected:0x800000007fffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834120(0x0:DIMM >> Unknown): read:0x04000000fb00ff00, reread:0x04000000fb00ff00 >> expected:0x04000000fbffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834128(0x0:DIMM >> Unknown): read:0x02000000fd00ff00, reread:0x02000000fd00ff00 >> expected:0x02000000fdffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834130(0x0:DIMM >> Unknown): read:0x01000000fe00ff00, reread:0x01000000fe00ff00 >> expected:0x01000000feffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834138(0x0:DIMM >> Unknown): read:0x00800000007fff00, reread:0x00800000007fff00 >> expected:0x00800000ff7fffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834160(0x0:DIMM >> Unknown): read:0x0004000000fbff00, reread:0x0004000000fbff00 >> expected:0x00040000fffbffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834168(0x0:DIMM >> Unknown): read:0x0002000000fdff00, reread:0x0002000000fdff00 >> expected:0x00020000fffdffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834170(0x0:DIMM >> Unknown): read:0x0001000000feff00, reread:0x0001000000feff00 >> expected:0x00010000fffeffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834178(0x0:DIMM >> Unknown): read:0x0000800000007f00, reread:0x0000800000007f00 >> expected:0x00008000ffff7fff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341a0(0x0:DIMM >> Unknown): read:0x000004000000fb00, reread:0x000004000000fb00 >> expected:0x00000400fffffbff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341a8(0x0:DIMM >> Unknown): read:0x000002000000fd00, reread:0x000002000000fd00 >> expected:0x00000200fffffdff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341b0(0x0:DIMM >> Unknown): read:0x000001000000fe00, reread:0x000001000000fe00 >> expected:0x00000100fffffeff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341b8(0x0:DIMM >> Unknown): read:0x000000800000ff7f, reread:0x000000800000ff7f >> expected:0x00000080ffffff7f >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341e0(0x0:DIMM >> Unknown): read:0x000000040000fffb, reread:0x000000040000fffb >> expected:0x00000004fffffffb >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341e8(0x0:DIMM >> Unknown): read:0x000000020000fffd, reread:0x000000020000fffd >> expected:0x00000002fffffffd >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00008(0x0:DIMM >> Unknown): read:0xfffffff7000000fb, reread:0xfffffff7000000fb >> expected:0xfffffff7fffffffb >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00018(0x0:DIMM >> Unknown): read:0xffffff7f000000bf, reread:0xffffff7f000000bf >> expected:0xffffff7fffffffbf >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00028(0x0:DIMM >> Unknown): read:0xfffff7ff0000fb00, reread:0xfffff7ff0000fb00 >> expected:0xfffff7fffffffbff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00038(0x0:DIMM >> Unknown): read:0xffff7fff0000bf00, reread:0xffff7fff0000bf00 >> expected:0xffff7fffffffbfff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00048(0x0:DIMM >> Unknown): read:0xfff7ffff00fb0000, reread:0xfff7ffff00fb0000 >> expected:0xfff7fffffffbffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00058(0x0:DIMM >> Unknown): read:0xff7fffff00bf0000, reread:0xff7fffff00bf0000 >> expected:0xff7fffffffbfffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00068(0x0:DIMM >> Unknown): read:0xf7fffffffb000000, reread:0xf7fffffffb000000 >> expected:0xf7fffffffbffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00078(0x0:DIMM >> Unknown): read:0x7fffffffbf000000, reread:0x7fffffffbf000000 >> expected:0x7fffffffbfffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00088(0x0:DIMM >> Unknown): read:0xf7ffffffef000000, reread:0xf7ffffffef000000 >> expected:0xf7ffffffefffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00098(0x0:DIMM >> Unknown): read:0xff7ffffffe000000, reread:0xff7ffffffe000000 >> expected:0xff7ffffffeffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000a8(0x0:DIMM >> Unknown): read:0xfff7ffff00ef0000, reread:0xfff7ffff00ef0000 >> expected:0xfff7ffffffefffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000b8(0x0:DIMM >> Unknown): read:0xffff7fff00fe0000, reread:0xffff7fff00fe0000 >> expected:0xffff7ffffffeffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000c8(0x0:DIMM >> Unknown): read:0xfffff7ff0000ef00, reread:0xfffff7ff0000ef00 >> expected:0xfffff7ffffffefff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000d8(0x0:DIMM >> Unknown): read:0xffffff7f0000fe00, reread:0xffffff7f0000fe00 >> expected:0xffffff7ffffffeff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000e8(0x0:DIMM >> Unknown): read:0xfffffff7000000ef, reread:0xfffffff7000000ef >> expected:0xfffffff7ffffffef >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000f8(0x0:DIMM >> Unknown): read:0xffffffff000000fe, reread:0xffffffff000000fe >> expected:0xfffffffffffffffe >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00108(0x0:DIMM >> Unknown): read:0xfffffff7000000fb, reread:0xfffffff7000000fb >> expected:0xfffffff7fffffffb >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00118(0x0:DIMM >> Unknown): read:0xffffff7f000000bf, reread:0xffffff7f000000bf >> expected:0xffffff7fffffffbf >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00128(0x0:DIMM >> Unknown): read:0xfffff7ff0000fb00, reread:0xfffff7ff0000fb00 >> expected:0xfffff7fffffffbff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00138(0x0:DIMM >> Unknown): read:0xffff7fff0000bf00, reread:0xffff7fff0000bf00 >> expected:0xffff7fffffffbfff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00148(0x0:DIMM >> Unknown): read:0xfff7ffff00fb0000, reread:0xfff7ffff00fb0000 >> expected:0xfff7fffffffbffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00158(0x0:DIMM >> Unknown): read:0xff7fffff00bf0000, reread:0xff7fffff00bf0000 >> expected:0xff7fffffffbfffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00168(0x0:DIMM >> Unknown): read:0xf7fffffffb000000, reread:0xf7fffffffb000000 >> expected:0xf7fffffffbffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00178(0x0:DIMM >> Unknown): read:0x7fffffffbf000000, reread:0x7fffffffbf000000 >> expected:0x7fffffffbfffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00188(0x0:DIMM >> Unknown): read:0xf7ffffffef000000, reread:0xf7ffffffef000000 >> expected:0xf7ffffffefffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00198(0x0:DIMM >> Unknown): read:0xff7ffffffe000000, reread:0xff7ffffffe000000 >> expected:0xff7ffffffeffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001a8(0x0:DIMM >> Unknown): read:0xfff7ffff00ef0000, reread:0xfff7ffff00ef0000 >> expected:0xfff7ffffffefffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001b8(0x0:DIMM >> Unknown): read:0xffff7fff00fe0000, reread:0xffff7fff00fe0000 >> expected:0xffff7ffffffeffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001c8(0x0:DIMM >> Unknown): read:0xfffff7ff0000ef00, reread:0xfffff7ff0000ef00 >> expected:0xfffff7ffffffefff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001d8(0x0:DIMM >> Unknown): read:0xffffff7f0000fe00, reread:0xffffff7f0000fe00 >> expected:0xffffff7ffffffeff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200010(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200018(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200050(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200058(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200090(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200098(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72000d0(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72000d8(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200110(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200118(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200150(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200158(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200190(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200198(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72001d0(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72001d8(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200210(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200218(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200250(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200258(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200290(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200298(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72002d0(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72002d8(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200310(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200318(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200350(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200358(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200390(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200398(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00000(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00008(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00010(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00018(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00040(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00048(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00050(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00058(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00080(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00088(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00090(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00098(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000c0(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000c8(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000d0(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000d8(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00100(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00108(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00110(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00118(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00140(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00148(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00150(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00158(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00180(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00188(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00190(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00198(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa001c0(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Report Error: miscompare : DIMM Unknown : 1 : 572s >> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa001c8(0x0:DIMM >> Unknown): read:0x0000000000000000, reread:0x0000000000000000 >> expected:0xffffffffffffffff >> Log: Thread 61 found 85568 hardware incidents >> Log: Thread 62 found 169344 hardware incidents >> Log: Thread 63 found 44544 hardware incidents >> Log: Thread 64 found 149504 hardware incidents >> Log: Thread 65 found 131968 hardware incidents >> Log: Thread 66 found 150528 hardware incidents >> Log: Thread 67 found 144384 hardware incidents >> Log: Thread 68 found 149888 hardware incidents >> Stats: Found 1025728 hardware incidents >> Stats: Completed: 9176812.00M in 524.63s 17491.92MB/s, with 1025728 >> hardware incidents, 0 errors >> Stats: Memory Copy: 4890244.00M at 9402.74MB/s >> Stats: File Copy: 0.00M at 0.00MB/s >> Stats: Net Copy: 0.00M at 0.00MB/s >> Stats: Data Check: 0.00M at 0.00MB/s >> Stats: Invert Data: 4286568.00M at 8242.44MB/s >> Stats: Disk: 0.00M at 0.00MB/s >> >> Status: FAIL - test discovered HW problems >> ----------------------------------------------------------------------------------- >> >> Stefan >> Am 07.02.2014 10:37, schrieb Stefan Priebe - Profihost AG: >>> >>> Am 07.02.2014 10:31, schrieb Dr. David Alan Gilbert: >>>> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: >>>>> >>>>> Am 07.02.2014 10:15, schrieb Dr. David Alan Gilbert: >>>>>> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: >>>>>>> Am 07.02.2014 09:15, schrieb Alexandre DERUMIER: >>>>>>>> >>>>>>>> do you use xbzrle for live migration ? >>>>>>> >>>>>>> no - i'm really stucked right now with this. Biggest problem i can't >>>>>>> reproduce with test machines ;-( >>>>>> >>>>>> Only being able to test on your production VMs isn't fun; >>>>>> is it possible or you to run an extra program on these VMs - e.g. >>>>>> if we came up with a simple (userland) memory test? >>>>> >>>>> You mean to reproduce? >>>> >>>> I'm more interested in seeing what type of corruption is happening; >>>> if you've got a test VM that corrupts memory and we can run a program >>>> in that vm that writes a known pattern into memory and checks it >>>> then see what changed after migration, it might give a clue. >>>> >>>> But obviously this would only be of any use if run on the VM that actually >>>> fails. >>> >>> Right that makes sense - sadly i still don't know how to reproduce? Any >>> app ideas i can try? >>> >>> >>>>> I already tried https://code.google.com/p/stressapptest/ while migrating >>>>> on a test VM but this works fine. >>>>> >>>>> I also tried running mysql bench while migrating on a test vm and this >>>>> works too ;-( >>>> >>>> >>>> Dave >>>> -- >>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >>>> > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 12:30 ` Stefan Priebe - Profihost AG @ 2014-02-07 12:44 ` Paolo Bonzini 2014-02-07 13:04 ` Stefan Priebe - Profihost AG 0 siblings, 1 reply; 50+ messages in thread From: Paolo Bonzini @ 2014-02-07 12:44 UTC (permalink / raw) To: Stefan Priebe - Profihost AG, Dr. David Alan Gilbert Cc: m.gibula, qemu-devel, Alexandre DERUMIER Il 07/02/2014 13:30, Stefan Priebe - Profihost AG ha scritto: >> >> i was able to reproduce with a longer running test VM running the google >> >> stress test. >> >> Hmm that's quite a fun set of differences; I think I'd like >> to understand whether the pattern is related to the pattern of what >> the test is doing. Stefan, can you try to reproduce it: - with Unix migration between two QEMUs on the same host - with different hosts - with a different network (e.g. just a cross cable between two machines) Paolo ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 12:44 ` Paolo Bonzini @ 2014-02-07 13:04 ` Stefan Priebe - Profihost AG 2014-02-07 13:08 ` Dr. David Alan Gilbert 2014-02-07 13:19 ` Paolo Bonzini 0 siblings, 2 replies; 50+ messages in thread From: Stefan Priebe - Profihost AG @ 2014-02-07 13:04 UTC (permalink / raw) To: Paolo Bonzini, Dr. David Alan Gilbert Cc: m.gibula, qemu-devel, Alexandre DERUMIER Hi, Am 07.02.2014 13:44, schrieb Paolo Bonzini: > Il 07/02/2014 13:30, Stefan Priebe - Profihost AG ha scritto: >>> >> i was able to reproduce with a longer running test VM running the >>> google >>> >> stress test. >>> >>> Hmm that's quite a fun set of differences; I think I'd like >>> to understand whether the pattern is related to the pattern of what >>> the test is doing. > > Stefan, can you try to reproduce it: first of all i've now a memory image of a VM where i can reproduce it. reproducing does NOT work if i boot the VM freshly i need to let it run for some hours. Then just when the migration finishes there is a short time frame where the google stress app reports memory errors than when the migration finishes it runs fine again. It seems to me it is related to pause and unpause/resume? > - with Unix migration between two QEMUs on the same host now tested => same issue > - with different hosts already tested => same issue > - with a different network (e.g. just a cross cable between two machines) already tested => same issue Greets, Stefan ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 13:04 ` Stefan Priebe - Profihost AG @ 2014-02-07 13:08 ` Dr. David Alan Gilbert 2014-02-07 13:10 ` Stefan Priebe - Profihost AG 2014-02-07 13:19 ` Paolo Bonzini 1 sibling, 1 reply; 50+ messages in thread From: Dr. David Alan Gilbert @ 2014-02-07 13:08 UTC (permalink / raw) To: Stefan Priebe - Profihost AG Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: > Hi, > > Am 07.02.2014 13:44, schrieb Paolo Bonzini: > > Il 07/02/2014 13:30, Stefan Priebe - Profihost AG ha scritto: > >>> >> i was able to reproduce with a longer running test VM running the > >>> google > >>> >> stress test. > >>> > >>> Hmm that's quite a fun set of differences; I think I'd like > >>> to understand whether the pattern is related to the pattern of what > >>> the test is doing. > > > > Stefan, can you try to reproduce it: > > first of all i've now a memory image of a VM where i can reproduce it. > reproducing does NOT work if i boot the VM freshly i need to let it run > for some hours. > > Then just when the migration finishes there is a short time frame where > the google stress app reports memory errors than when the migration > finishes it runs fine again. > > It seems to me it is related to pause and unpause/resume? But do you have to pause/resume it to cause the error? Have you got cases where you boot it and then leave it running for a few hours and then it fails if you migrate it? Dave -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 13:08 ` Dr. David Alan Gilbert @ 2014-02-07 13:10 ` Stefan Priebe - Profihost AG 2014-02-07 13:15 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 50+ messages in thread From: Stefan Priebe - Profihost AG @ 2014-02-07 13:10 UTC (permalink / raw) To: Dr. David Alan Gilbert; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER Hi, Am 07.02.2014 14:08, schrieb Dr. David Alan Gilbert: > * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: >> Hi, >> >> Am 07.02.2014 13:44, schrieb Paolo Bonzini: >>> Il 07/02/2014 13:30, Stefan Priebe - Profihost AG ha scritto: >>>>>>> i was able to reproduce with a longer running test VM running the >>>>> google >>>>>>> stress test. >>>>> >>>>> Hmm that's quite a fun set of differences; I think I'd like >>>>> to understand whether the pattern is related to the pattern of what >>>>> the test is doing. >>> >>> Stefan, can you try to reproduce it: >> >> first of all i've now a memory image of a VM where i can reproduce it. >> reproducing does NOT work if i boot the VM freshly i need to let it run >> for some hours. >> >> Then just when the migration finishes there is a short time frame where >> the google stress app reports memory errors than when the migration >> finishes it runs fine again. >> >> It seems to me it is related to pause and unpause/resume? > > But do you have to pause/resume it to cause the error? Have you got cases > where you boot it and then leave it running for a few hours and then it > fails if you migrate it? Yes but isn't migration always a pause / unpause at the end? I thought migration_downtime is the value a very small pause unpause is allowed. Stefan ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 13:10 ` Stefan Priebe - Profihost AG @ 2014-02-07 13:15 ` Dr. David Alan Gilbert 2014-02-07 13:21 ` Stefan Priebe - Profihost AG 0 siblings, 1 reply; 50+ messages in thread From: Dr. David Alan Gilbert @ 2014-02-07 13:15 UTC (permalink / raw) To: Stefan Priebe - Profihost AG Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: > Hi, > > Am 07.02.2014 14:08, schrieb Dr. David Alan Gilbert: > > * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: > >> first of all i've now a memory image of a VM where i can reproduce it. > >> reproducing does NOT work if i boot the VM freshly i need to let it run > >> for some hours. > >> > >> Then just when the migration finishes there is a short time frame where > >> the google stress app reports memory errors than when the migration > >> finishes it runs fine again. > >> > >> It seems to me it is related to pause and unpause/resume? > > > > But do you have to pause/resume it to cause the error? Have you got cases > > where you boot it and then leave it running for a few hours and then it > > fails if you migrate it? > > Yes but isn't migration always a pause / unpause at the end? I thought > migration_downtime is the value a very small pause unpause is allowed. There's a heck of a lot of other stuff that goes on in migration, and that downtime isn't quite the same. If it can be reproduced with just suspend/resume stuff then that's a different place to start looking than if it's migration only. Dave -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 13:15 ` Dr. David Alan Gilbert @ 2014-02-07 13:21 ` Stefan Priebe - Profihost AG 0 siblings, 0 replies; 50+ messages in thread From: Stefan Priebe - Profihost AG @ 2014-02-07 13:21 UTC (permalink / raw) To: Dr. David Alan Gilbert; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER Hi, Am 07.02.2014 14:15, schrieb Dr. David Alan Gilbert: > * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: >> Hi, >> >> Am 07.02.2014 14:08, schrieb Dr. David Alan Gilbert: >>> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: > >>>> first of all i've now a memory image of a VM where i can reproduce it. >>>> reproducing does NOT work if i boot the VM freshly i need to let it run >>>> for some hours. >>>> >>>> Then just when the migration finishes there is a short time frame where >>>> the google stress app reports memory errors than when the migration >>>> finishes it runs fine again. >>>> >>>> It seems to me it is related to pause and unpause/resume? >>> >>> But do you have to pause/resume it to cause the error? Have you got cases >>> where you boot it and then leave it running for a few hours and then it >>> fails if you migrate it? >> >> Yes but isn't migration always a pause / unpause at the end? I thought >> migration_downtime is the value a very small pause unpause is allowed. > > There's a heck of a lot of other stuff that goes on in migration, and that > downtime isn't quite the same. > > If it can be reproduced with just suspend/resume stuff then that's a different > place to start looking than if it's migration only. ah OK now i got it. No i can't reproduce with suspend resume. But while migrating it happens directly at the end when the switch from host a to b happens. > Dave > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 13:04 ` Stefan Priebe - Profihost AG 2014-02-07 13:08 ` Dr. David Alan Gilbert @ 2014-02-07 13:19 ` Paolo Bonzini 2014-02-07 13:39 ` Stefan Priebe - Profihost AG 1 sibling, 1 reply; 50+ messages in thread From: Paolo Bonzini @ 2014-02-07 13:19 UTC (permalink / raw) To: Stefan Priebe - Profihost AG, Dr. David Alan Gilbert Cc: m.gibula, qemu-devel, Alexandre DERUMIER Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto: > first of all i've now a memory image of a VM where i can reproduce it. You mean you start that VM with -incoming 'exec:cat /path/to/vm.img'? But google stress test doesn't report any error until you start migration _and_ it finishes? That sounds good enough. Can you upload the image somewhere (doesn't have to be a public place, you can contact David or others offlist)? > reproducing does NOT work if i boot the VM freshly i need to let it run > for some hours. > > Then just when the migration finishes there is a short time frame where > the google stress app reports memory errors than when the migration > finishes it runs fine again. > > It seems to me it is related to pause and unpause/resume? > >> > - with Unix migration between two QEMUs on the same host > now tested => same issue > >> > - with different hosts > already tested => same issue > >> > - with a different network (e.g. just a cross cable between two machines) > already tested => same issue Another test: - start the VM with -S, migrate, do errors appear on the destination? Thanks, Paolo ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 13:19 ` Paolo Bonzini @ 2014-02-07 13:39 ` Stefan Priebe - Profihost AG 2014-02-07 13:45 ` Stefan Priebe - Profihost AG 0 siblings, 1 reply; 50+ messages in thread From: Stefan Priebe - Profihost AG @ 2014-02-07 13:39 UTC (permalink / raw) To: Paolo Bonzini, Dr. David Alan Gilbert Cc: m.gibula, qemu-devel, Alexandre DERUMIER Hi, Am 07.02.2014 14:19, schrieb Paolo Bonzini: > Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto: >> first of all i've now a memory image of a VM where i can reproduce it. > > You mean you start that VM with -incoming 'exec:cat /path/to/vm.img'? > But google stress test doesn't report any error until you start > migration _and_ it finishes? Sorry no i meant i have a VM where i saved the memory to disk - so i don't need to wait hours until i can reproduce as it does not happen with a fresh started VM. So it's a state file i think. > Another test: > > - start the VM with -S, migrate, do errors appear on the destination? I started with -S and the errors appear AFTER resuming/unpause the VM. So it is fine until i resume it on the "new" host. Stefan ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 13:39 ` Stefan Priebe - Profihost AG @ 2014-02-07 13:45 ` Stefan Priebe - Profihost AG 2014-02-07 19:21 ` Stefan Priebe 0 siblings, 1 reply; 50+ messages in thread From: Stefan Priebe - Profihost AG @ 2014-02-07 13:45 UTC (permalink / raw) To: Paolo Bonzini, Dr. David Alan Gilbert Cc: m.gibula, qemu-devel, Alexandre DERUMIER it's always the same "pattern" there are too many 0 instead of X. only seen: read:0x0000000000000000 ... expected:0xffffffffffffffff or read:0xffffffff00000000 ... expected:0xffffffffffffffff or read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff or read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5 no idea if this helps. Stefan Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG: > Hi, > Am 07.02.2014 14:19, schrieb Paolo Bonzini: >> Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto: >>> first of all i've now a memory image of a VM where i can reproduce it. >> >> You mean you start that VM with -incoming 'exec:cat /path/to/vm.img'? >> But google stress test doesn't report any error until you start >> migration _and_ it finishes? > > Sorry no i meant i have a VM where i saved the memory to disk - so i > don't need to wait hours until i can reproduce as it does not happen > with a fresh started VM. So it's a state file i think. > >> Another test: >> >> - start the VM with -S, migrate, do errors appear on the destination? > > I started with -S and the errors appear AFTER resuming/unpause the VM. > So it is fine until i resume it on the "new" host. > > Stefan > ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 13:45 ` Stefan Priebe - Profihost AG @ 2014-02-07 19:21 ` Stefan Priebe 2014-02-07 20:02 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 50+ messages in thread From: Stefan Priebe @ 2014-02-07 19:21 UTC (permalink / raw) To: Paolo Bonzini, Dr. David Alan Gilbert Cc: m.gibula, qemu-devel, Alexandre DERUMIER anything i could try or debug? to help to find the problem? Stefan Am 07.02.2014 14:45, schrieb Stefan Priebe - Profihost AG: > it's always the same "pattern" there are too many 0 instead of X. > > only seen: > > read:0x0000000000000000 ... expected:0xffffffffffffffff > > or > > read:0xffffffff00000000 ... expected:0xffffffffffffffff > > or > > read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff > > or > > read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5 > > no idea if this helps. > > Stefan > > Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG: >> Hi, >> Am 07.02.2014 14:19, schrieb Paolo Bonzini: >>> Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto: >>>> first of all i've now a memory image of a VM where i can reproduce it. >>> >>> You mean you start that VM with -incoming 'exec:cat /path/to/vm.img'? >>> But google stress test doesn't report any error until you start >>> migration _and_ it finishes? >> >> Sorry no i meant i have a VM where i saved the memory to disk - so i >> don't need to wait hours until i can reproduce as it does not happen >> with a fresh started VM. So it's a state file i think. >> >>> Another test: >>> >>> - start the VM with -S, migrate, do errors appear on the destination? >> >> I started with -S and the errors appear AFTER resuming/unpause the VM. >> So it is fine until i resume it on the "new" host. >> >> Stefan >> ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 19:21 ` Stefan Priebe @ 2014-02-07 20:02 ` Dr. David Alan Gilbert 2014-02-07 20:10 ` Stefan Priebe 0 siblings, 1 reply; 50+ messages in thread From: Dr. David Alan Gilbert @ 2014-02-07 20:02 UTC (permalink / raw) To: Stefan Priebe; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER * Stefan Priebe (s.priebe@profihost.ag) wrote: > anything i could try or debug? to help to find the problem? I think the most useful would be to see if the problem is a new problem in the 1.7 you're using or has existed for a while; depending on the machine type you used, it might be possible to load that image on an earlier (or newer) qemu and try the same test, however if the problem doesn't repeat reliably it can be hard. If you have any way of simplifying the configuration of the VM it would be good; e.g. if you could get a failure on something without graphics (-nographic) and USB. Dave > > Stefan > > Am 07.02.2014 14:45, schrieb Stefan Priebe - Profihost AG: > >it's always the same "pattern" there are too many 0 instead of X. > > > >only seen: > > > >read:0x0000000000000000 ... expected:0xffffffffffffffff > > > >or > > > >read:0xffffffff00000000 ... expected:0xffffffffffffffff > > > >or > > > >read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff > > > >or > > > >read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5 > > > >no idea if this helps. > > > >Stefan > > > >Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG: > >>Hi, > >>Am 07.02.2014 14:19, schrieb Paolo Bonzini: > >>>Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto: > >>>>first of all i've now a memory image of a VM where i can reproduce it. > >>> > >>>You mean you start that VM with -incoming 'exec:cat /path/to/vm.img'? > >>>But google stress test doesn't report any error until you start > >>>migration _and_ it finishes? > >> > >>Sorry no i meant i have a VM where i saved the memory to disk - so i > >>don't need to wait hours until i can reproduce as it does not happen > >>with a fresh started VM. So it's a state file i think. > >> > >>>Another test: > >>> > >>>- start the VM with -S, migrate, do errors appear on the destination? > >> > >>I started with -S and the errors appear AFTER resuming/unpause the VM. > >>So it is fine until i resume it on the "new" host. > >> > >>Stefan > >> > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 20:02 ` Dr. David Alan Gilbert @ 2014-02-07 20:10 ` Stefan Priebe 2014-02-08 19:23 ` Stefan Priebe 0 siblings, 1 reply; 50+ messages in thread From: Stefan Priebe @ 2014-02-07 20:10 UTC (permalink / raw) To: Dr. David Alan Gilbert; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER Am 07.02.2014 21:02, schrieb Dr. David Alan Gilbert: > * Stefan Priebe (s.priebe@profihost.ag) wrote: >> anything i could try or debug? to help to find the problem? > > I think the most useful would be to see if the problem is > a new problem in the 1.7 you're using or has existed > for a while; depending on the machine type you used, it might > be possible to load that image on an earlier (or newer) qemu > and try the same test, however if the problem doesn't > repeat reliably it can be hard. I've seen this first with Qemu 1.5 but was not able to reproduce it for month. 1.4 was working fine. > If you have any way of simplifying the configuration of the > VM it would be good; e.g. if you could get a failure on > something without graphics (-nographic) and USB. Sadly not ;-( > Dave > >> >> Stefan >> >> Am 07.02.2014 14:45, schrieb Stefan Priebe - Profihost AG: >>> it's always the same "pattern" there are too many 0 instead of X. >>> >>> only seen: >>> >>> read:0x0000000000000000 ... expected:0xffffffffffffffff >>> >>> or >>> >>> read:0xffffffff00000000 ... expected:0xffffffffffffffff >>> >>> or >>> >>> read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff >>> >>> or >>> >>> read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5 >>> >>> no idea if this helps. >>> >>> Stefan >>> >>> Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG: >>>> Hi, >>>> Am 07.02.2014 14:19, schrieb Paolo Bonzini: >>>>> Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto: >>>>>> first of all i've now a memory image of a VM where i can reproduce it. >>>>> >>>>> You mean you start that VM with -incoming 'exec:cat /path/to/vm.img'? >>>>> But google stress test doesn't report any error until you start >>>>> migration _and_ it finishes? >>>> >>>> Sorry no i meant i have a VM where i saved the memory to disk - so i >>>> don't need to wait hours until i can reproduce as it does not happen >>>> with a fresh started VM. So it's a state file i think. >>>> >>>>> Another test: >>>>> >>>>> - start the VM with -S, migrate, do errors appear on the destination? >>>> >>>> I started with -S and the errors appear AFTER resuming/unpause the VM. >>>> So it is fine until i resume it on the "new" host. >>>> >>>> Stefan >>>> >> > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 20:10 ` Stefan Priebe @ 2014-02-08 19:23 ` Stefan Priebe 2014-02-10 9:30 ` Dr. David Alan Gilbert ` (2 more replies) 0 siblings, 3 replies; 50+ messages in thread From: Stefan Priebe @ 2014-02-08 19:23 UTC (permalink / raw) To: Dr. David Alan Gilbert; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER i could fix it by explicitly disable xbzrle - it seems its automatically on if i do not set the migration caps to false. So it seems to be a xbzrle bug. Stefan Am 07.02.2014 21:10, schrieb Stefan Priebe: > Am 07.02.2014 21:02, schrieb Dr. David Alan Gilbert: >> * Stefan Priebe (s.priebe@profihost.ag) wrote: >>> anything i could try or debug? to help to find the problem? >> >> I think the most useful would be to see if the problem is >> a new problem in the 1.7 you're using or has existed >> for a while; depending on the machine type you used, it might >> be possible to load that image on an earlier (or newer) qemu >> and try the same test, however if the problem doesn't >> repeat reliably it can be hard. > > I've seen this first with Qemu 1.5 but was not able to reproduce it for > month. 1.4 was working fine. > >> If you have any way of simplifying the configuration of the >> VM it would be good; e.g. if you could get a failure on >> something without graphics (-nographic) and USB. > > Sadly not ;-( > >> Dave >> >>> >>> Stefan >>> >>> Am 07.02.2014 14:45, schrieb Stefan Priebe - Profihost AG: >>>> it's always the same "pattern" there are too many 0 instead of X. >>>> >>>> only seen: >>>> >>>> read:0x0000000000000000 ... expected:0xffffffffffffffff >>>> >>>> or >>>> >>>> read:0xffffffff00000000 ... expected:0xffffffffffffffff >>>> >>>> or >>>> >>>> read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff >>>> >>>> or >>>> >>>> read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5 >>>> >>>> no idea if this helps. >>>> >>>> Stefan >>>> >>>> Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG: >>>>> Hi, >>>>> Am 07.02.2014 14:19, schrieb Paolo Bonzini: >>>>>> Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto: >>>>>>> first of all i've now a memory image of a VM where i can >>>>>>> reproduce it. >>>>>> >>>>>> You mean you start that VM with -incoming 'exec:cat /path/to/vm.img'? >>>>>> But google stress test doesn't report any error until you start >>>>>> migration _and_ it finishes? >>>>> >>>>> Sorry no i meant i have a VM where i saved the memory to disk - so i >>>>> don't need to wait hours until i can reproduce as it does not happen >>>>> with a fresh started VM. So it's a state file i think. >>>>> >>>>>> Another test: >>>>>> >>>>>> - start the VM with -S, migrate, do errors appear on the destination? >>>>> >>>>> I started with -S and the errors appear AFTER resuming/unpause the VM. >>>>> So it is fine until i resume it on the "new" host. >>>>> >>>>> Stefan >>>>> >>> >> -- >> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >> ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-08 19:23 ` Stefan Priebe @ 2014-02-10 9:30 ` Dr. David Alan Gilbert 2014-02-10 16:07 ` Dr. David Alan Gilbert 2014-02-11 13:32 ` Orit Wasserman 2 siblings, 0 replies; 50+ messages in thread From: Dr. David Alan Gilbert @ 2014-02-10 9:30 UTC (permalink / raw) To: Stefan Priebe; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER * Stefan Priebe (s.priebe@profihost.ag) wrote: > i could fix it by explicitly disable xbzrle - it seems its > automatically on if i do not set the migration caps to false. > > So it seems to be a xbzrle bug. Ah right, yes that would make sense for the type of errors you're seeing, and does make it easier to tie down. Dave > > Stefan > > Am 07.02.2014 21:10, schrieb Stefan Priebe: > >Am 07.02.2014 21:02, schrieb Dr. David Alan Gilbert: > >>* Stefan Priebe (s.priebe@profihost.ag) wrote: > >>>anything i could try or debug? to help to find the problem? > >> > >>I think the most useful would be to see if the problem is > >>a new problem in the 1.7 you're using or has existed > >>for a while; depending on the machine type you used, it might > >>be possible to load that image on an earlier (or newer) qemu > >>and try the same test, however if the problem doesn't > >>repeat reliably it can be hard. > > > >I've seen this first with Qemu 1.5 but was not able to reproduce it for > >month. 1.4 was working fine. > > > >>If you have any way of simplifying the configuration of the > >>VM it would be good; e.g. if you could get a failure on > >>something without graphics (-nographic) and USB. > > > >Sadly not ;-( > > > >>Dave > >> > >>> > >>>Stefan > >>> > >>>Am 07.02.2014 14:45, schrieb Stefan Priebe - Profihost AG: > >>>>it's always the same "pattern" there are too many 0 instead of X. > >>>> > >>>>only seen: > >>>> > >>>>read:0x0000000000000000 ... expected:0xffffffffffffffff > >>>> > >>>>or > >>>> > >>>>read:0xffffffff00000000 ... expected:0xffffffffffffffff > >>>> > >>>>or > >>>> > >>>>read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff > >>>> > >>>>or > >>>> > >>>>read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5 > >>>> > >>>>no idea if this helps. > >>>> > >>>>Stefan > >>>> > >>>>Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG: > >>>>>Hi, > >>>>>Am 07.02.2014 14:19, schrieb Paolo Bonzini: > >>>>>>Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto: > >>>>>>>first of all i've now a memory image of a VM where i can > >>>>>>>reproduce it. > >>>>>> > >>>>>>You mean you start that VM with -incoming 'exec:cat /path/to/vm.img'? > >>>>>>But google stress test doesn't report any error until you start > >>>>>>migration _and_ it finishes? > >>>>> > >>>>>Sorry no i meant i have a VM where i saved the memory to disk - so i > >>>>>don't need to wait hours until i can reproduce as it does not happen > >>>>>with a fresh started VM. So it's a state file i think. > >>>>> > >>>>>>Another test: > >>>>>> > >>>>>>- start the VM with -S, migrate, do errors appear on the destination? > >>>>> > >>>>>I started with -S and the errors appear AFTER resuming/unpause the VM. > >>>>>So it is fine until i resume it on the "new" host. > >>>>> > >>>>>Stefan > >>>>> > >>> > >>-- > >>Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > >> -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-08 19:23 ` Stefan Priebe 2014-02-10 9:30 ` Dr. David Alan Gilbert @ 2014-02-10 16:07 ` Dr. David Alan Gilbert 2014-02-10 18:53 ` Stefan Priebe 2014-02-11 13:32 ` Orit Wasserman 2 siblings, 1 reply; 50+ messages in thread From: Dr. David Alan Gilbert @ 2014-02-10 16:07 UTC (permalink / raw) To: Stefan Priebe; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER * Stefan Priebe (s.priebe@profihost.ag) wrote: > i could fix it by explicitly disable xbzrle - it seems its > automatically on if i do not set the migration caps to false. > > So it seems to be a xbzrle bug. Stefan can you give me some more info on your hardware and migration setup; that stressapptest (which is a really nice find!) really batters the memory and it means the migration isn't converging for me, so I'm curious what your setup is. What CPU have you got? How many cores are you giving each guest? What network technology are you migrating over? Other than xbzrle what else do you have enabled? How long is the migrate taking for you? Thanks, Dave -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-10 16:07 ` Dr. David Alan Gilbert @ 2014-02-10 18:53 ` Stefan Priebe 2014-02-13 20:06 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 50+ messages in thread From: Stefan Priebe @ 2014-02-10 18:53 UTC (permalink / raw) To: Dr. David Alan Gilbert; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER Am 10.02.2014 17:07, schrieb Dr. David Alan Gilbert: > * Stefan Priebe (s.priebe@profihost.ag) wrote: >> i could fix it by explicitly disable xbzrle - it seems its >> automatically on if i do not set the migration caps to false. >> >> So it seems to be a xbzrle bug. > > Stefan can you give me some more info on your hardware and > migration setup; that stressapptest (which is a really nice > find!) really batters the memory and it means the migration > isn't converging for me, so I'm curious what your setup is. That one is devlopment by google and known to me since a few years. Google has detected that memtest and co are not good enough to stress test memory. > What CPU have you got? Dual Xeon E5-2695v2 > How many cores are you giving each guest? 16 > What network technology are you migrating over? 10Gb/s > Other than xbzrle what else do you have enabled? nothing > How long is the migrate taking for you? with migration_downtime => 4s around 10s Stefan > Thanks, > > Dave > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-10 18:53 ` Stefan Priebe @ 2014-02-13 20:06 ` Dr. David Alan Gilbert 2014-02-13 20:26 ` Stefan Priebe ` (2 more replies) 0 siblings, 3 replies; 50+ messages in thread From: Dr. David Alan Gilbert @ 2014-02-13 20:06 UTC (permalink / raw) To: Stefan Priebe; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER, owasserm * Stefan Priebe (s.priebe@profihost.ag) wrote: > Am 10.02.2014 17:07, schrieb Dr. David Alan Gilbert: > >* Stefan Priebe (s.priebe@profihost.ag) wrote: > >>i could fix it by explicitly disable xbzrle - it seems its > >>automatically on if i do not set the migration caps to false. > >> > >>So it seems to be a xbzrle bug. > > > >Stefan can you give me some more info on your hardware and > >migration setup; that stressapptest (which is a really nice > >find!) really batters the memory and it means the migration > >isn't converging for me, so I'm curious what your setup is. > > That one is devlopment by google and known to me since a few years. > Google has detected that memtest and co are not good enough to > stress test memory. Hi Stefan, I've just posted a patch to qemu-devel that fixes two bugs that we found; I've only tried a small stressapptest run and it seems to survive with them (where it didn't before); you might like to try it if you're up for rebuilding qemu. It's the one entitled ' [PATCH] Fix two XBZRLE corruption issues' I'll try and get a larger run done myself, but I'd be interested to hear if it fixes it for you (or anyone else who hit the problem). Dave -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-13 20:06 ` Dr. David Alan Gilbert @ 2014-02-13 20:26 ` Stefan Priebe 2014-02-13 20:31 ` Stefan Priebe 2014-02-13 21:18 ` Stefan Priebe 2 siblings, 0 replies; 50+ messages in thread From: Stefan Priebe @ 2014-02-13 20:26 UTC (permalink / raw) To: Dr. David Alan Gilbert Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER, owasserm Am 13.02.2014 21:06, schrieb Dr. David Alan Gilbert: > * Stefan Priebe (s.priebe@profihost.ag) wrote: >> Am 10.02.2014 17:07, schrieb Dr. David Alan Gilbert: >>> * Stefan Priebe (s.priebe@profihost.ag) wrote: >>>> i could fix it by explicitly disable xbzrle - it seems its >>>> automatically on if i do not set the migration caps to false. >>>> >>>> So it seems to be a xbzrle bug. >>> >>> Stefan can you give me some more info on your hardware and >>> migration setup; that stressapptest (which is a really nice >>> find!) really batters the memory and it means the migration >>> isn't converging for me, so I'm curious what your setup is. >> >> That one is devlopment by google and known to me since a few years. >> Google has detected that memtest and co are not good enough to >> stress test memory. > > Hi Stefan, > I've just posted a patch to qemu-devel that fixes two bugs that > we found; I've only tried a small stressapptest run and it seems > to survive with them (where it didn't before); you might like to try > it if you're up for rebuilding qemu. > > It's the one entitled ' [PATCH] Fix two XBZRLE corruption issues' Thanks! Really would love to try them but nor google nor myself can find them. http://osdir.com/ml/qemu-devel/2014-02/ Stefan > I'll try and get a larger run done myself, but I'd be interested to > hear if it fixes it for you (or anyone else who hit the problem). > > Dave > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-13 20:06 ` Dr. David Alan Gilbert 2014-02-13 20:26 ` Stefan Priebe @ 2014-02-13 20:31 ` Stefan Priebe 2014-02-13 21:18 ` Stefan Priebe 2 siblings, 0 replies; 50+ messages in thread From: Stefan Priebe @ 2014-02-13 20:31 UTC (permalink / raw) To: Dr. David Alan Gilbert Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER, owasserm got it here: http://lists.nongnu.org/archive/html/qemu-devel/2014-02/msg02341.html will try asap Am 13.02.2014 21:06, schrieb Dr. David Alan Gilbert: > * Stefan Priebe (s.priebe@profihost.ag) wrote: >> Am 10.02.2014 17:07, schrieb Dr. David Alan Gilbert: >>> * Stefan Priebe (s.priebe@profihost.ag) wrote: >>>> i could fix it by explicitly disable xbzrle - it seems its >>>> automatically on if i do not set the migration caps to false. >>>> >>>> So it seems to be a xbzrle bug. >>> >>> Stefan can you give me some more info on your hardware and >>> migration setup; that stressapptest (which is a really nice >>> find!) really batters the memory and it means the migration >>> isn't converging for me, so I'm curious what your setup is. >> >> That one is devlopment by google and known to me since a few years. >> Google has detected that memtest and co are not good enough to >> stress test memory. > > Hi Stefan, > I've just posted a patch to qemu-devel that fixes two bugs that > we found; I've only tried a small stressapptest run and it seems > to survive with them (where it didn't before); you might like to try > it if you're up for rebuilding qemu. > > It's the one entitled ' [PATCH] Fix two XBZRLE corruption issues' > > I'll try and get a larger run done myself, but I'd be interested to > hear if it fixes it for you (or anyone else who hit the problem). > > Dave > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-13 20:06 ` Dr. David Alan Gilbert 2014-02-13 20:26 ` Stefan Priebe 2014-02-13 20:31 ` Stefan Priebe @ 2014-02-13 21:18 ` Stefan Priebe 2014-02-14 9:06 ` Dr. David Alan Gilbert 2 siblings, 1 reply; 50+ messages in thread From: Stefan Priebe @ 2014-02-13 21:18 UTC (permalink / raw) To: Dr. David Alan Gilbert Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER, owasserm Am 13.02.2014 21:06, schrieb Dr. David Alan Gilbert: > * Stefan Priebe (s.priebe@profihost.ag) wrote: >> Am 10.02.2014 17:07, schrieb Dr. David Alan Gilbert: >>> * Stefan Priebe (s.priebe@profihost.ag) wrote: >>>> i could fix it by explicitly disable xbzrle - it seems its >>>> automatically on if i do not set the migration caps to false. >>>> >>>> So it seems to be a xbzrle bug. >>> >>> Stefan can you give me some more info on your hardware and >>> migration setup; that stressapptest (which is a really nice >>> find!) really batters the memory and it means the migration >>> isn't converging for me, so I'm curious what your setup is. >> >> That one is devlopment by google and known to me since a few years. >> Google has detected that memtest and co are not good enough to >> stress test memory. > > Hi Stefan, > I've just posted a patch to qemu-devel that fixes two bugs that > we found; I've only tried a small stressapptest run and it seems > to survive with them (where it didn't before); you might like to try > it if you're up for rebuilding qemu. > > It's the one entitled ' [PATCH] Fix two XBZRLE corruption issues' > > I'll try and get a larger run done myself, but I'd be interested to > hear if it fixes it for you (or anyone else who hit the problem). Yes works fine - now no crash but it's sower than without XBZRLE ;-) Without XBZRLE: i needed migrate_downtime 4 around 60s With XBZRLE: i needed migrate_downtime 16 and 240s > Dave > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-13 21:18 ` Stefan Priebe @ 2014-02-14 9:06 ` Dr. David Alan Gilbert 0 siblings, 0 replies; 50+ messages in thread From: Dr. David Alan Gilbert @ 2014-02-14 9:06 UTC (permalink / raw) To: Stefan Priebe; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER, owasserm * Stefan Priebe (s.priebe@profihost.ag) wrote: > > Am 13.02.2014 21:06, schrieb Dr. David Alan Gilbert: > >* Stefan Priebe (s.priebe@profihost.ag) wrote: > >>Am 10.02.2014 17:07, schrieb Dr. David Alan Gilbert: > >>>* Stefan Priebe (s.priebe@profihost.ag) wrote: > >>>>i could fix it by explicitly disable xbzrle - it seems its > >>>>automatically on if i do not set the migration caps to false. > >>>> > >>>>So it seems to be a xbzrle bug. > >>> > >>>Stefan can you give me some more info on your hardware and > >>>migration setup; that stressapptest (which is a really nice > >>>find!) really batters the memory and it means the migration > >>>isn't converging for me, so I'm curious what your setup is. > >> > >>That one is devlopment by google and known to me since a few years. > >>Google has detected that memtest and co are not good enough to > >>stress test memory. > > > >Hi Stefan, > > I've just posted a patch to qemu-devel that fixes two bugs that > >we found; I've only tried a small stressapptest run and it seems > >to survive with them (where it didn't before); you might like to try > >it if you're up for rebuilding qemu. > > > >It's the one entitled ' [PATCH] Fix two XBZRLE corruption issues' > > > >I'll try and get a larger run done myself, but I'd be interested to > >hear if it fixes it for you (or anyone else who hit the problem). > > Yes works fine - now no crash but it's sower than without XBZRLE ;-) > > Without XBZRLE: i needed migrate_downtime 4 around 60s > With XBZRLE: i needed migrate_downtime 16 and 240s Hmm; how did that compare with the previous (broken) with XBZRLE time? (i.e. was XBZRLE always slower for you?) If you're driving this from the hmp/command interface then the result of the info migrate command at the end of each of those runs would be interesting. Another thing you could try is changing the xbzrle_cache_zero_page in arch_init.c that I added so it reads as: static void xbzrle_cache_zero_page(ram_addr_t current_addr) { if (ram_bulk_stage || !migrate_use_xbzrle()) { return; } if (!cache_is_cached(XBZRLE.cache, current_addr)) { return; } /* We don't care if this fails to allocate a new cache page * as long as it updated an old one */ cache_insert(XBZRLE.cache, current_addr, ZERO_TARGET_PAGE); } Dave -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-08 19:23 ` Stefan Priebe 2014-02-10 9:30 ` Dr. David Alan Gilbert 2014-02-10 16:07 ` Dr. David Alan Gilbert @ 2014-02-11 13:32 ` Orit Wasserman 2014-02-11 13:33 ` Stefan Priebe - Profihost AG 2 siblings, 1 reply; 50+ messages in thread From: Orit Wasserman @ 2014-02-11 13:32 UTC (permalink / raw) To: Stefan Priebe, Dr. David Alan Gilbert Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER On 02/08/2014 09:23 PM, Stefan Priebe wrote: > i could fix it by explicitly disable xbzrle - it seems its automatically on if i do not set the migration caps to false. > > So it seems to be a xbzrle bug. > XBZRLE is disabled by default (actually all capabilities are off by default) What version of QEMU are you using that you need to disable it explicitly? Maybe you run migration with XBZRLE and canceled it, so it stays on? Orit > Stefan > > Am 07.02.2014 21:10, schrieb Stefan Priebe: >> Am 07.02.2014 21:02, schrieb Dr. David Alan Gilbert: >>> * Stefan Priebe (s.priebe@profihost.ag) wrote: >>>> anything i could try or debug? to help to find the problem? >>> >>> I think the most useful would be to see if the problem is >>> a new problem in the 1.7 you're using or has existed >>> for a while; depending on the machine type you used, it might >>> be possible to load that image on an earlier (or newer) qemu >>> and try the same test, however if the problem doesn't >>> repeat reliably it can be hard. >> >> I've seen this first with Qemu 1.5 but was not able to reproduce it for >> month. 1.4 was working fine. >> >>> If you have any way of simplifying the configuration of the >>> VM it would be good; e.g. if you could get a failure on >>> something without graphics (-nographic) and USB. >> >> Sadly not ;-( >> >>> Dave >>> >>>> >>>> Stefan >>>> >>>> Am 07.02.2014 14:45, schrieb Stefan Priebe - Profihost AG: >>>>> it's always the same "pattern" there are too many 0 instead of X. >>>>> >>>>> only seen: >>>>> >>>>> read:0x0000000000000000 ... expected:0xffffffffffffffff >>>>> >>>>> or >>>>> >>>>> read:0xffffffff00000000 ... expected:0xffffffffffffffff >>>>> >>>>> or >>>>> >>>>> read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff >>>>> >>>>> or >>>>> >>>>> read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5 >>>>> >>>>> no idea if this helps. >>>>> >>>>> Stefan >>>>> >>>>> Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG: >>>>>> Hi, >>>>>> Am 07.02.2014 14:19, schrieb Paolo Bonzini: >>>>>>> Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto: >>>>>>>> first of all i've now a memory image of a VM where i can >>>>>>>> reproduce it. >>>>>>> >>>>>>> You mean you start that VM with -incoming 'exec:cat /path/to/vm.img'? >>>>>>> But google stress test doesn't report any error until you start >>>>>>> migration _and_ it finishes? >>>>>> >>>>>> Sorry no i meant i have a VM where i saved the memory to disk - so i >>>>>> don't need to wait hours until i can reproduce as it does not happen >>>>>> with a fresh started VM. So it's a state file i think. >>>>>> >>>>>>> Another test: >>>>>>> >>>>>>> - start the VM with -S, migrate, do errors appear on the destination? >>>>>> >>>>>> I started with -S and the errors appear AFTER resuming/unpause the VM. >>>>>> So it is fine until i resume it on the "new" host. >>>>>> >>>>>> Stefan >>>>>> >>>> >>> -- >>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >>> > ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-11 13:32 ` Orit Wasserman @ 2014-02-11 13:33 ` Stefan Priebe - Profihost AG 2014-02-11 13:45 ` Orit Wasserman 0 siblings, 1 reply; 50+ messages in thread From: Stefan Priebe - Profihost AG @ 2014-02-11 13:33 UTC (permalink / raw) To: Orit Wasserman, Dr. David Alan Gilbert Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER Am 11.02.2014 14:32, schrieb Orit Wasserman: > On 02/08/2014 09:23 PM, Stefan Priebe wrote: >> i could fix it by explicitly disable xbzrle - it seems its >> automatically on if i do not set the migration caps to false. >> >> So it seems to be a xbzrle bug. >> > > XBZRLE is disabled by default (actually all capabilities are off by > default) > What version of QEMU are you using that you need to disable it explicitly? > Maybe you run migration with XBZRLE and canceled it, so it stays on? No real idea why this happens - but yes this seems to be a problem for me. But the bug in XBZRLE is still there ;-) Stefan > Orit > >> Stefan >> >> Am 07.02.2014 21:10, schrieb Stefan Priebe: >>> Am 07.02.2014 21:02, schrieb Dr. David Alan Gilbert: >>>> * Stefan Priebe (s.priebe@profihost.ag) wrote: >>>>> anything i could try or debug? to help to find the problem? >>>> >>>> I think the most useful would be to see if the problem is >>>> a new problem in the 1.7 you're using or has existed >>>> for a while; depending on the machine type you used, it might >>>> be possible to load that image on an earlier (or newer) qemu >>>> and try the same test, however if the problem doesn't >>>> repeat reliably it can be hard. >>> >>> I've seen this first with Qemu 1.5 but was not able to reproduce it for >>> month. 1.4 was working fine. >>> >>>> If you have any way of simplifying the configuration of the >>>> VM it would be good; e.g. if you could get a failure on >>>> something without graphics (-nographic) and USB. >>> >>> Sadly not ;-( >>> >>>> Dave >>>> >>>>> >>>>> Stefan >>>>> >>>>> Am 07.02.2014 14:45, schrieb Stefan Priebe - Profihost AG: >>>>>> it's always the same "pattern" there are too many 0 instead of X. >>>>>> >>>>>> only seen: >>>>>> >>>>>> read:0x0000000000000000 ... expected:0xffffffffffffffff >>>>>> >>>>>> or >>>>>> >>>>>> read:0xffffffff00000000 ... expected:0xffffffffffffffff >>>>>> >>>>>> or >>>>>> >>>>>> read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff >>>>>> >>>>>> or >>>>>> >>>>>> read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5 >>>>>> >>>>>> no idea if this helps. >>>>>> >>>>>> Stefan >>>>>> >>>>>> Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG: >>>>>>> Hi, >>>>>>> Am 07.02.2014 14:19, schrieb Paolo Bonzini: >>>>>>>> Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto: >>>>>>>>> first of all i've now a memory image of a VM where i can >>>>>>>>> reproduce it. >>>>>>>> >>>>>>>> You mean you start that VM with -incoming 'exec:cat >>>>>>>> /path/to/vm.img'? >>>>>>>> But google stress test doesn't report any error until you start >>>>>>>> migration _and_ it finishes? >>>>>>> >>>>>>> Sorry no i meant i have a VM where i saved the memory to disk - so i >>>>>>> don't need to wait hours until i can reproduce as it does not happen >>>>>>> with a fresh started VM. So it's a state file i think. >>>>>>> >>>>>>>> Another test: >>>>>>>> >>>>>>>> - start the VM with -S, migrate, do errors appear on the >>>>>>>> destination? >>>>>>> >>>>>>> I started with -S and the errors appear AFTER resuming/unpause >>>>>>> the VM. >>>>>>> So it is fine until i resume it on the "new" host. >>>>>>> >>>>>>> Stefan >>>>>>> >>>>> >>>> -- >>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >>>> >> > ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-11 13:33 ` Stefan Priebe - Profihost AG @ 2014-02-11 13:45 ` Orit Wasserman 2014-02-11 14:49 ` Stefan Priebe - Profihost AG 0 siblings, 1 reply; 50+ messages in thread From: Orit Wasserman @ 2014-02-11 13:45 UTC (permalink / raw) To: Stefan Priebe - Profihost AG, Dr. David Alan Gilbert Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER On 02/11/2014 03:33 PM, Stefan Priebe - Profihost AG wrote: > > Am 11.02.2014 14:32, schrieb Orit Wasserman: >> On 02/08/2014 09:23 PM, Stefan Priebe wrote: >>> i could fix it by explicitly disable xbzrle - it seems its >>> automatically on if i do not set the migration caps to false. >>> >>> So it seems to be a xbzrle bug. >>> >> >> XBZRLE is disabled by default (actually all capabilities are off by >> default) >> What version of QEMU are you using that you need to disable it explicitly? >> Maybe you run migration with XBZRLE and canceled it, so it stays on? > > No real idea why this happens - but yes this seems to be a problem for me. > I checked upstream QEMU and it is still off by default (always been) > But the bug in XBZRLE is still there ;-) > We need to understand the exact scenario in order to understand the problem. What exact version of Qemu are you using? Can you try with the latest upstream version, there were some fixes to the XBZRLE code? > Stefan > >> Orit >> >>> Stefan >>> >>> Am 07.02.2014 21:10, schrieb Stefan Priebe: >>>> Am 07.02.2014 21:02, schrieb Dr. David Alan Gilbert: >>>>> * Stefan Priebe (s.priebe@profihost.ag) wrote: >>>>>> anything i could try or debug? to help to find the problem? >>>>> >>>>> I think the most useful would be to see if the problem is >>>>> a new problem in the 1.7 you're using or has existed >>>>> for a while; depending on the machine type you used, it might >>>>> be possible to load that image on an earlier (or newer) qemu >>>>> and try the same test, however if the problem doesn't >>>>> repeat reliably it can be hard. >>>> >>>> I've seen this first with Qemu 1.5 but was not able to reproduce it for >>>> month. 1.4 was working fine. >>>> >>>>> If you have any way of simplifying the configuration of the >>>>> VM it would be good; e.g. if you could get a failure on >>>>> something without graphics (-nographic) and USB. >>>> >>>> Sadly not ;-( >>>> >>>>> Dave >>>>> >>>>>> >>>>>> Stefan >>>>>> >>>>>> Am 07.02.2014 14:45, schrieb Stefan Priebe - Profihost AG: >>>>>>> it's always the same "pattern" there are too many 0 instead of X. >>>>>>> >>>>>>> only seen: >>>>>>> >>>>>>> read:0x0000000000000000 ... expected:0xffffffffffffffff >>>>>>> >>>>>>> or >>>>>>> >>>>>>> read:0xffffffff00000000 ... expected:0xffffffffffffffff >>>>>>> >>>>>>> or >>>>>>> >>>>>>> read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff >>>>>>> >>>>>>> or >>>>>>> >>>>>>> read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5 >>>>>>> >>>>>>> no idea if this helps. >>>>>>> >>>>>>> Stefan >>>>>>> >>>>>>> Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG: >>>>>>>> Hi, >>>>>>>> Am 07.02.2014 14:19, schrieb Paolo Bonzini: >>>>>>>>> Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto: >>>>>>>>>> first of all i've now a memory image of a VM where i can >>>>>>>>>> reproduce it. >>>>>>>>> >>>>>>>>> You mean you start that VM with -incoming 'exec:cat >>>>>>>>> /path/to/vm.img'? >>>>>>>>> But google stress test doesn't report any error until you start >>>>>>>>> migration _and_ it finishes? >>>>>>>> >>>>>>>> Sorry no i meant i have a VM where i saved the memory to disk - so i >>>>>>>> don't need to wait hours until i can reproduce as it does not happen >>>>>>>> with a fresh started VM. So it's a state file i think. >>>>>>>> >>>>>>>>> Another test: >>>>>>>>> >>>>>>>>> - start the VM with -S, migrate, do errors appear on the >>>>>>>>> destination? >>>>>>>> >>>>>>>> I started with -S and the errors appear AFTER resuming/unpause >>>>>>>> the VM. >>>>>>>> So it is fine until i resume it on the "new" host. >>>>>>>> >>>>>>>> Stefan >>>>>>>> >>>>>> >>>>> -- >>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >>>>> >>> >> ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-11 13:45 ` Orit Wasserman @ 2014-02-11 14:49 ` Stefan Priebe - Profihost AG 0 siblings, 0 replies; 50+ messages in thread From: Stefan Priebe - Profihost AG @ 2014-02-11 14:49 UTC (permalink / raw) To: Orit Wasserman, Dr. David Alan Gilbert Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER Am 11.02.2014 14:45, schrieb Orit Wasserman: > On 02/11/2014 03:33 PM, Stefan Priebe - Profihost AG wrote: >> >> Am 11.02.2014 14:32, schrieb Orit Wasserman: >>> On 02/08/2014 09:23 PM, Stefan Priebe wrote: >>>> i could fix it by explicitly disable xbzrle - it seems its >>>> automatically on if i do not set the migration caps to false. >>>> >>>> So it seems to be a xbzrle bug. >>>> >>> >>> XBZRLE is disabled by default (actually all capabilities are off by >>> default) >>> What version of QEMU are you using that you need to disable it >>> explicitly? >>> Maybe you run migration with XBZRLE and canceled it, so it stays on? >> >> No real idea why this happens - but yes this seems to be a problem for >> me. >> > > I checked upstream QEMU and it is still off by default (always been) May be i had it on in the past and the VM was still running from an older migration. >> But the bug in XBZRLE is still there ;-) >> > > We need to understand the exact scenario in order to understand the > problem. > > What exact version of Qemu are you using? Qemu 1.7.0 > Can you try with the latest upstream version, there were some fixes to the > XBZRLE code? Sadly not - i have some custom patches (not related to xbzrle) which won't apply to current upstream. But i could cherry-pick the ones you have in mind - if you give me the commit ids. Stefan >> Stefan >> >>> Orit >>> >>>> Stefan >>>> >>>> Am 07.02.2014 21:10, schrieb Stefan Priebe: >>>>> Am 07.02.2014 21:02, schrieb Dr. David Alan Gilbert: >>>>>> * Stefan Priebe (s.priebe@profihost.ag) wrote: >>>>>>> anything i could try or debug? to help to find the problem? >>>>>> >>>>>> I think the most useful would be to see if the problem is >>>>>> a new problem in the 1.7 you're using or has existed >>>>>> for a while; depending on the machine type you used, it might >>>>>> be possible to load that image on an earlier (or newer) qemu >>>>>> and try the same test, however if the problem doesn't >>>>>> repeat reliably it can be hard. >>>>> >>>>> I've seen this first with Qemu 1.5 but was not able to reproduce it >>>>> for >>>>> month. 1.4 was working fine. >>>>> >>>>>> If you have any way of simplifying the configuration of the >>>>>> VM it would be good; e.g. if you could get a failure on >>>>>> something without graphics (-nographic) and USB. >>>>> >>>>> Sadly not ;-( >>>>> >>>>>> Dave >>>>>> >>>>>>> >>>>>>> Stefan >>>>>>> >>>>>>> Am 07.02.2014 14:45, schrieb Stefan Priebe - Profihost AG: >>>>>>>> it's always the same "pattern" there are too many 0 instead of X. >>>>>>>> >>>>>>>> only seen: >>>>>>>> >>>>>>>> read:0x0000000000000000 ... expected:0xffffffffffffffff >>>>>>>> >>>>>>>> or >>>>>>>> >>>>>>>> read:0xffffffff00000000 ... expected:0xffffffffffffffff >>>>>>>> >>>>>>>> or >>>>>>>> >>>>>>>> read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff >>>>>>>> >>>>>>>> or >>>>>>>> >>>>>>>> read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5 >>>>>>>> >>>>>>>> no idea if this helps. >>>>>>>> >>>>>>>> Stefan >>>>>>>> >>>>>>>> Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG: >>>>>>>>> Hi, >>>>>>>>> Am 07.02.2014 14:19, schrieb Paolo Bonzini: >>>>>>>>>> Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto: >>>>>>>>>>> first of all i've now a memory image of a VM where i can >>>>>>>>>>> reproduce it. >>>>>>>>>> >>>>>>>>>> You mean you start that VM with -incoming 'exec:cat >>>>>>>>>> /path/to/vm.img'? >>>>>>>>>> But google stress test doesn't report any error until you start >>>>>>>>>> migration _and_ it finishes? >>>>>>>>> >>>>>>>>> Sorry no i meant i have a VM where i saved the memory to disk - >>>>>>>>> so i >>>>>>>>> don't need to wait hours until i can reproduce as it does not >>>>>>>>> happen >>>>>>>>> with a fresh started VM. So it's a state file i think. >>>>>>>>> >>>>>>>>>> Another test: >>>>>>>>>> >>>>>>>>>> - start the VM with -S, migrate, do errors appear on the >>>>>>>>>> destination? >>>>>>>>> >>>>>>>>> I started with -S and the errors appear AFTER resuming/unpause >>>>>>>>> the VM. >>>>>>>>> So it is fine until i resume it on the "new" host. >>>>>>>>> >>>>>>>>> Stefan >>>>>>>>> >>>>>>> >>>>>> -- >>>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >>>>>> >>>> >>> > ^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 2014-02-07 9:31 ` Dr. David Alan Gilbert 2014-02-07 9:37 ` Stefan Priebe - Profihost AG @ 2014-02-07 9:59 ` Marcin Gibuła 1 sibling, 0 replies; 50+ messages in thread From: Marcin Gibuła @ 2014-02-07 9:59 UTC (permalink / raw) To: Dr. David Alan Gilbert, Stefan Priebe - Profihost AG Cc: qemu-devel, Alexandre DERUMIER >> You mean to reproduce? > > I'm more interested in seeing what type of corruption is happening; > if you've got a test VM that corrupts memory and we can run a program > in that vm that writes a known pattern into memory and checks it > then see what changed after migration, it might give a clue. > > But obviously this would only be of any use if run on the VM that actually > fails. Hi, Seeing similar issue in my company I would be happy to run such tests. Do you have any test suite I could run or some leads how to write it? -- mg ^ permalink raw reply [flat|nested] 50+ messages in thread
end of thread, other threads:[~2014-02-14 9:06 UTC | newest] Thread overview: 50+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-02-05 17:51 [Qemu-devel] QEMU LIve Migration - swap_free: Bad swap file entry Stefan Priebe 2014-02-05 20:15 ` Dr. David Alan Gilbert 2014-02-06 7:20 ` Stefan Priebe - Profihost AG 2014-02-06 10:22 ` Orit Wasserman 2014-02-06 10:49 ` Stefan Priebe - Profihost AG 2014-02-06 11:14 ` [Qemu-devel] [pve-devel] " Alexandre DERUMIER 2014-02-06 11:19 ` Stefan Priebe - Profihost AG 2014-02-06 11:40 ` Alexandre DERUMIER 2014-02-06 12:10 ` Stefan Priebe - Profihost AG 2014-02-06 14:03 ` Stefan Priebe - Profihost AG 2014-02-06 14:12 ` Marcin Gibuła 2014-02-06 19:51 ` Dr. David Alan Gilbert 2014-02-06 20:00 ` Stefan Priebe 2014-02-07 8:15 ` Alexandre DERUMIER 2014-02-07 8:17 ` Stefan Priebe - Profihost AG 2014-02-07 9:15 ` Dr. David Alan Gilbert 2014-02-07 9:20 ` Stefan Priebe - Profihost AG 2014-02-07 9:29 ` Marcin Gibuła 2014-02-07 9:30 ` Stefan Priebe - Profihost AG 2014-02-07 9:31 ` Dr. David Alan Gilbert 2014-02-07 9:37 ` Stefan Priebe - Profihost AG 2014-02-07 12:02 ` Stefan Priebe - Profihost AG 2014-02-07 12:21 ` Dr. David Alan Gilbert 2014-02-07 12:30 ` Stefan Priebe - Profihost AG 2014-02-07 12:44 ` Paolo Bonzini 2014-02-07 13:04 ` Stefan Priebe - Profihost AG 2014-02-07 13:08 ` Dr. David Alan Gilbert 2014-02-07 13:10 ` Stefan Priebe - Profihost AG 2014-02-07 13:15 ` Dr. David Alan Gilbert 2014-02-07 13:21 ` Stefan Priebe - Profihost AG 2014-02-07 13:19 ` Paolo Bonzini 2014-02-07 13:39 ` Stefan Priebe - Profihost AG 2014-02-07 13:45 ` Stefan Priebe - Profihost AG 2014-02-07 19:21 ` Stefan Priebe 2014-02-07 20:02 ` Dr. David Alan Gilbert 2014-02-07 20:10 ` Stefan Priebe 2014-02-08 19:23 ` Stefan Priebe 2014-02-10 9:30 ` Dr. David Alan Gilbert 2014-02-10 16:07 ` Dr. David Alan Gilbert 2014-02-10 18:53 ` Stefan Priebe 2014-02-13 20:06 ` Dr. David Alan Gilbert 2014-02-13 20:26 ` Stefan Priebe 2014-02-13 20:31 ` Stefan Priebe 2014-02-13 21:18 ` Stefan Priebe 2014-02-14 9:06 ` Dr. David Alan Gilbert 2014-02-11 13:32 ` Orit Wasserman 2014-02-11 13:33 ` Stefan Priebe - Profihost AG 2014-02-11 13:45 ` Orit Wasserman 2014-02-11 14:49 ` Stefan Priebe - Profihost AG 2014-02-07 9:59 ` Marcin Gibuła
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.