[Qemu-devel] QEMU LIve Migration - swap

All of lore.kernel.org
 help / color / mirror / Atom feed

* [Qemu-devel] QEMU LIve Migration - swap_free: Bad swap file entry
@ 2014-02-05 17:51 Stefan Priebe
  2014-02-05 20:15 ` Dr. David Alan Gilbert
  2014-02-06 11:14 ` [Qemu-devel] [pve-devel] " Alexandre DERUMIER
  0 siblings, 2 replies; 50+ messages in thread
From: Stefan Priebe @ 2014-02-05 17:51 UTC (permalink / raw)
  To: pve-devel, qemu-devel

Hello,

after live migrating machines with a lot of memory (32GB, 48GB, ...) i 
see pretty often crashing services after migration and the guest kernel 
prints:

[1707620.031806] swap_free: Bad swap file entry 00377410
[1707620.031806] swap_free: Bad swap file entry 00593c48
[1707620.031807] swap_free: Bad swap file entry 03201430
[1707620.031807] swap_free: Bad swap file entry 01bc5900
[1707620.031807] swap_free: Bad swap file entry 0173ce40
[1707620.031808] swap_free: Bad swap file entry 011c0270
[1707620.031808] swap_free: Bad swap file entry 03c58ae8
[1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:1 
val:1536
[1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:2 
val:-1536

Qemu is 1.7

Does anybody know a fix?

Greets,
Stefan

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-05 17:51 [Qemu-devel] QEMU LIve Migration - swap_free: Bad swap file entry Stefan Priebe
@ 2014-02-05 20:15 ` Dr. David Alan Gilbert
  2014-02-06  7:20   ` Stefan Priebe - Profihost AG
  2014-02-06 11:14 ` [Qemu-devel] [pve-devel] " Alexandre DERUMIER
  1 sibling, 1 reply; 50+ messages in thread
From: Dr. David Alan Gilbert @ 2014-02-05 20:15 UTC (permalink / raw)
  To: Stefan Priebe; +Cc: qemu-devel, pve-devel

* Stefan Priebe (s.priebe@profihost.ag) wrote:
> Hello,
> 
> after live migrating machines with a lot of memory (32GB, 48GB, ...)
> i see pretty often crashing services after migration and the guest
> kernel prints:
> 
> [1707620.031806] swap_free: Bad swap file entry 00377410
> [1707620.031806] swap_free: Bad swap file entry 00593c48
> [1707620.031807] swap_free: Bad swap file entry 03201430
> [1707620.031807] swap_free: Bad swap file entry 01bc5900
> [1707620.031807] swap_free: Bad swap file entry 0173ce40
> [1707620.031808] swap_free: Bad swap file entry 011c0270
> [1707620.031808] swap_free: Bad swap file entry 03c58ae8
> [1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380
> idx:1 val:1536
> [1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380
> idx:2 val:-1536
> 
> Qemu is 1.7
> 
> Does anybody know a fix?

I don't, but some more information about:
    1) What guest you're running
    2) The configuration of your hosts
    3) The command line (or XML if you're running libvirt) for
       your qemu so we can see what devices you're running.

would probably help people.

Do you get any messages on either the source or destination
qemu during the migrate?

Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-05 20:15 ` Dr. David Alan Gilbert
@ 2014-02-06  7:20   ` Stefan Priebe - Profihost AG
  2014-02-06 10:22     ` Orit Wasserman
  0 siblings, 1 reply; 50+ messages in thread
From: Stefan Priebe - Profihost AG @ 2014-02-06  7:20 UTC (permalink / raw)
  To: Dr. David Alan Gilbert; +Cc: qemu-devel, pve-devel

Am 05.02.2014 21:15, schrieb Dr. David Alan Gilbert:
> * Stefan Priebe (s.priebe@profihost.ag) wrote:
>> Hello,
>>
>> after live migrating machines with a lot of memory (32GB, 48GB, ...)
>> i see pretty often crashing services after migration and the guest
>> kernel prints:
>>
>> [1707620.031806] swap_free: Bad swap file entry 00377410
>> [1707620.031806] swap_free: Bad swap file entry 00593c48
>> [1707620.031807] swap_free: Bad swap file entry 03201430
>> [1707620.031807] swap_free: Bad swap file entry 01bc5900
>> [1707620.031807] swap_free: Bad swap file entry 0173ce40
>> [1707620.031808] swap_free: Bad swap file entry 011c0270
>> [1707620.031808] swap_free: Bad swap file entry 03c58ae8
>> [1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380
>> idx:1 val:1536
>> [1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380
>> idx:2 val:-1536
>>
>> Qemu is 1.7
>>
>> Does anybody know a fix?
> 
> I don't, but some more information about:
>     1) What guest you're running

Linux guest the output is also from the guest. Kernel 3.10.26

>     2) The configuration of your hosts

What do you mean by that?

>     3) The command line (or XML if you're running libvirt) for
>        your qemu so we can see what devices you're running.

qemu -chardev
socket,id=qmp,path=/var/run/qemu-server/179.qmp,server,nowait -mon
chardev=qmp,mode=control -vnc
unix:/var/run/qemu-server/179.vnc,x509,password -pidfile
/var/run/qemu-server/179.pid -daemonize -name K31953 -smp
sockets=1,cores=16 -nodefaults -boot
menu=on,strict=on,reboot-timeout=1000 -vga cirrus -cpu
kvm64,+lahf_lm,+x2apic,+sep -k de -m 32768 -device
piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device
usb-tablet,id=tablet,bus=uhci.0,port=1 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -drive
if=none,id=drive-ide2,media=cdrom,aio=native -device
ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -device
virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive
file=rbd:...,if=none,id=drive-scsi0,iops_rd=1000,iops_wr=500,bps_rd=314572800,bps_wr=209715200,aio=native,discard=on
-device
scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100
-netdev
type=tap,id=net0,ifname=tap179i0,script=/var/lib/qemu-server/pve-bridge,vhost=on
-device
virtio-net-pci,mac=CA:CA:23:AC:2D:C5,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
-rtc base=localtime -machine type=pc-i440fx-1.7

> Do you get any messages on either the source or destination
> qemu during the migrate?

no

Stefan

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-06  7:20   ` Stefan Priebe - Profihost AG
@ 2014-02-06 10:22     ` Orit Wasserman
  2014-02-06 10:49       ` Stefan Priebe - Profihost AG
  0 siblings, 1 reply; 50+ messages in thread
From: Orit Wasserman @ 2014-02-06 10:22 UTC (permalink / raw)
  To: Stefan Priebe - Profihost AG, Dr. David Alan Gilbert
  Cc: qemu-devel, pve-devel

On 02/06/2014 09:20 AM, Stefan Priebe - Profihost AG wrote:
> Am 05.02.2014 21:15, schrieb Dr. David Alan Gilbert:
>> * Stefan Priebe (s.priebe@profihost.ag) wrote:
>>> Hello,
>>>
>>> after live migrating machines with a lot of memory (32GB, 48GB, ...)
>>> i see pretty often crashing services after migration and the guest
>>> kernel prints:
>>>
>>> [1707620.031806] swap_free: Bad swap file entry 00377410
>>> [1707620.031806] swap_free: Bad swap file entry 00593c48
>>> [1707620.031807] swap_free: Bad swap file entry 03201430
>>> [1707620.031807] swap_free: Bad swap file entry 01bc5900
>>> [1707620.031807] swap_free: Bad swap file entry 0173ce40
>>> [1707620.031808] swap_free: Bad swap file entry 011c0270
>>> [1707620.031808] swap_free: Bad swap file entry 03c58ae8
>>> [1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380
>>> idx:1 val:1536
>>> [1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380
>>> idx:2 val:-1536
>>>

Is this live migration with shared storage? what kind of shared storage?
Does this happens with smaller guests?

>>> Qemu is 1.7
>>>
>>> Does anybody know a fix?
>>
>> I don't, but some more information about:
>>      1) What guest you're running
>
> Linux guest the output is also from the guest. Kernel 3.10.26
>
>>      2) The configuration of your hosts
>
> What do you mean by that?
>
>>      3) The command line (or XML if you're running libvirt) for
>>         your qemu so we can see what devices you're running.
>
> qemu -chardev
> socket,id=qmp,path=/var/run/qemu-server/179.qmp,server,nowait -mon
> chardev=qmp,mode=control -vnc
> unix:/var/run/qemu-server/179.vnc,x509,password -pidfile
> /var/run/qemu-server/179.pid -daemonize -name K31953 -smp
> sockets=1,cores=16 -nodefaults -boot
> menu=on,strict=on,reboot-timeout=1000 -vga cirrus -cpu
> kvm64,+lahf_lm,+x2apic,+sep -k de -m 32768 -device
> piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device
> usb-tablet,id=tablet,bus=uhci.0,port=1 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -drive
> if=none,id=drive-ide2,media=cdrom,aio=native -device
> ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -device
> virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive
> file=rbd:...,if=none,id=drive-scsi0,iops_rd=1000,iops_wr=500,bps_rd=314572800,bps_wr=209715200,aio=native,discard=on
> -device
> scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100
> -netdev
> type=tap,id=net0,ifname=tap179i0,script=/var/lib/qemu-server/pve-bridge,vhost=on
> -device
> virtio-net-pci,mac=CA:CA:23:AC:2D:C5,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
> -rtc base=localtime -machine type=pc-i440fx-1.7
>
>> Do you get any messages on either the source or destination
>> qemu during the migrate?
>
> no
>
> Stefan
>

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-06 10:22     ` Orit Wasserman
@ 2014-02-06 10:49       ` Stefan Priebe - Profihost AG
  0 siblings, 0 replies; 50+ messages in thread
From: Stefan Priebe - Profihost AG @ 2014-02-06 10:49 UTC (permalink / raw)
  To: Orit Wasserman, Dr. David Alan Gilbert; +Cc: qemu-devel, pve-devel


Am 06.02.2014 11:22, schrieb Orit Wasserman:
> On 02/06/2014 09:20 AM, Stefan Priebe - Profihost AG wrote:
>> Am 05.02.2014 21:15, schrieb Dr. David Alan Gilbert:
>>> * Stefan Priebe (s.priebe@profihost.ag) wrote:
>>>> Hello,
>>>>
>>>> after live migrating machines with a lot of memory (32GB, 48GB, ...)
>>>> i see pretty often crashing services after migration and the guest
>>>> kernel prints:
>>>>
>>>> [1707620.031806] swap_free: Bad swap file entry 00377410
>>>> [1707620.031806] swap_free: Bad swap file entry 00593c48
>>>> [1707620.031807] swap_free: Bad swap file entry 03201430
>>>> [1707620.031807] swap_free: Bad swap file entry 01bc5900
>>>> [1707620.031807] swap_free: Bad swap file entry 0173ce40
>>>> [1707620.031808] swap_free: Bad swap file entry 011c0270
>>>> [1707620.031808] swap_free: Bad swap file entry 03c58ae8
>>>> [1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380
>>>> idx:1 val:1536
>>>> [1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380
>>>> idx:2 val:-1536
>>>>
> 
> Is this live migration with shared storage? what kind of shared storage?

Yes - i'm using ceph / rbd.

> Does this happens with smaller guests?

Never seen that. Always with guest having memory > 16GB.

>>>> Qemu is 1.7
>>>>
>>>> Does anybody know a fix?
>>>
>>> I don't, but some more information about:
>>>      1) What guest you're running
>>
>> Linux guest the output is also from the guest. Kernel 3.10.26
>>
>>>      2) The configuration of your hosts
>>
>> What do you mean by that?
>>
>>>      3) The command line (or XML if you're running libvirt) for
>>>         your qemu so we can see what devices you're running.
>>
>> qemu -chardev
>> socket,id=qmp,path=/var/run/qemu-server/179.qmp,server,nowait -mon
>> chardev=qmp,mode=control -vnc
>> unix:/var/run/qemu-server/179.vnc,x509,password -pidfile
>> /var/run/qemu-server/179.pid -daemonize -name K31953 -smp
>> sockets=1,cores=16 -nodefaults -boot
>> menu=on,strict=on,reboot-timeout=1000 -vga cirrus -cpu
>> kvm64,+lahf_lm,+x2apic,+sep -k de -m 32768 -device
>> piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device
>> usb-tablet,id=tablet,bus=uhci.0,port=1 -device
>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -drive
>> if=none,id=drive-ide2,media=cdrom,aio=native -device
>> ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -device
>> virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive
>> file=rbd:...,if=none,id=drive-scsi0,iops_rd=1000,iops_wr=500,bps_rd=314572800,bps_wr=209715200,aio=native,discard=on
>>
>> -device
>> scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100
>>
>> -netdev
>> type=tap,id=net0,ifname=tap179i0,script=/var/lib/qemu-server/pve-bridge,vhost=on
>>
>> -device
>> virtio-net-pci,mac=CA:CA:23:AC:2D:C5,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
>>
>> -rtc base=localtime -machine type=pc-i440fx-1.7
>>
>>> Do you get any messages on either the source or destination
>>> qemu during the migrate?

no

Stefan

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-05 17:51 [Qemu-devel] QEMU LIve Migration - swap_free: Bad swap file entry Stefan Priebe
  2014-02-05 20:15 ` Dr. David Alan Gilbert
@ 2014-02-06 11:14 ` Alexandre DERUMIER
  2014-02-06 11:19   ` Stefan Priebe - Profihost AG
  1 sibling, 1 reply; 50+ messages in thread
From: Alexandre DERUMIER @ 2014-02-06 11:14 UTC (permalink / raw)
  To: Stefan Priebe; +Cc: qemu-devel, pve-devel

Do you force rbd_cache=true in ceph.conf?

if yes, do you use cache=writeback ?

according to ceph doc:
http://ceph.com/docs/next/rbd/qemu-rbd/

"Important If you set rbd_cache=true, you must set cache=writeback or risk data loss. Without cache=writeback, QEMU will not send flush requests to librbd. If QEMU exits uncleanly in this configuration, filesystems on top of rbd can be corrupted."



----- Mail original -----

De: "Stefan Priebe" <s.priebe@profihost.ag>
À: pve-devel@pve.proxmox.com, "qemu-devel" <qemu-devel@nongnu.org>
Envoyé: Mercredi 5 Février 2014 18:51:15
Objet: [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry

Hello,

after live migrating machines with a lot of memory (32GB, 48GB, ...) i
see pretty often crashing services after migration and the guest kernel
prints:

[1707620.031806] swap_free: Bad swap file entry 00377410
[1707620.031806] swap_free: Bad swap file entry 00593c48
[1707620.031807] swap_free: Bad swap file entry 03201430
[1707620.031807] swap_free: Bad swap file entry 01bc5900
[1707620.031807] swap_free: Bad swap file entry 0173ce40
[1707620.031808] swap_free: Bad swap file entry 011c0270
[1707620.031808] swap_free: Bad swap file entry 03c58ae8
[1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:1
val:1536
[1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:2
val:-1536

Qemu is 1.7

Does anybody know a fix?

Greets,
Stefan
_______________________________________________
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-06 11:14 ` [Qemu-devel] [pve-devel] " Alexandre DERUMIER
@ 2014-02-06 11:19   ` Stefan Priebe - Profihost AG
  2014-02-06 11:40     ` Alexandre DERUMIER
  0 siblings, 1 reply; 50+ messages in thread
From: Stefan Priebe - Profihost AG @ 2014-02-06 11:19 UTC (permalink / raw)
  To: Alexandre DERUMIER; +Cc: qemu-devel, pve-devel


Am 06.02.2014 12:14, schrieb Alexandre DERUMIER:
> Do you force rbd_cache=true in ceph.conf?

no

> if yes, do you use cache=writeback ?

yes

So this should be safe.

PS: all my guests do not even have !!SWAP!!

# free|grep Swap
Swap:            0          0          0

Stefan

> according to ceph doc:
> http://ceph.com/docs/next/rbd/qemu-rbd/
> 
> "Important If you set rbd_cache=true, you must set cache=writeback or risk data loss. Without cache=writeback, QEMU will not send flush requests to librbd. If QEMU exits uncleanly in this configuration, filesystems on top of rbd can be corrupted."
> 
> 
> 
> ----- Mail original ----- 
> 
> De: "Stefan Priebe" <s.priebe@profihost.ag> 
> À: pve-devel@pve.proxmox.com, "qemu-devel" <qemu-devel@nongnu.org> 
> Envoyé: Mercredi 5 Février 2014 18:51:15 
> Objet: [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 
> 
> Hello, 
> 
> after live migrating machines with a lot of memory (32GB, 48GB, ...) i 
> see pretty often crashing services after migration and the guest kernel 
> prints: 
> 
> [1707620.031806] swap_free: Bad swap file entry 00377410 
> [1707620.031806] swap_free: Bad swap file entry 00593c48 
> [1707620.031807] swap_free: Bad swap file entry 03201430 
> [1707620.031807] swap_free: Bad swap file entry 01bc5900 
> [1707620.031807] swap_free: Bad swap file entry 0173ce40 
> [1707620.031808] swap_free: Bad swap file entry 011c0270 
> [1707620.031808] swap_free: Bad swap file entry 03c58ae8 
> [1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:1 
> val:1536 
> [1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:2 
> val:-1536 
> 
> Qemu is 1.7 
> 
> Does anybody know a fix? 
> 
> Greets, 
> Stefan 
> _______________________________________________ 
> pve-devel mailing list 
> pve-devel@pve.proxmox.com 
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 
> 

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-06 11:19   ` Stefan Priebe - Profihost AG
@ 2014-02-06 11:40     ` Alexandre DERUMIER
  2014-02-06 12:10       ` Stefan Priebe - Profihost AG
  0 siblings, 1 reply; 50+ messages in thread
From: Alexandre DERUMIER @ 2014-02-06 11:40 UTC (permalink / raw)
  To: Stefan Priebe - Profihost AG; +Cc: qemu-devel, pve-devel

>>PS: all my guests do not even have !!SWAP!!

Not sure is related to swap file.

I found an similar problem here, triggered with suspend/resume on ext4

http://lkml.indiana.edu/hypermail/linux/kernel/1106.3/01340.html


Maybe is it a guest kernel bug ?

----- Mail original -----

De: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag>
À: "Alexandre DERUMIER" <aderumier@odiso.com>
Cc: pve-devel@pve.proxmox.com, "qemu-devel" <qemu-devel@nongnu.org>
Envoyé: Jeudi 6 Février 2014 12:19:36
Objet: Re: [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry


Am 06.02.2014 12:14, schrieb Alexandre DERUMIER:
> Do you force rbd_cache=true in ceph.conf?

no

> if yes, do you use cache=writeback ?

yes

So this should be safe.

PS: all my guests do not even have !!SWAP!!

# free|grep Swap
Swap: 0 0 0

Stefan

> according to ceph doc:
> http://ceph.com/docs/next/rbd/qemu-rbd/
>
> "Important If you set rbd_cache=true, you must set cache=writeback or risk data loss. Without cache=writeback, QEMU will not send flush requests to librbd. If QEMU exits uncleanly in this configuration, filesystems on top of rbd can be corrupted."
>
>
>
> ----- Mail original -----
>
> De: "Stefan Priebe" <s.priebe@profihost.ag>
> À: pve-devel@pve.proxmox.com, "qemu-devel" <qemu-devel@nongnu.org>
> Envoyé: Mercredi 5 Février 2014 18:51:15
> Objet: [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
>
> Hello,
>
> after live migrating machines with a lot of memory (32GB, 48GB, ...) i
> see pretty often crashing services after migration and the guest kernel
> prints:
>
> [1707620.031806] swap_free: Bad swap file entry 00377410
> [1707620.031806] swap_free: Bad swap file entry 00593c48
> [1707620.031807] swap_free: Bad swap file entry 03201430
> [1707620.031807] swap_free: Bad swap file entry 01bc5900
> [1707620.031807] swap_free: Bad swap file entry 0173ce40
> [1707620.031808] swap_free: Bad swap file entry 011c0270
> [1707620.031808] swap_free: Bad swap file entry 03c58ae8
> [1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:1
> val:1536
> [1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:2
> val:-1536
>
> Qemu is 1.7
>
> Does anybody know a fix?
>
> Greets,
> Stefan
> _______________________________________________
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-06 11:40     ` Alexandre DERUMIER
@ 2014-02-06 12:10       ` Stefan Priebe - Profihost AG
  2014-02-06 14:03         ` Stefan Priebe - Profihost AG
  0 siblings, 1 reply; 50+ messages in thread
From: Stefan Priebe - Profihost AG @ 2014-02-06 12:10 UTC (permalink / raw)
  To: Alexandre DERUMIER; +Cc: qemu-devel, pve-devel

May be,

sadly i've no idea. Only using 3.10 Kernel with XFS.

Stefan

Am 06.02.2014 12:40, schrieb Alexandre DERUMIER:
>>> PS: all my guests do not even have !!SWAP!! 
> 
> Not sure is related to swap file.
> 
> I found an similar problem here, triggered with suspend/resume on ext4
> 
> http://lkml.indiana.edu/hypermail/linux/kernel/1106.3/01340.html
> 
> 
> Maybe is it a guest kernel bug ?
> 
> ----- Mail original ----- 
> 
> De: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag> 
> À: "Alexandre DERUMIER" <aderumier@odiso.com> 
> Cc: pve-devel@pve.proxmox.com, "qemu-devel" <qemu-devel@nongnu.org> 
> Envoyé: Jeudi 6 Février 2014 12:19:36 
> Objet: Re: [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 
> 
> 
> Am 06.02.2014 12:14, schrieb Alexandre DERUMIER: 
>> Do you force rbd_cache=true in ceph.conf? 
> 
> no 
> 
>> if yes, do you use cache=writeback ? 
> 
> yes 
> 
> So this should be safe. 
> 
> PS: all my guests do not even have !!SWAP!! 
> 
> # free|grep Swap 
> Swap: 0 0 0 
> 
> Stefan 
> 
>> according to ceph doc: 
>> http://ceph.com/docs/next/rbd/qemu-rbd/ 
>>
>> "Important If you set rbd_cache=true, you must set cache=writeback or risk data loss. Without cache=writeback, QEMU will not send flush requests to librbd. If QEMU exits uncleanly in this configuration, filesystems on top of rbd can be corrupted." 
>>
>>
>>
>> ----- Mail original ----- 
>>
>> De: "Stefan Priebe" <s.priebe@profihost.ag> 
>> À: pve-devel@pve.proxmox.com, "qemu-devel" <qemu-devel@nongnu.org> 
>> Envoyé: Mercredi 5 Février 2014 18:51:15 
>> Objet: [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 
>>
>> Hello, 
>>
>> after live migrating machines with a lot of memory (32GB, 48GB, ...) i 
>> see pretty often crashing services after migration and the guest kernel 
>> prints: 
>>
>> [1707620.031806] swap_free: Bad swap file entry 00377410 
>> [1707620.031806] swap_free: Bad swap file entry 00593c48 
>> [1707620.031807] swap_free: Bad swap file entry 03201430 
>> [1707620.031807] swap_free: Bad swap file entry 01bc5900 
>> [1707620.031807] swap_free: Bad swap file entry 0173ce40 
>> [1707620.031808] swap_free: Bad swap file entry 011c0270 
>> [1707620.031808] swap_free: Bad swap file entry 03c58ae8 
>> [1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:1 
>> val:1536 
>> [1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:2 
>> val:-1536 
>>
>> Qemu is 1.7 
>>
>> Does anybody know a fix? 
>>
>> Greets, 
>> Stefan 
>> _______________________________________________ 
>> pve-devel mailing list 
>> pve-devel@pve.proxmox.com 
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 
>>

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-06 12:10       ` Stefan Priebe - Profihost AG
@ 2014-02-06 14:03         ` Stefan Priebe - Profihost AG
  2014-02-06 14:12           ` Marcin Gibuła
  2014-02-06 19:51           ` Dr. David Alan Gilbert
  0 siblings, 2 replies; 50+ messages in thread
From: Stefan Priebe - Profihost AG @ 2014-02-06 14:03 UTC (permalink / raw)
  To: Alexandre DERUMIER; +Cc: qemu-devel, pve-devel

some more things which happen during migration:

php5.2[20258]: segfault at a0 ip 0000000000740656 sp 00007fff53b694a0
error 4 in php-cgi[400000+6d7000]

php5.2[20249]: segfault at c ip 00007f1fb8ecb2b8 sp 00007fff642d9c20
error 4 in ZendOptimizer.so[7f1fb8e71000+147000]

cron[3154]: segfault at 7f0008a70ed4 ip 00007fc890b9d440 sp
00007fff08a6f9b0 error 4 in libc-2.13.so[7fc890b67000+182000]

Stefan

Am 06.02.2014 13:10, schrieb Stefan Priebe - Profihost AG:
> May be,
> 
> sadly i've no idea. Only using 3.10 Kernel with XFS.
> 
> Stefan
> 
> Am 06.02.2014 12:40, schrieb Alexandre DERUMIER:
>>>> PS: all my guests do not even have !!SWAP!! 
>>
>> Not sure is related to swap file.
>>
>> I found an similar problem here, triggered with suspend/resume on ext4
>>
>> http://lkml.indiana.edu/hypermail/linux/kernel/1106.3/01340.html
>>
>>
>> Maybe is it a guest kernel bug ?
>>
>> ----- Mail original ----- 
>>
>> De: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag> 
>> À: "Alexandre DERUMIER" <aderumier@odiso.com> 
>> Cc: pve-devel@pve.proxmox.com, "qemu-devel" <qemu-devel@nongnu.org> 
>> Envoyé: Jeudi 6 Février 2014 12:19:36 
>> Objet: Re: [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 
>>
>>
>> Am 06.02.2014 12:14, schrieb Alexandre DERUMIER: 
>>> Do you force rbd_cache=true in ceph.conf? 
>>
>> no 
>>
>>> if yes, do you use cache=writeback ? 
>>
>> yes 
>>
>> So this should be safe. 
>>
>> PS: all my guests do not even have !!SWAP!! 
>>
>> # free|grep Swap 
>> Swap: 0 0 0 
>>
>> Stefan 
>>
>>> according to ceph doc: 
>>> http://ceph.com/docs/next/rbd/qemu-rbd/ 
>>>
>>> "Important If you set rbd_cache=true, you must set cache=writeback or risk data loss. Without cache=writeback, QEMU will not send flush requests to librbd. If QEMU exits uncleanly in this configuration, filesystems on top of rbd can be corrupted." 
>>>
>>>
>>>
>>> ----- Mail original ----- 
>>>
>>> De: "Stefan Priebe" <s.priebe@profihost.ag> 
>>> À: pve-devel@pve.proxmox.com, "qemu-devel" <qemu-devel@nongnu.org> 
>>> Envoyé: Mercredi 5 Février 2014 18:51:15 
>>> Objet: [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 
>>>
>>> Hello, 
>>>
>>> after live migrating machines with a lot of memory (32GB, 48GB, ...) i 
>>> see pretty often crashing services after migration and the guest kernel 
>>> prints: 
>>>
>>> [1707620.031806] swap_free: Bad swap file entry 00377410 
>>> [1707620.031806] swap_free: Bad swap file entry 00593c48 
>>> [1707620.031807] swap_free: Bad swap file entry 03201430 
>>> [1707620.031807] swap_free: Bad swap file entry 01bc5900 
>>> [1707620.031807] swap_free: Bad swap file entry 0173ce40 
>>> [1707620.031808] swap_free: Bad swap file entry 011c0270 
>>> [1707620.031808] swap_free: Bad swap file entry 03c58ae8 
>>> [1707660.749059] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:1 
>>> val:1536 
>>> [1707660.749937] BUG: Bad rss-counter state mm:ffff88064d09f380 idx:2 
>>> val:-1536 
>>>
>>> Qemu is 1.7 
>>>
>>> Does anybody know a fix? 
>>>
>>> Greets, 
>>> Stefan 
>>> _______________________________________________ 
>>> pve-devel mailing list 
>>> pve-devel@pve.proxmox.com 
>>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 
>>>

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-06 14:03         ` Stefan Priebe - Profihost AG
@ 2014-02-06 14:12           ` Marcin Gibuła
  2014-02-06 19:51           ` Dr. David Alan Gilbert
  1 sibling, 0 replies; 50+ messages in thread
From: Marcin Gibuła @ 2014-02-06 14:12 UTC (permalink / raw)
  To: qemu-devel

On 06.02.2014 15:03, Stefan Priebe - Profihost AG wrote:
> some more things which happen during migration:
>
> php5.2[20258]: segfault at a0 ip 0000000000740656 sp 00007fff53b694a0
> error 4 in php-cgi[400000+6d7000]
>
> php5.2[20249]: segfault at c ip 00007f1fb8ecb2b8 sp 00007fff642d9c20
> error 4 in ZendOptimizer.so[7f1fb8e71000+147000]
>
> cron[3154]: segfault at 7f0008a70ed4 ip 00007fc890b9d440 sp
> 00007fff08a6f9b0 error 4 in libc-2.13.so[7fc890b67000+182000]

Hi,

I've seen memory corruptions after live (and offline) migrations as 
well. But in our enviroment its mostly (but not only) seen as timer 
corruption - guest hangs or have insane date in future. But I've seen 
segfaults and oopses as well.

Sadly it's very hard for me to reproduce it reliably but it occures on 
all types of linux guests - all versions of ubuntu, centos, debian, etc, 
so it doesn't seem to be connected to a specific guest kernel version. 
I've never seen windows crashing though. There was another guy here on 
qemu-devel who had similar issue and fixed it by running guest with 
no-kvmclock.

I've tested qemu 1.4 - 1.6 and kernels 3.4 - 3.10.

-- 
mg

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-06 14:03         ` Stefan Priebe - Profihost AG
  2014-02-06 14:12           ` Marcin Gibuła
@ 2014-02-06 19:51           ` Dr. David Alan Gilbert
  2014-02-06 20:00             ` Stefan Priebe
  1 sibling, 1 reply; 50+ messages in thread
From: Dr. David Alan Gilbert @ 2014-02-06 19:51 UTC (permalink / raw)
  To: Stefan Priebe - Profihost AG; +Cc: qemu-devel, Alexandre DERUMIER

* Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
> some more things which happen during migration:
> 
> php5.2[20258]: segfault at a0 ip 0000000000740656 sp 00007fff53b694a0
> error 4 in php-cgi[400000+6d7000]
> 
> php5.2[20249]: segfault at c ip 00007f1fb8ecb2b8 sp 00007fff642d9c20
> error 4 in ZendOptimizer.so[7f1fb8e71000+147000]
> 
> cron[3154]: segfault at 7f0008a70ed4 ip 00007fc890b9d440 sp
> 00007fff08a6f9b0 error 4 in libc-2.13.so[7fc890b67000+182000]

OK, so lets just assume some part of memory (or CPU state, or memory
loaded off disk...)

You said before that it was happening on a 32GB image - is it *only*
happening on a 32GB or bigger VM, or is it just more likely?

I think you also said you were using 1.7; have you tried an older
version - i.e. is this a regression in 1.7 or don't we know?

Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-06 19:51           ` Dr. David Alan Gilbert
@ 2014-02-06 20:00             ` Stefan Priebe
  2014-02-07  8:15               ` Alexandre DERUMIER
  0 siblings, 1 reply; 50+ messages in thread
From: Stefan Priebe @ 2014-02-06 20:00 UTC (permalink / raw)
  To: Dr. David Alan Gilbert; +Cc: qemu-devel, Alexandre DERUMIER

Hi,
Am 06.02.2014 20:51, schrieb Dr. David Alan Gilbert:
> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
>> some more things which happen during migration:
>>
>> php5.2[20258]: segfault at a0 ip 0000000000740656 sp 00007fff53b694a0
>> error 4 in php-cgi[400000+6d7000]
>>
>> php5.2[20249]: segfault at c ip 00007f1fb8ecb2b8 sp 00007fff642d9c20
>> error 4 in ZendOptimizer.so[7f1fb8e71000+147000]
>>
>> cron[3154]: segfault at 7f0008a70ed4 ip 00007fc890b9d440 sp
>> 00007fff08a6f9b0 error 4 in libc-2.13.so[7fc890b67000+182000]
>
> OK, so lets just assume some part of memory (or CPU state, or memory
> loaded off disk...)
>
> You said before that it was happening on a 32GB image - is it *only*
> happening on a 32GB or bigger VM, or is it just more likely?

Not image, memory. I've only seen this with vms having more than 16GB or 
32GB memory. But maybe this also indicates that just the migration takes 
longer.

> I think you also said you were using 1.7; have you tried an older
> version - i.e. is this a regression in 1.7 or don't we know?
Don't know. Sadly i cannot reproduce this with test VMs only with 
production ones.

Stefan

> Dave
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-06 20:00             ` Stefan Priebe
@ 2014-02-07  8:15               ` Alexandre DERUMIER
  2014-02-07  8:17                 ` Stefan Priebe - Profihost AG
  0 siblings, 1 reply; 50+ messages in thread
From: Alexandre DERUMIER @ 2014-02-07  8:15 UTC (permalink / raw)
  To: Stefan Priebe; +Cc: qemu-devel, Dr. David Alan Gilbert


do you use xbzrle for live migration ?



----- Mail original -----

De: "Stefan Priebe" <s.priebe@profihost.ag>
À: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
Cc: "Alexandre DERUMIER" <aderumier@odiso.com>, "qemu-devel" <qemu-devel@nongnu.org>
Envoyé: Jeudi 6 Février 2014 21:00:27
Objet: Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry

Hi,
Am 06.02.2014 20:51, schrieb Dr. David Alan Gilbert:
> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
>> some more things which happen during migration:
>>
>> php5.2[20258]: segfault at a0 ip 0000000000740656 sp 00007fff53b694a0
>> error 4 in php-cgi[400000+6d7000]
>>
>> php5.2[20249]: segfault at c ip 00007f1fb8ecb2b8 sp 00007fff642d9c20
>> error 4 in ZendOptimizer.so[7f1fb8e71000+147000]
>>
>> cron[3154]: segfault at 7f0008a70ed4 ip 00007fc890b9d440 sp
>> 00007fff08a6f9b0 error 4 in libc-2.13.so[7fc890b67000+182000]
>
> OK, so lets just assume some part of memory (or CPU state, or memory
> loaded off disk...)
>
> You said before that it was happening on a 32GB image - is it *only*
> happening on a 32GB or bigger VM, or is it just more likely?

Not image, memory. I've only seen this with vms having more than 16GB or
32GB memory. But maybe this also indicates that just the migration takes
longer.

> I think you also said you were using 1.7; have you tried an older
> version - i.e. is this a regression in 1.7 or don't we know?
Don't know. Sadly i cannot reproduce this with test VMs only with
production ones.

Stefan

> Dave
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07  8:15               ` Alexandre DERUMIER
@ 2014-02-07  8:17                 ` Stefan Priebe - Profihost AG
  2014-02-07  9:15                   ` Dr. David Alan Gilbert
  0 siblings, 1 reply; 50+ messages in thread
From: Stefan Priebe - Profihost AG @ 2014-02-07  8:17 UTC (permalink / raw)
  To: Alexandre DERUMIER; +Cc: qemu-devel, Dr. David Alan Gilbert

Am 07.02.2014 09:15, schrieb Alexandre DERUMIER:
> 
> do you use xbzrle for live migration ?

no - i'm really stucked right now with this. Biggest problem i can't
reproduce with test machines ;-(

Stefan


> 
> ----- Mail original ----- 
> 
> De: "Stefan Priebe" <s.priebe@profihost.ag> 
> À: "Dr. David Alan Gilbert" <dgilbert@redhat.com> 
> Cc: "Alexandre DERUMIER" <aderumier@odiso.com>, "qemu-devel" <qemu-devel@nongnu.org> 
> Envoyé: Jeudi 6 Février 2014 21:00:27 
> Objet: Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 
> 
> Hi, 
> Am 06.02.2014 20:51, schrieb Dr. David Alan Gilbert: 
>> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: 
>>> some more things which happen during migration: 
>>>
>>> php5.2[20258]: segfault at a0 ip 0000000000740656 sp 00007fff53b694a0 
>>> error 4 in php-cgi[400000+6d7000] 
>>>
>>> php5.2[20249]: segfault at c ip 00007f1fb8ecb2b8 sp 00007fff642d9c20 
>>> error 4 in ZendOptimizer.so[7f1fb8e71000+147000] 
>>>
>>> cron[3154]: segfault at 7f0008a70ed4 ip 00007fc890b9d440 sp 
>>> 00007fff08a6f9b0 error 4 in libc-2.13.so[7fc890b67000+182000] 
>>
>> OK, so lets just assume some part of memory (or CPU state, or memory 
>> loaded off disk...) 
>>
>> You said before that it was happening on a 32GB image - is it *only* 
>> happening on a 32GB or bigger VM, or is it just more likely? 
> 
> Not image, memory. I've only seen this with vms having more than 16GB or 
> 32GB memory. But maybe this also indicates that just the migration takes 
> longer. 
> 
>> I think you also said you were using 1.7; have you tried an older 
>> version - i.e. is this a regression in 1.7 or don't we know? 
> Don't know. Sadly i cannot reproduce this with test VMs only with 
> production ones. 
> 
> Stefan 
> 
>> Dave 
>> -- 
>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK 
>>

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07  8:17                 ` Stefan Priebe - Profihost AG
@ 2014-02-07  9:15                   ` Dr. David Alan Gilbert
  2014-02-07  9:20                     ` Stefan Priebe - Profihost AG
  0 siblings, 1 reply; 50+ messages in thread
From: Dr. David Alan Gilbert @ 2014-02-07  9:15 UTC (permalink / raw)
  To: Stefan Priebe - Profihost AG; +Cc: qemu-devel, Alexandre DERUMIER

* Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
> Am 07.02.2014 09:15, schrieb Alexandre DERUMIER:
> > 
> > do you use xbzrle for live migration ?
> 
> no - i'm really stucked right now with this. Biggest problem i can't
> reproduce with test machines ;-(

Only being able to test on your production VMs isn't fun;
is it possible or you to run an extra program on these VMs - e.g.
if we came up with a simple (userland) memory test?

Dave

> 
> Stefan
> 
> 
> > 
> > ----- Mail original ----- 
> > 
> > De: "Stefan Priebe" <s.priebe@profihost.ag> 
> > À: "Dr. David Alan Gilbert" <dgilbert@redhat.com> 
> > Cc: "Alexandre DERUMIER" <aderumier@odiso.com>, "qemu-devel" <qemu-devel@nongnu.org> 
> > Envoyé: Jeudi 6 Février 2014 21:00:27 
> > Objet: Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 
> > 
> > Hi, 
> > Am 06.02.2014 20:51, schrieb Dr. David Alan Gilbert: 
> >> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: 
> >>> some more things which happen during migration: 
> >>>
> >>> php5.2[20258]: segfault at a0 ip 0000000000740656 sp 00007fff53b694a0 
> >>> error 4 in php-cgi[400000+6d7000] 
> >>>
> >>> php5.2[20249]: segfault at c ip 00007f1fb8ecb2b8 sp 00007fff642d9c20 
> >>> error 4 in ZendOptimizer.so[7f1fb8e71000+147000] 
> >>>
> >>> cron[3154]: segfault at 7f0008a70ed4 ip 00007fc890b9d440 sp 
> >>> 00007fff08a6f9b0 error 4 in libc-2.13.so[7fc890b67000+182000] 
> >>
> >> OK, so lets just assume some part of memory (or CPU state, or memory 
> >> loaded off disk...) 
> >>
> >> You said before that it was happening on a 32GB image - is it *only* 
> >> happening on a 32GB or bigger VM, or is it just more likely? 
> > 
> > Not image, memory. I've only seen this with vms having more than 16GB or 
> > 32GB memory. But maybe this also indicates that just the migration takes 
> > longer. 
> > 
> >> I think you also said you were using 1.7; have you tried an older 
> >> version - i.e. is this a regression in 1.7 or don't we know? 
> > Don't know. Sadly i cannot reproduce this with test VMs only with 
> > production ones. 
> > 
> > Stefan 
> > 
> >> Dave 
> >> -- 
> >> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK 
> >>
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07  9:15                   ` Dr. David Alan Gilbert
@ 2014-02-07  9:20                     ` Stefan Priebe - Profihost AG
  2014-02-07  9:29                       ` Marcin Gibuła
  2014-02-07  9:31                       ` Dr. David Alan Gilbert
  0 siblings, 2 replies; 50+ messages in thread
From: Stefan Priebe - Profihost AG @ 2014-02-07  9:20 UTC (permalink / raw)
  To: Dr. David Alan Gilbert; +Cc: qemu-devel, Alexandre DERUMIER


Am 07.02.2014 10:15, schrieb Dr. David Alan Gilbert:
> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
>> Am 07.02.2014 09:15, schrieb Alexandre DERUMIER:
>>>
>>> do you use xbzrle for live migration ?
>>
>> no - i'm really stucked right now with this. Biggest problem i can't
>> reproduce with test machines ;-(
> 
> Only being able to test on your production VMs isn't fun;
> is it possible or you to run an extra program on these VMs - e.g.
> if we came up with a simple (userland) memory test?

You mean to reproduce?

I already tried https://code.google.com/p/stressapptest/ while migrating
on a test VM but this works fine.

I also tried running mysql bench while migrating on a test vm and this
works too ;-(

Stefan

> Dave
> 
>>
>> Stefan
>>
>>
>>>
>>> ----- Mail original ----- 
>>>
>>> De: "Stefan Priebe" <s.priebe@profihost.ag> 
>>> À: "Dr. David Alan Gilbert" <dgilbert@redhat.com> 
>>> Cc: "Alexandre DERUMIER" <aderumier@odiso.com>, "qemu-devel" <qemu-devel@nongnu.org> 
>>> Envoyé: Jeudi 6 Février 2014 21:00:27 
>>> Objet: Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry 
>>>
>>> Hi, 
>>> Am 06.02.2014 20:51, schrieb Dr. David Alan Gilbert: 
>>>> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote: 
>>>>> some more things which happen during migration: 
>>>>>
>>>>> php5.2[20258]: segfault at a0 ip 0000000000740656 sp 00007fff53b694a0 
>>>>> error 4 in php-cgi[400000+6d7000] 
>>>>>
>>>>> php5.2[20249]: segfault at c ip 00007f1fb8ecb2b8 sp 00007fff642d9c20 
>>>>> error 4 in ZendOptimizer.so[7f1fb8e71000+147000] 
>>>>>
>>>>> cron[3154]: segfault at 7f0008a70ed4 ip 00007fc890b9d440 sp 
>>>>> 00007fff08a6f9b0 error 4 in libc-2.13.so[7fc890b67000+182000] 
>>>>
>>>> OK, so lets just assume some part of memory (or CPU state, or memory 
>>>> loaded off disk...) 
>>>>
>>>> You said before that it was happening on a 32GB image - is it *only* 
>>>> happening on a 32GB or bigger VM, or is it just more likely? 
>>>
>>> Not image, memory. I've only seen this with vms having more than 16GB or 
>>> 32GB memory. But maybe this also indicates that just the migration takes 
>>> longer. 
>>>
>>>> I think you also said you were using 1.7; have you tried an older 
>>>> version - i.e. is this a regression in 1.7 or don't we know? 
>>> Don't know. Sadly i cannot reproduce this with test VMs only with 
>>> production ones. 
>>>
>>> Stefan 
>>>
>>>> Dave 
>>>> -- 
>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK 
>>>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07  9:20                     ` Stefan Priebe - Profihost AG
@ 2014-02-07  9:29                       ` Marcin Gibuła
  2014-02-07  9:30                         ` Stefan Priebe - Profihost AG
  2014-02-07  9:31                       ` Dr. David Alan Gilbert
  1 sibling, 1 reply; 50+ messages in thread
From: Marcin Gibuła @ 2014-02-07  9:29 UTC (permalink / raw)
  To: Stefan Priebe - Profihost AG, Dr. David Alan Gilbert
  Cc: qemu-devel, Alexandre DERUMIER

>>>> do you use xbzrle for live migration ?
>>>
>>> no - i'm really stucked right now with this. Biggest problem i can't
>>> reproduce with test machines ;-(
>>
>> Only being able to test on your production VMs isn't fun;
>> is it possible or you to run an extra program on these VMs - e.g.
>> if we came up with a simple (userland) memory test?
>
> You mean to reproduce?
>
> I already tried https://code.google.com/p/stressapptest/ while migrating
> on a test VM but this works fine.
>
> I also tried running mysql bench while migrating on a test vm and this
> works too ;-(

Have you tried to let test VM run idle for some time before migrating? 
(like 18-24 hours)

Having the same (or very similar) problem, I had bigger luck with 
reproducing it by not using freshly started VMs.

-- 
mg

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07  9:29                       ` Marcin Gibuła
@ 2014-02-07  9:30                         ` Stefan Priebe - Profihost AG
  0 siblings, 0 replies; 50+ messages in thread
From: Stefan Priebe - Profihost AG @ 2014-02-07  9:30 UTC (permalink / raw)
  To: Marcin Gibuła, Dr. David Alan Gilbert; +Cc: qemu-devel, Alexandre DERUMIER

Hi,
Am 07.02.2014 10:29, schrieb Marcin Gibuła:
>>>>> do you use xbzrle for live migration ?
>>>>
>>>> no - i'm really stucked right now with this. Biggest problem i can't
>>>> reproduce with test machines ;-(
>>>
>>> Only being able to test on your production VMs isn't fun;
>>> is it possible or you to run an extra program on these VMs - e.g.
>>> if we came up with a simple (userland) memory test?
>>
>> You mean to reproduce?
>>
>> I already tried https://code.google.com/p/stressapptest/ while migrating
>> on a test VM but this works fine.
>>
>> I also tried running mysql bench while migrating on a test vm and this
>> works too ;-(
> 
> Have you tried to let test VM run idle for some time before migrating?
> (like 18-24 hours)
> 
> Having the same (or very similar) problem, I had bigger luck with
> reproducing it by not using freshly started VMs.

no i haven't tried this will do so soon.

Stefan

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07  9:20                     ` Stefan Priebe - Profihost AG
  2014-02-07  9:29                       ` Marcin Gibuła
@ 2014-02-07  9:31                       ` Dr. David Alan Gilbert
  2014-02-07  9:37                         ` Stefan Priebe - Profihost AG
  2014-02-07  9:59                         ` Marcin Gibuła
  1 sibling, 2 replies; 50+ messages in thread
From: Dr. David Alan Gilbert @ 2014-02-07  9:31 UTC (permalink / raw)
  To: Stefan Priebe - Profihost AG; +Cc: qemu-devel, Alexandre DERUMIER

* Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
> 
> Am 07.02.2014 10:15, schrieb Dr. David Alan Gilbert:
> > * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
> >> Am 07.02.2014 09:15, schrieb Alexandre DERUMIER:
> >>>
> >>> do you use xbzrle for live migration ?
> >>
> >> no - i'm really stucked right now with this. Biggest problem i can't
> >> reproduce with test machines ;-(
> > 
> > Only being able to test on your production VMs isn't fun;
> > is it possible or you to run an extra program on these VMs - e.g.
> > if we came up with a simple (userland) memory test?
> 
> You mean to reproduce?

I'm more interested in seeing what type of corruption is happening;
if you've got a test VM that corrupts memory and we can run a program
in that vm that writes a known pattern into memory and checks it
then see what changed after migration, it might give a clue.

But obviously this would only be of any use if run on the VM that actually
fails.

> I already tried https://code.google.com/p/stressapptest/ while migrating
> on a test VM but this works fine.
> 
> I also tried running mysql bench while migrating on a test vm and this
> works too ;-(


Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07  9:31                       ` Dr. David Alan Gilbert
@ 2014-02-07  9:37                         ` Stefan Priebe - Profihost AG
  2014-02-07 12:02                           ` Stefan Priebe - Profihost AG
  2014-02-07  9:59                         ` Marcin Gibuła
  1 sibling, 1 reply; 50+ messages in thread
From: Stefan Priebe - Profihost AG @ 2014-02-07  9:37 UTC (permalink / raw)
  To: Dr. David Alan Gilbert; +Cc: qemu-devel, Alexandre DERUMIER


Am 07.02.2014 10:31, schrieb Dr. David Alan Gilbert:
> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
>>
>> Am 07.02.2014 10:15, schrieb Dr. David Alan Gilbert:
>>> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
>>>> Am 07.02.2014 09:15, schrieb Alexandre DERUMIER:
>>>>>
>>>>> do you use xbzrle for live migration ?
>>>>
>>>> no - i'm really stucked right now with this. Biggest problem i can't
>>>> reproduce with test machines ;-(
>>>
>>> Only being able to test on your production VMs isn't fun;
>>> is it possible or you to run an extra program on these VMs - e.g.
>>> if we came up with a simple (userland) memory test?
>>
>> You mean to reproduce?
> 
> I'm more interested in seeing what type of corruption is happening;
> if you've got a test VM that corrupts memory and we can run a program
> in that vm that writes a known pattern into memory and checks it
> then see what changed after migration, it might give a clue.
> 
> But obviously this would only be of any use if run on the VM that actually
> fails.

Right that makes sense - sadly i still don't know how to reproduce? Any
app ideas i can try?


>> I already tried https://code.google.com/p/stressapptest/ while migrating
>> on a test VM but this works fine.
>>
>> I also tried running mysql bench while migrating on a test vm and this
>> works too ;-(
> 
> 
> Dave
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07  9:37                         ` Stefan Priebe - Profihost AG
@ 2014-02-07 12:02                           ` Stefan Priebe - Profihost AG
  2014-02-07 12:21                             ` Dr. David Alan Gilbert
  0 siblings, 1 reply; 50+ messages in thread
From: Stefan Priebe - Profihost AG @ 2014-02-07 12:02 UTC (permalink / raw)
  To: Dr. David Alan Gilbert; +Cc: m.gibula, qemu-devel, Alexandre DERUMIER

Hi,

i was able to reproduce with a longer running test VM running the google
stress test.

And it happens exacly when the migration finishes it does not happen
while the migration is running.

Google Stress Output displays Memory errors:
------------------------------------------------
Page Error: miscompare on CPU 5(0xffff) at 0x7f52431341c0(0x0:DIMM
Unknown): read:0x00000040000000bf, reread:0x00000040000000bf
expected:0x00000040ffffffbf
Report Error: miscompare : DIMM Unknown : 1 : 571s
Page Error: miscompare on CPU 5(0xffff) at 0x7f52431341c8(0x0:DIMM
Unknown): read:0x00000020000000df, reread:0x00000020000000df
expected:0x00000020ffffffdf
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34020(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34028(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34060(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34068(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340a0(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340a8(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340e0(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340e8(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34120(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34128(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34160(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34168(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341a0(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341a8(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341e0(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341e8(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34220(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34228(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34260(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34268(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342a0(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342a8(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342e0(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342e8(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34320(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34328(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34360(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34368(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c343a0(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 571s
Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c343a8(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235030(0x0:DIMM
Unknown): read:0xfffffffd000000fd, reread:0xfffffffd000000fd
expected:0xfffffffdfffffffd
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235070(0x0:DIMM
Unknown): read:0xfffffff7000000f7, reread:0xfffffff7000000f7
expected:0xfffffff7fffffff7
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2350b0(0x0:DIMM
Unknown): read:0xffffffdf000000df, reread:0xffffffdf000000df
expected:0xffffffdfffffffdf
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2350f0(0x0:DIMM
Unknown): read:0xffffff7f0000007f, reread:0xffffff7f0000007f
expected:0xffffff7fffffff7f
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235130(0x0:DIMM
Unknown): read:0xfffffdff0000fd00, reread:0xfffffdff0000fd00
expected:0xfffffdfffffffdff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235170(0x0:DIMM
Unknown): read:0xfffff7ff0000f700, reread:0xfffff7ff0000f700
expected:0xfffff7fffffff7ff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2351b0(0x0:DIMM
Unknown): read:0xffffdfff0000df00, reread:0xffffdfff0000df00
expected:0xffffdfffffffdfff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2351f0(0x0:DIMM
Unknown): read:0xffff7fff00007f00, reread:0xffff7fff00007f00
expected:0xffff7fffffff7fff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235230(0x0:DIMM
Unknown): read:0xfffdffff00fd0000, reread:0xfffdffff00fd0000
expected:0xfffdfffffffdffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235270(0x0:DIMM
Unknown): read:0xfff7ffff00f70000, reread:0xfff7ffff00f70000
expected:0xfff7fffffff7ffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2352b0(0x0:DIMM
Unknown): read:0xffdfffff00df0000, reread:0xffdfffff00df0000
expected:0xffdfffffffdfffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2352f0(0x0:DIMM
Unknown): read:0xff7fffff007f0000, reread:0xff7fffff007f0000
expected:0xff7fffffff7fffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235330(0x0:DIMM
Unknown): read:0xfdfffffffd000000, reread:0xfdfffffffd000000
expected:0xfdfffffffdffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235370(0x0:DIMM
Unknown): read:0xf7fffffff7000000, reread:0xf7fffffff7000000
expected:0xf7fffffff7ffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2353b0(0x0:DIMM
Unknown): read:0xdfffffffdf000000, reread:0xdfffffffdf000000
expected:0xdfffffffdfffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2353f0(0x0:DIMM
Unknown): read:0x7fffffff7f000000, reread:0x7fffffff7f000000
expected:0x7fffffff7fffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235430(0x0:DIMM
Unknown): read:0xdfffffffdf000000, reread:0xdfffffffdf000000
expected:0xdfffffffdfffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235470(0x0:DIMM
Unknown): read:0xf7fffffff7000000, reread:0xf7fffffff7000000
expected:0xf7fffffff7ffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2354b0(0x0:DIMM
Unknown): read:0xfdfffffffd000000, reread:0xfdfffffffd000000
expected:0xfdfffffffdffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2354f0(0x0:DIMM
Unknown): read:0xff7fffff007f0000, reread:0xff7fffff007f0000
expected:0xff7fffffff7fffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235530(0x0:DIMM
Unknown): read:0xffdfffff00df0000, reread:0xffdfffff00df0000
expected:0xffdfffffffdfffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235570(0x0:DIMM
Unknown): read:0xfff7ffff00f70000, reread:0xfff7ffff00f70000
expected:0xfff7fffffff7ffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2355b0(0x0:DIMM
Unknown): read:0xfffdffff00fd0000, reread:0xfffdffff00fd0000
expected:0xfffdfffffffdffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2355f0(0x0:DIMM
Unknown): read:0xffff7fff00007f00, reread:0xffff7fff00007f00
expected:0xffff7fffffff7fff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235630(0x0:DIMM
Unknown): read:0xffffdfff0000df00, reread:0xffffdfff0000df00
expected:0xffffdfffffffdfff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235670(0x0:DIMM
Unknown): read:0xfffff7ff0000f700, reread:0xfffff7ff0000f700
expected:0xfffff7fffffff7ff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2356b0(0x0:DIMM
Unknown): read:0xfffffdff0000fd00, reread:0xfffffdff0000fd00
expected:0xfffffdfffffffdff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2356f0(0x0:DIMM
Unknown): read:0xffffff7f0000007f, reread:0xffffff7f0000007f
expected:0xffffff7fffffff7f
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235730(0x0:DIMM
Unknown): read:0xffffffdf000000df, reread:0xffffffdf000000df
expected:0xffffffdfffffffdf
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235770(0x0:DIMM
Unknown): read:0xfffffff7000000f7, reread:0xfffffff7000000f7
expected:0xfffffff7fffffff7
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834020(0x0:DIMM
Unknown): read:0x00000010000000ef, reread:0x00000010000000ef
expected:0x00000010ffffffef
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834028(0x0:DIMM
Unknown): read:0x00000020000000df, reread:0x00000020000000df
expected:0x00000020ffffffdf
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834030(0x0:DIMM
Unknown): read:0x00000040000000bf, reread:0x00000040000000bf
expected:0x00000040ffffffbf
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834038(0x0:DIMM
Unknown): read:0x000000800000007f, reread:0x000000800000007f
expected:0x00000080ffffff7f
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834060(0x0:DIMM
Unknown): read:0x000010000000efff, reread:0x000010000000efff
expected:0x00001000ffffefff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834068(0x0:DIMM
Unknown): read:0x000020000000dfff, reread:0x000020000000dfff
expected:0x00002000ffffdfff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834070(0x0:DIMM
Unknown): read:0x000040000000bfff, reread:0x000040000000bfff
expected:0x00004000ffffbfff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834078(0x0:DIMM
Unknown): read:0x0000800000007fff, reread:0x0000800000007fff
expected:0x00008000ffff7fff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340a0(0x0:DIMM
Unknown): read:0x0010000000ef00ff, reread:0x0010000000ef00ff
expected:0x00100000ffefffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340a8(0x0:DIMM
Unknown): read:0x0020000000df00ff, reread:0x0020000000df00ff
expected:0x00200000ffdfffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340b0(0x0:DIMM
Unknown): read:0x0040000000bf00ff, reread:0x0040000000bf00ff
expected:0x00400000ffbfffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340b8(0x0:DIMM
Unknown): read:0x00800000007f00ff, reread:0x00800000007f00ff
expected:0x00800000ff7fffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340e0(0x0:DIMM
Unknown): read:0x10000000ef0000ff, reread:0x10000000ef0000ff
expected:0x10000000efffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340e8(0x0:DIMM
Unknown): read:0x20000000df0000ff, reread:0x20000000df0000ff
expected:0x20000000dfffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340f0(0x0:DIMM
Unknown): read:0x40000000bf0000ff, reread:0x40000000bf0000ff
expected:0x40000000bfffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340f8(0x0:DIMM
Unknown): read:0x800000007f0000ff, reread:0x800000007f0000ff
expected:0x800000007fffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834120(0x0:DIMM
Unknown): read:0x04000000fb00ff00, reread:0x04000000fb00ff00
expected:0x04000000fbffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834128(0x0:DIMM
Unknown): read:0x02000000fd00ff00, reread:0x02000000fd00ff00
expected:0x02000000fdffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834130(0x0:DIMM
Unknown): read:0x01000000fe00ff00, reread:0x01000000fe00ff00
expected:0x01000000feffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834138(0x0:DIMM
Unknown): read:0x00800000007fff00, reread:0x00800000007fff00
expected:0x00800000ff7fffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834160(0x0:DIMM
Unknown): read:0x0004000000fbff00, reread:0x0004000000fbff00
expected:0x00040000fffbffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834168(0x0:DIMM
Unknown): read:0x0002000000fdff00, reread:0x0002000000fdff00
expected:0x00020000fffdffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834170(0x0:DIMM
Unknown): read:0x0001000000feff00, reread:0x0001000000feff00
expected:0x00010000fffeffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834178(0x0:DIMM
Unknown): read:0x0000800000007f00, reread:0x0000800000007f00
expected:0x00008000ffff7fff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341a0(0x0:DIMM
Unknown): read:0x000004000000fb00, reread:0x000004000000fb00
expected:0x00000400fffffbff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341a8(0x0:DIMM
Unknown): read:0x000002000000fd00, reread:0x000002000000fd00
expected:0x00000200fffffdff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341b0(0x0:DIMM
Unknown): read:0x000001000000fe00, reread:0x000001000000fe00
expected:0x00000100fffffeff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341b8(0x0:DIMM
Unknown): read:0x000000800000ff7f, reread:0x000000800000ff7f
expected:0x00000080ffffff7f
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341e0(0x0:DIMM
Unknown): read:0x000000040000fffb, reread:0x000000040000fffb
expected:0x00000004fffffffb
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341e8(0x0:DIMM
Unknown): read:0x000000020000fffd, reread:0x000000020000fffd
expected:0x00000002fffffffd
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00008(0x0:DIMM
Unknown): read:0xfffffff7000000fb, reread:0xfffffff7000000fb
expected:0xfffffff7fffffffb
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00018(0x0:DIMM
Unknown): read:0xffffff7f000000bf, reread:0xffffff7f000000bf
expected:0xffffff7fffffffbf
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00028(0x0:DIMM
Unknown): read:0xfffff7ff0000fb00, reread:0xfffff7ff0000fb00
expected:0xfffff7fffffffbff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00038(0x0:DIMM
Unknown): read:0xffff7fff0000bf00, reread:0xffff7fff0000bf00
expected:0xffff7fffffffbfff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00048(0x0:DIMM
Unknown): read:0xfff7ffff00fb0000, reread:0xfff7ffff00fb0000
expected:0xfff7fffffffbffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00058(0x0:DIMM
Unknown): read:0xff7fffff00bf0000, reread:0xff7fffff00bf0000
expected:0xff7fffffffbfffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00068(0x0:DIMM
Unknown): read:0xf7fffffffb000000, reread:0xf7fffffffb000000
expected:0xf7fffffffbffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00078(0x0:DIMM
Unknown): read:0x7fffffffbf000000, reread:0x7fffffffbf000000
expected:0x7fffffffbfffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00088(0x0:DIMM
Unknown): read:0xf7ffffffef000000, reread:0xf7ffffffef000000
expected:0xf7ffffffefffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00098(0x0:DIMM
Unknown): read:0xff7ffffffe000000, reread:0xff7ffffffe000000
expected:0xff7ffffffeffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000a8(0x0:DIMM
Unknown): read:0xfff7ffff00ef0000, reread:0xfff7ffff00ef0000
expected:0xfff7ffffffefffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000b8(0x0:DIMM
Unknown): read:0xffff7fff00fe0000, reread:0xffff7fff00fe0000
expected:0xffff7ffffffeffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000c8(0x0:DIMM
Unknown): read:0xfffff7ff0000ef00, reread:0xfffff7ff0000ef00
expected:0xfffff7ffffffefff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000d8(0x0:DIMM
Unknown): read:0xffffff7f0000fe00, reread:0xffffff7f0000fe00
expected:0xffffff7ffffffeff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000e8(0x0:DIMM
Unknown): read:0xfffffff7000000ef, reread:0xfffffff7000000ef
expected:0xfffffff7ffffffef
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000f8(0x0:DIMM
Unknown): read:0xffffffff000000fe, reread:0xffffffff000000fe
expected:0xfffffffffffffffe
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00108(0x0:DIMM
Unknown): read:0xfffffff7000000fb, reread:0xfffffff7000000fb
expected:0xfffffff7fffffffb
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00118(0x0:DIMM
Unknown): read:0xffffff7f000000bf, reread:0xffffff7f000000bf
expected:0xffffff7fffffffbf
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00128(0x0:DIMM
Unknown): read:0xfffff7ff0000fb00, reread:0xfffff7ff0000fb00
expected:0xfffff7fffffffbff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00138(0x0:DIMM
Unknown): read:0xffff7fff0000bf00, reread:0xffff7fff0000bf00
expected:0xffff7fffffffbfff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00148(0x0:DIMM
Unknown): read:0xfff7ffff00fb0000, reread:0xfff7ffff00fb0000
expected:0xfff7fffffffbffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00158(0x0:DIMM
Unknown): read:0xff7fffff00bf0000, reread:0xff7fffff00bf0000
expected:0xff7fffffffbfffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00168(0x0:DIMM
Unknown): read:0xf7fffffffb000000, reread:0xf7fffffffb000000
expected:0xf7fffffffbffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00178(0x0:DIMM
Unknown): read:0x7fffffffbf000000, reread:0x7fffffffbf000000
expected:0x7fffffffbfffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00188(0x0:DIMM
Unknown): read:0xf7ffffffef000000, reread:0xf7ffffffef000000
expected:0xf7ffffffefffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00198(0x0:DIMM
Unknown): read:0xff7ffffffe000000, reread:0xff7ffffffe000000
expected:0xff7ffffffeffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001a8(0x0:DIMM
Unknown): read:0xfff7ffff00ef0000, reread:0xfff7ffff00ef0000
expected:0xfff7ffffffefffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001b8(0x0:DIMM
Unknown): read:0xffff7fff00fe0000, reread:0xffff7fff00fe0000
expected:0xffff7ffffffeffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001c8(0x0:DIMM
Unknown): read:0xfffff7ff0000ef00, reread:0xfffff7ff0000ef00
expected:0xfffff7ffffffefff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001d8(0x0:DIMM
Unknown): read:0xffffff7f0000fe00, reread:0xffffff7f0000fe00
expected:0xffffff7ffffffeff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200010(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200018(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200050(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200058(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200090(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200098(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72000d0(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72000d8(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200110(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200118(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200150(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200158(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200190(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200198(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72001d0(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72001d8(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200210(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200218(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200250(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200258(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200290(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200298(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72002d0(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72002d8(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200310(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200318(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200350(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200358(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200390(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200398(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00000(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00008(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00010(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00018(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00040(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00048(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00050(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00058(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00080(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00088(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00090(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00098(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000c0(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000c8(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000d0(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000d8(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00100(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00108(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00110(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00118(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00140(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00148(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00150(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00158(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00180(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00188(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00190(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00198(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa001c0(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Report Error: miscompare : DIMM Unknown : 1 : 572s
Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa001c8(0x0:DIMM
Unknown): read:0x0000000000000000, reread:0x0000000000000000
expected:0xffffffffffffffff
Log: Thread 61 found 85568 hardware incidents
Log: Thread 62 found 169344 hardware incidents
Log: Thread 63 found 44544 hardware incidents
Log: Thread 64 found 149504 hardware incidents
Log: Thread 65 found 131968 hardware incidents
Log: Thread 66 found 150528 hardware incidents
Log: Thread 67 found 144384 hardware incidents
Log: Thread 68 found 149888 hardware incidents
Stats: Found 1025728 hardware incidents
Stats: Completed: 9176812.00M in 524.63s 17491.92MB/s, with 1025728
hardware incidents, 0 errors
Stats: Memory Copy: 4890244.00M at 9402.74MB/s
Stats: File Copy: 0.00M at 0.00MB/s
Stats: Net Copy: 0.00M at 0.00MB/s
Stats: Data Check: 0.00M at 0.00MB/s
Stats: Invert Data: 4286568.00M at 8242.44MB/s
Stats: Disk: 0.00M at 0.00MB/s

Status: FAIL - test discovered HW problems
-----------------------------------------------------------------------------------

Stefan
Am 07.02.2014 10:37, schrieb Stefan Priebe - Profihost AG:
> 
> Am 07.02.2014 10:31, schrieb Dr. David Alan Gilbert:
>> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
>>>
>>> Am 07.02.2014 10:15, schrieb Dr. David Alan Gilbert:
>>>> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
>>>>> Am 07.02.2014 09:15, schrieb Alexandre DERUMIER:
>>>>>>
>>>>>> do you use xbzrle for live migration ?
>>>>>
>>>>> no - i'm really stucked right now with this. Biggest problem i can't
>>>>> reproduce with test machines ;-(
>>>>
>>>> Only being able to test on your production VMs isn't fun;
>>>> is it possible or you to run an extra program on these VMs - e.g.
>>>> if we came up with a simple (userland) memory test?
>>>
>>> You mean to reproduce?
>>
>> I'm more interested in seeing what type of corruption is happening;
>> if you've got a test VM that corrupts memory and we can run a program
>> in that vm that writes a known pattern into memory and checks it
>> then see what changed after migration, it might give a clue.
>>
>> But obviously this would only be of any use if run on the VM that actually
>> fails.
> 
> Right that makes sense - sadly i still don't know how to reproduce? Any
> app ideas i can try?
> 
> 
>>> I already tried https://code.google.com/p/stressapptest/ while migrating
>>> on a test VM but this works fine.
>>>
>>> I also tried running mysql bench while migrating on a test vm and this
>>> works too ;-(
>>
>>
>> Dave
>> --
>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07 12:02                           ` Stefan Priebe - Profihost AG
@ 2014-02-07 12:21                             ` Dr. David Alan Gilbert
  2014-02-07 12:30                               ` Stefan Priebe - Profihost AG
  0 siblings, 1 reply; 50+ messages in thread
From: Dr. David Alan Gilbert @ 2014-02-07 12:21 UTC (permalink / raw)
  To: Stefan Priebe - Profihost AG; +Cc: m.gibula, qemu-devel, Alexandre DERUMIER

* Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
> Hi,
> 
> i was able to reproduce with a longer running test VM running the google
> stress test.

Hmm that's quite a fun set of differences; I think I'd like
to understand whether the pattern is related to the pattern of what
the test is doing.

Can you just give an explanation of exactly how you ran that test?
   What you installed, how exactly you ran it.

Then Marcin and I can try and replicate it.

Dave

> And it happens exacly when the migration finishes it does not happen
> while the migration is running.
> 
> Google Stress Output displays Memory errors:
> ------------------------------------------------
> Page Error: miscompare on CPU 5(0xffff) at 0x7f52431341c0(0x0:DIMM
> Unknown): read:0x00000040000000bf, reread:0x00000040000000bf
> expected:0x00000040ffffffbf
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Page Error: miscompare on CPU 5(0xffff) at 0x7f52431341c8(0x0:DIMM
> Unknown): read:0x00000020000000df, reread:0x00000020000000df
> expected:0x00000020ffffffdf
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34020(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34028(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34060(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34068(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340a0(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340a8(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340e0(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340e8(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34120(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34128(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34160(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34168(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341a0(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341a8(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341e0(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341e8(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34220(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34228(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34260(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34268(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342a0(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342a8(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342e0(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342e8(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34320(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34328(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34360(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34368(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c343a0(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 571s
> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c343a8(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235030(0x0:DIMM
> Unknown): read:0xfffffffd000000fd, reread:0xfffffffd000000fd
> expected:0xfffffffdfffffffd
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235070(0x0:DIMM
> Unknown): read:0xfffffff7000000f7, reread:0xfffffff7000000f7
> expected:0xfffffff7fffffff7
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2350b0(0x0:DIMM
> Unknown): read:0xffffffdf000000df, reread:0xffffffdf000000df
> expected:0xffffffdfffffffdf
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2350f0(0x0:DIMM
> Unknown): read:0xffffff7f0000007f, reread:0xffffff7f0000007f
> expected:0xffffff7fffffff7f
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235130(0x0:DIMM
> Unknown): read:0xfffffdff0000fd00, reread:0xfffffdff0000fd00
> expected:0xfffffdfffffffdff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235170(0x0:DIMM
> Unknown): read:0xfffff7ff0000f700, reread:0xfffff7ff0000f700
> expected:0xfffff7fffffff7ff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2351b0(0x0:DIMM
> Unknown): read:0xffffdfff0000df00, reread:0xffffdfff0000df00
> expected:0xffffdfffffffdfff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2351f0(0x0:DIMM
> Unknown): read:0xffff7fff00007f00, reread:0xffff7fff00007f00
> expected:0xffff7fffffff7fff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235230(0x0:DIMM
> Unknown): read:0xfffdffff00fd0000, reread:0xfffdffff00fd0000
> expected:0xfffdfffffffdffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235270(0x0:DIMM
> Unknown): read:0xfff7ffff00f70000, reread:0xfff7ffff00f70000
> expected:0xfff7fffffff7ffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2352b0(0x0:DIMM
> Unknown): read:0xffdfffff00df0000, reread:0xffdfffff00df0000
> expected:0xffdfffffffdfffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2352f0(0x0:DIMM
> Unknown): read:0xff7fffff007f0000, reread:0xff7fffff007f0000
> expected:0xff7fffffff7fffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235330(0x0:DIMM
> Unknown): read:0xfdfffffffd000000, reread:0xfdfffffffd000000
> expected:0xfdfffffffdffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235370(0x0:DIMM
> Unknown): read:0xf7fffffff7000000, reread:0xf7fffffff7000000
> expected:0xf7fffffff7ffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2353b0(0x0:DIMM
> Unknown): read:0xdfffffffdf000000, reread:0xdfffffffdf000000
> expected:0xdfffffffdfffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2353f0(0x0:DIMM
> Unknown): read:0x7fffffff7f000000, reread:0x7fffffff7f000000
> expected:0x7fffffff7fffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235430(0x0:DIMM
> Unknown): read:0xdfffffffdf000000, reread:0xdfffffffdf000000
> expected:0xdfffffffdfffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235470(0x0:DIMM
> Unknown): read:0xf7fffffff7000000, reread:0xf7fffffff7000000
> expected:0xf7fffffff7ffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2354b0(0x0:DIMM
> Unknown): read:0xfdfffffffd000000, reread:0xfdfffffffd000000
> expected:0xfdfffffffdffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2354f0(0x0:DIMM
> Unknown): read:0xff7fffff007f0000, reread:0xff7fffff007f0000
> expected:0xff7fffffff7fffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235530(0x0:DIMM
> Unknown): read:0xffdfffff00df0000, reread:0xffdfffff00df0000
> expected:0xffdfffffffdfffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235570(0x0:DIMM
> Unknown): read:0xfff7ffff00f70000, reread:0xfff7ffff00f70000
> expected:0xfff7fffffff7ffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2355b0(0x0:DIMM
> Unknown): read:0xfffdffff00fd0000, reread:0xfffdffff00fd0000
> expected:0xfffdfffffffdffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2355f0(0x0:DIMM
> Unknown): read:0xffff7fff00007f00, reread:0xffff7fff00007f00
> expected:0xffff7fffffff7fff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235630(0x0:DIMM
> Unknown): read:0xffffdfff0000df00, reread:0xffffdfff0000df00
> expected:0xffffdfffffffdfff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235670(0x0:DIMM
> Unknown): read:0xfffff7ff0000f700, reread:0xfffff7ff0000f700
> expected:0xfffff7fffffff7ff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2356b0(0x0:DIMM
> Unknown): read:0xfffffdff0000fd00, reread:0xfffffdff0000fd00
> expected:0xfffffdfffffffdff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2356f0(0x0:DIMM
> Unknown): read:0xffffff7f0000007f, reread:0xffffff7f0000007f
> expected:0xffffff7fffffff7f
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235730(0x0:DIMM
> Unknown): read:0xffffffdf000000df, reread:0xffffffdf000000df
> expected:0xffffffdfffffffdf
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235770(0x0:DIMM
> Unknown): read:0xfffffff7000000f7, reread:0xfffffff7000000f7
> expected:0xfffffff7fffffff7
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834020(0x0:DIMM
> Unknown): read:0x00000010000000ef, reread:0x00000010000000ef
> expected:0x00000010ffffffef
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834028(0x0:DIMM
> Unknown): read:0x00000020000000df, reread:0x00000020000000df
> expected:0x00000020ffffffdf
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834030(0x0:DIMM
> Unknown): read:0x00000040000000bf, reread:0x00000040000000bf
> expected:0x00000040ffffffbf
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834038(0x0:DIMM
> Unknown): read:0x000000800000007f, reread:0x000000800000007f
> expected:0x00000080ffffff7f
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834060(0x0:DIMM
> Unknown): read:0x000010000000efff, reread:0x000010000000efff
> expected:0x00001000ffffefff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834068(0x0:DIMM
> Unknown): read:0x000020000000dfff, reread:0x000020000000dfff
> expected:0x00002000ffffdfff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834070(0x0:DIMM
> Unknown): read:0x000040000000bfff, reread:0x000040000000bfff
> expected:0x00004000ffffbfff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834078(0x0:DIMM
> Unknown): read:0x0000800000007fff, reread:0x0000800000007fff
> expected:0x00008000ffff7fff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340a0(0x0:DIMM
> Unknown): read:0x0010000000ef00ff, reread:0x0010000000ef00ff
> expected:0x00100000ffefffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340a8(0x0:DIMM
> Unknown): read:0x0020000000df00ff, reread:0x0020000000df00ff
> expected:0x00200000ffdfffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340b0(0x0:DIMM
> Unknown): read:0x0040000000bf00ff, reread:0x0040000000bf00ff
> expected:0x00400000ffbfffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340b8(0x0:DIMM
> Unknown): read:0x00800000007f00ff, reread:0x00800000007f00ff
> expected:0x00800000ff7fffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340e0(0x0:DIMM
> Unknown): read:0x10000000ef0000ff, reread:0x10000000ef0000ff
> expected:0x10000000efffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340e8(0x0:DIMM
> Unknown): read:0x20000000df0000ff, reread:0x20000000df0000ff
> expected:0x20000000dfffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340f0(0x0:DIMM
> Unknown): read:0x40000000bf0000ff, reread:0x40000000bf0000ff
> expected:0x40000000bfffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340f8(0x0:DIMM
> Unknown): read:0x800000007f0000ff, reread:0x800000007f0000ff
> expected:0x800000007fffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834120(0x0:DIMM
> Unknown): read:0x04000000fb00ff00, reread:0x04000000fb00ff00
> expected:0x04000000fbffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834128(0x0:DIMM
> Unknown): read:0x02000000fd00ff00, reread:0x02000000fd00ff00
> expected:0x02000000fdffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834130(0x0:DIMM
> Unknown): read:0x01000000fe00ff00, reread:0x01000000fe00ff00
> expected:0x01000000feffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834138(0x0:DIMM
> Unknown): read:0x00800000007fff00, reread:0x00800000007fff00
> expected:0x00800000ff7fffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834160(0x0:DIMM
> Unknown): read:0x0004000000fbff00, reread:0x0004000000fbff00
> expected:0x00040000fffbffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834168(0x0:DIMM
> Unknown): read:0x0002000000fdff00, reread:0x0002000000fdff00
> expected:0x00020000fffdffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834170(0x0:DIMM
> Unknown): read:0x0001000000feff00, reread:0x0001000000feff00
> expected:0x00010000fffeffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834178(0x0:DIMM
> Unknown): read:0x0000800000007f00, reread:0x0000800000007f00
> expected:0x00008000ffff7fff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341a0(0x0:DIMM
> Unknown): read:0x000004000000fb00, reread:0x000004000000fb00
> expected:0x00000400fffffbff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341a8(0x0:DIMM
> Unknown): read:0x000002000000fd00, reread:0x000002000000fd00
> expected:0x00000200fffffdff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341b0(0x0:DIMM
> Unknown): read:0x000001000000fe00, reread:0x000001000000fe00
> expected:0x00000100fffffeff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341b8(0x0:DIMM
> Unknown): read:0x000000800000ff7f, reread:0x000000800000ff7f
> expected:0x00000080ffffff7f
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341e0(0x0:DIMM
> Unknown): read:0x000000040000fffb, reread:0x000000040000fffb
> expected:0x00000004fffffffb
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341e8(0x0:DIMM
> Unknown): read:0x000000020000fffd, reread:0x000000020000fffd
> expected:0x00000002fffffffd
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00008(0x0:DIMM
> Unknown): read:0xfffffff7000000fb, reread:0xfffffff7000000fb
> expected:0xfffffff7fffffffb
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00018(0x0:DIMM
> Unknown): read:0xffffff7f000000bf, reread:0xffffff7f000000bf
> expected:0xffffff7fffffffbf
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00028(0x0:DIMM
> Unknown): read:0xfffff7ff0000fb00, reread:0xfffff7ff0000fb00
> expected:0xfffff7fffffffbff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00038(0x0:DIMM
> Unknown): read:0xffff7fff0000bf00, reread:0xffff7fff0000bf00
> expected:0xffff7fffffffbfff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00048(0x0:DIMM
> Unknown): read:0xfff7ffff00fb0000, reread:0xfff7ffff00fb0000
> expected:0xfff7fffffffbffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00058(0x0:DIMM
> Unknown): read:0xff7fffff00bf0000, reread:0xff7fffff00bf0000
> expected:0xff7fffffffbfffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00068(0x0:DIMM
> Unknown): read:0xf7fffffffb000000, reread:0xf7fffffffb000000
> expected:0xf7fffffffbffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00078(0x0:DIMM
> Unknown): read:0x7fffffffbf000000, reread:0x7fffffffbf000000
> expected:0x7fffffffbfffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00088(0x0:DIMM
> Unknown): read:0xf7ffffffef000000, reread:0xf7ffffffef000000
> expected:0xf7ffffffefffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00098(0x0:DIMM
> Unknown): read:0xff7ffffffe000000, reread:0xff7ffffffe000000
> expected:0xff7ffffffeffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000a8(0x0:DIMM
> Unknown): read:0xfff7ffff00ef0000, reread:0xfff7ffff00ef0000
> expected:0xfff7ffffffefffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000b8(0x0:DIMM
> Unknown): read:0xffff7fff00fe0000, reread:0xffff7fff00fe0000
> expected:0xffff7ffffffeffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000c8(0x0:DIMM
> Unknown): read:0xfffff7ff0000ef00, reread:0xfffff7ff0000ef00
> expected:0xfffff7ffffffefff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000d8(0x0:DIMM
> Unknown): read:0xffffff7f0000fe00, reread:0xffffff7f0000fe00
> expected:0xffffff7ffffffeff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000e8(0x0:DIMM
> Unknown): read:0xfffffff7000000ef, reread:0xfffffff7000000ef
> expected:0xfffffff7ffffffef
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000f8(0x0:DIMM
> Unknown): read:0xffffffff000000fe, reread:0xffffffff000000fe
> expected:0xfffffffffffffffe
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00108(0x0:DIMM
> Unknown): read:0xfffffff7000000fb, reread:0xfffffff7000000fb
> expected:0xfffffff7fffffffb
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00118(0x0:DIMM
> Unknown): read:0xffffff7f000000bf, reread:0xffffff7f000000bf
> expected:0xffffff7fffffffbf
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00128(0x0:DIMM
> Unknown): read:0xfffff7ff0000fb00, reread:0xfffff7ff0000fb00
> expected:0xfffff7fffffffbff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00138(0x0:DIMM
> Unknown): read:0xffff7fff0000bf00, reread:0xffff7fff0000bf00
> expected:0xffff7fffffffbfff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00148(0x0:DIMM
> Unknown): read:0xfff7ffff00fb0000, reread:0xfff7ffff00fb0000
> expected:0xfff7fffffffbffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00158(0x0:DIMM
> Unknown): read:0xff7fffff00bf0000, reread:0xff7fffff00bf0000
> expected:0xff7fffffffbfffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00168(0x0:DIMM
> Unknown): read:0xf7fffffffb000000, reread:0xf7fffffffb000000
> expected:0xf7fffffffbffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00178(0x0:DIMM
> Unknown): read:0x7fffffffbf000000, reread:0x7fffffffbf000000
> expected:0x7fffffffbfffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00188(0x0:DIMM
> Unknown): read:0xf7ffffffef000000, reread:0xf7ffffffef000000
> expected:0xf7ffffffefffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00198(0x0:DIMM
> Unknown): read:0xff7ffffffe000000, reread:0xff7ffffffe000000
> expected:0xff7ffffffeffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001a8(0x0:DIMM
> Unknown): read:0xfff7ffff00ef0000, reread:0xfff7ffff00ef0000
> expected:0xfff7ffffffefffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001b8(0x0:DIMM
> Unknown): read:0xffff7fff00fe0000, reread:0xffff7fff00fe0000
> expected:0xffff7ffffffeffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001c8(0x0:DIMM
> Unknown): read:0xfffff7ff0000ef00, reread:0xfffff7ff0000ef00
> expected:0xfffff7ffffffefff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001d8(0x0:DIMM
> Unknown): read:0xffffff7f0000fe00, reread:0xffffff7f0000fe00
> expected:0xffffff7ffffffeff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200010(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200018(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200050(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200058(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200090(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200098(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72000d0(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72000d8(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200110(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200118(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200150(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200158(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200190(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200198(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72001d0(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72001d8(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200210(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200218(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200250(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200258(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200290(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200298(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72002d0(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72002d8(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200310(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200318(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200350(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200358(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200390(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200398(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00000(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00008(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00010(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00018(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00040(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00048(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00050(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00058(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00080(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00088(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00090(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00098(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000c0(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000c8(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000d0(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000d8(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00100(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00108(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00110(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00118(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00140(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00148(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00150(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00158(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00180(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00188(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00190(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00198(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa001c0(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Report Error: miscompare : DIMM Unknown : 1 : 572s
> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa001c8(0x0:DIMM
> Unknown): read:0x0000000000000000, reread:0x0000000000000000
> expected:0xffffffffffffffff
> Log: Thread 61 found 85568 hardware incidents
> Log: Thread 62 found 169344 hardware incidents
> Log: Thread 63 found 44544 hardware incidents
> Log: Thread 64 found 149504 hardware incidents
> Log: Thread 65 found 131968 hardware incidents
> Log: Thread 66 found 150528 hardware incidents
> Log: Thread 67 found 144384 hardware incidents
> Log: Thread 68 found 149888 hardware incidents
> Stats: Found 1025728 hardware incidents
> Stats: Completed: 9176812.00M in 524.63s 17491.92MB/s, with 1025728
> hardware incidents, 0 errors
> Stats: Memory Copy: 4890244.00M at 9402.74MB/s
> Stats: File Copy: 0.00M at 0.00MB/s
> Stats: Net Copy: 0.00M at 0.00MB/s
> Stats: Data Check: 0.00M at 0.00MB/s
> Stats: Invert Data: 4286568.00M at 8242.44MB/s
> Stats: Disk: 0.00M at 0.00MB/s
> 
> Status: FAIL - test discovered HW problems
> -----------------------------------------------------------------------------------
> 
> Stefan
> Am 07.02.2014 10:37, schrieb Stefan Priebe - Profihost AG:
> > 
> > Am 07.02.2014 10:31, schrieb Dr. David Alan Gilbert:
> >> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
> >>>
> >>> Am 07.02.2014 10:15, schrieb Dr. David Alan Gilbert:
> >>>> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
> >>>>> Am 07.02.2014 09:15, schrieb Alexandre DERUMIER:
> >>>>>>
> >>>>>> do you use xbzrle for live migration ?
> >>>>>
> >>>>> no - i'm really stucked right now with this. Biggest problem i can't
> >>>>> reproduce with test machines ;-(
> >>>>
> >>>> Only being able to test on your production VMs isn't fun;
> >>>> is it possible or you to run an extra program on these VMs - e.g.
> >>>> if we came up with a simple (userland) memory test?
> >>>
> >>> You mean to reproduce?
> >>
> >> I'm more interested in seeing what type of corruption is happening;
> >> if you've got a test VM that corrupts memory and we can run a program
> >> in that vm that writes a known pattern into memory and checks it
> >> then see what changed after migration, it might give a clue.
> >>
> >> But obviously this would only be of any use if run on the VM that actually
> >> fails.
> > 
> > Right that makes sense - sadly i still don't know how to reproduce? Any
> > app ideas i can try?
> > 
> > 
> >>> I already tried https://code.google.com/p/stressapptest/ while migrating
> >>> on a test VM but this works fine.
> >>>
> >>> I also tried running mysql bench while migrating on a test vm and this
> >>> works too ;-(
> >>
> >>
> >> Dave
> >> --
> >> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07 12:21                             ` Dr. David Alan Gilbert
@ 2014-02-07 12:30                               ` Stefan Priebe - Profihost AG
  2014-02-07 12:44                                 ` Paolo Bonzini
  0 siblings, 1 reply; 50+ messages in thread
From: Stefan Priebe - Profihost AG @ 2014-02-07 12:30 UTC (permalink / raw)
  To: Dr. David Alan Gilbert; +Cc: m.gibula, qemu-devel, Alexandre DERUMIER

Hi,
Am 07.02.2014 13:21, schrieb Dr. David Alan Gilbert:
> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
>> Hi,
>>
>> i was able to reproduce with a longer running test VM running the google
>> stress test.
> 
> Hmm that's quite a fun set of differences; I think I'd like
> to understand whether the pattern is related to the pattern of what
> the test is doing.
> 
> Can you just give an explanation of exactly how you ran that test?
>    What you installed, how exactly you ran it.

While migrating i've still no reliable way to reproduce but i'll try to.

I can force the problem without migration when start with:
bin/stressapptest -s 3600 -m 20 -i 20 -C 20 --force_errors

=> inject false errors to test error handling

Stefan

> Then Marcin and I can try and replicate it.
> 
> Dave
> 
>> And it happens exacly when the migration finishes it does not happen
>> while the migration is running.
>>
>> Google Stress Output displays Memory errors:
>> ------------------------------------------------
>> Page Error: miscompare on CPU 5(0xffff) at 0x7f52431341c0(0x0:DIMM
>> Unknown): read:0x00000040000000bf, reread:0x00000040000000bf
>> expected:0x00000040ffffffbf
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Page Error: miscompare on CPU 5(0xffff) at 0x7f52431341c8(0x0:DIMM
>> Unknown): read:0x00000020000000df, reread:0x00000020000000df
>> expected:0x00000020ffffffdf
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34020(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34028(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34060(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34068(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340a0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340a8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340e0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340e8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34120(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34128(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34160(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34168(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341a0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341a8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341e0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341e8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34220(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34228(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34260(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34268(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342a0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342a8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342e0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342e8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34320(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34328(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34360(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34368(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c343a0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c343a8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235030(0x0:DIMM
>> Unknown): read:0xfffffffd000000fd, reread:0xfffffffd000000fd
>> expected:0xfffffffdfffffffd
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235070(0x0:DIMM
>> Unknown): read:0xfffffff7000000f7, reread:0xfffffff7000000f7
>> expected:0xfffffff7fffffff7
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2350b0(0x0:DIMM
>> Unknown): read:0xffffffdf000000df, reread:0xffffffdf000000df
>> expected:0xffffffdfffffffdf
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2350f0(0x0:DIMM
>> Unknown): read:0xffffff7f0000007f, reread:0xffffff7f0000007f
>> expected:0xffffff7fffffff7f
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235130(0x0:DIMM
>> Unknown): read:0xfffffdff0000fd00, reread:0xfffffdff0000fd00
>> expected:0xfffffdfffffffdff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235170(0x0:DIMM
>> Unknown): read:0xfffff7ff0000f700, reread:0xfffff7ff0000f700
>> expected:0xfffff7fffffff7ff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2351b0(0x0:DIMM
>> Unknown): read:0xffffdfff0000df00, reread:0xffffdfff0000df00
>> expected:0xffffdfffffffdfff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2351f0(0x0:DIMM
>> Unknown): read:0xffff7fff00007f00, reread:0xffff7fff00007f00
>> expected:0xffff7fffffff7fff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235230(0x0:DIMM
>> Unknown): read:0xfffdffff00fd0000, reread:0xfffdffff00fd0000
>> expected:0xfffdfffffffdffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235270(0x0:DIMM
>> Unknown): read:0xfff7ffff00f70000, reread:0xfff7ffff00f70000
>> expected:0xfff7fffffff7ffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2352b0(0x0:DIMM
>> Unknown): read:0xffdfffff00df0000, reread:0xffdfffff00df0000
>> expected:0xffdfffffffdfffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2352f0(0x0:DIMM
>> Unknown): read:0xff7fffff007f0000, reread:0xff7fffff007f0000
>> expected:0xff7fffffff7fffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235330(0x0:DIMM
>> Unknown): read:0xfdfffffffd000000, reread:0xfdfffffffd000000
>> expected:0xfdfffffffdffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235370(0x0:DIMM
>> Unknown): read:0xf7fffffff7000000, reread:0xf7fffffff7000000
>> expected:0xf7fffffff7ffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2353b0(0x0:DIMM
>> Unknown): read:0xdfffffffdf000000, reread:0xdfffffffdf000000
>> expected:0xdfffffffdfffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2353f0(0x0:DIMM
>> Unknown): read:0x7fffffff7f000000, reread:0x7fffffff7f000000
>> expected:0x7fffffff7fffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235430(0x0:DIMM
>> Unknown): read:0xdfffffffdf000000, reread:0xdfffffffdf000000
>> expected:0xdfffffffdfffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235470(0x0:DIMM
>> Unknown): read:0xf7fffffff7000000, reread:0xf7fffffff7000000
>> expected:0xf7fffffff7ffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2354b0(0x0:DIMM
>> Unknown): read:0xfdfffffffd000000, reread:0xfdfffffffd000000
>> expected:0xfdfffffffdffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2354f0(0x0:DIMM
>> Unknown): read:0xff7fffff007f0000, reread:0xff7fffff007f0000
>> expected:0xff7fffffff7fffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235530(0x0:DIMM
>> Unknown): read:0xffdfffff00df0000, reread:0xffdfffff00df0000
>> expected:0xffdfffffffdfffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235570(0x0:DIMM
>> Unknown): read:0xfff7ffff00f70000, reread:0xfff7ffff00f70000
>> expected:0xfff7fffffff7ffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2355b0(0x0:DIMM
>> Unknown): read:0xfffdffff00fd0000, reread:0xfffdffff00fd0000
>> expected:0xfffdfffffffdffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2355f0(0x0:DIMM
>> Unknown): read:0xffff7fff00007f00, reread:0xffff7fff00007f00
>> expected:0xffff7fffffff7fff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235630(0x0:DIMM
>> Unknown): read:0xffffdfff0000df00, reread:0xffffdfff0000df00
>> expected:0xffffdfffffffdfff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235670(0x0:DIMM
>> Unknown): read:0xfffff7ff0000f700, reread:0xfffff7ff0000f700
>> expected:0xfffff7fffffff7ff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2356b0(0x0:DIMM
>> Unknown): read:0xfffffdff0000fd00, reread:0xfffffdff0000fd00
>> expected:0xfffffdfffffffdff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2356f0(0x0:DIMM
>> Unknown): read:0xffffff7f0000007f, reread:0xffffff7f0000007f
>> expected:0xffffff7fffffff7f
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235730(0x0:DIMM
>> Unknown): read:0xffffffdf000000df, reread:0xffffffdf000000df
>> expected:0xffffffdfffffffdf
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235770(0x0:DIMM
>> Unknown): read:0xfffffff7000000f7, reread:0xfffffff7000000f7
>> expected:0xfffffff7fffffff7
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834020(0x0:DIMM
>> Unknown): read:0x00000010000000ef, reread:0x00000010000000ef
>> expected:0x00000010ffffffef
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834028(0x0:DIMM
>> Unknown): read:0x00000020000000df, reread:0x00000020000000df
>> expected:0x00000020ffffffdf
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834030(0x0:DIMM
>> Unknown): read:0x00000040000000bf, reread:0x00000040000000bf
>> expected:0x00000040ffffffbf
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834038(0x0:DIMM
>> Unknown): read:0x000000800000007f, reread:0x000000800000007f
>> expected:0x00000080ffffff7f
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834060(0x0:DIMM
>> Unknown): read:0x000010000000efff, reread:0x000010000000efff
>> expected:0x00001000ffffefff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834068(0x0:DIMM
>> Unknown): read:0x000020000000dfff, reread:0x000020000000dfff
>> expected:0x00002000ffffdfff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834070(0x0:DIMM
>> Unknown): read:0x000040000000bfff, reread:0x000040000000bfff
>> expected:0x00004000ffffbfff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834078(0x0:DIMM
>> Unknown): read:0x0000800000007fff, reread:0x0000800000007fff
>> expected:0x00008000ffff7fff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340a0(0x0:DIMM
>> Unknown): read:0x0010000000ef00ff, reread:0x0010000000ef00ff
>> expected:0x00100000ffefffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340a8(0x0:DIMM
>> Unknown): read:0x0020000000df00ff, reread:0x0020000000df00ff
>> expected:0x00200000ffdfffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340b0(0x0:DIMM
>> Unknown): read:0x0040000000bf00ff, reread:0x0040000000bf00ff
>> expected:0x00400000ffbfffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340b8(0x0:DIMM
>> Unknown): read:0x00800000007f00ff, reread:0x00800000007f00ff
>> expected:0x00800000ff7fffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340e0(0x0:DIMM
>> Unknown): read:0x10000000ef0000ff, reread:0x10000000ef0000ff
>> expected:0x10000000efffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340e8(0x0:DIMM
>> Unknown): read:0x20000000df0000ff, reread:0x20000000df0000ff
>> expected:0x20000000dfffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340f0(0x0:DIMM
>> Unknown): read:0x40000000bf0000ff, reread:0x40000000bf0000ff
>> expected:0x40000000bfffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340f8(0x0:DIMM
>> Unknown): read:0x800000007f0000ff, reread:0x800000007f0000ff
>> expected:0x800000007fffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834120(0x0:DIMM
>> Unknown): read:0x04000000fb00ff00, reread:0x04000000fb00ff00
>> expected:0x04000000fbffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834128(0x0:DIMM
>> Unknown): read:0x02000000fd00ff00, reread:0x02000000fd00ff00
>> expected:0x02000000fdffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834130(0x0:DIMM
>> Unknown): read:0x01000000fe00ff00, reread:0x01000000fe00ff00
>> expected:0x01000000feffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834138(0x0:DIMM
>> Unknown): read:0x00800000007fff00, reread:0x00800000007fff00
>> expected:0x00800000ff7fffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834160(0x0:DIMM
>> Unknown): read:0x0004000000fbff00, reread:0x0004000000fbff00
>> expected:0x00040000fffbffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834168(0x0:DIMM
>> Unknown): read:0x0002000000fdff00, reread:0x0002000000fdff00
>> expected:0x00020000fffdffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834170(0x0:DIMM
>> Unknown): read:0x0001000000feff00, reread:0x0001000000feff00
>> expected:0x00010000fffeffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834178(0x0:DIMM
>> Unknown): read:0x0000800000007f00, reread:0x0000800000007f00
>> expected:0x00008000ffff7fff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341a0(0x0:DIMM
>> Unknown): read:0x000004000000fb00, reread:0x000004000000fb00
>> expected:0x00000400fffffbff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341a8(0x0:DIMM
>> Unknown): read:0x000002000000fd00, reread:0x000002000000fd00
>> expected:0x00000200fffffdff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341b0(0x0:DIMM
>> Unknown): read:0x000001000000fe00, reread:0x000001000000fe00
>> expected:0x00000100fffffeff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341b8(0x0:DIMM
>> Unknown): read:0x000000800000ff7f, reread:0x000000800000ff7f
>> expected:0x00000080ffffff7f
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341e0(0x0:DIMM
>> Unknown): read:0x000000040000fffb, reread:0x000000040000fffb
>> expected:0x00000004fffffffb
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341e8(0x0:DIMM
>> Unknown): read:0x000000020000fffd, reread:0x000000020000fffd
>> expected:0x00000002fffffffd
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00008(0x0:DIMM
>> Unknown): read:0xfffffff7000000fb, reread:0xfffffff7000000fb
>> expected:0xfffffff7fffffffb
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00018(0x0:DIMM
>> Unknown): read:0xffffff7f000000bf, reread:0xffffff7f000000bf
>> expected:0xffffff7fffffffbf
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00028(0x0:DIMM
>> Unknown): read:0xfffff7ff0000fb00, reread:0xfffff7ff0000fb00
>> expected:0xfffff7fffffffbff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00038(0x0:DIMM
>> Unknown): read:0xffff7fff0000bf00, reread:0xffff7fff0000bf00
>> expected:0xffff7fffffffbfff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00048(0x0:DIMM
>> Unknown): read:0xfff7ffff00fb0000, reread:0xfff7ffff00fb0000
>> expected:0xfff7fffffffbffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00058(0x0:DIMM
>> Unknown): read:0xff7fffff00bf0000, reread:0xff7fffff00bf0000
>> expected:0xff7fffffffbfffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00068(0x0:DIMM
>> Unknown): read:0xf7fffffffb000000, reread:0xf7fffffffb000000
>> expected:0xf7fffffffbffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00078(0x0:DIMM
>> Unknown): read:0x7fffffffbf000000, reread:0x7fffffffbf000000
>> expected:0x7fffffffbfffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00088(0x0:DIMM
>> Unknown): read:0xf7ffffffef000000, reread:0xf7ffffffef000000
>> expected:0xf7ffffffefffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00098(0x0:DIMM
>> Unknown): read:0xff7ffffffe000000, reread:0xff7ffffffe000000
>> expected:0xff7ffffffeffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000a8(0x0:DIMM
>> Unknown): read:0xfff7ffff00ef0000, reread:0xfff7ffff00ef0000
>> expected:0xfff7ffffffefffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000b8(0x0:DIMM
>> Unknown): read:0xffff7fff00fe0000, reread:0xffff7fff00fe0000
>> expected:0xffff7ffffffeffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000c8(0x0:DIMM
>> Unknown): read:0xfffff7ff0000ef00, reread:0xfffff7ff0000ef00
>> expected:0xfffff7ffffffefff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000d8(0x0:DIMM
>> Unknown): read:0xffffff7f0000fe00, reread:0xffffff7f0000fe00
>> expected:0xffffff7ffffffeff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000e8(0x0:DIMM
>> Unknown): read:0xfffffff7000000ef, reread:0xfffffff7000000ef
>> expected:0xfffffff7ffffffef
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000f8(0x0:DIMM
>> Unknown): read:0xffffffff000000fe, reread:0xffffffff000000fe
>> expected:0xfffffffffffffffe
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00108(0x0:DIMM
>> Unknown): read:0xfffffff7000000fb, reread:0xfffffff7000000fb
>> expected:0xfffffff7fffffffb
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00118(0x0:DIMM
>> Unknown): read:0xffffff7f000000bf, reread:0xffffff7f000000bf
>> expected:0xffffff7fffffffbf
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00128(0x0:DIMM
>> Unknown): read:0xfffff7ff0000fb00, reread:0xfffff7ff0000fb00
>> expected:0xfffff7fffffffbff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00138(0x0:DIMM
>> Unknown): read:0xffff7fff0000bf00, reread:0xffff7fff0000bf00
>> expected:0xffff7fffffffbfff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00148(0x0:DIMM
>> Unknown): read:0xfff7ffff00fb0000, reread:0xfff7ffff00fb0000
>> expected:0xfff7fffffffbffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00158(0x0:DIMM
>> Unknown): read:0xff7fffff00bf0000, reread:0xff7fffff00bf0000
>> expected:0xff7fffffffbfffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00168(0x0:DIMM
>> Unknown): read:0xf7fffffffb000000, reread:0xf7fffffffb000000
>> expected:0xf7fffffffbffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00178(0x0:DIMM
>> Unknown): read:0x7fffffffbf000000, reread:0x7fffffffbf000000
>> expected:0x7fffffffbfffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00188(0x0:DIMM
>> Unknown): read:0xf7ffffffef000000, reread:0xf7ffffffef000000
>> expected:0xf7ffffffefffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00198(0x0:DIMM
>> Unknown): read:0xff7ffffffe000000, reread:0xff7ffffffe000000
>> expected:0xff7ffffffeffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001a8(0x0:DIMM
>> Unknown): read:0xfff7ffff00ef0000, reread:0xfff7ffff00ef0000
>> expected:0xfff7ffffffefffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001b8(0x0:DIMM
>> Unknown): read:0xffff7fff00fe0000, reread:0xffff7fff00fe0000
>> expected:0xffff7ffffffeffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001c8(0x0:DIMM
>> Unknown): read:0xfffff7ff0000ef00, reread:0xfffff7ff0000ef00
>> expected:0xfffff7ffffffefff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001d8(0x0:DIMM
>> Unknown): read:0xffffff7f0000fe00, reread:0xffffff7f0000fe00
>> expected:0xffffff7ffffffeff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200010(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200018(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200050(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200058(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200090(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200098(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72000d0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72000d8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200110(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200118(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200150(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200158(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200190(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200198(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72001d0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72001d8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200210(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200218(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200250(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200258(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200290(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200298(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72002d0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72002d8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200310(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200318(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200350(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200358(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200390(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200398(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00000(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00008(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00010(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00018(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00040(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00048(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00050(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00058(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00080(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00088(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00090(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00098(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000c0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000c8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000d0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000d8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00100(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00108(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00110(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00118(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00140(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00148(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00150(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00158(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00180(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00188(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00190(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00198(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa001c0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa001c8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Log: Thread 61 found 85568 hardware incidents
>> Log: Thread 62 found 169344 hardware incidents
>> Log: Thread 63 found 44544 hardware incidents
>> Log: Thread 64 found 149504 hardware incidents
>> Log: Thread 65 found 131968 hardware incidents
>> Log: Thread 66 found 150528 hardware incidents
>> Log: Thread 67 found 144384 hardware incidents
>> Log: Thread 68 found 149888 hardware incidents
>> Stats: Found 1025728 hardware incidents
>> Stats: Completed: 9176812.00M in 524.63s 17491.92MB/s, with 1025728
>> hardware incidents, 0 errors
>> Stats: Memory Copy: 4890244.00M at 9402.74MB/s
>> Stats: File Copy: 0.00M at 0.00MB/s
>> Stats: Net Copy: 0.00M at 0.00MB/s
>> Stats: Data Check: 0.00M at 0.00MB/s
>> Stats: Invert Data: 4286568.00M at 8242.44MB/s
>> Stats: Disk: 0.00M at 0.00MB/s
>>
>> Status: FAIL - test discovered HW problems
>> -----------------------------------------------------------------------------------
>>
>> Stefan
>> Am 07.02.2014 10:37, schrieb Stefan Priebe - Profihost AG:
>>>
>>> Am 07.02.2014 10:31, schrieb Dr. David Alan Gilbert:
>>>> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
>>>>>
>>>>> Am 07.02.2014 10:15, schrieb Dr. David Alan Gilbert:
>>>>>> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
>>>>>>> Am 07.02.2014 09:15, schrieb Alexandre DERUMIER:
>>>>>>>>
>>>>>>>> do you use xbzrle for live migration ?
>>>>>>>
>>>>>>> no - i'm really stucked right now with this. Biggest problem i can't
>>>>>>> reproduce with test machines ;-(
>>>>>>
>>>>>> Only being able to test on your production VMs isn't fun;
>>>>>> is it possible or you to run an extra program on these VMs - e.g.
>>>>>> if we came up with a simple (userland) memory test?
>>>>>
>>>>> You mean to reproduce?
>>>>
>>>> I'm more interested in seeing what type of corruption is happening;
>>>> if you've got a test VM that corrupts memory and we can run a program
>>>> in that vm that writes a known pattern into memory and checks it
>>>> then see what changed after migration, it might give a clue.
>>>>
>>>> But obviously this would only be of any use if run on the VM that actually
>>>> fails.
>>>
>>> Right that makes sense - sadly i still don't know how to reproduce? Any
>>> app ideas i can try?
>>>
>>>
>>>>> I already tried https://code.google.com/p/stressapptest/ while migrating
>>>>> on a test VM but this works fine.
>>>>>
>>>>> I also tried running mysql bench while migrating on a test vm and this
>>>>> works too ;-(
>>>>
>>>>
>>>> Dave
>>>> --
>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07 12:30                               ` Stefan Priebe - Profihost AG
@ 2014-02-07 12:44                                 ` Paolo Bonzini
  2014-02-07 13:04                                   ` Stefan Priebe - Profihost AG
  0 siblings, 1 reply; 50+ messages in thread
From: Paolo Bonzini @ 2014-02-07 12:44 UTC (permalink / raw)
  To: Stefan Priebe - Profihost AG, Dr. David Alan Gilbert
  Cc: m.gibula, qemu-devel, Alexandre DERUMIER

Il 07/02/2014 13:30, Stefan Priebe - Profihost AG ha scritto:
>> >> i was able to reproduce with a longer running test VM running the google
>> >> stress test.
>>
>> Hmm that's quite a fun set of differences; I think I'd like
>> to understand whether the pattern is related to the pattern of what
>> the test is doing.

Stefan, can you try to reproduce it:

- with Unix migration between two QEMUs on the same host

- with different hosts

- with a different network (e.g. just a cross cable between two machines)

Paolo

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07 12:44                                 ` Paolo Bonzini
@ 2014-02-07 13:04                                   ` Stefan Priebe - Profihost AG
  2014-02-07 13:08                                     ` Dr. David Alan Gilbert
  2014-02-07 13:19                                     ` Paolo Bonzini
  0 siblings, 2 replies; 50+ messages in thread
From: Stefan Priebe - Profihost AG @ 2014-02-07 13:04 UTC (permalink / raw)
  To: Paolo Bonzini, Dr. David Alan Gilbert
  Cc: m.gibula, qemu-devel, Alexandre DERUMIER

Hi,

Am 07.02.2014 13:44, schrieb Paolo Bonzini:
> Il 07/02/2014 13:30, Stefan Priebe - Profihost AG ha scritto:
>>> >> i was able to reproduce with a longer running test VM running the
>>> google
>>> >> stress test.
>>>
>>> Hmm that's quite a fun set of differences; I think I'd like
>>> to understand whether the pattern is related to the pattern of what
>>> the test is doing.
> 
> Stefan, can you try to reproduce it:

first of all i've now a memory image of a VM where i can reproduce it.
reproducing does NOT work if i boot the VM freshly i need to let it run
for some hours.

Then just when the migration finishes there is a short time frame where
the google stress app reports memory errors than when the migration
finishes it runs fine again.

It seems to me it is related to pause and unpause/resume?

> - with Unix migration between two QEMUs on the same host
now tested => same issue

> - with different hosts
already tested => same issue

> - with a different network (e.g. just a cross cable between two machines)
already tested => same issue

Greets,
Stefan

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07 13:04                                   ` Stefan Priebe - Profihost AG
@ 2014-02-07 13:08                                     ` Dr. David Alan Gilbert
  2014-02-07 13:10                                       ` Stefan Priebe - Profihost AG
  2014-02-07 13:19                                     ` Paolo Bonzini
  1 sibling, 1 reply; 50+ messages in thread
From: Dr. David Alan Gilbert @ 2014-02-07 13:08 UTC (permalink / raw)
  To: Stefan Priebe - Profihost AG
  Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER

* Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
> Hi,
> 
> Am 07.02.2014 13:44, schrieb Paolo Bonzini:
> > Il 07/02/2014 13:30, Stefan Priebe - Profihost AG ha scritto:
> >>> >> i was able to reproduce with a longer running test VM running the
> >>> google
> >>> >> stress test.
> >>>
> >>> Hmm that's quite a fun set of differences; I think I'd like
> >>> to understand whether the pattern is related to the pattern of what
> >>> the test is doing.
> > 
> > Stefan, can you try to reproduce it:
> 
> first of all i've now a memory image of a VM where i can reproduce it.
> reproducing does NOT work if i boot the VM freshly i need to let it run
> for some hours.
> 
> Then just when the migration finishes there is a short time frame where
> the google stress app reports memory errors than when the migration
> finishes it runs fine again.
> 
> It seems to me it is related to pause and unpause/resume?

But do you have to pause/resume it to cause the error? Have you got cases
where you boot it and then leave it running for a few hours and then it 
fails if you migrate it?

Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07 13:08                                     ` Dr. David Alan Gilbert
@ 2014-02-07 13:10                                       ` Stefan Priebe - Profihost AG
  2014-02-07 13:15                                         ` Dr. David Alan Gilbert
  0 siblings, 1 reply; 50+ messages in thread
From: Stefan Priebe - Profihost AG @ 2014-02-07 13:10 UTC (permalink / raw)
  To: Dr. David Alan Gilbert; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER

Hi,

Am 07.02.2014 14:08, schrieb Dr. David Alan Gilbert:
> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
>> Hi,
>>
>> Am 07.02.2014 13:44, schrieb Paolo Bonzini:
>>> Il 07/02/2014 13:30, Stefan Priebe - Profihost AG ha scritto:
>>>>>>> i was able to reproduce with a longer running test VM running the
>>>>> google
>>>>>>> stress test.
>>>>>
>>>>> Hmm that's quite a fun set of differences; I think I'd like
>>>>> to understand whether the pattern is related to the pattern of what
>>>>> the test is doing.
>>>
>>> Stefan, can you try to reproduce it:
>>
>> first of all i've now a memory image of a VM where i can reproduce it.
>> reproducing does NOT work if i boot the VM freshly i need to let it run
>> for some hours.
>>
>> Then just when the migration finishes there is a short time frame where
>> the google stress app reports memory errors than when the migration
>> finishes it runs fine again.
>>
>> It seems to me it is related to pause and unpause/resume?
> 
> But do you have to pause/resume it to cause the error? Have you got cases
> where you boot it and then leave it running for a few hours and then it 
> fails if you migrate it?

Yes but isn't migration always a pause / unpause at the end? I thought
migration_downtime is the value a very small pause unpause is allowed.

Stefan

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07 13:10                                       ` Stefan Priebe - Profihost AG
@ 2014-02-07 13:15                                         ` Dr. David Alan Gilbert
  2014-02-07 13:21                                           ` Stefan Priebe - Profihost AG
  0 siblings, 1 reply; 50+ messages in thread
From: Dr. David Alan Gilbert @ 2014-02-07 13:15 UTC (permalink / raw)
  To: Stefan Priebe - Profihost AG
  Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER

* Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
> Hi,
> 
> Am 07.02.2014 14:08, schrieb Dr. David Alan Gilbert:
> > * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:

> >> first of all i've now a memory image of a VM where i can reproduce it.
> >> reproducing does NOT work if i boot the VM freshly i need to let it run
> >> for some hours.
> >>
> >> Then just when the migration finishes there is a short time frame where
> >> the google stress app reports memory errors than when the migration
> >> finishes it runs fine again.
> >>
> >> It seems to me it is related to pause and unpause/resume?
> > 
> > But do you have to pause/resume it to cause the error? Have you got cases
> > where you boot it and then leave it running for a few hours and then it 
> > fails if you migrate it?
> 
> Yes but isn't migration always a pause / unpause at the end? I thought
> migration_downtime is the value a very small pause unpause is allowed.

There's a heck of a lot of other stuff that goes on in migration, and that
downtime isn't quite the same.

If it can be reproduced with just suspend/resume stuff then that's a different
place to start looking than if it's migration only.

Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07 13:15                                         ` Dr. David Alan Gilbert
@ 2014-02-07 13:21                                           ` Stefan Priebe - Profihost AG
  0 siblings, 0 replies; 50+ messages in thread
From: Stefan Priebe - Profihost AG @ 2014-02-07 13:21 UTC (permalink / raw)
  To: Dr. David Alan Gilbert; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER

Hi,

Am 07.02.2014 14:15, schrieb Dr. David Alan Gilbert:
> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
>> Hi,
>>
>> Am 07.02.2014 14:08, schrieb Dr. David Alan Gilbert:
>>> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
> 
>>>> first of all i've now a memory image of a VM where i can reproduce it.
>>>> reproducing does NOT work if i boot the VM freshly i need to let it run
>>>> for some hours.
>>>>
>>>> Then just when the migration finishes there is a short time frame where
>>>> the google stress app reports memory errors than when the migration
>>>> finishes it runs fine again.
>>>>
>>>> It seems to me it is related to pause and unpause/resume?
>>>
>>> But do you have to pause/resume it to cause the error? Have you got cases
>>> where you boot it and then leave it running for a few hours and then it 
>>> fails if you migrate it?
>>
>> Yes but isn't migration always a pause / unpause at the end? I thought
>> migration_downtime is the value a very small pause unpause is allowed.
> 
> There's a heck of a lot of other stuff that goes on in migration, and that
> downtime isn't quite the same.
> 
> If it can be reproduced with just suspend/resume stuff then that's a different
> place to start looking than if it's migration only.

ah OK now i got it. No i can't reproduce with suspend resume. But while
migrating it happens directly at the end when the switch from host a to
b happens.

> Dave
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07 13:04                                   ` Stefan Priebe - Profihost AG
  2014-02-07 13:08                                     ` Dr. David Alan Gilbert
@ 2014-02-07 13:19                                     ` Paolo Bonzini
  2014-02-07 13:39                                       ` Stefan Priebe - Profihost AG
  1 sibling, 1 reply; 50+ messages in thread
From: Paolo Bonzini @ 2014-02-07 13:19 UTC (permalink / raw)
  To: Stefan Priebe - Profihost AG, Dr. David Alan Gilbert
  Cc: m.gibula, qemu-devel, Alexandre DERUMIER

Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto:
> first of all i've now a memory image of a VM where i can reproduce it.

You mean you start that VM with -incoming 'exec:cat /path/to/vm.img'? 
But google stress test doesn't report any error until you start 
migration _and_ it finishes?

That sounds good enough.  Can you upload the image somewhere (doesn't 
have to be a public place, you can contact David or others offlist)?

> reproducing does NOT work if i boot the VM freshly i need to let it run
> for some hours.
>
> Then just when the migration finishes there is a short time frame where
> the google stress app reports memory errors than when the migration
> finishes it runs fine again.
>
> It seems to me it is related to pause and unpause/resume?
>
>> > - with Unix migration between two QEMUs on the same host
> now tested => same issue
>
>> > - with different hosts
> already tested => same issue
>
>> > - with a different network (e.g. just a cross cable between two machines)
> already tested => same issue

Another test:

- start the VM with -S, migrate, do errors appear on the destination?

Thanks,

Paolo

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07 13:19                                     ` Paolo Bonzini
@ 2014-02-07 13:39                                       ` Stefan Priebe - Profihost AG
  2014-02-07 13:45                                         ` Stefan Priebe - Profihost AG
  0 siblings, 1 reply; 50+ messages in thread
From: Stefan Priebe - Profihost AG @ 2014-02-07 13:39 UTC (permalink / raw)
  To: Paolo Bonzini, Dr. David Alan Gilbert
  Cc: m.gibula, qemu-devel, Alexandre DERUMIER

Hi,
Am 07.02.2014 14:19, schrieb Paolo Bonzini:
> Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto:
>> first of all i've now a memory image of a VM where i can reproduce it.
> 
> You mean you start that VM with -incoming 'exec:cat /path/to/vm.img'?
> But google stress test doesn't report any error until you start
> migration _and_ it finishes?

Sorry no i meant i have a VM where i saved the memory to disk - so i
don't need to wait hours until i can reproduce as it does not happen
with a fresh started VM. So it's a state file i think.

> Another test:
> 
> - start the VM with -S, migrate, do errors appear on the destination?

I started with -S and the errors appear AFTER resuming/unpause the VM.
So it is fine until i resume it on the "new" host.

Stefan

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07 13:39                                       ` Stefan Priebe - Profihost AG
@ 2014-02-07 13:45                                         ` Stefan Priebe - Profihost AG
  2014-02-07 19:21                                           ` Stefan Priebe
  0 siblings, 1 reply; 50+ messages in thread
From: Stefan Priebe - Profihost AG @ 2014-02-07 13:45 UTC (permalink / raw)
  To: Paolo Bonzini, Dr. David Alan Gilbert
  Cc: m.gibula, qemu-devel, Alexandre DERUMIER

it's always the same "pattern" there are too many 0 instead of X.

only seen:

read:0x0000000000000000 ... expected:0xffffffffffffffff

or

read:0xffffffff00000000 ... expected:0xffffffffffffffff

or

read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff

or

read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5

no idea if this helps.

Stefan

Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG:
> Hi,
> Am 07.02.2014 14:19, schrieb Paolo Bonzini:
>> Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto:
>>> first of all i've now a memory image of a VM where i can reproduce it.
>>
>> You mean you start that VM with -incoming 'exec:cat /path/to/vm.img'?
>> But google stress test doesn't report any error until you start
>> migration _and_ it finishes?
> 
> Sorry no i meant i have a VM where i saved the memory to disk - so i
> don't need to wait hours until i can reproduce as it does not happen
> with a fresh started VM. So it's a state file i think.
> 
>> Another test:
>>
>> - start the VM with -S, migrate, do errors appear on the destination?
> 
> I started with -S and the errors appear AFTER resuming/unpause the VM.
> So it is fine until i resume it on the "new" host.
> 
> Stefan
> 

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07 13:45                                         ` Stefan Priebe - Profihost AG
@ 2014-02-07 19:21                                           ` Stefan Priebe
  2014-02-07 20:02                                             ` Dr. David Alan Gilbert
  0 siblings, 1 reply; 50+ messages in thread
From: Stefan Priebe @ 2014-02-07 19:21 UTC (permalink / raw)
  To: Paolo Bonzini, Dr. David Alan Gilbert
  Cc: m.gibula, qemu-devel, Alexandre DERUMIER

anything i could try or debug? to help to find the problem?

Stefan

Am 07.02.2014 14:45, schrieb Stefan Priebe - Profihost AG:
> it's always the same "pattern" there are too many 0 instead of X.
>
> only seen:
>
> read:0x0000000000000000 ... expected:0xffffffffffffffff
>
> or
>
> read:0xffffffff00000000 ... expected:0xffffffffffffffff
>
> or
>
> read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff
>
> or
>
> read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5
>
> no idea if this helps.
>
> Stefan
>
> Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG:
>> Hi,
>> Am 07.02.2014 14:19, schrieb Paolo Bonzini:
>>> Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto:
>>>> first of all i've now a memory image of a VM where i can reproduce it.
>>>
>>> You mean you start that VM with -incoming 'exec:cat /path/to/vm.img'?
>>> But google stress test doesn't report any error until you start
>>> migration _and_ it finishes?
>>
>> Sorry no i meant i have a VM where i saved the memory to disk - so i
>> don't need to wait hours until i can reproduce as it does not happen
>> with a fresh started VM. So it's a state file i think.
>>
>>> Another test:
>>>
>>> - start the VM with -S, migrate, do errors appear on the destination?
>>
>> I started with -S and the errors appear AFTER resuming/unpause the VM.
>> So it is fine until i resume it on the "new" host.
>>
>> Stefan
>>

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07 19:21                                           ` Stefan Priebe
@ 2014-02-07 20:02                                             ` Dr. David Alan Gilbert
  2014-02-07 20:10                                               ` Stefan Priebe
  0 siblings, 1 reply; 50+ messages in thread
From: Dr. David Alan Gilbert @ 2014-02-07 20:02 UTC (permalink / raw)
  To: Stefan Priebe; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER

* Stefan Priebe (s.priebe@profihost.ag) wrote:
> anything i could try or debug? to help to find the problem?

I think the most useful would be to see if the problem is 
a new problem in the 1.7 you're using or has existed
for a while; depending on the machine type you used, it might
be possible to load that image on an earlier (or newer) qemu
and try the same test, however if the problem doesn't
repeat reliably it can be hard.

If you have any way of simplifying the configuration of the
VM it would be good; e.g. if you could get a failure on
something without graphics (-nographic) and USB.

Dave

> 
> Stefan
> 
> Am 07.02.2014 14:45, schrieb Stefan Priebe - Profihost AG:
> >it's always the same "pattern" there are too many 0 instead of X.
> >
> >only seen:
> >
> >read:0x0000000000000000 ... expected:0xffffffffffffffff
> >
> >or
> >
> >read:0xffffffff00000000 ... expected:0xffffffffffffffff
> >
> >or
> >
> >read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff
> >
> >or
> >
> >read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5
> >
> >no idea if this helps.
> >
> >Stefan
> >
> >Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG:
> >>Hi,
> >>Am 07.02.2014 14:19, schrieb Paolo Bonzini:
> >>>Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto:
> >>>>first of all i've now a memory image of a VM where i can reproduce it.
> >>>
> >>>You mean you start that VM with -incoming 'exec:cat /path/to/vm.img'?
> >>>But google stress test doesn't report any error until you start
> >>>migration _and_ it finishes?
> >>
> >>Sorry no i meant i have a VM where i saved the memory to disk - so i
> >>don't need to wait hours until i can reproduce as it does not happen
> >>with a fresh started VM. So it's a state file i think.
> >>
> >>>Another test:
> >>>
> >>>- start the VM with -S, migrate, do errors appear on the destination?
> >>
> >>I started with -S and the errors appear AFTER resuming/unpause the VM.
> >>So it is fine until i resume it on the "new" host.
> >>
> >>Stefan
> >>
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07 20:02                                             ` Dr. David Alan Gilbert
@ 2014-02-07 20:10                                               ` Stefan Priebe
  2014-02-08 19:23                                                 ` Stefan Priebe
  0 siblings, 1 reply; 50+ messages in thread
From: Stefan Priebe @ 2014-02-07 20:10 UTC (permalink / raw)
  To: Dr. David Alan Gilbert; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER

Am 07.02.2014 21:02, schrieb Dr. David Alan Gilbert:
> * Stefan Priebe (s.priebe@profihost.ag) wrote:
>> anything i could try or debug? to help to find the problem?
>
> I think the most useful would be to see if the problem is
> a new problem in the 1.7 you're using or has existed
> for a while; depending on the machine type you used, it might
> be possible to load that image on an earlier (or newer) qemu
> and try the same test, however if the problem doesn't
> repeat reliably it can be hard.

I've seen this first with Qemu 1.5 but was not able to reproduce it for 
month. 1.4 was working fine.

> If you have any way of simplifying the configuration of the
> VM it would be good; e.g. if you could get a failure on
> something without graphics (-nographic) and USB.

Sadly not ;-(

> Dave
>
>>
>> Stefan
>>
>> Am 07.02.2014 14:45, schrieb Stefan Priebe - Profihost AG:
>>> it's always the same "pattern" there are too many 0 instead of X.
>>>
>>> only seen:
>>>
>>> read:0x0000000000000000 ... expected:0xffffffffffffffff
>>>
>>> or
>>>
>>> read:0xffffffff00000000 ... expected:0xffffffffffffffff
>>>
>>> or
>>>
>>> read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff
>>>
>>> or
>>>
>>> read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5
>>>
>>> no idea if this helps.
>>>
>>> Stefan
>>>
>>> Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG:
>>>> Hi,
>>>> Am 07.02.2014 14:19, schrieb Paolo Bonzini:
>>>>> Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto:
>>>>>> first of all i've now a memory image of a VM where i can reproduce it.
>>>>>
>>>>> You mean you start that VM with -incoming 'exec:cat /path/to/vm.img'?
>>>>> But google stress test doesn't report any error until you start
>>>>> migration _and_ it finishes?
>>>>
>>>> Sorry no i meant i have a VM where i saved the memory to disk - so i
>>>> don't need to wait hours until i can reproduce as it does not happen
>>>> with a fresh started VM. So it's a state file i think.
>>>>
>>>>> Another test:
>>>>>
>>>>> - start the VM with -S, migrate, do errors appear on the destination?
>>>>
>>>> I started with -S and the errors appear AFTER resuming/unpause the VM.
>>>> So it is fine until i resume it on the "new" host.
>>>>
>>>> Stefan
>>>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07 20:10                                               ` Stefan Priebe
@ 2014-02-08 19:23                                                 ` Stefan Priebe
  2014-02-10  9:30                                                   ` Dr. David Alan Gilbert
                                                                     ` (2 more replies)
  0 siblings, 3 replies; 50+ messages in thread
From: Stefan Priebe @ 2014-02-08 19:23 UTC (permalink / raw)
  To: Dr. David Alan Gilbert; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER

i could fix it by explicitly disable xbzrle - it seems its automatically 
on if i do not set the migration caps to false.

So it seems to be a xbzrle bug.

Stefan

Am 07.02.2014 21:10, schrieb Stefan Priebe:
> Am 07.02.2014 21:02, schrieb Dr. David Alan Gilbert:
>> * Stefan Priebe (s.priebe@profihost.ag) wrote:
>>> anything i could try or debug? to help to find the problem?
>>
>> I think the most useful would be to see if the problem is
>> a new problem in the 1.7 you're using or has existed
>> for a while; depending on the machine type you used, it might
>> be possible to load that image on an earlier (or newer) qemu
>> and try the same test, however if the problem doesn't
>> repeat reliably it can be hard.
>
> I've seen this first with Qemu 1.5 but was not able to reproduce it for
> month. 1.4 was working fine.
>
>> If you have any way of simplifying the configuration of the
>> VM it would be good; e.g. if you could get a failure on
>> something without graphics (-nographic) and USB.
>
> Sadly not ;-(
>
>> Dave
>>
>>>
>>> Stefan
>>>
>>> Am 07.02.2014 14:45, schrieb Stefan Priebe - Profihost AG:
>>>> it's always the same "pattern" there are too many 0 instead of X.
>>>>
>>>> only seen:
>>>>
>>>> read:0x0000000000000000 ... expected:0xffffffffffffffff
>>>>
>>>> or
>>>>
>>>> read:0xffffffff00000000 ... expected:0xffffffffffffffff
>>>>
>>>> or
>>>>
>>>> read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff
>>>>
>>>> or
>>>>
>>>> read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5
>>>>
>>>> no idea if this helps.
>>>>
>>>> Stefan
>>>>
>>>> Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG:
>>>>> Hi,
>>>>> Am 07.02.2014 14:19, schrieb Paolo Bonzini:
>>>>>> Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto:
>>>>>>> first of all i've now a memory image of a VM where i can
>>>>>>> reproduce it.
>>>>>>
>>>>>> You mean you start that VM with -incoming 'exec:cat /path/to/vm.img'?
>>>>>> But google stress test doesn't report any error until you start
>>>>>> migration _and_ it finishes?
>>>>>
>>>>> Sorry no i meant i have a VM where i saved the memory to disk - so i
>>>>> don't need to wait hours until i can reproduce as it does not happen
>>>>> with a fresh started VM. So it's a state file i think.
>>>>>
>>>>>> Another test:
>>>>>>
>>>>>> - start the VM with -S, migrate, do errors appear on the destination?
>>>>>
>>>>> I started with -S and the errors appear AFTER resuming/unpause the VM.
>>>>> So it is fine until i resume it on the "new" host.
>>>>>
>>>>> Stefan
>>>>>
>>>
>> --
>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-08 19:23                                                 ` Stefan Priebe
@ 2014-02-10  9:30                                                   ` Dr. David Alan Gilbert
  2014-02-10 16:07                                                   ` Dr. David Alan Gilbert
  2014-02-11 13:32                                                   ` Orit Wasserman
  2 siblings, 0 replies; 50+ messages in thread
From: Dr. David Alan Gilbert @ 2014-02-10  9:30 UTC (permalink / raw)
  To: Stefan Priebe; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER

* Stefan Priebe (s.priebe@profihost.ag) wrote:
> i could fix it by explicitly disable xbzrle - it seems its
> automatically on if i do not set the migration caps to false.
> 
> So it seems to be a xbzrle bug.

Ah right, yes that would make sense for the type of errors
you're seeing, and does make it easier to tie down.

Dave

> 
> Stefan
> 
> Am 07.02.2014 21:10, schrieb Stefan Priebe:
> >Am 07.02.2014 21:02, schrieb Dr. David Alan Gilbert:
> >>* Stefan Priebe (s.priebe@profihost.ag) wrote:
> >>>anything i could try or debug? to help to find the problem?
> >>
> >>I think the most useful would be to see if the problem is
> >>a new problem in the 1.7 you're using or has existed
> >>for a while; depending on the machine type you used, it might
> >>be possible to load that image on an earlier (or newer) qemu
> >>and try the same test, however if the problem doesn't
> >>repeat reliably it can be hard.
> >
> >I've seen this first with Qemu 1.5 but was not able to reproduce it for
> >month. 1.4 was working fine.
> >
> >>If you have any way of simplifying the configuration of the
> >>VM it would be good; e.g. if you could get a failure on
> >>something without graphics (-nographic) and USB.
> >
> >Sadly not ;-(
> >
> >>Dave
> >>
> >>>
> >>>Stefan
> >>>
> >>>Am 07.02.2014 14:45, schrieb Stefan Priebe - Profihost AG:
> >>>>it's always the same "pattern" there are too many 0 instead of X.
> >>>>
> >>>>only seen:
> >>>>
> >>>>read:0x0000000000000000 ... expected:0xffffffffffffffff
> >>>>
> >>>>or
> >>>>
> >>>>read:0xffffffff00000000 ... expected:0xffffffffffffffff
> >>>>
> >>>>or
> >>>>
> >>>>read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff
> >>>>
> >>>>or
> >>>>
> >>>>read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5
> >>>>
> >>>>no idea if this helps.
> >>>>
> >>>>Stefan
> >>>>
> >>>>Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG:
> >>>>>Hi,
> >>>>>Am 07.02.2014 14:19, schrieb Paolo Bonzini:
> >>>>>>Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto:
> >>>>>>>first of all i've now a memory image of a VM where i can
> >>>>>>>reproduce it.
> >>>>>>
> >>>>>>You mean you start that VM with -incoming 'exec:cat /path/to/vm.img'?
> >>>>>>But google stress test doesn't report any error until you start
> >>>>>>migration _and_ it finishes?
> >>>>>
> >>>>>Sorry no i meant i have a VM where i saved the memory to disk - so i
> >>>>>don't need to wait hours until i can reproduce as it does not happen
> >>>>>with a fresh started VM. So it's a state file i think.
> >>>>>
> >>>>>>Another test:
> >>>>>>
> >>>>>>- start the VM with -S, migrate, do errors appear on the destination?
> >>>>>
> >>>>>I started with -S and the errors appear AFTER resuming/unpause the VM.
> >>>>>So it is fine until i resume it on the "new" host.
> >>>>>
> >>>>>Stefan
> >>>>>
> >>>
> >>--
> >>Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-08 19:23                                                 ` Stefan Priebe
  2014-02-10  9:30                                                   ` Dr. David Alan Gilbert
@ 2014-02-10 16:07                                                   ` Dr. David Alan Gilbert
  2014-02-10 18:53                                                     ` Stefan Priebe
  2014-02-11 13:32                                                   ` Orit Wasserman
  2 siblings, 1 reply; 50+ messages in thread
From: Dr. David Alan Gilbert @ 2014-02-10 16:07 UTC (permalink / raw)
  To: Stefan Priebe; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER

* Stefan Priebe (s.priebe@profihost.ag) wrote:
> i could fix it by explicitly disable xbzrle - it seems its
> automatically on if i do not set the migration caps to false.
> 
> So it seems to be a xbzrle bug.

Stefan can you give me some more info on your hardware and
migration setup;   that stressapptest (which is a really nice
find!) really batters the memory and it means the migration
isn't converging for me, so I'm curious what your setup is.

  What CPU have you got?
  How many cores are you giving each guest?
  What network technology are you migrating over?
  Other than xbzrle what else do you have enabled?
  How long is the migrate taking for you?

Thanks,

Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-10 16:07                                                   ` Dr. David Alan Gilbert
@ 2014-02-10 18:53                                                     ` Stefan Priebe
  2014-02-13 20:06                                                       ` Dr. David Alan Gilbert
  0 siblings, 1 reply; 50+ messages in thread
From: Stefan Priebe @ 2014-02-10 18:53 UTC (permalink / raw)
  To: Dr. David Alan Gilbert; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER

Am 10.02.2014 17:07, schrieb Dr. David Alan Gilbert:
> * Stefan Priebe (s.priebe@profihost.ag) wrote:
>> i could fix it by explicitly disable xbzrle - it seems its
>> automatically on if i do not set the migration caps to false.
>>
>> So it seems to be a xbzrle bug.
>
> Stefan can you give me some more info on your hardware and
> migration setup;   that stressapptest (which is a really nice
> find!) really batters the memory and it means the migration
> isn't converging for me, so I'm curious what your setup is.

That one is devlopment by google and known to me since a few years. 
Google has detected that memtest and co are not good enough to stress 
test memory.

>    What CPU have you got?

Dual Xeon E5-2695v2

>    How many cores are you giving each guest?

16

>    What network technology are you migrating over?

10Gb/s

>    Other than xbzrle what else do you have enabled?

nothing

>    How long is the migrate taking for you?

with migration_downtime => 4s around 10s

Stefan

> Thanks,
>
> Dave
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-10 18:53                                                     ` Stefan Priebe
@ 2014-02-13 20:06                                                       ` Dr. David Alan Gilbert
  2014-02-13 20:26                                                         ` Stefan Priebe
                                                                           ` (2 more replies)
  0 siblings, 3 replies; 50+ messages in thread
From: Dr. David Alan Gilbert @ 2014-02-13 20:06 UTC (permalink / raw)
  To: Stefan Priebe; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER, owasserm

* Stefan Priebe (s.priebe@profihost.ag) wrote:
> Am 10.02.2014 17:07, schrieb Dr. David Alan Gilbert:
> >* Stefan Priebe (s.priebe@profihost.ag) wrote:
> >>i could fix it by explicitly disable xbzrle - it seems its
> >>automatically on if i do not set the migration caps to false.
> >>
> >>So it seems to be a xbzrle bug.
> >
> >Stefan can you give me some more info on your hardware and
> >migration setup;   that stressapptest (which is a really nice
> >find!) really batters the memory and it means the migration
> >isn't converging for me, so I'm curious what your setup is.
> 
> That one is devlopment by google and known to me since a few years.
> Google has detected that memtest and co are not good enough to
> stress test memory.

Hi Stefan,
  I've just posted a patch to qemu-devel that fixes two bugs that
we found; I've only tried a small stressapptest run and it seems
to survive with them (where it didn't before);  you might like to try
it if you're up for rebuilding qemu.

It's the one entitled ' [PATCH] Fix two XBZRLE corruption issues'

I'll try and get a larger run done myself, but I'd be interested to
hear if it fixes it for you (or anyone else who hit the problem).

Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-13 20:06                                                       ` Dr. David Alan Gilbert
@ 2014-02-13 20:26                                                         ` Stefan Priebe
  2014-02-13 20:31                                                         ` Stefan Priebe
  2014-02-13 21:18                                                         ` Stefan Priebe
  2 siblings, 0 replies; 50+ messages in thread
From: Stefan Priebe @ 2014-02-13 20:26 UTC (permalink / raw)
  To: Dr. David Alan Gilbert
  Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER, owasserm


Am 13.02.2014 21:06, schrieb Dr. David Alan Gilbert:
> * Stefan Priebe (s.priebe@profihost.ag) wrote:
>> Am 10.02.2014 17:07, schrieb Dr. David Alan Gilbert:
>>> * Stefan Priebe (s.priebe@profihost.ag) wrote:
>>>> i could fix it by explicitly disable xbzrle - it seems its
>>>> automatically on if i do not set the migration caps to false.
>>>>
>>>> So it seems to be a xbzrle bug.
>>>
>>> Stefan can you give me some more info on your hardware and
>>> migration setup;   that stressapptest (which is a really nice
>>> find!) really batters the memory and it means the migration
>>> isn't converging for me, so I'm curious what your setup is.
>>
>> That one is devlopment by google and known to me since a few years.
>> Google has detected that memtest and co are not good enough to
>> stress test memory.
>
> Hi Stefan,
>    I've just posted a patch to qemu-devel that fixes two bugs that
> we found; I've only tried a small stressapptest run and it seems
> to survive with them (where it didn't before);  you might like to try
> it if you're up for rebuilding qemu.
>
> It's the one entitled ' [PATCH] Fix two XBZRLE corruption issues'

Thanks!

Really would love to try them but nor google nor myself can find them.

http://osdir.com/ml/qemu-devel/2014-02/

Stefan


> I'll try and get a larger run done myself, but I'd be interested to
> hear if it fixes it for you (or anyone else who hit the problem).
>
> Dave
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-13 20:06                                                       ` Dr. David Alan Gilbert
  2014-02-13 20:26                                                         ` Stefan Priebe
@ 2014-02-13 20:31                                                         ` Stefan Priebe
  2014-02-13 21:18                                                         ` Stefan Priebe
  2 siblings, 0 replies; 50+ messages in thread
From: Stefan Priebe @ 2014-02-13 20:31 UTC (permalink / raw)
  To: Dr. David Alan Gilbert
  Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER, owasserm

got it here:
http://lists.nongnu.org/archive/html/qemu-devel/2014-02/msg02341.html

will try asap

Am 13.02.2014 21:06, schrieb Dr. David Alan Gilbert:
> * Stefan Priebe (s.priebe@profihost.ag) wrote:
>> Am 10.02.2014 17:07, schrieb Dr. David Alan Gilbert:
>>> * Stefan Priebe (s.priebe@profihost.ag) wrote:
>>>> i could fix it by explicitly disable xbzrle - it seems its
>>>> automatically on if i do not set the migration caps to false.
>>>>
>>>> So it seems to be a xbzrle bug.
>>>
>>> Stefan can you give me some more info on your hardware and
>>> migration setup;   that stressapptest (which is a really nice
>>> find!) really batters the memory and it means the migration
>>> isn't converging for me, so I'm curious what your setup is.
>>
>> That one is devlopment by google and known to me since a few years.
>> Google has detected that memtest and co are not good enough to
>> stress test memory.
>
> Hi Stefan,
>    I've just posted a patch to qemu-devel that fixes two bugs that
> we found; I've only tried a small stressapptest run and it seems
> to survive with them (where it didn't before);  you might like to try
> it if you're up for rebuilding qemu.
>
> It's the one entitled ' [PATCH] Fix two XBZRLE corruption issues'
>
> I'll try and get a larger run done myself, but I'd be interested to
> hear if it fixes it for you (or anyone else who hit the problem).
>
> Dave
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-13 20:06                                                       ` Dr. David Alan Gilbert
  2014-02-13 20:26                                                         ` Stefan Priebe
  2014-02-13 20:31                                                         ` Stefan Priebe
@ 2014-02-13 21:18                                                         ` Stefan Priebe
  2014-02-14  9:06                                                           ` Dr. David Alan Gilbert
  2 siblings, 1 reply; 50+ messages in thread
From: Stefan Priebe @ 2014-02-13 21:18 UTC (permalink / raw)
  To: Dr. David Alan Gilbert
  Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER, owasserm


Am 13.02.2014 21:06, schrieb Dr. David Alan Gilbert:
> * Stefan Priebe (s.priebe@profihost.ag) wrote:
>> Am 10.02.2014 17:07, schrieb Dr. David Alan Gilbert:
>>> * Stefan Priebe (s.priebe@profihost.ag) wrote:
>>>> i could fix it by explicitly disable xbzrle - it seems its
>>>> automatically on if i do not set the migration caps to false.
>>>>
>>>> So it seems to be a xbzrle bug.
>>>
>>> Stefan can you give me some more info on your hardware and
>>> migration setup;   that stressapptest (which is a really nice
>>> find!) really batters the memory and it means the migration
>>> isn't converging for me, so I'm curious what your setup is.
>>
>> That one is devlopment by google and known to me since a few years.
>> Google has detected that memtest and co are not good enough to
>> stress test memory.
>
> Hi Stefan,
>    I've just posted a patch to qemu-devel that fixes two bugs that
> we found; I've only tried a small stressapptest run and it seems
> to survive with them (where it didn't before);  you might like to try
> it if you're up for rebuilding qemu.
>
> It's the one entitled ' [PATCH] Fix two XBZRLE corruption issues'
>
> I'll try and get a larger run done myself, but I'd be interested to
> hear if it fixes it for you (or anyone else who hit the problem).

Yes works fine - now no crash but it's sower than without XBZRLE ;-)

Without XBZRLE: i needed migrate_downtime 4 around 60s
With XBZRLE: i needed migrate_downtime 16 and 240s


> Dave
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-13 21:18                                                         ` Stefan Priebe
@ 2014-02-14  9:06                                                           ` Dr. David Alan Gilbert
  0 siblings, 0 replies; 50+ messages in thread
From: Dr. David Alan Gilbert @ 2014-02-14  9:06 UTC (permalink / raw)
  To: Stefan Priebe; +Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER, owasserm

* Stefan Priebe (s.priebe@profihost.ag) wrote:
> 
> Am 13.02.2014 21:06, schrieb Dr. David Alan Gilbert:
> >* Stefan Priebe (s.priebe@profihost.ag) wrote:
> >>Am 10.02.2014 17:07, schrieb Dr. David Alan Gilbert:
> >>>* Stefan Priebe (s.priebe@profihost.ag) wrote:
> >>>>i could fix it by explicitly disable xbzrle - it seems its
> >>>>automatically on if i do not set the migration caps to false.
> >>>>
> >>>>So it seems to be a xbzrle bug.
> >>>
> >>>Stefan can you give me some more info on your hardware and
> >>>migration setup;   that stressapptest (which is a really nice
> >>>find!) really batters the memory and it means the migration
> >>>isn't converging for me, so I'm curious what your setup is.
> >>
> >>That one is devlopment by google and known to me since a few years.
> >>Google has detected that memtest and co are not good enough to
> >>stress test memory.
> >
> >Hi Stefan,
> >   I've just posted a patch to qemu-devel that fixes two bugs that
> >we found; I've only tried a small stressapptest run and it seems
> >to survive with them (where it didn't before);  you might like to try
> >it if you're up for rebuilding qemu.
> >
> >It's the one entitled ' [PATCH] Fix two XBZRLE corruption issues'
> >
> >I'll try and get a larger run done myself, but I'd be interested to
> >hear if it fixes it for you (or anyone else who hit the problem).
> 
> Yes works fine - now no crash but it's sower than without XBZRLE ;-)
> 
> Without XBZRLE: i needed migrate_downtime 4 around 60s
> With XBZRLE: i needed migrate_downtime 16 and 240s

Hmm; how did that compare with the previous (broken) with XBZRLE
time?   (i.e. was XBZRLE always slower for you?)

If you're driving this from the hmp/command interface then
the result of the
      info migrate

command at the end of each of those runs would be interesting.

Another thing you could try is changing the xbzrle_cache_zero_page
in arch_init.c that I added so it reads as:

static void xbzrle_cache_zero_page(ram_addr_t current_addr)
{
    if (ram_bulk_stage || !migrate_use_xbzrle()) {
        return;
    }

    if (!cache_is_cached(XBZRLE.cache, current_addr)) {
        return;
    }

    /* We don't care if this fails to allocate a new cache page
     * as long as it updated an old one */
    cache_insert(XBZRLE.cache, current_addr, ZERO_TARGET_PAGE);
}

Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-08 19:23                                                 ` Stefan Priebe
  2014-02-10  9:30                                                   ` Dr. David Alan Gilbert
  2014-02-10 16:07                                                   ` Dr. David Alan Gilbert
@ 2014-02-11 13:32                                                   ` Orit Wasserman
  2014-02-11 13:33                                                     ` Stefan Priebe - Profihost AG
  2 siblings, 1 reply; 50+ messages in thread
From: Orit Wasserman @ 2014-02-11 13:32 UTC (permalink / raw)
  To: Stefan Priebe, Dr. David Alan Gilbert
  Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER

On 02/08/2014 09:23 PM, Stefan Priebe wrote:
> i could fix it by explicitly disable xbzrle - it seems its automatically on if i do not set the migration caps to false.
>
> So it seems to be a xbzrle bug.
>

XBZRLE is disabled by default (actually all capabilities are off by default)
What version of QEMU are you using that you need to disable it explicitly?
Maybe you run migration with XBZRLE and canceled it, so it stays on?

Orit

> Stefan
>
> Am 07.02.2014 21:10, schrieb Stefan Priebe:
>> Am 07.02.2014 21:02, schrieb Dr. David Alan Gilbert:
>>> * Stefan Priebe (s.priebe@profihost.ag) wrote:
>>>> anything i could try or debug? to help to find the problem?
>>>
>>> I think the most useful would be to see if the problem is
>>> a new problem in the 1.7 you're using or has existed
>>> for a while; depending on the machine type you used, it might
>>> be possible to load that image on an earlier (or newer) qemu
>>> and try the same test, however if the problem doesn't
>>> repeat reliably it can be hard.
>>
>> I've seen this first with Qemu 1.5 but was not able to reproduce it for
>> month. 1.4 was working fine.
>>
>>> If you have any way of simplifying the configuration of the
>>> VM it would be good; e.g. if you could get a failure on
>>> something without graphics (-nographic) and USB.
>>
>> Sadly not ;-(
>>
>>> Dave
>>>
>>>>
>>>> Stefan
>>>>
>>>> Am 07.02.2014 14:45, schrieb Stefan Priebe - Profihost AG:
>>>>> it's always the same "pattern" there are too many 0 instead of X.
>>>>>
>>>>> only seen:
>>>>>
>>>>> read:0x0000000000000000 ... expected:0xffffffffffffffff
>>>>>
>>>>> or
>>>>>
>>>>> read:0xffffffff00000000 ... expected:0xffffffffffffffff
>>>>>
>>>>> or
>>>>>
>>>>> read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff
>>>>>
>>>>> or
>>>>>
>>>>> read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5
>>>>>
>>>>> no idea if this helps.
>>>>>
>>>>> Stefan
>>>>>
>>>>> Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG:
>>>>>> Hi,
>>>>>> Am 07.02.2014 14:19, schrieb Paolo Bonzini:
>>>>>>> Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto:
>>>>>>>> first of all i've now a memory image of a VM where i can
>>>>>>>> reproduce it.
>>>>>>>
>>>>>>> You mean you start that VM with -incoming 'exec:cat /path/to/vm.img'?
>>>>>>> But google stress test doesn't report any error until you start
>>>>>>> migration _and_ it finishes?
>>>>>>
>>>>>> Sorry no i meant i have a VM where i saved the memory to disk - so i
>>>>>> don't need to wait hours until i can reproduce as it does not happen
>>>>>> with a fresh started VM. So it's a state file i think.
>>>>>>
>>>>>>> Another test:
>>>>>>>
>>>>>>> - start the VM with -S, migrate, do errors appear on the destination?
>>>>>>
>>>>>> I started with -S and the errors appear AFTER resuming/unpause the VM.
>>>>>> So it is fine until i resume it on the "new" host.
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>
>>> --
>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>
>

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-11 13:32                                                   ` Orit Wasserman
@ 2014-02-11 13:33                                                     ` Stefan Priebe - Profihost AG
  2014-02-11 13:45                                                       ` Orit Wasserman
  0 siblings, 1 reply; 50+ messages in thread
From: Stefan Priebe - Profihost AG @ 2014-02-11 13:33 UTC (permalink / raw)
  To: Orit Wasserman, Dr. David Alan Gilbert
  Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER


Am 11.02.2014 14:32, schrieb Orit Wasserman:
> On 02/08/2014 09:23 PM, Stefan Priebe wrote:
>> i could fix it by explicitly disable xbzrle - it seems its
>> automatically on if i do not set the migration caps to false.
>>
>> So it seems to be a xbzrle bug.
>>
> 
> XBZRLE is disabled by default (actually all capabilities are off by
> default)
> What version of QEMU are you using that you need to disable it explicitly?
> Maybe you run migration with XBZRLE and canceled it, so it stays on?

No real idea why this happens - but yes this seems to be a problem for me.

But the bug in XBZRLE is still there ;-)

Stefan

> Orit
> 
>> Stefan
>>
>> Am 07.02.2014 21:10, schrieb Stefan Priebe:
>>> Am 07.02.2014 21:02, schrieb Dr. David Alan Gilbert:
>>>> * Stefan Priebe (s.priebe@profihost.ag) wrote:
>>>>> anything i could try or debug? to help to find the problem?
>>>>
>>>> I think the most useful would be to see if the problem is
>>>> a new problem in the 1.7 you're using or has existed
>>>> for a while; depending on the machine type you used, it might
>>>> be possible to load that image on an earlier (or newer) qemu
>>>> and try the same test, however if the problem doesn't
>>>> repeat reliably it can be hard.
>>>
>>> I've seen this first with Qemu 1.5 but was not able to reproduce it for
>>> month. 1.4 was working fine.
>>>
>>>> If you have any way of simplifying the configuration of the
>>>> VM it would be good; e.g. if you could get a failure on
>>>> something without graphics (-nographic) and USB.
>>>
>>> Sadly not ;-(
>>>
>>>> Dave
>>>>
>>>>>
>>>>> Stefan
>>>>>
>>>>> Am 07.02.2014 14:45, schrieb Stefan Priebe - Profihost AG:
>>>>>> it's always the same "pattern" there are too many 0 instead of X.
>>>>>>
>>>>>> only seen:
>>>>>>
>>>>>> read:0x0000000000000000 ... expected:0xffffffffffffffff
>>>>>>
>>>>>> or
>>>>>>
>>>>>> read:0xffffffff00000000 ... expected:0xffffffffffffffff
>>>>>>
>>>>>> or
>>>>>>
>>>>>> read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff
>>>>>>
>>>>>> or
>>>>>>
>>>>>> read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5
>>>>>>
>>>>>> no idea if this helps.
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>>> Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG:
>>>>>>> Hi,
>>>>>>> Am 07.02.2014 14:19, schrieb Paolo Bonzini:
>>>>>>>> Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto:
>>>>>>>>> first of all i've now a memory image of a VM where i can
>>>>>>>>> reproduce it.
>>>>>>>>
>>>>>>>> You mean you start that VM with -incoming 'exec:cat
>>>>>>>> /path/to/vm.img'?
>>>>>>>> But google stress test doesn't report any error until you start
>>>>>>>> migration _and_ it finishes?
>>>>>>>
>>>>>>> Sorry no i meant i have a VM where i saved the memory to disk - so i
>>>>>>> don't need to wait hours until i can reproduce as it does not happen
>>>>>>> with a fresh started VM. So it's a state file i think.
>>>>>>>
>>>>>>>> Another test:
>>>>>>>>
>>>>>>>> - start the VM with -S, migrate, do errors appear on the
>>>>>>>> destination?
>>>>>>>
>>>>>>> I started with -S and the errors appear AFTER resuming/unpause
>>>>>>> the VM.
>>>>>>> So it is fine until i resume it on the "new" host.
>>>>>>>
>>>>>>> Stefan
>>>>>>>
>>>>>
>>>> -- 
>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>>
>>
> 

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-11 13:33                                                     ` Stefan Priebe - Profihost AG
@ 2014-02-11 13:45                                                       ` Orit Wasserman
  2014-02-11 14:49                                                         ` Stefan Priebe - Profihost AG
  0 siblings, 1 reply; 50+ messages in thread
From: Orit Wasserman @ 2014-02-11 13:45 UTC (permalink / raw)
  To: Stefan Priebe - Profihost AG, Dr. David Alan Gilbert
  Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER

On 02/11/2014 03:33 PM, Stefan Priebe - Profihost AG wrote:
>
> Am 11.02.2014 14:32, schrieb Orit Wasserman:
>> On 02/08/2014 09:23 PM, Stefan Priebe wrote:
>>> i could fix it by explicitly disable xbzrle - it seems its
>>> automatically on if i do not set the migration caps to false.
>>>
>>> So it seems to be a xbzrle bug.
>>>
>>
>> XBZRLE is disabled by default (actually all capabilities are off by
>> default)
>> What version of QEMU are you using that you need to disable it explicitly?
>> Maybe you run migration with XBZRLE and canceled it, so it stays on?
>
> No real idea why this happens - but yes this seems to be a problem for me.
>

I checked upstream QEMU and it is still off by default (always been)

> But the bug in XBZRLE is still there ;-)
>

We need to understand the exact scenario in order to understand the problem.

What exact version of Qemu are you using?
Can you try with the latest upstream version, there were some fixes to the
XBZRLE code?

> Stefan
>
>> Orit
>>
>>> Stefan
>>>
>>> Am 07.02.2014 21:10, schrieb Stefan Priebe:
>>>> Am 07.02.2014 21:02, schrieb Dr. David Alan Gilbert:
>>>>> * Stefan Priebe (s.priebe@profihost.ag) wrote:
>>>>>> anything i could try or debug? to help to find the problem?
>>>>>
>>>>> I think the most useful would be to see if the problem is
>>>>> a new problem in the 1.7 you're using or has existed
>>>>> for a while; depending on the machine type you used, it might
>>>>> be possible to load that image on an earlier (or newer) qemu
>>>>> and try the same test, however if the problem doesn't
>>>>> repeat reliably it can be hard.
>>>>
>>>> I've seen this first with Qemu 1.5 but was not able to reproduce it for
>>>> month. 1.4 was working fine.
>>>>
>>>>> If you have any way of simplifying the configuration of the
>>>>> VM it would be good; e.g. if you could get a failure on
>>>>> something without graphics (-nographic) and USB.
>>>>
>>>> Sadly not ;-(
>>>>
>>>>> Dave
>>>>>
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>>> Am 07.02.2014 14:45, schrieb Stefan Priebe - Profihost AG:
>>>>>>> it's always the same "pattern" there are too many 0 instead of X.
>>>>>>>
>>>>>>> only seen:
>>>>>>>
>>>>>>> read:0x0000000000000000 ... expected:0xffffffffffffffff
>>>>>>>
>>>>>>> or
>>>>>>>
>>>>>>> read:0xffffffff00000000 ... expected:0xffffffffffffffff
>>>>>>>
>>>>>>> or
>>>>>>>
>>>>>>> read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff
>>>>>>>
>>>>>>> or
>>>>>>>
>>>>>>> read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5
>>>>>>>
>>>>>>> no idea if this helps.
>>>>>>>
>>>>>>> Stefan
>>>>>>>
>>>>>>> Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG:
>>>>>>>> Hi,
>>>>>>>> Am 07.02.2014 14:19, schrieb Paolo Bonzini:
>>>>>>>>> Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto:
>>>>>>>>>> first of all i've now a memory image of a VM where i can
>>>>>>>>>> reproduce it.
>>>>>>>>>
>>>>>>>>> You mean you start that VM with -incoming 'exec:cat
>>>>>>>>> /path/to/vm.img'?
>>>>>>>>> But google stress test doesn't report any error until you start
>>>>>>>>> migration _and_ it finishes?
>>>>>>>>
>>>>>>>> Sorry no i meant i have a VM where i saved the memory to disk - so i
>>>>>>>> don't need to wait hours until i can reproduce as it does not happen
>>>>>>>> with a fresh started VM. So it's a state file i think.
>>>>>>>>
>>>>>>>>> Another test:
>>>>>>>>>
>>>>>>>>> - start the VM with -S, migrate, do errors appear on the
>>>>>>>>> destination?
>>>>>>>>
>>>>>>>> I started with -S and the errors appear AFTER resuming/unpause
>>>>>>>> the VM.
>>>>>>>> So it is fine until i resume it on the "new" host.
>>>>>>>>
>>>>>>>> Stefan
>>>>>>>>
>>>>>>
>>>>> --
>>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>>>
>>>
>>

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-11 13:45                                                       ` Orit Wasserman
@ 2014-02-11 14:49                                                         ` Stefan Priebe - Profihost AG
  0 siblings, 0 replies; 50+ messages in thread
From: Stefan Priebe - Profihost AG @ 2014-02-11 14:49 UTC (permalink / raw)
  To: Orit Wasserman, Dr. David Alan Gilbert
  Cc: Paolo Bonzini, qemu-devel, Alexandre DERUMIER

Am 11.02.2014 14:45, schrieb Orit Wasserman:
> On 02/11/2014 03:33 PM, Stefan Priebe - Profihost AG wrote:
>>
>> Am 11.02.2014 14:32, schrieb Orit Wasserman:
>>> On 02/08/2014 09:23 PM, Stefan Priebe wrote:
>>>> i could fix it by explicitly disable xbzrle - it seems its
>>>> automatically on if i do not set the migration caps to false.
>>>>
>>>> So it seems to be a xbzrle bug.
>>>>
>>>
>>> XBZRLE is disabled by default (actually all capabilities are off by
>>> default)
>>> What version of QEMU are you using that you need to disable it
>>> explicitly?
>>> Maybe you run migration with XBZRLE and canceled it, so it stays on?
>>
>> No real idea why this happens - but yes this seems to be a problem for
>> me.
>>
> 
> I checked upstream QEMU and it is still off by default (always been)

May be i had it on in the past and the VM was still running from an
older migration.

>> But the bug in XBZRLE is still there ;-)
>>
> 
> We need to understand the exact scenario in order to understand the
> problem.
> 
> What exact version of Qemu are you using?

Qemu 1.7.0

> Can you try with the latest upstream version, there were some fixes to the
> XBZRLE code?

Sadly not - i have some custom patches (not related to xbzrle) which
won't apply to current upstream.

But i could cherry-pick the ones you have in mind - if you give me the
commit ids.

Stefan

>> Stefan
>>
>>> Orit
>>>
>>>> Stefan
>>>>
>>>> Am 07.02.2014 21:10, schrieb Stefan Priebe:
>>>>> Am 07.02.2014 21:02, schrieb Dr. David Alan Gilbert:
>>>>>> * Stefan Priebe (s.priebe@profihost.ag) wrote:
>>>>>>> anything i could try or debug? to help to find the problem?
>>>>>>
>>>>>> I think the most useful would be to see if the problem is
>>>>>> a new problem in the 1.7 you're using or has existed
>>>>>> for a while; depending on the machine type you used, it might
>>>>>> be possible to load that image on an earlier (or newer) qemu
>>>>>> and try the same test, however if the problem doesn't
>>>>>> repeat reliably it can be hard.
>>>>>
>>>>> I've seen this first with Qemu 1.5 but was not able to reproduce it
>>>>> for
>>>>> month. 1.4 was working fine.
>>>>>
>>>>>> If you have any way of simplifying the configuration of the
>>>>>> VM it would be good; e.g. if you could get a failure on
>>>>>> something without graphics (-nographic) and USB.
>>>>>
>>>>> Sadly not ;-(
>>>>>
>>>>>> Dave
>>>>>>
>>>>>>>
>>>>>>> Stefan
>>>>>>>
>>>>>>> Am 07.02.2014 14:45, schrieb Stefan Priebe - Profihost AG:
>>>>>>>> it's always the same "pattern" there are too many 0 instead of X.
>>>>>>>>
>>>>>>>> only seen:
>>>>>>>>
>>>>>>>> read:0x0000000000000000 ... expected:0xffffffffffffffff
>>>>>>>>
>>>>>>>> or
>>>>>>>>
>>>>>>>> read:0xffffffff00000000 ... expected:0xffffffffffffffff
>>>>>>>>
>>>>>>>> or
>>>>>>>>
>>>>>>>> read:0x0000bf000000bf00 ... expected:0xffffbfffffffbfff
>>>>>>>>
>>>>>>>> or
>>>>>>>>
>>>>>>>> read:0x0000000000000000 ... expected:0xb5b5b5b5b5b5b5b5
>>>>>>>>
>>>>>>>> no idea if this helps.
>>>>>>>>
>>>>>>>> Stefan
>>>>>>>>
>>>>>>>> Am 07.02.2014 14:39, schrieb Stefan Priebe - Profihost AG:
>>>>>>>>> Hi,
>>>>>>>>> Am 07.02.2014 14:19, schrieb Paolo Bonzini:
>>>>>>>>>> Il 07/02/2014 14:04, Stefan Priebe - Profihost AG ha scritto:
>>>>>>>>>>> first of all i've now a memory image of a VM where i can
>>>>>>>>>>> reproduce it.
>>>>>>>>>>
>>>>>>>>>> You mean you start that VM with -incoming 'exec:cat
>>>>>>>>>> /path/to/vm.img'?
>>>>>>>>>> But google stress test doesn't report any error until you start
>>>>>>>>>> migration _and_ it finishes?
>>>>>>>>>
>>>>>>>>> Sorry no i meant i have a VM where i saved the memory to disk -
>>>>>>>>> so i
>>>>>>>>> don't need to wait hours until i can reproduce as it does not
>>>>>>>>> happen
>>>>>>>>> with a fresh started VM. So it's a state file i think.
>>>>>>>>>
>>>>>>>>>> Another test:
>>>>>>>>>>
>>>>>>>>>> - start the VM with -S, migrate, do errors appear on the
>>>>>>>>>> destination?
>>>>>>>>>
>>>>>>>>> I started with -S and the errors appear AFTER resuming/unpause
>>>>>>>>> the VM.
>>>>>>>>> So it is fine until i resume it on the "new" host.
>>>>>>>>>
>>>>>>>>> Stefan
>>>>>>>>>
>>>>>>>
>>>>>> -- 
>>>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>>>>
>>>>
>>>
> 

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry
  2014-02-07  9:31                       ` Dr. David Alan Gilbert
  2014-02-07  9:37                         ` Stefan Priebe - Profihost AG
@ 2014-02-07  9:59                         ` Marcin Gibuła
  1 sibling, 0 replies; 50+ messages in thread
From: Marcin Gibuła @ 2014-02-07  9:59 UTC (permalink / raw)
  To: Dr. David Alan Gilbert, Stefan Priebe - Profihost AG
  Cc: qemu-devel, Alexandre DERUMIER

>> You mean to reproduce?
>
> I'm more interested in seeing what type of corruption is happening;
> if you've got a test VM that corrupts memory and we can run a program
> in that vm that writes a known pattern into memory and checks it
> then see what changed after migration, it might give a clue.
>
> But obviously this would only be of any use if run on the VM that actually
> fails.

Hi,

Seeing similar issue in my company I would be happy to run such tests. 
Do you have any test suite I could run or some leads how to write it?

-- 
mg

^ permalink raw reply	[flat|nested] 50+ messages in thread

end of thread, other threads:[~2014-02-14  9:06 UTC | newest]

Thread overview: 50+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-02-05 17:51 [Qemu-devel] QEMU LIve Migration - swap_free: Bad swap file entry Stefan Priebe
2014-02-05 20:15 ` Dr. David Alan Gilbert
2014-02-06  7:20   ` Stefan Priebe - Profihost AG
2014-02-06 10:22     ` Orit Wasserman
2014-02-06 10:49       ` Stefan Priebe - Profihost AG
2014-02-06 11:14 ` [Qemu-devel] [pve-devel] " Alexandre DERUMIER
2014-02-06 11:19   ` Stefan Priebe - Profihost AG
2014-02-06 11:40     ` Alexandre DERUMIER
2014-02-06 12:10       ` Stefan Priebe - Profihost AG
2014-02-06 14:03         ` Stefan Priebe - Profihost AG
2014-02-06 14:12           ` Marcin Gibuła
2014-02-06 19:51           ` Dr. David Alan Gilbert
2014-02-06 20:00             ` Stefan Priebe
2014-02-07  8:15               ` Alexandre DERUMIER
2014-02-07  8:17                 ` Stefan Priebe - Profihost AG
2014-02-07  9:15                   ` Dr. David Alan Gilbert
2014-02-07  9:20                     ` Stefan Priebe - Profihost AG
2014-02-07  9:29                       ` Marcin Gibuła
2014-02-07  9:30                         ` Stefan Priebe - Profihost AG
2014-02-07  9:31                       ` Dr. David Alan Gilbert
2014-02-07  9:37                         ` Stefan Priebe - Profihost AG
2014-02-07 12:02                           ` Stefan Priebe - Profihost AG
2014-02-07 12:21                             ` Dr. David Alan Gilbert
2014-02-07 12:30                               ` Stefan Priebe - Profihost AG
2014-02-07 12:44                                 ` Paolo Bonzini
2014-02-07 13:04                                   ` Stefan Priebe - Profihost AG
2014-02-07 13:08                                     ` Dr. David Alan Gilbert
2014-02-07 13:10                                       ` Stefan Priebe - Profihost AG
2014-02-07 13:15                                         ` Dr. David Alan Gilbert
2014-02-07 13:21                                           ` Stefan Priebe - Profihost AG
2014-02-07 13:19                                     ` Paolo Bonzini
2014-02-07 13:39                                       ` Stefan Priebe - Profihost AG
2014-02-07 13:45                                         ` Stefan Priebe - Profihost AG
2014-02-07 19:21                                           ` Stefan Priebe
2014-02-07 20:02                                             ` Dr. David Alan Gilbert
2014-02-07 20:10                                               ` Stefan Priebe
2014-02-08 19:23                                                 ` Stefan Priebe
2014-02-10  9:30                                                   ` Dr. David Alan Gilbert
2014-02-10 16:07                                                   ` Dr. David Alan Gilbert
2014-02-10 18:53                                                     ` Stefan Priebe
2014-02-13 20:06                                                       ` Dr. David Alan Gilbert
2014-02-13 20:26                                                         ` Stefan Priebe
2014-02-13 20:31                                                         ` Stefan Priebe
2014-02-13 21:18                                                         ` Stefan Priebe
2014-02-14  9:06                                                           ` Dr. David Alan Gilbert
2014-02-11 13:32                                                   ` Orit Wasserman
2014-02-11 13:33                                                     ` Stefan Priebe - Profihost AG
2014-02-11 13:45                                                       ` Orit Wasserman
2014-02-11 14:49                                                         ` Stefan Priebe - Profihost AG
2014-02-07  9:59                         ` Marcin Gibuła

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.