Re: [Qemu-devel] E5-2620v2 - emulation stop error

[Andrey Korolyov <andrey@xdel.ru>]

[-- Attachment #1: Type: text/plain, Size: 5196 bytes --]

On Thu, Mar 26, 2015 at 5:47 AM, Bandan Das <bsd@redhat.com> wrote:
> Hi Andrey,
>
> Andrey Korolyov <andrey@xdel.ru> writes:
>
>> On Mon, Mar 16, 2015 at 10:17 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
>>> For now, it looks like bug have a mixed Murphy-Heisenberg nature, as
>>> it appearance is very rare (compared to the number of actual launches)
>>> and most probably bounded to the physical characteristics of my
>>> production nodes. As soon as I reach any reproducible path for a
>>> regular workstation environment, I`ll let everyone know. Also I am
>>> starting to think that issue can belong to the particular motherboard
>>> firmware revision, despite fact that the CPU microcode is the same
>>> everywhere.
>
> I will take the risk and say this - "could it be a processor bug ?" :)
>
>>
>> Hello everyone, I`ve managed to reproduce this issue
>> *deterministically* with latest seabios with smp fix and 3.18.3. The
>> error occuring just *once* per vm until hypervisor reboots, at least
>> in my setup, this is definitely crazy...
>>
>> - launch two VMs (Centos 7 in my case),
>> - wait a little while they are booting,
>> - attach serial console (I am using virsh list for this exact purpose),
>> - issue acpi reboot or reset, does not matter,
>> - VM always hangs at boot, most times with sgabios initialization
>> string printed out [1], but sometimes it hangs a bit later [2],
>> - no matter how many times I try to relaunch the QEMU afterwards, the
>> issue does not appear on VM which experienced problem once;
>> - trace and sample args can be seen in [3] and [4] respectively.
>
> My system is a Dell R720 dual socket which has 2620v2s. I tried your
> setup but couldn't reproduce (my qemu cmdline isn't exactly the same
> as yours), although, if you could simplify your command line a bit,
> I can try again.
>
> Bandan
>
>> 1)
>> Google, Inc.
>> Serial Graphics Adapter 06/11/14
>> SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $
>> (pbuilder@zorak) Wed Jun 11 05:57:34 UTC 2014
>> Term: 211x62
>> 4 0
>>
>> 2)
>> Google, Inc.
>> Serial Graphics Adapter 06/11/14
>> SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $
>> (pbuilder@zorak) Wed Jun 11 05:57:34 UTC 2014
>> Term: 211x62
>> 4 0
>> [...empty screen...]
>> SeaBIOS (version 1.8.1-20150325_230423-testnode)
>> Machine UUID 3c78721f-7317-4f85-bcbe-f5ad46d293a1
>>
>>
>> iPXE (http://ipxe.org) 00:02.0 C100 PCI2.10 PnP PMM+3FF95BA0+3FEF5BA0 C10
>>
>> 3)
>>
>> KVM internal error. Suberror: 2
>> extra data[0]: 800000ef
>> extra data[1]: 80000b0d
>> EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
>> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006d2c
>> EIP=0000d331 EFL=00010202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>> ES =0000 00000000 0000ffff 00009300
>> CS =f000 000f0000 0000ffff 00009b00
>> SS =0000 00000000 0000ffff 00009300
>> DS =0000 00000000 0000ffff 00009300
>> FS =0000 00000000 0000ffff 00009300
>> GS =0000 00000000 0000ffff 00009300
>> LDT=0000 00000000 0000ffff 00008200
>> TR =0000 00000000 0000ffff 00008b00
>> GDT=     000f6cb0 00000037
>> IDT=     00000000 000003ff
>> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>> DR3=0000000000000000
>> DR6=00000000ffff0ff0 DR7=0000000000000400
>> EFER=0000000000000000
>> Code=66 c3 cd 02 cb cd 10 cb cd 13 cb cd 15 cb cd 16 cb cd 18 cb <cd>
>> 19 cb cd 1c cb cd 4a cb fa fc 66 ba 47 d3 0f 00 e9 ad fe f3 90 f0 0f
>> ba 2d d4 fe fb 3f
>>
>> 4)
>> /usr/bin/qemu-system-x86_64 -name centos71 -S -machine
>> pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -bios
>> /usr/share/seabios/bios.bin -m 1024 -realtime mlock=off -smp
>> 12,sockets=1,cores=12,threads=12 -uuid
>> 3c78721f-7317-4f85-bcbe-f5ad46d293a1 -nographic -no-user-config
>> -nodefaults -device sga -chardev
>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/centos71.monitor,server,nowait
>> -mon chardev=charmonitor,id=monitor,mode=control -rtc
>> base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard
>> -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
>> PIIX4_PM.disable_s4=1 -boot strict=on -device
>> nec-usb-xhci,id=usb,bus=pci.0,addr=0x3 -device
>> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
>> file=rbd:dev-rack2/centos7-1.raw:id=qemukvm:key=XXXXXXXXXXXXXXXXXXXXXXXXXX:auth_supported=cephx\;none:mon_host=10.6.0.1\:6789\;10.6.0.3\:6789\;10.6.0.4\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writeback,aio=native
>> -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>> -chardev pty,id=charserial0 -device
>> isa-serial,chardev=charserial0,id=serial0 -chardev
>> socket,id=charchannel0,path=/var/lib/libvirt/qemu/centos71.sock,server,nowait
>> -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.1
>> -msg timestamp=on

Hehe, 2.2 works just perfectly but 2.1 isn`t. I`ll bisect the issue in
a next couple of days and post the right commit (but as can remember
none of commits b/w 2.1 and 2.2 can fix simular issue by a purpose).
I`ve attached a reference xml to simplify playing with libvirt if
anyone willing to do so.

[-- Attachment #2: centos71.xml --]
[-- Type: text/xml, Size: 3392 bytes --]

<domain type='kvm' id='2' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>centos71</name>
  <uuid>3c78721f-7317-4f85-bcbe-f5ad46d293a1</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static' cpuset='0-11'>12</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
    <loader>/usr/share/seabios/bios.bin</loader>
    <boot dev='hd'/>
    <bios useserial='yes'/>
  </os>
  <features>
    <acpi/>
    <apic eoi='on'/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>SandyBridge</model>
    <vendor>Intel</vendor>
    <topology sockets='1' cores='12' threads='12'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback' io='native'/>
      <auth username='qemukvm'>
        <secret type='ceph' uuid='ec833836-71ae-f516-0666-7d2f6e123097'/>
      </auth>
      <source protocol='rbd' name='dev-rack2/centos7-1.raw'>
        <host name='10.6.0.1' port='6789'/>
        <host name='10.6.0.3' port='6789'/>
        <host name='10.6.0.4' port='6789'/>
      </source>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <iotune>
        <read_bytes_sec>30000000</read_bytes_sec>
        <write_bytes_sec>15000000</write_bytes_sec>
        <read_iops_sec>100</read_iops_sec>
        <write_iops_sec>50</write_iops_sec>
      </iotune>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='nec-xhci'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target type='isa-serial' port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/centos71.sock'/>
      <target type='virtio' name='org.qemu.guest_agent.1'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <memballoon model='none'>
      <alias name='balloon0'/>
    </memballoon>
  </devices>
  <seclabel type='none'/>
  <qemu:commandline>
    <qemu:arg value='-chardev'/>
    <qemu:arg value='file,path=/tmp/seabioslog-vm1.log,id=seabios'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='isa-debugcon,iobase=0x402,chardev=seabios'/>
  </qemu:commandline>
</domain>