Re: [Qemu-devel] E5-2620v2 - emulation stop error

From: Bandan Das <bsd@redhat.com>
To: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>,
	kraxel@redhat.com, Andrey Korolyov <andrey@xdel.ru>,
	"qemu-devel\@nongnu.org" <qemu-devel@nongnu.org>,
	"kvm\@vger.kernel.org" <kvm@vger.kernel.org>
Subject: Re: [Qemu-devel] E5-2620v2 - emulation stop error
Date: Tue, 10 Mar 2015 22:38:57 -0400	[thread overview]
Message-ID: <jpgbnk01cn2.fsf@redhat.com> (raw)
In-Reply-To: <20150310202958.GR2338@work-vm> (David Alan Gilbert's message of "Tue, 10 Mar 2015 20:29:58 +0000")

"Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:

> * Paolo Bonzini (pbonzini@redhat.com) wrote:
>> 
>> 
>> On 10/03/2015 19:21, Bandan Das wrote:
>> > Paolo Bonzini <pbonzini@redhat.com> writes:
>> > 
>> >> On 10/03/2015 17:57, Dr. David Alan Gilbert wrote:
>> >>> I'm seeing something similar; it's very intermittent and generally
>> >>> happening right at boot of the guest;   I'm running this on qemu
>> >>> head+my postcopy world (but it's happening right at boot before postcopy
>> >>> gets a chance), and I'm using a 3.19ish kernel. Xeon E5-2407 in my case
>> >>> but hey maybe I'm seeing a different bug.
>> > 
>> > Probably a tangent but is the qemu trace identical to what Andrey is seeing ?
>> > From a cursory look and my limited understanding, it seems his failure is #GP
>> > when executing video bios.
>> > 
>> >> Same here on 3.16 + Xeon E5 v3 kernel.
>> > 
>> > I will try to reproduce this on a v2.
>> 
>> I see several failures, usually mine have suberror 1.  With a 32-VCPU
>> guest I can reproduce it roughly half of the time.
>> 
>> Paolo
>
> while true; do (sleep 5; echo -e '\001cq\n')|/opt/qemu-try-world3/bin/qemu-system-x86_64 -machine pc-i440fx-2.0,accel=kvm -m 1024 -smp 128 -nographic -device sga 2>&1 | tee /tmp/qemu.op; grep "internal error" /tmp/qemu.op -q && break; done
>
> (and leave about 2mins of runs before declaring good)
>
> bad: cd2946607b42636d6c8cf6dbf94bce0273507b17
> bad: 041ccc922ee474693a2869d4e3b59e920c739bc0
> bad: 2559db069628981bfdc90637fac5bf1b4f4e8ef5
> bad: 21f5826a04d38e19488f917e1eef22751490c769
> good:e95d24ff40c77fbfd71396834a2eb99375f8bcc4
> good: 7781a492fa5a2eff53d06b25b93f0186ad3226c9
> good: c3edd62851098e6417786193ed9e9341781fcf57
> good: c5c6d7f81a6950d8e32a3b5a0bafd37bfa5a8e88
> good: 73104fd399c6778112f64fe0d439319f24508d9a
> good: 92013cf8ca10adafec9a92deb5df993e7df22cb9
> good: 4478aa768ccefcc5b234c23d035435fd71b932f6
> good: 2.2.0
>
> [root@virtlab413 qemu-world3]# git bisect bad
> 21f5826a04d38e19488f917e1eef22751490c769 is the first bad commit

I can reproduce this on E5-2620 v2 with  David's "while true" test.
(The emulation failure I mean, not the suberror 2 that Andrey is seeing)
The commit that seems to have introduced this is -

commit 0673b7870063a3affbad9046fb6d385a4e734c19
Author: Kevin O'Connor <kevin@koconnor.net>
Date:   Sat May 24 10:49:50 2014 -0400

    smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.

    Change the multi-processor init code to trampoline into 32bit mode on
    each of the additional processors.  Implement an atomic lock so that
    each processor performs its initialization serially.

I am not sure what in that change could cause this though..
Also, in my testing, "unrestricted_guest=0" avoids the failure.

> commit 21f5826a04d38e19488f917e1eef22751490c769
> Author: Gerd Hoffmann <kraxel@redhat.com>
> Date:   Thu Feb 19 09:33:03 2015 +0100
>
>     seabios: update to 1.8.0 release
>     
>     'git shortlog 8936dbb2..4c59f5d8' for seabios repo:
>     
>     David Woodhouse (4):
>           Update EFI_COMPATIBILITY16_TABLE to match 0.98 spec update
>           build: use -m16 where available instead of asm(".code16gcc")
>           romlayout: Use .code16 not .code16gcc
>           vgabios: Use .code16 not .code16gcc
>     
>     Gerd Hoffmann (2):
>           add scripts/tarball.sh
>           build: set LC_ALL=C
>     
>     Hannes Reinecke (1):
>           megasas: read addional PCI I/O bar
>     
>     Ian Campbell (1):
>           romlayout: Use "rep ; nop" not "rep nop".
>     
>     Kevin O'Connor (139):
>           vgabios: Return from handle_1011() if handler found.
>           edd: Move EDD get drive parameters (int 1348) logic from disk.c to block.c.
>           edd: Use sectors==-1 to detect removable media.
>           edd: Separate out ATA and virtio specific parts of fill_edd().
>           cdemu: store internal cdemu fields in standard "el-torito" spec format.
>           Move cdemu call interface and disk_ret helper code to disk.c.
>           smm: Replace SMI assembler code with C code.
>           smm: Use a C struct to define the layout of the SMM area.
>           smp: Replace QEMU SMP init assembler code with C; run only in 32bit mode.
>           Don't enable thread preemption during S3 resume vga option rom execution.
>           Remove old Bochs bios fixed address string at 0xfff00.
>           Move most of the VAR16FIXED() defs to misc.c.
>           build: Avoid absolute paths during "whole-program" compiling.
>           Make sure handle_smi() and handle_smp() are compiled out if not enabled.
>           Remove the TODO file.
>           Abstract reset call (and possible 16bit mode switch) into reset() function.
>           build: Remove unused function getSectionsStart() from layoutrom.py.
>           build: Extract section visiting logic in layoutrom.py.
>           build: Refactor layoutrom.py gc() function.
>           build: Use customized entry point for each type of build.
>           build: Refactor findInit() function.
>           build: Rework getRelocs() to use a hash instead of categories in layoutrom.py
>           build: Keep segmented sections separate until final link step.
>           build: Use fileid instead of category to write sections in layoutrom.py.
>           build: Only export needed fields in LayoutInfo in layoutrom.py.
>           build: Get fixed address variables from 32bit compile pass (not 16bit)
>           build: Minor - fix comments referring to old tools/ directory.
>           xhci: Update the times for usb command timeouts.
>           ehci: Update usb command timeouts to use usb_xfer_time()
>           uhci: Update usb command timeouts to use usb_xfer_time()
>           ohci: Update usb command timeouts to use usb_xfer_time()
>           vgabios: Fix broken build resulting from e5749978.
>           boot: Change ":rom%d" boot order rom instance to ":rom%x"
>           Minor - remove stray tab from src/fw/smm.c.
>           build: Update kconfig to version in Linux 3.16.
>           usb: Fix usb_xfer_time() to work when called in 16bit mode.
>           xhci: Call usb_desc2pipe() on xhci_update_pipe().
>           xhci: Remove 16bit code wrappers.
>           xhci: Use high memory instead of low memory for internal storage.
>           xhci: Move root hub and setup code to top of file.
>           xhci: Add xhci_check_ports() and xhci_free_pipes() functions.
>           ehci: Move port power up from ehci_hub_detect() to check_ehci_ports().
>           usb-hub: Enable power to all ports prior to calling usb_enumerate().
>           xhci: Change xhci_hub_detect() to use connect status instead of link state.
>           uhci: Repeatedly poll for device detect for 100ms.
>           ohci: Repeatedly poll for device detect for 100ms.
>           ehci: Stall uhci/ohci init only until default port routing is done.
>           usb: Perform device detect polling on all usb controllers.
>           ehci: Fix bug in hub port assignment
>           Revert "Use the extra stack for 16bit USB and PS2 keyboard/mouse commands."
>           pmm: Fix entry point to support non-zero %ss
>           Move stack hop code below call32/call16 code in stacks.c
>           Add need_hop_back() call that determines if stack_hop_back is needed
>           Update invoke_mouse_handler() to use need_hop_back()
>           Update stack_hop_back() to jump to 16bit mode if called in 32bit mode.
>           Track when entering via call32() and use the same mode for stack_hop_back()
>           Simplify farcall16 code
>           Update reset() to use call16_back()
>           build: Support declaring 32bit C functions that must reside in the f-segment
>           Move call16() functions from romlayout.S to inline assembler in stacks.c
>           Break up call32() into call32() and call32_sloppy()
>           Fully restore 16bit state during call16_sloppy()
>           Implement call32 mechanism using SMIs.
>           Move a20 code from system.c and ps2port.h to x86.h
>           Backup and restore a20 on call32_sloppy()
>           usb: Rename ?hci_control() to ?hci_send_control()
>           usb: Rename usb_getFrameExp() to usb_get_period()
>           usb: Rename findEndPointDesc() to usb_find_desc()
>           usb: Rename send_default_control() to usb_send_default_control()
>           usb: Rename free_pipe() to usb_free_pipe()
>           usb: Clarify usb freelist manipulations
>           xhci: Change xhci_update_pipe() to xhci_realloc_pipe() and use for alloc too
>           uhci: Export uhci_realloc_pipe() instead of uhci_alloc_pipe()
>           ohci: Export ohci_realloc_pipe() instead of ohci_alloc_pipe()
>           ehci: Export ehci_realloc_pipe() instead of ehci_alloc_pipe()
>           usb: Use usb_realloc_pipe for pipe alloc, update, and free.
>           Use 32bit memcpy in int1587 when applicable
>           Don't clobber %ax on ENTRY_INTO32 macro
>           Create assembler macros for saving and restoring 'struct bregs'
>           Do full BREGS backup/restore for pmm, pnp, and irqentry_extrastack
>           Remove unused macro ENTRY_ST
>           vgabios: Don't declare custom internal BDA storage in std/bda.h
>           vgabios: Cache a pointer to the current mode struct in the BDA
>           vgabios: Don't pass vmode_g to vgafb_move_chars() / vgafb_clear_chars()
>           vgabios: Rename vbe_flags to flags
>           vgabios: Set cursor shape fixes
>           vgabios: Refactor get/set_cursor_shape() code
>           vgabios: Only init BDA device details in init_bios_area()
>           vgabios: Only set the dcc_index=8 if stdvga ports are available
>           vgabios: Move standard table definitions to std/vga.h
>           vgabios: Fill in available legacy modes in video_func_static at runtime
>           vgabios: Add support for reading framebuffer in "direct" mode
>           Fix PNP regression introduced in 99cb8f3e due to missed conversion
>           Minor - move PORT_PS2_CTRLB from hw/ps2port.h to hw/timer.c
>           vgabios: Support emulating text mode attributes while in graphics mode
>           vgabios: Add software cursor capability
>           Use an aligned stack offset when entering on the extra stack
>           Minor - comment updates in romlayout.S
>           Fix build issue on gcc34
>           pciinit: Fix build warning in mch_pci_slot_get_irq()
>           floppy: Make sure to yield() during floppy PIO
>           Minor - be consistent in placement of .code16/32 in romlayout.S
>           Use macros for .code16/32 mode switches in inline asm in stacks.c
>           Eliminate FUNCFSEG - only force portions of inline asm to f-segment
>           usb: Update USB hub code to support super speed hubs
>           Simplify README files - point to online documentation instead
>           sdcard: Initial support for SD cards on PCI SDHCI controllers on QEMU
>           Add wiki documentation to repository
>           docs: Don't point to repo README files
>           docs: Add info on MODE16/MODESEGMENT compile time flags
>           docs: Add page describing SeaBIOS final object linking
>           scsi: Move cdb_* functions above scsi_* functions
>           scsi: Move process_scsi_op() to hw/blockcmd.c and rename
>           cdrom: call scsi_process_op() instead of cdb_read()
>           scsi: Don't export cdb_* functions
>           cdrom: Break up very large read requests into smaller requests
>           block: Check for read/write requests over 64K
>           usb: Add support for OHCI bulk transfers
>           readserial: Enhance pipe support
>           docs: Add documentation on using readserial.py script
>           uhci: Enable "depth" tree traversal for bulk transfers
>           uhci: Increase bulk transfer STACKTDS to 16
>           vgabios: Support emulated text in gfx_read_char()
>           ehci: No need to support td array wrapping
>           ehci: Simplify fillTDbuffer() and rename
>           ehci: Merge ehci_send_control with ehci_send_bulk
>           ohci: Merge ohci_send_control with ohci_send_bulk
>           uhci: Merge uhci_send_control with uhci_send_bulk
>           xhci: Merge xhci_send_control with xhci_send_bulk
>           usb: Use usb_send_pipe() now that all drivers have x_send_pipe()
>           xhci: Move xhci_xfer_x() functions together
>           xhci: Merge some xhci_xfer_x() functions into xhci_send_pipe()
>           usb: Control transfers always have an 8 byte command size
>           usb: Minor - properly free memory on get_device_config() error path
>           checkstack: Handle callw instruction
>           docs: Document why v1.6.3 release came after v0.6.2
>           docs: Update release history with dates of stable releases
>           docs: There is only one VAR16 flag now
>           docs: Note v1.8.0 release
>     
>     Marcel Apfelbaum (1):
>           hw/pci: reserve IO and mem for pci express downstream ports with no devices attached
>     
>     Markus Armbruster (1):
>           boot: Fix boot order for SCSI target, lun > 9
>     
>     Paolo Bonzini (5):
>           piix: add and use dev-piix.h
>           smm: complete SMM setup
>           smm: unify SMM handlers
>           vgabios: fix graphics operation with Bochs VGA in non-DISPI modes
>           vgabios: implement read char in graphics mode
>     
>     zhanghailiang (1):
>           acpi: use specified macro instead of magic-number
>     
>     Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
>
>
>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK