[Qemu-devel] [Bug 1156313] [NEW] X86-64 flags handling broken

[Qemu-devel] [Bug 1156313] [NEW] X86-64 flags handling broken

[Torbjorn Granlund]

Public bug reported:

The current qemu sources cause improper handling of flags on x86-64.
This bug seems to have shown up a few weeks ago.

A plain install of Debian GNU/Linux makes user processes catch
spurious signals.  The kernel seems to run stably, though.

The ADX feature works very poorly.  It might be related; at least it
allows for reproducibly provoking invalid behaviour.

Here is a test case:

================================================================
qemumain.c
#include <stdio.h>
long adx();
int
main ()
{
  printf ("%lx\n", adx (0xffbeef, 17));
  return 0;
}
================================================================
qemuadx.s:
        .globl  adx
adx:    xor     %rax, %rax
1:      dec     %rdi
        jnz     1b
        .byte 0xf3, 0x48, 0x0f, 0x38, 0xf6, 0xc0        # adox  %rax, %rax
        .byte 0x66, 0x48, 0x0f, 0x38, 0xf6, 0xc0        # adcx  %rax, %rax
        ret
================================================================

Compile and execute:
$ gcc -m64 qemumain.c qemuadx.s
$ a.out
ffffff8000378cd8

Expected output is simply "0".  The garbage value varies between qemu
compiles and guest systems.

Note that one needs a recent GNU assembler in order to handle adox and
adcx.  For convenience I have supplied them as byte sequences.

Exaplanation and feeble analysis:

The 0xffbeef argument is a loop count.  It is necessary to loop for a
while in order to trigger this bug.  If the loop count is decreased,
the bug will seen intermittently; the lower the count, the less
frequent the invalid behaviour.

It seems like a reasonable assumption that this bug is related to
flags handling at context switch.  Presumably, qemu keeps flags state
in some internal format, then recomputes then when needing to form the
eflags register, as needed for example for context switching.

I haven't tried to reproduce this bug using qemu-x86_64 and SYSROOT,
but I strongly suspect that to be impossible.  I use
qemu-system-x86_64 and the guest Debian GNU/Linux x86_64 (version
6.0.6) .

The bug happens also with the guest FreeBSD x86_64 version 9.1.  (The
iteration count for triggering the problem 50% of the runs is not the
same when using the kernel Linux and FreeBSD's kernel, presumably due
to different ticks.)

The bug happens much more frequently for a loaded system; in fact, the
loop count can be radically decreased if two instances of the trigger
program are run in parallel.

** Affects: qemu
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1156313

Title:
  X86-64 flags handling broken

Status in QEMU:
  New

Bug description:
  The current qemu sources cause improper handling of flags on x86-64.
  This bug seems to have shown up a few weeks ago.

  A plain install of Debian GNU/Linux makes user processes catch
  spurious signals.  The kernel seems to run stably, though.

  The ADX feature works very poorly.  It might be related; at least it
  allows for reproducibly provoking invalid behaviour.

  Here is a test case:

  ================================================================
  qemumain.c
  #include <stdio.h>
  long adx();
  int
  main ()
  {
    printf ("%lx\n", adx (0xffbeef, 17));
    return 0;
  }
  ================================================================
  qemuadx.s:
          .globl  adx
  adx:    xor     %rax, %rax
  1:      dec     %rdi
          jnz     1b
          .byte 0xf3, 0x48, 0x0f, 0x38, 0xf6, 0xc0        # adox  %rax, %rax
          .byte 0x66, 0x48, 0x0f, 0x38, 0xf6, 0xc0        # adcx  %rax, %rax
          ret
  ================================================================

  Compile and execute:
  $ gcc -m64 qemumain.c qemuadx.s
  $ a.out
  ffffff8000378cd8

  Expected output is simply "0".  The garbage value varies between qemu
  compiles and guest systems.

  Note that one needs a recent GNU assembler in order to handle adox and
  adcx.  For convenience I have supplied them as byte sequences.

  Exaplanation and feeble analysis:

  The 0xffbeef argument is a loop count.  It is necessary to loop for a
  while in order to trigger this bug.  If the loop count is decreased,
  the bug will seen intermittently; the lower the count, the less
  frequent the invalid behaviour.

  It seems like a reasonable assumption that this bug is related to
  flags handling at context switch.  Presumably, qemu keeps flags state
  in some internal format, then recomputes then when needing to form the
  eflags register, as needed for example for context switching.

  I haven't tried to reproduce this bug using qemu-x86_64 and SYSROOT,
  but I strongly suspect that to be impossible.  I use
  qemu-system-x86_64 and the guest Debian GNU/Linux x86_64 (version
  6.0.6) .

  The bug happens also with the guest FreeBSD x86_64 version 9.1.  (The
  iteration count for triggering the problem 50% of the runs is not the
  same when using the kernel Linux and FreeBSD's kernel, presumably due
  to different ticks.)

  The bug happens much more frequently for a loaded system; in fact, the
  loop count can be radically decreased if two instances of the trigger
  program are run in parallel.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1156313/+subscriptions

Re: [Qemu-devel] [Bug 1156313] [NEW] X86-64 flags handling broken

[Richard Henderson]

On 03/17/2013 12:34 PM, Torbjorn Granlund wrote:
> Public bug reported:
> 
> The current qemu sources cause improper handling of flags on x86-64.
> This bug seems to have shown up a few weeks ago.
> 
> A plain install of Debian GNU/Linux makes user processes catch
> spurious signals.  The kernel seems to run stably, though.
> 
> The ADX feature works very poorly.  It might be related; at least it
> allows for reproducibly provoking invalid behaviour.

Patch at http://patchwork.ozlabs.org/patch/229139/

Knowing what the problem is, I'm able to reliably trigger the problem
with the userland emulator.  It's simply a matter of understanding
how to force the right garbage values into the qemu state.  E.g.

	.globl	adx, main
	.type	adx, @function
	.type	main, @function

adx:	mov	$0xdeadbeef, %eax
	add	$0, %eax
	mov	$0, %eax
	jmp	1f
1:	adox	%rax, %rax
	adcx	%rax, %rax
	ret

main:	call	adx
	lea	string(%rip), %rdi
	mov	%rax, %rsi
	call	printf
	xor	%eax, %eax
	ret

.rodata:
string:	.asciz	"result %lx\n"

will always print deadbeef before the fix.  Thus while having a context
switch inside the loop tended to cause unpredictable behaviour, I believe
it to be a red herring.


r~

Re: [Qemu-devel] [Bug 1156313] [NEW] X86-64 flags handling broken

[Torbjorn Granlund]

[-- Attachment #1: Type: text/plain, Size: 751 bytes --]

Richard Henderson <rth@twiddle.net> writes:

  Patch at http://patchwork.ozlabs.org/patch/229139/
  
Thanks.  I can confirm that this fixes the bug triggered by my test case
(and yours).  However, the instability of Debian GNU/Linux x86_64 has
not improved.

The exact same Debian version (debian "testing") updated at the same
time runs well on hardware.

My qemu Debian system now got messed up, since I attempted an upgrade in
the buggy qemu, which segfaulted several times during the upgrade.  I
need to reinstall, and then rely on -snapshot.

There is a problem with denorms which is reproducible, but whether that
is a qemu bug, and whether it can actually cause the observed
instability, is questionable.  Here is a testcase for that problem:


[-- Attachment #2: qemu-denorm-problem.s --]
[-- Type: application/octet-stream, Size: 279 bytes --]

	.text
	.globl	main
	.type	main, @function
main:
	mov	$0x10000000000000, %rax		# DBL_MIN
	movq	%rax, %xmm0
	movabsq	$0x4000000000000000, %rax	# 2.0
	movq	%rax, %xmm1
	divsd	%xmm1, %xmm0
	xorpd	%xmm1, %xmm1

1:	ucomisd	%xmm0, %xmm1
	je	1b

	mov	$0, %eax
	ret

	.size	main, .-main

[-- Attachment #3: Type: text/plain, Size: 669 bytes --]


It should terminate.  The observed buggy behaviour is that it hangs.

The instability problem can be observed at gmplib.org/devel/tm-date.html.
hwl-deb.gmplib.org is Debian under qemu with -cpu Haswell,+adx.

Not that the exact same qemu runs FreeBSD flawlessly (hwl.gmplib.org).
It is neither instable nor does it run the denorms testcase poorly.

I fully realise this is a hopeless bug report, but I am sure you can
reproduce it, since it is far from GMP specific.  After all apt-get
update; apt-get upgrade triggered it.  Debugging it will be a nightmare.

Qemu version: main git repo from less than a week ago + Richard ADX
patch.

-- 
Torbjörn

Re: [Qemu-devel] [Bug 1156313] [NEW] X86-64 flags handling broken

[Torbjorn Granlund]

After the latest patch of Richard's, qemu-system-x86_64 now runs
GNU/Linux properly.  The denorm problem is gone, and there seem to be no
(or at least much fewer) spurious segfaults.

This ends a multi-month period of qemu-system-x86_64 instability.

Still qemu-system-mips64, qemu-system-mips64el, and qemu-system-arm
give spurious errors.  The former two have never (at least not the last
3-4 years) been able to run a GMP

  configure && make && make check

reliably.   No regression there, just maintained buggyness...

-- 
Torbjörn

Re: [Qemu-devel] [Bug 1156313] [NEW] X86-64 flags handling broken

[Peter Maydell]

On 25 March 2013 14:06, Torbjorn Granlund <tg@gmplib.org> wrote:
> Still qemu-system-mips64, qemu-system-mips64el, and qemu-system-arm
> give spurious errors.

If you can provide a reasonably sized test case for
qemu-system-arm I can investigate that one.

thanks
-- PMM

Re: [Qemu-devel] [Bug 1156313] [NEW] X86-64 flags handling broken

[Torbjorn Granlund]

Peter Maydell <peter.maydell@linaro.org> writes:

  On 25 March 2013 14:06, Torbjorn Granlund <tg@gmplib.org> wrote:
  > Still qemu-system-mips64, qemu-system-mips64el, and qemu-system-arm
  > give spurious errors.
  
  If you can provide a reasonably sized test case for
  qemu-system-arm I can investigate that one.
  
I don't know how to make a nice bug report about instability.  Here is
my best effort:

A Debian install, adding the packages gcc, g++, m4, and make, then
downloading GMP (e.g., the latest release) then attempt a GMP build:

     configure && make

This will fail most of the time and in non-reproducible ways.

Last night's build give a segfault and an illop in sh during configure:

http://gmplib.org/devel/testmachines/build/failure/army.gmplib.org:standard.txt

This is certainly not GMP-related, since it is (1) during build and (2)
non-reproducible.

What a "Debian install" means is of course quite unclear.  It is a
fiddly process with many dead ends, not helped by misleading and
incomplete instructions with dead links on the Net.  And passing any
-cpu option to qemu typically leads to early hangs.

I followed Aurélien Jarno's instructions (unfortunately with many dead
links) at http://www.aurel32.net/info/debian_arm_qemu.php, then managed
a painful upgrade route from there to a less obsolete system.  The
upgrade is painful because so many things that ought to work just causes
non-boot.  The newest Debian-packaged kernel that I lastly managed to
make work at all with qemu was "3.0.0-1".

I boot the instable system using this qemu command:

qemu-system-arm -M versatilepb -m 256 \
  -net nic -net tap \
  -kernel vmlinuz-3.0.0-1-versatile -initrd initrd.img-3.0.0-1-versatile \
  -hda arm.img -append "root=/dev/sda1" \
  -display vnc=localhost:30

(Note that my interest is not in playing with qemu testing and compiling
custom kernels.  My interest is improving the test environment for GMP.
Since qemu is very fragile, where the many options are a veritable
minefield, finding some setup which both installs and then boots is very
time consuming.  I tried a fresh install now, to create something more
obviously reproducible, before sending this message, but I didn't manage
to navigate through the minefield today, and have now run out of time.
My working install of arm is not new, and upgrading it also hits mines.
Sorry about that.)

-- 
Torbjörn

[Qemu-devel] [Bug 1156313] Re: X86-64 flags handling broken

[Peter Maydell]

It looks from this bug that we fixed the initial ADOX bug in commit
c53de1a2896cc (2013), and I've just tried the 'qemu-denorm-problem.s'
test case from comment #1 and it works OK, so I think we've fixed that
denormals bug too. Given that, and that this bug report is 4 years old,
I'm going to close it. If you're still having problems with recent
versions of QEMU, please open a new bug.


** Changed in: qemu
       Status: New => Fix Released

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1156313

Title:
  X86-64 flags handling broken

Status in QEMU:
  Fix Released

Bug description:
  The current qemu sources cause improper handling of flags on x86-64.
  This bug seems to have shown up a few weeks ago.

  A plain install of Debian GNU/Linux makes user processes catch
  spurious signals.  The kernel seems to run stably, though.

  The ADX feature works very poorly.  It might be related; at least it
  allows for reproducibly provoking invalid behaviour.

  Here is a test case:

  ================================================================
  qemumain.c
  #include <stdio.h>
  long adx();
  int
  main ()
  {
    printf ("%lx\n", adx (0xffbeef, 17));
    return 0;
  }
  ================================================================
  qemuadx.s:
          .globl  adx
  adx:    xor     %rax, %rax
  1:      dec     %rdi
          jnz     1b
          .byte 0xf3, 0x48, 0x0f, 0x38, 0xf6, 0xc0        # adox  %rax, %rax
          .byte 0x66, 0x48, 0x0f, 0x38, 0xf6, 0xc0        # adcx  %rax, %rax
          ret
  ================================================================

  Compile and execute:
  $ gcc -m64 qemumain.c qemuadx.s
  $ a.out
  ffffff8000378cd8

  Expected output is simply "0".  The garbage value varies between qemu
  compiles and guest systems.

  Note that one needs a recent GNU assembler in order to handle adox and
  adcx.  For convenience I have supplied them as byte sequences.

  Exaplanation and feeble analysis:

  The 0xffbeef argument is a loop count.  It is necessary to loop for a
  while in order to trigger this bug.  If the loop count is decreased,
  the bug will seen intermittently; the lower the count, the less
  frequent the invalid behaviour.

  It seems like a reasonable assumption that this bug is related to
  flags handling at context switch.  Presumably, qemu keeps flags state
  in some internal format, then recomputes then when needing to form the
  eflags register, as needed for example for context switching.

  I haven't tried to reproduce this bug using qemu-x86_64 and SYSROOT,
  but I strongly suspect that to be impossible.  I use
  qemu-system-x86_64 and the guest Debian GNU/Linux x86_64 (version
  6.0.6) .

  The bug happens also with the guest FreeBSD x86_64 version 9.1.  (The
  iteration count for triggering the problem 50% of the runs is not the
  same when using the kernel Linux and FreeBSD's kernel, presumably due
  to different ticks.)

  The bug happens much more frequently for a loaded system; in fact, the
  loop count can be radically decreased if two instances of the trigger
  program are run in parallel.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1156313/+subscriptions