[Bug 1748296] Re: TCG throws Invalid Opcode when executing x86 BMI shlx instruction

[David Greenaway <1748296@bugs.launchpad.net>]

I hit this today on QEMU head. The problem appears to crop up when:

  1. Decoding a VEX instruction (see [1]) that uses the 0x66 mandatory
     prefix; and

  2. The OSFXSR bit in CR4 is clear (that is, SSE is disabled)

This means that x86_64 instructions such as:

     c4 e2 f9 f7 c0                shlxq   %rax, %rax, %rax

fail. Similar instructions the use a different mandatory prefix
(such as `shrxq`, which uses prefix 0xf2) work fine.

Most operating systems presumably set the OSFXSR bit fairly early on, which I
guess is why this problem isn't likely to be seen except in low-level or early
boot code.

The culprit appears to be the block of code in `gen_sse` [2]:

    if (is_xmm
        && !(s->flags & HF_OSFXSR_MASK)
        && ((b != 0x38 && b != 0x3a) || (s->prefix & PREFIX_DATA))) {
        goto unknown_op;
    }

Removing the check `... || (s->prefix & DATA_DATA)` causes QEMU to correctly
translate the instruction, and allows doug16k's test above to pass.

I must confess, I'm not clear what this clause was testing for. My best guess
is that early code (e.g. 4242b1bd8ac) required it to avoid accessing invalid
opcode tables, but we seem to be handling that more gracefully today (e.g.
[3]), so I suspect it is no longer needed.

[1]: https://wiki.osdev.org/X86-64_Instruction_Encoding#VEX.2FXOP_opcodes
[2]: https://github.com/qemu/qemu/blob/6b63d126121a9535784003924fcb67f574a6afc0/target/i386/tcg/translate.c#L3078
[3]: https://github.com/qemu/qemu/blob/6b63d126121a9535784003924fcb67f574a6afc0/target/i386/tcg/translate.c#L3696-L3700

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1748296

Title:
  TCG throws Invalid Opcode when executing x86 BMI shlx instruction

Status in QEMU:
  New

Bug description:
  I am unable to use BMI in my project when running under TCG. I
  narrowed the problem down to incorrect instruction decoding for BMI
  instructions (which have a 2 byte VEX prefix). The gen_sse function in
  translate.c reaches the goto label do_0f_38_fx, but b does not equal
  0x1f7, 0x2f7, or 0x3f7, so the switch takes the default path and
  raises an invalid opcode exception.

  The code executes correctly and passes the test under KVM.

  I have created a complete repro here: https://github.com/doug65536
  /qemu-bmibug

  The makefile has the following utility targets:

  debug-kvm: Build and run the VM using KVM and wait for gdbstub attach

  run: Run the test case with TCG, make fails if the test fails. (It
  will fail)

  run-kvm: Run the test case with KVM, make fails if the test fails. (It
  will succeed)

  debug: Build and run the VM with TCG and wait for GDB attach

  attach-gdb: Run GDB and attach to QEMU gdbstub

  The VM runs with -cpu max. CPUID reports support for BMI, BMI2, and
  ABM.

  You can quickly verify the issue by executing `make run-kvm` to
  confirm that KVM passes, then `make run` to confirm that TCG fails.

  I believe the bug affects other BMI, BMI2, and ABM instructions, but I
  have only completely verified incorrect execution of SHLX.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1748296/+subscriptions