[Qemu-devel] QEMU testing methodology & results

[Qemu-devel] QEMU testing methodology & results

[Roberto Paleari]

Dear QEMU developers,

we are a group of researchers working at the University of Milan,
Italy. During the last year we focused on automatic techniques to find
defects inside CPU emulators and virtualizers. Our work has been
published in different conference papers [1][2][3], and the testing
methodologies we developed allowed us to find defects in several
emulators and virtualizers, including QEMU.

In these days we were asked to publicly release our experimental
results. As these results also include several defects in QEMU, we
believed it was better to contact you before releasing this material
to the public.

For this reason, we ask to whom it may concern to contact us privately
at emufuzzer@security.dico.unimi.it to discuss about the disclosure of
these results.

Thank you very much for your help,

Footnotes:
[1]  Testing CPU Emulators (http://roberto.greyhats.it/pubs/issta09.pdf)
[2]  Testing system virtual machines
(http://roberto.greyhats.it/pubs/issta10-kemufuzzer.pdf)
[3]  A fistful of red-pills: How to automatically generate procedures
to detect CPU emulators (http://roberto.greyhats.it/pubs/woot09.pdf)

-- 
Roberto Paleari
http://roberto.greyhats.it/
PGP Key Fingerprint: F94C 1933 F61C 3948 7CDD  2C61 38C7 6116 833D D6A0

Re: [Qemu-devel] QEMU testing methodology & results

[Blue Swirl]

On Fri, Apr 8, 2011 at 10:18 AM, Roberto Paleari
<roberto@security.dico.unimi.it> wrote:
> Dear QEMU developers,
>
> we are a group of researchers working at the University of Milan,
> Italy. During the last year we focused on automatic techniques to find
> defects inside CPU emulators and virtualizers. Our work has been
> published in different conference papers [1][2][3], and the testing
> methodologies we developed allowed us to find defects in several
> emulators and virtualizers, including QEMU.

Very interesting!

The test case generation is a bit like crashme program, but more
intelligent. It would be nice to integrate something like this to QEMU
as a test suite instead of manually written assembly programs.

KEmuFuzzer seems to be more general. The approach of the patch is a
bit intrusive. But there are similarities with it and GDB interface,
tracepoints and other instrumentation needs, so it may be possible to
work out a common solution.

I don't think it is possible to avoid red pills. Even with the fastest
hardware assisted emulator, it may be possible to make a program to
detect systematic distortions in the clock speed. Lack of cache
emulation may be easy to detect. The devices that QEMU provides are so
old that a machine with those devices can be considered to be QEMU by
the red pill. And so on.

> In these days we were asked to publicly release our experimental
> results. As these results also include several defects in QEMU, we
> believed it was better to contact you before releasing this material
> to the public.
>
> For this reason, we ask to whom it may concern to contact us privately
> at emufuzzer@security.dico.unimi.it to discuss about the disclosure of
> these results.

I'd vote for full disclosure and open discussion on this list, but I
suppose the distro people may have business interests to protect.
Though the papers may already give the black hats enough ideas how to
find the defects.

Re: [Qemu-devel] QEMU testing methodology & results

[Roberto Paleari]

On Fri, Apr 8, 2011 at 9:56 PM, Blue Swirl <blauwirbel@gmail.com> wrote:
> Very interesting!

Thank you!

> KEmuFuzzer seems to be more general. The approach of the patch is a
> bit intrusive. But there are similarities with it and GDB interface,
> tracepoints and other instrumentation needs, so it may be possible to
> work out a common solution.

Yes, KEmuFuzzer is more general and more effective than EmuFuzzer, as
it can be also used to test privileged instruction.

> I don't think it is possible to avoid red pills. Even with the fastest
> hardware assisted emulator, it may be possible to make a program to
> detect systematic distortions in the clock speed. Lack of cache
> emulation may be easy to detect. The devices that QEMU provides are so
> old that a machine with those devices can be considered to be QEMU by
> the red pill. And so on.

I agree: it would be really difficult to avoid red pills at all.
However, in EmuFuzzer and KEmuFuzzer we are interested only in
"instruction-level" red pills. I mean, we focused only on possibile
defects in the implementation of the instruction semantics, so we do
not take into account possibile red pills caused by clock skews,
device IDs, and so on.

> I'd vote for full disclosure and open discussion on this list, but I
> suppose the distro people may have business interests to protect.
> Though the papers may already give the black hats enough ideas how to
> find the defects.

I agree: the papers already disclose some details that could allow
"black hats" to re-implement the same approaches and eventualy find
emulation defects. However, we prefered to contact QEMU developers
before disclosing the details of the defects we found.

Thank you!
Roberto

-- 
Roberto Paleari
http://roberto.greyhats.it/
PGP Key Fingerprint: F94C 1933 F61C 3948 7CDD  2C61 38C7 6116 833D D6A0

Re: [Qemu-devel] QEMU testing methodology & results

[Stefan Hajnoczi]

On Fri, Apr 8, 2011 at 8:18 AM, Roberto Paleari
<roberto@security.dico.unimi.it> wrote:
> For this reason, we ask to whom it may concern to contact us privately
> at emufuzzer@security.dico.unimi.it to discuss about the disclosure of
> these results.

Hi Roberto,
Any update?  Did a qemu.git committer contact you in order to handle
the issues you found?

Stefan

Re: [Qemu-devel] QEMU testing methodology & results

[Roberto Paleari]

Hi Stefan,

Not yet. I have not received any reply besides Blue Swirl's message..

Roberto

On Wed, Apr 27, 2011 at 4:46 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> Hi Roberto,
> Any update?  Did a qemu.git committer contact you in order to handle
> the issues you found?

Re: [Qemu-devel] QEMU testing methodology & results

[Blue Swirl]

On Wed, Apr 27, 2011 at 6:31 PM, Roberto Paleari
<roberto@idea.sec.dico.unimi.it> wrote:
> Hi Stefan,
>
> Not yet. I have not received any reply besides Blue Swirl's message..

Therefore, please publish the problems you found on this list so we
can start fixing them.

Re: [Qemu-devel] QEMU testing methodology & results

[Anthony Liguori]

On 04/08/2011 02:18 AM, Roberto Paleari wrote:
> Dear QEMU developers,
>
> we are a group of researchers working at the University of Milan,
> Italy. During the last year we focused on automatic techniques to find
> defects inside CPU emulators and virtualizers. Our work has been
> published in different conference papers [1][2][3], and the testing
> methodologies we developed allowed us to find defects in several
> emulators and virtualizers, including QEMU.
>
> In these days we were asked to publicly release our experimental
> results. As these results also include several defects in QEMU, we
> believed it was better to contact you before releasing this material
> to the public.

Just to be clear, at least for x86 CPU emulation, QEMU does not attempt 
to achieve perfect fidelity and has some pretty well known "security" 
issues.  For instance, it does not emulate segment register limits in 
any meaningful way.

This does not mean that the guest can break into the host, just that the 
guest protection mechanisms aren't always enforced when using TCG.

This is not an issue with KVM though.

Regards,

Anthony Liguori

>
> For this reason, we ask to whom it may concern to contact us privately
> at emufuzzer@security.dico.unimi.it to discuss about the disclosure of
> these results.
>
> Thank you very much for your help,
>
> Footnotes:
> [1]  Testing CPU Emulators (http://roberto.greyhats.it/pubs/issta09.pdf)
> [2]  Testing system virtual machines
> (http://roberto.greyhats.it/pubs/issta10-kemufuzzer.pdf)
> [3]  A fistful of red-pills: How to automatically generate procedures
> to detect CPU emulators (http://roberto.greyhats.it/pubs/woot09.pdf)
>

Re: [Qemu-devel] QEMU testing methodology & results

[Peter Maydell]

On 28 April 2011 20:44, Anthony Liguori <anthony@codemonkey.ws> wrote:
> Just to be clear, at least for x86 CPU emulation, QEMU does not attempt to
> achieve perfect fidelity

Also true for ARM CPU emulation. The theoretical aim there as far
as I'm concerned is architectural correctness -- in other words we
should be a valid implementation of the architecture, but where the
architecture says things are UNPREDICTABLE or IMPLEMENTATION DEFINED
we don't necessarily do what any particular real hardware
implementation does. Device accuracy [ie "if modelling an A8,
do what an A8 does"] is not really done at all beyond the really
obvious things like ID registers.

-- PMM

Re: [Qemu-devel] QEMU testing methodology & results

[Paolo Bonzini]

On 04/29/2011 02:17 AM, Peter Maydell wrote:
> The theoretical aim there as far
> as I'm concerned is architectural correctness -- in other words we
> should be a valid implementation of the architecture,

That's not even the case for x86.  It should be a goal, however, that 
with mainstream kernels user programs shouldn't be able to see the 
difference.  There are some known issues besides bugs, for example the 
"instruction pointer of the last FP instruction" register (!) is not 
implemented.

Paolo

Re: [Qemu-devel] QEMU testing methodology & results

[Blue Swirl]

On Fri, Apr 29, 2011 at 11:33 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> On 04/29/2011 02:17 AM, Peter Maydell wrote:
>>
>> The theoretical aim there as far
>> as I'm concerned is architectural correctness -- in other words we
>> should be a valid implementation of the architecture,
>
> That's not even the case for x86.  It should be a goal, however, that with
> mainstream kernels user programs shouldn't be able to see the difference.
>  There are some known issues besides bugs, for example the "instruction
> pointer of the last FP instruction" register (!) is not implemented.

For Sparc32 userland, I think the emulator is pretty close to correct
(or at least it could be very precise) since the instruction set is so
simple and there are very few corner cases. Maybe also for Sparc64,
where the set is not so simple anymore.

Still, things like order of exceptions may be tricky to implement
correctly. FPUs in QEMU execute synchronously to integer units.

Another case is that TCG needs to keep TBs and the instructions which
were used to generate the TBs in synch to support PC search (usually
this is needed anyway for example to support SMC on x86), but on Sparc
it should be possible to execute old code from cache while the
instructions have been modified in memory and no barrier instructions
have been issued.