Re: Qemu TCG Plugins - how to access guest registers?

Re: Qemu TCG Plugins - how to access guest registers?

[Alex Bennée]

Lukas Straub <lukasstraub2@web.de> writes:

> On Fri, 27 Mar 2020 15:40:38 -0600
> Benjamin <benjamin@thatjames.org> wrote:
>
>> Qemu version 4.2.0 includes new functionality for something called TCG
>> Plugins. There are a few examples in the tests/plugins directory, and the
>> API is more or less defined in qemu-plugin.h.
>> 
>> This file defines two enumerated types, "qemu_plugin_cb_flags" and
>> "qemu_plugin_mem_rw", which are passed into functions that register
>> callbacks. These enums seem to indicate whether the callbacks will read or
>> write CPU registers or memory. However, all of the example plugins use
>> "QEMU_PLUGIN_CB_NO_REGS", and only 2 of the plugins use the memory access
>> enum. hotpages.c and mem.c use "QEMU_PLUGIN_MEM_RW" as the default for
>> registering a memory callback (qemu_plugin_register_vcpu_mem_cb). mem.c has
>> an argument when the plugin is loaded to choose if it's read or write,
>> however, it doesn't seem to make any difference in the callback
>> function.

You are quite right the flags exist so the TCG can be told if the
callback is going to mess with machine state - they map to the internal
TCG_CALL_* flags which are used for the same reason for the internal
helper functions.

>> My question is, how do I access the guest memory and registers from the
>> plugin callback function? The API seems to indicate that it is possible,
>> since the callback registering requires you to say if you will access them,
>> and if it's RW or just read.

Currently we document what plugins can do in docs/devel/tcg-plugins.rst:

  TCG plugins are unable to change the system state only monitor it
  passively. However they can do this down to an individual instruction
  granularity including potentially subscribing to all load and store
  operations.

This was a conscious decision to avoid plugins being used as an end-run
around the GPL to implement proprietary out-of-tree extensions.

That is not to say we might relax the rules in the future - it was a
topic of discussion at the maintainers summit and we wanted to document
some guidelines around interfacing with QEMU so people didn't get the
wrong idea about what these APIs provide (c.f. multi-process qemu and
vhost-user which could also be abused in similar ways).

Indeed Emilio's original tree did contain these more invasive hooks and
while we deliberate upstream it will hopefully not be as hard to
maintain a light out-of-tree fork should you need such functionality now.

>> Are there any examples of using this part of the API? I realize this is a
>> very new part of Qemu functionality.

So far the examples represent the totality of the API that has been
exercised and I'm keen we add more as new extensions to the API are
added. As to the specific question about accessing register values that
is exactly down to the newness of the API.

Register access is a tricky problem because of the fact that QEMU
supports so many guest architectures. I wasn't keen on slapping in a
functional but ugly API that hard-coded values like gdb register ids so
I left it out for the time being. I'd happily accept patches to add that
functionality assuming it meets the projects quality and taste
guidelines.

Some approaches we could take include:

  - hook into tcg_global_*_new and allow plugin to introspect registers
  - hook into gdbstub in some way

The first approach elides over the fact that a lot of registers aren't
actually known to the TCG - pretty much all vector registers tend to be
loaded into anonymous TCG temps as we go. Maybe this could just be fixed
by just providing a few more registrations functions at the start even
if the TCG itself wouldn't use that knowledge.

The gdbstub provides a lot more visibility of register state and for
some architectures this can be quite comprehensive - for example in
system mode you can access pretty much every ARM co-processor register.
However the gdb interface is a little clunky as there are a lot of magic
register id numbers and the canonical way to enumerate this is to chew
through a bunch of XML that each target generates (see
gdb_register_coprocessor). I'm not keen on exposing that pile of XML via
the register interface. Maybe there is scope to improve our internal
APIs so the enumeration of registers can be handled by helpers that
record mappings and ids and generate the XML for the gdbstub centrally?

There may be other approaches we could take and I'm open to suggestions.

>> 
>> Thanks
>
> CC'ing the maintainer of TCG Plugins.


-- 
Alex Bennée

Re: Qemu TCG Plugins - how to access guest registers?

[Benjamin]

[-- Attachment #1: Type: text/plain, Size: 4855 bytes --]

Thanks for your quick response.

On Mon, Mar 30, 2020 at 9:15 AM Alex Bennée <alex.bennee@linaro.org> wrote:

>
> Lukas Straub <lukasstraub2@web.de> writes:
>
> >> My question is, how do I access the guest memory and registers from the
> >> plugin callback function? The API seems to indicate that it is possible,
> >> since the callback registering requires you to say if you will access
> them,
> >> and if it's RW or just read.
>
> Currently we document what plugins can do in docs/devel/tcg-plugins.rst:
>
>   TCG plugins are unable to change the system state only monitor it
>   passively. However they can do this down to an individual instruction
>   granularity including potentially subscribing to all load and store
>   operations.
>
> This was a conscious decision to avoid plugins being used as an end-run
> around the GPL to implement proprietary out-of-tree extensions.
>
> That is not to say we might relax the rules in the future - it was a
> topic of discussion at the maintainers summit and we wanted to document
> some guidelines around interfacing with QEMU so people didn't get the
> wrong idea about what these APIs provide (c.f. multi-process qemu and
> vhost-user which could also be abused in similar ways).
>

I understand restricting the API so that the system state cannot be
changed, only inspected.  I should have been more specific in my question.
I am attempting to create a plugin that tracks all memory accesses, so I
can emulate cache behavior.  The reason I would like to read the registers,
is because many load/store instructions depend on register values, which I
can only be sure of at run-time.
Some of the concepts you mentioned I am not very familiar with; I am simply
emulating an ARM A9 processor in bare-metal mode, to give you a point of
reference of my use case.


> Indeed Emilio's original tree did contain these more invasive hooks and
> while we deliberate upstream it will hopefully not be as hard to
> maintain a light out-of-tree fork should you need such functionality now.
>

Could you please point me towards this tree?  I haven't run across it yet
in my investigating of all of this.


> >> Are there any examples of using this part of the API? I realize this is
> a
> >> very new part of Qemu functionality.
>
> So far the examples represent the totality of the API that has been
> exercised and I'm keen we add more as new extensions to the API are
> added. As to the specific question about accessing register values that
> is exactly down to the newness of the API.
>
> Register access is a tricky problem because of the fact that QEMU
> supports so many guest architectures. I wasn't keen on slapping in a
> functional but ugly API that hard-coded values like gdb register ids so
> I left it out for the time being. I'd happily accept patches to add that
> functionality assuming it meets the projects quality and taste
> guidelines.
>
> Some approaches we could take include:
>
>   - hook into tcg_global_*_new and allow plugin to introspect registers
>   - hook into gdbstub in some way
>
> The first approach elides over the fact that a lot of registers aren't
> actually known to the TCG - pretty much all vector registers tend to be
> loaded into anonymous TCG temps as we go. Maybe this could just be fixed
> by just providing a few more registrations functions at the start even
> if the TCG itself wouldn't use that knowledge.
>
> The gdbstub provides a lot more visibility of register state and for
> some architectures this can be quite comprehensive - for example in
> system mode you can access pretty much every ARM co-processor register.
> However the gdb interface is a little clunky as there are a lot of magic
> register id numbers and the canonical way to enumerate this is to chew
> through a bunch of XML that each target generates (see
> gdb_register_coprocessor). I'm not keen on exposing that pile of XML via
> the register interface. Maybe there is scope to improve our internal
> APIs so the enumeration of registers can be handled by helpers that
> record mappings and ids and generate the XML for the gdbstub centrally?
>
> There may be other approaches we could take and I'm open to suggestions.
>

I'd be happy to look into ways to implement this functionality.  However, I
just started using Qemu this year, so these things sound like they would
have a steep learning curve for me.
The gdbstub approach seems like it would provide the most introspection
ability.  What would you suggest as a starting point for looking into this?
All of this being said, if you think my project is too complicated, to
implement a cache emulator with TCG plugins, then I could always try just
hacking together some custom helper functions.


> --
> Alex Bennée
>

Thanks

[-- Attachment #2: Type: text/html, Size: 5965 bytes --]

Re: Qemu TCG Plugins - how to access guest registers?

[Alex Bennée]

Benjamin <benjamin@thatjames.org> writes:

> Thanks for your quick response.
>
> On Mon, Mar 30, 2020 at 9:15 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>
>>
>> Lukas Straub <lukasstraub2@web.de> writes:
>>
>> >> My question is, how do I access the guest memory and registers from the
>> >> plugin callback function? The API seems to indicate that it is possible,
>> >> since the callback registering requires you to say if you will access
>> them,
>> >> and if it's RW or just read.
>>
>> Currently we document what plugins can do in docs/devel/tcg-plugins.rst:
>>
>>   TCG plugins are unable to change the system state only monitor it
>>   passively. However they can do this down to an individual instruction
>>   granularity including potentially subscribing to all load and store
>>   operations.
>>
>> This was a conscious decision to avoid plugins being used as an end-run
>> around the GPL to implement proprietary out-of-tree extensions.
>>
>> That is not to say we might relax the rules in the future - it was a
>> topic of discussion at the maintainers summit and we wanted to document
>> some guidelines around interfacing with QEMU so people didn't get the
>> wrong idea about what these APIs provide (c.f. multi-process qemu and
>> vhost-user which could also be abused in similar ways).
>>
>
> I understand restricting the API so that the system state cannot be
> changed, only inspected.  I should have been more specific in my question.
> I am attempting to create a plugin that tracks all memory accesses, so I
> can emulate cache behavior.

Emulate or measure? By the way there is a GSoC project proposal to add
this:

  https://wiki.qemu.org/Internships/ProjectIdeas/CacheModelling

> The reason I would like to read the registers,
> is because many load/store instructions depend on register values, which I
> can only be sure of at run-time.

You don't need the registers at that point because at run time QEMU will
have already resolved the address and will pass it via the
qemu_plugin_register_vcpu_mem_cb. The hotpages and mem plugin examples
demonstrate the use of the API.


> Some of the concepts you mentioned I am not very familiar with; I am simply
> emulating an ARM A9 processor in bare-metal mode, to give you a point of
> reference of my use case.
>
>
>> Indeed Emilio's original tree did contain these more invasive hooks and
>> while we deliberate upstream it will hopefully not be as hard to
>> maintain a light out-of-tree fork should you need such functionality now.
>>
>
> Could you please point me towards this tree?  I haven't run across it yet
> in my investigating of all of this.

His tree is at:

  https://github.com/cota/qemu

But I'm not sure what his latest branch is. I've CC'd him.

>
>
>> >> Are there any examples of using this part of the API? I realize this is
>> a
>> >> very new part of Qemu functionality.
>>
>> So far the examples represent the totality of the API that has been
>> exercised and I'm keen we add more as new extensions to the API are
>> added. As to the specific question about accessing register values that
>> is exactly down to the newness of the API.
>>
>> Register access is a tricky problem because of the fact that QEMU
>> supports so many guest architectures. I wasn't keen on slapping in a
>> functional but ugly API that hard-coded values like gdb register ids so
>> I left it out for the time being. I'd happily accept patches to add that
>> functionality assuming it meets the projects quality and taste
>> guidelines.
>>
>> Some approaches we could take include:
>>
>>   - hook into tcg_global_*_new and allow plugin to introspect registers
>>   - hook into gdbstub in some way
>>
>> The first approach elides over the fact that a lot of registers aren't
>> actually known to the TCG - pretty much all vector registers tend to be
>> loaded into anonymous TCG temps as we go. Maybe this could just be fixed
>> by just providing a few more registrations functions at the start even
>> if the TCG itself wouldn't use that knowledge.
>>
>> The gdbstub provides a lot more visibility of register state and for
>> some architectures this can be quite comprehensive - for example in
>> system mode you can access pretty much every ARM co-processor register.
>> However the gdb interface is a little clunky as there are a lot of magic
>> register id numbers and the canonical way to enumerate this is to chew
>> through a bunch of XML that each target generates (see
>> gdb_register_coprocessor). I'm not keen on exposing that pile of XML via
>> the register interface. Maybe there is scope to improve our internal
>> APIs so the enumeration of registers can be handled by helpers that
>> record mappings and ids and generate the XML for the gdbstub centrally?
>>
>> There may be other approaches we could take and I'm open to suggestions.
>>
>
> I'd be happy to look into ways to implement this functionality.  However, I
> just started using Qemu this year, so these things sound like they would
> have a steep learning curve for me.
> The gdbstub approach seems like it would provide the most introspection
> ability.  What would you suggest as a starting point for looking into this?
> All of this being said, if you think my project is too complicated, to
> implement a cache emulator with TCG plugins, then I could always try just
> hacking together some custom helper functions.

As I said above I don't think you need register values to do cache
emulation as you have the addresses. You will need to decode some of the
cache management instructions though. Fortunately you can do that at
translation time and only instrument the ones you need. See howvec for
examples.

-- 
Alex Bennée

Re: Qemu TCG Plugins - how to access guest registers?

[Benjamin]

[-- Attachment #1: Type: text/plain, Size: 7973 bytes --]

On Mon, Mar 30, 2020 at 1:37 PM Alex Bennée <alex.bennee@linaro.org> wrote:

>
> Benjamin <benjamin@thatjames.org> writes:
>
> > Thanks for your quick response.
> >
> > On Mon, Mar 30, 2020 at 9:15 AM Alex Bennée <alex.bennee@linaro.org>
> wrote:
> >
> >>
> >> Lukas Straub <lukasstraub2@web.de> writes:
> >>
> >> >> My question is, how do I access the guest memory and registers from
> the
> >> >> plugin callback function? The API seems to indicate that it is
> possible,
> >> >> since the callback registering requires you to say if you will access
> >> them,
> >> >> and if it's RW or just read.
> >>
> >> Currently we document what plugins can do in docs/devel/tcg-plugins.rst:
> >>
> >>   TCG plugins are unable to change the system state only monitor it
> >>   passively. However they can do this down to an individual instruction
> >>   granularity including potentially subscribing to all load and store
> >>   operations.
> >>
> >> This was a conscious decision to avoid plugins being used as an end-run
> >> around the GPL to implement proprietary out-of-tree extensions.
> >>
> >> That is not to say we might relax the rules in the future - it was a
> >> topic of discussion at the maintainers summit and we wanted to document
> >> some guidelines around interfacing with QEMU so people didn't get the
> >> wrong idea about what these APIs provide (c.f. multi-process qemu and
> >> vhost-user which could also be abused in similar ways).
> >>
> >
> > I understand restricting the API so that the system state cannot be
> > changed, only inspected.  I should have been more specific in my
> question.
> > I am attempting to create a plugin that tracks all memory accesses, so I
> > can emulate cache behavior.
>
> Emulate or measure? By the way there is a GSoC project proposal to add
> this:
>
>   https://wiki.qemu.org/Internships/ProjectIdeas/CacheModelling
>

Definitely emulate - I already have C code that can track which memory
addresses have been loaded into the cache (though not the actual data).
The idea for using QEMU for this project came from a research paper I read
awhile ago, but they were using version 0.12:
https://ieeexplore.ieee.org/document/6945730
I started trying to create this with helper functions, but when I heard
about the plugins, that sounded like a cleaner way to go.
The end goal is being able to use QEMU as a fault injection platform, to
test the effects of Single-bit upsets on baremetal C code.  So I would like
some way to interact with the plugin as QEMU is running; I have some ideas
on how to do this, but none fleshed out yet.
I actually saw this morning on qemu-discuss that someone was asking if
something like Valgrind could be created using the TCG plugins, so there is
certainly interest in this kind of thing.


> > The reason I would like to read the registers,
> > is because many load/store instructions depend on register values, which
> I
> > can only be sure of at run-time.
>
> You don't need the registers at that point because at run time QEMU will
> have already resolved the address and will pass it via the
> qemu_plugin_register_vcpu_mem_cb. The hotpages and mem plugin examples
> demonstrate the use of the API.
>

The way you explained this, although it might seem simple, really helped me
understand better the level at which the TCG plugins operate.
I went and changed my code to be more based on the code in hotpages.c, and
it is much simpler now.
I'm going to go look at that Dinero Cache Simulator you linked to see if I
can get any ideas on how to improve my cache code.


> > Some of the concepts you mentioned I am not very familiar with; I am
> simply
> > emulating an ARM A9 processor in bare-metal mode, to give you a point of
> > reference of my use case.
> >
> >
> >> Indeed Emilio's original tree did contain these more invasive hooks and
> >> while we deliberate upstream it will hopefully not be as hard to
> >> maintain a light out-of-tree fork should you need such functionality
> now.
> >>
> >
> > Could you please point me towards this tree?  I haven't run across it yet
> > in my investigating of all of this.
>
> His tree is at:
>
>   https://github.com/cota/qemu
>
> But I'm not sure what his latest branch is. I've CC'd him.
>
> >
> >
> >> >> Are there any examples of using this part of the API? I realize this
> is
> >> a
> >> >> very new part of Qemu functionality.
> >>
> >> So far the examples represent the totality of the API that has been
> >> exercised and I'm keen we add more as new extensions to the API are
> >> added. As to the specific question about accessing register values that
> >> is exactly down to the newness of the API.
> >>
> >> Register access is a tricky problem because of the fact that QEMU
> >> supports so many guest architectures. I wasn't keen on slapping in a
> >> functional but ugly API that hard-coded values like gdb register ids so
> >> I left it out for the time being. I'd happily accept patches to add that
> >> functionality assuming it meets the projects quality and taste
> >> guidelines.
> >>
> >> Some approaches we could take include:
> >>
> >>   - hook into tcg_global_*_new and allow plugin to introspect registers
> >>   - hook into gdbstub in some way
> >>
> >> The first approach elides over the fact that a lot of registers aren't
> >> actually known to the TCG - pretty much all vector registers tend to be
> >> loaded into anonymous TCG temps as we go. Maybe this could just be fixed
> >> by just providing a few more registrations functions at the start even
> >> if the TCG itself wouldn't use that knowledge.
> >>
> >> The gdbstub provides a lot more visibility of register state and for
> >> some architectures this can be quite comprehensive - for example in
> >> system mode you can access pretty much every ARM co-processor register.
> >> However the gdb interface is a little clunky as there are a lot of magic
> >> register id numbers and the canonical way to enumerate this is to chew
> >> through a bunch of XML that each target generates (see
> >> gdb_register_coprocessor). I'm not keen on exposing that pile of XML via
> >> the register interface. Maybe there is scope to improve our internal
> >> APIs so the enumeration of registers can be handled by helpers that
> >> record mappings and ids and generate the XML for the gdbstub centrally?
> >>
> >> There may be other approaches we could take and I'm open to suggestions.
> >>
> >
> > I'd be happy to look into ways to implement this functionality.
> However, I
> > just started using Qemu this year, so these things sound like they would
> > have a steep learning curve for me.
> > The gdbstub approach seems like it would provide the most introspection
> > ability.  What would you suggest as a starting point for looking into
> this?
> > All of this being said, if you think my project is too complicated, to
> > implement a cache emulator with TCG plugins, then I could always try just
> > hacking together some custom helper functions.
>
> As I said above I don't think you need register values to do cache
> emulation as you have the addresses. You will need to decode some of the
> cache management instructions though. Fortunately you can do that at
> translation time and only instrument the ones you need. See howvec for
> examples.
>

I'm not familiar with cache management instructions.  What exactly do you
mean by that?  It sounds like something that would be dependent on the
guest architecture.
Or maybe it's things like pre-fetching hints?  Then the plugin would need
to take into account cache latencies, something my code doesn't deal with
right now.


> --
> Alex Bennée
>

I would be glad to share my implementation once it's in a better working
state.
Where can I find guidelines on the coding standard expected of QEMU
software?
Thanks

[-- Attachment #2: Type: text/html, Size: 10242 bytes --]

Re: Qemu TCG Plugins - how to access guest registers?

[Alex Bennée]

Benjamin <benjamin@thatjames.org> writes:

<snip>
>>
>> You don't need the registers at that point because at run time QEMU will
>> have already resolved the address and will pass it via the
>> qemu_plugin_register_vcpu_mem_cb. The hotpages and mem plugin examples
>> demonstrate the use of the API.
>>
>
> The way you explained this, although it might seem simple, really helped me
> understand better the level at which the TCG plugins operate.
> I went and changed my code to be more based on the code in hotpages.c, and
> it is much simpler now.
> I'm going to go look at that Dinero Cache Simulator you linked to see if I
> can get any ideas on how to improve my cache code.

Glad it was of help ;-)
>> > All of this being said, if you think my project is too complicated, to
>> > implement a cache emulator with TCG plugins, then I could always try just
>> > hacking together some custom helper functions.
>>
>> As I said above I don't think you need register values to do cache
>> emulation as you have the addresses. You will need to decode some of the
>> cache management instructions though. Fortunately you can do that at
>> translation time and only instrument the ones you need. See howvec for
>> examples.
>>
>
> I'm not familiar with cache management instructions.  What exactly do you
> mean by that?  It sounds like something that would be dependent on the
> guest architecture.
> Or maybe it's things like pre-fetching hints?  Then the plugin would need
> to take into account cache latencies, something my code doesn't deal with
> right now.

There are architecture dependent instructions that do things like flush
d and icaches. As QEMU doesn't expose those semantics to the plugin
(behaviour is very specific to each guest architecture) then the plugin
would have to deal with it itself. You would do this by checking each
translated instruction and inserting a call back in front of those that
do cache operations to make whatever internal changes you need to do
inside of the plugin.

> I would be glad to share my implementation once it's in a better working
> state.
> Where can I find guidelines on the coding standard expected of QEMU
> software?
> Thanks

CODING_STYLE.rst in the top level of the source tree is the canonical
document. Also "make check-tcg" will include a plugin test for each
plugin in the tree for each test (assuming you have compilers available
for each guest architecture - see docs/devel/testing.rst).

Good luck!

-- 
Alex Bennée