Access target TranslatorOps

Access target TranslatorOps

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 255 bytes --]

Hello,

I would like to be able to, from the linux-user/main.c, access the target's
registered TranslatorOps instance. How would I do that when 1) the TCG is
correctly initialized and ready to run 2) before QEMU starts to run or when
it is safely paused?

[-- Attachment #2: Type: text/html, Size: 299 bytes --]

Re: Access target TranslatorOps

[Alex Bennée]

Kenneth Adam Miller <kennethadammiller@gmail.com> writes:

> Hello,
>
> I would like to be able to, from the linux-user/main.c, access the target's registered TranslatorOps instance. How would I
> do that when 1) the TCG is correctly initialized and ready to run 2)
> before QEMU starts to run or when it is safely paused?

Why would you want to mess with the TranslatorOps?

If you want to do some sort of analysis you might want to consider:

  https://qemu.readthedocs.io/en/latest/devel/tcg-plugins.html

At which point you can hook into any translation or individual instructions.

-- 
Alex Bennée

Re: Access target TranslatorOps

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 1129 bytes --]

That I know of, the TCG plugins do not allow me to feed the QEMU instance
dynamically changing opcodes. I wouldn't use TranslatorOps if I don't have
to. I want to facilitate a use case in which the contents of the target
being emulated are changing, but it is not a self modifying target. I have
to query and interact with the TCG to find out what opcodes are supported
or not.

On Wed, Jul 20, 2022 at 4:46 AM Alex Bennée <alex.bennee@linaro.org> wrote:

>
> Kenneth Adam Miller <kennethadammiller@gmail.com> writes:
>
> > Hello,
> >
> > I would like to be able to, from the linux-user/main.c, access the
> target's registered TranslatorOps instance. How would I
> > do that when 1) the TCG is correctly initialized and ready to run 2)
> > before QEMU starts to run or when it is safely paused?
>
> Why would you want to mess with the TranslatorOps?
>
> If you want to do some sort of analysis you might want to consider:
>
>   https://qemu.readthedocs.io/en/latest/devel/tcg-plugins.html
>
> At which point you can hook into any translation or individual
> instructions.
>
> --
> Alex Bennée
>

[-- Attachment #2: Type: text/html, Size: 1646 bytes --]

Re: Access target TranslatorOps

[Peter Maydell]

On Wed, 20 Jul 2022 at 17:39, Kenneth Adam Miller
<kennethadammiller@gmail.com> wrote:
> That I know of, the TCG plugins do not allow me to feed the
> QEMU instance dynamically changing opcodes. I wouldn't use
> TranslatorOps if I don't have to. I want to facilitate a
> use case in which the contents of the target being emulated
> are changing, but it is not a self modifying target. I have
> to query and interact with the TCG to find out what opcodes
> are supported or not.

I agree that feeding opcodes into the translator isn't what
TCG plugins are intended for.

I'm definitely not clear on what you're trying to do here,
so it's hard to suggest some other approach, but linux-user
code shouldn't be messing with the internals of the translator
by grabbing the TranslatorOps struct. Among other things,
linux-user code is runtime and TranslatorOps is for
translate-time.

Sometimes code in linux-user needs to be a bit over-familiar
with the CPU state, but we try to keep that to a minimum.
Generally that involves code in target/foo/ providing some
set of interface functions that code in linux-user/foo/
can work with, typically passing it the CPU state struct.

thanks
-- PMM

Re: Access target TranslatorOps

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 2099 bytes --]

I need to determine the set of instruction encodings that the TCG can
support for a given platform. I am not bothered whether the target runs at
all, and in fact it is better if it doesn't, so runtime or translate time
doesn't bother me.

Imagine I were adding support for more instructions for a given platform. I
would like to check that I'm using the API right. It's amazing that it's
been so far and there's no way to check that the correct behavior occurs
when a given encoding is encountered regarding the TCG. A boolean result
from a can_translate called just when the target encounters the instruction
would be good. Additionally, the ability to force the translation of
arbitrary encodings would be good. I would like to not have to engineer
some binary file format.

On Wed, Jul 20, 2022 at 1:37 PM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Wed, 20 Jul 2022 at 17:39, Kenneth Adam Miller
> <kennethadammiller@gmail.com> wrote:
> > That I know of, the TCG plugins do not allow me to feed the
> > QEMU instance dynamically changing opcodes. I wouldn't use
> > TranslatorOps if I don't have to. I want to facilitate a
> > use case in which the contents of the target being emulated
> > are changing, but it is not a self modifying target. I have
> > to query and interact with the TCG to find out what opcodes
> > are supported or not.
>
> I agree that feeding opcodes into the translator isn't what
> TCG plugins are intended for.
>
> I'm definitely not clear on what you're trying to do here,
> so it's hard to suggest some other approach, but linux-user
> code shouldn't be messing with the internals of the translator
> by grabbing the TranslatorOps struct. Among other things,
> linux-user code is runtime and TranslatorOps is for
> translate-time.
>
> Sometimes code in linux-user needs to be a bit over-familiar
> with the CPU state, but we try to keep that to a minimum.
> Generally that involves code in target/foo/ providing some
> set of interface functions that code in linux-user/foo/
> can work with, typically passing it the CPU state struct.
>
> thanks
> -- PMM
>

[-- Attachment #2: Type: text/html, Size: 2637 bytes --]

Re: Access target TranslatorOps

[Alex Bennée]

Kenneth Adam Miller <kennethadammiller@gmail.com> writes:

> I need to determine the set of instruction encodings that the TCG can support for a given platform. I am not bothered
> whether the target runs at all, and in fact it is better if it
> doesn't, so runtime or translate time doesn't bother me.

Which architectures are you interested in? For the ones that have been
converted to use decode tree it should be easy enough to update the
script to emit the uncovered opcode space. However decode tree targets
regular encoding - I think it has gained support for multiple encoding
modes but I don't know if it can handle the irregular madness of x86.

> Imagine I were adding support for more instructions for a given platform. I would like to check that I'm using the API
> right. It's amazing that it's been so far and there's no way to check that the correct behavior occurs when a given
> encoding is encountered regarding the TCG. A boolean result from a can_translate called just when the target encounters
> the instruction would be good.

Generally when the translator encounters an instruction it can't
translate it would emit a illegal instruction exception. While you might
be able to peek into the TCG opcode stream to see such calls to the
relevant helpers I doubt it would be up-streamable as each front end
will deal with illegal instructions their own way (including
instructions that are illegal due to the current CPU operating mode).

> Additionally, the ability to force the translation of arbitrary encodings would be good. I
> would like to not have to engineer some binary file format.

You don't need a new binary file format - just to construct an ELF with
the stream you want. A possibly adjacent project you might want to look
at is RISU:

  https://git.linaro.org/people/peter.maydell/risu.git/about/

which we've used for testing the range of the translator for a number of
architectures.

>
> On Wed, Jul 20, 2022 at 1:37 PM Peter Maydell <peter.maydell@linaro.org> wrote:
>
>  On Wed, 20 Jul 2022 at 17:39, Kenneth Adam Miller
>  <kennethadammiller@gmail.com> wrote:
>  > That I know of, the TCG plugins do not allow me to feed the
>  > QEMU instance dynamically changing opcodes. I wouldn't use
>  > TranslatorOps if I don't have to. I want to facilitate a
>  > use case in which the contents of the target being emulated
>  > are changing, but it is not a self modifying target. I have
>  > to query and interact with the TCG to find out what opcodes
>  > are supported or not.
>
>  I agree that feeding opcodes into the translator isn't what
>  TCG plugins are intended for.
>
>  I'm definitely not clear on what you're trying to do here,
>  so it's hard to suggest some other approach, but linux-user
>  code shouldn't be messing with the internals of the translator
>  by grabbing the TranslatorOps struct. Among other things,
>  linux-user code is runtime and TranslatorOps is for
>  translate-time.
>
>  Sometimes code in linux-user needs to be a bit over-familiar
>  with the CPU state, but we try to keep that to a minimum.
>  Generally that involves code in target/foo/ providing some
>  set of interface functions that code in linux-user/foo/
>  can work with, typically passing it the CPU state struct.
>
>  thanks
>  -- PMM


-- 
Alex Bennée

Re: Access target TranslatorOps

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 3980 bytes --]

Oh whoa, I thought I could have an architecture neutral way to interface
with the TCG to find this out.

Yes, I do have to use the decode tree, and converting the script to output
the codes would suffice for my case. However, I do not know how to do that
at the moment. I've tried my best to understand the TCG documentation but
this appears to not be too straightforward.


On Fri, Jul 22, 2022 at 5:31 AM Alex Bennée <alex.bennee@linaro.org> wrote:

>
> Kenneth Adam Miller <kennethadammiller@gmail.com> writes:
>
> > I need to determine the set of instruction encodings that the TCG can
> support for a given platform. I am not bothered
> > whether the target runs at all, and in fact it is better if it
> > doesn't, so runtime or translate time doesn't bother me.
>
> Which architectures are you interested in? For the ones that have been
> converted to use decode tree it should be easy enough to update the
> script to emit the uncovered opcode space. However decode tree targets
> regular encoding - I think it has gained support for multiple encoding
> modes but I don't know if it can handle the irregular madness of x86.
>
> > Imagine I were adding support for more instructions for a given
> platform. I would like to check that I'm using the API
> > right. It's amazing that it's been so far and there's no way to check
> that the correct behavior occurs when a given
> > encoding is encountered regarding the TCG. A boolean result from a
> can_translate called just when the target encounters
> > the instruction would be good.
>
> Generally when the translator encounters an instruction it can't
> translate it would emit a illegal instruction exception. While you might
> be able to peek into the TCG opcode stream to see such calls to the
> relevant helpers I doubt it would be up-streamable as each front end
> will deal with illegal instructions their own way (including
> instructions that are illegal due to the current CPU operating mode).
>
> > Additionally, the ability to force the translation of arbitrary
> encodings would be good. I
> > would like to not have to engineer some binary file format.
>
> You don't need a new binary file format - just to construct an ELF with
> the stream you want. A possibly adjacent project you might want to look
> at is RISU:
>
>   https://git.linaro.org/people/peter.maydell/risu.git/about/
>
> which we've used for testing the range of the translator for a number of
> architectures.
>
> >
> > On Wed, Jul 20, 2022 at 1:37 PM Peter Maydell <peter.maydell@linaro.org>
> wrote:
> >
> >  On Wed, 20 Jul 2022 at 17:39, Kenneth Adam Miller
> >  <kennethadammiller@gmail.com> wrote:
> >  > That I know of, the TCG plugins do not allow me to feed the
> >  > QEMU instance dynamically changing opcodes. I wouldn't use
> >  > TranslatorOps if I don't have to. I want to facilitate a
> >  > use case in which the contents of the target being emulated
> >  > are changing, but it is not a self modifying target. I have
> >  > to query and interact with the TCG to find out what opcodes
> >  > are supported or not.
> >
> >  I agree that feeding opcodes into the translator isn't what
> >  TCG plugins are intended for.
> >
> >  I'm definitely not clear on what you're trying to do here,
> >  so it's hard to suggest some other approach, but linux-user
> >  code shouldn't be messing with the internals of the translator
> >  by grabbing the TranslatorOps struct. Among other things,
> >  linux-user code is runtime and TranslatorOps is for
> >  translate-time.
> >
> >  Sometimes code in linux-user needs to be a bit over-familiar
> >  with the CPU state, but we try to keep that to a minimum.
> >  Generally that involves code in target/foo/ providing some
> >  set of interface functions that code in linux-user/foo/
> >  can work with, typically passing it the CPU state struct.
> >
> >  thanks
> >  -- PMM
>
>
> --
> Alex Bennée
>

[-- Attachment #2: Type: text/html, Size: 4957 bytes --]

Re: Access target TranslatorOps

[Alex Bennée]

Kenneth Adam Miller <kennethadammiller@gmail.com> writes:

> Oh whoa, I thought I could have an architecture neutral way to
> interface with the TCG to find this out.

While the TCG intermediates are architecture neutral there are enough
difference between the various guest architectures in the way exceptions
are raised there is no common API. Most will generate an exception with
a front end specific helper. We only define a few common exception types
that all CPUs might generate:

  #define EXCP_INTERRUPT  0x10000 /* async interruption */
  #define EXCP_HLT        0x10001 /* hlt instruction reached */
  #define EXCP_DEBUG      0x10002 /* cpu stopped after a breakpoint or singlestep */
  #define EXCP_HALTED     0x10003 /* cpu is halted (waiting for external event) */
  #define EXCP_YIELD      0x10004 /* cpu wants to yield timeslice to another */
  #define EXCP_ATOMIC     0x10005 /* stop-the-world and emulate atomic */

with the front-ends free to generate any others as they see fit.

>
> Yes, I do have to use the decode tree, and converting the script to output the codes would suffice for my case. However,
> I do not know how to do that at the moment. I've tried my best to understand the TCG documentation but this appears to
> not be too straightforward.

We've slowly been moving more stuff into the RST documentation which you
can see rendered here:

  https://qemu.readthedocs.io/en/latest/devel/index-tcg.html

but we could certainly do with adding some more to describe the general
flow of translation and execution. However if there are things that
aren't clear please to ask here and we can do our best to answer.

>
> On Fri, Jul 22, 2022 at 5:31 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>
>  Kenneth Adam Miller <kennethadammiller@gmail.com> writes:
>
>  > I need to determine the set of instruction encodings that the TCG can support for a given platform. I am not
>  bothered
>  > whether the target runs at all, and in fact it is better if it
>  > doesn't, so runtime or translate time doesn't bother me.
>
>  Which architectures are you interested in? For the ones that have been
>  converted to use decode tree it should be easy enough to update the
>  script to emit the uncovered opcode space. However decode tree targets
>  regular encoding - I think it has gained support for multiple encoding
>  modes but I don't know if it can handle the irregular madness of x86.
>
>  > Imagine I were adding support for more instructions for a given platform. I would like to check that I'm using the
>  API
>  > right. It's amazing that it's been so far and there's no way to check that the correct behavior occurs when a given
>  > encoding is encountered regarding the TCG. A boolean result from a can_translate called just when the target
>  encounters
>  > the instruction would be good.
>
>  Generally when the translator encounters an instruction it can't
>  translate it would emit a illegal instruction exception. While you might
>  be able to peek into the TCG opcode stream to see such calls to the
>  relevant helpers I doubt it would be up-streamable as each front end
>  will deal with illegal instructions their own way (including
>  instructions that are illegal due to the current CPU operating mode).
>
>  > Additionally, the ability to force the translation of arbitrary encodings would be good. I
>  > would like to not have to engineer some binary file format.
>
>  You don't need a new binary file format - just to construct an ELF with
>  the stream you want. A possibly adjacent project you might want to look
>  at is RISU:
>
>    https://git.linaro.org/people/peter.maydell/risu.git/about/
>
>  which we've used for testing the range of the translator for a number of
>  architectures.
>
>  >
>  > On Wed, Jul 20, 2022 at 1:37 PM Peter Maydell <peter.maydell@linaro.org> wrote:
>  >
>  >  On Wed, 20 Jul 2022 at 17:39, Kenneth Adam Miller
>  >  <kennethadammiller@gmail.com> wrote:
>  >  > That I know of, the TCG plugins do not allow me to feed the
>  >  > QEMU instance dynamically changing opcodes. I wouldn't use
>  >  > TranslatorOps if I don't have to. I want to facilitate a
>  >  > use case in which the contents of the target being emulated
>  >  > are changing, but it is not a self modifying target. I have
>  >  > to query and interact with the TCG to find out what opcodes
>  >  > are supported or not.
>  >
>  >  I agree that feeding opcodes into the translator isn't what
>  >  TCG plugins are intended for.
>  >
>  >  I'm definitely not clear on what you're trying to do here,
>  >  so it's hard to suggest some other approach, but linux-user
>  >  code shouldn't be messing with the internals of the translator
>  >  by grabbing the TranslatorOps struct. Among other things,
>  >  linux-user code is runtime and TranslatorOps is for
>  >  translate-time.
>  >
>  >  Sometimes code in linux-user needs to be a bit over-familiar
>  >  with the CPU state, but we try to keep that to a minimum.
>  >  Generally that involves code in target/foo/ providing some
>  >  set of interface functions that code in linux-user/foo/
>  >  can work with, typically passing it the CPU state struct.
>  >
>  >  thanks
>  >  -- PMM
>
>  -- 
>  Alex Bennée


-- 
Alex Bennée

Re: Access target TranslatorOps

[Peter Maydell]

On Fri, 22 Jul 2022 at 06:09, Kenneth Adam Miller
<kennethadammiller@gmail.com> wrote:
>
> I need to determine the set of instruction encodings that the TCG can support for a given platform. I am not bothered whether the target runs at all, and in fact it is better if it doesn't, so runtime or translate time doesn't bother me.

So, something like "does the emulated CPU support guest architecture
feature X" ? Look at how eg arm handles setting the Linux hwcap bits,
for instance.

> Imagine I were adding support for more instructions for a given platform.
> I would like to check that I'm using the API right. It's amazing that
> it's been so far and there's no way to check that the correct behavior
> occurs when a given encoding is encountered regarding the TCG.

The way to test "is the emulation correct" is to have test programs.
For Arm we use 'risu' to generate random instruction sequences and
check their behaviour against some golden reference, which can catch
things like "is this insn supposed to undef in this configuration".

> A boolean result from a can_translate called just when the target
> encounters the instruction would be good. Additionally, the ability
> to force the translation of arbitrary encodings would be good.

I am completely confused about what you want to do here, because
these requests just sound completely bizarre to me. The translator
is its own self-contained, and linux-user should have no requirement at
all to be told whether an instruction happens to translate to
"raise an exception" or "generate code to do something".

> Additionally, the ability
> to force the translation of arbitrary encodings would be good.

This is easy -- just put the right bytes into the test binary.

-- PMM