QEMU System and User targets

QEMU System and User targets

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 590 bytes --]

If I am right, the softmmu/system build target for each
architecture subsumes the source of the user target. It's what I see in the
build of i386. But then, I think the user targets have some dedicated user
layer which allow a foreign architecture user binary to be relayed through
in order to map to the host. What I was wondering is, if the user layer of
qemu is used by the user target, and by user layer I mean the using
contents of linux-user, then how does the system target receive the user
programs instead of the qemu user emulation layer even within a guest that
is an entire OS?

[-- Attachment #2: Type: text/html, Size: 622 bytes --]

Re: QEMU System and User targets

[Peter Maydell]

On Thu, 15 Jul 2021 at 16:59, Kenneth Adam Miller
<kennethadammiller@gmail.com> wrote:
>
> If I am right, the softmmu/system build target for each architecture subsumes the source of the user target.

I'm not sure what you mean by "subsumes" here. Some code in
QEMU is compiled into both the system and usermode emulators
(eg most of the CPU emulation code). Some is system mode only
(eg the device models). Some is usermode only (eg the emulation
of various system calls).

> What I was wondering is, if the user layer of qemu is used by the
> user target, and by user layer I mean the using contents of linux-user,
> then how does the system target receive the user programs instead
> of the qemu user emulation layer even within a guest that is an
> entire OS?

qemu-i386 and qemu-system-i386 are different executables.
They're built differently, and what happens when a usermode
program running inside a Linux guest inside qemu-system-i386
executes a syscall instruction is completely different from
what happens when a program running on qemu-i386 executes that
instruction. Specifically, there are different versions of the
x86_cpu_do_interrupt() function: the one for system emulation does
"update the guest CPU state in the way that the real CPU does when
an int 0x80 is executed", and the one for usermode emulation does
"raise a fake exception that will cause execution to return from
the call to cpu_exec() in linux-user/i386/cpu_loop.c:cpu_loop()
so it can emulate a syscall".

thanks
-- PMM

Re: QEMU System and User targets

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 2199 bytes --]

Well certainly, I know they are different executables. I'm just trying to
understand how the different targets work.

By subsumes, I mean that just looking at the meson.build for i386, you can
see that there are files added to the i386_ss, but not visibly added to the
softmmu target. But the softmmu target has those files built whenever you
configure and build it.

And right that's what I thought. What I'm thinking is that the i386_ss is
compiled with a different implementation for those call backs based on the
target and the user emulation.

On Thu, Jul 15, 2021 at 11:17 AM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Thu, 15 Jul 2021 at 16:59, Kenneth Adam Miller
> <kennethadammiller@gmail.com> wrote:
> >
> > If I am right, the softmmu/system build target for each architecture
> subsumes the source of the user target.
>
> I'm not sure what you mean by "subsumes" here. Some code in
> QEMU is compiled into both the system and usermode emulators
> (eg most of the CPU emulation code). Some is system mode only
> (eg the device models). Some is usermode only (eg the emulation
> of various system calls).
>
> > What I was wondering is, if the user layer of qemu is used by the
> > user target, and by user layer I mean the using contents of linux-user,
> > then how does the system target receive the user programs instead
> > of the qemu user emulation layer even within a guest that is an
> > entire OS?
>
> qemu-i386 and qemu-system-i386 are different executables.
> They're built differently, and what happens when a usermode
> program running inside a Linux guest inside qemu-system-i386
> executes a syscall instruction is completely different from
> what happens when a program running on qemu-i386 executes that
> instruction. Specifically, there are different versions of the
> x86_cpu_do_interrupt() function: the one for system emulation does
> "update the guest CPU state in the way that the real CPU does when
> an int 0x80 is executed", and the one for usermode emulation does
> "raise a fake exception that will cause execution to return from
> the call to cpu_exec() in linux-user/i386/cpu_loop.c:cpu_loop()
> so it can emulate a syscall".
>
> thanks
> -- PMM
>

[-- Attachment #2: Type: text/html, Size: 2733 bytes --]

Re: QEMU System and User targets

[Peter Maydell]

On Thu, 15 Jul 2021 at 17:25, Kenneth Adam Miller
<kennethadammiller@gmail.com> wrote:
>
> Well certainly, I know they are different executables. I'm just trying to understand how the different targets work.
>
> By subsumes, I mean that just looking at the meson.build for i386, you can see that there are files added to the i386_ss, but not visibly added to the softmmu target. But the softmmu target has those files built whenever you configure and build it.

In the meson.build files, i386_ss is files built for both softmmu and user;
i386_user_ss is files built for usermode only; i386_softmmu_ss is files
built for softmmu only. target/i386/meson.build sets target_arch,
target_softmmu_arch and target_user_arch to these sourcesets.
The top level meson.build adds the relevant target_* sourcesets to the
set of sources required to build the various executables.

Some source files also use #ifdefs: you can look for ifdefs on
CONFIG_USER_ONLY and CONFIG_SOFTMMU to find code that's conditionally
compiled in different ways for the two executables.

-- PMM

Re: QEMU System and User targets

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 1384 bytes --]

Oh I didn't know that there was a i386_user_ss in order to see that it was
intended that they were shared that way, so I initially thought that
i386_ss was user only until I saw it in the build.

On Thu, Jul 15, 2021 at 11:35 AM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Thu, 15 Jul 2021 at 17:25, Kenneth Adam Miller
> <kennethadammiller@gmail.com> wrote:
> >
> > Well certainly, I know they are different executables. I'm just trying
> to understand how the different targets work.
> >
> > By subsumes, I mean that just looking at the meson.build for i386, you
> can see that there are files added to the i386_ss, but not visibly added to
> the softmmu target. But the softmmu target has those files built whenever
> you configure and build it.
>
> In the meson.build files, i386_ss is files built for both softmmu and user;
> i386_user_ss is files built for usermode only; i386_softmmu_ss is files
> built for softmmu only. target/i386/meson.build sets target_arch,
> target_softmmu_arch and target_user_arch to these sourcesets.
> The top level meson.build adds the relevant target_* sourcesets to the
> set of sources required to build the various executables.
>
> Some source files also use #ifdefs: you can look for ifdefs on
> CONFIG_USER_ONLY and CONFIG_SOFTMMU to find code that's conditionally
> compiled in different ways for the two executables.
>
> -- PMM
>

[-- Attachment #2: Type: text/html, Size: 1814 bytes --]

Re: QEMU System and User targets

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 2397 bytes --]

When I go to build the qemu softmmu target the shared files - the i386_ss
of my arch - gives problems where the build system isn't specifying the
include headers for the compiler to find the surrounding headers that
belong to different parts of the qemu library. I was able to edit my own
source only so much before recursive header dependencies had included their
own respective qemu library components subdirectories, at which point the
build fails with cannot find header. I want to know, are these shared
source set files not supposed to include those qemu subcomponents? Or
resolve a different way for each target using #ifdef guards? I would think
that this would be modularized from within the qemu subcomponent library.
And I think that configure and meson are working together to generate the
correct build commands to these shared components. So I'm unsure what to do.

On Thu, Jul 15, 2021 at 11:39 AM Kenneth Adam Miller <
kennethadammiller@gmail.com> wrote:

> Oh I didn't know that there was a i386_user_ss in order to see that it was
> intended that they were shared that way, so I initially thought that
> i386_ss was user only until I saw it in the build.
>
> On Thu, Jul 15, 2021 at 11:35 AM Peter Maydell <peter.maydell@linaro.org>
> wrote:
>
>> On Thu, 15 Jul 2021 at 17:25, Kenneth Adam Miller
>> <kennethadammiller@gmail.com> wrote:
>> >
>> > Well certainly, I know they are different executables. I'm just trying
>> to understand how the different targets work.
>> >
>> > By subsumes, I mean that just looking at the meson.build for i386, you
>> can see that there are files added to the i386_ss, but not visibly added to
>> the softmmu target. But the softmmu target has those files built whenever
>> you configure and build it.
>>
>> In the meson.build files, i386_ss is files built for both softmmu and
>> user;
>> i386_user_ss is files built for usermode only; i386_softmmu_ss is files
>> built for softmmu only. target/i386/meson.build sets target_arch,
>> target_softmmu_arch and target_user_arch to these sourcesets.
>> The top level meson.build adds the relevant target_* sourcesets to the
>> set of sources required to build the various executables.
>>
>> Some source files also use #ifdefs: you can look for ifdefs on
>> CONFIG_USER_ONLY and CONFIG_SOFTMMU to find code that's conditionally
>> compiled in different ways for the two executables.
>>
>> -- PMM
>>
>

[-- Attachment #2: Type: text/html, Size: 3096 bytes --]

Re: QEMU System and User targets

[Peter Maydell]

On Fri, 16 Jul 2021 at 16:16, Kenneth Adam Miller
<kennethadammiller@gmail.com> wrote:
>
> When I go to build the qemu softmmu target the shared files - the i386_ss of my arch - gives problems where the build system isn't specifying the include headers for the compiler to find the surrounding headers that belong to different parts of the qemu library. I was able to edit my own source only so much before recursive header dependencies had included their own respective qemu library components subdirectories, at which point the build fails with cannot find header. I want to know, are these shared source set files not supposed to include those qemu subcomponents? Or resolve a different way for each target using #ifdef guards? I would think that this would be modularized from within the qemu subcomponent library. And I think that configure and meson are working together to generate the correct build commands to these shared components. So I'm unsure what to do.

Upstream QEMU builds fine for me, so you'll need to be more
specific about what you're trying to do and how it fails.
The top level meson.build always includes the current source
directory and the top level include/ directory on the compiler
include path, which should be all you need.

thanks
-- PMM

Re: QEMU System and User targets

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 2276 bytes --]

There's a lot of files and I don't want to muddy up the discussion with too
many details.

And for sure, this is not a problem with the upstream qemu. I'm working on
adding a target, and this is just what I'm experiencing. As for my target,
it has includes that correspond to finds within sub-directories of qemu
components, and I just mean that the include directives are only the file
name (no path prefix), but such file can be found only in folders other
than the include directory. One example is qemu.h; it is in linux-user. You
can get the compilation to find exactly just that file, but it includes
other files, and it isn't reasonable to edit anything outside of my own
architecture implementation. I'm wondering if perhaps anything that makes
an include to linux-user would need to be moved into the user target source
set, because currently it is in the shared.

On Fri, Jul 16, 2021 at 10:38 AM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Fri, 16 Jul 2021 at 16:16, Kenneth Adam Miller
> <kennethadammiller@gmail.com> wrote:
> >
> > When I go to build the qemu softmmu target the shared files - the
> i386_ss of my arch - gives problems where the build system isn't specifying
> the include headers for the compiler to find the surrounding headers that
> belong to different parts of the qemu library. I was able to edit my own
> source only so much before recursive header dependencies had included their
> own respective qemu library components subdirectories, at which point the
> build fails with cannot find header. I want to know, are these shared
> source set files not supposed to include those qemu subcomponents? Or
> resolve a different way for each target using #ifdef guards? I would think
> that this would be modularized from within the qemu subcomponent library.
> And I think that configure and meson are working together to generate the
> correct build commands to these shared components. So I'm unsure what to do.
>
> Upstream QEMU builds fine for me, so you'll need to be more
> specific about what you're trying to do and how it fails.
> The top level meson.build always includes the current source
> directory and the top level include/ directory on the compiler
> include path, which should be all you need.
>
> thanks
> -- PMM
>

[-- Attachment #2: Type: text/html, Size: 2704 bytes --]

Re: QEMU System and User targets

[Peter Maydell]

On Fri, 16 Jul 2021 at 18:50, Kenneth Adam Miller
<kennethadammiller@gmail.com> wrote:
> There's a lot of files and I don't want to muddy up the discussion with too many details.

If you don't provide details, you get vague answers. Your choice :-)

> And for sure, this is not a problem with the upstream qemu. I'm working on adding a target, and this is just what I'm experiencing. As for my target, it has includes that correspond to finds within sub-directories of qemu components, and I just mean that the include directives are only the file name (no path prefix), but such file can be found only in folders other than the include directory. One example is qemu.h; it is in linux-user. You can get the compilation to find exactly just that file, but it includes other files, and it isn't reasonable to edit anything outside of my own architecture implementation. I'm wondering if perhaps anything that makes an include to linux-user would need to be moved into the user target source set, because currently it is in the shared.

The broad-strokes answer is "your code in target/whatever should generally
not be including files that are neither in include/ nor in target/whatever".
If you find yourself doing that you've probably structured something wrong
(otherwise other targets would also have run into this).

For linux-user/qemu.h in particular, the top level meson.build does
add linux-user/ to the include path, but only for when it is building
files for the linux-user targets. (It makes no sense to include that header
file into code built for system emulation.)

Of the 4 targets that #include "qemu.h" in target/whatever code, 3 of them
(m68k, nios2, arm) do it only for their semihosting .c file, and there
the #include "qemu.h" is inside an #ifdef CONFIG_USER_ONLY. (Semihosting
is a bit of an odd thing which works differently for usermode and
system emulation mode, which is why it needs this linux-user header.)
The 4th is hexagon, and that is a bug in the hexagon code which is only
going unnoticed because hexagon currently supports only the linux-user
target and not system emulation.

thanks
-- PMM

Re: QEMU System and User targets

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 2624 bytes --]

Right, that's what I was thinking, that I shouldn't be building that for
the system target. That's why I started out with the question that I did,
because I was thinking that it probably hard codes it to user emulation.
Currently though, understanding qemu internals is not so clear to me as I'm
just becoming familiar with the code base.

On Fri, Jul 16, 2021 at 1:05 PM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Fri, 16 Jul 2021 at 18:50, Kenneth Adam Miller
> <kennethadammiller@gmail.com> wrote:
> > There's a lot of files and I don't want to muddy up the discussion with
> too many details.
>
> If you don't provide details, you get vague answers. Your choice :-)
>
> > And for sure, this is not a problem with the upstream qemu. I'm working
> on adding a target, and this is just what I'm experiencing. As for my
> target, it has includes that correspond to finds within sub-directories of
> qemu components, and I just mean that the include directives are only the
> file name (no path prefix), but such file can be found only in folders
> other than the include directory. One example is qemu.h; it is in
> linux-user. You can get the compilation to find exactly just that file, but
> it includes other files, and it isn't reasonable to edit anything outside
> of my own architecture implementation. I'm wondering if perhaps anything
> that makes an include to linux-user would need to be moved into the user
> target source set, because currently it is in the shared.
>
> The broad-strokes answer is "your code in target/whatever should generally
> not be including files that are neither in include/ nor in
> target/whatever".
> If you find yourself doing that you've probably structured something wrong
> (otherwise other targets would also have run into this).
>
> For linux-user/qemu.h in particular, the top level meson.build does
> add linux-user/ to the include path, but only for when it is building
> files for the linux-user targets. (It makes no sense to include that header
> file into code built for system emulation.)
>
> Of the 4 targets that #include "qemu.h" in target/whatever code, 3 of them
> (m68k, nios2, arm) do it only for their semihosting .c file, and there
> the #include "qemu.h" is inside an #ifdef CONFIG_USER_ONLY. (Semihosting
> is a bit of an odd thing which works differently for usermode and
> system emulation mode, which is why it needs this linux-user header.)
> The 4th is hexagon, and that is a bug in the hexagon code which is only
> going unnoticed because hexagon currently supports only the linux-user
> target and not system emulation.
>
> thanks
> -- PMM
>

[-- Attachment #2: Type: text/html, Size: 3130 bytes --]

Re: QEMU System and User targets

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 2972 bytes --]

After checking around, I don't see any <target>_user_ss in any target
directory. And I only see *_user_ss in the linux-user subdirectory. Were
you talking about that meson.build in linux-user?

On Fri, Jul 16, 2021 at 1:20 PM Kenneth Adam Miller <
kennethadammiller@gmail.com> wrote:

> Right, that's what I was thinking, that I shouldn't be building that for
> the system target. That's why I started out with the question that I did,
> because I was thinking that it probably hard codes it to user emulation.
> Currently though, understanding qemu internals is not so clear to me as I'm
> just becoming familiar with the code base.
>
> On Fri, Jul 16, 2021 at 1:05 PM Peter Maydell <peter.maydell@linaro.org>
> wrote:
>
>> On Fri, 16 Jul 2021 at 18:50, Kenneth Adam Miller
>> <kennethadammiller@gmail.com> wrote:
>> > There's a lot of files and I don't want to muddy up the discussion with
>> too many details.
>>
>> If you don't provide details, you get vague answers. Your choice :-)
>>
>> > And for sure, this is not a problem with the upstream qemu. I'm working
>> on adding a target, and this is just what I'm experiencing. As for my
>> target, it has includes that correspond to finds within sub-directories of
>> qemu components, and I just mean that the include directives are only the
>> file name (no path prefix), but such file can be found only in folders
>> other than the include directory. One example is qemu.h; it is in
>> linux-user. You can get the compilation to find exactly just that file, but
>> it includes other files, and it isn't reasonable to edit anything outside
>> of my own architecture implementation. I'm wondering if perhaps anything
>> that makes an include to linux-user would need to be moved into the user
>> target source set, because currently it is in the shared.
>>
>> The broad-strokes answer is "your code in target/whatever should generally
>> not be including files that are neither in include/ nor in
>> target/whatever".
>> If you find yourself doing that you've probably structured something wrong
>> (otherwise other targets would also have run into this).
>>
>> For linux-user/qemu.h in particular, the top level meson.build does
>> add linux-user/ to the include path, but only for when it is building
>> files for the linux-user targets. (It makes no sense to include that
>> header
>> file into code built for system emulation.)
>>
>> Of the 4 targets that #include "qemu.h" in target/whatever code, 3 of them
>> (m68k, nios2, arm) do it only for their semihosting .c file, and there
>> the #include "qemu.h" is inside an #ifdef CONFIG_USER_ONLY. (Semihosting
>> is a bit of an odd thing which works differently for usermode and
>> system emulation mode, which is why it needs this linux-user header.)
>> The 4th is hexagon, and that is a bug in the hexagon code which is only
>> going unnoticed because hexagon currently supports only the linux-user
>> target and not system emulation.
>>
>> thanks
>> -- PMM
>>
>

[-- Attachment #2: Type: text/html, Size: 3727 bytes --]

Re: QEMU System and User targets

[Peter Maydell]

On Fri, 16 Jul 2021 at 21:06, Kenneth Adam Miller
<kennethadammiller@gmail.com> wrote:
>
> After checking around, I don't see any <target>_user_ss in any target directory. And I only see *_user_ss in the linux-user subdirectory. Were you talking about that meson.build in linux-user?

$ git grep _user_ss target
target/i386/meson.build:i386_user_ss = ss.source_set()
target/i386/meson.build:target_user_arch += {'i386': i386_user_ss}
target/i386/tcg/user/meson.build:i386_user_ss.add(when: ['CONFIG_TCG',
'CONFIG_USER_ONLY'], if_true: files(
target/mips/meson.build:mips_user_ss = ss.source_set()
target/mips/meson.build:target_user_arch += {'mips': mips_user_ss}
target/mips/tcg/user/meson.build:mips_user_ss.add(files(
target/s390x/meson.build:s390x_user_ss = ss.source_set()
target/s390x/meson.build:s390x_user_ss.add(files(
target/s390x/meson.build:target_user_arch += {'s390x': s390x_user_ss}

i386, mips and s390 all have a foo_user_ss sourceset defined in
their target/foo/meson.build files.

You don't need to use the *_user_ss sourceset if you don't have
any files which should only be compiled for the user emulator,
which is why most of the targets don't use it. Having files
to be compiled only for softmmu is much more common.

-- PMM