Building QEMU as a shared library

Building QEMU as a shared library

[Amir Gonnen]

[-- Attachment #1: Type: text/plain, Size: 2574 bytes --]

Hi,

Before sending a patch, I would like to check if it's of interest to the community.

My goal is to simulate a mixed architecture system.
Today QEMU strongly assumes that the simulated system is a *single architecture*.
Changing this assumption and supporting mixed architecture in QEMU proved to be
non-trivial and may require significant development effort. Common code such as
TCG and others explicitly include architecture specific header files, for example.

A possible solution, discussed on https://stackoverflow.com/q/63229262/619493 is to
separate the simulation to multiple processes (as done by Xilinx) and to use some form
of Interprocess Communication channel between them.
Such solution has several disadvantages:

- Harder to synchronize simulation between processes
- Performance impact of Interprocess Communication
- Harder to debug, profile and maintain

Instead, I would like to suggest a new approach we use at Neuroblade to achieve this:
Build QEMU as a shared library that can be loaded and used directly in a larger simulation.
Today we build qemu-system-nios2 shared library and load it from qemu-system-x86_64 in order
to simulate an x86_64 system that also consists of multiple nios2 cores.
In our simulation, two independent "main" functions are running on different threads, and
simulation synchronization is reduced to synchronizing threads.

To achieve this, I needed to do the following changes in QEMU:

1. Avoid Glib global context. Use a different context (g_main_context_new) for each QEMU instance.
2. Change meson.build to build QEMU as a shared library (with PIC enabled for static libraries)
3. Define a C API for the library and export it (with a -Wl,--version-script)

These changes seem enough for simulating mixed architecture system on a single process.

If this approach sounds useful, I'll be happy to send patches.
I'd appreciate if you could provide your feedback!

Thanks,
Amir


The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited.

[-- Attachment #2: Type: text/html, Size: 5725 bytes --]

Re: Building QEMU as a shared library

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2658 bytes --]

On Wed, Dec 15, 2021 at 08:18:53AM +0000, Amir Gonnen wrote:
> Before sending a patch, I would like to check if it's of interest to the community.
> 
> My goal is to simulate a mixed architecture system.
> Today QEMU strongly assumes that the simulated system is a *single architecture*.
> Changing this assumption and supporting mixed architecture in QEMU proved to be
> non-trivial and may require significant development effort. Common code such as
> TCG and others explicitly include architecture specific header files, for example.

Hi Amir,
Simulating heterogenous machines comes up from periodically. So far no
one has upstreamed a solution but there is definitely interest.

I suggest going ahead and posting the code even if it's not cleaned up.

> A possible solution, discussed on https://stackoverflow.com/q/63229262/619493 is to
> separate the simulation to multiple processes (as done by Xilinx) and to use some form
> of Interprocess Communication channel between them.
> Such solution has several disadvantages:
> 
> - Harder to synchronize simulation between processes
> - Performance impact of Interprocess Communication
> - Harder to debug, profile and maintain
> 
> Instead, I would like to suggest a new approach we use at Neuroblade to achieve this:
> Build QEMU as a shared library that can be loaded and used directly in a larger simulation.
> Today we build qemu-system-nios2 shared library and load it from qemu-system-x86_64 in order
> to simulate an x86_64 system that also consists of multiple nios2 cores.
> In our simulation, two independent "main" functions are running on different threads, and
> simulation synchronization is reduced to synchronizing threads.
> 
> To achieve this, I needed to do the following changes in QEMU:
> 
> 1. Avoid Glib global context. Use a different context (g_main_context_new) for each QEMU instance.
> 2. Change meson.build to build QEMU as a shared library (with PIC enabled for static libraries)
> 3. Define a C API for the library and export it (with a -Wl,--version-script)
> 
> These changes seem enough for simulating mixed architecture system on a single process.
> 
> If this approach sounds useful, I'll be happy to send patches.
> I'd appreciate if you could provide your feedback!

I'm curious how much synchronization and IPC there is between the QEMU
shared libraries? I would have guessed that the pain of making
communication work efficiently between processes would be less than the
pain of solving global state bugs related to shared libraries within a
single process.

Were there issues with POSIX signal handlers?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Building QEMU as a shared library

[Peter Maydell]

On Wed, 15 Dec 2021 at 08:18, Amir Gonnen <amir.gonnen@neuroblade.ai> wrote:
> My goal is to simulate a mixed architecture system.
>
> Today QEMU strongly assumes that the simulated system is a *single architecture*.
> Changing this assumption and supporting mixed architecture in QEMU proved to be
> non-trivial and may require significant development effort. Common code such as
> TCG and others explicitly include architecture specific header files, for example.

Yeah. This is definitely something we'd like to fix some day. It's
the approach I would prefer for getting multi-architecture machines.

> Instead, I would like to suggest a new approach we use at Neuroblade to achieve this:
> Build QEMU as a shared library that can be loaded and used directly in a larger simulation.
> Today we build qemu-system-nios2 shared library and load it from qemu-system-x86_64 in order
> to simulate an x86_64 system that also consists of multiple nios2 cores.
> In our simulation, two independent "main" functions are running on different threads, and
> simulation synchronization is reduced to synchronizing threads.

I agree with Stefan that you should go ahead and send the code as
an RFC patchset, but I feel like there is a lot of work required
to really get the codebase into a state where it is a clean
shared library...

-- PMM

Re: Building QEMU as a shared library

[Daniel P. Berrangé]

On Wed, Dec 15, 2021 at 10:10:35AM +0000, Peter Maydell wrote:
> On Wed, 15 Dec 2021 at 08:18, Amir Gonnen <amir.gonnen@neuroblade.ai> wrote:
> > My goal is to simulate a mixed architecture system.
> >
> > Today QEMU strongly assumes that the simulated system is a *single architecture*.
> > Changing this assumption and supporting mixed architecture in QEMU proved to be
> > non-trivial and may require significant development effort. Common code such as
> > TCG and others explicitly include architecture specific header files, for example.
> 
> Yeah. This is definitely something we'd like to fix some day. It's
> the approach I would prefer for getting multi-architecture machines.
> 
> > Instead, I would like to suggest a new approach we use at Neuroblade to achieve this:
> > Build QEMU as a shared library that can be loaded and used directly in a larger simulation.
> > Today we build qemu-system-nios2 shared library and load it from qemu-system-x86_64 in order
> > to simulate an x86_64 system that also consists of multiple nios2 cores.
> > In our simulation, two independent "main" functions are running on different threads, and
> > simulation synchronization is reduced to synchronizing threads.
> 
> I agree with Stefan that you should go ahead and send the code as
> an RFC patchset, but I feel like there is a lot of work required
> to really get the codebase into a state where it is a clean
> shared library...

I expect there could end up being a big difference between

   "clean for use with QEMU CLI config X"

vs

   "clean for use with all possible QEMU CLI configs"


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Building QEMU as a shared library

[Daniel P. Berrangé]

On Wed, Dec 15, 2021 at 09:45:56AM +0000, Stefan Hajnoczi wrote:
> On Wed, Dec 15, 2021 at 08:18:53AM +0000, Amir Gonnen wrote:
> > Before sending a patch, I would like to check if it's of interest to the community.
> > 
> > My goal is to simulate a mixed architecture system.
> > Today QEMU strongly assumes that the simulated system is a *single architecture*.
> > Changing this assumption and supporting mixed architecture in QEMU proved to be
> > non-trivial and may require significant development effort. Common code such as
> > TCG and others explicitly include architecture specific header files, for example.
> 
> Hi Amir,
> Simulating heterogenous machines comes up from periodically. So far no
> one has upstreamed a solution but there is definitely interest.
> 
> I suggest going ahead and posting the code even if it's not cleaned up.
> 
> > A possible solution, discussed on https://stackoverflow.com/q/63229262/619493 is to
> > separate the simulation to multiple processes (as done by Xilinx) and to use some form
> > of Interprocess Communication channel between them.
> > Such solution has several disadvantages:
> > 
> > - Harder to synchronize simulation between processes
> > - Performance impact of Interprocess Communication
> > - Harder to debug, profile and maintain
> > 
> > Instead, I would like to suggest a new approach we use at Neuroblade to achieve this:
> > Build QEMU as a shared library that can be loaded and used directly in a larger simulation.
> > Today we build qemu-system-nios2 shared library and load it from qemu-system-x86_64 in order
> > to simulate an x86_64 system that also consists of multiple nios2 cores.
> > In our simulation, two independent "main" functions are running on different threads, and
> > simulation synchronization is reduced to synchronizing threads.
> > 
> > To achieve this, I needed to do the following changes in QEMU:
> > 
> > 1. Avoid Glib global context. Use a different context (g_main_context_new) for each QEMU instance.
> > 2. Change meson.build to build QEMU as a shared library (with PIC enabled for static libraries)
> > 3. Define a C API for the library and export it (with a -Wl,--version-script)

I'm curious what kind of scope the C API has and whether it is likely to
conflict with any general plans we have for future evolution to QEMU's
design, especially around configuration / CLI.

In your example where the client loading the QEMU shared library is
QEMU itself, the licensing doesn't become an issue, as everything
is within the same project codebase.

If the shared library were exposed to consumers outside of the
QEMU project, then licensing is a significant stumbling block,
as any consumer has to be GPL-v2-only compatible. It is pretty
easy to get into a license incompatibility situation with this
requirement. I fear alot of people would just ignore this as an
"inconvenience" and knowingly use incompatible code.

Both of these issues could be avoided if we make any shuared
library a QEMU-internal thing, by not installing any headers
and doing something to ensure its ABI changes on every build.

> > These changes seem enough for simulating mixed architecture system on a single process.
> > 
> > If this approach sounds useful, I'll be happy to send patches.
> > I'd appreciate if you could provide your feedback!
> 
> I'm curious how much synchronization and IPC there is between the QEMU
> shared libraries? I would have guessed that the pain of making
> communication work efficiently between processes would be less than the
> pain of solving global state bugs related to shared libraries within a
> single process.

We've already got ability to have device backends run in separate
processes with vhost-user, and have various other efforts underway
to support more generalized out of tree device implementations.
With this direction in mind, it feels conceptually desirable  if
we are able to support heterogenous CPU emulation using co-operating
processes too, as opposed to a monolithic process architecture.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

RE: Building QEMU as a shared library

[Amir Gonnen]

Hi Stefan,

Easier/faster synchronization is just one side of the issue.
It's much easier to debug a single process or profile it, for example. It's also easier to deploy and maintain a single process.

For now, the only "global state" problem I had to fix was Glib global context.
I didn't see an issue with posix signals. Any other global state you think I should take care of?
Of course, I only tested it for my own simple use case (qemu-system-nios2 + qemu-system-x86_64 with no GUI)

As a first step , I intend to send a patch that removes the Glib context global state.

Thanks,
Amir

-----Original Message-----
From: Stefan Hajnoczi <stefanha@redhat.com>
Sent: Wednesday, December 15, 2021 11:46 AM
To: Amir Gonnen <amir.gonnen@neuroblade.ai>
Cc: qemu-devel@nongnu.org; peter.maydell@linaro.org; f4bug@amsat.org
Subject: Re: Building QEMU as a shared library

On Wed, Dec 15, 2021 at 08:18:53AM +0000, Amir Gonnen wrote:
> Before sending a patch, I would like to check if it's of interest to the community.
>
> My goal is to simulate a mixed architecture system.
> Today QEMU strongly assumes that the simulated system is a *single architecture*.
> Changing this assumption and supporting mixed architecture in QEMU
> proved to be non-trivial and may require significant development
> effort. Common code such as TCG and others explicitly include architecture specific header files, for example.

Hi Amir,
Simulating heterogenous machines comes up from periodically. So far no one has upstreamed a solution but there is definitely interest.

I suggest going ahead and posting the code even if it's not cleaned up.

> A possible solution, discussed on
> https://stackoverflow.com/q/63229262/619493 is to separate the
> simulation to multiple processes (as done by Xilinx) and to use some form of Interprocess Communication channel between them.
> Such solution has several disadvantages:
>
> - Harder to synchronize simulation between processes
> - Performance impact of Interprocess Communication
> - Harder to debug, profile and maintain
>
> Instead, I would like to suggest a new approach we use at Neuroblade to achieve this:
> Build QEMU as a shared library that can be loaded and used directly in a larger simulation.
> Today we build qemu-system-nios2 shared library and load it from
> qemu-system-x86_64 in order to simulate an x86_64 system that also consists of multiple nios2 cores.
> In our simulation, two independent "main" functions are running on
> different threads, and simulation synchronization is reduced to synchronizing threads.
>
> To achieve this, I needed to do the following changes in QEMU:
>
> 1. Avoid Glib global context. Use a different context (g_main_context_new) for each QEMU instance.
> 2. Change meson.build to build QEMU as a shared library (with PIC
> enabled for static libraries) 3. Define a C API for the library and
> export it (with a -Wl,--version-script)
>
> These changes seem enough for simulating mixed architecture system on a single process.
>
> If this approach sounds useful, I'll be happy to send patches.
> I'd appreciate if you could provide your feedback!

I'm curious how much synchronization and IPC there is between the QEMU shared libraries? I would have guessed that the pain of making communication work efficiently between processes would be less than the pain of solving global state bugs related to shared libraries within a single process.

Were there issues with POSIX signal handlers?

Stefan

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited.

Re: Building QEMU as a shared library

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1784 bytes --]

On Wed, Dec 15, 2021 at 12:18:16PM +0000, Amir Gonnen wrote:
> Easier/faster synchronization is just one side of the issue.
> It's much easier to debug a single process or profile it, for example. It's also easier to deploy and maintain a single process.
> 
> For now, the only "global state" problem I had to fix was Glib global context.
> I didn't see an issue with posix signals. Any other global state you think I should take care of?

Signal handlers are interesting because they are per-process, not
per-thread. Luckily QEMU doesn't use signals that much. Accelerators
like KVM use SIGUSR1 while TCG doesn't rely on signals.

One example is what happens when you send SIGINT (Ctrl+C) to the
process. Only one of the shared libraries will handle the signal. The
other one will not be aware that SIGINT happened.

The signal handlers that execute in specific threads (e.g. vCPU threads)
are likely to crash or behave in weird ways. For example,
softmmu/cpus.c:

  static void sigbus_handler(int n, siginfo_t *siginfo, void *ctx)
  {
      if (siginfo->si_code != BUS_MCEERR_AO && siginfo->si_code != BUS_MCEERR_AR) {
          sigbus_reraise();
      }
  
      if (current_cpu) {
          ^^^^^^^^^^^

current_cpu is a thread-local variable. The problem is that the
current_cpu symbol belongs to a specific shared library (nios2 or
x86_64).

If nios2 installs this signal handler then x86_64 vCPUs will not be able
to handle SIGBUS because the current_cpu variable is trying to fetch the
thread-local storage allocated to the other library.

Global variables are also broken in the same way. If the other shared
library's version of the signal handler function is executed we'll
access the other global variable and not our own.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Building QEMU as a shared library

[Peter Maydell]

On Wed, 15 Dec 2021 at 13:23, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> On Wed, Dec 15, 2021 at 12:18:16PM +0000, Amir Gonnen wrote:
> > Easier/faster synchronization is just one side of the issue.
> > It's much easier to debug a single process or profile it, for example. It's also easier to deploy and maintain a single process.
> >
> > For now, the only "global state" problem I had to fix was Glib global context.
> > I didn't see an issue with posix signals. Any other global state you think I should take care of?
>
> Signal handlers are interesting because they are per-process, not
> per-thread. Luckily QEMU doesn't use signals that much. Accelerators
> like KVM use SIGUSR1 while TCG doesn't rely on signals.

I think TCG also uses SIGUSR1 for SIG_IPI.

-- PMM

Re: Building QEMU as a shared library

[Philippe Mathieu-Daudé]

Hi Peter,

On 12/15/21 11:10, Peter Maydell wrote:
> On Wed, 15 Dec 2021 at 08:18, Amir Gonnen <amir.gonnen@neuroblade.ai> wrote:
>> My goal is to simulate a mixed architecture system.
>>
>> Today QEMU strongly assumes that the simulated system is a *single architecture*.
>> Changing this assumption and supporting mixed architecture in QEMU proved to be
>> non-trivial and may require significant development effort. Common code such as
>> TCG and others explicitly include architecture specific header files, for example.
> 
> Yeah. This is definitely something we'd like to fix some day. It's
> the approach I would prefer for getting multi-architecture machines.

Am I understanding correctly your preference would be *not* using shared
libraries, but having a monolithic process able to use any configuration
of heterogeneous architectures?

What are your thoughts on Daniel idea to where (IIUC) cores can are
external processes wired via vhost-user. One problem is not all
operating systems supported provide this possibility.

>> Instead, I would like to suggest a new approach we use at Neuroblade to achieve this:
>> Build QEMU as a shared library that can be loaded and used directly in a larger simulation.
>> Today we build qemu-system-nios2 shared library and load it from qemu-system-x86_64 in order
>> to simulate an x86_64 system that also consists of multiple nios2 cores.
>> In our simulation, two independent "main" functions are running on different threads, and
>> simulation synchronization is reduced to synchronizing threads.
> 
> I agree with Stefan that you should go ahead and send the code as
> an RFC patchset, but I feel like there is a lot of work required
> to really get the codebase into a state where it is a clean
> shared library...
> 
> -- PMM
> 

Re: Building QEMU as a shared library

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 857 bytes --]

On Thu, Dec 23, 2021 at 10:49:46AM +0100, Philippe Mathieu-Daudé wrote:
> What are your thoughts on Daniel idea to where (IIUC) cores can are
> external processes wired via vhost-user. One problem is not all
> operating systems supported provide this possibility.

There is no fundamental limitation that prevents vhost-user from being
used on non-Linux OSes. Eventfds can be replaced with pipes or other
file descriptors. Shared memory fd passing can be replaced with an
OS-specific shared memory API.

If the OS does not support these things:

- Johannes Berg added in-band notifications ("docs: vhost-user: add
  in-band kick/call messages"), so eventfds aren't strictly necessary
  anymore.

- David Gilbert is working on in-band DMA, reducing the need for shared
  memory (at the cost of extra inter-process communication).

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Building QEMU as a shared library

[Peter Maydell]

On Thu, 23 Dec 2021 at 09:49, Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>
> Hi Peter,
>
> On 12/15/21 11:10, Peter Maydell wrote:
> > On Wed, 15 Dec 2021 at 08:18, Amir Gonnen <amir.gonnen@neuroblade.ai> wrote:
> >> My goal is to simulate a mixed architecture system.
> >>
> >> Today QEMU strongly assumes that the simulated system is a *single architecture*.
> >> Changing this assumption and supporting mixed architecture in QEMU proved to be
> >> non-trivial and may require significant development effort. Common code such as
> >> TCG and others explicitly include architecture specific header files, for example.
> >
> > Yeah. This is definitely something we'd like to fix some day. It's
> > the approach I would prefer for getting multi-architecture machines.
>
> Am I understanding correctly your preference would be *not* using shared
> libraries, but having a monolithic process able to use any configuration
> of heterogeneous architectures?

That would be my preference, yes. On the other hand I know there's
a lot of work in trying to get there, so I don't want to rule out the
idea that maybe pragmatically we do something else instead.

> What are your thoughts on Daniel idea to where (IIUC) cores can are
> external processes wired via vhost-user.

It sounds a bit awkward to me -- you end up with a system where
QEMU's scheduler between vCPUs is no longer the only thing deciding
what gets to run. (eg, how do you emulate atomic accesses? currently
we do those by "stop all other CPUs, do the thing, restart" for the
cases where we can't rely on the host's atomic insns.)
The multiple-host-processes model can certainly be made to work,
though -- AIUI this is how Xilinx's out-of-tree stuff works.

-- PMM