Re: QEMU 2.2.0 in Xen 4.6

Re: QEMU 2.2.0 in Xen 4.6

[Eric Shelton]

On Fri, 16 Jan 2015, Stefano Stabellini wrote:
> On Thu, 15 Jan 2015, Don Slutz wrote:
> > Now that Xen 4.5 has been released, I would like to see at least QEMU
> > 2.2.0 in qemu-upstream-stable on the master branch.
>
> Agreed

Is there any concrete plan regarding if and how stub domains will be
implemented for qemu-upstream in 4.6?  If not, would you entertain the
idea of someone continuing Anthony's earlier work with a Linux-based
stub domain?

Eric

Re: QEMU 2.2.0 in Xen 4.6

[Stefano Stabellini]

On Thu, 22 Jan 2015, Eric Shelton wrote:
> On Fri, 16 Jan 2015, Stefano Stabellini wrote:
> > On Thu, 15 Jan 2015, Don Slutz wrote:
> > > Now that Xen 4.5 has been released, I would like to see at least QEMU
> > > 2.2.0 in qemu-upstream-stable on the master branch.
> >
> > Agreed
> 
> Is there any concrete plan regarding if and how stub domains will be
> implemented for qemu-upstream in 4.6?  If not, would you entertain the
> idea of someone continuing Anthony's earlier work with a Linux-based
> stub domain?

I think it would be fantastic if you could pick up Anthony's work on
Linux based stubdoms and complete it!  I would really like to have
upstream QEMU based stubdoms in Xen 4.6!

FYI there have been some efforts toward getting upstream QEMU building
on rumpkernel (http://wiki.netbsd.org/rumpkernel/), but the work isn't
completed, it is difficult to estimate when it is going to be finished
and it won't make 4.6.  So I think that Linux-based stubdoms would be a
great start.

Re: QEMU 2.2.0 in Xen 4.6

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 4631 bytes --]

On Thu, 22 Jan 2015, Eric Shelton wrote:
> On Jan 22, 2015 9:39 AM, "Stefano Stabellini" <stefano.stabellini@eu.citrix.com> wrote:
> >
> > On Thu, 22 Jan 2015, Eric Shelton wrote:
> > > On Fri, 16 Jan 2015, Stefano Stabellini wrote:
> > > > On Thu, 15 Jan 2015, Don Slutz wrote:
> > > > > Now that Xen 4.5 has been released, I would like to see at least QEMU
> > > > > 2.2.0 in qemu-upstream-stable on the master branch.
> > > >
> > > > Agreed
> > >
> > > Is there any concrete plan regarding if and how stub domains will be
> > > implemented for qemu-upstream in 4.6?  If not, would you entertain the
> > > idea of someone continuing Anthony's earlier work with a Linux-based
> > > stub domain?
> >
> > I think it would be fantastic if you could pick up Anthony's work on
> > Linux based stubdoms and complete it!  I would really like to have
> > upstream QEMU based stubdoms in Xen 4.6!
> >
> > FYI there have been some efforts toward getting upstream QEMU building
> > on rumpkernel (http://wiki.netbsd.org/rumpkernel/), but the work isn't
> > completed, it is difficult to estimate when it is going to be finished
> > and it won't make 4.6.  So I think that Linux-based stubdoms would be a
> > great start.
> 
> Stefano,
> 
> Just a brief question, that I did not think needed to go via xen-devel

I think that it is a very reasonable question and might be useful to
others, so I am CC'ing xen-devel back.


> There are several components interacting with each other in the current qemu stubdom:
> - dom0 qemu process
> - stub domain with MiniOS
> - qemu device model running under MiniOS
> - hvm domain with guest OS
> 
> Is there something, or would you mind taking a minute to write up something, that briefly describes this architecture, and how,
> for example, the two qemu instances cooperate and the two domains must cooperate with each other?  I can go about figuring this
> out, but a brief primer would speed things up.
 
Sure.

Like you wrote, device emulation is done by QEMU running on top of
MiniOS within the stubdom.

The stubdom itself is a PV guest and as such it connects to PV backends
in Dom0 to get access to disk, network and other PV protocols. But in
the stubdom case, they are the disk and network of the guest. The
stubdom doesn't actually have disk or network for itself. (However it
has a couple of other PV interfaces for special purposes, I'll
talk about them later.)

As usual the PV backends might be running in the Linux kernel or in
userspace in QEMU. Therefore you might have a second QEMU instance
running in Dom0 to provide the PV backends.


HVM guest  ->  QEMU Stubdom (device emulation)  ->  QEMU in Dom0 and Linux (PV backends for stubdom)


For example when the HVM guest tries to read from the disk via the
emulated IDE interface, it will trap in Xen and the IO request is going
to be forwarded to QEMU running in the stubdom. QEMU will do the
emulation and eventually will issue a PV read request via MiniOS to the
PV block backend to read from the guest disk. If the PV block backend
is in QEMU, for example because the guest disk is a QCOW2 file, then the
read request will reach the QEMU instance running in Dom0.

In the Linux stubdom case, QEMU running in the stubdom would simply use
the POSIX read syscall to read from the guest disk as usual, then the
Linux kernel in the stubdom would issue the PV block read request to the
backend via xen-blkfront. From QEMU's point of view there should be no
differences between running on a Linux stubdom or in Dom0. Similarly it
is business as usual for the Linux kernel: it is running just as a
regular PV guest.

In addition the stubdom is provided with 3 "special" PV console
interfaces, the first one is used for logging the stubdom's own output,
the second one is used to save the state of emulated devices and the
third one if used to read the state of emulated devices. The last two
are only used for save/restore purposes. See
tools/libxl/libxl_dm.c:spawn_stub_launch_dm.

To complicate matter further, the HVM guest could be running PV
frontends of its own.  For example the Windows PV Drivers. They would be
connecting to a separate set of PV backends in Dom0.  Again if one or
more PV backends are running in QEMU, there is going to a third QEMU
instance to provide PV backends to the guest.


HVM guest ---------(PV)----------> QEMU in Dom0 for guest
    |
    --(emulation)--> QEMU Stubdom-(syscall)->Linux Stubdom---(PV)--> QEMU Dom0 for stubdom


I hope I managed to make things clearer for you.

Cheers,

Stefano

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: QEMU 2.2.0 in Xen 4.6

[Ian Campbell]

On Fri, 2015-01-23 at 14:42 +0000, Stefano Stabellini wrote:

> HVM guest ---------(PV)----------> QEMU in Dom0 for guest
>     |
>     --(emulation)--> QEMU Stubdom-(syscall)->Linux Stubdom---(PV)--> QEMU Dom0 for stubdom

Here, and throughout what you said I think, "QEMU in Dom0 for guest"
could equally well be e.g. "blkback in driver domain for guest" likewise
the "... for stubdom" too.

i.e. the PV backend for the stubdom or the guest doesn't necessarily
need to be QEMU and doesn't necessarily need to be in dom0.

Ian.

Re: QEMU 2.2.0 in Xen 4.6

[Stefano Stabellini]

On Fri, 23 Jan 2015, Ian Campbell wrote:
> On Fri, 2015-01-23 at 14:42 +0000, Stefano Stabellini wrote:
> 
> > HVM guest ---------(PV)----------> QEMU in Dom0 for guest
> >     |
> >     --(emulation)--> QEMU Stubdom-(syscall)->Linux Stubdom---(PV)--> QEMU Dom0 for stubdom
> 
> Here, and throughout what you said I think, "QEMU in Dom0 for guest"
> could equally well be e.g. "blkback in driver domain for guest" likewise
> the "... for stubdom" too.
>
> i.e. the PV backend for the stubdom or the guest doesn't necessarily
> need to be QEMU and doesn't necessarily need to be in dom0.
 
Indeed

Re: QEMU 2.2.0 in Xen 4.6

[Eric Shelton]

[-- Attachment #1.1: Type: text/plain, Size: 1136 bytes --]

On Jan 23, 2015 10:10 AM, "Stefano Stabellini" <
stefano.stabellini@eu.citrix.com> wrote:
>
> On Fri, 23 Jan 2015, Ian Campbell wrote:
> > On Fri, 2015-01-23 at 14:42 +0000, Stefano Stabellini wrote:
> >
> > > HVM guest ---------(PV)----------> QEMU in Dom0 for guest
> > >     |
> > >     --(emulation)--> QEMU Stubdom-(syscall)->Linux Stubdom---(PV)-->
QEMU Dom0 for stubdom
> >
> > Here, and throughout what you said I think, "QEMU in Dom0 for guest"
> > could equally well be e.g. "blkback in driver domain for guest" likewise
> > the "... for stubdom" too.
> >
> > i.e. the PV backend for the stubdom or the guest doesn't necessarily
> > need to be QEMU and doesn't necessarily need to be in dom0.
>
> Indeed
>

Thank you both.

There is one other thing that would be helpful to understand.  Anthony had
to patch the Linux kernel running in the stubdom to allow memory mapping.
What mappings are needed between dom0 and the stub domain, and between the
stub domain and the HVM guest domain?  I am guessing this comes into play
for the display, as it seems that the dom0 qemu instance is running the VNC
server.

Thanks again,
Eric

[-- Attachment #1.2: Type: text/html, Size: 1511 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: QEMU 2.2.0 in Xen 4.6

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1331 bytes --]

On Fri, 23 Jan 2015, Eric Shelton wrote:
> On Jan 23, 2015 10:10 AM, "Stefano Stabellini" <stefano.stabellini@eu.citrix.com> wrote:
> >
> > On Fri, 23 Jan 2015, Ian Campbell wrote:
> > > On Fri, 2015-01-23 at 14:42 +0000, Stefano Stabellini wrote:
> > >
> > > > HVM guest ---------(PV)----------> QEMU in Dom0 for guest
> > > >     |
> > > >     --(emulation)--> QEMU Stubdom-(syscall)->Linux Stubdom---(PV)--> QEMU Dom0 for stubdom
> > >
> > > Here, and throughout what you said I think, "QEMU in Dom0 for guest"
> > > could equally well be e.g. "blkback in driver domain for guest" likewise
> > > the "... for stubdom" too.
> > >
> > > i.e. the PV backend for the stubdom or the guest doesn't necessarily
> > > need to be QEMU and doesn't necessarily need to be in dom0.
> >
> > Indeed
> >
> 
> Thank you both.
> 
> There is one other thing that would be helpful to understand.  Anthony had to patch the Linux kernel running in the stubdom to
> allow memory mapping.  What mappings are needed between dom0 and the stub domain, and between the stub domain and the HVM guest
> domain?  I am guessing this comes into play for the display, as it seems that the dom0 qemu instance is running the VNC server.

I don't remember the details of it anymore. CC'ing Anthony that might
have a better idea.

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: QEMU 2.2.0 in Xen 4.6

[Eric Shelton]

On Mon, Jan 26, 2015 at 9:37 AM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Fri, 23 Jan 2015, Eric Shelton wrote:
>> On Jan 23, 2015 10:10 AM, "Stefano Stabellini" <stefano.stabellini@eu.citrix.com> wrote:
>> >
>> > On Fri, 23 Jan 2015, Ian Campbell wrote:
>> > > On Fri, 2015-01-23 at 14:42 +0000, Stefano Stabellini wrote:
>> > >
>> > > > HVM guest ---------(PV)----------> QEMU in Dom0 for guest
>> > > >     |
>> > > >     --(emulation)--> QEMU Stubdom-(syscall)->Linux Stubdom---(PV)--> QEMU Dom0 for stubdom
>> > >
>> > > Here, and throughout what you said I think, "QEMU in Dom0 for guest"
>> > > could equally well be e.g. "blkback in driver domain for guest" likewise
>> > > the "... for stubdom" too.
>> > >
>> > > i.e. the PV backend for the stubdom or the guest doesn't necessarily
>> > > need to be QEMU and doesn't necessarily need to be in dom0.
>> >
>> > Indeed
>> >
>>
>> Thank you both.
>>
>> There is one other thing that would be helpful to understand.  Anthony had to patch the Linux kernel running in the stubdom to
>> allow memory mapping.  What mappings are needed between dom0 and the stub domain, and between the stub domain and the HVM guest
>> domain?  I am guessing this comes into play for the display, as it seems that the dom0 qemu instance is running the VNC server.
>
> I don't remember the details of it anymore. CC'ing Anthony that might
> have a better idea.

Actually, it looks like what I need to understand is how the startup
of (1) QEMU in Dom0, (2) QEMU in stubdom, and (3) the HVM domain are
synchronized (or supposed to be synchronized).  I assume the desired
order is (2) -> (1) -> (3).  It looks like
/local/domain/0/device-model/{hvm-domid}/state is used to notify xl
that QEMU in Dom0 is running, which prompts xl to unpause the HVM
domain, making sure that (1) occurs before (3).  The problem I am
running into is that, at least with QEMU 2.0, nothing ensures that
QEMU in stubdom is up and running before unpausing the HVM domain.
This causes hvmloader to fail, as (3) occurs before (2), and there is
no device model in place yet.

So, what mechanism is being used with qemu-traditional to ensure QEMU
in stubdom is running before the HVM domain is unpaused?

Thanks,
Eric

Re: QEMU 2.2.0 in Xen 4.6

[Stefano Stabellini]

On Mon, 26 Jan 2015, Eric Shelton wrote:
> On Mon, Jan 26, 2015 at 9:37 AM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Fri, 23 Jan 2015, Eric Shelton wrote:
> >> On Jan 23, 2015 10:10 AM, "Stefano Stabellini" <stefano.stabellini@eu.citrix.com> wrote:
> >> >
> >> > On Fri, 23 Jan 2015, Ian Campbell wrote:
> >> > > On Fri, 2015-01-23 at 14:42 +0000, Stefano Stabellini wrote:
> >> > >
> >> > > > HVM guest ---------(PV)----------> QEMU in Dom0 for guest
> >> > > >     |
> >> > > >     --(emulation)--> QEMU Stubdom-(syscall)->Linux Stubdom---(PV)--> QEMU Dom0 for stubdom
> >> > >
> >> > > Here, and throughout what you said I think, "QEMU in Dom0 for guest"
> >> > > could equally well be e.g. "blkback in driver domain for guest" likewise
> >> > > the "... for stubdom" too.
> >> > >
> >> > > i.e. the PV backend for the stubdom or the guest doesn't necessarily
> >> > > need to be QEMU and doesn't necessarily need to be in dom0.
> >> >
> >> > Indeed
> >> >
> >>
> >> Thank you both.
> >>
> >> There is one other thing that would be helpful to understand.  Anthony had to patch the Linux kernel running in the stubdom to
> >> allow memory mapping.  What mappings are needed between dom0 and the stub domain, and between the stub domain and the HVM guest
> >> domain?  I am guessing this comes into play for the display, as it seems that the dom0 qemu instance is running the VNC server.
> >
> > I don't remember the details of it anymore. CC'ing Anthony that might
> > have a better idea.
> 
> Actually, it looks like what I need to understand is how the startup
> of (1) QEMU in Dom0, (2) QEMU in stubdom, and (3) the HVM domain are
> synchronized (or supposed to be synchronized).  I assume the desired
> order is (2) -> (1) -> (3).

We have two QEMUs in Dom0, one to provide backends to the stubdoms and
one to provide backends to the HVM guest. Which one is (1)?

If (1) is the QEMU providing the backends for the stubdomain, then you
want (1)->(2)->(3)


> It looks like
> /local/domain/0/device-model/{hvm-domid}/state is used to notify xl
> that QEMU in Dom0 is running, which prompts xl to unpause the HVM
> domain, making sure that (1) occurs before (3).  The problem I am
> running into is that, at least with QEMU 2.0, nothing ensures that
> QEMU in stubdom is up and running before unpausing the HVM domain.
> This causes hvmloader to fail, as (3) occurs before (2), and there is
> no device model in place yet.
> 
> So, what mechanism is being used with qemu-traditional to ensure QEMU
> in stubdom is running before the HVM domain is unpaused?

Actually I know that is confusing but I am pretty sure that

/local/domain/0/device-model/{hvm-domid}/state

is to notify that QEMU in the stubdom is up and running. The QEMU
writing to that path, is the one running in the stubdom.

I think that we are probably missing a comparable mechanism for upstream
QEMU.  The toolstack (xl and libxl) talks to qemu-traditional via
xenstore, and talks to upstream QEMU via QMP over a unix socket.  I
think that you need to get QMP working in a stubdom first, maybe by
using a PV console as transport for it.  Afterward you could use QMP to
check the state of QEMU and pause and unpause the domain.

QEMU 2.2.0 in Xen 4.6

[Don Slutz]

Now that Xen 4.5 has been released, I would like to see at least QEMU
2.2.0 in qemu-upstream-stable on the master branch.

   -Don Slutz

Re: QEMU 2.2.0 in Xen 4.6

[Stefano Stabellini]

On Thu, 15 Jan 2015, Don Slutz wrote:
> Now that Xen 4.5 has been released, I would like to see at least QEMU
> 2.2.0 in qemu-upstream-stable on the master branch.

Agreed