Questions regarding ivshmem spec

Questions regarding ivshmem spec

[Sasha Levin]

Hello,

I am looking to implement an ivshmem device for KVM tools, the purpose
is to provide same functionality as QEMU and interoperability with QEMU.

Going through the spec (I found here:
https://gitorious.org/nahanni/guest-code/blobs/master/device_spec.txt )
and the code in QEMU I have gathered several questions, I'll be happy
for some help with it.

1. File handles and guest IDs are passed between the server and the
peers using sockets, is the protocol itself documented anywhere? I would
like to be able to work alongside QEMU servers/peers.

2. The spec describes DOORBELL as an array of DWORDs, when one guest
wants to poke a different guest it would write something into the offset
of the other guest in the DOORBELL array.
Looking at the implementation in QEMU, DOORBELL is one DWORD, when
writing to it the upper WORD is the guest id and the lower WORD is the
value.
What am I missing here?

3. There are 3 ways for guests to communicate between each other, and
I'm assuming all guests using the same SHM block must use the same
method. Is it safe to assume we'll always use ioeventfds as in the
implementation now?

Thanks!
	
-- 

Sasha.

[Qemu-devel] Questions regarding ivshmem spec

[Sasha Levin]

Hello,

I am looking to implement an ivshmem device for KVM tools, the purpose
is to provide same functionality as QEMU and interoperability with QEMU.

Going through the spec (I found here:
https://gitorious.org/nahanni/guest-code/blobs/master/device_spec.txt )
and the code in QEMU I have gathered several questions, I'll be happy
for some help with it.

1. File handles and guest IDs are passed between the server and the
peers using sockets, is the protocol itself documented anywhere? I would
like to be able to work alongside QEMU servers/peers.

2. The spec describes DOORBELL as an array of DWORDs, when one guest
wants to poke a different guest it would write something into the offset
of the other guest in the DOORBELL array.
Looking at the implementation in QEMU, DOORBELL is one DWORD, when
writing to it the upper WORD is the guest id and the lower WORD is the
value.
What am I missing here?

3. There are 3 ways for guests to communicate between each other, and
I'm assuming all guests using the same SHM block must use the same
method. Is it safe to assume we'll always use ioeventfds as in the
implementation now?

Thanks!
	
-- 

Sasha.

Re: [Qemu-devel] Questions regarding ivshmem spec

[Avi Kivity]

On 08/25/2011 04:29 PM, Sasha Levin wrote:
> 2. The spec describes DOORBELL as an array of DWORDs, when one guest
> wants to poke a different guest it would write something into the offset
> of the other guest in the DOORBELL array.
> Looking at the implementation in QEMU, DOORBELL is one DWORD, when
> writing to it the upper WORD is the guest id and the lower WORD is the
> value.
> What am I missing here?
>

The spec in qemu.git is accurate.  The intent is to use an ioeventfd 
bound into an irqfd so a write into the doorbell injects an interrupt 
directly into the other guest, without going through qemu^Wkvm tool.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] Questions regarding ivshmem spec

[Avi Kivity]

On 08/25/2011 04:29 PM, Sasha Levin wrote:
> 2. The spec describes DOORBELL as an array of DWORDs, when one guest
> wants to poke a different guest it would write something into the offset
> of the other guest in the DOORBELL array.
> Looking at the implementation in QEMU, DOORBELL is one DWORD, when
> writing to it the upper WORD is the guest id and the lower WORD is the
> value.
> What am I missing here?
>

The spec in qemu.git is accurate.  The intent is to use an ioeventfd 
bound into an irqfd so a write into the doorbell injects an interrupt 
directly into the other guest, without going through qemu^Wkvm tool.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] Questions regarding ivshmem spec

[Sasha Levin]

On Thu, 2011-08-25 at 17:00 +0300, Avi Kivity wrote:
> On 08/25/2011 04:29 PM, Sasha Levin wrote:
> > 2. The spec describes DOORBELL as an array of DWORDs, when one guest
> > wants to poke a different guest it would write something into the offset
> > of the other guest in the DOORBELL array.
> > Looking at the implementation in QEMU, DOORBELL is one DWORD, when
> > writing to it the upper WORD is the guest id and the lower WORD is the
> > value.
> > What am I missing here?
> >
> 
> The spec in qemu.git is accurate.  The intent is to use an ioeventfd 
> bound into an irqfd so a write into the doorbell injects an interrupt 
> directly into the other guest, without going through qemu^Wkvm tool.
> 

But the doorbell is a single DWORD, so if a guest writes to it we'd
still need to figure out which guest/vector he wants to poke from
userspace, no?

If it was an array of doorbells then yes, we could assign an ioeventfd
to each offset - but now I don't quite see how we can avoid passing
through the userspace.

-- 

Sasha.

Re: [Qemu-devel] Questions regarding ivshmem spec

[Sasha Levin]

On Thu, 2011-08-25 at 17:00 +0300, Avi Kivity wrote:
> On 08/25/2011 04:29 PM, Sasha Levin wrote:
> > 2. The spec describes DOORBELL as an array of DWORDs, when one guest
> > wants to poke a different guest it would write something into the offset
> > of the other guest in the DOORBELL array.
> > Looking at the implementation in QEMU, DOORBELL is one DWORD, when
> > writing to it the upper WORD is the guest id and the lower WORD is the
> > value.
> > What am I missing here?
> >
> 
> The spec in qemu.git is accurate.  The intent is to use an ioeventfd 
> bound into an irqfd so a write into the doorbell injects an interrupt 
> directly into the other guest, without going through qemu^Wkvm tool.
> 

But the doorbell is a single DWORD, so if a guest writes to it we'd
still need to figure out which guest/vector he wants to poke from
userspace, no?

If it was an array of doorbells then yes, we could assign an ioeventfd
to each offset - but now I don't quite see how we can avoid passing
through the userspace.

-- 

Sasha.

Re: [Qemu-devel] Questions regarding ivshmem spec

[Avi Kivity]

On 08/25/2011 05:39 PM, Sasha Levin wrote:
> On Thu, 2011-08-25 at 17:00 +0300, Avi Kivity wrote:
> >  On 08/25/2011 04:29 PM, Sasha Levin wrote:
> >  >  2. The spec describes DOORBELL as an array of DWORDs, when one guest
> >  >  wants to poke a different guest it would write something into the offset
> >  >  of the other guest in the DOORBELL array.
> >  >  Looking at the implementation in QEMU, DOORBELL is one DWORD, when
> >  >  writing to it the upper WORD is the guest id and the lower WORD is the
> >  >  value.
> >  >  What am I missing here?
> >  >
> >
> >  The spec in qemu.git is accurate.  The intent is to use an ioeventfd
> >  bound into an irqfd so a write into the doorbell injects an interrupt
> >  directly into the other guest, without going through qemu^Wkvm tool.
> >
>
> But the doorbell is a single DWORD, so if a guest writes to it we'd
> still need to figure out which guest/vector he wants to poke from
> userspace, no?
>
> If it was an array of doorbells then yes, we could assign an ioeventfd
> to each offset - but now I don't quite see how we can avoid passing
> through the userspace.
>

Use the datamatch facility.

We didn't want an array of registers to avoid scaling issues (PIO space 
is quite small).


-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] Questions regarding ivshmem spec

[Avi Kivity]

On 08/25/2011 05:39 PM, Sasha Levin wrote:
> On Thu, 2011-08-25 at 17:00 +0300, Avi Kivity wrote:
> >  On 08/25/2011 04:29 PM, Sasha Levin wrote:
> >  >  2. The spec describes DOORBELL as an array of DWORDs, when one guest
> >  >  wants to poke a different guest it would write something into the offset
> >  >  of the other guest in the DOORBELL array.
> >  >  Looking at the implementation in QEMU, DOORBELL is one DWORD, when
> >  >  writing to it the upper WORD is the guest id and the lower WORD is the
> >  >  value.
> >  >  What am I missing here?
> >  >
> >
> >  The spec in qemu.git is accurate.  The intent is to use an ioeventfd
> >  bound into an irqfd so a write into the doorbell injects an interrupt
> >  directly into the other guest, without going through qemu^Wkvm tool.
> >
>
> But the doorbell is a single DWORD, so if a guest writes to it we'd
> still need to figure out which guest/vector he wants to poke from
> userspace, no?
>
> If it was an array of doorbells then yes, we could assign an ioeventfd
> to each offset - but now I don't quite see how we can avoid passing
> through the userspace.
>

Use the datamatch facility.

We didn't want an array of registers to avoid scaling issues (PIO space 
is quite small).


-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: Questions regarding ivshmem spec

[Sasha Levin]

On Thu, 2011-08-25 at 17:40 +0300, Avi Kivity wrote:
> On 08/25/2011 05:39 PM, Sasha Levin wrote:
> > On Thu, 2011-08-25 at 17:00 +0300, Avi Kivity wrote:
> > >  On 08/25/2011 04:29 PM, Sasha Levin wrote:
> > >  >  2. The spec describes DOORBELL as an array of DWORDs, when one guest
> > >  >  wants to poke a different guest it would write something into the offset
> > >  >  of the other guest in the DOORBELL array.
> > >  >  Looking at the implementation in QEMU, DOORBELL is one DWORD, when
> > >  >  writing to it the upper WORD is the guest id and the lower WORD is the
> > >  >  value.
> > >  >  What am I missing here?
> > >  >
> > >
> > >  The spec in qemu.git is accurate.  The intent is to use an ioeventfd
> > >  bound into an irqfd so a write into the doorbell injects an interrupt
> > >  directly into the other guest, without going through qemu^Wkvm tool.
> > >
> >
> > But the doorbell is a single DWORD, so if a guest writes to it we'd
> > still need to figure out which guest/vector he wants to poke from
> > userspace, no?
> >
> > If it was an array of doorbells then yes, we could assign an ioeventfd
> > to each offset - but now I don't quite see how we can avoid passing
> > through the userspace.
> >
> 
> Use the datamatch facility.
> 
> We didn't want an array of registers to avoid scaling issues (PIO space 
> is quite small).
> 
> 

Ah, right.

Thanks!

-- 

Sasha.

Re: [Qemu-devel] Questions regarding ivshmem spec

[Sasha Levin]

On Thu, 2011-08-25 at 17:40 +0300, Avi Kivity wrote:
> On 08/25/2011 05:39 PM, Sasha Levin wrote:
> > On Thu, 2011-08-25 at 17:00 +0300, Avi Kivity wrote:
> > >  On 08/25/2011 04:29 PM, Sasha Levin wrote:
> > >  >  2. The spec describes DOORBELL as an array of DWORDs, when one guest
> > >  >  wants to poke a different guest it would write something into the offset
> > >  >  of the other guest in the DOORBELL array.
> > >  >  Looking at the implementation in QEMU, DOORBELL is one DWORD, when
> > >  >  writing to it the upper WORD is the guest id and the lower WORD is the
> > >  >  value.
> > >  >  What am I missing here?
> > >  >
> > >
> > >  The spec in qemu.git is accurate.  The intent is to use an ioeventfd
> > >  bound into an irqfd so a write into the doorbell injects an interrupt
> > >  directly into the other guest, without going through qemu^Wkvm tool.
> > >
> >
> > But the doorbell is a single DWORD, so if a guest writes to it we'd
> > still need to figure out which guest/vector he wants to poke from
> > userspace, no?
> >
> > If it was an array of doorbells then yes, we could assign an ioeventfd
> > to each offset - but now I don't quite see how we can avoid passing
> > through the userspace.
> >
> 
> Use the datamatch facility.
> 
> We didn't want an array of registers to avoid scaling issues (PIO space 
> is quite small).
> 
> 

Ah, right.

Thanks!

-- 

Sasha.

Re: Questions regarding ivshmem spec

[Sasha Levin]

On Thu, 2011-08-25 at 16:29 +0300, Sasha Levin wrote:
> Hello,
> 
> I am looking to implement an ivshmem device for KVM tools, the purpose
> is to provide same functionality as QEMU and interoperability with QEMU.

[snip]

> 1. File handles and guest IDs are passed between the server and the
> peers using sockets, is the protocol itself documented anywhere? I would
> like to be able to work alongside QEMU servers/peers.	

I'm still wondering if someone could do a quick sketch of the
client-server protocol or possibly point me to something that documents
it?

-- 

Sasha.

Re: [Qemu-devel] Questions regarding ivshmem spec

[Sasha Levin]

On Thu, 2011-08-25 at 16:29 +0300, Sasha Levin wrote:
> Hello,
> 
> I am looking to implement an ivshmem device for KVM tools, the purpose
> is to provide same functionality as QEMU and interoperability with QEMU.

[snip]

> 1. File handles and guest IDs are passed between the server and the
> peers using sockets, is the protocol itself documented anywhere? I would
> like to be able to work alongside QEMU servers/peers.	

I'm still wondering if someone could do a quick sketch of the
client-server protocol or possibly point me to something that documents
it?

-- 

Sasha.

Re: Questions regarding ivshmem spec

[Cam Macdonell]

On Mon, Aug 29, 2011 at 9:25 AM, Sasha Levin <levinsasha928@gmail.com> wrote:
> On Thu, 2011-08-25 at 16:29 +0300, Sasha Levin wrote:
>> Hello,
>>
>> I am looking to implement an ivshmem device for KVM tools, the purpose
>> is to provide same functionality as QEMU and interoperability with QEMU.
>
> [snip]
>
>> 1. File handles and guest IDs are passed between the server and the
>> peers using sockets, is the protocol itself documented anywhere? I would
>> like to be able to work alongside QEMU servers/peers.
>
> I'm still wondering if someone could do a quick sketch of the
> client-server protocol or possibly point me to something that documents
> it?

Hi Sasha,

I have something like that.  I'll be in touch when I find it.

Cam

>
> --
>
> Sasha.
>
>

Re: [Qemu-devel] Questions regarding ivshmem spec

[Cam Macdonell]

On Mon, Aug 29, 2011 at 9:25 AM, Sasha Levin <levinsasha928@gmail.com> wrote:
> On Thu, 2011-08-25 at 16:29 +0300, Sasha Levin wrote:
>> Hello,
>>
>> I am looking to implement an ivshmem device for KVM tools, the purpose
>> is to provide same functionality as QEMU and interoperability with QEMU.
>
> [snip]
>
>> 1. File handles and guest IDs are passed between the server and the
>> peers using sockets, is the protocol itself documented anywhere? I would
>> like to be able to work alongside QEMU servers/peers.
>
> I'm still wondering if someone could do a quick sketch of the
> client-server protocol or possibly point me to something that documents
> it?

Hi Sasha,

I have something like that.  I'll be in touch when I find it.

Cam

>
> --
>
> Sasha.
>
>

Re: Questions regarding ivshmem spec

[Cam Macdonell]

On Thu, Aug 25, 2011 at 7:29 AM, Sasha Levin <levinsasha928@gmail.com> wrote:
> Hello,
>
> I am looking to implement an ivshmem device for KVM tools, the purpose
> is to provide same functionality as QEMU and interoperability with QEMU.
>
> Going through the spec (I found here:
> https://gitorious.org/nahanni/guest-code/blobs/master/device_spec.txt )
> and the code in QEMU I have gathered several questions, I'll be happy
> for some help with it.
>
> 1. File handles and guest IDs are passed between the server and the
> peers using sockets, is the protocol itself documented anywhere? I would
> like to be able to work alongside QEMU servers/peers.
>
> 2. The spec describes DOORBELL as an array of DWORDs, when one guest
> wants to poke a different guest it would write something into the offset
> of the other guest in the DOORBELL array.
> Looking at the implementation in QEMU, DOORBELL is one DWORD, when
> writing to it the upper WORD is the guest id and the lower WORD is the
> value.
> What am I missing here?
>
> 3. There are 3 ways for guests to communicate between each other, and
> I'm assuming all guests using the same SHM block must use the same
> method. Is it safe to assume we'll always use ioeventfds as in the
> implementation now?

Guests can either access the shared memory region directly or use the
server, so I would count that as 2 ways to share memory via ivshmem.
Can you clarify what you mean by "ways for guests to communicate"?

As for ioeventfds, they are an optional optimization to eventfds.  We
use eventfds currently, but since the guest VMs only see file
descriptors (passed from the server) another mechanism could replace
eventfds if there was some reason to.

Cam

Re: [Qemu-devel] Questions regarding ivshmem spec

[Cam Macdonell]

On Thu, Aug 25, 2011 at 7:29 AM, Sasha Levin <levinsasha928@gmail.com> wrote:
> Hello,
>
> I am looking to implement an ivshmem device for KVM tools, the purpose
> is to provide same functionality as QEMU and interoperability with QEMU.
>
> Going through the spec (I found here:
> https://gitorious.org/nahanni/guest-code/blobs/master/device_spec.txt )
> and the code in QEMU I have gathered several questions, I'll be happy
> for some help with it.
>
> 1. File handles and guest IDs are passed between the server and the
> peers using sockets, is the protocol itself documented anywhere? I would
> like to be able to work alongside QEMU servers/peers.
>
> 2. The spec describes DOORBELL as an array of DWORDs, when one guest
> wants to poke a different guest it would write something into the offset
> of the other guest in the DOORBELL array.
> Looking at the implementation in QEMU, DOORBELL is one DWORD, when
> writing to it the upper WORD is the guest id and the lower WORD is the
> value.
> What am I missing here?
>
> 3. There are 3 ways for guests to communicate between each other, and
> I'm assuming all guests using the same SHM block must use the same
> method. Is it safe to assume we'll always use ioeventfds as in the
> implementation now?

Guests can either access the shared memory region directly or use the
server, so I would count that as 2 ways to share memory via ivshmem.
Can you clarify what you mean by "ways for guests to communicate"?

As for ioeventfds, they are an optional optimization to eventfds.  We
use eventfds currently, but since the guest VMs only see file
descriptors (passed from the server) another mechanism could replace
eventfds if there was some reason to.

Cam