[Qemu-devel] virtIO question

[Qemu-devel] virtIO question

[zhunxun]

who can explain the means of idx in VRingUsed and VRingAvail structure about virtIO??
thanks!



zhunxun@gmail.com

Re: [Qemu-devel] virtIO question

[jitendra kumar khasdev]

Hi All,

I am also exploring virtio implementation in qemu, I try to understand
virtio in linux(guest) and qemu, but I am not able to understand I/O stack
between linux to qemu.

here is what I find difficulty in understanding the code,

1. Qemu virtio datastructures are not so clear.
2. I didn't find any sort high level design with respective qemu for virtIO.
3. How qemu virtIO and linux virtIO does shared memoru communication.

I will deeply thankful, if somebody help me to kick start this because I am
really willing to work on virtIO. I am hoping somebody in community will
help to give me some direction to achive the goal.

----
Jitendra





On Sat, Nov 5, 2016 at 4:20 PM, zhunxun@gmail.com <zhunxun@gmail.com> wrote:

> who can explain the means of idx in VRingUsed and VRingAvail structure
> about virtIO??
> thanks!
>
>
>
> zhunxun@gmail.com
>

Re: [Qemu-devel] virtIO question

[Peter Maydell]

On 5 November 2016 at 14:46, jitendra kumar khasdev <jkhasdev@gmail.com> wrote:
> here is what I find difficulty in understanding the code,
>
> 1. Qemu virtio datastructures are not so clear.
> 2. I didn't find any sort high level design with respective qemu for virtIO.
> 3. How qemu virtIO and linux virtIO does shared memoru communication.
>
> I will deeply thankful, if somebody help me to kick start this because I am
> really willing to work on virtIO. I am hoping somebody in community will
> help to give me some direction to achive the goal.

Have you looked at the virtio specification? This describes
the overall structure and communication mechanism, which
QEMU and Linux each only implement one half of:
http://docs.oasis-open.org/virtio/virtio/v1.0/virtio-v1.0.html

Hopefully it will provide the higher level overview
that you are looking for. Many of the data structures
and their purposes are also directly required and
described by the spec.

thanks
-- PMM

Re: [Qemu-devel] virtIO question

[jitendra kumar khasdev]

>
> Have you looked at the virtio specification?


No.


> This describes
> the overall structure and communication mechanism, which
> QEMU and Linux each only implement one half of:
> http://docs.oasis-open.org/virtio/virtio/v1.0/virtio-v1.0.html


Thanks Peter. This doc looks me interesting.

---
Jitendra

Re: [Qemu-devel] virtIO question

[zhunxun]

I want to ask a another question,why a virt_queue in virtio include in_sgs and out_sgs,for example,send_queue of virtIO net driver have in_sgs and out_sgs，when transmit data,It add buffer to out_sgs of send_queue,but how it to use in_sgs??



zhunxun@gmail.com
 
From: jitendra kumar khasdev
Date: 2016-11-05 23:41
To: Peter Maydell
CC: zhunxun@gmail.com; qemu-devel
Subject: Re: [Qemu-devel] virtIO question
Have you looked at the virtio specification?

No.
 
This describes
the overall structure and communication mechanism, which
QEMU and Linux each only implement one half of:
http://docs.oasis-open.org/virtio/virtio/v1.0/virtio-v1.0.html

Thanks Peter. This doc looks me interesting. 

---
Jitendra 

Re: [Qemu-devel] virtIO question

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1327 bytes --]

On Wed, Nov 09, 2016 at 06:58:16PM +0800, zhunxun@gmail.com wrote:
> I want to ask a another question,why a virt_queue in virtio include in_sgs and out_sgs,for example,send_queue of virtIO net driver have in_sgs and out_sgs，when transmit data,It add buffer to out_sgs of send_queue,but how it to use in_sgs??

You can think of every virtqueue buffer as having two scatter-gather
lists:
1. out_sgs are driver->device buffers (e.g. tx packet payload)
2. in_sgs are device->driver buffers (e.g. rx packet payload)

Look at the virtio-net ctrl virtqueue (see spec and
virtio_net_handle_ctrl() for details).  Each buffer has:

1. struct virtio_net_ctrl_hdr (out_sgs)
2. request-specific fields (out_sgs)
3. virtio_net_ctrl_ack status byte (in_sgs)

The device parses the request and performs the operation.  Then it fills
in the result (success or error code) in the status byte.

Processing ctrl virtqueue buffers therefore requires both guest memory
reads (out_sgs) and writes (in_sgs).  Most of the other virtio devices
also use bi-directional buffers.

This may not be obvious if you only consider the virtio-net tx
virtqueue, for example, where buffers use out_sgs only.

Hope this makes sense.  If not, look at the specification again and
think about how virtio-net ctrl request processing works.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] virtIO question

[zhunxun]

From this point of view ,I think it make sense well, thank you very much!
 but I have another question about notify mechanism between virtIO driver and qemu.
according the source code of Linux and qemu,
when driver add a sg buffer to send queue named sq,
sq->vq->vring.avail->idx++
vq->num_added++
and then use virtqueue_kick_prepare to make sure if need notify qemu.
it (new_idx-event_idx)<(new_idx-old_idx)
if it is true,then notify other side.
However,every time driver add a sg,then virtqueue_kick_prepare is called,and vq->num_added  is reseted to 0,so in fact ,I think vq->num_added is always 0 or 1。
as to qemu side,every time when pop a elem from virtqueue,it set VRingUsed.ring[vring.num] to the lastest VRingAvail.idx, this according the arithmetic ((new_idx-event_idx)<(new_idx-old_idx)),it seems that this mechanism does not make sense
I do not know if I describe it clearly.or can you give me an example to prove how it make sense!!
thanks a lot!


zhunxun@gmail.com
 
From: Stefan Hajnoczi
Date: 2016-11-10 18:32
To: zhunxun@gmail.com
CC: jkhasdev; qemu
Subject: Re: [Qemu-devel] virtIO question
On Wed, Nov 09, 2016 at 06:58:16PM +0800, zhunxun@gmail.com wrote:
> I want to ask a another question,why a virt_queue in virtio include in_sgs and out_sgs,for example,send_queue of virtIO net driver have in_sgs and out_sgs，when transmit data,It add buffer to out_sgs of send_queue,but how it to use in_sgs??
 
You can think of every virtqueue buffer as having two scatter-gather
lists:
1. out_sgs are driver->device buffers (e.g. tx packet payload)
2. in_sgs are device->driver buffers (e.g. rx packet payload)
 
Look at the virtio-net ctrl virtqueue (see spec and
virtio_net_handle_ctrl() for details).  Each buffer has:
 
1. struct virtio_net_ctrl_hdr (out_sgs)
2. request-specific fields (out_sgs)
3. virtio_net_ctrl_ack status byte (in_sgs)
 
The device parses the request and performs the operation.  Then it fills
in the result (success or error code) in the status byte.
 
Processing ctrl virtqueue buffers therefore requires both guest memory
reads (out_sgs) and writes (in_sgs).  Most of the other virtio devices
also use bi-directional buffers.
 
This may not be obvious if you only consider the virtio-net tx
virtqueue, for example, where buffers use out_sgs only.
 
Hope this makes sense.  If not, look at the specification again and
think about how virtio-net ctrl request processing works.

Re: [Qemu-devel] virtIO question

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2094 bytes --]

On Thu, Nov 10, 2016 at 08:16:38PM +0800, zhunxun@gmail.com wrote:
> From this point of view ,I think it make sense well, thank you very much!
>  but I have another question about notify mechanism between virtIO driver and qemu.
> according the source code of Linux and qemu,
> when driver add a sg buffer to send queue named sq,
> sq->vq->vring.avail->idx++
> vq->num_added++
> and then use virtqueue_kick_prepare to make sure if need notify qemu.
> it (new_idx-event_idx)<(new_idx-old_idx)

This expression is wrong.  The specification and Linux code both say:

  (u16)(new_idx - event_idx - 1) < (u16)(new_idx - old_idx)

Both the (u16) and the -1 matter.  Maybe that's why you are confused by
this?

> if it is true,then notify other side.
> However,every time driver add a sg,then virtqueue_kick_prepare is called,and vq->num_added  is reseted to 0,so in fact ,I think vq->num_added is always 0 or 1。

A driver may add multiple buffers to the virtqueue by calling
virtqueue_add_sgs() or similar functions multiple times before kicking.
Therefore vq->num_added > 1 is possible.

> as to qemu side,every time when pop a elem from virtqueue,it set VRingUsed.ring[vring.num] to the lastest VRingAvail.idx, this according the arithmetic ((new_idx-event_idx)<(new_idx-old_idx)),it seems that this mechanism does not make sense

You are basically asking "how does event_idx work?".  The specification
says:

  "The driver can ask the device to delay interrupts until an entry with
  an index specified by the “used_event” field is written in the used ring
  (equivalently, until the idx field in the used ring will reach the
  value used_event + 1)."

and:

  "The device can ask the driver to delay notifications until an entry
  with an index specified by the “avail_event” field is written in the
  available ring (equivalently, until the idx field in the used ring will
  reach the value avail_event + 1)."

Whenever the device or driver wants to notify, it first checks if the
index update crossed the event index set by the other side.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] virtIO question

[zhunxun]

Thanks,the expression is not the key problem,I just write it wrong,the key problem is that what I get from the code is everytime dirver add a sg ,it will call virtqueue_kick,such as network driver,in start_xmit function ,it called xmit_skb generate a sg list and add it to the queue,then called virtqueue_kick ,why it handle like this??can you explain it to me??thank you very much!!!



zhunxun@gmail.com
 
From: Stefan Hajnoczi
Date: 2016-11-11 20:03
To: zhunxun@gmail.com
CC: qemu
Subject: Re: Re: [Qemu-devel] virtIO question
On Thu, Nov 10, 2016 at 08:16:38PM +0800, zhunxun@gmail.com wrote:
> From this point of view ,I think it make sense well, thank you very much!
>  but I have another question about notify mechanism between virtIO driver and qemu.
> according the source code of Linux and qemu,
> when driver add a sg buffer to send queue named sq,
> sq->vq->vring.avail->idx++
> vq->num_added++
> and then use virtqueue_kick_prepare to make sure if need notify qemu.
> it (new_idx-event_idx)<(new_idx-old_idx)
 
This expression is wrong.  The specification and Linux code both say:
 
  (u16)(new_idx - event_idx - 1) < (u16)(new_idx - old_idx)
 
Both the (u16) and the -1 matter.  Maybe that's why you are confused by
this?
 
> if it is true,then notify other side.
> However,every time driver add a sg,then virtqueue_kick_prepare is called,and vq->num_added  is reseted to 0,so in fact ,I think vq->num_added is always 0 or 1。
 
A driver may add multiple buffers to the virtqueue by calling
virtqueue_add_sgs() or similar functions multiple times before kicking.
Therefore vq->num_added > 1 is possible.
 
> as to qemu side,every time when pop a elem from virtqueue,it set VRingUsed.ring[vring.num] to the lastest VRingAvail.idx, this according the arithmetic ((new_idx-event_idx)<(new_idx-old_idx)),it seems that this mechanism does not make sense
 
You are basically asking "how does event_idx work?".  The specification
says:
 
  "The driver can ask the device to delay interrupts until an entry with
  an index specified by the “used_event” field is written in the used ring
  (equivalently, until the idx field in the used ring will reach the
  value used_event + 1)."
 
and:
 
  "The device can ask the driver to delay notifications until an entry
  with an index specified by the “avail_event” field is written in the
  available ring (equivalently, until the idx field in the used ring will
  reach the value avail_event + 1)."
 
Whenever the device or driver wants to notify, it first checks if the
index update crossed the event index set by the other side.

Re: [Qemu-devel] virtIO question

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1072 bytes --]

On Sat, Nov 12, 2016 at 04:43:21PM +0800, zhunxun@gmail.com wrote:
> Thanks,the expression is not the key problem,I just write it wrong,the key problem is that what I get from the code is everytime dirver add a sg ,it will call virtqueue_kick,such as network driver,in start_xmit function ,it called xmit_skb generate a sg list and add it to the queue,then called virtqueue_kick ,why it handle like this??can you explain it to me??thank you very much!!!

I can see there *are* cases in Linux 4.9-rc1 virtio_net.c:start_xmit()
where virtqueue_kick() is skipped so that multiple tx packets can be
added to the virtqueue in a single kick.

static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev)
{
        ...
        bool kick = !skb->xmit_more;

        ...
        if (kick || netif_xmit_stopped(txq))
                virtqueue_kick(sq->vq);
        ...
}

Also keep in mind that the virtio driver APIs in Linux are used by
multiple device drivers (virtio_net.ko, virtio_blk.ko, etc).
virtio_net.ko does not use all features offered by the API.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] virtIO question

[zhunxun]

Hello，teacher,I may be found where the driver add mutiple buffer to virtqueue and then kick qemu side. It is when driver use NAPI to poll the device to get buffers,and it is in receive queue.but in transmit queue,every time driver add a buffer to virtqueue,then kick function is called!!!why ??is qemu handle buffer faster than driver add it??

thank you very much!



zhunxun@gmail.com
 
From: Stefan Hajnoczi
Date: 2016-11-11 20:03
To: zhunxun@gmail.com
CC: qemu
Subject: Re: Re: [Qemu-devel] virtIO question
On Thu, Nov 10, 2016 at 08:16:38PM +0800, zhunxun@gmail.com wrote:
> From this point of view ,I think it make sense well, thank you very much!
>  but I have another question about notify mechanism between virtIO driver and qemu.
> according the source code of Linux and qemu,
> when driver add a sg buffer to send queue named sq,
> sq->vq->vring.avail->idx++
> vq->num_added++
> and then use virtqueue_kick_prepare to make sure if need notify qemu.
> it (new_idx-event_idx)<(new_idx-old_idx)
 
This expression is wrong.  The specification and Linux code both say:
 
  (u16)(new_idx - event_idx - 1) < (u16)(new_idx - old_idx)
 
Both the (u16) and the -1 matter.  Maybe that's why you are confused by
this?
 
> if it is true,then notify other side.
> However,every time driver add a sg,then virtqueue_kick_prepare is called,and vq->num_added  is reseted to 0,so in fact ,I think vq->num_added is always 0 or 1。
 
A driver may add multiple buffers to the virtqueue by calling
virtqueue_add_sgs() or similar functions multiple times before kicking.
Therefore vq->num_added > 1 is possible.
 
> as to qemu side,every time when pop a elem from virtqueue,it set VRingUsed.ring[vring.num] to the lastest VRingAvail.idx, this according the arithmetic ((new_idx-event_idx)<(new_idx-old_idx)),it seems that this mechanism does not make sense
 
You are basically asking "how does event_idx work?".  The specification
says:
 
  "The driver can ask the device to delay interrupts until an entry with
  an index specified by the “used_event” field is written in the used ring
  (equivalently, until the idx field in the used ring will reach the
  value used_event + 1)."
 
and:
 
  "The device can ask the driver to delay notifications until an entry
  with an index specified by the “avail_event” field is written in the
  available ring (equivalently, until the idx field in the used ring will
  reach the value avail_event + 1)."
 
Whenever the device or driver wants to notify, it first checks if the
index update crossed the event index set by the other side.

Re: [Qemu-devel] virtIO question

[jack]

   thanks a lot!I will read the document carefully!

   æ�¥è‡ª é…æ—� MX5

   -------- 原始邮件 --------
   �件人：jitendra kumar khasdev <jkhasdev@gmail.com>
   æ—¶é—´ï¼šå‘¨å… 11月5æ—¥ 23:11
   收件人：Peter Maydell <peter.maydell@linaro.org>
   抄�：zhunxun@gmail.com,qemu-devel <qemu-devel@nongnu.org>
   主题：Re: [Qemu-devel] virtIO question

     Have you looked at the virtio specification?

   No.
   Â

     This describes
     the overall structure and communication mechanism, which
     QEMU and Linux each only implement one half of:
     [1]http://docs.oasis-open.org/virtio/virtio/v1.0/virtio-v1.0.html

   Thanks Peter. This doc looks me interesting.Â
   ---
   JitendraÂ

References

   1. http://docs.oasis-open.org/virtio/virtio/v1.0/virtio-v1.0.html

Re: [Qemu-devel] virtIO question

[jack]

[-- Attachment #1: Type: text/plain, Size: 2697 bytes --]

   thanks,but i really do not found when the driver will add mutiple
   buffers then call kick function,this is the key problem .

   æ�¥è‡ª é…æ—� MX5

   -------- 原始邮件 --------
   �件人：Stefan Hajnoczi <stefanha@gmail.com>
   时间：周五 11月11日 20:03
   收件人：zhunxun@gmail.com
   抄�：qemu <qemu-devel@nongnu.org>
   主题：Re: Re: [Qemu-devel] virtIO question
   On Thu, Nov 10, 2016 at 08:16:38PM +0800, zhunxun@gmail.com wrote:
   > From this point of view ,I think it make sense well, thank you very
   much!
   >  but I have another question about notify mechanism between virtIO
   driver and qemu.
   > according the source code of Linux and qemu,
   > when driver add a sg buffer to send queue named sq,
   > sq->vq->vring.avail->idx++
   > vq->num_added++
   > and then use virtqueue_kick_prepare to make sure if need notify qemu.
   > it (new_idx-event_idx)<(new_idx-old_idx)
   This expression is wrong.  The specification and Linux code both say:
     (u16)(new_idx - event_idx - 1) < (u16)(new_idx - old_idx)
   Both the (u16) and the -1 matter.  Maybe that's why you are confused by
   this?
   > if it is true,then notify other side.
   > However,every time driver add a sg,then virtqueue_kick_prepare is
   called,and vq->num_added  is reseted to 0,so in fact ,I think
   vq->num_added is always 0 or 1。
   A driver may add multiple buffers to the virtqueue by calling
   virtqueue_add_sgs() or similar functions multiple times before kicking.
   Therefore vq->num_added > 1 is possible.
   > as to qemu side,every time when pop a elem from virtqueue,it set
   VRingUsed.ring[vring.num] to the lastest VRingAvail.idx, this according
   the arithmetic ((new_idx-event_idx)<(new_idx-old_idx)),it seems that
   this mechanism does not make sense
   You are basically asking "how does event_idx work?".  The specification
   says:
     "The driver can ask the device to delay interrupts until an entry
   with
     an index specified by the “used_event� field is written in the
   used ring
     (equivalently, until the idx field in the used ring will reach the
     value used_event + 1)."
   and:
     "The device can ask the driver to delay noti�cations until an entry
     with an index specified by the “avail_event� field is written in
   the
     available ring (equivalently, until the idx field in the used ring
   will
     reach the value avail_event + 1)."
   Whenever the device or driver wants to notify, it first checks if the
   index update crossed the event index set by the other side.

[-- Attachment #2: signature.asc --]
[-- Type: application/octet-stream, Size: 23 bytes --]



解码邮件时出错

[Qemu-devel] virtIO question

[zhunxun]

I have a question about qemu.is it a bug in qemu version 1.2?
in qemu version 1.2 ,it set avail event by the code :
 if (vq->vdev->guest_features & (1 << VIRTIO_RING_F_EVENT_IDX)) {
        vring_avail_event(vq, vring_avail_idx(vq));
        }
 and in version 2.7 the code is
 if (virtio_vdev_has_feature(vdev, VIRTIO_RING_F_EVENT_IDX)) {
        vring_set_avail_event(vq, vq->last_avail_idx);
    }

a big difference of this is the value.vring_avail_idx(vq)is the latest value of VRingAvail.idx,and vq->last_avail_idx is not, I think it really different with the two different values,and I think the later is right,is it??
thanks a lot!!



zhunxun@gmail.com

Re: [Qemu-devel] virtIO question

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 993 bytes --]

On Mon, Nov 14, 2016 at 08:36:39PM +0800, zhunxun@gmail.com wrote:
> I have a question about qemu.is it a bug in qemu version 1.2?
> in qemu version 1.2 ,it set avail event by the code :
>  if (vq->vdev->guest_features & (1 << VIRTIO_RING_F_EVENT_IDX)) {
>         vring_avail_event(vq, vring_avail_idx(vq));
>         }
>  and in version 2.7 the code is
>  if (virtio_vdev_has_feature(vdev, VIRTIO_RING_F_EVENT_IDX)) {
>         vring_set_avail_event(vq, vq->last_avail_idx);
>     }
> 
> a big difference of this is the value.vring_avail_idx(vq)is the latest value of VRingAvail.idx,and vq->last_avail_idx is not, I think it really different with the two different values,and I think the later is right,is it??

qemu.git/master has both vring_set_avail_event(vq, vring_avail_idx(vq))
and vring_set_avail_event(vq, vq->last_avail_idx) so I'm not sure what
you are referring to.

Please use git-blame(1) and git-log(1) to investigate changes to the
code yourself.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]