[virtio-comment] multibuffer in packed queue

[virtio-comment] multibuffer in packed queue

[Krasnov Arseniy]

[-- Attachment #1: Type: text/plain, Size: 1015 bytes --]

Hellow, I've got question about chapter 2.7.9:

*  2.7.9 Multi-buffer requests Some devices combine multiple buffers as
part of processing of a single request. These devices always mark the
descriptor corresponding to the first buffer in the request used after the
rest of the descriptors (corresponding to rest of the buffers) in the
request - which follow the first descriptor in ring order - has been marked
used and written out into the ring. This guarantees that the driver will
never observe a partial request in the ring.  *

Does it mean, that device can use sequence(chain) of descriptor unil
descriptor with VIRTQ_DESC_F_NEXT  == 0 bit is found, OR device can use
descriptors as many as it needs, don't care about  VIRTQ_DESC_F_NEXT bit,
and finally set  VIRTQ_DESC_F_NEXT bit in all used descriptors.

Also, in packed queue device can't modify descriptors except USED bit. In
split queue, device also can't modify descriptor(it touches used buffer
with index and length fields). Is that true?

Thank You

[-- Attachment #2: Type: text/html, Size: 1189 bytes --]

Re: [virtio-comment] multibuffer in packed queue

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3428 bytes --]

On Thu, Aug 13, 2020 at 10:57:52AM +0300, Krasnov Arseniy wrote:
> Hellow, I've got question about chapter 2.7.9:
> 
> *  2.7.9 Multi-buffer requests Some devices combine multiple buffers as
> part of processing of a single request. These devices always mark the
> descriptor corresponding to the first buffer in the request used after the
> rest of the descriptors (corresponding to rest of the buffers) in the
> request - which follow the first descriptor in ring order - has been marked
> used and written out into the ring. This guarantees that the driver will
> never observe a partial request in the ring.  *
> 
> Does it mean, that device can use sequence(chain) of descriptor unil
> descriptor with VIRTQ_DESC_F_NEXT  == 0 bit is found, OR device can use
> descriptors as many as it needs, don't care about  VIRTQ_DESC_F_NEXT bit,
> and finally set  VIRTQ_DESC_F_NEXT bit in all used descriptors.

2.7.9 talks about 3 concepts:
1. Requests (highest-level concept)
2. Buffers
3. Descriptors (lowest-level concept)

They are distinct. A request is something like a received packet from a
network card. A buffer is a descriptor chain (1 or more descriptors with
the F_NEXT bit set). A descriptor is a single memory [start, end) range
that the device may read from or write to.

The paragraph descibes how some devices complete one request by
combining multiple buffers. If a 8KB network packet arrives on a NIC and
the driver provided two 4KB buffers consisting of 1 4KB descriptor each
then the device may "merge" those two buffers (see virtio-net for
details). When doing so it must ensure that the first 4KB buffer of the
packet is marked used and written out to the ring *after* the second 4KB
buffer so that the driver sees the beginning of the network packet and
not the second half with the first half missing.

Merging buffers is done at the device type-level (e.g. virtio-net), not
at a virtqueue-level (i.e. core virtio code). The device type must
specify exactly how merging works and device type-specific header fields are
be used by the device to indicate which buffers have been merged into a
single request (see virtio-net's VIRTIO_NET_F_MRG_RXBUF feature bit and
num_buffers header field).

> Also, in packed queue device can't modify descriptors except USED bit.

No, the device *must* overwrite descriptors in the packed ring. Imagine
a device that completes buffers out-of-order:

1. The driver submits two buffers (F_NEXT is not set because a chain of
   descriptors would make the ASCII diagram more complex) and waits for
   the device:

  | buf#1 !USED LEN=4KB | buf#2 !USED LEN=4KB |

2. The device completes buf#2 first (1KB of the 4KB descriptor was
   written) and writes the used descriptor to the ring:

  | buf#2 USED LEN=1KB | buf#2 !USED LEN=4KB |

3. Let's say the driver waits for buf#1 to complete (3KB of the 4KB
   descriptor was written):

  | buf#2 USED LEN=1KB | buf#1 USED LEN=3KB |

You can see that the ring was overwritten by the device and the
descriptors were modified (e.g. the 4KB avail length was replaced by the
1KB used length value).

> In
> split queue, device also can't modify descriptor(it touches used buffer
> with index and length fields). Is that true?

Yes, the device does not modify split queue descriptors. It only fills
in used ring elements and updates the used ring index.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [virtio-comment] multibuffer in packed queue

[Krasnov Arseniy]

[-- Attachment #1: Type: text/plain, Size: 5176 bytes --]

чт, 13 авг. 2020 г. в 17:29, Krasnov Arseniy <oxffffaa@gmail.com>:

> Ok, i understand about how virtio-net merge works. I asked about the
> following thing, consider we have two buffers(two descriptors each):
>
> buffer 1(two descriptors):
> | buf_1_1, NEXT, !USED, 1KB | | buf_1_2, !NEXT, !USED, 1KB|
>
> buffer 2(two descriptors):
>  | buf_2_1, NEXT, !USED, 1KB | | buf_2_2, !NEXT, !USED, 1KB|
>
> Both four descriptors placed sequentially in queue: buf_1_1, buf_1_2,
> buf_2_1, buf_2_2.
>
> Then 3KB network packet arrived and device fills buffers:
>   | buf_1_1, NEXT, USED, 0KB | | buf_1_2, !NEXT, USED, 0KB|
>
>
>
>  | buf_2_1, NEXT, USED, 0KB | | buf_2_2, !NEXT, USED, 512B|
>
> Is device allowed to flip NEXT bit of buf_1_2, two make chain of four
> descriptors(linking buf_1_2 and buf_2_1)?
>
> чт, 13 авг. 2020 г. в 15:16, Stefan Hajnoczi <stefanha@redhat.com>:
>
>> On Thu, Aug 13, 2020 at 10:57:52AM +0300, Krasnov Arseniy wrote:
>>
>>
>> > Hellow, I've got question about chapter 2.7.9:
>>
>>
>> >
>>
>>
>> > *  2.7.9 Multi-buffer requests Some devices combine multiple buffers as
>>
>>
>> > part of processing of a single request. These devices always mark the
>>
>>
>> > descriptor corresponding to the first buffer in the request used after
>> the
>>
>>
>> > rest of the descriptors (corresponding to rest of the buffers) in the
>>
>>
>> > request - which follow the first descriptor in ring order - has been
>> marked
>>
>>
>> > used and written out into the ring. This guarantees that the driver will
>>
>>
>> > never observe a partial request in the ring.  *
>>
>>
>> >
>>
>>
>> > Does it mean, that device can use sequence(chain) of descriptor unil
>>
>>
>> > descriptor with VIRTQ_DESC_F_NEXT  == 0 bit is found, OR device can use
>>
>>
>> > descriptors as many as it needs, don't care about  VIRTQ_DESC_F_NEXT
>> bit,
>>
>>
>> > and finally set  VIRTQ_DESC_F_NEXT bit in all used descriptors.
>>
>>
>>
>>
>>
>> 2.7.9 talks about 3 concepts:
>>
>>
>> 1. Requests (highest-level concept)
>>
>>
>> 2. Buffers
>>
>>
>> 3. Descriptors (lowest-level concept)
>>
>>
>>
>>
>>
>> They are distinct. A request is something like a received packet from a
>>
>>
>> network card. A buffer is a descriptor chain (1 or more descriptors with
>>
>>
>> the F_NEXT bit set). A descriptor is a single memory [start, end) range
>>
>>
>> that the device may read from or write to.
>>
>>
>>
>>
>>
>> The paragraph descibes how some devices complete one request by
>>
>>
>> combining multiple buffers. If a 8KB network packet arrives on a NIC and
>>
>>
>> the driver provided two 4KB buffers consisting of 1 4KB descriptor each
>>
>>
>> then the device may "merge" those two buffers (see virtio-net for
>>
>>
>> details). When doing so it must ensure that the first 4KB buffer of the
>>
>>
>> packet is marked used and written out to the ring *after* the second 4KB
>>
>>
>> buffer so that the driver sees the beginning of the network packet and
>>
>>
>> not the second half with the first half missing.
>>
>>
>>
>>
>>
>> Merging buffers is done at the device type-level (e.g. virtio-net), not
>>
>>
>> at a virtqueue-level (i.e. core virtio code). The device type must
>>
>>
>> specify exactly how merging works and device type-specific header fields
>> are
>>
>>
>> be used by the device to indicate which buffers have been merged into a
>>
>>
>> single request (see virtio-net's VIRTIO_NET_F_MRG_RXBUF feature bit and
>>
>>
>> num_buffers header field).
>>
>>
>>
>>
>>
>> > Also, in packed queue device can't modify descriptors except USED bit.
>>
>>
>>
>>
>>
>> No, the device *must* overwrite descriptors in the packed ring. Imagine
>>
>>
>> a device that completes buffers out-of-order:
>>
>>
>>
>>
>>
>> 1. The driver submits two buffers (F_NEXT is not set because a chain of
>>
>>
>>    descriptors would make the ASCII diagram more complex) and waits for
>>
>>
>>    the device:
>>
>>
>>
>>
>>
>>   | buf#1 !USED LEN=4KB | buf#2 !USED LEN=4KB |
>>
>>
>>
>>
>>
>> 2. The device completes buf#2 first (1KB of the 4KB descriptor was
>>
>>
>>    written) and writes the used descriptor to the ring:
>>
>>
>>
>>
>>
>>   | buf#2 USED LEN=1KB | buf#2 !USED LEN=4KB |
>>
>>
>>
>>
>>
>> 3. Let's say the driver waits for buf#1 to complete (3KB of the 4KB
>>
>>
>>    descriptor was written):
>>
>>
>>
>>
>>
>>   | buf#2 USED LEN=1KB | buf#1 USED LEN=3KB |
>>
>>
>>
>>
>>
>> You can see that the ring was overwritten by the device and the
>>
>>
>> descriptors were modified (e.g. the 4KB avail length was replaced by the
>>
>>
>> 1KB used length value).
>>
>>
>>
>>
>>
>> > In
>>
>>
>> > split queue, device also can't modify descriptor(it touches used buffer
>>
>>
>> > with index and length fields). Is that true?
>>
>>
>>
>>
>>
>> Yes, the device does not modify split queue descriptors. It only fills
>>
>>
>> in used ring elements and updates the used ring index.
>>
>>
>>
>>
>>
>> Stefan
>>
>>
>>
>
>

[-- Attachment #2: Type: text/html, Size: 5913 bytes --]