Re: lkvm: virtio-net-rx general protection error

Re: lkvm: virtio-net-rx general protection error

[Pekka Enberg]

On 10/21/13 1:35 PM, Milan Kocian wrote:
> hi,
>
> sorry for writing it directly to you but I didn't find better recipient.
> Does exist some mailing-list about lkvm?
>
> I found the crash in virtio-net-rx thread (I can reproduce it every time
> by 'aptitude update' in VM):
>
> traps: virtio-net-rx[28933] general protection ip:7f00dda3d107 sp:7f00c58f4de8 error:0 in libc-2.17.so[7f00dd90f000+1a2000]
>
> gdb backtrace:
>
> (gdb) bt
> #0  0x00007fb6a548e107 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #1  0x000000000041259c in memcpy_toiovecend (iov=0x7fb68d346ea0, iov@entry=0x7fb68d345e90,
>      kdata=<optimized out>, kdata@entry=0x7fb68d346e90 "", offset=<optimized out>, len=<optimized out>)
>      at util/iovec.c:70
> #2  0x000000000040c66d in virtio_net_rx_thread (p=0x23688a0) at virtio/net.c:117
> #3  0x00007fb6a5b2ee0e in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007fb6a54489ed in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
>
> I tried to add some printf to diagnose it but it isn't clear to me:
>
> virtio_net_rx_thread: before memcpy_toiovecend; copied: 0, len: 18890, iovsize: 4096, realiovsize: 4096
> memcpy_toiovecend: offset: 0, len: 4096
> memcpy_toiovecend: iov_len: 4096, len: 4096
> virtio_net_rx_thread: before memcpy_toiovecend; copied: 4096, len: 18890, iovsize: 4096, realiovsize: 4096
> memcpy_toiovecend: offset: 4096, len: 4096
> memcpy_toiovecend: iov_len: 4096, len: 4096
> memcpy_toiovecend: iov_len: 0, len: 4096
> memcpy_toiovecend: iov_len: 0, len: 4096
> .
> N x memcpy_toiovecend: iov_len: 0, len: 4096
> .
> memcpy_toiovecend: iov_len: 0, len: 4096
> memcpy_toiovecend: iov_len: 0, len: 4096
> memcpy_toiovecend: iov_len: 1519143547641528320, len: 4096
> memcpy_toiovecend: iov_len: 193827583623176, len: 4096
> ./runlkvm.sh: line 2: 16090 Segmentation fault
>
>
> IMHO problem come when received len size is bigger than maximum
> of the dst iovec (realiovsize). Only iovec size is copied and in the next
> run isn't place to copy the rest of len size.
>
> So solution may be increase dst iovec size or send data in dst iovec
> to user (but i don't know how, I am not virtio expert :-)).

I'm CC'ing Asias, Sasha and others.

                 Pekka

Re: lkvm: virtio-net-rx general protection error

[Asias He]

[-- Attachment #1: Type: text/plain, Size: 2422 bytes --]

On Mon, Oct 21, 2013 at 8:18 PM, Pekka Enberg <penberg@iki.fi> wrote:
> On 10/21/13 1:35 PM, Milan Kocian wrote:
>>
>> hi,
>>
>> sorry for writing it directly to you but I didn't find better recipient.
>> Does exist some mailing-list about lkvm?
>>
>> I found the crash in virtio-net-rx thread (I can reproduce it every time
>> by 'aptitude update' in VM):
>>
>> traps: virtio-net-rx[28933] general protection ip:7f00dda3d107
>> sp:7f00c58f4de8 error:0 in libc-2.17.so[7f00dd90f000+1a2000]
>>
>> gdb backtrace:
>>
>> (gdb) bt
>> #0  0x00007fb6a548e107 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x000000000041259c in memcpy_toiovecend (iov=0x7fb68d346ea0,
>> iov@entry=0x7fb68d345e90,
>>      kdata=<optimized out>, kdata@entry=0x7fb68d346e90 "",
>> offset=<optimized out>, len=<optimized out>)
>>      at util/iovec.c:70
>> #2  0x000000000040c66d in virtio_net_rx_thread (p=0x23688a0) at
>> virtio/net.c:117
>> #3  0x00007fb6a5b2ee0e in start_thread () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> #4  0x00007fb6a54489ed in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>
>>
>> I tried to add some printf to diagnose it but it isn't clear to me:
>>
>> virtio_net_rx_thread: before memcpy_toiovecend; copied: 0, len: 18890,
>> iovsize: 4096, realiovsize: 4096
>> memcpy_toiovecend: offset: 0, len: 4096
>> memcpy_toiovecend: iov_len: 4096, len: 4096
>> virtio_net_rx_thread: before memcpy_toiovecend; copied: 4096, len: 18890,
>> iovsize: 4096, realiovsize: 4096
>> memcpy_toiovecend: offset: 4096, len: 4096
>> memcpy_toiovecend: iov_len: 4096, len: 4096
>> memcpy_toiovecend: iov_len: 0, len: 4096
>> memcpy_toiovecend: iov_len: 0, len: 4096
>> .
>> N x memcpy_toiovecend: iov_len: 0, len: 4096
>> .
>> memcpy_toiovecend: iov_len: 0, len: 4096
>> memcpy_toiovecend: iov_len: 0, len: 4096
>> memcpy_toiovecend: iov_len: 1519143547641528320, len: 4096
>> memcpy_toiovecend: iov_len: 193827583623176, len: 4096
>> ./runlkvm.sh: line 2: 16090 Segmentation fault
>>
>>
>> IMHO problem come when received len size is bigger than maximum
>> of the dst iovec (realiovsize). Only iovec size is copied and in the next
>> run isn't place to copy the rest of len size.
>>
>> So solution may be increase dst iovec size or send data in dst iovec
>> to user (but i don't know how, I am not virtio expert :-)).
>
>
> I'm CC'ing Asias, Sasha and others.

Hello Milan,

Does the attached patch fix your problem?

-- 
Asias

[-- Attachment #2: 0001-kvm-tools-Fix-virtio-net-iov-memcpy.patch --]
[-- Type: text/x-patch, Size: 966 bytes --]

From b48eaeff7250bf7476c771e82cdbf20c3e85c4c9 Mon Sep 17 00:00:00 2001
From: Asias He <asias.hejun@gmail.com>
Date: Mon, 28 Oct 2013 15:02:54 +0800
Subject: [PATCH 1/1] kvm-tools: Fix virtio-net iov memcpy

We should skip copied bytes from the buffer not from the iov itself
which memcpy_toiovecend does.

Signed-off-by: Asias He <asias.hejun@gmail.com>
---
 tools/kvm/virtio/net.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/kvm/virtio/net.c b/tools/kvm/virtio/net.c
index 2c34996..3715aaf 100644
--- a/tools/kvm/virtio/net.c
+++ b/tools/kvm/virtio/net.c
@@ -114,7 +114,7 @@ static void *virtio_net_rx_thread(void *p)
 			while (copied < len) {
 				size_t iovsize = min(len - copied, iov_size(iov, in));
 
-				memcpy_toiovecend(iov, buffer, copied, iovsize);
+				memcpy_toiovec(iov, buffer + copied, iovsize);
 				copied += iovsize;
 				if (has_virtio_feature(ndev, VIRTIO_NET_F_MRG_RXBUF))
 					hdr->num_buffers++;
-- 
1.8.3.1

Re: lkvm: virtio-net-rx general protection error

[Milan Kocian]

hello,

On Mon, Oct 28, 2013 at 04:28:57PM +0800, Asias He wrote:
> 
> Hello Milan,
> 
> Does the attached patch fix your problem?
> 
> -- 
> Asias

> From b48eaeff7250bf7476c771e82cdbf20c3e85c4c9 Mon Sep 17 00:00:00 2001
> From: Asias He <asias.hejun@gmail.com>
> Date: Mon, 28 Oct 2013 15:02:54 +0800
> Subject: [PATCH 1/1] kvm-tools: Fix virtio-net iov memcpy
> 
> We should skip copied bytes from the buffer not from the iov itself
> which memcpy_toiovecend does.
> 
> Signed-off-by: Asias He <asias.hejun@gmail.com>
> ---
>  tools/kvm/virtio/net.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tools/kvm/virtio/net.c b/tools/kvm/virtio/net.c
> index 2c34996..3715aaf 100644
> --- a/tools/kvm/virtio/net.c
> +++ b/tools/kvm/virtio/net.c
> @@ -114,7 +114,7 @@ static void *virtio_net_rx_thread(void *p)
>  			while (copied < len) {
>  				size_t iovsize = min(len - copied, iov_size(iov, in));
>  
> -				memcpy_toiovecend(iov, buffer, copied, iovsize);
> +				memcpy_toiovec(iov, buffer + copied, iovsize);
>  				copied += iovsize;
>  				if (has_virtio_feature(ndev, VIRTIO_NET_F_MRG_RXBUF))
>  					hdr->num_buffers++;
> -- 
> 1.8.3.1
> 

Excellent, this patch fixes the problem. Feel free to add: 

Tested-by: Milan Kocian <milon@wq.cz>

Many thanks.

-- 
Milan Kocian