kvm network latency, higher with virtio ?

kvm network latency, higher with virtio ?

[Luca Bigliardi]

Hi,
I'm running some tests between two linux instances bridged together.

If I try to ping 10 times I obtain the following results:

-net nic,model=virtio -net tap :
    rtt min/avg/max/mdev = 0.756/0.967/2.115/0.389 ms

-net nic,model=rtl8139 -net tap :
    rtt min/avg/max/mdev = 0.301/0.449/1.173/0.248 ms

So it seems with virtio the latency is higher. Is it normal?

The results I'm reporting were obtained with
- host
  qemu-kvm 0.11-rc2
  kvm-kmod-2.6.30.1
  kernel: 2.6.30.5 (HIGH_RES_TIMERS=y as suggested in
                    http://www.linux-kvm.org/page/Virtio )
- guest
  kernel: 2.6.31

but I also tested older versions always obtaining latency values at least two
times higher than rtl8139/e1000 .

Thank you,
Luca

Re: kvm network latency, higher with virtio ?

[Michael S. Tsirkin]

On Tue, Sep 15, 2009 at 05:15:09PM +0200, Luca Bigliardi wrote:
> Hi,
> I'm running some tests between two linux instances bridged together.
> 
> If I try to ping 10 times I obtain the following results:
> 
> -net nic,model=virtio -net tap :
>     rtt min/avg/max/mdev = 0.756/0.967/2.115/0.389 ms
> 
> -net nic,model=rtl8139 -net tap :
>     rtt min/avg/max/mdev = 0.301/0.449/1.173/0.248 ms
> 
> So it seems with virtio the latency is higher. Is it normal?

Yes, the main reason is the TX timer it uses for interrupt/vm exit mitigation.

> The results I'm reporting were obtained with
> - host
>   qemu-kvm 0.11-rc2
>   kvm-kmod-2.6.30.1
>   kernel: 2.6.30.5 (HIGH_RES_TIMERS=y as suggested in
>                     http://www.linux-kvm.org/page/Virtio )
> - guest
>   kernel: 2.6.31
> 
> but I also tested older versions always obtaining latency values at least two
> times higher than rtl8139/e1000 .
> 
> Thank you,
> Luca
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kvm network latency, higher with virtio ?

[Dor Laor]

On 09/16/2009 10:27 AM, Michael S. Tsirkin wrote:
> On Tue, Sep 15, 2009 at 05:15:09PM +0200, Luca Bigliardi wrote:
>> Hi,
>> I'm running some tests between two linux instances bridged together.
>>
>> If I try to ping 10 times I obtain the following results:
>>
>> -net nic,model=virtio -net tap :
>>      rtt min/avg/max/mdev = 0.756/0.967/2.115/0.389 ms
>>
>> -net nic,model=rtl8139 -net tap :
>>      rtt min/avg/max/mdev = 0.301/0.449/1.173/0.248 ms
>>
>> So it seems with virtio the latency is higher. Is it normal?
>
> Yes, the main reason is the TX timer it uses for interrupt/vm exit mitigation.

Originally we used the tx mitigation timer in order to provide better 
throughput on the expense of latency.
Measurements of older versions of virtio proved that we can cancel this 
timer and achieve better latency while not hurting throughput.

Vhost wouldn't use it. For the time being until be get vhost, we should 
probably remove it from qemu.

>
>> The results I'm reporting were obtained with
>> - host
>>    qemu-kvm 0.11-rc2
>>    kvm-kmod-2.6.30.1
>>    kernel: 2.6.30.5 (HIGH_RES_TIMERS=y as suggested in
>>                      http://www.linux-kvm.org/page/Virtio )
>> - guest
>>    kernel: 2.6.31
>>
>> but I also tested older versions always obtaining latency values at least two
>> times higher than rtl8139/e1000 .
>>
>> Thank you,
>> Luca
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kvm network latency, higher with virtio ?

[Luca Bigliardi]

On Wed, Sep 16, 2009 at 11:47:07AM +0300, Dor Laor wrote:

Hi!
Thank you for explaining.

> Measurements of older versions of virtio proved that we can cancel this  
> timer and achieve better latency while not hurting throughput.

Well, I did try the patch Michael is suggesting here:
http://lkml.org/lkml/2009/8/17/140

And I should say that in my testing environment not only the throughput doesn't
decrease, but in some cases it grows (tested using netperf with udp-stream).

> Vhost wouldn't use it. For the time being until be get vhost, we should  
> probably remove it from qemu.

Yes please, check with your tests and consider this.


Thank you,

Luca