Networking performance on a KVM Host (with no guests)

Networking performance on a KVM Host (with no guests)

[Chegu Vinod]

Hello,

Perhaps this query was answered in the past. If yes kindly point me to 
the same.

We noticed differences in networking performance (measured via netperf 
over a 10G  NIC) on an X86_64 server between the following two 
configurations :

1) Server run as a KVM Host (but with no KVM guests created on it (or) no 
extra  bridges created other than the default vibr0 bridge)  

vs. 

2) The same server running the same version of Linux but without any of the 
virtualization software installed on it. 

Config #2 performed much better ! 
Is this expected ? What are the reasons behind this ? 

Is there any way to gain back the loss in performance on the KVM host 
without having to uninstall the virtualization software (i.e. in Config 1).

Thanks!
Vinod

Re: Networking performance on a KVM Host (with no guests)

[Gleb Natapov]

On Thu, Apr 19, 2012 at 03:53:39AM +0000, Chegu Vinod wrote:
> 
> Hello,
> 
> Perhaps this query was answered in the past. If yes kindly point me to 
> the same.
> 
> We noticed differences in networking performance (measured via netperf 
> over a 10G  NIC) on an X86_64 server between the following two 
> configurations :
> 
> 1) Server run as a KVM Host (but with no KVM guests created on it (or) no 
> extra  bridges created other than the default vibr0 bridge)  
> 
> vs. 
> 
> 2) The same server running the same version of Linux but without any of the 
> virtualization software installed on it. 
> 
> Config #2 performed much better ! 
> Is this expected ? What are the reasons behind this ? 
> 
> Is there any way to gain back the loss in performance on the KVM host 
> without having to uninstall the virtualization software (i.e. in Config 1).
> 
Can you list exactly what you need to uninstall to get performance back?

--
			Gleb.

Re: Networking performance on a KVM Host (with no guests)

[Jean-Philippe Menil]

Le 19/04/2012 07:43, Gleb Natapov a écrit :
> On Thu, Apr 19, 2012 at 03:53:39AM +0000, Chegu Vinod wrote:
>> Hello,
>>
>> Perhaps this query was answered in the past. If yes kindly point me to
>> the same.
>>
>> We noticed differences in networking performance (measured via netperf
>> over a 10G  NIC) on an X86_64 server between the following two
>> configurations :
>>
>> 1) Server run as a KVM Host (but with no KVM guests created on it (or) no
>> extra  bridges created other than the default vibr0 bridge)
>>
>> vs.
>>
>> 2) The same server running the same version of Linux but without any of the
>> virtualization software installed on it.
>>
>> Config #2 performed much better !
>> Is this expected ? What are the reasons behind this ?
>>
>> Is there any way to gain back the loss in performance on the KVM host
>> without having to uninstall the virtualization software (i.e. in Config 1).
>>
> Can you list exactly what you need to uninstall to get performance back?
>
> --
> 			Gleb.
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
Hi,

can you precise your ethernet card, and if in the second case, you have 
activate the virtualiation in the bios.

Regards.

-- 
Jean-Philippe Menil - Pôle réseau Service IRTS
DSI Université de Nantes
jean-philippe.menil@univ-nantes.fr
Tel : 02.53.48.49.27 - Fax : 02.53.48.49.09

Re: Networking performance on a KVM Host (with no guests)

[Chegu Vinod]

On 4/18/2012 10:43 PM, Gleb Natapov wrote:
> On Thu, Apr 19, 2012 at 03:53:39AM +0000, Chegu Vinod wrote:
>> Hello,
>>
>> Perhaps this query was answered in the past. If yes kindly point me to
>> the same.
>>
>> We noticed differences in networking performance (measured via netperf
>> over a 10G  NIC) on an X86_64 server between the following two
>> configurations :
>>
>> 1) Server run as a KVM Host (but with no KVM guests created on it (or) no
>> extra  bridges created other than the default vibr0 bridge)
>>
>> vs.
>>
>> 2) The same server running the same version of Linux but without any of the
>> virtualization software installed on it.
>>
>> Config #2 performed much better !
>> Is this expected ? What are the reasons behind this ?
>>
>> Is there any way to gain back the loss in performance on the KVM host
>> without having to uninstall the virtualization software (i.e. in Config 1).
>>
>>

Just an observation   :

Removing the "intel_iommu=on" boot time parameter  in the Config 1 case 
seemed to help
gain back the networking performance on the KVM host... i.e. performed 
nearly the same as
in Config2.   Didn't have to change any of the VT settings in the BIOS...

Trying to understand why that's the case...   Any pointers ?

Vinod

> --
> 			Gleb.
>

Re: Networking performance on a KVM Host (with no guests)

[Nadav Har'El]

On Fri, Apr 20, 2012, Chegu Vinod wrote about "Re: Networking performance on a KVM Host (with no guests)":
> Removing the "intel_iommu=on" boot time parameter  in the Config 1
> case seemed to help

"intel_iommu=on" is essential with you're mostly running guests *and*
using device assignment.

However, unfortunately, it also has a serious performance penalty for
I/O in *host* processes:

When intel_iommu=on, Linux (completely unrelated to KVM) adds a new level
of protection which didn't exist without an IOMMU - the network card, which
without an IOMMU could write (via DMA) to any memory location, now is
not allowed - the card can only write to memory locates which the OS
wanted it to write. Theoretically, this can protect the OS against
various kinds of attacks. But what happens now is that every time that
Linux passes a new buffer to the card, it needs to change the IOMMU
mappings. This noticably slows down I/O, unfortunately.

-- 
Nadav Har'El                        |                    Friday, Apr 20 2012, 
nyh@math.technion.ac.il             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |A bird in the hand is safer than one
http://nadav.harel.org.il           |overhead.

Re: Networking performance on a KVM Host (with no guests)

[Chegu Vinod]

Nadav Har'El <nyh <at> math.technion.ac.il> writes:

> 
> On Fri, Apr 20, 2012, Chegu Vinod wrote about "Re: Networking performance on a 
KVM Host (with no guests)":
> > Removing the "intel_iommu=on" boot time parameter  in the Config 1
> > case seemed to help
> 
> "intel_iommu=on" is essential with you're mostly running guests *and*
> using device assignment.
> 

Yes. I do understand that...

> However, unfortunately, it also has a serious performance penalty for
> I/O in *host* processes:
> 
> When intel_iommu=on, Linux (completely unrelated to KVM) adds a new level
> of protection which didn't exist without an IOMMU - the network card, which
> without an IOMMU could write (via DMA) to any memory location, now is
> not allowed - the card can only write to memory locates which the OS
> wanted it to write. Theoretically, this can protect the OS against
> various kinds of attacks. But what happens now is that every time that
> Linux passes a new buffer to the card, it needs to change the IOMMU
> mappings. This noticably slows down I/O, unfortunately.
> 

Hmm...  So if one were to have a private link setup between two hosts to drive 
some traffic through between the hosts... then we can't expect to get line rate 
on that private NIC. 

Thanks much for your response !
Vinod

Re: Networking performance on a KVM Host (with no guests)

[Nadav Har'El]

On Sat, Apr 21, 2012, Chegu Vinod wrote about "Re: Networking performance on a KVM Host (with no guests)":
> some traffic through between the hosts... then we can't expect to get line rate 
> on that private NIC. 

If you're not using device assignment, then just disable intel_iommu.

If you *are* using device assignment, try setting intel_iommu to something
other then on. Maybe intel_iommu=pt will work (see
http://lwn.net/Articles/329174/). But I actually never tried this myself -
so let me know if it works ;-)

Is anyone else aware of a different "best practice", on how to enable
VT-d for device assignment, without any changes (in performance or
security) in the host?

-- 
Nadav Har'El                        |                    Sunday, Apr 22 2012, 
nyh@math.technion.ac.il             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |Sign in zoo: Do not feed the animals. If
http://nadav.harel.org.il           |you have food give it to the guard on duty