vif interface dropping 80% packets

vif interface dropping 80% packets

[Reese Faucette]

[-- Attachment #1.1: Type: text/plain, Size: 1438 bytes --]

I'm trying to characterize the performance of 10G ethernet to a user domain.  
The machine has 2 configured ethernet cards, eth0 is a 1G card, and eth2 is a 10G card.  (eth1 is another 1G card and eth3 is another 10G, but these are not configured)
I have network bridge set up on eth2 with defaults, nothing special, just:
   /etc/xen/scripts/network-bridge start vifnum=2
I have a user domain that attaches eth1 to this vif, and when I try netperf tests, I get tons of packet loss in the vif interface.

vif3.1    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:165602 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12982847 errors:0 dropped:48987465 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:8679444 (8.2 MiB)  TX bytes:20888548648 (19.4 GiB)

As can be seen above, vif3.1 drops about 80% of the packets it should transmit.  Is this expected due to something in the dom0 -> domU interface just not being able to keep up, or is there something I can do about it?  It almost looks like the dom0->domU link is throttled at about 1Gb/s.  I say this because the 10G link is running about 50-60% capacity when the drop rate is 80%.

Everything has MTU 1500, btw - I noticed that Xen is pretty unhappy with an MTU of 9000.

thanks,
-reese

[-- Attachment #1.2: Type: text/html, Size: 2783 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: vif interface dropping 80% packets

[Reese Faucette]

[-- Attachment #1.1: Type: text/plain, Size: 428 bytes --]

sorry - missing some possibly useful info:  2.6.18.5 with the fedora patch, Xen 3.0.3, no other mods.  dual 3GHz Xeon.  2 machines connected back-to-back, no ethernet switch.
  I'm trying to characterize the performance of 10G ethernet to a user domain.  
  The machine has 2 configured ethernet cards, eth0 is a 1G card, and eth2 is a 10G card.  (eth1 is another 1G card and eth3 is another 10G, but these are not configured)

[-- Attachment #1.2: Type: text/html, Size: 1017 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: vif interface dropping 80% packets

[Ian Pratt]

> I'm trying to characterize the performance of 10G ethernet to a user
> domain.
> The machine has 2 configured ethernet cards, eth0 is a 1G card, and
eth2 is
> a 10G card.  (eth1 is another 1G card and eth3 is another 10G, but
these
> are not configured)
> I have network bridge set up on eth2 with defaults, nothing special,
just:
>    /etc/xen/scripts/network-bridge start vifnum=2
> I have a user domain that attaches eth1 to this vif, and when I try
netperf
> tests, I get tons of packet loss in the vif interface.
>
> As can be seen above, vif3.1 drops about 80% of the packets it should
> transmit.  Is this expected due to something in the dom0 -> domU
interface
> just not being able to keep up, or is there something I can do about
it?
> It almost looks like the dom0->domU link is throttled at about 1Gb/s.
I
> say this because the 10G link is running about 50-60% capacity when
the
> drop rate is 80%.

First off, try switching to 3.0.4 and use the included 2.6.16-xen
kernel. 

How are you measuring the bandwidth? It's much more realistic to use TCP
as open-loop UDP blasts aren't used by any real protocols. 

> Everything has MTU 1500, btw - I noticed that Xen is pretty unhappy
with an
> MTU of 9000.

In what way?  I'd expect larger MTUs to work OK, though I haven't tested
them myself.

Does the larger MTU work in dom0, but not through to other domU's?

You may need to set the MTU on the bridge and all vif's.


Ian
 

RE: vif interface dropping 80% packets

[Santos, Jose Renato G]

[-- Attachment #1.1: Type: text/plain, Size: 2551 bytes --]

Most likely your dom0 CPU is saturated.. It is unrealistic to expect
full throughput of a 10 Gig NIC. I would be surprised if even linux
could keep up with a receive rate of 10 Gb/s. Did you try the same
experiment on native linux? In  my experiments, on a 4-way 2.8 Ghz Xeon,
dom0 consumes ~75% of one cpu for processing receive packets while the
guest consume ~55% of another CPU. It is  expected that your dual CPU
system saturates at a rate slighlty above 1 Ghz. (You are also consuming
extra cycles to drop packets)
You could confirm that the CPU is saturating by running xentop while
running your experiments.
You could avoid dropping packets and get a more realistic experiment by
running a TCP experiment as Ian suggested
 
Regards
 
Renato


________________________________

	From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Reese
Faucette
	Sent: Monday, February 05, 2007 4:31 PM
	To: xen-devel@lists.xensource.com
	Subject: [Xen-devel] vif interface dropping 80% packets
	
	
	I'm trying to characterize the performance of 10G ethernet to a
user domain.  
	The machine has 2 configured ethernet cards, eth0 is a 1G card,
and eth2 is a 10G card.  (eth1 is another 1G card and eth3 is another
10G, but these are not configured)
	I have network bridge set up on eth2 with defaults, nothing
special, just:
	   /etc/xen/scripts/network-bridge start vifnum=2
	I have a user domain that attaches eth1 to this vif, and when I
try netperf tests, I get tons of packet loss in the vif interface.
	 
	vif3.1    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
	          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
	          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
	          RX packets:165602 errors:0 dropped:0 overruns:0
frame:0
	          TX packets:12982847 errors:0 dropped:48987465
overruns:0 carrier:0
	          collisions:0 txqueuelen:0
	          RX bytes:8679444 (8.2 MiB)  TX bytes:20888548648 (19.4
GiB)
	
	As can be seen above, vif3.1 drops about 80% of the packets it
should transmit.  Is this expected due to something in the dom0 -> domU
interface just not being able to keep up, or is there something I can do
about it?  It almost looks like the dom0->domU link is throttled at
about 1Gb/s.  I say this because the 10G link is running about 50-60%
capacity when the drop rate is 80%.
	 
	Everything has MTU 1500, btw - I noticed that Xen is pretty
unhappy with an MTU of 9000.
	 
	thanks,
	-reese
	 


[-- Attachment #1.2: Type: text/html, Size: 5296 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: vif interface dropping 80% packets

[Reese Faucette]

[-- Attachment #1.1: Type: text/plain, Size: 1077 bytes --]



  Most likely your dom0 CPU is saturated.. It is unrealistic to expect full throughput of a 10 Gig NIC. I would be surprised if even linux could keep up with a receive rate of 10 Gb/s. Did you try the same experiment on native linux? In  my experiments, on a 4-way 2.8 Ghz Xeon, dom0 consumes ~75% of one cpu for processing receive packets while the guest consume ~55% of another CPU.
With native->native or even native->dom0, I get 9.88 Gb/s with 21% utilization of receive side CPU.  It's a nice machine and we make nice NICs ;-)  So, it's not unrealistic.
  It is  expected that your dual CPU system saturates at a rate slighlty above 1 Ghz. (You are also consuming extra cycles to drop packets)
  You could confirm that the CPU is saturating by running xentop while running your experiments.
  You could avoid dropping packets and get a more realistic experiment by running a TCP experiment as Ian suggested
i have a pending note on that, TCP test is actually what I tried first, and the perf was terrible due to the same packet loss.  more info to come.
-reese

[-- Attachment #1.2: Type: text/html, Size: 2926 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: vif interface dropping 80% packets

[Santos, Jose Renato G]

[-- Attachment #1.1: Type: text/plain, Size: 1961 bytes --]

You are probably using jumbo packets for native, right? What is the MTU
when you get 9.88 Gb/s? I was thinking MTU=1500  when I said
unrealistic. Not sure why you cannot get higher MTU in Xen to work. I
have not tried MTU > 1500 bytes in Xen myself  but I believe Herbert Xu
has done that successfully.
 
If you get larger MTU to work, keep in mind that in Xen there is an
extra data copy to transfer the packet from dom0 to guest which will
have a higher impact on performance for larger MTUs when compared to
linux. 
 
Renato


________________________________

	From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Reese
Faucette
	Sent: Tuesday, February 06, 2007 10:59 AM
	To: xen-devel@lists.xensource.com
	Subject: Re: [Xen-devel] vif interface dropping 80% packets
	
	
	
	 

		Most likely your dom0 CPU is saturated.. It is
unrealistic to expect full throughput of a 10 Gig NIC. I would be
surprised if even linux could keep up with a receive rate of 10 Gb/s.
Did you try the same experiment on native linux? In  my experiments, on
a 4-way 2.8 Ghz Xeon, dom0 consumes ~75% of one cpu for processing
receive packets while the guest consume ~55% of another CPU.

	With native->native or even native->dom0, I get 9.88 Gb/s with
21% utilization of receive side CPU.  It's a nice machine and we make
nice NICs ;-)  So, it's not unrealistic.

		It is  expected that your dual CPU system saturates at a
rate slighlty above 1 Ghz. (You are also consuming extra cycles to drop
packets)
		You could confirm that the CPU is saturating by running
xentop while running your experiments.
		You could avoid dropping packets and get a more
realistic experiment by running a TCP experiment as Ian suggested

	i have a pending note on that, TCP test is actually what I tried
first, and the perf was terrible due to the same packet loss.  more info
to come.
	-reese
	 


[-- Attachment #1.2: Type: text/html, Size: 4853 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: vif interface dropping 80% packets

[Reese Faucette]

> You are probably using jumbo packets for native, right? What is the MTU 
> when you get
> 9.88 Gb/s? I was thinking MTU=1500  when I said unrealistic. Not sure why 
> you cannot
> get higher MTU in Xen to work. I have not tried MTU > 1500 bytes in Xen 
> myself  but
> I believe Herbert Xu has done that successfully.

ok, true - got me there.  TCP test with native linux sending to dom0 and 
1500 MTU is 9.1GB/s and 23% of 1 CPU used on receiver.

> If you get larger MTU to work, keep in mind that in Xen there is an extra 
> data copy
> to transfer the packet from dom0 to guest which will have a higher impact 
> on performance
> for larger MTUs when compared to linux.

of course - i don't expect there to be no impact, but the numbers i was 
getting with mtu=1500 were horrid, like 70Mb/s using netperf with 
TCP_SENDFILE.  a slower sender actually resulted in better throughput due to 
fewer dropped packets.  (e.g. using TCP_STREAM got 220Mb/s, better but still 
stinky)

update:
MTU=9000 does work, I had failed to set one of the vif interfaces - plenty 
of interfaces to change ;-)
The performance is better with 9k MTU, as expected:
TCP_STREAM and TCP_SENDFILE both get ~1.8Gb/s, xentop reports 87% 
utilization for dom0, so I guess that's about all I can expect without code 
changes.

If I understand the Xen bridge network flow, though, it's a real bummer that

> First off, try switching to 3.0.4 and use the included 2.6.16-xen kernel.

I can try, but the issue i had before was that 2.6.16 did not have the
ethernet driver I need for the main ethernet connection:
0000:06:00.1 Ethernet controller: Intel Corporation Enterprise Southbridge 
DPT LAN Copper (rev 01)
It comes with 2.6.18, but not 2.6.16, and the driver does not compile under
2.6.16, so i was stuck with 2.6.18.

before I spend a lot of time converting to 3.0.4, a couple of questions:
1) any reason to expect better perf on 3.0.4 ?
2) if yes, Will the patch "fedora-36252.patch" work on the 2.6.16 kernel 
included with 3.0.4, or have conflicts crept in?

thanks!
-reese

RE: vif interface dropping 80% packets

[Ian Pratt]

> update:
> MTU=9000 does work, I had failed to set one of the vif interfaces -
plenty
> of interfaces to change ;-)
> The performance is better with 9k MTU, as expected:
> TCP_STREAM and TCP_SENDFILE both get ~1.8Gb/s, xentop reports 87%
> utilization for dom0, so I guess that's about all I can expect without
code
> changes.

I've seen rather better figures than this with another 10G card.

Does your NIC/driver support TSO and LRO?

> before I spend a lot of time converting to 3.0.4, a couple of
questions:
> 1) any reason to expect better perf on 3.0.4 ?

Yes, there were changes in the RX path.

> 2) if yes, Will the patch "fedora-36252.patch" work on the 2.6.16
kernel
> included with 3.0.4, or have conflicts crept in?

No idea. What does the patch do?

Ian

Re: vif interface dropping 80% packets

[Reese Faucette]

>> update:
>> The performance is better with 9k MTU, as expected:
>> TCP_STREAM and TCP_SENDFILE both get ~1.8Gb/s, xentop reports 87%
>> utilization for dom0, so I guess that's about all I can expect without
>> code changes.
>
> I've seen rather better figures than this with another 10G card.
> Does your NIC/driver support TSO and LRO?

yes to both, and have tried with LRO both enabled and disabled.  again, the 
dom0 performance is fine, it's just to domU that is slower.  I have network 
bridging set up on this interface, i.e. peth2, eth2, xenbr2, etc. - is that 
not the way I should be doing it?

>> 2) if yes, Will the patch "fedora-36252.patch" work on the 2.6.16
>> kernel included with 3.0.4, or have conflicts crept in?
>
> No idea. What does the patch do?

This was the mythical patch to make 2.6.18 into a Xen-aware kernel.  Took me 
forever to track it down - is there a more obvious place to look?  Sorry to 
just refer to the file by name, after googling forever i decided i was the 
only person in the Xen world not intimately familiar with this patch.
-reese

RE: vif interface dropping 80% packets

[Ian Pratt]

> > I've seen rather better figures than this with another 10G card.
> > Does your NIC/driver support TSO and LRO?
> 
> yes to both, and have tried with LRO both enabled and disabled.
again, the
> dom0 performance is fine, it's just to domU that is slower.  I have
network
> bridging set up on this interface, i.e. peth2, eth2, xenbr2, etc. - is
that
> not the way I should be doing it?

I presume this is a box with >1 core? Try running dom0 and the domU with
one VCPU to keep things simple. The scheduler should put them on
different physical CPUs.

The packets are getting dropped being passed from the bridge into the
domU, because the domU isn't keeping up and creating slots in the ring
buffer. For a TCP connection, this is surprising as you'd have to have a
big socket buffer to saturate the ring buffers. Switching to 3.0.4 or
unstable will certainly help. 

You could try putting netback.queue_length=256 on the module or dom0
kernel command line, but a better fix would be to increase the ring
buffer size.

> >> 2) if yes, Will the patch "fedora-36252.patch" work on the 2.6.16
> >> kernel included with 3.0.4, or have conflicts crept in?
> >
> > No idea. What does the patch do?
> 
> This was the mythical patch to make 2.6.18 into a Xen-aware kernel.
Took
> me
> forever to track it down - is there a more obvious place to look?
Sorry to
> just refer to the file by name, after googling forever i decided i was
the
> only person in the Xen world not intimately familiar with this patch.

3.0.4 and xen-unstable currently come with a patched linux kernel src
for convenience (though the intention is to move this out of tree right
after 3.0.5 is released).

Ian 

RE: vif interface dropping 80% packets

[Santos, Jose Renato G]

> This was the mythical patch to make 2.6.18 into a Xen-aware 
> kernel.  Took me forever to track it down - is there a more 
> obvious place to look?  Sorry to just refer to the file by 
> name, after googling forever i decided i was the only person 
> in the Xen world not intimately familiar with this patch.
> -reese
> 
  You should consider trying xen-unstable which currently includes linux
2.6.18.
  Renato
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: vif interface dropping 80% packets

[Reese Faucette]

Indeed, xen-unstable is MUCH better - TCP_SENDFILE looks more like 6.5Gb/s 
on my first try from native to domU with no fiddling.
The fact that xen-unstable is based on 2.6.18 is a big help, also.

Ian, is this more in line with what you have seen before?

thanks for the pointers,
-reese.

RE: vif interface dropping 80% packets

[Ian Pratt]

> Indeed, xen-unstable is MUCH better - TCP_SENDFILE looks more like
6.5Gb/s
> on my first try from native to domU with no fiddling.
> The fact that xen-unstable is based on 2.6.18 is a big help, also.
> 
> Ian, is this more in line with what you have seen before?

That's more like it. If you work with us and do some profiling etc we
can probably help you improve it some more. 

If you want really good network performance in domU's you should write a
driver that gives guests direct access to a virtual interface on the
card in a safe and protected manner. The solarflare guys have got some
excellent results by doing this, and have made their design and patches
available. We're hoping to get something based on their proposed
architecture into mainline Xen, and are looking for interested parties
to help...

Ian