Constantly varying download rate with a complex xen networking setup, why?

Constantly varying download rate with a complex xen networking setup, why?

[Raphael Hertzog]

[ Please put me in CC of your answers ]

Hello,

I have a weird problem and I'm not sure if it comes from linux or from
Xen. Thus I'd like your opinion and maybe your help so that I can
do a proper bugreport where it belongs. I can do supplementary tests on
request of course.

I probably should try to reproduce it with kvm and a newer kernel but the
hardware where this got tested doesn't have the CPU instructions for
virtualization. If you don't have any idea of what can be wrong in this
setup, I will try to arrange that test but otherwise I would like your
feedback before I do more tests.

(Note the full description below has also been submitted to
http://serverfault.com/questions/22219/)

The network configuration in my Xen setup:
- the dom0 has 3 network cards (eth0, eth1, eth2), 3 brigdes (xenbrE,
  xenbrI, xenbrD) and each brigde integrates the corresponding network
  card. Only xenbrD has an IP address configured (192.168.78.2, a private
  LAN) so that it can discuss with all domU.
- there's a domU that is a firewall/router and it also contains 3 virtual
  cards (eth0, eth1, eth2). It does masquerading for traffic going out on
  eth0 (the external interface which is part of xenbrE).
  # grep vif /etc/xen/xm.slis 
  vif = [ 'mac=00:16:3e:14:85:11, bridge=xenbrE', 'mac=00:16:3e:14:85:12, bridge=xenbrI', 'mac=00:16:3e:14:85:13, bridge=xenbrD' ]

My problem is that when I download a big file from the internet by HTTP in
the dom0, the download rate is not stable. It goes up progressively and then
stalls for a few seconds, and restart again going up progressively (and
all this in loop until the download is complete). During the stalls, it looks
like all networking is blocked on the machine (noticed on interactive SSH
sessions).

    dom0                             │domU
         wget                        │
           ↕                         │
    eth2↔xenbrD(192.168.78.2)↔vif2.2←┼→eth2(192.168.78.1/24)
                                     │   ↕ masquerading
    eth0↔xenbrE↔vif2.0←——————————————┼→eth0(192.168.1.20/24)
     ↕
    internet

If I do the same download but uses a (non-caching) HTTP proxy that runs in
the firewall domU, the download rate is stable at its maximum value.

How can I avoid this problem?

I suspect it's a bug in the networking stack but I would like assistance
to diagnose it more precisely (and maybe find a work-around).

This is a Debian Etch system with Xen 3.2 and the 2.6.26-xen-686 kernel of
Debian Lenny (backports). The bridges are created with
/etc/network/interfaces:

    auto lo
    iface lo inet loopback

    auto xenbrE
    iface xenbrE inet manual
            bridge_ports eth0
            bridge_maxwait 0

    auto xenbrI
    iface xenbrI inet manual
            bridge_ports eth1
            bridge_maxwait 0

    auto xenbrD
    iface xenbrD inet static
            address 192.168.78.2
            netmask 255.255.255.0
            gateway 192.168.78.1
            bridge_ports eth2
            bridge_maxwait 0

The xend configuration is not complicated:

    # grep '^(' /etc/xen/xend-config.sxp 
    (network-script network-dummy)
    (vif-script vif-bridge)
    (dom0-min-mem 150)
    (dom0-cpus 0)
    (vncpasswd '')

And the only routing in dom0 redirects to the domU via xenbrD:

    # route -n
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    192.168.78.0    0.0.0.0         255.255.255.0   U     0      0        0 xenbrD
    0.0.0.0         192.168.78.1    0.0.0.0         UG    0      0        0 xenbrD

In the domU, the only iptables configuration done is `iptables -t nat -A
POSTROUTING -s 192.168.78.0/24 -o eth0 -j MASQUERADE`.

Thank you for your help.
-- 
Raphaël Hertzog -+- http://www.ouaza.com

Freexian : des développeurs Debian au service des entreprises
http://www.freexian.com

Re: Constantly varying download rate with a complex xen networking setup, why?

[Ilpo Järvinen]

On Mon, 15 Jun 2009, Raphael Hertzog wrote:

> My problem is that when I download a big file from the internet by HTTP in
> the dom0, the download rate is not stable. It goes up progressively and then
> stalls for a few seconds, and restart again going up progressively (and
> all this in loop until the download is complete).

My guess: TCP sawtooth pattern and the associated loss recovery...? What 
you describe here seems to match exactly to what is expected to happen 
because of it (I see this happening quite often actually). What is the 
size of the bottleneck buffer and base RTTs of the relevant flows (HTTP, 
ssh)?

> During the stalls, it looks like all networking is blocked on the 
> machine (noticed on interactive SSH sessions).

They too might be recovering from losses if the bottleneck is shared? 
Please verify "blocking" using tcpdump or so. You may find out e.g. that 
network is not stalled but that out-of-order arrivals prevent 
applications from making progress.

If this matches with your problem, deploying AQM at the bottleneck would 
help.


-- 
 i.

Re: Constantly varying download rate with a complex xen networking setup, why?

[Raphael Hertzog]

Le lundi 15 juin 2009, Ilpo Järvinen a écrit :
> On Mon, 15 Jun 2009, Raphael Hertzog wrote:
> 
> > My problem is that when I download a big file from the internet by HTTP in
> > the dom0, the download rate is not stable. It goes up progressively and then
> > stalls for a few seconds, and restart again going up progressively (and
> > all this in loop until the download is complete).
> 
> My guess: TCP sawtooth pattern and the associated loss recovery...? What 
> you describe here seems to match exactly to what is expected to happen 
> because of it (I see this happening quite often actually).

Why would this not happen for the same transfer done through a proxy (that
runs on the same machine but different domU)?

> What is the size of the bottleneck buffer and base RTTs of the relevant
> flows (HTTP, ssh)?

How can I collect those information?

> Please verify "blocking" using tcpdump or so. You may find out e.g. that 

I'll try to do so and follow-up.

> network is not stalled but that out-of-order arrivals prevent 
> applications from making progress.
> 
> If this matches with your problem, deploying AQM at the bottleneck would 
> help.

The bottleneck in my test setup is the ADSL connection and I don't control
the ADSL router.

Cheers,
-- 
Raphaël Hertzog -+- http://www.ouaza.com

Freexian : des développeurs Debian au service des entreprises
http://www.freexian.com

Re: Constantly varying download rate with a complex xen networking setup, why?

[Ilpo Järvinen]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2284 bytes --]

On Mon, 15 Jun 2009, Raphael Hertzog wrote:

> Le lundi 15 juin 2009, Ilpo Järvinen a écrit :
> > On Mon, 15 Jun 2009, Raphael Hertzog wrote:
> > 
> > > My problem is that when I download a big file from the internet by HTTP in
> > > the dom0, the download rate is not stable. It goes up progressively and then
> > > stalls for a few seconds, and restart again going up progressively (and
> > > all this in loop until the download is complete).
> > 
> > My guess: TCP sawtooth pattern and the associated loss recovery...? What 
> > you describe here seems to match exactly to what is expected to happen 
> > because of it (I see this happening quite often actually).
> 
> Why would this not happen for the same transfer done through a proxy (that
> runs on the same machine but different domU)?

Maybe the proxy interferes there somehow... I don't know enough about the
details to say but I suppose the proxy at least breaks your tcp connection 
to two parts.

It could well be something else. I was just saying that this 
increasing+stall+repeat pattern itself is quite typical also during normal 
operation (with short rtts one can't see that though due to humanly 
limitations in perception :-) and because of the granularity the 
reporting).

> > What is the size of the bottleneck buffer and base RTTs of the relevant
> > flows (HTTP, ssh)?
> 
> How can I collect those information?

Perhaps it's enough to just know if sshs that stall go over the adsl link 
too or are they local? If they're local it's sort of pointing to something 
else (of which I probably have no clue).

> > Please verify "blocking" using tcpdump or so. You may find out e.g. that 
> 
> I'll try to do so and follow-up.
> 
> > network is not stalled but that out-of-order arrivals prevent 
> > applications from making progress.
> > 
> > If this matches with your problem, deploying AQM at the bottleneck would 
> > help.
> 
> The bottleneck in my test setup is the ADSL connection and I don't control
> the ADSL router.

Right, those non-controllable things need a bit more sophisticated 
control. Lartc has some pre-made script which basically puts an articifial 
cap on the receiver side slightly below the downlinkg rate to create 
bottleneck there which in turn allows controlling the queue.


-- 
 i.

Re: Constantly varying download rate with a complex xen networking setup, why?

[Raphael Hertzog]

Le lundi 15 juin 2009, Ilpo Järvinen a écrit :
> Maybe the proxy interferes there somehow... I don't know enough about the
> details to say but I suppose the proxy at least breaks your tcp connection 
> to two parts.

Indeed. Is there some processing done in a simple linux bridge where the
reapperance of the same TCP packet that has been created and sent on another
local interface could create problem?

> > How can I collect those information?
> 
> Perhaps it's enough to just know if sshs that stall go over the adsl link 
> too or are they local? If they're local it's sort of pointing to something 
> else (of which I probably have no clue).

The ssh connection was local and did not go over the ADSL link.

Cheers,
-- 
Raphaël Hertzog -+- http://www.ouaza.com

Freexian : des développeurs Debian au service des entreprises
http://www.freexian.com

Re: Constantly varying download rate with a complex xen networking setup, why?

[Ilpo Järvinen]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 672 bytes --]

On Wed, 17 Jun 2009, Raphael Hertzog wrote:

> Le lundi 15 juin 2009, Ilpo Järvinen a écrit :
> > Maybe the proxy interferes there somehow... I don't know enough about the
> > details to say but I suppose the proxy at least breaks your tcp connection 
> > to two parts.
> 
> Indeed. Is there some processing done in a simple linux bridge where the
> reapperance of the same TCP packet that has been created and sent on another
> local interface could create problem?

I thought out had a http proxy in between? I suppose it is certainly doing 
more than bridging. Anyway, I'll be week away, so no quick responses are 
to be expected from my part after this mail.


-- 
 i.

Re: Constantly varying download rate with a complex xen networking setup, why?

[Raphael Hertzog]

Le mercredi 17 juin 2009, Ilpo Järvinen a écrit :
> On Wed, 17 Jun 2009, Raphael Hertzog wrote:
> 
> > Le lundi 15 juin 2009, Ilpo Järvinen a écrit :
> > > Maybe the proxy interferes there somehow... I don't know enough about the
> > > details to say but I suppose the proxy at least breaks your tcp connection 
> > > to two parts.
> > 
> > Indeed. Is there some processing done in a simple linux bridge where the
> > reapperance of the same TCP packet that has been created and sent on another
> > local interface could create problem?
> 
> I thought out had a http proxy in between? I suppose it is certainly doing 
> more than bridging. Anyway, I'll be week away, so no quick responses are 
> to be expected from my part after this mail.

Well, I have the problem when I don't use the proxy... if I use it, the
download is split over two TCP connections and things are fine.

Hence my question was if something could be confused by the fact that the
same packet is seen twice on the same machine once (in output) via eth2/xenbrD
and once (in forward) via xenbrE (the routing between both bridges is done
by the domU independently of the dom0 network config).

Cheers,
-- 
Raphaël Hertzog -+- http://www.ouaza.com

Freexian : des développeurs Debian au service des entreprises
http://www.freexian.com

Re: Constantly varying download rate with a complex xen networking setup, why?

[Ilpo Järvinen]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1390 bytes --]

On Wed, 17 Jun 2009, Raphael Hertzog wrote:

> Le mercredi 17 juin 2009, Ilpo Järvinen a écrit :
>> On Wed, 17 Jun 2009, Raphael Hertzog wrote:
>>
>>> Le lundi 15 juin 2009, Ilpo Järvinen a écrit :
>>>> Maybe the proxy interferes there somehow... I don't know enough about the
>>>> details to say but I suppose the proxy at least breaks your tcp connection
>>>> to two parts.
>>>
>>> Indeed. Is there some processing done in a simple linux bridge where the
>>> reapperance of the same TCP packet that has been created and sent on another
>>> local interface could create problem?
>>
>> I thought out had a http proxy in between? I suppose it is certainly doing
>> more than bridging. Anyway, I'll be week away, so no quick responses are
>> to be expected from my part after this mail.
>
> Well, I have the problem when I don't use the proxy... if I use it, the
> download is split over two TCP connections and things are fine.
>
> Hence my question was if something could be confused by the fact that the
> same packet is seen twice on the same machine once (in output) via 
> eth2/xenbrD and once (in forward) via xenbrE (the routing between both 
> bridges is done by the domU independently of the dom0 network config).

Did you ever get tcpdumps btw? Looking into your setup it would probably 
be useful take it from all the interfaces where the packets are supposed 
to pass.


-- 
  i.

Re: Constantly varying download rate with a complex xen networking setup, why?

[Bill Fink]

On Tue, 21 Jul 2009, Ilpo Järvinen wrote:

> On Wed, 17 Jun 2009, Raphael Hertzog wrote:
> 
> > Le mercredi 17 juin 2009, Ilpo Järvinen a écrit :
> >> On Wed, 17 Jun 2009, Raphael Hertzog wrote:
> >>
> >>> Le lundi 15 juin 2009, Ilpo Järvinen a écrit :
> >>>> Maybe the proxy interferes there somehow... I don't know enough about the
> >>>> details to say but I suppose the proxy at least breaks your tcp connection
> >>>> to two parts.
> >>>
> >>> Indeed. Is there some processing done in a simple linux bridge where the
> >>> reapperance of the same TCP packet that has been created and sent on another
> >>> local interface could create problem?
> >>
> >> I thought out had a http proxy in between? I suppose it is certainly doing
> >> more than bridging. Anyway, I'll be week away, so no quick responses are
> >> to be expected from my part after this mail.
> >
> > Well, I have the problem when I don't use the proxy... if I use it, the
> > download is split over two TCP connections and things are fine.
> >
> > Hence my question was if something could be confused by the fact that the
> > same packet is seen twice on the same machine once (in output) via 
> > eth2/xenbrD and once (in forward) via xenbrE (the routing between both 
> > bridges is done by the domU independently of the dom0 network config).
> 
> Did you ever get tcpdumps btw? Looking into your setup it would probably 
> be useful take it from all the interfaces where the packets are supposed 
> to pass.

I don't know if the following has any applicability to your specific
situation, but I thought I'd mention it.

It has to do with what I call the non-transitivity of TCP network
performance.  If you have something like the following scenario:

	A <----LAN----> B <----WAN----> C
	     low RTT        high RTT

Just because you have good TCP network performance from A to B, and
good TCP network performance from B to C, it does not necessarily
follow that you will also have good TCP network performance from A to C.

Consider the case where the B<->C path is clean, but there are problems
on the A<->B path (causing TCP retransmissions).  The low RTT on the
A<->B path can mask the problems (basically TCP performs well in spite
of the problems because of the low RTT), but a transfer directly from
A to C performs poorly because the high RTT results in a very slow
ramp up in performance after a loss event.

Again, I have no idea if this applies in any way to your case.

						-Bill

Constantly varying download rate with a complex xen networking setup, why?

[Raphael Hertzog]

[ Please CC me ]

Hello,

I have a weird problem and I'm not sure if it comes from linux or from
Xen. Thus I'd like your opinion and maybe your help so that I can
do a proper bugreport where it belongs. I can do supplementary tests on
request of course.

(Note the full description below has also been submitted to
http://serverfault.com/questions/22219/)

The network configuration in my Xen setup:
- the dom0 has 3 network cards (eth0, eth1, eth2), 3 brigdes (xenbrE,
  xenbrI, xenbrD) and each brigde integrates the corresponding network
  card. Only xenbrD has an IP address configured (192.168.78.2, a private
  LAN) so that it can discuss with all domU.
- there's a domU that is a firewall/router and it also contains 3 virtual
  cards (eth0, eth1, eth2). It does masquerading for traffic going out on
  eth0 (the external interface which is part of xenbrE).
  # grep vif /etc/xen/xm.slis 
  vif = [ 'mac=00:16:3e:14:85:11, bridge=xenbrE', 'mac=00:16:3e:14:85:12, bridge=xenbrI', 'mac=00:16:3e:14:85:13, bridge=xenbrD' ]

My problem is that when I download a big file from the internet by HTTP in
the dom0, the download rate is not stable. It goes up progressively and then
stalls for a few seconds, and restart again going up progressively (and
all this in loop until the download is complete). During the stalls, it looks
all networking is blocked on the machine (noticed on interactive SSH sessions).

    dom0                             │domU
         wget                        │
           ↕                         │
    eth2↔xenbrD(192.168.78.2)↔vif2.2←┼→eth2(192.168.78.1/24)
                                     │   ↕ masquerading
    eth0↔xenbrE↔vif2.0←——————————————┼→eth0(192.168.1.20/24)
     ↕
    internet

If I do the same download but uses a (non-caching) HTTP proxy that runs in
the firewall domU, the download rate is stable at its maximum value.

How can I avoid this problem?

I suspect it's a bug in the networking stack but I would like assistance
to diagnose it more precisely (and maybe find a work-around).

This is a Debian Etch system with Xen 3.2 and the 2.6.26-xen-686 kernel of
Debian Lenny (backports). The bridges are created with
/etc/network/interfaces:

    auto lo
    iface lo inet loopback

    auto xenbrE
    iface xenbrE inet manual
            bridge_ports eth0
            bridge_maxwait 0

    auto xenbrI
    iface xenbrI inet manual
            bridge_ports eth1
            bridge_maxwait 0

    auto xenbrD
    iface xenbrD inet static
            address 192.168.78.2
            netmask 255.255.255.0
            gateway 192.168.78.1
            bridge_ports eth2
            bridge_maxwait 0

The xend configuration is not complicated:

    # grep '^(' /etc/xen/xend-config.sxp 
    (network-script network-dummy)
    (vif-script vif-bridge)
    (dom0-min-mem 150)
    (dom0-cpus 0)
    (vncpasswd '')

And the only routing in dom0 redirects to the domU via xenbrD:

    # route -n
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    192.168.78.0    0.0.0.0         255.255.255.0   U     0      0        0 xenbrD
    0.0.0.0         192.168.78.1    0.0.0.0         UG    0      0        0 xenbrD

In the domU, the only iptables configuration done is `iptables -t nat -A
POSTROUTING -s 192.168.78.0/24 -o eth0 -j MASQUERADE`.

Thank you for your help.
-- 
Raphaël Hertzog -+- http://www.ouaza.com

Freexian : des développeurs Debian au service des entreprises
http://www.freexian.com

Constantly varying download rate with a complex xen networking setup, why?

[Raphael Hertzog]

[ Please CC me I'm not subscribed ]

Hello,

I have a weird problem and I'm not sure if it comes from linux or from
Xen. Thus I'd like your opinion and maybe your help so that I can
do a proper bugreport where it belongs.

(Note the full description below has also been submitted to
http://serverfault.com/questions/22219/)

The network configuration in my Xen setup:
- the dom0 has 3 network cards (eth0, eth1, eth2), 3 brigdes (xenbrE,
  xenbrI, xenbrD) and each brigde integrates the corresponding network
  card. Only xenbrD has an IP address configured (192.168.78.2, a private
  LAN) so that it can discuss with all domU.
- there's a domU that is a firewall/router and it also contains 3 virtual
  cards (eth0, eth1, eth2). It does masquerading for traffic going out on
  eth0 (the external interface which is part of xenbrE).
  # grep vif /etc/xen/xm.slis 
  vif = [ 'mac=00:16:3e:14:85:11, bridge=xenbrE', 'mac=00:16:3e:14:85:12, bridge=xenbrI', 'mac=00:16:3e:14:85:13, bridge=xenbrD' ]

My problem is that when I download a big file from the internet by HTTP in
the dom0, the download rate is not stable. It goes up progressively and then
stalls for a few seconds, and restart again going up progressively (and
all this in loop until the download is complete). During the stalls, it looks
all networking is blocked on the machine (noticed on interactive SSH sessions).

    dom0                             │domU
         wget                        │
           ↕                         │
    eth2↔xenbrD(192.168.78.2)↔vif2.2←┼→eth2(192.168.78.1/24)
                                     │   ↕ masquerading
    eth0↔xenbrE↔vif2.0←——————————————┼→eth0(192.168.1.20/24)
     ↕
    internet

If I do the same download but uses a (non-caching) HTTP proxy that runs in
the firewall domU, the download rate is stable at its maximum value.

How can I avoid this problem?

I suspect it's a bug in the networking stack but I would like assistance
to diagnose it more precisely (and maybe find a work-around).

This is a Debian Etch system with Xen 3.2 and the 2.6.26-xen-686 kernel of
Debian Lenny (backports). The bridges are created with
/etc/network/interfaces:

    auto lo
    iface lo inet loopback

    auto xenbrE
    iface xenbrE inet manual
            bridge_ports eth0
            bridge_maxwait 0

    auto xenbrI
    iface xenbrI inet manual
            bridge_ports eth1
            bridge_maxwait 0

    auto xenbrD
    iface xenbrD inet static
            address 192.168.78.2
            netmask 255.255.255.0
            gateway 192.168.78.1
            bridge_ports eth2
            bridge_maxwait 0

The xend configuration is not complicated:

    # grep '^(' /etc/xen/xend-config.sxp 
    (network-script network-dummy)
    (vif-script vif-bridge)
    (dom0-min-mem 150)
    (dom0-cpus 0)
    (vncpasswd '')

And the only routing in dom0 redirects to the domU via xenbrD:

    # route -n
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    192.168.78.0    0.0.0.0         255.255.255.0   U     0      0        0 xenbrD
    0.0.0.0         192.168.78.1    0.0.0.0         UG    0      0        0 xenbrD

In the domU, the only iptables configuration done is `iptables -t nat -A
POSTROUTING -s 192.168.78.0/24 -o eth0 -j MASQUERADE`.

Thank you for your help.
-- 
Raphaël Hertzog -+- http://www.ouaza.com

Freexian : des développeurs Debian au service des entreprises
http://www.freexian.com