Dropping Packets in 2.6.17

Dropping Packets in 2.6.17

[Danial Thom]

I'm trying to make a case for using linux as a
network appliance, but I can't find any
combination of settings that will keep it from
dropping packets at an unacceptably high rate.
The test system is a 1.8Ghz Opteron with intel
gigE cards running 2.6.17. I'm passing about 70K
pps through the box, which is a light load, but
userland activities (such as building a kernel)
cause it to lose packets, even with backlog set
to 20000. I had the same problem with 2.6.12 and
abandoned the effort. Has anything been done
since to give priority to networking? You can't
have a network appliance drop packets when some
application is gathering stats or a user is
looking at a graph. What tunings are available?

DT


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Dropping Packets in 2.6.17

[Arjan van de Ven]

On Thu, 2006-06-22 at 04:31 -0700, Danial Thom wrote:
> I'm trying to make a case for using linux as a
> network appliance, but I can't find any
> combination of settings that will keep it from
> dropping packets at an unacceptably high rate.
> The test system is a 1.8Ghz Opteron with intel
> gigE cards running 2.6.17. I'm passing about 70K
> pps through the box, which is a light load, but
> userland activities (such as building a kernel)
> cause it to lose packets, even with backlog set
> to 20000. I had the same problem with 2.6.12 and
> abandoned the effort. Has anything been done
> since to give priority to networking? You can't
> have a network appliance drop packets when some
> application is gathering stats or a user is
> looking at a graph. What tunings are available?

Hi Danial,

the most likely tunable that will help you is

/proc/sys/vm/min_free_kbytes

For the router kind of device that one usually needs bumping a bit;
without the bumping the VM doesn't see enough "normal" activity to tune
it's emergency/interrupt handling buffers (and most networking
allocations happen in interrupt context), and then ends up failing
allocations in interrupt context, which leads to dropped packets.

Greetings,
   Arjan van de Ven

Re: Dropping Packets in 2.6.17

[Danial Thom]

--- Arjan van de Ven <arjan@infradead.org> wrote:

> On Thu, 2006-06-22 at 04:31 -0700, Danial Thom
> wrote:
> > I'm trying to make a case for using linux as
> a
> > network appliance, but I can't find any
> > combination of settings that will keep it
> from
> > dropping packets at an unacceptably high
> rate.
> > The test system is a 1.8Ghz Opteron with
> intel
> > gigE cards running 2.6.17. I'm passing about
> 70K
> > pps through the box, which is a light load,
> but
> > userland activities (such as building a
> kernel)
> > cause it to lose packets, even with backlog
> set
> > to 20000. I had the same problem with 2.6.12
> and
> > abandoned the effort. Has anything been done
> > since to give priority to networking? You
> can't
> > have a network appliance drop packets when
> some
> > application is gathering stats or a user is
> > looking at a graph. What tunings are
> available?
> 
> Hi Danial,
> 
> the most likely tunable that will help you is
> 
> /proc/sys/vm/min_free_kbytes
> 
> For the router kind of device that one usually
> needs bumping a bit;
> without the bumping the VM doesn't see enough
> "normal" activity to tune
> it's emergency/interrupt handling buffers (and
> most networking
> allocations happen in interrupt context), and
> then ends up failing
> allocations in interrupt context, which leads
> to dropped packets.

I don't think thats the problem, as I've tracked
the problem to packets being dropped because of
excessive backlog (ie, they are being dropped
gracefully). However with a backlog of 20000, and
a traffic level of about 75,000pps, that means
almost 1/3 of a second that the system doesn't
process packets, which is just unacceptable.

I'll try changing the setting, but running out of
memory doesn't seem to be the issue. I think what
I need is some mechanism to make interrupts a
priority, like it was back in the days when
networking was more important then mp3 playback.

Danial

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Dropping Packets in 2.6.17

[Pádraig Brady]

Danial Thom wrote:
> I'm trying to make a case for using linux as a
> network appliance, but I can't find any
> combination of settings that will keep it from
> dropping packets at an unacceptably high rate.
> The test system is a 1.8Ghz Opteron with intel
> gigE cards running 2.6.17. I'm passing about 70K
> pps through the box, which is a light load, but
> userland activities (such as building a kernel)
> cause it to lose packets, even with backlog set
> to 20000. I had the same problem with 2.6.12 and
> abandoned the effort. Has anything been done
> since to give priority to networking? You can't
> have a network appliance drop packets when some
> application is gathering stats or a user is
> looking at a graph. What tunings are available?

For reference with 2.4.20 on a dual 3.4GHz xeon
and 2 x e1000 cards, I was able to capture, classify
and do sophisticated statistical calculations on
625Kpps per interface (1.3 million packets per second).
The bottleneck at this point was memory bandwidth.
Allowing some drops the average rate went up to the
PCI bottleneck of about 850kpps/port.
Classification and Computation was done in userspace.

Note there is a max interrupt rate of around 80K/s
on x86 at least (not sure about opteron), so make
sure you're using NAPI. /proc/interrupts will
show your interrupt rate.

If the packets go to userspace, make sure you're using
CONFIG_PACKET_MMAP

Pádraig.

Re: Dropping Packets in 2.6.17

[Miquel van Smoorenburg]

In article <449A9ADC.9070800@draigBrady.com>,
Pádraig Brady  <P@draigBrady.com> wrote:
>Note there is a max interrupt rate of around 80K/s
>on x86 at least (not sure about opteron), so make
>sure you're using NAPI. /proc/interrupts will
>show your interrupt rate.

The e1000 driver has some more knobs you can turn. I have this
in my /etc/modules file:

e1000 RxAbsIntDelay=256,256 TxAbsIntDelay=256,256 TxDescriptors=1024,1024 RxDescriptors=1024,1024

(this is for two cards)

and this in /etc/sysctl.conf:

# Tune back swappiness on 2.6
vm.swappiness = 10

# Lots of kernel memory needed for e1000
vm.min_free_kbytes = 65535

This box accepts ~250 mbit/sec, stores that on disk, and streams it
out again to multiple peers at ~1300 mbit/sec total. Kernel 2.6.14.2,
dual xeon em64t in 64 bits mode, 4 GB memory.

Mike.

Re: Dropping Packets in 2.6.17

[Pádraig Brady]

Miquel van Smoorenburg wrote:
> In article <449A9ADC.9070800@draigBrady.com>,
> Pádraig Brady  <P@draigBrady.com> wrote:
> 
>>Note there is a max interrupt rate of around 80K/s
>>on x86 at least (not sure about opteron), so make
>>sure you're using NAPI. /proc/interrupts will
>>show your interrupt rate.
> 
> 
> The e1000 driver has some more knobs you can turn. I have this
> in my /etc/modules file:
> 
> e1000 RxAbsIntDelay=256,256 TxAbsIntDelay=256,256 TxDescriptors=1024,1024 RxDescriptors=1024,1024

Yep sorry I should have mentioned those.
I set the descriptors up to the max of 4096
(as the statistical calcs only needed 5ms timestamping accuracy).

I got a large gain also by modifying the e1000 driver to do skb recycling

Pádraig.

Re: Dropping Packets in 2.6.17

[Danial Thom]

--- Pádraig Brady <P@draigBrady.com> wrote:

> Danial Thom wrote:
> > I'm trying to make a case for using linux as
> a
> > network appliance, but I can't find any
> > combination of settings that will keep it
> from
> > dropping packets at an unacceptably high
> rate.
> > The test system is a 1.8Ghz Opteron with
> intel
> > gigE cards running 2.6.17. I'm passing about
> 70K
> > pps through the box, which is a light load,
> but
> > userland activities (such as building a
> kernel)
> > cause it to lose packets, even with backlog
> set
> > to 20000. I had the same problem with 2.6.12
> and
> > abandoned the effort. Has anything been done
> > since to give priority to networking? You
> can't
> > have a network appliance drop packets when
> some
> > application is gathering stats or a user is
> > looking at a graph. What tunings are
> available?
> 
> For reference with 2.4.20 on a dual 3.4GHz xeon
> and 2 x e1000 cards, I was able to capture,
> classify
> and do sophisticated statistical calculations
> on
> 625Kpps per interface (1.3 million packets per
> second).
> The bottleneck at this point was memory
> bandwidth.
> Allowing some drops the average rate went up to
> the
> PCI bottleneck of about 850kpps/port.
> Classification and Computation was done in
> userspace.
> 
> Note there is a max interrupt rate of around
> 80K/s
> on x86 at least (not sure about opteron), so
> make
> sure you're using NAPI. /proc/interrupts will
> show your interrupt rate.
> 
> If the packets go to userspace, make sure
> you're using
> CONFIG_PACKET_MMAP

Unfortunately I can do that much with FreeBSD 4.x
with 1 2.0Ghz opteron, so its not a very
compelling case to have to spend twice as much on
hardware to use LINUX. However 2.4 seemed much
better than 2.6 in this regard. 2.6 wants to drop
a lot more packets. The goal of using 2.6 is to
utilize DP better, but it obviously has to
perform better than a UP Freebsd box.

What ITR setting are using for the e1000 driver?

># Lots of kernel memory needed for e1000 
>vm.min_free_kbytes = 65535 

I'm curious as to why a vm setting is useful, as
it doesn't seem that the e1000 driver uses
virtual memory? Since rings are replenished with
sk_buffs, and sk_buffs have to be contiguous, how
does vm come into play?

DT


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Dropping Packets in 2.6.17

[Pádraig Brady]

Danial Thom wrote:
> 
> --- Pádraig Brady <P@draigBrady.com> wrote:
> 
>>For reference with 2.4.20 on a dual 3.4GHz xeon
>>and 2 x e1000 cards, I was able to capture,
>>classify and do statistical calculations
>>on 625Kpps per interface (1.3 million pps).
> 
> Unfortunately I can do that much with FreeBSD 4.x
> with 1 2.0Ghz opteron, so its not a very
> compelling case to have to spend twice as much on
> hardware to use LINUX. However 2.4 seemed much
> better than 2.6 in this regard. 2.6 wants to drop
> a lot more packets. The goal of using 2.6 is to
> utilize DP better, but it obviously has to
> perform better than a UP Freebsd box.

NC.

> What ITR setting are using for the e1000 driver?

I didn't use ITR, I used NAPI.

>># Lots of kernel memory needed for e1000 
>>vm.min_free_kbytes = 65535 
> 
> 
> I'm curious as to why a vm setting is useful, as
> it doesn't seem that the e1000 driver uses
> virtual memory? Since rings are replenished with
> sk_buffs, and sk_buffs have to be contiguous, how
> does vm come into play?

Contiguous? The [tr]x descriptors contain
pointers to the skbufs.
Anyway I bypassed the large allocation overhead
by using skb recycling.

Pádraig.

Re: Dropping Packets in 2.6.17

[Danial Thom]

--- Pádraig Brady <P@draigBrady.com> wrote:

> Danial Thom wrote:
> > 
> > --- Pádraig Brady <P@draigBrady.com> wrote:
> > 
> >>For reference with 2.4.20 on a dual 3.4GHz
> xeon
> >>and 2 x e1000 cards, I was able to capture,
> >>classify and do statistical calculations
> >>on 625Kpps per interface (1.3 million pps).
> > 
> > Unfortunately I can do that much with FreeBSD
> 4.x
> > with 1 2.0Ghz opteron, so its not a very
> > compelling case to have to spend twice as
> much on
> > hardware to use LINUX. However 2.4 seemed
> much
> > better than 2.6 in this regard. 2.6 wants to
> drop
> > a lot more packets. The goal of using 2.6 is
> to
> > utilize DP better, but it obviously has to
> > perform better than a UP Freebsd box.
> 
> NC.
> 
> > What ITR setting are using for the e1000
> driver?
> 
> I didn't use ITR, I used NAPI.
> 

If thats the case then your "system" would have
the same problem that I'm encountering, since
polling results in buckets of packets being
dropped with heavy userland activity.

Since you can control exactly how many interrupts
are generated and how often, there should be no
advantage to polling,  unless LINUX interrupt
processing is badly broken, which seems like a
strong possibility.

This should work quite naturally without having
to use kludges like polling, enormous receive
rings or memory cheating. 

DT

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Dropping Packets in 2.6.17

[Danial Thom]

--- Robert Hancock <hancockr@shaw.ca> wrote:

> Danial Thom wrote:
> >>
> >> I didn't use ITR, I used NAPI.
> >>
> > 
> > If thats the case then your "system" would
> have
> > the same problem that I'm encountering, since
> > polling results in buckets of packets being
> > dropped with heavy userland activity.
> 
> This is to some extent by design. If you
> processed all packets purely in 
> interrupt context, at some point you can start
> receiving so many packets 
> that you never leave interrupt context to
> perform any other useful work, 
> no matter if your adapter can avoid generating
> an interrupt for every 
> packet. Packet floods can completely hang the
> machine. Pushing the work 
> into a softirq and disabling NIC interrupts in
> the interim allows the 
> machine to continue performing other useful
> work.

You guys keep talking about generating an int for
every packet, and like no controller designed
after 1998 does it, so why are you still talking
about it?

Yes, this is the "design" thats flawed. Apps like
MySQL and compilers always try to use all of the
cpu, and if it results in the system *thinking*
that its being flooded then its not working
properly. A network stream that is using less
that 20% of the cpu should *never* drop any
packets. 

> If you want to give more priority to processing
> network packets at the 
> expense of user processes then you likely need
> to increase the priority 
> of the ksoftirqd thread(s). These compete for
> CPU time like any other 
> processes.

I've tried this and it doesn't do much in terms
of alleviate the problem. It makes a marginal
difference.

I think much of the problem is that the network
guys are spending so much time on frivilous
things like polling that they have no interest in
getting the hardware to work the way its supposed
to. Ironically, if the harware worked properly
there would be no reason to have polling.

Detecting a livelock condition is easy. There's
no reason to drop packets randomly. Additionally
a workload tunable would deflect the argument
against allowing near-livelock conditions.
Dropping packets should be a last resort; not the
norm.

DT

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Dropping Packets in 2.6.17

[Brian F. G. Bidulock]

Robert,

On Thu, 22 Jun 2006, Robert Hancock wrote:
> 
> If you want to give more priority to processing network packets at the 
> expense of user processes then you likely need to increase the priority 
> of the ksoftirqd thread(s). These compete for CPU time like any other 
> processes.
> 

I don't think that's a fair statement:

 - "any other process" does not execute when returning from an interrupt
    as do softirq threads

 - "any other process" does not execute upon local_bh_enable().

 - "any other process" is blockable (which at softirq is a big BUG()).

Under moderate to heavy load, throttling (or disabling) hard interrupts
effectively reduces the priority of ksoftirqd threads (they have less
opportunity to run because interrupts are returning less often).

Re: Dropping Packets in 2.6.17

[Robert Hancock]

Danial Thom wrote:
>>
>> I didn't use ITR, I used NAPI.
>>
> 
> If thats the case then your "system" would have
> the same problem that I'm encountering, since
> polling results in buckets of packets being
> dropped with heavy userland activity.

This is to some extent by design. If you processed all packets purely in 
interrupt context, at some point you can start receiving so many packets 
that you never leave interrupt context to perform any other useful work, 
no matter if your adapter can avoid generating an interrupt for every 
packet. Packet floods can completely hang the machine. Pushing the work 
into a softirq and disabling NIC interrupts in the interim allows the 
machine to continue performing other useful work.

If you want to give more priority to processing network packets at the 
expense of user processes then you likely need to increase the priority 
of the ksoftirqd thread(s). These compete for CPU time like any other 
processes.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/