Re: Linux Route Cache performance tests

[Eric Dumazet <eric.dumazet@gmail.com>]

Le dimanche 06 novembre 2011 à 20:20 +0100, Paweł Staszewski a écrit :
> W dniu 2011-11-06 19:48, Eric Dumazet pisze:
> > Le dimanche 06 novembre 2011 à 19:28 +0100, Paweł Staszewski a écrit :
> >> W dniu 2011-11-06 18:29, Eric Dumazet pisze:
> >>> Le dimanche 06 novembre 2011 à 16:57 +0100, Paweł Staszewski a écrit :
> >>>> Hello
> >>>>
> >>>>
> >>>>
> >>>> I make some networking performance tests for Linux 3.1
> >>>>
> >>>> Configuration:
> >>>>
> >>>> Linux (pktget) ---->   Linux (router) ---->   Linux (Sink)
> >>>>
> >>>> pktgen config:
> >>>> clone_skb 32
> >>>> pkt_size 64
> >>>> delay 0
> >>>>
> >>>> pgset "flag IPDST_RND"
> >>>> pgset "dst_min 10.0.0.0"
> >>>> pgset "dst_max 10.18.255.255"
> >>>> pgset "config 1"
> >>>> pgset "flows 256"
> >>>> pgset "flowlen 8"
> >>>>
> >>>> TX performance for this host:
> >>>> eth0:            RX: 0.00 P/s      TX: 12346107.73 P/s      TOTAL:
> >>>> 12346107.73 P/s
> >>>>
> >>>> On Linux (router):
> >>>> grep . /proc/sys/net/ipv4/route/*
> >>>> /proc/sys/net/ipv4/route/error_burst:500
> >>>> /proc/sys/net/ipv4/route/error_cost:100
> >>>> grep: /proc/sys/net/ipv4/route/flush: Permission denied
> >>>> /proc/sys/net/ipv4/route/gc_elasticity:4
> >>>> /proc/sys/net/ipv4/route/gc_interval:60
> >>>> /proc/sys/net/ipv4/route/gc_min_interval:0
> >>>> /proc/sys/net/ipv4/route/gc_min_interval_ms:500
> >>>> /proc/sys/net/ipv4/route/gc_thresh:2000000
> >>>> /proc/sys/net/ipv4/route/gc_timeout:60
> >>>> /proc/sys/net/ipv4/route/max_size:8388608
> >>>> /proc/sys/net/ipv4/route/min_adv_mss:256
> >>>> /proc/sys/net/ipv4/route/min_pmtu:552
> >>>> /proc/sys/net/ipv4/route/mtu_expires:600
> >>>> /proc/sys/net/ipv4/route/redirect_load:2
> >>>> /proc/sys/net/ipv4/route/redirect_number:9
> >>>> /proc/sys/net/ipv4/route/redirect_silence:2048
> >>>>
> >>>> For the first 30secs maybee more router is forwarding ~5Mpps to the
> >>>> Linux (Sink)
> >>>> and some stats for this forst 30secs in attached image:
> >>>>
> >>>> http://imageshack.us/photo/my-images/684/test1ih.png/
> >>>>
> >>>> Left up - pktgen linux
> >>>> left down - Linux router (htop)
> >>>> Right up - Linux router (bwm-ng - showing pps)
> >>>> Right down - Linux router (lnstat)
> >>>>
> >>>>
> >>>> And all is good - performance 5Mpps until Linux router will reach ~1kk
> >>>> entries
> >>>> What You can see on next attached image:
> >>>>
> >>>> http://imageshack.us/photo/my-images/24/test2id.png/
> >>>>
> >>>> Forwarding performance drops from 5Mpps to 1,8Mpps
> >>>> And after 3 - 4 minutes it will stop on 0,7Mpps
> >>>>
> >>>>
> >>>> After flushing the route cache performance increase from 0.7Mpps to 6Mpps
> >>>> What You can see on next attached image:
> >>>>
> >>>> http://imageshack.us/photo/my-images/197/test3r.png/
> >>>>
> >>>> Is it possible to turn off route cache ? and see what performance will
> >>>> be without caching
> >>>>
> >>> Route cache cannot handle DDOS situation, since it will be filled,
> >>> unless you have a lot of memory.
> >> hmm
> >> but what is DDOS situation for route cache ? new entries per sec ? total
> >> amount of entries 1,2kk in my tests ?
> >> Look sometimes in normal scenario You can hit
> >> 1245072 route cache entries
> >> This is normal for BGP configurations.
> >>
> > Then figure out the right tunables for your machine ?
> >
> > Its not a laptop or average server setup, so you need to allow your
> > kernel to consume a fair amount of memory for the route cache.
> Yes this parameters  was special not tuned :)
> To see what is the route cache performance limit
> 

Hmm, I thought you were asking for help on netdev ?

> Because there was no optimal parameters for this test :)
> no matter what i tuned results are always the same
> performance drops from 5Mpps to 0.7Mpps without tuning sysctl
> 
> And with tuned parameters i can reach the same as turning off route 
> cache - when running this tests.
> So Yes Tuned performance is better
> performance drops from 5Mpps to 0.7Mpps - without tuning
> and from 5Mpps to 3,7Mpps with tuned sysctl - so a little less than with 
> turned off route cache
> 
> So the point of this test was figure out how much of route cache entries 
> Linux can handle without dropping performance.

No need to even do a bench, its pretty easy to understand how a hash
table is handled.

Allowing long chains is not good.

With your 512k slots hash table, you cannot expect handling 1.4M routes
with optimal performance. End of story.

Since route hash table is allocated at boot time, only way to change its
size is using "rhash_entries=2097152" boot parameter.

If it still doesnt fly, try with "rhash_entries=4194304"