16-CPU #s for lockfree rtcache (rt_rcu)

16-CPU #s for lockfree rtcache (rt_rcu)

[Dipankar Sarma]

As promised, here is the kernprof data from the simulated
test suggested by Dave. The test uses 32 processes and dest
addresses for rtcache lookup are random, changing after every 5 packets.
The measurements were done in a 16 CPU NUMA-Q.

2.5.3 : ip_route_output_key [c01bab8c]: 12166
2.5.3+rt_rcu : ip_route_output_key [c01bb084]: 6027

I have seen moderately significant profile counts
for ip_route_input() in preliminary webserver benchmark runs.
It is not however clear to me that bucket lock cache line
bouncing is the reason behind it. That one needs more investigation.

Thanks
-- 
Dipankar Sarma  <dipankar@in.ibm.com> http://lse.sourceforge.net
Linux Technology Center, IBM Software Lab, Bangalore, India.

Re: 16-CPU #s for lockfree rtcache (rt_rcu)

[David S. Miller]

   From: Dipankar Sarma <dipankar@in.ibm.com>
   Date: Fri, 17 May 2002 19:21:16 +0530
   
   2.5.3 : ip_route_output_key [c01bab8c]: 12166
   2.5.3+rt_rcu : ip_route_output_key [c01bb084]: 6027
   
Thanks for doing the testing.  Are you able to do this
test on some 4 or 8 processor non-NUMA system?

Basically halfing the profile hits for this function
is wonderful and I'd love to see how much of this translates to a
non-NUMA system.

   I have seen moderately significant profile counts
   for ip_route_input() in preliminary webserver benchmark runs.
   It is not however clear to me that bucket lock cache line
   bouncing is the reason behind it. That one needs more investigation.

This is where most of the routing heavy work is done on
a web server, so this doesn't surprise me.  Once packet
is input and routed, we have destination and just grab a reference to
and use it for output back to that remote host.

Re: 16-CPU #s for lockfree rtcache (rt_rcu)

[Dipankar Sarma]

On Fri, May 17, 2002 at 06:49:21AM -0700, David S. Miller wrote:
>    From: Dipankar Sarma <dipankar@in.ibm.com>
>    Date: Fri, 17 May 2002 19:21:16 +0530
>    
>    2.5.3 : ip_route_output_key [c01bab8c]: 12166
>    2.5.3+rt_rcu : ip_route_output_key [c01bb084]: 6027
>    
> Thanks for doing the testing.  Are you able to do this
> test on some 4 or 8 processor non-NUMA system?

Yes, but may not have been the same test. We have been doing various 
configurations for this test. One fallout of using large number of 
dest addresses is that we have frequent neighbor table garbage collection 
which results in a lot of lock contentions. By slowing down
the packet rate and/or increasing the gc threshold, we can
avoid this. How realistic is this ? If we avoid frequent
gc, we see better gains in route lookup. With frequent gc,
the speedup was of about 22% for an 8 cpu SMP, IIRC. I will rerun
the tests tomorrow or monday to get both sets of numbers for 8-cpu
SMP.

Thanks
-- 
Dipankar Sarma  <dipankar@in.ibm.com> http://lse.sourceforge.net
Linux Technology Center, IBM Software Lab, Bangalore, India.

Re: 16-CPU #s for lockfree rtcache (rt_rcu)

[David S. Miller]

   From: Dipankar Sarma <dipankar@in.ibm.com>
   Date: Fri, 17 May 2002 21:44:33 +0530
   
   I will rerun the tests tomorrow or monday to get both sets of
   numbers for 8-cpu  SMP.

Provide the data, it will be interesting.

Re: 16-CPU #s for lockfree rtcache (rt_rcu)

[Andi Kleen]

"David S. Miller" <davem@redhat.com> writes:

>    From: Dipankar Sarma <dipankar@in.ibm.com>
>    Date: Fri, 17 May 2002 21:44:33 +0530
>    
>    I will rerun the tests tomorrow or monday to get both sets of
>    numbers for 8-cpu  SMP.
> 
> Provide the data, it will be interesting.

I bet the numbers would be much better if the x86 
do_gettimeofday() was converted to a lockless version first ...
Currently it is bouncing around its readlock for every incoming packet.

-Andi

Re: 16-CPU #s for lockfree rtcache (rt_rcu)

[David S. Miller]

   From: Andi Kleen <ak@muc.de>
   Date: 17 May 2002 21:25:16 +0200

   "David S. Miller" <davem@redhat.com> writes:
   
   > Provide the data, it will be interesting.
   
   I bet the numbers would be much better if the x86 
   do_gettimeofday() was converted to a lockless version first ...
   Currently it is bouncing around its readlock for every incoming packet.

That is true.  But right now we are trying to analyze the effects of
his patch all by itself.

Re: 16-CPU #s for lockfree rtcache (rt_rcu)

[Dipankar Sarma]

On Fri, May 17, 2002 at 12:25:19PM -0700, David S. Miller wrote:
>    From: Andi Kleen <ak@muc.de>
>    Date: 17 May 2002 21:25:16 +0200
> 
>    "David S. Miller" <davem@redhat.com> writes:
>    
>    > Provide the data, it will be interesting.
>    
>    I bet the numbers would be much better if the x86 
>    do_gettimeofday() was converted to a lockless version first ...
>    Currently it is bouncing around its readlock for every incoming packet.
> 
> That is true.  But right now we are trying to analyze the effects of
> his patch all by itself.

Yes, that is a another problem needs addressing.

BTW, do_gettimeofday() also shows up moderately significant in profile
of 8-CPU webserver benchmark. I will address xtime_lock separately.

Thanks
-- 
Dipankar Sarma  <dipankar@in.ibm.com> http://lse.sourceforge.net
Linux Technology Center, IBM Software Lab, Bangalore, India.