Re: Kernel 4.19 network performance - forwarding/routing normal users traffic

[Jesper Dangaard Brouer <brouer@redhat.com>]

On Sat, 3 Nov 2018 01:16:08 +0100
Paweł Staszewski <pstaszewski@itcare.pl> wrote:

> W dniu 02.11.2018 o 20:02, Paweł Staszewski pisze:
> >
> >
> > W dniu 02.11.2018 o 15:20, Aaron Lu pisze:  
> >> On Fri, Nov 02, 2018 at 12:40:37PM +0100, Jesper Dangaard Brouer wrote:  
> >>> On Fri, 2 Nov 2018 13:23:56 +0800
> >>> Aaron Lu <aaron.lu@intel.com> wrote:
> >>>  
> >>>> On Thu, Nov 01, 2018 at 08:23:19PM +0000, Saeed Mahameed wrote:  
> >>>>> On Thu, 2018-11-01 at 23:27 +0800, Aaron Lu wrote:  
> >>>>>> On Thu, Nov 01, 2018 at 10:22:13AM +0100, Jesper Dangaard Brouer
> >>>>>> wrote:
> >>>>>> ... ...  
[...]
> >>> TL;DR: this is order-0 pages (code-walk trough proof below)
> >>>
> >>> To Aaron, the network stack *can* call __free_pages_ok() with order-0
> >>> pages, via:
[...]
> >>  
> >> I think here is a problem - order 0 pages are freed directly to buddy,
> >> bypassing per-cpu-pages. This might be the reason lock contention
> >> appeared on free path. Can someone apply below diff and see if lock
> >> contention is gone?  
> >>
> > Will test it tonight
> >  
> Patch applied
> perf report:
> https://ufile.io/sytfh
> 
> 
> But i need to wait also with more traffic currently cpu's are sleeping
> 

Well, that would be the expected result, that the CPUs get more time to
sleep, if the lock contention is gone...

What is the measured bandwidth now?

Notice, you might still be limited by the PCIe bandwidth, but then your
CPUs might actually decide to sleep, as they are getting data fast
enough.


[...]
> >>
> >> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> >> index e2ef1c17942f..65c0ae13215a 100644
> >> --- a/mm/page_alloc.c
> >> +++ b/mm/page_alloc.c
> >> @@ -4554,8 +4554,14 @@ void page_frag_free(void *addr)
> >>   {
> >>       struct page *page = virt_to_head_page(addr);
> >>   -    if (unlikely(put_page_testzero(page)))
> >> -        __free_pages_ok(page, compound_order(page));
> >> +    if (unlikely(put_page_testzero(page))) {
> >> +        unsigned int order = compound_order(page);
> >> +
> >> +        if (order == 0)
> >> +            free_unref_page(page);
> >> +        else
> >> +            __free_pages_ok(page, order);
> >> +    }



-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer