Re: [RFC PATCH v4] mm/slub: Optimize slub memory usage

[Vlastimil Babka <vbabka@suse.cz>]

On 8/24/23 12:52, Jay Patel wrote:
> On Fri, 2023-08-11 at 17:43 +0200, Vlastimil Babka wrote:
>> On 8/10/23 19:54, Hyeonggon Yoo wrote:
>> > >                         order = calc_slab_order(size,
>> > > min_objects,
>> > >                                         slub_max_order,
>> > > fraction);
>> > > @@ -4159,14 +4164,6 @@ static inline int calculate_order(unsigned
>> > > int size)
>> > >                 min_objects--;
>> > >         }
>> > > -       /*
>> > > -        * We were unable to place multiple objects in a slab.
>> > > Now
>> > > -        * lets see if we can place a single object there.
>> > > -        */
>> > > -       order = calc_slab_order(size, 1, slub_max_order, 1);
>> > > -       if (order <= slub_max_order)
>> > > -               return order;
>> > 
>> > I'm not sure if it's okay to remove this?
>> > It was fine in v2 because the least wasteful order was chosen
>> > regardless of fraction but that's not true anymore.
>> > 
>> > Otherwise, everything looks fine to me. I'm too dumb to anticipate
>> > the outcome of increasing the slab order :P but this patch does not
>> > sound crazy to me.
>> 
>> I wanted to have a better idea how the orders change so I hacked up a
>> patch
>> to print them for all sizes up to 1MB (unnecessarily large I guess)
>> and also
>> for various page sizes and nr_cpus (that's however rather invasive
>> and prone
>> to me missing some helper being used that still relies on real
>> PAGE_SHIFT),
>> then I applied v4 (needed some conflict fixups with my hack) on top:
>> 
>> https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slab-orders
>> 
>> As expected, things didn't change with 4k PAGE_SIZE. With 64k
>> PAGE_SIZE, I
>> thought the patch in v4 form would result in lower orders, but seems
>> not always?
>> 
>> I.e. I can see before the patch:
>> 
>>  Calculated slab orders for page_shift 16 nr_cpus 1:
>>           8       0
>>        4376       1
>> 
>> (so until 4368 bytes it keeps order at 0)
>> 
>> And after:
>>           8       0
>>        2264       1
>>        2272       0
>>        2344       1
>>        2352       0
>>        2432       1
>> 
>> Not sure this kind of "oscillation" is helpful with a small machine
>> (1CPU),
>> and 64kB pages so the unused part of page is quite small.
>> 
> Hi Vlastimil,
>  
> With patch. it will cause the fraction_size to rise to 32
> when utilizing a 64k page size. As a result, the maximum wastage cap
> for each slab cache will be 2k (64k divided by 32). Any object size
> exceeding this cap will be moved to order 1 or beyond due to which this
> oscillation is seen.

Hi, sorry for the late reply.

>> With 16 cpus, AFAICS the orders are also larger for some sizes.
>> Hm but you reported reduction of total slab memory which suggests
>> lower
>> orders were selected somewhere, so maybe I did some mistake.A
> 
> AFAIK total slab memory is reduce because of two reason (with this
> patch for larger page size) 
> 1) order for some slab cache is reduce (by increasing fraction_size)

How can increased fraction_size ever result in a lower order? I think it can
only result in increased order (or same order). And the simulations with my
hack patch don't seem to counter example that. Note previously I did expect
the order to be lower (or same) and was surprised by my results, but now I
realized I misunderstood the v4 patch.

> 2) Have also seen reduction in overall slab cache numbers as because of
> increasing page order

I think your results might be just due to randomness and could turn out
different with repeating the test, or converge to be the same if you average
multiple runs. You posted them for "160 CPUs with 64K Page size" and if I
add that combination to my hack print, I see the same result before and
after your patch:

Calculated slab orders for page_shift 16 nr_cpus 160:
         8       0
      1824       1
      3648       2
      7288       3
    174768       2
    196608       3
    524296       4

Still, I might have a bug there. Can you confirm there are actual
differences with a /proc/slabinfo before/after your patch? If there are
none, any differences observed have to be due to randomness, not differences
in order.

Going back to the idea behind your patch, I don't think it makes sense to
try increase the fraction only for higher-orders. Yes, with 1/16 fraction,
the waste with 64kB page can be 4kB, while with 1/32 it will be just 2kB,
and with 4kB this is only 256 vs 128bytes. However the object sizes and
counts don't differ with page size, so with 4kB pages we'll have more slabs
to host the same number of objects, and the waste will accumulate
accordingly - i.e. the fraction metric should be independent of page size
wrt resulting total kilobytes of waste.

So maybe the only thing we need to do is to try setting it to 32 initial
value instead of 16 regardless of page size. That should hopefully again
show a good tradeoff for 4kB as one of the earlier versions, while on 64kB
it shouldn't cause much difference (again, none at all with 160 cpus, some
difference with less than 128 cpus, if my simulations were correct).

>> 
>> Anyway my point here is that this evaluation approach might be
>> useful, even
>> if it's a non-upstreamable hack, and some postprocessing of the
>> output is
>> needed for easier comparison of before/after, so feel free to try
>> that out.
> 
> Thank you for this details test :) 
>> 
>> BTW I'll be away for 2 weeks from now, so further feedback will have
>> to come
>> from others in that time...
>> 
> Do we have any additional feedback from others on the same matter?
> 
> Thank
> 
> Jay Patel
>> > Thanks!
>> > --
>> > Hyeonggon
> 
>