Re: [PATCH 0/4] Memory controller soft limit patches (v3)

[KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>]

On Mon, 2 Mar 2009 12:06:49 +0530
Balbir Singh <balbir@linux.vnet.ibm.com> wrote:

> * KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> [2009-03-02 15:21:28]:
> 
> > On Mon, 2 Mar 2009 11:35:19 +0530
> > Balbir Singh <balbir@linux.vnet.ibm.com> wrote:
> > 
> > > * KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> [2009-03-02 14:32:50]:
> > > 
> > > > On Mon, 2 Mar 2009 10:10:43 +0530
> > > > Balbir Singh <balbir@linux.vnet.ibm.com> wrote:
> > > > 
> > > > > * KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> [2009-03-02 09:24:04]:
> > > > > 
> > > > > > On Sun, 01 Mar 2009 11:59:59 +0530
> > > > > > Balbir Singh <balbir@linux.vnet.ibm.com> wrote:
> > > > > > 
> > > > > > > 
> > > > > > > From: Balbir Singh <balbir@linux.vnet.ibm.com>
> > > > 
> > > > > > 
> > > > > > At first, it's said "When cgroup people adds something, the kernel gets slow".
> > > > > > This is my start point of reviewing. Below is comments to this version of patch.
> > > > > > 
> > > > > >  1. I think it's bad to add more hooks to res_counter. It's enough slow to give up
> > > > > >     adding more fancy things..
> > > > > 
> > > > > res_counters was desgined to be extensible, why is adding anything to
> > > > > it going to make it slow, unless we turn on soft_limits?
> > > > > 
> > > > You inserted new "if" logic in the core loop.
> > > > (What I want to say here is not that this is definitely bad but that "isn't there
> > > >  any alternatives which is less overhead.)
> > > > 
> > > > 
> > > > > > 
> > > > > >  2. please avoid to add hooks to hot-path. In your patch, especially a hook to
> > > > > >     mem_cgroup_uncharge_common() is annoying me.
> > > > > 
> > > > > If soft limits are not enabled, the function does a small check and
> > > > > leaves. 
> > > > > 
> > > > &soft_fail_res is passed always even if memory.soft_limit==ULONG_MAX
> > > > res_counter_soft_limit_excess() adds one more function call and spinlock, and irq-off.
> > > >
> > > 
> > > OK, I see that overhead.. I'll figure out a way to work around it.
> > >  
> > > > > > 
> > > > > >  3. please avoid to use global spinlock more. 
> > > > > >     no lock is best. mutex is better, maybe.
> > > > > > 
> > > > > 
> > > > > No lock to update a tree which is update concurrently?
> > > > > 
> > > > Using tree/sort itself is nonsense, I believe.
> > > > 
> > > 
> > > I tried using prio trees in the past, but they are not easy to update
> > > either. I won't mind asking for suggestions for a data structure that
> > > can scaled well, allow quick insert/delete and search.
> > > 
> > Now, because the routine is called by kswapd() not by try_to_free.....
> > 
> > It's not necessary to be very very fast. That's my point.
> >
> 
> OK, I get your point, but whay does that make RB-Tree data structure non-sense?
>  

 1. Until memory-shortage, rb-tree is kept to be updated and the users(kernel)
    has to pay its maintainace/check cost, whici is unnecessary.
    Considering trade-off, paying cost only when memory-shortage happens tend to
    be reasonable way.

 2. Current "exceed" just shows "How much we got over my soft limit" but doesn't
    tell any information per-node/zone. Considering this, this rb-tree
    information will not be able to help kswapd (on NUMA).
    But maintain per-node information uses too much resource.

 Considering above 2, it's not bad to find victim by proper logic
 from balance_pgdat() by using mem_cgroup_select_victim().
 like this:
==
 struct mem_cgroup *select_vicitim_at_soft_limit_via_balance_pgdat(int nid, int zid)
 {
     while (?) {
        vitcim = mem_cgroup_select_victim(init_mem_cgroup);  #need some modification.
        if (victim is not over soft-limit)
             continue;
        /* Ok this is candidate */
        usage = mem_cgroup_nid_zid_usage(mem, nid, zid); #get sum of active/inactive
        if (usage_is_enough_big)
              return victim;
     }
 }
 balance_pgdat()
 ...... find target zone....
 ...
 mem = select_victime_at_soft_limit_via_balance_pgdat(nid, zid)
 if (mem)
   sc->mem = mem;
 shrink_zone();
 if (mem) {
   sc->mem = NULL;
   css_put(&mem->css);
 }
==

 We have to pay scan cost but it will not be too big(if there are not thousands of memcg.)
 Under above, round-robin rotation is used rather than sort.
 Maybe I can show you sample.....(but I'm a bit busy.)
 
Thanks,
-Kame