mm/slab: extremly high slab cache problem

mm/slab: extremly high slab cache problem

[lulu he]

Hi,

I'am not used to ask but I have followed every piece of information I
could not find solution, so I'am trying here for help.

On one of my server, I have some memory/disk KV service,
Memory KV behave like memcached, ask for a big trunk of memory(10GB)
when initialized,
Disk Kv behave like leveldbd, its random read and sequential write,
and it frequently reads a lot files.
Memory are all alloced using libc malloc.

My KV server process do not consume a lot of memory as below (since
lack of memory, I have killed memory KV, leaving only disk KV, but
free memory still goes down):

:~$top
PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20 0 5030m 3.9g 2772 S 8 6.1 10430:52 tair_server
20 0 4833m 3.9g 4560 S 8 6.1 10171:07 tair_server
20 0 4844m 3.9g 3844 S 38 6.1 10073:32 tair_server
20 0 4765m 3.8g 4144 S 8 6.0 10552:39 tair_server
20 0 2941m 2.4g 9.8m S 0 3.8 256:45.70 tair_server
20 0 2953m 2.4g 12m S 1 3.7 276:54.64 tair_server

BUT, my memory are gone.
$free -m
             total       used       free     shared    buffers     cached
Mem:         64552      57778       6774          0         16        326
-/+ buffers/cache:      57435       7117
Swap:            0          0          0


I can see slab consume lots of my memory, and it's unreclaimable.
$cat /proc/meminfo
MemTotal:       66101892 kB
MemFree:         6816228 kB
Buffers:           17024 kB
Cached:           456640 kB
SwapCached:            0 kB
Active:         19697712 kB
Inactive:        3197312 kB
Active(anon):   19546504 kB
Inactive(anon):  2875632 kB
Active(file):     151208 kB
Inactive(file):   321680 kB
Unevictable:          48 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:              6612 kB
Writeback:            72 kB
AnonPages:      22421152 kB
Mapped:            54408 kB
Shmem:               332 kB
Slab:           28870400 kB
SReclaimable:     213344 kB
SUnreclaim:     28657056 kB
KernelStack:       30000 kB
PageTables:        62776 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    33050944 kB
Committed_AS:   37517224 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      388624 kB
VmallocChunk:   34324313700 kB
HardwareCorrupted:     0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        5696 kB
DirectMap2M:     2082816 kB
DirectMap1G:    65011712 kB

Here is the slab info.
$slabtop -s c
 OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
69842766 69838389  38%    0.19K 1663025       42  13304200K kmalloc-192
69314912 69314796  38%    0.12K 2166091       32   8664364K kmalloc-128
70866624 70866323  39%    0.06K 1107291       64   4429164K kmalloc-64
69299968 69299592  38%    0.03K 541406      128   2165624K kmalloc-32
128388  72434  56%    0.99K   4230       32    135360K ext4_inode_cache
208782  94112  45%    0.19K   4971       42     39768K dentry


I don't who and why consume such lots memory, and how to solve this.
Can this be an interval kernel error?
OR it's a glibc problem, it do'not return memory back to system, due
to frequently disk read?

thanks in advance, any help will be appreciated.


--
loloseed