* extreme system load [kswapd]
@ 2012-03-20 9:08 Karol Šebesta
2012-03-20 9:47 ` Nicolas Maupu
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Karol Šebesta @ 2012-03-20 9:08 UTC (permalink / raw)
To: linux-kernel; +Cc: linux-admin
Hello @All,
We have a problem on our production machine with high CPU utilization
caused by kswapd3 daemon. Server is 128GB of physical memory and 81GB
of SWAP.
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.6 (Tikanga)
#
# uptime
09:52:13 up 38 days, 17:01, 14 users, load average: 43.93, 50.02, 51.36
#
# free -m
total used free shared buffers cached
Mem: 128989 75577 53412 0 416 57131
-/+ buffers/cache: 18029 110960
Swap: 81919 31310 50609
#
### from TOP:
Tasks: 1590 total, 3 running, 1578 sleeping, 1 stopped, 8 zombie
Cpu(s): 1.1%us, 1.9%sy, 0.0%ni, 74.4%id, 22.4%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 132085528k total, 77397544k used, 54687984k free, 426472k buffers
Swap: 83886064k total, 32043168k used, 51842896k free, 58475692k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1816 root 10 -5 0 0 0 D 203.0 0.0 1124:39 kswapd3
10801 root 15 0 113m 23m 1680 S 158.6 0.0 1694:13 vxconfigd
3925 root RT 0 162m 15m 3368 S 140.1 0.0 1313:32 multipathd
21399 wsy5067 16 0 22768 3192 1704 S 50.3 0.0 0:12.96 top
10583 root 15 0 0 0 0 S 43.8 0.0 58:56.55 dmp_daemon
16673 root 16 0 17720 1220 792 S 37.3 0.0 732:23.49 cmafcad
24636 oracle 15 0 3314m 41m 40m S 26.6 0.0 29:31.84 oracle
487 oracle 15 0 3301m 90m 88m S 21.4 0.1 1:05.68 oracle
14197 root 18 0 315m 15m 3984 S 21.1 0.0 346:15.70 vxsvc
26071 root 15 0 23900 6788 2092 S 19.8 0.0 12:31.44 MountAgent
16418 root 15 0 5308 3656 592 S 19.1 0.0 544:54.78 cmahostd
485 oracle 15 0 3301m 87m 87m S 18.8 0.1 1:03.92 oracle
21456 root 15 0 0 0 0 S 18.5 0.0 14:16.09 pdflush
4699 root 22 0 65204 8420 1568 S 15.6 0.0 137:15.75 vxdclid
16603 root 18 0 8820 6808 332 D 14.3 0.0 1290:32 cmaperfd
22047 tlmcron1 18 0 1227m 115m 6812 S 12.3 0.1 554:55.86 java
6157 oracle 16 0 3302m 118m 115m D 11.4 0.1 0:01.29 oracle
30669 oracle 16 0 2066m 141m 134m D 10.7 0.1 0:20.19 oracle
357 tlmtst3 18 0 101m 15m 9292 S 9.7 0.0 0:29.43 stlfetch
15005 oracle 16 0 2046m 84m 75m D 9.7 0.1 0:01.75 oracle
32744 tlmtst3 18 0 101m 15m 9292 S 8.8 0.0 0:31.52 stlfetch
26077 root 15 0 12116 4228 1936 S 7.8 0.0 4:26.71 VolumeAgent
17955 oracle 16 0 3301m 28m 25m D 7.1 0.0 0:00.22 oracle
26076 root 15 0 23268 5284 2224 S 6.8 0.0 4:00.50 OracleAgent
111 root 10 -5 0 0 0 S 6.5 0.0 4:57.32 events/13
5112 oracle 16 0 3314m 35m 35m S 6.5 0.0 10:38.55 oracle
10440 oracle 16 0 3314m 36m 36m S 6.5 0.0 10:54.65 oracle
26074 root 15 0 23012 4904 2136 S 6.5 0.0 3:09.46 NetlsnrAgent
8631 oracle 16 0 3315m 37m 36m S 6.2 0.0 12:04.57 oracle
26636 oracle 15 0 3316m 40m 39m S 6.2 0.0 13:07.75 oracle
10780 oracle 16 0 6386m 39m 39m D 5.8 0.0 10:57.01 oracle
13091 tlmtst5 15 0 423m 18m 13m S 5.8 0.0 0:02.76 gedit
17212 oracle 15 0 2050m 39m 39m S 5.8 0.0 11:30.82 oracle
20283 oracle 15 0 3314m 41m 40m S 5.8 0.0 10:49.00 oracle
13147 oracle 15 0 3314m 43m 43m S 5.5 0.0 10:43.12 oracle
16875 oracle 15 0 3316m 42m 40m S 5.5 0.0 22:52.66 oracle
28143 root 15 0 111m 6700 3396 S 5.5 0.0 143:25.46 opcmona
5718 oracle 16 0 3314m 38m 36m S 5.2 0.0 11:30.95 oracle
22163 oracle 15 0 3316m 39m 38m S 5.2 0.0 32:51.94 oracle
26065 root 15 0 22960 4724 2164 S 4.5 0.0 1:53.23 ApplicationAgen
26612 oracle 18 0 3302m 22m 19m S 4.2 0.0 23:31.48 oracle
8597 oracle 18 0 3302m 18m 16m S 3.9 0.0 23:30.62 oracle
10690 oracle 18 0 6373m 20m 16m S 3.9 0.0 23:28.74 oracle
11822 oracle 18 0 2038m 29m 25m S 3.9 0.0 0:05.17 oracle
Why the kswapd daemon causing so high CPU utilization? Can it be
caused by some process which try to access memory pages which are not
in memory, but in swap? Or can it be cause by swapping for exapmple
some inactive oracle database SGA from memory to swap?
Thank you for help.
Regards,
Karol Sebesta
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: extreme system load [kswapd]
2012-03-20 9:08 extreme system load [kswapd] Karol Šebesta
@ 2012-03-20 9:47 ` Nicolas Maupu
2012-03-20 18:59 ` Peter Zijlstra
2012-03-21 14:58 ` Rik van Riel
2 siblings, 0 replies; 6+ messages in thread
From: Nicolas Maupu @ 2012-03-20 9:47 UTC (permalink / raw)
To: Karol Šebesta; +Cc: linux-kernel, linux-admin
Hello,
Maybe, try to track IO on your machine (iostat) to see if it is an IO
problem (i.e. write to disk) or not.
On Tue, Mar 20, 2012 at 10:08, Karol Šebesta <sebesta.karol@gmail.com> wrote:
>
> Hello @All,
>
> We have a problem on our production machine with high CPU utilization
> caused by kswapd3 daemon. Server is 128GB of physical memory and 81GB
> of SWAP.
>
>
> # cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 5.6 (Tikanga)
> #
>
>
> # uptime
> 09:52:13 up 38 days, 17:01, 14 users, load average: 43.93, 50.02, 51.36
> #
>
>
> # free -m
> total used free shared buffers cached
> Mem: 128989 75577 53412 0 416 57131
> -/+ buffers/cache: 18029 110960
> Swap: 81919 31310 50609
> #
>
> ### from TOP:
>
> Tasks: 1590 total, 3 running, 1578 sleeping, 1 stopped, 8 zombie
> Cpu(s): 1.1%us, 1.9%sy, 0.0%ni, 74.4%id, 22.4%wa, 0.0%hi, 0.2%si, 0.0%st
> Mem: 132085528k total, 77397544k used, 54687984k free, 426472k buffers
> Swap: 83886064k total, 32043168k used, 51842896k free, 58475692k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 1816 root 10 -5 0 0 0 D 203.0 0.0 1124:39 kswapd3
> 10801 root 15 0 113m 23m 1680 S 158.6 0.0 1694:13 vxconfigd
> 3925 root RT 0 162m 15m 3368 S 140.1 0.0 1313:32 multipathd
> 21399 wsy5067 16 0 22768 3192 1704 S 50.3 0.0 0:12.96 top
> 10583 root 15 0 0 0 0 S 43.8 0.0 58:56.55 dmp_daemon
> 16673 root 16 0 17720 1220 792 S 37.3 0.0 732:23.49 cmafcad
> 24636 oracle 15 0 3314m 41m 40m S 26.6 0.0 29:31.84 oracle
> 487 oracle 15 0 3301m 90m 88m S 21.4 0.1 1:05.68 oracle
> 14197 root 18 0 315m 15m 3984 S 21.1 0.0 346:15.70 vxsvc
> 26071 root 15 0 23900 6788 2092 S 19.8 0.0 12:31.44 MountAgent
> 16418 root 15 0 5308 3656 592 S 19.1 0.0 544:54.78 cmahostd
> 485 oracle 15 0 3301m 87m 87m S 18.8 0.1 1:03.92 oracle
> 21456 root 15 0 0 0 0 S 18.5 0.0 14:16.09 pdflush
> 4699 root 22 0 65204 8420 1568 S 15.6 0.0 137:15.75 vxdclid
> 16603 root 18 0 8820 6808 332 D 14.3 0.0 1290:32 cmaperfd
> 22047 tlmcron1 18 0 1227m 115m 6812 S 12.3 0.1 554:55.86 java
> 6157 oracle 16 0 3302m 118m 115m D 11.4 0.1 0:01.29 oracle
> 30669 oracle 16 0 2066m 141m 134m D 10.7 0.1 0:20.19 oracle
> 357 tlmtst3 18 0 101m 15m 9292 S 9.7 0.0 0:29.43 stlfetch
> 15005 oracle 16 0 2046m 84m 75m D 9.7 0.1 0:01.75 oracle
> 32744 tlmtst3 18 0 101m 15m 9292 S 8.8 0.0 0:31.52 stlfetch
> 26077 root 15 0 12116 4228 1936 S 7.8 0.0 4:26.71 VolumeAgent
> 17955 oracle 16 0 3301m 28m 25m D 7.1 0.0 0:00.22 oracle
> 26076 root 15 0 23268 5284 2224 S 6.8 0.0 4:00.50 OracleAgent
> 111 root 10 -5 0 0 0 S 6.5 0.0 4:57.32 events/13
> 5112 oracle 16 0 3314m 35m 35m S 6.5 0.0 10:38.55 oracle
> 10440 oracle 16 0 3314m 36m 36m S 6.5 0.0 10:54.65 oracle
> 26074 root 15 0 23012 4904 2136 S 6.5 0.0 3:09.46 NetlsnrAgent
> 8631 oracle 16 0 3315m 37m 36m S 6.2 0.0 12:04.57 oracle
> 26636 oracle 15 0 3316m 40m 39m S 6.2 0.0 13:07.75 oracle
> 10780 oracle 16 0 6386m 39m 39m D 5.8 0.0 10:57.01 oracle
> 13091 tlmtst5 15 0 423m 18m 13m S 5.8 0.0 0:02.76 gedit
> 17212 oracle 15 0 2050m 39m 39m S 5.8 0.0 11:30.82 oracle
> 20283 oracle 15 0 3314m 41m 40m S 5.8 0.0 10:49.00 oracle
> 13147 oracle 15 0 3314m 43m 43m S 5.5 0.0 10:43.12 oracle
> 16875 oracle 15 0 3316m 42m 40m S 5.5 0.0 22:52.66 oracle
> 28143 root 15 0 111m 6700 3396 S 5.5 0.0 143:25.46 opcmona
> 5718 oracle 16 0 3314m 38m 36m S 5.2 0.0 11:30.95 oracle
> 22163 oracle 15 0 3316m 39m 38m S 5.2 0.0 32:51.94 oracle
> 26065 root 15 0 22960 4724 2164 S 4.5 0.0 1:53.23 ApplicationAgen
> 26612 oracle 18 0 3302m 22m 19m S 4.2 0.0 23:31.48 oracle
> 8597 oracle 18 0 3302m 18m 16m S 3.9 0.0 23:30.62 oracle
> 10690 oracle 18 0 6373m 20m 16m S 3.9 0.0 23:28.74 oracle
> 11822 oracle 18 0 2038m 29m 25m S 3.9 0.0 0:05.17 oracle
>
> Why the kswapd daemon causing so high CPU utilization? Can it be
> caused by some process which try to access memory pages which are not
> in memory, but in swap? Or can it be cause by swapping for exapmple
> some inactive oracle database SGA from memory to swap?
>
> Thank you for help.
>
>
> Regards,
>
> Karol Sebesta
> --
> To unsubscribe from this list: send the line "unsubscribe linux-admin" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Nicolas Maupu - SS2J - Excilys
Tél :+33 (0) 1 41 24 43 27
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: extreme system load [kswapd]
2012-03-20 9:08 extreme system load [kswapd] Karol Šebesta
2012-03-20 9:47 ` Nicolas Maupu
@ 2012-03-20 18:59 ` Peter Zijlstra
2012-03-21 14:58 ` Rik van Riel
2 siblings, 0 replies; 6+ messages in thread
From: Peter Zijlstra @ 2012-03-20 18:59 UTC (permalink / raw)
To: Karol Šebesta; +Cc: linux-kernel, linux-admin, Rik van Riel
On Tue, 2012-03-20 at 10:08 +0100, Karol Šebesta wrote:
> We have a problem on our production machine with high CPU utilization
> caused by kswapd3 daemon. Server is 128GB of physical memory and 81GB
> of SWAP.
>
>
> # cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 5.6 (Tikanga)
That's a very very old kernel, I would suggest you upgrade to something
that has the reclaim rewrite Rik did to deal with large memory systems.
I think they're in RHEL6, but I'm sure Rik knows.
Also, since you're running this dinosaur, contact RHT, they're the only
ones that care about it -- but I guess you're going to get the same
suggestion.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: extreme system load [kswapd]
2012-03-20 9:08 extreme system load [kswapd] Karol Šebesta
2012-03-20 9:47 ` Nicolas Maupu
2012-03-20 18:59 ` Peter Zijlstra
@ 2012-03-21 14:58 ` Rik van Riel
2012-04-12 11:35 ` Karol Šebesta
2 siblings, 1 reply; 6+ messages in thread
From: Rik van Riel @ 2012-03-21 14:58 UTC (permalink / raw)
To: Karol Šebesta; +Cc: linux-kernel, linux-admin
On 03/20/2012 05:08 AM, Karol Šebesta wrote:
> Hello @All,
>
> We have a problem on our production machine with high CPU utilization
> caused by kswapd3 daemon. Server is 128GB of physical memory and 81GB
> of SWAP.
> # free -m
> total used free shared buffers cached
> Mem: 128989 75577 53412 0 416 57131
> -/+ buffers/cache: 18029 110960
> Swap: 81919 31310 50609
> #
Looks like a combination of NUMA and the workload thrown
at the system. You did not post any vmstat output, or
info on the size of your Oracle SGA, so I will take some
wild guesses here :)
Not only are you 31GB in swap, you also have 53GB of
memory free. Additionally, only kswapd3 is very busy,
while kswapd on the other NUMA nodes do not even show
up in top.
I would guess that the value of /proc/sys/vm/zone_reclaim_mode
is 1, causing the system to reclaim memory from the NUMA node
where things are running, instead of overflowing memory
allocations into other NUMA nodes.
Setting zone_reclaim_mode to 0 could resolve some of your
issues.
Another, more fundamental, issue is that on older kernels
we mix page cache and process pages on the same LRU lists.
This causes the pageout code to scan over many pages that
we do not want to evict, increasing CPU use by kswapd and
other processes invoking the pageout code.
That issue got fixed in newer kernels, including the
kernel in RHEL 6.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: extreme system load [kswapd]
2012-03-21 14:58 ` Rik van Riel
@ 2012-04-12 11:35 ` Karol Šebesta
2012-04-12 11:36 ` Karol Šebesta
0 siblings, 1 reply; 6+ messages in thread
From: Karol Šebesta @ 2012-04-12 11:35 UTC (permalink / raw)
To: Rik van Riel, linux-kernel
Hello Rik,
Thank you for answer, but now we are facing to another interesting
issue on our redhat 5.6 server.
It seems that linux dropped almost all disk cache and now it trying to
gain disk cache space by swapping. That is the reason why kswapd
process loads the system.
### as you can see almost all RAM is free ( that is bad ) - some time
ago there was all RAM cached in disk cache
[root@dusu0216 ~]# free -m
total used free shared buffers cached
Mem: 128861 29814 99047 0 108 20186
-/+ buffers/cache: 9519 119342
Swap: 81919 16614 65305
[root@dusu0216 ~]#
Do you think that can be a problem with NUMA nodes or what can be the
reason that Linux itself dropped almost all disk cache?
Thank you.
Regards,
Karol Sebesta
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: extreme system load [kswapd]
2012-04-12 11:35 ` Karol Šebesta
@ 2012-04-12 11:36 ` Karol Šebesta
0 siblings, 0 replies; 6+ messages in thread
From: Karol Šebesta @ 2012-04-12 11:36 UTC (permalink / raw)
To: Rik van Riel, linux-kernel
# cat /proc/sys/vm/zone_reclaim_mode
0
#
The value is set to zero, and we have these problems.... interesting...
Regards,
Karol Sebesta
2012/4/12 Karol Šebesta <sebesta.karol@gmail.com>:
> Hello Rik,
>
> Thank you for answer, but now we are facing to another interesting
> issue on our redhat 5.6 server.
>
> It seems that linux dropped almost all disk cache and now it trying to
> gain disk cache space by swapping. That is the reason why kswapd
> process loads the system.
>
> ### as you can see almost all RAM is free ( that is bad ) - some time
> ago there was all RAM cached in disk cache
>
> [root@dusu0216 ~]# free -m
> total used free shared buffers cached
> Mem: 128861 29814 99047 0 108 20186
> -/+ buffers/cache: 9519 119342
> Swap: 81919 16614 65305
> [root@dusu0216 ~]#
>
>
> Do you think that can be a problem with NUMA nodes or what can be the
> reason that Linux itself dropped almost all disk cache?
>
> Thank you.
>
> Regards,
>
> Karol Sebesta
--
S pozdravom
Karol Šebesta
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2012-04-12 11:36 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-03-20 9:08 extreme system load [kswapd] Karol Šebesta
2012-03-20 9:47 ` Nicolas Maupu
2012-03-20 18:59 ` Peter Zijlstra
2012-03-21 14:58 ` Rik van Riel
2012-04-12 11:35 ` Karol Šebesta
2012-04-12 11:36 ` Karol Šebesta
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).