* 2.6.0: something is leaking memory
@ 2004-01-04 18:57 Erik Hensema
2004-01-04 21:00 ` Linus Torvalds
2004-01-04 23:18 ` Mike Fedyk
0 siblings, 2 replies; 16+ messages in thread
From: Erik Hensema @ 2004-01-04 18:57 UTC (permalink / raw)
To: linux-kernel
Hi,
About a month ago I installed 2.6.0-test11 on my home server (NAT
gateway, NFS server, news, mail, ntp, ldap, you name it). From
that moment there's some memory leak in the system. It could be
userspace, but there are no changes there since before the
upgrade.
The leak can be most easily seen in my rrdtool graphs of memory
usage: http://dexter.hensema.net/~erik/stats/mem-mon.gif and
http://dexter.hensema.net/~erik/stats/mem-year.gif - try to guess
when I switched to 2.6.0-test11 ;-)
At Dec 31 I upgraded to 2.6.0-final.
Output from ps aux, /proc/vmstat and /proc/meminfo, are attached.
My .config isn't compiled in and I haven't got it at hand
currently. If anyone is interested I can post it tomorrow.
The machine is running SuSE 8.2, fairly standard. INN is compiled
from source, it's 2.4.0, but the other daemons came with the
distribution.
LVM, procps, modutils are all updated. All filesystems are
reiserfs on LVM (except for the root partition, which is a
regular partition, and /boot which is ext2 on a partition).
It's a Duron 700 with 256 MB ram, IDE disk. No preempt, no
framebuffer. Three network cards: de4x5, tulip and 8139too
drivers.
Modules:
Module Size Used by
i2c_dev 8256 0
eeprom 5960 0
i2c_isa 1664 0
tun 6784 1
nfsd 94768 8
exportfs 5056 1 nfsd
ip6t_MARK 1664 5
ipt_MARK 1664 4
ipt_mark 1344 1
ipt_TOS 1920 5
ipt_length 1344 5
ipt_tos 1280 5
ip6table_mangle 1920 1
iptable_mangle 2112 1
cls_fw 3520 4
sch_sfq 4480 4
sch_htb 22016 1
ip6t_LOG 4672 1
ip6table_filter 2048 1
ip6_tables 16000 4 ip6t_MARK,ip6table_mangle,ip6t_LOG,ip6table_filter
ipt_MASQUERADE 2752 4
ip_nat_ftp 3952 0
ipt_REJECT 5376 4
ipt_state 1408 4
ipt_LOG 4736 8
ipt_limit 1856 5
ipt_ULOG 5480 21
iptable_filter 2112 1
ip_nat_irc 3312 0
iptable_nat 18668 4 ipt_MASQUERADE,ip_nat_ftp,ip_nat_irc
ip_conntrack_irc 70260 1 ip_nat_irc
ip_conntrack_ftp 70964 1 ip_nat_ftp
ip_conntrack 26160 7 ipt_MASQUERADE,ip_nat_ftp,ipt_state,ip_nat_irc,iptable_nat,ip_conntrack_irc,ip_conntrack_ftp
ip_tables 15104 14 ipt_MARK,ipt_mark,ipt_TOS,ipt_length,ipt_tos,iptable_mangle,ipt_MASQUERADE,ipt_REJECT,ipt_state,ipt_LOG,ipt_limit,ipt_ULOG,iptable_filter,iptable_nat
8139too 19136 0
mii 4096 1 8139too
tulip 43872 0
de4x5 48416 0
via686a 18376 0
lm75 6532 0
i2c_sensor 2304 3 eeprom,via686a,lm75
i2c_viapro 5900 0
i2c_core 20808 7 i2c_dev,eeprom,i2c_isa,via686a,lm75,i2c_sensor,i2c_viapro
The following dumps were made just before rebooting on Dec 31:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 500 72 ? S Dec02 0:14 init [3]
root 2 0.0 0.0 0 0 ? SWN Dec02 0:00 [ksoftirqd/0]
root 3 0.0 0.0 0 0 ? SW< Dec02 0:00 [events/0]
root 4 0.0 0.0 0 0 ? SW< Dec02 2:05 [kblockd/0]
root 7 0.0 0.0 0 0 ? SW Dec02 2:57 [kswapd0]
root 8 0.0 0.0 0 0 ? SW< Dec02 0:00 [aio/0]
root 9 0.0 0.0 0 0 ? SW Dec02 0:00 [kseriod]
root 10 0.0 0.0 0 0 ? SW< Dec02 0:00 [reiserfs/0]
root 213 0.0 0.1 1488 444 ttyS0 S Dec02 0:01 /sbin/powerd
root 350 0.0 0.1 1516 340 ? S Dec02 2:54 /sbin/dhcpcd -D -R -N -Y -t 999999 -h cc78409-a eth0
root 689 0.0 0.2 1788 628 ? S Dec02 22:24 /sbin/syslogd -a /var/lib/dhcp/dev/log -a /var/lib/named/dev/log
root 692 0.0 0.1 2632 440 ? S Dec02 0:07 /sbin/klogd -c 1 -2
root 873 0.0 0.5 2344 1300 ? S Dec02 17:17 /usr/local/sbin/ulog-acctd
root 883 0.0 0.1 1524 380 ? S Dec02 0:00 /sbin/resmgrd
bin 893 0.0 0.1 1528 440 ? S Dec02 0:01 /sbin/portmap
root 906 0.0 0.2 2764 568 ? S Dec02 0:03 /usr/sbin/zebra -d
root 924 0.0 0.0 0 0 ? SW Dec02 4:09 [nfsd]
root 925 0.0 0.0 0 0 ? SW Dec02 4:15 [nfsd]
root 926 0.0 0.0 0 0 ? SW Dec02 4:45 [nfsd]
root 927 0.0 0.0 0 0 ? SW Dec02 4:30 [nfsd]
root 928 0.0 0.0 0 0 ? SW Dec02 4:55 [nfsd]
root 932 0.0 0.0 0 0 ? SW Dec02 0:01 [lockd]
root 933 0.0 0.0 0 0 ? SW Dec02 0:00 [rpciod]
root 929 0.0 0.0 0 0 ? SW Dec02 4:02 [nfsd]
root 930 0.0 0.0 0 0 ? SW Dec02 4:05 [nfsd]
root 931 0.0 0.0 0 0 ? SW Dec02 4:20 [nfsd]
root 947 0.0 0.2 1804 648 ? S Dec02 0:01 /usr/sbin/rpc.mountd
root 1118 0.0 0.1 2608 472 ? S Dec02 0:00 /bin/sh /usr/bin/safe_mysqld --user=mysql --pid-file=/var/lib/mysql/mysqld.pid --socket=/var/lib/mysql/mysql.sock --datadir=/var/lib/mysql
mysql 1155 0.0 0.5 26244 1472 ? S Dec02 3:34 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/mysqld.pid --skip-locking
mysql 1156 0.0 0.5 26244 1472 ? S Dec02 0:13 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/mysqld.pid --skip-locking
mysql 1157 0.0 0.5 26244 1472 ? S Dec02 0:03 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/mysqld.pid --skip-locking
root 1210 0.0 0.2 4388 688 ? S Dec03 0:18 /usr/sbin/sshd -o PidFile /var/run/sshd.init.pid
root 1235 0.1 0.3 3152 792 ? S Dec03 57:14 /usr/local/sbin/tincd -n wlan
ntp 1307 0.0 0.8 2164 2156 ? SL Dec03 0:09 /usr/sbin/ntpd -U ntp
at 1377 0.0 0.0 1656 112 ? S Dec03 0:00 /usr/sbin/atd
root 1467 0.0 0.0 1640 168 ? S Dec03 0:05 /usr/sbin/cron
root 1472 0.0 0.1 2064 392 ? S Dec03 0:17 /usr/local/bin/fetchmail -f /etc/fetchmailrc
root 1483 0.0 0.0 1548 224 ? S Dec03 0:01 /usr/sbin/inetd
root 1495 0.0 0.2 1756 552 ? S Dec03 0:00 /sbin/rpc.statd
root 1500 0.0 0.0 1624 72 tty1 S Dec03 0:00 /sbin/mingetty --noclear tty1
root 1501 0.0 0.1 1624 364 tty2 S Dec03 0:00 /sbin/mingetty tty2
root 1502 0.0 0.0 1624 72 tty3 S Dec03 0:00 /sbin/mingetty tty3
root 1503 0.0 0.0 1624 72 tty4 S Dec03 0:00 /sbin/mingetty tty4
root 1504 0.0 0.0 1624 72 tty5 S Dec03 0:00 /sbin/mingetty tty5
root 1505 0.0 0.0 1624 72 tty6 S Dec03 0:00 /sbin/mingetty tty6
logger 23531 0.0 0.0 6924 252 ? S Dec03 0:06 SCREEN irssi
logger 23532 0.0 0.5 8292 1320 pts/1 S Dec03 2:03 irssi
erik 9850 0.0 0.0 7308 88 ? S Dec07 0:00 /usr/bin/perl -w /home/erik/swentrap.pl
erik 9851 0.0 0.0 1644 72 ? S Dec07 0:00 whois -h whois.abuse.net smtp4.hy.skanova.net.
erik 9852 0.0 0.0 7176 88 ? S Dec07 0:00 /usr/bin/perl -w /home/erik/swentrap.pl
erik 9853 0.0 0.0 1644 72 ? S Dec07 0:00 whois -h whois.abuse.net smtp5.clb.oleane.net.
erik 30466 0.0 0.0 6460 88 ? S Dec08 0:00 /usr/bin/perl ./siggen.pl
erik 30467 0.0 0.0 6460 88 ? S Dec08 0:00 /usr/bin/perl ./siggen.pl
erik 30468 0.0 0.0 6460 88 ? S Dec08 0:00 /usr/bin/perl ./siggen.pl
erik 30469 0.0 0.0 6460 88 ? S Dec08 0:00 /usr/bin/perl ./siggen.pl
erik 30470 0.0 0.0 6460 88 ? S Dec08 0:00 /usr/bin/perl ./siggen.pl
erik 5387 0.0 0.0 7176 88 ? S Dec09 0:00 /usr/bin/perl -w /home/erik/swentrap.pl
erik 5388 0.0 0.0 1644 72 ? S Dec09 0:00 whois -h whois.abuse.net outbound04.telus.net.
root 22934 0.0 0.2 5404 632 ? S Dec13 0:31 sendmail: accepting connections
mail 22938 0.0 0.1 4872 428 ? S Dec13 0:00 sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue
root 1532 0.0 0.1 2724 408 ? S Dec21 0:09 /usr/sbin/dhcpd eth1 eth2
named 3932 0.0 2.0 21292 5348 ? S Dec22 0:00 /usr/sbin/named -t /var/named -u named
named 3933 0.0 2.0 21292 5352 ? S Dec22 0:00 /usr/sbin/named -t /var/named -u named
named 3934 0.0 2.0 21292 5352 ? S Dec22 12:04 /usr/sbin/named -t /var/named -u named
named 3935 0.0 2.0 21292 5352 ? S Dec22 0:06 /usr/sbin/named -t /var/named -u named
named 3936 0.0 2.0 21292 5352 ? S Dec22 2:54 /usr/sbin/named -t /var/named -u named
ldap 24050 0.0 1.1 187432 3012 ? S Dec25 0:00 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 24051 0.0 1.1 187432 3012 ? S Dec25 0:00 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 24052 0.0 1.1 187432 3012 ? S Dec25 1:51 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 24083 0.0 1.1 187432 3012 ? S Dec25 0:29 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 24084 0.0 1.1 187432 3012 ? S Dec25 0:29 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 24085 0.0 1.1 187432 3012 ? S Dec25 0:28 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 24086 0.0 1.1 187432 3012 ? S Dec25 0:29 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 24087 0.0 1.1 187432 3012 ? S Dec25 0:29 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 24088 0.0 1.1 187432 3012 ? S Dec25 0:28 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
root 24562 0.0 0.0 95704 236 ? S Dec25 0:02 /usr/sbin/httpd -f /etc/httpd/httpd.conf
wwwrun 24564 0.0 0.4 96128 1140 ? S Dec25 0:00 /usr/sbin/httpd -f /etc/httpd/httpd.conf
wwwrun 24565 0.0 0.4 96132 1160 ? S Dec25 0:00 /usr/sbin/httpd -f /etc/httpd/httpd.conf
ldap 24568 0.0 1.1 187432 3012 ? S Dec25 0:30 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 24595 0.0 1.1 187432 3012 ? S Dec25 0:29 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 24596 0.0 1.1 187432 3012 ? S Dec25 0:28 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 24597 0.0 1.1 187432 3012 ? S Dec25 0:29 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 24598 0.0 1.1 187432 3012 ? S Dec25 0:28 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 24599 0.0 1.1 187432 3012 ? S Dec25 0:29 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 25146 0.0 1.1 187432 3012 ? S Dec25 0:28 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 25416 0.0 1.1 187432 3012 ? S Dec25 0:29 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 25565 0.0 1.1 187432 3012 ? S Dec25 0:29 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 25566 0.0 1.1 187432 3012 ? S Dec25 0:28 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 26104 0.0 1.1 187432 3012 ? S Dec25 0:29 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 26105 0.0 1.1 187432 3012 ? S Dec25 0:27 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 26106 0.0 1.1 187432 3012 ? S Dec25 0:28 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 26107 0.0 1.1 187432 3012 ? S Dec25 0:30 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 26108 0.0 1.1 187432 3012 ? S Dec25 0:29 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 26109 0.0 1.1 187432 3012 ? S Dec25 0:29 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 26503 0.0 1.1 187432 3012 ? S Dec25 0:28 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 26504 0.0 1.1 187432 3012 ? S Dec25 0:28 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 27006 0.0 1.1 187432 3012 ? S Dec25 0:28 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 27117 0.0 1.1 187432 3012 ? S Dec25 0:29 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 27120 0.0 1.1 187432 3012 ? S Dec25 0:28 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 27392 0.0 1.1 187432 3012 ? S Dec25 0:29 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 27393 0.0 1.1 187432 3012 ? S Dec25 0:29 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 27394 0.0 1.1 187432 3012 ? S Dec25 0:28 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 27395 0.0 1.1 187432 3012 ? S Dec25 0:29 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
ldap 27396 0.0 1.1 187432 3012 ? S Dec25 0:28 /usr/lib/openldap/slapd -h ldap:/// -u ldap -g ldap
news 4451 0.1 1.7 15324 4492 ? S Dec30 1:56 /usr/lib/news/bin/innd -p 3,4 -C
news 4452 0.0 0.0 4728 80 ? S Dec30 0:00 /bin/sh /usr/lib/news/bin/rc.news start
news 4456 0.0 0.2 2832 572 ? SN Dec30 0:15 /usr/lib/news/bin/innfeed
news 4457 0.0 0.1 4872 276 ? SN Dec30 0:00 /usr/bin/perl -w /usr/lib/news/bin/controlchan
news 4458 0.0 0.2 3620 536 ? SN Dec30 0:10 /usr/bin/perl -w /etc/news/bin/indexchan
news 4459 0.0 0.0 2476 76 ? SN Dec30 0:00 /bin/sh /etc/news/autocancel/startautocancel
news 4460 0.0 0.3 3620 788 ? SN Dec30 0:10 /usr/bin/perl -w /home/erik/src/cancelchan/cancelchan.pl
news 4463 0.0 0.5 6116 1456 ? SN Dec30 0:55 /etc/news/autocancel/autocancel
news 4514 0.2 0.2 4856 644 ? S Dec30 2:35 /bin/sh /usr/lib/news/bin/innwatch
root 30876 0.0 0.4 16676 1092 ? S 2004 0:01 /usr/sbin/nscd
root 30877 0.0 0.4 16676 1092 ? S 2004 0:00 /usr/sbin/nscd
root 30878 0.0 0.4 16676 1092 ? S 2004 0:00 /usr/sbin/nscd
root 30879 0.0 0.4 16676 1092 ? S 2004 0:00 /usr/sbin/nscd
root 30880 0.0 0.4 16676 1092 ? S 2004 0:00 /usr/sbin/nscd
root 30881 0.0 0.4 16676 1092 ? S 2004 0:00 /usr/sbin/nscd
root 30882 0.0 0.4 16676 1092 ? S 2004 0:00 /usr/sbin/nscd
root 5176 0.0 0.0 0 0 ? SW 2004 0:01 [pdflush]
root 26637 0.0 0.0 0 0 ? SW 2004 0:10 [pdflush]
news 17883 0.0 0.5 20544 1356 ? SN 2004 0:00 - nnrpd: bender.ipv6.hensema.net ARTICLE
root 17919 0.0 0.8 8324 2172 ? S 2004 0:00 /usr/sbin/sshd -o PidFile /var/run/sshd.init.pid
erik 17921 0.0 0.8 8336 2288 ? S 2004 0:00 /usr/sbin/sshd -o PidFile /var/run/sshd.init.pid
erik 17922 0.0 0.6 4792 1564 pts/0 S 2004 0:00 -bash
news 19089 0.0 0.2 4084 552 ? S 2004 0:00 sleep 120
erik 19102 0.0 0.2 2780 708 pts/0 R 2004 0:00 ps aux
nr_dirty 35
nr_writeback 0
nr_unstable 0
nr_page_table_pages 286
nr_mapped 6113
nr_slab 49479
pgpgin 70810333
pgpgout 164438264
pswpin 1046860
pswpout 591665
pgalloc 512977858
pgfree 512979155
pgactivate 16200881
pgdeactivate 14230409
pgfault 1132924331
pgmajfault 729065
pgscan 72463905
pgrefill 37275398
pgsteal 34042946
pginodesteal 0
kswapd_steal 33967753
kswapd_inodesteal 155251
pageoutrun 108132
allocstall 2656
pgrotated 1446342
MemTotal: 256012 kB
MemFree: 5088 kB
Buffers: 2988 kB
Cached: 16948 kB
SwapCached: 15332 kB
Active: 35976 kB
Inactive: 1916 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 256012 kB
LowFree: 5088 kB
SwapTotal: 500464 kB
SwapFree: 451456 kB
Dirty: 72 kB
Writeback: 0 kB
Mapped: 24444 kB
Slab: 197940 kB
Committed_AS: 379592 kB
PageTables: 1144 kB
VmallocTotal: 778168 kB
VmallocUsed: 4100 kB
VmallocChunk: 773940 kB
--
Erik Hensema <erik@hensema.net>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 2.6.0: something is leaking memory
2004-01-04 18:57 2.6.0: something is leaking memory Erik Hensema
@ 2004-01-04 21:00 ` Linus Torvalds
2004-01-04 21:31 ` Erik Hensema
2004-01-04 23:18 ` Mike Fedyk
1 sibling, 1 reply; 16+ messages in thread
From: Linus Torvalds @ 2004-01-04 21:00 UTC (permalink / raw)
To: Erik Hensema; +Cc: linux-kernel
On Sun, 4 Jan 2004, Erik Hensema wrote:
>
> The leak can be most easily seen in my rrdtool graphs of memory
> usage: http://dexter.hensema.net/~erik/stats/mem-mon.gif and
> http://dexter.hensema.net/~erik/stats/mem-year.gif - try to guess
> when I switched to 2.6.0-test11 ;-)
>
> At Dec 31 I upgraded to 2.6.0-final.
>
> Output from ps aux, /proc/vmstat and /proc/meminfo, are attached.
> My .config isn't compiled in and I haven't got it at hand
> currently. If anyone is interested I can post it tomorrow.
Can you do /proc/slabinfo too?
Clearly the memory leak isn't in the page cache, so the most likely source
is network buffers, and most likely in iptables connection tracking or
similar. If you actually _use_ IPv6, then that is also more likely to have
leaks just due to less testing.
David & co fixed a number of leaks just before 2.6.0-final, but that
doesn't mean that they are all gone - rather it means that there
definitely were still leaks around just a few weeks ago.
Linus
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 2.6.0: something is leaking memory
2004-01-04 21:00 ` Linus Torvalds
@ 2004-01-04 21:31 ` Erik Hensema
2004-01-04 21:39 ` Roland Dreier
` (3 more replies)
0 siblings, 4 replies; 16+ messages in thread
From: Erik Hensema @ 2004-01-04 21:31 UTC (permalink / raw)
To: linux-kernel
Linus Torvalds (torvalds@osdl.org) wrote:
>
>
> On Sun, 4 Jan 2004, Erik Hensema wrote:
>>
>> The leak can be most easily seen in my rrdtool graphs of memory
>> usage: http://dexter.hensema.net/~erik/stats/mem-mon.gif and
>> http://dexter.hensema.net/~erik/stats/mem-year.gif - try to guess
>> when I switched to 2.6.0-test11 ;-)
>>
>> At Dec 31 I upgraded to 2.6.0-final.
>>
>> Output from ps aux, /proc/vmstat and /proc/meminfo, are attached.
>> My .config isn't compiled in and I haven't got it at hand
>> currently. If anyone is interested I can post it tomorrow.
>
> Can you do /proc/slabinfo too?
Sure, this is of course my currently running system, 4 days, 9:53
uptime.
slabinfo - version: 2.0
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <batchcount> <limit> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
ip_conntrack 66 156 320 12 1 : tunables 54 27 0 : slabdata 13 13 0
ip_fib_hash 30 202 16 202 1 : tunables 120 60 0 : slabdata 1 1 0
rpc_buffers 8 8 2048 2 1 : tunables 24 12 0 : slabdata 4 4 0
rpc_tasks 8 20 192 20 1 : tunables 120 60 0 : slabdata 1 1 0
rpc_inode_cache 0 0 384 10 1 : tunables 54 27 0 : slabdata 0 0 0
fib6_nodes 42 112 32 112 1 : tunables 120 60 0 : slabdata 1 1 0
ip6_dst_cache 38 60 256 15 1 : tunables 120 60 0 : slabdata 4 4 0
ndisc_cache 10 20 192 20 1 : tunables 120 60 0 : slabdata 1 1 0
raw6_sock 0 0 640 6 1 : tunables 54 27 0 : slabdata 0 0 0
udp6_sock 2 6 640 6 1 : tunables 54 27 0 : slabdata 1 1 0
tcp6_sock 19729 19732 1024 4 1 : tunables 54 27 0 : slabdata 4933 4933 0
unix_sock 42 50 384 10 1 : tunables 54 27 0 : slabdata 5 5 0
tcp_tw_bucket 5 30 128 30 1 : tunables 120 60 0 : slabdata 1 1 0
tcp_bind_bucket 107 202 16 202 1 : tunables 120 60 0 : slabdata 1 1 0
tcp_open_request 0 0 128 30 1 : tunables 120 60 0 : slabdata 0 0 0
inet_peer_cache 10 59 64 59 1 : tunables 120 60 0 : slabdata 1 1 0
secpath_cache 0 0 128 30 1 : tunables 120 60 0 : slabdata 0 0 0
xfrm_dst_cache 0 0 256 15 1 : tunables 120 60 0 : slabdata 0 0 0
ip_dst_cache 259 300 256 15 1 : tunables 120 60 0 : slabdata 20 20 0
arp_cache 4 30 128 30 1 : tunables 120 60 0 : slabdata 1 1 0
raw4_sock 0 0 512 7 1 : tunables 54 27 0 : slabdata 0 0 0
udp_sock 14 14 512 7 1 : tunables 54 27 0 : slabdata 2 2 0
tcp_sock 35 36 896 4 1 : tunables 54 27 0 : slabdata 9 9 0
flow_cache 0 0 128 30 1 : tunables 120 60 0 : slabdata 0 0 0
dm_io 784 1010 16 202 1 : tunables 120 60 0 : slabdata 5 5 0
reiser_inode_cache 3090 11050 384 10 1 : tunables 54 27 0 : slabdata 1105 1105 0
nfs_write_data 36 36 448 9 1 : tunables 54 27 0 : slabdata 4 4 0
nfs_read_data 32 36 448 9 1 : tunables 54 27 0 : slabdata 4 4 0
nfs_inode_cache 0 0 640 6 1 : tunables 54 27 0 : slabdata 0 0 0
nfs_page 0 0 128 30 1 : tunables 120 60 0 : slabdata 0 0 0
ext2_inode_cache 1 9 448 9 1 : tunables 54 27 0 : slabdata 1 1 0
eventpoll_pwq 0 0 36 101 1 : tunables 120 60 0 : slabdata 0 0 0
eventpoll_epi 0 0 128 30 1 : tunables 120 60 0 : slabdata 0 0 0
kioctx 0 0 192 20 1 : tunables 120 60 0 : slabdata 0 0 0
kiocb 0 0 192 20 1 : tunables 120 60 0 : slabdata 0 0 0
dnotify_cache 0 0 20 169 1 : tunables 120 60 0 : slabdata 0 0 0
file_lock_cache 9 42 92 42 1 : tunables 120 60 0 : slabdata 1 1 0
fasync_cache 0 0 16 202 1 : tunables 120 60 0 : slabdata 0 0 0
shmem_inode_cache 8 9 448 9 1 : tunables 54 27 0 : slabdata 1 1 0
idr_layer_cache 0 0 136 28 1 : tunables 120 60 0 : slabdata 0 0 0
posix_timers_cache 0 0 80 48 1 : tunables 120 60 0 : slabdata 0 0 0
uid_cache 10 112 32 112 1 : tunables 120 60 0 : slabdata 1 1 0
deadline_drq 0 0 52 72 1 : tunables 120 60 0 : slabdata 0 0 0
as_arq 21 118 64 59 1 : tunables 120 60 0 : slabdata 2 2 0
blkdev_requests 21 72 160 24 1 : tunables 120 60 0 : slabdata 3 3 0
biovec-BIO_MAX_PAGES 256 260 3072 5 4 : tunables 24 12 0 : slabdata 52 52 0
biovec-128 256 260 1536 5 2 : tunables 24 12 0 : slabdata 52 52 0
biovec-64 256 260 768 5 1 : tunables 54 27 0 : slabdata 52 52 0
biovec-16 256 260 192 20 1 : tunables 120 60 0 : slabdata 13 13 0
biovec-4 258 295 64 59 1 : tunables 120 60 0 : slabdata 5 5 0
biovec-1 300 606 16 202 1 : tunables 120 60 0 : slabdata 3 3 0
bio 273 413 64 59 1 : tunables 120 60 0 : slabdata 7 7 0
sock_inode_cache 208 230 384 10 1 : tunables 54 27 0 : slabdata 23 23 0
skbuff_head_cache 300 360 192 20 1 : tunables 120 60 0 : slabdata 18 18 0
sock 7 12 320 12 1 : tunables 54 27 0 : slabdata 1 1 0
proc_inode_cache 980 980 384 10 1 : tunables 54 27 0 : slabdata 98 98 0
sigqueue 16 27 144 27 1 : tunables 120 60 0 : slabdata 1 1 0
radix_tree_node 2680 2925 260 15 1 : tunables 54 27 0 : slabdata 195 195 0
bdev_cache 9 9 448 9 1 : tunables 54 27 0 : slabdata 1 1 0
mnt_cache 17 59 64 59 1 : tunables 120 60 0 : slabdata 1 1 0
inode_cache 917 930 384 10 1 : tunables 54 27 0 : slabdata 93 93 0
dentry_cache 7348 20240 192 20 1 : tunables 120 60 0 : slabdata 1012 1012 0
filp 1048 1290 128 30 1 : tunables 120 60 0 : slabdata 43 43 0
names_cache 6 6 4096 1 1 : tunables 24 12 0 : slabdata 6 6 0
buffer_head 28856 35064 52 72 1 : tunables 120 60 0 : slabdata 487 487 0
mm_struct 58 70 512 7 1 : tunables 54 27 0 : slabdata 10 10 0
vm_area_struct 2337 2832 64 59 1 : tunables 120 60 0 : slabdata 48 48 0
fs_cache 66 112 32 112 1 : tunables 120 60 0 : slabdata 1 1 0
files_cache 58 72 448 9 1 : tunables 54 27 0 : slabdata 8 8 0
signal_cache 124 177 64 59 1 : tunables 120 60 0 : slabdata 3 3 0
sighand_cache 73 81 1344 3 1 : tunables 24 12 0 : slabdata 27 27 0
task_struct 129 140 1568 5 2 : tunables 24 12 0 : slabdata 28 28 0
pte_chain 9102 10679 64 59 1 : tunables 120 60 0 : slabdata 181 181 0
pgd 58 58 4096 1 1 : tunables 24 12 0 : slabdata 58 58 0
size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0
size-131072 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0
size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0 : slabdata 0 0 0
size-65536 0 0 65536 1 16 : tunables 8 4 0 : slabdata 0 0 0
size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0 : slabdata 0 0 0
size-32768 0 0 32768 1 8 : tunables 8 4 0 : slabdata 0 0 0
size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0 : slabdata 0 0 0
size-16384 0 0 16384 1 4 : tunables 8 4 0 : slabdata 0 0 0
size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0 : slabdata 0 0 0
size-8192 131 131 8192 1 2 : tunables 8 4 0 : slabdata 131 131 0
size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 0 : slabdata 0 0 0
size-4096 120 120 4096 1 1 : tunables 24 12 0 : slabdata 120 120 0
size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 0 : slabdata 0 0 0
size-2048 160 174 2048 2 1 : tunables 24 12 0 : slabdata 87 87 0
size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 0 : slabdata 0 0 0
size-1024 109 128 1024 4 1 : tunables 54 27 0 : slabdata 32 32 0
size-512(DMA) 0 0 512 8 1 : tunables 54 27 0 : slabdata 0 0 0
size-512 288 288 512 8 1 : tunables 54 27 0 : slabdata 36 36 0
size-256(DMA) 0 0 256 15 1 : tunables 120 60 0 : slabdata 0 0 0
size-256 394 660 256 15 1 : tunables 120 60 0 : slabdata 44 44 0
size-192(DMA) 0 0 192 20 1 : tunables 120 60 0 : slabdata 0 0 0
size-192 32 40 192 20 1 : tunables 120 60 0 : slabdata 2 2 0
size-128(DMA) 0 0 128 30 1 : tunables 120 60 0 : slabdata 0 0 0
size-128 2675 2730 128 30 1 : tunables 120 60 0 : slabdata 91 91 0
size-64(DMA) 0 0 64 59 1 : tunables 120 60 0 : slabdata 0 0 0
size-64 5724 5782 64 59 1 : tunables 120 60 0 : slabdata 98 98 0
size-32(DMA) 0 0 32 112 1 : tunables 120 60 0 : slabdata 0 0 0
size-32 820 896 32 112 1 : tunables 120 60 0 : slabdata 8 8 0
kmem_cache 132 132 116 33 1 : tunables 120 60 0 : slabdata 4 4 0
> Clearly the memory leak isn't in the page cache, so the most likely source
> is network buffers, and most likely in iptables connection tracking or
> similar. If you actually _use_ IPv6, then that is also more likely to have
> leaks just due to less testing.
I do use IPv6. I've got three active tunnels and native IPv6 over
ethernet.
I've always had problems with nscd leaking filedescriptors, all
IPv6 connections to my LDAP server. This started after upgrading
suse 8.0 to 8.2 (I think the problem is in nss_ldap).
I'm restarting nscd using a cronjob every night now. Output of
netstat --inet6 -avpn is below. All sockets in CLOSE_WAIT are
leaked and will go away after a nscd restart.
> David & co fixed a number of leaks just before 2.6.0-final, but that
> doesn't mean that they are all gone - rather it means that there
> definitely were still leaks around just a few weeks ago.
The server isn't very critical, but I do need it. I'm willing to
try some patches (or do an upgrade to -mm), but nothing to wild.
netstat --inet6 -avpn
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 :::22 :::* LISTEN 1208/sshd
tcp 0 0 :::119 :::* LISTEN 1364/innd
tcp 0 0 :::25 :::* LISTEN 1433/sendmail: acce
tcp 0 0 :::953 :::* LISTEN 1175/named
tcp 0 0 ::1:6010 :::* LISTEN 19900/sshd
tcp 0 0 ::1:6011 :::* LISTEN 20150/sshd
tcp 1 0 ::1:50565 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:50224 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 0 0 2001:888:10a1::1:389 ::1:55936 ESTABLISHED 1145/slapd
tcp 1 0 ::1:50343 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:50988 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:51181 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:51187 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:51186 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:50768 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:50774 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:50772 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:50788 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:50816 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:49454 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:49460 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:49458 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:49470 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:49619 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:49645 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:49644 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:49643 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:49592 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:32776 2001:888:10a1::1:389 CLOSE_WAIT 1510/httpd
tcp 1 0 ::1:32806 2001:888:10a1::1:389 CLOSE_WAIT 1777/SCREEN
tcp 0 0 192.168.1.1:119 192.168.1.1:56197 ESTABLISHED 20135/- nnrpd: dext
tcp 1 0 ::1:32862 2001:888:10a1::1:389 CLOSE_WAIT 2726/httpd
tcp 1 0 ::1:49291 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 0 0 2001:888:10a1::1:389 ::1:55527 ESTABLISHED 1145/slapd
tcp 0 0 212.120.97.185:119 62.212.82.150:58902 ESTABLISHED 1364/innd
tcp 1 0 ::1:52711 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:52263 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 0 0 2001:888:10a1::1:389 ::1:56203 ESTABLISHED 1145/slapd
tcp 1 0 ::1:52823 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:52839 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:52876 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:52888 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:51563 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:51460 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:51201 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:51420 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:51423 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:51417 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 0 0 2001:888:10a1::1:389 ::1:56204 ESTABLISHED 1145/slapd
tcp 1 0 ::1:52182 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 0 0 2001:888:10a1::1:389 ::1:56207 ESTABLISHED 1145/slapd
tcp 0 0 2001:888:10a1::1:389 ::1:56206 ESTABLISHED 1145/slapd
tcp 1 0 ::1:54612 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:54628 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:54366 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:54389 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:54370 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:54372 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 0 0 2001:888:10a1::1:389 ::1:56179 ESTABLISHED 1145/slapd
tcp 1 0 ::1:47004 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:47069 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 0 48 212.120.97.185:22 213.84.242.133:55760 ESTABLISHED 20146/sshd
tcp 1 0 ::1:53575 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:53571 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:53577 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:53722 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:53661 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:53695 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:53347 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:53447 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:54103 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:54087 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:54088 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:54127 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:54047 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 0 0 212.120.97.185:22 213.84.242.133:55733 ESTABLISHED 19807/sshd
tcp 1 0 ::1:53828 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:53867 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:53809 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:53974 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:53948 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:48465 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:48619 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:48233 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:48381 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:48658 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 1 0 ::1:48878 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 0 0 2001:888:10a1::1:389 ::1:56250 ESTABLISHED 1145/slapd
tcp 1 0 ::1:47368 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 0 0 2001:888:10a1::1:389 ::1:56188 ESTABLISHED 1145/slapd
tcp 0 0 ::1:55527 2001:888:10a1::1:389 ESTABLISHED 26536/nscd
tcp 0 0 212.120.97.185:119 131.211.28.48:47677 ESTABLISHED 1364/innd
tcp 0 0 ::1:56190 2001:888:10a1::1:389 ESTABLISHED 19900/sshd
tcp 0 0 ::1:56188 2001:888:10a1::1:389 ESTABLISHED 19807/sshd
tcp 0 0 ::1:56179 2001:888:10a1::1:389 ESTABLISHED 19807/sshd
tcp 0 0 ::1:56207 2001:888:10a1::1:389 ESTABLISHED 26536/nscd
tcp 0 0 ::1:56206 2001:888:10a1::1:389 ESTABLISHED 20150/sshd
tcp 0 0 ::1:56204 2001:888:10a1::1:389 ESTABLISHED 20146/sshd
tcp 0 0 ::1:56203 2001:888:10a1::1:389 ESTABLISHED 20146/sshd
tcp 0 0 ::1:56250 2001:888:10a1::1:389 ESTABLISHED 20670/su
tcp 0 0 ::1:56249 2001:888:10a1::1:389 TIME_WAIT -
tcp 0 0 ::1:56248 2001:888:10a1::1:389 TIME_WAIT -
tcp 0 0 2001:888:10a1::1:389 ::1:56190 ESTABLISHED 1145/slapd
tcp 1 0 ::1:47765 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
tcp 0 0 ::1:55936 2001:888:10a1::1:389 ESTABLISHED 26536/nscd
udp 0 0 :::514 :::* 688/syslogd
udp 0 0 :::32772 :::* 1175/named
udp 0 0 :::53 :::* 1175/named
raw 0 0 :::58 :::* 7 916/zebra
--
Erik Hensema <erik@hensema.net>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 2.6.0: something is leaking memory
2004-01-04 21:31 ` Erik Hensema
@ 2004-01-04 21:39 ` Roland Dreier
2004-01-05 16:18 ` Ricky Beam
2004-01-05 2:30 ` Linus Torvalds
` (2 subsequent siblings)
3 siblings, 1 reply; 16+ messages in thread
From: Roland Dreier @ 2004-01-04 21:39 UTC (permalink / raw)
To: erik; +Cc: linux-kernel
Yup, looks like IPv6 is leaking memory (since your netstat shows
nowhere near 19K sockets):
> tcp6_sock 19729 19732 1024 4 1 : tunables 54 27 0 : slabdata 4933 4933 0
Now to figure out why...
- R.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 2.6.0: something is leaking memory
2004-01-04 18:57 2.6.0: something is leaking memory Erik Hensema
2004-01-04 21:00 ` Linus Torvalds
@ 2004-01-04 23:18 ` Mike Fedyk
1 sibling, 0 replies; 16+ messages in thread
From: Mike Fedyk @ 2004-01-04 23:18 UTC (permalink / raw)
To: Erik Hensema; +Cc: linux-kernel
On Sun, Jan 04, 2004 at 06:57:59PM +0000, Erik Hensema wrote:
> The leak can be most easily seen in my rrdtool graphs of memory
> usage: http://dexter.hensema.net/~erik/stats/mem-mon.gif and
> http://dexter.hensema.net/~erik/stats/mem-year.gif - try to guess
> when I switched to 2.6.0-test11 ;-)
I saw something similair on my lrrd graphs.
http://www.matchmail.com/stats/lrrd/matchmail.com/mis-mike-wstn.matchmail.com-memory.html
But it also happens on 2.4.23, and it was a bug in rxvt which I won't be
able to try to reproduce until I get back to work tomorrow.
Mike
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 2.6.0: something is leaking memory
2004-01-04 21:31 ` Erik Hensema
2004-01-04 21:39 ` Roland Dreier
@ 2004-01-05 2:30 ` Linus Torvalds
2004-01-05 4:48 ` David S. Miller
2004-01-05 4:52 ` (usagi-core 16947) " YOSHIFUJI Hideaki / 吉藤英明
2004-03-29 15:43 ` YOSHIFUJI Hideaki / 吉藤英明
3 siblings, 1 reply; 16+ messages in thread
From: Linus Torvalds @ 2004-01-05 2:30 UTC (permalink / raw)
To: Erik Hensema; +Cc: netdev, David S. Miller
On Sun, 4 Jan 2004, Erik Hensema wrote:
> Linus Torvalds (torvalds@osdl.org) wrote:
> >
> > Can you do /proc/slabinfo too?
>
> Sure, this is of course my currently running system, 4 days, 9:53
> uptime.
>
> slabinfo - version: 2.0
> # name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <batchcount> <limit> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
> tcp6_sock 19729 19732 1024 4 1 : tunables 54 27 0 : slabdata 4933 4933 0
You've got 19 _megabytes_ allocated to "tcp6_sock", and they are all
marked as "active". That's almost certainly the leaking bug.
Everything else looks reasonably normal.
> I do use IPv6. I've got three active tunnels and native IPv6 over
> ethernet.
Yeah, but there is no way you have 19 MB worth of sockets active for three
tunnels.
David?
Linus
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 2.6.0: something is leaking memory
2004-01-05 2:30 ` Linus Torvalds
@ 2004-01-05 4:48 ` David S. Miller
2004-01-05 18:14 ` Erik Hensema
2004-01-06 15:59 ` Erik Hensema
0 siblings, 2 replies; 16+ messages in thread
From: David S. Miller @ 2004-01-05 4:48 UTC (permalink / raw)
To: Linus Torvalds; +Cc: erik, netdev
On Sun, 4 Jan 2004 18:30:21 -0800 (PST)
Linus Torvalds <torvalds@osdl.org> wrote:
> You've got 19 _megabytes_ allocated to "tcp6_sock", and they are all
> marked as "active". That's almost certainly the leaking bug.
>
> Everything else looks reasonably normal.
...
> David?
Fixed by changeset 1.1496.16.1 which is in 2.6.1-rc1
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: (usagi-core 16947) Re: 2.6.0: something is leaking memory
2004-01-04 21:31 ` Erik Hensema
2004-01-04 21:39 ` Roland Dreier
2004-01-05 2:30 ` Linus Torvalds
@ 2004-01-05 4:52 ` YOSHIFUJI Hideaki / 吉藤英明
2004-03-29 15:43 ` YOSHIFUJI Hideaki / 吉藤英明
3 siblings, 0 replies; 16+ messages in thread
From: YOSHIFUJI Hideaki / 吉藤英明 @ 2004-01-05 4:52 UTC (permalink / raw)
To: erik; +Cc: linux-kernel, netdev
In article <slrnbvh1hd.jl6.erik@dexter.hensema.net> (at Sun, 4 Jan 2004 21:31:26 +0000 (UTC)), Erik Hensema <erik@hensema.net> says:
> > Can you do /proc/slabinfo too?
>
> Sure, this is of course my currently running system, 4 days, 9:53
> uptime.
>
> slabinfo - version: 2.0
> # name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <batchcount> <limit> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
:
> tcp6_sock 19729 19732 1024 4 1 : tunables 54 27 0 : slabdata 4933 4933 0
:
> > Clearly the memory leak isn't in the page cache, so the most likely source
> > is network buffers, and most likely in iptables connection tracking or
> > similar. If you actually _use_ IPv6, then that is also more likely to have
> > leaks just due to less testing.
>
> I do use IPv6. I've got three active tunnels and native IPv6 over
> ethernet.
>
> I've always had problems with nscd leaking filedescriptors, all
> IPv6 connections to my LDAP server. This started after upgrading
> suse 8.0 to 8.2 (I think the problem is in nss_ldap).
> I'm restarting nscd using a cronjob every night now. Output of
> netstat --inet6 -avpn is below. All sockets in CLOSE_WAIT are
> leaked and will go away after a nscd restart.
How about /proc/slabinfo just after restarting nss_ldap?
> The server isn't very critical, but I do need it. I'm willing to
> try some patches (or do an upgrade to -mm), but nothing to wild.
>
> netstat --inet6 -avpn
>
> Active Internet connections (servers and established)
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
> tcp 0 0 :::22 :::* LISTEN 1208/sshd
> tcp 0 0 :::119 :::* LISTEN 1364/innd
> tcp 0 0 :::25 :::* LISTEN 1433/sendmail: acce
> tcp 0 0 :::953 :::* LISTEN 1175/named
> tcp 0 0 ::1:6010 :::* LISTEN 19900/sshd
> tcp 0 0 ::1:6011 :::* LISTEN 20150/sshd
> tcp 1 0 ::1:50565 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
> tcp 1 0 ::1:50224 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
> tcp 0 0 2001:888:10a1::1:389 ::1:55936 ESTABLISHED 1145/slapd
> tcp 1 0 ::1:50343 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
> tcp 1 0 ::1:50988 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
:
There're too many sockets in CLOSE_WAIT, but the number is very different from
"tcp6_sock."
And, what is happened when you use ipv4 in your nscd?
--
Hideaki YOSHIFUJI @ USAGI Project <yoshfuji@linux-ipv6.org>
GPG FP: 9022 65EB 1ECF 3AD1 0BDF 80D8 4807 F894 E062 0EEA
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 2.6.0: something is leaking memory
2004-01-04 21:39 ` Roland Dreier
@ 2004-01-05 16:18 ` Ricky Beam
2004-01-05 18:16 ` Erik Hensema
0 siblings, 1 reply; 16+ messages in thread
From: Ricky Beam @ 2004-01-05 16:18 UTC (permalink / raw)
To: Roland Dreier; +Cc: erik, linux-kernel
On 4 Jan 2004, Roland Dreier wrote:
>Yup, looks like IPv6 is leaking memory (since your netstat shows
>nowhere near 19K sockets):
>
> > tcp6_sock 19729 19732 1024 4 1 : tunables 54 27 0 : slabdata 4933 4933 0
>
>Now to figure out why...
Of course, there are a lot of sockets in CLOSE_WAIT. That's where I'd start
looking... if I actually cared about IPv6 :-) There used to be the same
sort of problem for IPv4 (sockets would never go away), but no one ever fixed
it -- a rewrite of the network stack made it go away.
--Ricky
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 2.6.0: something is leaking memory
2004-01-05 4:48 ` David S. Miller
@ 2004-01-05 18:14 ` Erik Hensema
2004-01-06 15:59 ` Erik Hensema
1 sibling, 0 replies; 16+ messages in thread
From: Erik Hensema @ 2004-01-05 18:14 UTC (permalink / raw)
To: David S. Miller; +Cc: Linus Torvalds, netdev
On Sun, Jan 04, 2004 at 08:48:34PM -0800, David S. Miller wrote:
> On Sun, 4 Jan 2004 18:30:21 -0800 (PST)
> Linus Torvalds <torvalds@osdl.org> wrote:
>
> > You've got 19 _megabytes_ allocated to "tcp6_sock", and they are all
> > marked as "active". That's almost certainly the leaking bug.
> >
> > Everything else looks reasonably normal.
> ...
> > David?
>
> Fixed by changeset 1.1496.16.1 which is in 2.6.1-rc1
Are you sure?
tcp6_sock 1110 1136 1024 4 1 : tunables 54 27 0
: slabdata 284 284 0
dexter:~ # netstat --inet6 | wc -l
21
dexter:~ # uptime
19:13:22 up 4:36, 1 user, load average: 0.02, 0.08, 0.09
This is 2.6.1-rc1.
I can't say anything definitive until after a few days of uptime, but it
sure seems the leak is still there.
--
Erik Hensema (erik@hensema.net)
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 2.6.0: something is leaking memory
2004-01-05 16:18 ` Ricky Beam
@ 2004-01-05 18:16 ` Erik Hensema
0 siblings, 0 replies; 16+ messages in thread
From: Erik Hensema @ 2004-01-05 18:16 UTC (permalink / raw)
To: Ricky Beam; +Cc: Roland Dreier, linux-kernel
On Mon, Jan 05, 2004 at 11:18:52AM -0500, Ricky Beam wrote:
> On 4 Jan 2004, Roland Dreier wrote:
> >Yup, looks like IPv6 is leaking memory (since your netstat shows
> >nowhere near 19K sockets):
> >
> > > tcp6_sock 19729 19732 1024 4 1 : tunables 54 27 0 : slabdata 4933 4933 0
> >
> >Now to figure out why...
>
> Of course, there are a lot of sockets in CLOSE_WAIT. That's where I'd
> start looking... if I actually cared about IPv6 :-) There used to be the
> same sort of problem for IPv4 (sockets would never go away), but no one
> ever fixed it -- a rewrite of the network stack made it go away.
Those sockets in CLOSE_WAIT state are due to a filedescriptor leak in nscd
(or nss_ldap), which is a userspace problem.
It does however make the leak in kernelspace more visible it seems.
--
Erik Hensema (erik@hensema.net)
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 2.6.0: something is leaking memory
2004-01-05 4:48 ` David S. Miller
2004-01-05 18:14 ` Erik Hensema
@ 2004-01-06 15:59 ` Erik Hensema
2004-01-06 17:59 ` David S. Miller
1 sibling, 1 reply; 16+ messages in thread
From: Erik Hensema @ 2004-01-06 15:59 UTC (permalink / raw)
To: David S. Miller; +Cc: Linus Torvalds, netdev
On Sun, Jan 04, 2004 at 08:48:34PM -0800, David S. Miller wrote:
> On Sun, 4 Jan 2004 18:30:21 -0800 (PST)
> Linus Torvalds <torvalds@osdl.org> wrote:
>
> > You've got 19 _megabytes_ allocated to "tcp6_sock", and they are all
> > marked as "active". That's almost certainly the leaking bug.
> >
> > Everything else looks reasonably normal.
> ...
> > David?
>
> Fixed by changeset 1.1496.16.1 which is in 2.6.1-rc1
The leak seems to be in 2.6.1-rc1 too and I can't find any IPv6 related
fixes wrt memory in the long format changelog of 2.6.1-rc1.
David: are you sure it was fixed in rc1?
It doesn't seem to be in -rc2 either.
This is after 26 hours uptime:
tcp6_sock 6246 6248 1024 4 1 : tunables 54 27 0
: slabdata 1562 1562 0
--
Erik Hensema (erik@hensema.net)
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 2.6.0: something is leaking memory
2004-01-06 15:59 ` Erik Hensema
@ 2004-01-06 17:59 ` David S. Miller
2004-01-06 19:03 ` Arnaldo Carvalho de Melo
2004-01-06 19:36 ` Erik Hensema
0 siblings, 2 replies; 16+ messages in thread
From: David S. Miller @ 2004-01-06 17:59 UTC (permalink / raw)
To: erik; +Cc: torvalds, netdev, acme
On Tue, 6 Jan 2004 16:59:34 +0100
Erik Hensema <erik@hensema.net> wrote:
> David: are you sure it was fixed in rc1?
>
> It doesn't seem to be in -rc2 either.
>
> This is after 26 hours uptime:
>
> tcp6_sock 6246 6248 1024 4 1 : tunables 54 27 0
> : slabdata 1562 1562 0
Someone mentioned about a bug in the userland program you're
using that is openning these sockets? Something about leaving sockets
not closed.
(Arnaldo, we aparently still have a TCP ipv6 socket leak...)
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 2.6.0: something is leaking memory
2004-01-06 17:59 ` David S. Miller
@ 2004-01-06 19:03 ` Arnaldo Carvalho de Melo
2004-01-06 19:36 ` Erik Hensema
1 sibling, 0 replies; 16+ messages in thread
From: Arnaldo Carvalho de Melo @ 2004-01-06 19:03 UTC (permalink / raw)
To: David S. Miller; +Cc: erik, torvalds, netdev
Em Tue, Jan 06, 2004 at 09:59:09AM -0800, David S. Miller escreveu:
> On Tue, 6 Jan 2004 16:59:34 +0100
> Erik Hensema <erik@hensema.net> wrote:
>
> > David: are you sure it was fixed in rc1?
> >
> > It doesn't seem to be in -rc2 either.
> >
> > This is after 26 hours uptime:
> >
> > tcp6_sock 6246 6248 1024 4 1 : tunables 54 27 0
> > : slabdata 1562 1562 0
>
> Someone mentioned about a bug in the userland program you're
> using that is openning these sockets? Something about leaving sockets
> not closed.
>
> (Arnaldo, we aparently still have a TCP ipv6 socket leak...)
I'll take a look at this.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: 2.6.0: something is leaking memory
2004-01-06 17:59 ` David S. Miller
2004-01-06 19:03 ` Arnaldo Carvalho de Melo
@ 2004-01-06 19:36 ` Erik Hensema
1 sibling, 0 replies; 16+ messages in thread
From: Erik Hensema @ 2004-01-06 19:36 UTC (permalink / raw)
To: David S. Miller; +Cc: torvalds, netdev, acme
On Tue, Jan 06, 2004 at 09:59:09AM -0800, David S. Miller wrote:
> On Tue, 6 Jan 2004 16:59:34 +0100
> Erik Hensema <erik@hensema.net> wrote:
>
> > David: are you sure it was fixed in rc1?
> >
> > It doesn't seem to be in -rc2 either.
> >
> > This is after 26 hours uptime:
> >
> > tcp6_sock 6246 6248 1024 4 1 : tunables 54 27 0
> > : slabdata 1562 1562 0
>
> Someone mentioned about a bug in the userland program you're
> using that is openning these sockets? Something about leaving sockets
> not closed.
That's correct. I suspect that this exposes the leak in the kernel more
promimently than on other systems.
The leak is in nscd, it doesn't properly close sockets to my LDAP server.
This is not a kernel problem I think, because it also leaks in 2.4.x. The
kernelspace leak however is not in 2.4, only in 2.6.
I'm restarting nscd in cron every night, which makes the CLOSE_WAIT sockets
go away. However the kernel resources are not freed it seems.
--
Erik Hensema (erik@hensema.net)
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: (usagi-core 16947) Re: 2.6.0: something is leaking memory
2004-01-04 21:31 ` Erik Hensema
` (2 preceding siblings ...)
2004-01-05 4:52 ` (usagi-core 16947) " YOSHIFUJI Hideaki / 吉藤英明
@ 2004-03-29 15:43 ` YOSHIFUJI Hideaki / 吉藤英明
3 siblings, 0 replies; 16+ messages in thread
From: YOSHIFUJI Hideaki / 吉藤英明 @ 2004-03-29 15:43 UTC (permalink / raw)
To: Administrator; +Cc: linux-kernel, netdev
In article <slrnbvh1hd.jl6.erik@dexter.hensema.net> (at Sun, 4 Jan 2004 21:31:26 +0000 (UTC)), Erik Hensema <erik@hensema.net> says:
> > Can you do /proc/slabinfo too?
>
> Sure, this is of course my currently running system, 4 days, 9:53
> uptime.
>
> slabinfo - version: 2.0
> # name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <batchcount> <limit> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
:
> tcp6_sock 19729 19732 1024 4 1 : tunables 54 27 0 : slabdata 4933 4933 0
:
> > Clearly the memory leak isn't in the page cache, so the most likely source
> > is network buffers, and most likely in iptables connection tracking or
> > similar. If you actually _use_ IPv6, then that is also more likely to have
> > leaks just due to less testing.
>
> I do use IPv6. I've got three active tunnels and native IPv6 over
> ethernet.
>
> I've always had problems with nscd leaking filedescriptors, all
> IPv6 connections to my LDAP server. This started after upgrading
> suse 8.0 to 8.2 (I think the problem is in nss_ldap).
> I'm restarting nscd using a cronjob every night now. Output of
> netstat --inet6 -avpn is below. All sockets in CLOSE_WAIT are
> leaked and will go away after a nscd restart.
How about /proc/slabinfo just after restarting nss_ldap?
> The server isn't very critical, but I do need it. I'm willing to
> try some patches (or do an upgrade to -mm), but nothing to wild.
>
> netstat --inet6 -avpn
>
> Active Internet connections (servers and established)
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
> tcp 0 0 :::22 :::* LISTEN 1208/sshd
> tcp 0 0 :::119 :::* LISTEN 1364/innd
> tcp 0 0 :::25 :::* LISTEN 1433/sendmail: acce
> tcp 0 0 :::953 :::* LISTEN 1175/named
> tcp 0 0 ::1:6010 :::* LISTEN 19900/sshd
> tcp 0 0 ::1:6011 :::* LISTEN 20150/sshd
> tcp 1 0 ::1:50565 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
> tcp 1 0 ::1:50224 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
> tcp 0 0 2001:888:10a1::1:389 ::1:55936 ESTABLISHED 1145/slapd
> tcp 1 0 ::1:50343 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
> tcp 1 0 ::1:50988 2001:888:10a1::1:389 CLOSE_WAIT 26536/nscd
:
There're too many sockets in CLOSE_WAIT, but the number is very different from
"tcp6_sock."
And, what is happened when you use ipv4 in your nscd?
--
Hideaki YOSHIFUJI @ USAGI Project <yoshfuji@linux-ipv6.org>
GPG FP: 9022 65EB 1ECF 3AD1 0BDF 80D8 4807 F894 E062 0EEA
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2004-03-29 15:43 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2004-01-04 18:57 2.6.0: something is leaking memory Erik Hensema
2004-01-04 21:00 ` Linus Torvalds
2004-01-04 21:31 ` Erik Hensema
2004-01-04 21:39 ` Roland Dreier
2004-01-05 16:18 ` Ricky Beam
2004-01-05 18:16 ` Erik Hensema
2004-01-05 2:30 ` Linus Torvalds
2004-01-05 4:48 ` David S. Miller
2004-01-05 18:14 ` Erik Hensema
2004-01-06 15:59 ` Erik Hensema
2004-01-06 17:59 ` David S. Miller
2004-01-06 19:03 ` Arnaldo Carvalho de Melo
2004-01-06 19:36 ` Erik Hensema
2004-01-05 4:52 ` (usagi-core 16947) " YOSHIFUJI Hideaki / 吉藤英明
2004-03-29 15:43 ` YOSHIFUJI Hideaki / 吉藤英明
2004-01-04 23:18 ` Mike Fedyk
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.