* Huge memory leak with 4.15.0-rc2+
@ 2017-12-11 21:23 Paweł Staszewski
2017-12-11 21:48 ` Paweł Staszewski
0 siblings, 1 reply; 6+ messages in thread
From: Paweł Staszewski @ 2017-12-11 21:23 UTC (permalink / raw)
To: Linux Kernel Network Developers
Hi
I just upgraded some testing host to 4.15.0-rc2+ kernel
And after some time of traffic processing - when traffic on all ports
reach about 3Mpps - memleak started.
Graph attached from memory usage: https://ibb.co/idK4zb
HW config:
Intel E5
8x Intel 82599 (used ixgbe driver from kernel)
Interfaces with vlans attached
All 8 ethernet ports are in one LAG group configured by team.
With current settings
(this host is acting as a router - and bgpd process is eating same
amount of memory from the beginning about 5.2GB)
cat /proc/meminfo
MemTotal: 32770588 kB
MemFree: 11342492 kB
MemAvailable: 10982752 kB
Buffers: 84704 kB
Cached: 83180 kB
SwapCached: 0 kB
Active: 5105320 kB
Inactive: 46252 kB
Active(anon): 4985448 kB
Inactive(anon): 1096 kB
Active(file): 119872 kB
Inactive(file): 45156 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4005280 kB
SwapFree: 4005280 kB
Dirty: 236 kB
Writeback: 0 kB
AnonPages: 4983752 kB
Mapped: 13556 kB
Shmem: 2852 kB
Slab: 1013124 kB
SReclaimable: 45876 kB
SUnreclaim: 967248 kB
KernelStack: 7152 kB
PageTables: 12164 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 20390572 kB
Committed_AS: 396568 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 1407572 kB
DirectMap2M: 20504576 kB
DirectMap1G: 13631488 kB
ps aux --sort -rss
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 6758 1.8 14.9 5044996 4886964 ? Sl 01:22 23:21
/usr/local/sbin/bgpd -d -u root -g root -I --ignore_warnings
root 6752 0.0 0.1 86272 61920 ? Ss 01:22 0:16
/usr/local/sbin/zebra -d -u root -g root -I --ignore_warnings
root 6766 12.6 0.0 51592 29196 ? S 01:22 157:48
/usr/sbin/snmpd -p /var/run/snmpd.pid -Ln
root 7494 0.0 0.0 708976 5896 ? Ssl 01:22 0:09
/opt/collectd/sbin/collectd
root 15531 0.0 0.0 67864 5056 ? Ss 21:57 0:00 sshd:
paol [priv]
root 4915 0.0 0.0 271912 4904 ? Ss 01:21 0:25
/usr/sbin/syslog-ng --persist-file /var/lib/syslog-ng/syslog-ng.persist
--cfgfile /etc/syslog-ng/syslog-ng.conf --pidfile /run/syslog-ng.pid
root 4278 0.0 0.0 37220 4164 ? Ss 01:21 0:00
/lib/systemd/systemd-udevd --daemon
root 5147 0.0 0.0 32072 3232 ? Ss 01:21 0:00
/usr/sbin/sshd
root 5203 0.0 0.0 28876 2436 ? S 01:21 0:00 teamd
-d -f /etc/teamd.conf
root 17372 0.0 0.0 17924 2388 pts/2 R+ 22:13 0:00 ps aux
--sort -rss
root 4789 0.0 0.0 5032 2176 ? Ss 01:21 0:00 mdadm
--monitor --scan --daemonise --pid-file /var/run/mdadm.pid --syslog
root 7511 0.0 0.0 12676 1920 tty4 Ss+ 01:22 0:00
/sbin/agetty 38400 tty4 linux
root 7510 0.0 0.0 12676 1896 tty3 Ss+ 01:22 0:00
/sbin/agetty 38400 tty3 linux
root 7512 0.0 0.0 12676 1860 tty5 Ss+ 01:22 0:00
/sbin/agetty 38400 tty5 linux
root 7513 0.0 0.0 12676 1836 tty6 Ss+ 01:22 0:00
/sbin/agetty 38400 tty6 linux
root 7509 0.0 0.0 12676 1832 tty2 Ss+ 01:22 0:00
/sbin/agetty 38400 tty2 linux
And latest kernel that everything was working is: 4.14.3
Some observations - when i disable tso on all cards there is more memleak.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Huge memory leak with 4.15.0-rc2+
2017-12-11 21:23 Huge memory leak with 4.15.0-rc2+ Paweł Staszewski
@ 2017-12-11 21:48 ` Paweł Staszewski
2017-12-11 22:15 ` John Fastabend
0 siblings, 1 reply; 6+ messages in thread
From: Paweł Staszewski @ 2017-12-11 21:48 UTC (permalink / raw)
To: Linux Kernel Network Developers
W dniu 2017-12-11 o 22:23, Paweł Staszewski pisze:
> Hi
>
>
> I just upgraded some testing host to 4.15.0-rc2+ kernel
>
> And after some time of traffic processing - when traffic on all ports
> reach about 3Mpps - memleak started.
>
> Graph attached from memory usage: https://ibb.co/idK4zb
>
>
>
> HW config:
>
> Intel E5
>
> 8x Intel 82599 (used ixgbe driver from kernel)
>
> Interfaces with vlans attached
>
> All 8 ethernet ports are in one LAG group configured by team.
>
> With current settings
>
> (this host is acting as a router - and bgpd process is eating same
> amount of memory from the beginning about 5.2GB)
>
> cat /proc/meminfo
> MemTotal: 32770588 kB
> MemFree: 11342492 kB
> MemAvailable: 10982752 kB
> Buffers: 84704 kB
> Cached: 83180 kB
> SwapCached: 0 kB
> Active: 5105320 kB
> Inactive: 46252 kB
> Active(anon): 4985448 kB
> Inactive(anon): 1096 kB
> Active(file): 119872 kB
> Inactive(file): 45156 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> SwapTotal: 4005280 kB
> SwapFree: 4005280 kB
> Dirty: 236 kB
> Writeback: 0 kB
> AnonPages: 4983752 kB
> Mapped: 13556 kB
> Shmem: 2852 kB
> Slab: 1013124 kB
> SReclaimable: 45876 kB
> SUnreclaim: 967248 kB
> KernelStack: 7152 kB
> PageTables: 12164 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 20390572 kB
> Committed_AS: 396568 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 0 kB
> VmallocChunk: 0 kB
> HardwareCorrupted: 0 kB
> AnonHugePages: 0 kB
> ShmemHugePages: 0 kB
> ShmemPmdMapped: 0 kB
> CmaTotal: 0 kB
> CmaFree: 0 kB
> HugePages_Total: 0
> HugePages_Free: 0
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
> DirectMap4k: 1407572 kB
> DirectMap2M: 20504576 kB
> DirectMap1G: 13631488 kB
>
> ps aux --sort -rss
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> root 6758 1.8 14.9 5044996 4886964 ? Sl 01:22 23:21
> /usr/local/sbin/bgpd -d -u root -g root -I --ignore_warnings
> root 6752 0.0 0.1 86272 61920 ? Ss 01:22 0:16
> /usr/local/sbin/zebra -d -u root -g root -I --ignore_warnings
> root 6766 12.6 0.0 51592 29196 ? S 01:22 157:48
> /usr/sbin/snmpd -p /var/run/snmpd.pid -Ln
> root 7494 0.0 0.0 708976 5896 ? Ssl 01:22 0:09
> /opt/collectd/sbin/collectd
> root 15531 0.0 0.0 67864 5056 ? Ss 21:57 0:00 sshd:
> paol [priv]
> root 4915 0.0 0.0 271912 4904 ? Ss 01:21 0:25
> /usr/sbin/syslog-ng --persist-file
> /var/lib/syslog-ng/syslog-ng.persist --cfgfile
> /etc/syslog-ng/syslog-ng.conf --pidfile /run/syslog-ng.pid
> root 4278 0.0 0.0 37220 4164 ? Ss 01:21 0:00
> /lib/systemd/systemd-udevd --daemon
> root 5147 0.0 0.0 32072 3232 ? Ss 01:21 0:00
> /usr/sbin/sshd
> root 5203 0.0 0.0 28876 2436 ? S 01:21 0:00 teamd
> -d -f /etc/teamd.conf
> root 17372 0.0 0.0 17924 2388 pts/2 R+ 22:13 0:00 ps
> aux --sort -rss
> root 4789 0.0 0.0 5032 2176 ? Ss 01:21 0:00 mdadm
> --monitor --scan --daemonise --pid-file /var/run/mdadm.pid --syslog
> root 7511 0.0 0.0 12676 1920 tty4 Ss+ 01:22 0:00
> /sbin/agetty 38400 tty4 linux
> root 7510 0.0 0.0 12676 1896 tty3 Ss+ 01:22 0:00
> /sbin/agetty 38400 tty3 linux
> root 7512 0.0 0.0 12676 1860 tty5 Ss+ 01:22 0:00
> /sbin/agetty 38400 tty5 linux
> root 7513 0.0 0.0 12676 1836 tty6 Ss+ 01:22 0:00
> /sbin/agetty 38400 tty6 linux
> root 7509 0.0 0.0 12676 1832 tty2 Ss+ 01:22 0:00
> /sbin/agetty 38400 tty2 linux
>
> And latest kernel that everything was working is: 4.14.3
>
>
> Some observations - when i disable tso on all cards there is more
> memleak.
>
>
>
>
>
When traffic starts to drop - there is less and less memleak
below link to memory usage graph:
https://ibb.co/hU97kG
And there is rising slab_unrecl - Amount of unreclaimable memory used
for slab kernel allocations
Forgot to add that im using hfsc and qdiscs like pfifo on classes.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Huge memory leak with 4.15.0-rc2+
2017-12-11 21:48 ` Paweł Staszewski
@ 2017-12-11 22:15 ` John Fastabend
2017-12-11 22:27 ` Paweł Staszewski
0 siblings, 1 reply; 6+ messages in thread
From: John Fastabend @ 2017-12-11 22:15 UTC (permalink / raw)
To: Paweł Staszewski, Linux Kernel Network Developers
On 12/11/2017 01:48 PM, Paweł Staszewski wrote:
>
>
> W dniu 2017-12-11 o 22:23, Paweł Staszewski pisze:
>> Hi
>>
>>
>> I just upgraded some testing host to 4.15.0-rc2+ kernel
>>
>> And after some time of traffic processing - when traffic on all ports
>> reach about 3Mpps - memleak started.
>>
[...]
>> Some observations - when i disable tso on all cards there is more
>> memleak.
>>
>>
>>
>>
>>
> When traffic starts to drop - there is less and less memleak
> below link to memory usage graph:
> https://ibb.co/hU97kG
>
> And there is rising slab_unrecl - Amount of unreclaimable memory used
> for slab kernel allocations
>
>
> Forgot to add that im using hfsc and qdiscs like pfifo on classes.
>
>
Maybe some error case I missed in the qdisc patches I'm looking into
it.
Thanks,
John
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Huge memory leak with 4.15.0-rc2+
2017-12-11 22:15 ` John Fastabend
@ 2017-12-11 22:27 ` Paweł Staszewski
2017-12-12 17:57 ` Paweł Staszewski
0 siblings, 1 reply; 6+ messages in thread
From: Paweł Staszewski @ 2017-12-11 22:27 UTC (permalink / raw)
To: John Fastabend, Linux Kernel Network Developers
W dniu 2017-12-11 o 23:15, John Fastabend pisze:
> On 12/11/2017 01:48 PM, Paweł Staszewski wrote:
>>
>> W dniu 2017-12-11 o 22:23, Paweł Staszewski pisze:
>>> Hi
>>>
>>>
>>> I just upgraded some testing host to 4.15.0-rc2+ kernel
>>>
>>> And after some time of traffic processing - when traffic on all ports
>>> reach about 3Mpps - memleak started.
>>>
>
> [...]
>
>>> Some observations - when i disable tso on all cards there is more
>>> memleak.
>>>
>>>
>>>
>>>
>>>
>> When traffic starts to drop - there is less and less memleak
>> below link to memory usage graph:
>> https://ibb.co/hU97kG
>>
>> And there is rising slab_unrecl - Amount of unreclaimable memory used
>> for slab kernel allocations
>>
>>
>> Forgot to add that im using hfsc and qdiscs like pfifo on classes.
>>
>>
> Maybe some error case I missed in the qdisc patches I'm looking into
> it.
>
> Thanks,
> John
>
>
This is how it looks like when corelated on graph - traffic vs mem
https://ibb.co/njpkqG
Typical hfsc class + qdisc:
### Client interface vlan1616
tc qdisc del dev vlan1616 root
tc qdisc add dev vlan1616 handle 1: root hfsc default 100
tc class add dev vlan1616 parent 1: classid 1:100 hfsc ls m2 200Mbit ul
m2 200Mbit
tc qdisc add dev vlan1616 parent 1:100 handle 100: pfifo limit 128
### End TM for client interface
tc qdisc del dev vlan1616 ingress
tc qdisc add dev vlan1616 handle ffff: ingress
tc filter add dev vlan1616 parent ffff: protocol ip prio 50 u32 match ip
src 0.0.0.0/0 police rate 200Mbit burst 200M mtu 32k drop flowid 1:1
And this is same for about 450 vlan interfaces
Good thing is that compared to 4.14.3 i have about 5% less cpu load on
4.15.0-rc2+
When hfsc will be lockless or tbf - then it will be really huge
difference in cpu load on x86 when using traffic shaping - so really
good job John.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Huge memory leak with 4.15.0-rc2+
2017-12-11 22:27 ` Paweł Staszewski
@ 2017-12-12 17:57 ` Paweł Staszewski
2017-12-13 21:05 ` John Fastabend
0 siblings, 1 reply; 6+ messages in thread
From: Paweł Staszewski @ 2017-12-12 17:57 UTC (permalink / raw)
To: John Fastabend, Linux Kernel Network Developers
W dniu 2017-12-11 o 23:27, Paweł Staszewski pisze:
>
>
> W dniu 2017-12-11 o 23:15, John Fastabend pisze:
>> On 12/11/2017 01:48 PM, Paweł Staszewski wrote:
>>>
>>> W dniu 2017-12-11 o 22:23, Paweł Staszewski pisze:
>>>> Hi
>>>>
>>>>
>>>> I just upgraded some testing host to 4.15.0-rc2+ kernel
>>>>
>>>> And after some time of traffic processing - when traffic on all ports
>>>> reach about 3Mpps - memleak started.
>>>>
>>
>> [...]
>>
>>>> Some observations - when i disable tso on all cards there is more
>>>> memleak.
>>>>
>>>>
>>>>
>>>>
>>>>
>>> When traffic starts to drop - there is less and less memleak
>>> below link to memory usage graph:
>>> https://ibb.co/hU97kG
>>>
>>> And there is rising slab_unrecl - Amount of unreclaimable memory used
>>> for slab kernel allocations
>>>
>>>
>>> Forgot to add that im using hfsc and qdiscs like pfifo on classes.
>>>
>>>
>> Maybe some error case I missed in the qdisc patches I'm looking into
>> it.
>>
>> Thanks,
>> John
>>
>>
> This is how it looks like when corelated on graph - traffic vs mem
> https://ibb.co/njpkqG
>
> Typical hfsc class + qdisc:
> ### Client interface vlan1616
> tc qdisc del dev vlan1616 root
> tc qdisc add dev vlan1616 handle 1: root hfsc default 100
> tc class add dev vlan1616 parent 1: classid 1:100 hfsc ls m2 200Mbit
> ul m2 200Mbit
> tc qdisc add dev vlan1616 parent 1:100 handle 100: pfifo limit 128
> ### End TM for client interface
> tc qdisc del dev vlan1616 ingress
> tc qdisc add dev vlan1616 handle ffff: ingress
> tc filter add dev vlan1616 parent ffff: protocol ip prio 50 u32 match
> ip src 0.0.0.0/0 police rate 200Mbit burst 200M mtu 32k drop flowid 1:1
>
> And this is same for about 450 vlan interfaces
>
>
> Good thing is that compared to 4.14.3 i have about 5% less cpu load on
> 4.15.0-rc2+
>
> When hfsc will be lockless or tbf - then it will be really huge
> difference in cpu load on x86 when using traffic shaping - so really
> good job John.
>
>
>
>
Yestarday changed kernel from
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git
to
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/?h=v4.15-rc3
And there is no memleak.
So yes probabbly lockless qdisc patches
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Huge memory leak with 4.15.0-rc2+
2017-12-12 17:57 ` Paweł Staszewski
@ 2017-12-13 21:05 ` John Fastabend
0 siblings, 0 replies; 6+ messages in thread
From: John Fastabend @ 2017-12-13 21:05 UTC (permalink / raw)
To: Paweł Staszewski, Linux Kernel Network Developers
On 12/12/2017 09:57 AM, Paweł Staszewski wrote:
>
>
> W dniu 2017-12-11 o 23:27, Paweł Staszewski pisze:
>>
>>
>> W dniu 2017-12-11 o 23:15, John Fastabend pisze:
>>> On 12/11/2017 01:48 PM, Paweł Staszewski wrote:
>>>>
>>>> W dniu 2017-12-11 o 22:23, Paweł Staszewski pisze:
>>>>> Hi
>>>>>
>>>>>
>>>>> I just upgraded some testing host to 4.15.0-rc2+ kernel
>>>>>
>>>>> And after some time of traffic processing - when traffic on all ports
>>>>> reach about 3Mpps - memleak started.
>>>>>
>>>
>>> [...]
>>>
>>>>> Some observations - when i disable tso on all cards there is more
>>>>> memleak.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> When traffic starts to drop - there is less and less memleak
>>>> below link to memory usage graph:
>>>> https://ibb.co/hU97kG
>>>>
>>>> And there is rising slab_unrecl - Amount of unreclaimable memory used
>>>> for slab kernel allocations
>>>>
>>>>
>>>> Forgot to add that im using hfsc and qdiscs like pfifo on classes.
>>>>
>>>>
>>> Maybe some error case I missed in the qdisc patches I'm looking into
>>> it.
>>>
>>> Thanks,
>>> John
>>>
>>>
>> This is how it looks like when corelated on graph - traffic vs mem
>> https://ibb.co/njpkqG
>>
>> Typical hfsc class + qdisc:
>> ### Client interface vlan1616
>> tc qdisc del dev vlan1616 root
>> tc qdisc add dev vlan1616 handle 1: root hfsc default 100
>> tc class add dev vlan1616 parent 1: classid 1:100 hfsc ls m2 200Mbit ul m2 200Mbit
>> tc qdisc add dev vlan1616 parent 1:100 handle 100: pfifo limit 128
>> ### End TM for client interface
>> tc qdisc del dev vlan1616 ingress
>> tc qdisc add dev vlan1616 handle ffff: ingress
>> tc filter add dev vlan1616 parent ffff: protocol ip prio 50 u32 match ip src 0.0.0.0/0 police rate 200Mbit burst 200M mtu 32k drop flowid 1:1
>>
>> And this is same for about 450 vlan interfaces
>>
>>
>> Good thing is that compared to 4.14.3 i have about 5% less cpu load on 4.15.0-rc2+
>>
>> When hfsc will be lockless or tbf - then it will be really huge difference in cpu load on x86 when using traffic shaping - so really good job John.
>>
>>
>>
>>
>
> Yestarday changed kernel from
> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git
>
> to
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/?h=v4.15-rc3
>
>
> And there is no memleak.
> So yes probabbly lockless qdisc patches
>
It seems I was able to produce a similar memleak with qdisc patches
reverted and running TCP traffic overnight. I guess we can do a bisect
and track it down. Will try to get a "good" run tonight.
Thanks,
John
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-12-13 21:05 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-12-11 21:23 Huge memory leak with 4.15.0-rc2+ Paweł Staszewski
2017-12-11 21:48 ` Paweł Staszewski
2017-12-11 22:15 ` John Fastabend
2017-12-11 22:27 ` Paweł Staszewski
2017-12-12 17:57 ` Paweł Staszewski
2017-12-13 21:05 ` John Fastabend
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.