From: Johannes Weiner <hannes@cmpxchg.org> To: "David S. Miller" <davem@davemloft.net>, Andrew Morton <akpm@linux-foundation.org> Cc: Michal Hocko <mhocko@suse.cz>, Vladimir Davydov <vdavydov@virtuozzo.com>, Tejun Heo <tj@kernel.org>, netdev@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 0/8] mm: memcontrol: account socket memory in unified hierarchy Date: Thu, 22 Oct 2015 00:21:28 -0400 [thread overview] Message-ID: <1445487696-21545-1-git-send-email-hannes@cmpxchg.org> (raw) Hi, this series adds socket buffer memory tracking and accounting to the unified hierarchy memory cgroup controller. [ Networking people, at this time please check the diffstat below to avoid going into convulsions. ] Socket buffer memory can make up a significant share of a workload's memory footprint, and so it needs to be accounted and tracked out of the box, along with other types of memory that can be directly linked to userspace activity, in order to provide useful resource isolation. Historically, socket buffers were accounted in a separate counter, without any pressure equalization between anonymous memory, page cache, and the socket buffers. When the socket buffer pool was exhausted, buffer allocations would fail hard and cause network performance to tank, regardless of whether there was still memory available to the group or not. Likewise, struggling anonymous or cache workingsets could not dip into an idle socket memory pool. Because of this, the feature was not usable for many real life applications. To not repeat this mistake, the new memory controller will account all types of memory pages it is tracking on behalf of a cgroup in a single pool. And upon pressure, the VM reclaims and shrinks whatever memory in that pool is within its reach. These patches add accounting for memory consumed by sockets associated with a cgroup to the existing pool of anonymous pages and page cache. Patch #3 reworks the existing memcg socket infrastructure. It has many provisions for future plans that won't materialize, and much of this simply evaporates. The networking people should be happy about this. Patch #5 adds accounting and tracking of socket memory to the unified hierarchy memory controller, as described above. It uses the existing per-cpu charge caches and triggers high limit reclaim asynchroneously. Patch #8 uses the vmpressure extension to equalize pressure between the pages tracked natively by the VM and socket buffer pages. As the pool is shared, it makes sense that while natively tracked pages are under duress the network transmit windows are also not increased. As per above, this is an essential part of the new memory controller's core functionality. With the unified hierarchy nearing release, please consider this for 4.4. include/linux/memcontrol.h | 90 +++++++++------- include/linux/page_counter.h | 6 +- include/net/sock.h | 139 ++---------------------- include/net/tcp.h | 5 +- include/net/tcp_memcontrol.h | 7 -- mm/backing-dev.c | 2 +- mm/hugetlb_cgroup.c | 3 +- mm/memcontrol.c | 235 ++++++++++++++++++++++++++--------------- mm/page_counter.c | 14 +-- mm/vmpressure.c | 29 ++++- mm/vmscan.c | 41 +++---- net/core/sock.c | 78 ++++---------- net/ipv4/sysctl_net_ipv4.c | 1 - net/ipv4/tcp.c | 3 +- net/ipv4/tcp_ipv4.c | 9 +- net/ipv4/tcp_memcontrol.c | 147 ++++---------------------- net/ipv4/tcp_output.c | 6 +- net/ipv6/tcp_ipv6.c | 3 - 18 files changed, 319 insertions(+), 499 deletions(-)
WARNING: multiple messages have this Message-ID (diff)
From: Johannes Weiner <hannes@cmpxchg.org> To: "David S. Miller" <davem@davemloft.net>, Andrew Morton <akpm@linux-foundation.org> Cc: Michal Hocko <mhocko@suse.cz>, Vladimir Davydov <vdavydov@virtuozzo.com>, Tejun Heo <tj@kernel.org>, netdev@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 0/8] mm: memcontrol: account socket memory in unified hierarchy Date: Thu, 22 Oct 2015 00:21:28 -0400 [thread overview] Message-ID: <1445487696-21545-1-git-send-email-hannes@cmpxchg.org> (raw) Hi, this series adds socket buffer memory tracking and accounting to the unified hierarchy memory cgroup controller. [ Networking people, at this time please check the diffstat below to avoid going into convulsions. ] Socket buffer memory can make up a significant share of a workload's memory footprint, and so it needs to be accounted and tracked out of the box, along with other types of memory that can be directly linked to userspace activity, in order to provide useful resource isolation. Historically, socket buffers were accounted in a separate counter, without any pressure equalization between anonymous memory, page cache, and the socket buffers. When the socket buffer pool was exhausted, buffer allocations would fail hard and cause network performance to tank, regardless of whether there was still memory available to the group or not. Likewise, struggling anonymous or cache workingsets could not dip into an idle socket memory pool. Because of this, the feature was not usable for many real life applications. To not repeat this mistake, the new memory controller will account all types of memory pages it is tracking on behalf of a cgroup in a single pool. And upon pressure, the VM reclaims and shrinks whatever memory in that pool is within its reach. These patches add accounting for memory consumed by sockets associated with a cgroup to the existing pool of anonymous pages and page cache. Patch #3 reworks the existing memcg socket infrastructure. It has many provisions for future plans that won't materialize, and much of this simply evaporates. The networking people should be happy about this. Patch #5 adds accounting and tracking of socket memory to the unified hierarchy memory controller, as described above. It uses the existing per-cpu charge caches and triggers high limit reclaim asynchroneously. Patch #8 uses the vmpressure extension to equalize pressure between the pages tracked natively by the VM and socket buffer pages. As the pool is shared, it makes sense that while natively tracked pages are under duress the network transmit windows are also not increased. As per above, this is an essential part of the new memory controller's core functionality. With the unified hierarchy nearing release, please consider this for 4.4. include/linux/memcontrol.h | 90 +++++++++------- include/linux/page_counter.h | 6 +- include/net/sock.h | 139 ++---------------------- include/net/tcp.h | 5 +- include/net/tcp_memcontrol.h | 7 -- mm/backing-dev.c | 2 +- mm/hugetlb_cgroup.c | 3 +- mm/memcontrol.c | 235 ++++++++++++++++++++++++++--------------- mm/page_counter.c | 14 +-- mm/vmpressure.c | 29 ++++- mm/vmscan.c | 41 +++---- net/core/sock.c | 78 ++++---------- net/ipv4/sysctl_net_ipv4.c | 1 - net/ipv4/tcp.c | 3 +- net/ipv4/tcp_ipv4.c | 9 +- net/ipv4/tcp_memcontrol.c | 147 ++++---------------------- net/ipv4/tcp_output.c | 6 +- net/ipv6/tcp_ipv6.c | 3 - 18 files changed, 319 insertions(+), 499 deletions(-) -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next reply other threads:[~2015-10-22 4:21 UTC|newest] Thread overview: 156+ messages / expand[flat|nested] mbox.gz Atom feed top 2015-10-22 4:21 Johannes Weiner [this message] 2015-10-22 4:21 ` [PATCH 0/8] mm: memcontrol: account socket memory in unified hierarchy Johannes Weiner 2015-10-22 4:21 ` [PATCH 1/8] mm: page_counter: let page_counter_try_charge() return bool Johannes Weiner 2015-10-22 4:21 ` Johannes Weiner 2015-10-23 11:31 ` Michal Hocko 2015-10-23 11:31 ` Michal Hocko 2015-10-22 4:21 ` [PATCH 2/8] mm: memcontrol: export root_mem_cgroup Johannes Weiner 2015-10-22 4:21 ` Johannes Weiner 2015-10-23 11:32 ` Michal Hocko 2015-10-23 11:32 ` Michal Hocko 2015-10-22 4:21 ` [PATCH 3/8] net: consolidate memcg socket buffer tracking and accounting Johannes Weiner 2015-10-22 4:21 ` Johannes Weiner 2015-10-22 18:46 ` Vladimir Davydov 2015-10-22 18:46 ` Vladimir Davydov 2015-10-22 18:46 ` Vladimir Davydov 2015-10-22 19:09 ` Johannes Weiner 2015-10-22 19:09 ` Johannes Weiner 2015-10-23 13:42 ` Vladimir Davydov 2015-10-23 13:42 ` Vladimir Davydov 2015-10-23 13:42 ` Vladimir Davydov 2015-10-23 12:38 ` Michal Hocko 2015-10-23 12:38 ` Michal Hocko 2015-10-22 4:21 ` [PATCH 4/8] mm: memcontrol: prepare for unified hierarchy socket accounting Johannes Weiner 2015-10-22 4:21 ` Johannes Weiner 2015-10-23 12:39 ` Michal Hocko 2015-10-23 12:39 ` Michal Hocko 2015-10-22 4:21 ` [PATCH 5/8] mm: memcontrol: account socket memory on unified hierarchy Johannes Weiner 2015-10-22 4:21 ` Johannes Weiner 2015-10-22 18:47 ` Vladimir Davydov 2015-10-22 18:47 ` Vladimir Davydov 2015-10-22 18:47 ` Vladimir Davydov 2015-10-23 13:19 ` Michal Hocko 2015-10-23 13:19 ` Michal Hocko 2015-10-23 13:19 ` Michal Hocko 2015-10-23 13:59 ` David Miller 2015-10-23 13:59 ` David Miller 2015-10-23 13:59 ` David Miller 2015-10-26 16:56 ` Johannes Weiner 2015-10-26 16:56 ` Johannes Weiner 2015-10-27 12:26 ` Michal Hocko 2015-10-27 12:26 ` Michal Hocko 2015-10-27 13:49 ` David Miller 2015-10-27 13:49 ` David Miller 2015-10-27 13:49 ` David Miller 2015-10-27 15:41 ` Johannes Weiner 2015-10-27 15:41 ` Johannes Weiner 2015-10-27 15:41 ` Johannes Weiner 2015-10-27 16:15 ` Michal Hocko 2015-10-27 16:15 ` Michal Hocko 2015-10-27 16:42 ` Johannes Weiner 2015-10-27 16:42 ` Johannes Weiner 2015-10-28 0:45 ` David Miller 2015-10-28 0:45 ` David Miller 2015-10-28 0:45 ` David Miller 2015-10-28 3:05 ` Johannes Weiner 2015-10-28 3:05 ` Johannes Weiner 2015-10-29 15:25 ` Michal Hocko 2015-10-29 15:25 ` Michal Hocko 2015-10-29 16:10 ` Johannes Weiner 2015-10-29 16:10 ` Johannes Weiner 2015-10-29 16:10 ` Johannes Weiner 2015-11-04 10:42 ` Michal Hocko 2015-11-04 10:42 ` Michal Hocko 2015-11-04 19:50 ` Johannes Weiner 2015-11-04 19:50 ` Johannes Weiner 2015-11-04 19:50 ` Johannes Weiner 2015-11-05 14:40 ` Michal Hocko 2015-11-05 14:40 ` Michal Hocko 2015-11-05 16:16 ` David Miller 2015-11-05 16:16 ` David Miller 2015-11-05 16:28 ` Michal Hocko 2015-11-05 16:28 ` Michal Hocko 2015-11-05 16:28 ` Michal Hocko 2015-11-05 16:30 ` David Miller 2015-11-05 16:30 ` David Miller 2015-11-05 22:32 ` Johannes Weiner 2015-11-05 22:32 ` Johannes Weiner 2015-11-05 22:32 ` Johannes Weiner 2015-11-06 12:51 ` Michal Hocko 2015-11-06 12:51 ` Michal Hocko 2015-11-05 20:55 ` Johannes Weiner 2015-11-05 20:55 ` Johannes Weiner 2015-11-05 22:52 ` Johannes Weiner 2015-11-05 22:52 ` Johannes Weiner 2015-11-05 22:52 ` Johannes Weiner 2015-11-06 10:57 ` Michal Hocko 2015-11-06 10:57 ` Michal Hocko 2015-11-06 16:19 ` Johannes Weiner 2015-11-06 16:19 ` Johannes Weiner 2015-11-06 16:46 ` Michal Hocko 2015-11-06 16:46 ` Michal Hocko 2015-11-06 16:46 ` Michal Hocko 2015-11-06 17:45 ` Johannes Weiner 2015-11-06 17:45 ` Johannes Weiner 2015-11-06 17:45 ` Johannes Weiner 2015-11-07 3:45 ` David Miller 2015-11-07 3:45 ` David Miller 2015-11-12 18:36 ` Mel Gorman 2015-11-12 18:36 ` Mel Gorman 2015-11-12 18:36 ` Mel Gorman 2015-11-12 19:12 ` Johannes Weiner 2015-11-12 19:12 ` Johannes Weiner 2015-11-06 9:05 ` Vladimir Davydov 2015-11-06 9:05 ` Vladimir Davydov 2015-11-06 9:05 ` Vladimir Davydov 2015-11-06 9:05 ` Vladimir Davydov 2015-11-06 13:29 ` Michal Hocko 2015-11-06 13:29 ` Michal Hocko 2015-11-06 16:35 ` Johannes Weiner 2015-11-06 16:35 ` Johannes Weiner 2015-11-06 13:21 ` Michal Hocko 2015-11-06 13:21 ` Michal Hocko 2015-10-22 4:21 ` [PATCH 6/8] mm: vmscan: simplify memcg vs. global shrinker invocation Johannes Weiner 2015-10-22 4:21 ` Johannes Weiner 2015-10-23 13:26 ` Michal Hocko 2015-10-23 13:26 ` Michal Hocko 2015-10-22 4:21 ` [PATCH 7/8] mm: vmscan: report vmpressure at the level of reclaim activity Johannes Weiner 2015-10-22 4:21 ` Johannes Weiner 2015-10-22 18:48 ` Vladimir Davydov 2015-10-22 18:48 ` Vladimir Davydov 2015-10-22 18:48 ` Vladimir Davydov 2015-10-22 18:48 ` Vladimir Davydov 2015-10-23 13:49 ` Michal Hocko 2015-10-23 13:49 ` Michal Hocko 2015-10-23 13:49 ` Michal Hocko 2015-10-22 4:21 ` [PATCH 8/8] mm: memcontrol: hook up vmpressure to socket pressure Johannes Weiner 2015-10-22 4:21 ` Johannes Weiner 2015-10-22 18:57 ` Vladimir Davydov 2015-10-22 18:57 ` Vladimir Davydov 2015-10-22 18:57 ` Vladimir Davydov 2015-10-22 18:45 ` [PATCH 0/8] mm: memcontrol: account socket memory in unified hierarchy Vladimir Davydov 2015-10-22 18:45 ` Vladimir Davydov 2015-10-22 18:45 ` Vladimir Davydov 2015-10-26 17:22 ` Johannes Weiner 2015-10-26 17:22 ` Johannes Weiner 2015-10-26 17:22 ` Johannes Weiner 2015-10-26 17:22 ` Johannes Weiner 2015-10-27 8:43 ` Vladimir Davydov 2015-10-27 8:43 ` Vladimir Davydov 2015-10-27 8:43 ` Vladimir Davydov 2015-10-27 16:01 ` Johannes Weiner 2015-10-27 16:01 ` Johannes Weiner 2015-10-28 8:20 ` Vladimir Davydov 2015-10-28 8:20 ` Vladimir Davydov 2015-10-28 8:20 ` Vladimir Davydov 2015-10-28 18:58 ` Johannes Weiner 2015-10-28 18:58 ` Johannes Weiner 2015-10-29 9:27 ` Vladimir Davydov 2015-10-29 9:27 ` Vladimir Davydov 2015-10-29 9:27 ` Vladimir Davydov 2015-10-29 17:52 ` Johannes Weiner 2015-10-29 17:52 ` Johannes Weiner 2015-10-29 17:52 ` Johannes Weiner 2015-11-02 14:47 ` Vladimir Davydov 2015-11-02 14:47 ` Vladimir Davydov 2015-11-02 14:47 ` Vladimir Davydov
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1445487696-21545-1-git-send-email-hannes@cmpxchg.org \ --to=hannes@cmpxchg.org \ --cc=akpm@linux-foundation.org \ --cc=cgroups@vger.kernel.org \ --cc=davem@davemloft.net \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=mhocko@suse.cz \ --cc=netdev@vger.kernel.org \ --cc=tj@kernel.org \ --cc=vdavydov@virtuozzo.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.