From: SeongJae Park <sjpark@amazon.com>
To: <davem@davemloft.net>
Cc: SeongJae Park <sjpark@amazon.de>, <kuba@kernel.org>,
<kuznet@ms2.inr.ac.ru>, <paulmck@kernel.org>,
<netdev@vger.kernel.org>, <rcu@vger.kernel.org>,
<linux-kernel@vger.kernel.orgor>
Subject: [PATCH 0/1] net: Reduce rcu_barrier() contentions from 'unshare(CLONE_NEWNET)'
Date: Tue, 8 Dec 2020 10:45:28 +0100 [thread overview]
Message-ID: <20201208094529.23266-1-sjpark@amazon.com> (raw)
From: SeongJae Park <sjpark@amazon.de>
On a few of our systems, I found frequent 'unshare(CLONE_NEWNET)' calls
make the number of active slab objects including 'sock_inode_cache' type
rapidly and continuously increase. As a result, memory pressure occurs.
'cleanup_net()' and 'fqdir_work_fn()' are functions that deallocate the
relevant memory objects. They are asynchronously invoked by the work
queues and internally use 'rcu_barrier()' to ensure safe destructions.
'cleanup_net()' works in a batched maneer in a single thread worker,
while 'fqdir_work_fn()' works for each 'fqdir_exit()' call in the
'system_wq'.
Therefore, 'fqdir_work_fn()' called frequently under the workload and
made the contention for 'rcu_barrier()' high. In more detail, the
global mutex, 'rcu_state.barrier_mutex' became the bottleneck.
I tried making 'fqdir_work_fn()' batched and confirmed it works. The
following patch is for the change. I think this is the right solution
for point fix of this issue, but someone might blame different parts.
1. User: Frequent 'unshare()' calls
From some point of view, such frequent 'unshare()' calls might seem only
insane.
2. Global mutex in 'rcu_barrier()'
Because of the global mutex, 'rcu_barrier()' callers could wait long
even after the callbacks started before the call finished. Therefore,
similar issues could happen in another 'rcu_barrier()' usages. Maybe we
can use some wait queue like mechanism to notify the waiters when the
desired time came.
I personally believe applying the point fix for now and making
'rcu_barrier()' improvement in longterm make sense. If I'm missing
something or you have different opinions, please feel free to let me
know.
SeongJae Park (1):
net/ipv4/inet_fragment: Batch fqdir destroy works
include/net/inet_frag.h | 2 +-
net/ipv4/inet_fragment.c | 28 ++++++++++++++++++++--------
2 files changed, 21 insertions(+), 9 deletions(-)
--
2.17.1
next reply other threads:[~2020-12-08 9:47 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-08 9:45 SeongJae Park [this message]
2020-12-08 9:45 ` [PATCH 1/1] net/ipv4/inet_fragment: Batch fqdir destroy works SeongJae Park
2020-12-09 23:16 ` Jakub Kicinski
2020-12-10 6:43 ` SeongJae Park
2020-12-10 0:17 ` Eric Dumazet
2020-12-10 7:27 ` SeongJae Park
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201208094529.23266-1-sjpark@amazon.com \
--to=sjpark@amazon.com \
--cc=davem@davemloft.net \
--cc=kuba@kernel.org \
--cc=kuznet@ms2.inr.ac.ru \
--cc=linux-kernel@vger.kernel.orgor \
--cc=netdev@vger.kernel.org \
--cc=paulmck@kernel.org \
--cc=rcu@vger.kernel.org \
--cc=sjpark@amazon.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).