From: Ivan Babrou <ivan@cloudflare.com>
To: Linux Kernel Network Developers <netdev@vger.kernel.org>
Cc: kernel-team <kernel-team@cloudflare.com>,
Eric Dumazet <edumazet@google.com>,
Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
linux-kernel <linux-kernel@vger.kernel.org>
Subject: wait_for_unix_gc can cause CPU overload for well behaved programs
Date: Thu, 19 Oct 2023 15:35:01 -0700 [thread overview]
Message-ID: <CABWYdi1kiu1g1mAq6DpQWczg78tMzaVFnytNMemZATFHqYSqYw@mail.gmail.com> (raw)
Hello,
We have observed this issue twice (2019 and 2023): a well behaved
service that doesn't pass any file descriptors around starts to spend
a ton of CPU time in wait_for_unix_gc.
The cause of this is that the unix send path unconditionally calls
wait_for_unix_gc, which is a global garbage collection. If any
misbehaved program exists on a system, it can force extra work for
well behaved programs.
This behavior is not new: 9915672d4127 ("af_unix: limit
unix_tot_inflight") is from 2010.
I managed to come up with a repro for this behavior:
* https://gist.github.com/bobrik/82e5722261920c9f23d9402b88a0bb27
It also includes a flamegraph illustrating the issue. It's all in one
program for convenience, but in reality the offender not picking up
SCM_RIGHTS messages and the suffering program just minding its own
business are separate.
It is also non-trivial to find the offender when this happens as it
can be completely idle while wrecking havoc for the rest of the
system.
I don't think it's fair to penalize every unix_stream_sendmsg like
this. The 16k threshold also doesn't feel very flexible, surely
computers are bigger these days and can handle more.
next reply other threads:[~2023-10-19 22:35 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-10-19 22:35 Ivan Babrou [this message]
[not found] ` <20231020104728.2060-1-hdanton@sina.com>
2023-10-20 17:25 ` wait_for_unix_gc can cause CPU overload for well behaved programs Ivan Babrou
[not found] ` <20231021012322.1799-1-hdanton@sina.com>
2023-10-23 23:22 ` Ivan Babrou
2023-10-23 23:45 ` Kuniyuki Iwashima
2023-11-17 23:38 ` Ivan Babrou
2023-11-20 19:29 ` Kuniyuki Iwashima
2023-11-20 22:30 ` Ivan Babrou
2023-11-20 23:53 ` Kuniyuki Iwashima
2023-10-20 22:05 ` Kuniyuki Iwashima
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CABWYdi1kiu1g1mAq6DpQWczg78tMzaVFnytNMemZATFHqYSqYw@mail.gmail.com \
--to=ivan@cloudflare.com \
--cc=edumazet@google.com \
--cc=kernel-team@cloudflare.com \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).