From: Michal Hocko <mhocko@suse.com>
To: Antoine Tenart <atenart@kernel.org>
Cc: davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com,
gregkh@linuxfoundation.org, ebiederm@xmission.com,
stephen@networkplumber.org, herbert@gondor.apana.org.au,
juri.lelli@redhat.com, netdev@vger.kernel.org,
Jiri Bohac <jbohac@suse.cz>
Subject: Re: [RFC PATCH net-next 0/9] Userspace spinning on net-sysfs access
Date: Wed, 6 Oct 2021 08:37:47 +0200 [thread overview]
Message-ID: <YV1EO9dsVSwWW7ua@dhcp22.suse.cz> (raw)
In-Reply-To: <20210928125500.167943-1-atenart@kernel.org>
On Tue 28-09-21 14:54:51, Antoine Tenart wrote:
> Hello,
Hi,
thanks for posting this. Coincidentally we have come across a similar
problem as well just recently.
> What made those syscalls to spin is the following construction (which is
> found a lot in net sysfs and sysctl code):
>
> if (!rtnl_trylock())
> return restart_syscall();
One of our customer is using Prometeus (https://github.com/prometheus/prometheus)
for monitoring and they have noticed that running several instances of
node-exporter can lead to a high CPU utilization. After some
investigation it has turned out that most instances are busy looping on
on of the sysfs files while one instance is processing sysfs speed file
for mlx driver which performs quite a convoluted set of operations (send
commands to the lower layers via workqueues) to talk to the device to
get the information.
The problem becomes more visible with more instance of node-exporter
running at parallel. This results in some monitoring alarms at the said
machine because the high CPU utilization is not expected.
I would appreciate if you CC me on next versions of this patchset.
Thankis for working on this!
--
Michal Hocko
SUSE Labs
next prev parent reply other threads:[~2021-10-06 6:38 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-28 12:54 [RFC PATCH net-next 0/9] Userspace spinning on net-sysfs access Antoine Tenart
2021-09-28 12:54 ` [RFC PATCH net-next 1/9] net-sysfs: try not to restart the syscall if it will fail eventually Antoine Tenart
2021-09-28 12:54 ` [RFC PATCH net-next 2/9] net: split unlisting the net device from unlisting its node name Antoine Tenart
2021-09-28 12:54 ` [RFC PATCH net-next 3/9] net: export netdev_name_node_lookup Antoine Tenart
2021-09-28 12:54 ` [RFC PATCH net-next 4/9] bonding: use the correct function to check for netdev name collision Antoine Tenart
2021-09-28 12:54 ` [RFC PATCH net-next 5/9] ppp: " Antoine Tenart
2021-09-28 12:54 ` [RFC PATCH net-next 6/9] net: " Antoine Tenart
2021-09-28 12:54 ` [RFC PATCH net-next 7/9] net: delay the removal of the name nodes until run_todo Antoine Tenart
2021-09-28 12:54 ` [RFC PATCH net-next 8/9] net: delay device_del " Antoine Tenart
2021-09-29 0:02 ` Jakub Kicinski
2021-09-29 8:26 ` Antoine Tenart
2021-09-29 13:31 ` Jakub Kicinski
2021-09-29 17:31 ` Antoine Tenart
2021-10-29 9:04 ` Antoine Tenart
2021-10-05 15:21 ` Antoine Tenart
2021-10-05 18:34 ` Jakub Kicinski
2021-09-28 12:55 ` [RFC PATCH net-next 9/9] net-sysfs: remove the use of rtnl_trylock/restart_syscall Antoine Tenart
2021-10-06 6:45 ` Michal Hocko
2021-10-06 8:03 ` Antoine Tenart
2021-10-06 8:55 ` Michal Hocko
2021-10-06 6:37 ` Michal Hocko [this message]
2021-10-06 7:59 ` [RFC PATCH net-next 0/9] Userspace spinning on net-sysfs access Antoine Tenart
2021-10-06 8:35 ` Michal Hocko
2021-10-29 14:33 ` Antoine Tenart
2021-10-29 15:45 ` Stephen Hemminger
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YV1EO9dsVSwWW7ua@dhcp22.suse.cz \
--to=mhocko@suse.com \
--cc=atenart@kernel.org \
--cc=davem@davemloft.net \
--cc=ebiederm@xmission.com \
--cc=gregkh@linuxfoundation.org \
--cc=herbert@gondor.apana.org.au \
--cc=jbohac@suse.cz \
--cc=juri.lelli@redhat.com \
--cc=kuba@kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=stephen@networkplumber.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.