From: Greg KH <gregkh@linuxfoundation.org>
To: Imran Khan <imran.f.khan@oracle.com>
Cc: tj@kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v3 2/2] kernfs: Reduce contention around global per-fs kernfs_rwsem.
Date: Thu, 13 Jan 2022 11:58:16 +0100 [thread overview]
Message-ID: <YeAFyOR61+c+FyMn@kroah.com> (raw)
In-Reply-To: <20220113104259.1584491-3-imran.f.khan@oracle.com>
On Thu, Jan 13, 2022 at 09:42:59PM +1100, Imran Khan wrote:
> Right now a global per file system based rwsem (kernfs_rwsem)
> synchronizes multiple kernfs operations. On a large system with
> few hundred CPUs and few hundred applications simultaenously trying
> to access sysfs, this results in multiple sys_open(s) contending on
> kernfs_rwsem via kernfs_iop_permission and kernfs_dop_revalidate.
>
> - 21.42% 21.34% showgids [kernel.kallsyms] [k] up_read
> 21.34% __libc_start_main
> __GI___libc_open
> entry_SYSCALL_64_after_hwframe
> do_syscall_64
> sys_open
> do_sys_open
> do_filp_open
> - path_openat
> - 20.05% link_path_walk
> - 9.76% walk_component
> lookup_fast
> - d_revalidate.part.24
> - 9.75% kernfs_dop_revalidate
> up_read
> - 9.46% inode_permission
> - __inode_permission
> - 9.46% kernfs_iop_permission
> up_read
> - 0.83% kernfs_iop_get_link
> up_read
> - 0.80% lookup_fast
> d_revalidate.part.24
> kernfs_dop_revalidate
> up_read
>
> - 21.31% 21.21% showgids [kernel.kallsyms] [k] down_read
> 21.21% __libc_start_main
> __GI___libc_open
> entry_SYSCALL_64_after_hwframe
> do_syscall_64
> sys_open
> do_sys_open
> do_filp_open
> - path_openat
> - 19.78% link_path_walk
> - 10.62% inode_permission
> - __inode_permission
> - 10.62% kernfs_iop_permission
> down_read
> - 8.45% walk_component
> lookup_fast
> - d_revalidate.part.24
> - 8.45% kernfs_dop_revalidate
> down_read
> - 0.71% kernfs_iop_get_link
> down_read
> - 0.72% lookup_fast
> - d_revalidate.part.24
> - 0.72% kernfs_dop_revalidate
> down_read
> - 0.71% may_open
> inode_permission
> __inode_permission
> kernfs_iop_permission
> down_read
>
> Since permission is specific to a kernfs_node we can use a hashed
> lock to access/modify permission. Also use kernfs reference counting
> to ensure we are accessing/modifying permissions for an existing
> kernfs_node object.
>
> Using this change brings down the above mentioned down_read/up_read
> numbers to ~8%, thus indicating that contention around kernfs_rwsem
> has reduced to about 1/3rd of earlier value.
Ah, nevermind, you do post the results here, I should have kept reading.
Nice work!
I'll look at these after 5.17-rc1 is out, thanks!
greg k-h
next prev parent reply other threads:[~2022-01-13 10:58 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-01-13 10:42 [PATCH v3 0/2] kernfs: use hashed mutex and spinlock in place of global ones Imran Khan
2022-01-13 10:42 ` [PATCH v3 1/2] " Imran Khan
2022-01-13 15:08 ` kernel test robot
2022-01-13 16:37 ` Tejun Heo
2022-01-17 18:54 ` kernel test robot
2022-01-13 10:42 ` [PATCH v3 2/2] kernfs: Reduce contention around global per-fs kernfs_rwsem Imran Khan
2022-01-13 10:58 ` Greg KH [this message]
2022-01-13 16:42 ` Tejun Heo
2022-01-14 17:08 ` Imran Khan
2022-01-14 17:44 ` Tejun Heo
2022-02-02 15:10 ` Imran Khan
2022-01-25 2:55 ` [kernfs] 8652224976: WARNING:at_kernel/locking/rwsem.c:#up_write kernel test robot
2022-01-25 2:55 ` kernel test robot
2022-01-13 10:57 ` [PATCH v3 0/2] kernfs: use hashed mutex and spinlock in place of global ones Greg KH
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YeAFyOR61+c+FyMn@kroah.com \
--to=gregkh@linuxfoundation.org \
--cc=imran.f.khan@oracle.com \
--cc=linux-kernel@vger.kernel.org \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.