All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 0/2] kernfs: speed up concurrency performance
@ 2020-12-02 14:58 Fox Chen
  2020-12-02 14:58 ` [PATCH 1/2] kernfs: replace the mutex in kernfs_iop_permission with a rwlock Fox Chen
                   ` (2 more replies)
  0 siblings, 3 replies; 16+ messages in thread
From: Fox Chen @ 2020-12-02 14:58 UTC (permalink / raw)
  To: gregkh, tj; +Cc: Fox Chen, linux-kernel

Hello,

kernfs is an important facillity to support pseudo file systems and cgroup. 
Currently, with a global mutex, reading files concurrently from kernfs (e.g. /sys) 
is very slow.

This problem is reported by Brice Goglin on thread:
Re: [PATCH 1/4] drivers core: Introduce CPU type sysfs interface
https://lore.kernel.org/lkml/X60dvJoT4fURcnsF@kroah.com/

I independently comfirmed this on a 96-core AWS c5.metal server.
Do open+read+write on /sys/devices/system/cpu/cpu15/topology/core_id 1000 times.
With a single thread it takes ~2.5 us for each open+read+close.
With one thread per core, 96 threads running simultaneously takes 540 us 
for each of the same operation (without much variation) -- 200x slower than the 
single thread one. 

The problem can only be observed in large machines (>=16 cores).
The more cores you have the slower it can be.

Perf shows that CPUs spend most of the time (>80%) waiting on mutex locks in 
kernfs_iop_permission and kernfs_dop_revalidate.

This patchset contains the following 2 patches:
0001-kernfs-replace-the-mutex-in-kernfs_iop_permission-wi.patch
0002-kernfs-remove-mutex-in-kernfs_dop_revalidate.patch

0001 replace the mutex lock in kernfs_iop_permission with a new rwlock and 
0002 removes the mutex lock in kernfs_dop_revalidate.

After applying this patchset, the multi-thread performance becomes linear with 
the fastest one at ~30 us to the worst at ~150 us, very similar as I tested it
on a normal ext4 file system with fastest one at ~20 us to slowest at ~100 us. 
And I believe that is largely due to spin_locks in filesystems which are normal.

Although it's still slower than single thread, users can benefit from this 
patchset, especially ones working on HPC realm with lots of cpu cores and want to
fetch system information from sysfs.

I tried my best to solve this problem. If there is stupid mistake, please kindly
point out. I would appreciate it greatly.

Fox

 fs/kernfs/dir.c        |  9 +++------
 fs/kernfs/inode.c      | 16 ++++++++--------
 include/linux/kernfs.h |  1 +
 3 files changed, 12 insertions(+), 14 deletions(-)

-- 
2.29.2


^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2020-12-18  8:02 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-02 14:58 [PATCH 0/2] kernfs: speed up concurrency performance Fox Chen
2020-12-02 14:58 ` [PATCH 1/2] kernfs: replace the mutex in kernfs_iop_permission with a rwlock Fox Chen
2020-12-02 18:27   ` Greg KH
2020-12-02 18:34   ` Tejun Heo
2020-12-02 18:37   ` Tejun Heo
2020-12-03  6:34     ` Fox Chen
2020-12-03  7:19   ` [kernfs] d680236464: BUG:sleeping_function_called_from_invalid_context_at_kernel/locking/mutex.c kernel test robot
2020-12-03  7:19     ` kernel test robot
2020-12-02 14:58 ` [PATCH 2/2] kernfs: remove mutex in kernfs_dop_revalidate Fox Chen
2020-12-02 18:27   ` Greg KH
2020-12-03  6:35     ` Fox Chen
2020-12-02 18:46   ` Tejun Heo
2020-12-03  6:44     ` Fox Chen
2020-12-18  8:02   ` [kernfs] 37746795a6: stress-ng.spawn.ops_per_sec 7.1% improvement kernel test robot
2020-12-02 18:29 ` [PATCH 0/2] kernfs: speed up concurrency performance Greg KH
2020-12-03  6:38   ` Fox Chen

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.