linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 0/4] kernfs: proposed locking and concurrency improvement
@ 2020-05-25  5:46 Ian Kent
  2020-05-25  5:47 ` [PATCH 1/4] kernfs: switch kernfs to use an rwsem Ian Kent
                   ` (4 more replies)
  0 siblings, 5 replies; 17+ messages in thread
From: Ian Kent @ 2020-05-25  5:46 UTC (permalink / raw)
  To: Andrew Morton
  Cc: Al Viro, Greg Kroah-Hartman, Tejun Heo, Rick Lindsley,
	Stephen Rothwell, David Howells, Miklos Szeredi, linux-fsdevel,
	Kernel Mailing List

For very large systems with hundreds of CPUs and TBs of RAM booting can
take a very long time.

Initial reports showed that booting a configuration of several hundred
CPUs and 64TB of RAM would take more than 30 minutes and require kernel
parameters of udev.children-max=1024 systemd.default_timeout_start_sec=3600
to prevent dropping into emergency mode.

Gathering information about what's happening during the boot is a bit
challenging. But two main issues appeared to be, a large number of path
lookups for non-existent files, and high lock contention in the VFS during
path walks particularly in the dentry allocation code path.

The underlying cause of this was believed to be the sheer number of sysfs
memory objects, 100,000+ for a 64TB memory configuration.

This patch series tries to reduce the locking needed during path walks
based on the assumption that there are many path walks with a fairly
large portion of those for non-existent paths.

This was done by adding kernfs negative dentry caching (non-existent
paths) to avoid continual alloc/free cycle of dentries and a read/write
semaphore introduced to increase kernfs concurrency during path walks.

With these changes the kernel parameters of udev.children-max=2048 and
systemd.default_timeout_start_sec=300 for are still needed to get the
fastest boot times and result in boot time of under 5 minutes.

There may be opportunities for further improvements but the series here
has seen a fair amount of testing. And thinking about what else could be
done, and discussing it with Rick Lindsay, I suspect improvements will
get more difficult to implement for somewhat less improvement so I think
what we have here is a good start for now.

I think what's needed now is patch review, and if we can get through
that, send them via linux-next for broader exposure and hopefully have
them merged into mainline.
---

Ian Kent (4):
      kernfs: switch kernfs to use an rwsem
      kernfs: move revalidate to be near lookup
      kernfs: improve kernfs path resolution
      kernfs: use revision to identify directory node changes


 fs/kernfs/dir.c             |  283 ++++++++++++++++++++++++++++---------------
 fs/kernfs/file.c            |    4 -
 fs/kernfs/inode.c           |   16 +-
 fs/kernfs/kernfs-internal.h |   29 ++++
 fs/kernfs/mount.c           |   12 +-
 fs/kernfs/symlink.c         |    4 -
 include/linux/kernfs.h      |    5 +
 7 files changed, 232 insertions(+), 121 deletions(-)

--
Ian


^ permalink raw reply	[flat|nested] 17+ messages in thread

end of thread, other threads:[~2020-06-11  3:02 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-05-25  5:46 [PATCH 0/4] kernfs: proposed locking and concurrency improvement Ian Kent
2020-05-25  5:47 ` [PATCH 1/4] kernfs: switch kernfs to use an rwsem Ian Kent
2020-06-06 15:52   ` [kernfs] ea7c5fc39a: stress-ng.stream.ops_per_sec 11827.2% improvement kernel test robot
2020-06-06 18:18     ` Greg Kroah-Hartman
2020-06-07  1:13       ` Ian Kent
2020-06-11  2:06         ` kernel test robot
2020-06-11  2:20           ` Rick Lindsley
2020-06-11  3:02           ` Ian Kent
2020-06-07  8:40   ` [PATCH 1/4] kernfs: switch kernfs to use an rwsem Ian Kent
2020-06-08  9:58     ` Ian Kent
2020-05-25  5:47 ` [PATCH 2/4] kernfs: move revalidate to be near lookup Ian Kent
2020-05-25  5:47 ` [PATCH 3/4] kernfs: improve kernfs path resolution Ian Kent
2020-05-25  5:47 ` [PATCH 4/4] kernfs: use revision to identify directory node changes Ian Kent
2020-05-25  6:16 ` [PATCH 0/4] kernfs: proposed locking and concurrency improvement Greg Kroah-Hartman
2020-05-25  7:23   ` Ian Kent
2020-05-25  7:31     ` Greg Kroah-Hartman
2020-05-27 12:44   ` Rick Lindsley

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).