archive mirror
 help / color / mirror / Atom feed
From: NeilBrown <>
To: Jeff Layton <>,
	Alexander Viro <>
Cc: "J. Bruce Fields" <>,
	Martin Wilck <>,,
	Frank Filz <>,
Subject: [PATCH 0/5 v2] locks: avoid thundering-herd wake-ups
Date: Tue, 14 Aug 2018 13:56:51 +1000	[thread overview]
Message-ID: <153421852728.24426.2111161640156686201.stgit@noble> (raw)

V2, which added wake_non_conflicts() was more broken than V1 - as
Bruce explained there is no transitivity in the blocking relation
between locks.
So this series takes a simpler approach.
It still attached waiters between other waiters as necessary to ensure
  - a waiter is blocked by it's parent (fl->blocker) and all further
    ancestors, and
  - the list of waiters on fl_blocked are mutually non-conflicting.

When a lock (the root of a tree of requests) is released, only its
immediate children (fl_blocked) are woken.
When any lock is woken (either because its fl_blocker was released
to due to a signal or similar) it with either:
 - be granted
 - be aborted
 - be re-queued beneath some other lock.

In the first case tree of blocked locks is moved across to the newly
created lock, and the invariants still hold.
In the order two cases, the tree or blocked waiters are all detached
and woken.

Note that this series has not received much testing yet.

Original description:
If you have a many-core machine, and have many threads all wanting to
briefly lock a give file (udev is known to do this), you can get quite
poor performance.

When one thread releases a lock, it wakes up all other threads that
are waiting (classic thundering-herd) - one will get the lock and the
others go to sleep.
When you have few cores, this is not very noticeable: by the time the
4th or 5th thread gets enough CPU time to try to claim the lock, the
earlier threads have claimed it, done what was needed, and released.
With 50+ cores, the contention can easily be measured.

This patchset creates a tree of pending lock request in which siblings
don't conflict and each lock request does conflict with its parent.
When a lock is released, only requests which don't conflict with each
other a woken.

Testing shows that lock-acquisitions-per-second is now fairly stable even
as number of contending process goes to 1000.  Without this patch,
locks-per-second drops off steeply after a few 10s of processes.

There is a small cost to this extra complexity.
At 20 processes running a particular test on 72 cores, the lock
acquisitions per second drops from 1.8 million to 1.4 million with
this patch.  For 100 processes, this patch still provides 1.4 million
while without this patch there are about 700,000.



NeilBrown (5):
      fs/locks: rename some lists and pointers.
      fs/locks: split out __locks_wake_up_blocks().
      fs/locks: allow a lock request to block other requests.
      fs/locks: change all *_conflict() functions to return bool.
      fs/locks: create a tree of dependent requests.

 fs/cifs/file.c                  |    2 -
 fs/locks.c                      |  156 ++++++++++++++++++++++++++-------------
 include/linux/fs.h              |    7 +-
 include/trace/events/filelock.h |   16 ++--
 4 files changed, 119 insertions(+), 62 deletions(-)


             reply	other threads:[~2018-08-14  3:56 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-08-14  3:56 NeilBrown [this message]
2018-08-14  3:56 ` [PATCH 5/5] fs/locks: create a tree of dependent requests NeilBrown
2018-08-14  3:56 ` [PATCH 3/5] fs/locks: allow a lock request to block other requests NeilBrown
2018-08-14  3:56 ` [PATCH 2/5] fs/locks: split out __locks_wake_up_blocks() NeilBrown
2018-08-14  3:56 ` [PATCH 1/5] fs/locks: rename some lists and pointers NeilBrown
2018-08-14  3:56 ` [PATCH 4/5] fs/locks: change all *_conflict() functions to return bool NeilBrown
2018-08-14 18:41 ` [PATCH 0/5 v2] locks: avoid thundering-herd wake-ups J. Bruce Fields
2018-08-14 19:12   ` Jeff Layton

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=153421852728.24426.2111161640156686201.stgit@noble \ \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).