ceph-devel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Jeff Layton <jlayton@kernel.org>
To: ceph-devel@vger.kernel.org
Cc: idryomov@gmail.com, mnelson@redhat.com
Subject: [RFC PATCH 0/2] libceph: submit new messages under spinlock instead of mutex
Date: Wed, 15 Sep 2021 09:26:54 -0400	[thread overview]
Message-ID: <20210915132656.30347-1-jlayton@kernel.org> (raw)

Several weeks ago, I was chatting with Mark Nelson and he mentioned that
his testing showed that the kclient would plateau out at a much lower
throughput than the hardware he was using was capable of.

While I don't have any details about the bottleneck he was hitting, I
noticed that libceph is heavily serialized on mutexes, the con->mutex,
in particular. This patchset is is an attempt to get the con->mutex out
of the outgoing message submission codepath.

In addition to potentially helping performance, this may also help to
close some potential races in the cephfs client. Right now, we have to
drop some spinlocks in order to send cap messages, but this may allow
us to avoid doing that.

My main concern with the set is whether this might open up some races
wrt changes to the con->state. I didn't hit any problems like that in
testing, but it's possibly an issue. We should be able to work around
that if it's a problem though, possibly by adding a new list_head to
stage submissions before adding them to the out_queue.

I've done some performance testing with the set, but the results were
inconclusive. I suspect my home test rig is hitting other bottlenecks
before con->mutex contention becomes an issue. It would be nice to do
some testing on a setup that allows for higher throughput to see if it
helps.

Comments and suggestions welcome...

Jeff Layton (2):
  libceph: defer clearing standby state to work function
  libceph: allow tasks to submit messages without taking con->mutex

 include/linux/ceph/messenger.h |  2 ++
 net/ceph/messenger.c           | 48 ++++++++++++++++++++--------------
 net/ceph/messenger_v1.c        | 35 +++++++++++++------------
 net/ceph/messenger_v2.c        |  5 ++++
 4 files changed, 54 insertions(+), 36 deletions(-)

-- 
2.31.1


             reply	other threads:[~2021-09-15 13:26 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-09-15 13:26 Jeff Layton [this message]
2021-09-15 13:26 ` [RFC PATCH 1/2] libceph: defer clearing standby state to work function Jeff Layton
2021-09-15 13:26 ` [RFC PATCH 2/2] libceph: allow tasks to submit messages without taking con->mutex Jeff Layton

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210915132656.30347-1-jlayton@kernel.org \
    --to=jlayton@kernel.org \
    --cc=ceph-devel@vger.kernel.org \
    --cc=idryomov@gmail.com \
    --cc=mnelson@redhat.com \
    --subject='Re: [RFC PATCH 0/2] libceph: submit new messages under spinlock instead of mutex' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).