All of lore.kernel.org
 help / color / mirror / Atom feed
From: Robert LeBlanc <robert@leblancnet.us>
To: linux-scsi@vger.kernel.org
Subject: Performance degradation on 4.6
Date: Tue, 31 May 2016 17:46:54 -0600	[thread overview]
Message-ID: <CAANLjFp28MkaMjWKwXnMGHPCX2S_g_RKOhXqHc9dYVJvMgqvVw@mail.gmail.com> (raw)

I've been working with iSER trying to maximize performance. I have two
FDR Infiniband cards (three ports total) and I've created multiple
address on the target and configured 12 paths from the initiator. The
iSER target is served up from a RAM disk. With the 4.4 kernel I'm able
to pretty much saturate the links (4.1 million seq 4K read IOPs 15.7
GB/s) using multipathd. With the same configuration on 4.6, I'm only
able to get 3.3 million IOPs or 12.8 GB/s.

Running perf top on the target and initiator, I'm seeing the target is
dominated by

73.89% [kernel] [k] memcpy_rems
 1.35% [kernel] [k] get_page_from_freelist
...

which is the same with 4.4. Perf top on the target has:

13.01% [kernel] [k] copy_user_enhanced_fast_string
 8.48% [kernel] [K] __blk_recalc_re_segments
 5.35% [kernel] [k] native_queued_spin_lock_slowpath
...

I don't recall what the initiator had on 4.4.

Single path performance has increase by about 5% which is nice, but
two path performance is about 50% that of the 4.4 kernel. Three paths
and greater are around 80% the 4.4 performance.

Any ideas on what could have caused the performance issues on 4.6? Is
there some additional information I can provide that would be useful?

Thanks,

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1

             reply	other threads:[~2016-05-31 23:46 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-05-31 23:46 Robert LeBlanc [this message]
2016-06-01 21:00 ` Performance degradation on 4.6 Nicholas A. Bellinger
2016-06-02 11:37   ` Christoph Hellwig
2016-06-02 15:18     ` Robert LeBlanc

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAANLjFp28MkaMjWKwXnMGHPCX2S_g_RKOhXqHc9dYVJvMgqvVw@mail.gmail.com \
    --to=robert@leblancnet.us \
    --cc=linux-scsi@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.