From: Sultan Alsawaf <sultan@kerneltoast.com>
To: unlisted-recipients:; (no To-header on input)
Cc: Sultan Alsawaf <sultan@kerneltoast.com>,
Jason Gunthorpe <jgg@ziepe.ca>,
"Martin K. Petersen" <martin.petersen@oracle.com>,
Thomas Hellstrom <thellstrom@vmware.com>,
Palmer Dabbelt <palmer@sifive.com>,
Sakari Ailus <sakari.ailus@linux.intel.com>,
Ming Lei <ming.lei@redhat.com>,
Gal Pressman <galpress@amazon.com>,
linux-kernel@vger.kernel.org
Subject: [PATCH] scatterlist: Speed up for_each_sg() loop macro
Date: Fri, 25 Oct 2019 14:33:58 -0700 [thread overview]
Message-ID: <20191025213359.7538-1-sultan@kerneltoast.com> (raw)
From: Sultan Alsawaf <sultan@kerneltoast.com>
Scatterlists are chained in predictable arrays of up to
SG_MAX_SINGLE_ALLOC sg structs in length. Using this knowledge, speed up
for_each_sg() by using constant operations to determine when to simply
increment the sg pointer by one or get the next sg array in the chain.
Rudimentary measurements with a trivial loop body show that this yields
roughly a 2x performance gain.
The following simple test module proves the correctness of the new loop
definition by testing all the different edge cases of sg chains:
#include <linux/module.h>
#include <linux/scatterlist.h>
#include <linux/slab.h>
static int __init test_for_each_sg(void)
{
static const gfp_t gfp_flags = GFP_KERNEL | __GFP_NOFAIL;
struct scatterlist *sg;
struct sg_table *table;
long old = 0, new = 0;
unsigned int i, nents;
table = kmalloc(sizeof(*table), gfp_flags);
for (nents = 1; nents <= 3 * SG_MAX_SINGLE_ALLOC; nents++) {
BUG_ON(sg_alloc_table(table, nents, gfp_flags));
for (sg = table->sgl; sg; sg = sg_next(sg))
old ^= (long)sg;
for_each_sg(table->sgl, sg, nents, i)
new ^= (long)sg;
sg_free_table(table);
}
BUG_ON(old != new);
kfree(table);
return 0;
}
module_init(test_for_each_sg);
Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
---
include/linux/scatterlist.h | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/include/linux/scatterlist.h b/include/linux/scatterlist.h
index 556ec1ea2574..73f7fd6702d7 100644
--- a/include/linux/scatterlist.h
+++ b/include/linux/scatterlist.h
@@ -146,7 +146,10 @@ static inline void sg_set_buf(struct scatterlist *sg, const void *buf,
* Loop over each sg element, following the pointer to a new list if necessary
*/
#define for_each_sg(sglist, sg, nr, __i) \
- for (__i = 0, sg = (sglist); __i < (nr); __i++, sg = sg_next(sg))
+ for (__i = 0, sg = (sglist); __i < (nr); \
+ likely(++__i % (SG_MAX_SINGLE_ALLOC - 1) || \
+ (__i + 1) >= (nr)) ? sg++ : \
+ (sg = sg_chain_ptr(sg + 1)))
/**
* sg_chain - Chain two sglists together
--
2.23.0
next reply other threads:[~2019-10-25 21:34 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-25 21:33 Sultan Alsawaf [this message]
2019-10-28 14:17 ` [PATCH] scatterlist: Speed up for_each_sg() loop macro Jason Gunthorpe
2019-10-28 16:18 ` Sultan Alsawaf
2019-10-28 16:23 ` Jason Gunthorpe
2019-10-28 16:28 ` Christoph Hellwig
2019-10-28 16:37 ` Sultan Alsawaf
2019-10-28 16:29 ` Sultan Alsawaf
2019-10-28 23:46 ` kbuild test robot
2019-10-29 2:25 ` Ming Lei
2019-11-01 3:33 ` [scatterlist] 8f39742f03: suspend_stress.fail kernel test robot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191025213359.7538-1-sultan@kerneltoast.com \
--to=sultan@kerneltoast.com \
--cc=galpress@amazon.com \
--cc=jgg@ziepe.ca \
--cc=linux-kernel@vger.kernel.org \
--cc=martin.petersen@oracle.com \
--cc=ming.lei@redhat.com \
--cc=palmer@sifive.com \
--cc=sakari.ailus@linux.intel.com \
--cc=thellstrom@vmware.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).