From: Peter Zijlstra <peterz@infradead.org>
To: will@kernel.org, boqun.feng@gmail.com
Cc: linux-kernel@vger.kernel.org, x86@kernel.org,
peterz@infradead.org, mark.rutland@arm.com, elver@google.com,
keescook@chromium.org, hch@infradead.org,
torvalds@linux-foundation.org, axboe@kernel.dk
Subject: [PATCH v2 9/9] refcount: Optimize __refcount_add_not_zero(.i=1)
Date: Fri, 10 Dec 2021 17:16:27 +0100 [thread overview]
Message-ID: <20211210162313.951869207@infradead.org> (raw)
In-Reply-To: 20211210161618.645249719@infradead.org
Allow the same off-by-one on the target range that refcount_inc()
already has by only testing the new value for overflow when the
increment is not constant-1.
Improves code-gen, for the common case from:
a887: 41 8b 14 24 mov (%r12),%edx
a88b: 83 fa ff cmp $0xffffffff,%edx
a88e: 74 1f je a8af <ring_buffer_get+0x3f>
a890: 8d 4a 01 lea 0x1(%rdx),%ecx
a893: 89 d0 mov %edx,%eax
a895: f0 41 0f b1 0c 24 lock cmpxchg %ecx,(%r12)
a89b: 75 35 jne a8d2 <ring_buffer_get+0x62>
a89d: 83 c2 02 add $0x2,%edx
a8a0: 09 ca or %ecx,%edx
a8a2: 78 19 js a8bd <ring_buffer_get+0x4d>
to:
a887: 41 8b 04 24 mov (%r12),%eax
a88b: 83 f8 ff cmp $0xffffffff,%eax
a88e: 74 1a je a8aa <ring_buffer_get+0x3a>
a890: 8d 50 01 lea 0x1(%rax),%edx
a893: f0 41 0f b1 14 24 lock cmpxchg %edx,(%r12)
a899: 75 f0 jne a88b <ring_buffer_get+0x1b>
a89b: 85 d2 test %edx,%edx
a89d: 78 19 js a8b8 <ring_buffer_get+0x48>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
include/linux/refcount.h | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
--- a/include/linux/refcount.h
+++ b/include/linux/refcount.h
@@ -161,7 +161,7 @@ static inline __must_check bool __refcou
if (oldp)
*oldp = old;
- if (unlikely(old < 0 || old + i < 0))
+ if (unlikely(old < 0 || (!(__builtin_constant_p(i) && i == 1) && old + i < 0)))
refcount_warn_saturate(r, REFCOUNT_ADD_NOT_ZERO_OVF);
return old;
next prev parent reply other threads:[~2021-12-10 16:27 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-12-10 16:16 [PATCH v2 0/9] refcount: Improve code-gen Peter Zijlstra
2021-12-10 16:16 ` [PATCH v2 1/9] atomic: Prepare scripts for macro ops Peter Zijlstra
2021-12-10 17:27 ` Mark Rutland
2021-12-10 17:43 ` Marco Elver
2021-12-10 16:16 ` [PATCH v2 2/9] atomic: Add xchg.tbl Peter Zijlstra
2021-12-13 9:50 ` Mark Rutland
2021-12-10 16:16 ` [PATCH v2 3/9] atomic: Introduce atomic_{inc,dec,dec_and_test}_overflow() Peter Zijlstra
2021-12-13 10:06 ` Mark Rutland
2021-12-13 10:57 ` Peter Zijlstra
2021-12-13 10:59 ` Peter Zijlstra
2021-12-13 11:09 ` Mark Rutland
2021-12-10 16:16 ` [PATCH v2 4/9] refcount: Use atomic_*_overflow() Peter Zijlstra
2021-12-13 10:35 ` Mark Rutland
2021-12-10 16:16 ` [PATCH v2 5/9] atomic,x86: Implement atomic_dec_and_test_overflow() Peter Zijlstra
2021-12-13 11:04 ` Mark Rutland
2021-12-10 16:16 ` [PATCH v2 6/9] refcount: Fix refcount_dec_not_one() Peter Zijlstra
2021-12-10 16:16 ` [PATCH v2 7/9] refcount: Prepare for atomic_*_overflow() offsets Peter Zijlstra
2021-12-10 16:16 ` [PATCH v2 8/9] atomic,x86: Alternative atomic_*_overflow() scheme Peter Zijlstra
2021-12-10 16:53 ` Linus Torvalds
2021-12-10 17:27 ` Linus Torvalds
2021-12-17 3:38 ` Herbert Xu
2021-12-13 16:43 ` Peter Zijlstra
2021-12-13 17:29 ` Marco Elver
2021-12-13 18:11 ` Linus Torvalds
2021-12-13 18:18 ` Marco Elver
2021-12-13 18:24 ` Linus Torvalds
2021-12-13 19:35 ` Marco Elver
2021-12-13 18:21 ` Linus Torvalds
2021-12-10 16:16 ` Peter Zijlstra [this message]
2021-12-10 19:37 ` [PATCH v2 0/9] refcount: Improve code-gen Peter Zijlstra
2021-12-13 12:15 ` [PATCH v2 10/9] atomic: Document the atomic_{}_overflow() functions Peter Zijlstra
2021-12-13 12:20 ` [PATCH v2 0/9] refcount: Improve code-gen Peter Zijlstra
2021-12-13 14:42 ` Marco Elver
2021-12-13 16:11 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20211210162313.951869207@infradead.org \
--to=peterz@infradead.org \
--cc=axboe@kernel.dk \
--cc=boqun.feng@gmail.com \
--cc=elver@google.com \
--cc=hch@infradead.org \
--cc=keescook@chromium.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=torvalds@linux-foundation.org \
--cc=will@kernel.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.