From: Peter Zijlstra <peterz@infradead.org>
To: will@kernel.org, boqun.feng@gmail.com
Cc: linux-kernel@vger.kernel.org, x86@kernel.org,
peterz@infradead.org, mark.rutland@arm.com, elver@google.com,
keescook@chromium.org, hch@infradead.org,
torvalds@linux-foundation.org, axboe@kernel.dk
Subject: [PATCH v2 8/9] atomic,x86: Alternative atomic_*_overflow() scheme
Date: Fri, 10 Dec 2021 17:16:26 +0100 [thread overview]
Message-ID: <20211210162313.857673010@infradead.org> (raw)
In-Reply-To: 20211210161618.645249719@infradead.org
Shift the overflow range from [0,INT_MIN] to [-1,INT_MIN], this allows
optimizing atomic_inc_overflow() to use "jle" to detect increment
from free-or-negative (with -1 being the new free and it's increment
being 0 which sets ZF).
This then obviously changes atomic_dec*_overflow() since it must now
detect the 0->-1 transition rather than the 1->0. Luckily this is
reflected in the carry flag (since we need to borrow to decrement 0).
However this means decrement must now use the SUB instruction with a
literal, since DEC doesn't set CF.
This then gives the following primitives:
[-1, INT_MIN] [0, INT_MIN]
inc() inc()
lock inc %[var] mov $-1, %[reg]
jle error-free-or-negative lock xadd %[reg], %[var]
test %[reg], %[reg]
jle error-zero-or-negative
dec() dec()
lock sub $1, %[var] lock dec %[var]
jc error-to-free jle error-zero-or-negative
jl error-from-negative
dec_and_test() dec_and_test()
lock sub $1, %[var] lock dec %[var]
jc do-free jl error-from-negative
jl error-from-negative je do-free
Make sure to set ATOMIC_OVERFLOW_OFFSET to 1 such that other code
interacting with these primitives can re-center 0.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
arch/x86/include/asm/atomic.h | 20 +++++++++++++++-----
1 file changed, 15 insertions(+), 5 deletions(-)
--- a/arch/x86/include/asm/atomic.h
+++ b/arch/x86/include/asm/atomic.h
@@ -263,21 +263,31 @@ static __always_inline int arch_atomic_f
}
#define arch_atomic_fetch_xor arch_atomic_fetch_xor
-#define arch_atomic_dec_overflow(_v, _label) \
- asm_volatile_goto(LOCK_PREFIX "decl %[var]\n\t" \
+#define ATOMIC_OVERFLOW_OFFSET 1
+
+#define arch_atomic_inc_overflow(_v, _label) \
+ asm_volatile_goto(LOCK_PREFIX "incl %[var]\n\r" \
"jle %l1" \
: : [var] "m" ((_v)->counter) \
: "memory" \
: _label)
+#define arch_atomic_dec_overflow(_v, _label) \
+ asm_volatile_goto(LOCK_PREFIX "subl $1, %[var]\n\t" \
+ "jc %l1\n\t" \
+ "jl %l1" \
+ : : [var] "m" ((_v)->counter) \
+ : "memory" \
+ : _label)
+
#define arch_atomic_dec_and_test_overflow(_v, _label) \
({ \
__label__ __zero; \
__label__ __out; \
bool __ret = false; \
- asm_volatile_goto(LOCK_PREFIX "decl %[var]\n\t" \
- "jl %l2\n\t" \
- "je %l[__zero]" \
+ asm_volatile_goto(LOCK_PREFIX "subl $1, %[var]\n\t" \
+ "jc %l[__zero]\n\t" \
+ "jl %l2" \
: : [var] "m" ((_v)->counter) \
: "memory" \
: __zero, _label); \
next prev parent reply other threads:[~2021-12-10 16:27 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-12-10 16:16 [PATCH v2 0/9] refcount: Improve code-gen Peter Zijlstra
2021-12-10 16:16 ` [PATCH v2 1/9] atomic: Prepare scripts for macro ops Peter Zijlstra
2021-12-10 17:27 ` Mark Rutland
2021-12-10 17:43 ` Marco Elver
2021-12-10 16:16 ` [PATCH v2 2/9] atomic: Add xchg.tbl Peter Zijlstra
2021-12-13 9:50 ` Mark Rutland
2021-12-10 16:16 ` [PATCH v2 3/9] atomic: Introduce atomic_{inc,dec,dec_and_test}_overflow() Peter Zijlstra
2021-12-13 10:06 ` Mark Rutland
2021-12-13 10:57 ` Peter Zijlstra
2021-12-13 10:59 ` Peter Zijlstra
2021-12-13 11:09 ` Mark Rutland
2021-12-10 16:16 ` [PATCH v2 4/9] refcount: Use atomic_*_overflow() Peter Zijlstra
2021-12-13 10:35 ` Mark Rutland
2021-12-10 16:16 ` [PATCH v2 5/9] atomic,x86: Implement atomic_dec_and_test_overflow() Peter Zijlstra
2021-12-13 11:04 ` Mark Rutland
2021-12-10 16:16 ` [PATCH v2 6/9] refcount: Fix refcount_dec_not_one() Peter Zijlstra
2021-12-10 16:16 ` [PATCH v2 7/9] refcount: Prepare for atomic_*_overflow() offsets Peter Zijlstra
2021-12-10 16:16 ` Peter Zijlstra [this message]
2021-12-10 16:53 ` [PATCH v2 8/9] atomic,x86: Alternative atomic_*_overflow() scheme Linus Torvalds
2021-12-10 17:27 ` Linus Torvalds
2021-12-17 3:38 ` Herbert Xu
2021-12-13 16:43 ` Peter Zijlstra
2021-12-13 17:29 ` Marco Elver
2021-12-13 18:11 ` Linus Torvalds
2021-12-13 18:18 ` Marco Elver
2021-12-13 18:24 ` Linus Torvalds
2021-12-13 19:35 ` Marco Elver
2021-12-13 18:21 ` Linus Torvalds
2021-12-10 16:16 ` [PATCH v2 9/9] refcount: Optimize __refcount_add_not_zero(.i=1) Peter Zijlstra
2021-12-10 19:37 ` [PATCH v2 0/9] refcount: Improve code-gen Peter Zijlstra
2021-12-13 12:15 ` [PATCH v2 10/9] atomic: Document the atomic_{}_overflow() functions Peter Zijlstra
2021-12-13 12:20 ` [PATCH v2 0/9] refcount: Improve code-gen Peter Zijlstra
2021-12-13 14:42 ` Marco Elver
2021-12-13 16:11 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20211210162313.857673010@infradead.org \
--to=peterz@infradead.org \
--cc=axboe@kernel.dk \
--cc=boqun.feng@gmail.com \
--cc=elver@google.com \
--cc=hch@infradead.org \
--cc=keescook@chromium.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=torvalds@linux-foundation.org \
--cc=will@kernel.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.