From: Uros Bizjak <ubizjak@gmail.com>
To: x86@kernel.org, linux-kernel@vger.kernel.org
Cc: Uros Bizjak <ubizjak@gmail.com>,
Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@kernel.org>, Borislav Petkov <bp@alien8.de>,
Dave Hansen <dave.hansen@linux.intel.com>,
"H. Peter Anvin" <hpa@zytor.com>,
Peter Zijlstra <peterz@infradead.org>
Subject: [PATCH 3/3] locking/atomic/x86: Introduce arch_try_cmpxchg64() for !CONFIG_X86_CMPXCHG64
Date: Mon, 8 Apr 2024 11:13:58 +0200 [thread overview]
Message-ID: <20240408091547.90111-4-ubizjak@gmail.com> (raw)
In-Reply-To: <20240408091547.90111-1-ubizjak@gmail.com>
Commit:
6d12c8d308e68 ("percpu: Wire up cmpxchg128")
improved emulated cmpxchg8b_emu() library function to return
success/failure in a ZF flag.
Define arch_try_cmpxchg64() for !CONFIG_X86_CMPXCHG64 targets
to override the generic archy_try_cmpxchg() with an optimized
target specific implementation that handles ZF flag.
The assembly code at the call sites improves from:
bf56d: e8 fc ff ff ff call cmpxchg8b_emu
bf572: 8b 74 24 28 mov 0x28(%esp),%esi
bf576: 89 c3 mov %eax,%ebx
bf578: 89 d1 mov %edx,%ecx
bf57a: 8b 7c 24 2c mov 0x2c(%esp),%edi
bf57e: 89 f0 mov %esi,%eax
bf580: 89 fa mov %edi,%edx
bf582: 31 d8 xor %ebx,%eax
bf584: 31 ca xor %ecx,%edx
bf586: 09 d0 or %edx,%eax
bf588: 0f 84 e3 01 00 00 je bf771 <...>
to:
bf572: e8 fc ff ff ff call cmpxchg8b_emu
bf577: 0f 84 b6 01 00 00 je bf733 <...>
No functional changes intended.
Signed-off-by: Uros Bizjak <ubizjak@gmail.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Peter Zijlstra <peterz@infradead.org>
---
arch/x86/include/asm/cmpxchg_32.h | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/arch/x86/include/asm/cmpxchg_32.h b/arch/x86/include/asm/cmpxchg_32.h
index fe40d0681ea8..9e0d330dd5d0 100644
--- a/arch/x86/include/asm/cmpxchg_32.h
+++ b/arch/x86/include/asm/cmpxchg_32.h
@@ -122,6 +122,34 @@ static __always_inline u64 arch_cmpxchg64_local(volatile u64 *ptr, u64 old, u64
}
#define arch_cmpxchg64_local arch_cmpxchg64_local
+#define __arch_try_cmpxchg64_emu(_ptr, _oldp, _new) \
+({ \
+ union __u64_halves o = { .full = *(_oldp), }, \
+ n = { .full = (_new), }; \
+ bool ret; \
+ \
+ asm volatile(ALTERNATIVE(LOCK_PREFIX_HERE \
+ "call cmpxchg8b_emu", \
+ "lock; cmpxchg8b %[ptr]", X86_FEATURE_CX8) \
+ CC_SET(e) \
+ : CC_OUT(e) (ret), \
+ [ptr] "+m" (*(_ptr)), \
+ "+a" (o.low), "+d" (o.high) \
+ : "b" (n.low), "c" (n.high), "S" (_ptr) \
+ : "memory"); \
+ \
+ if (unlikely(!ret)) \
+ *(_oldp) = o.full; \
+ \
+ likely(ret); \
+})
+
+static __always_inline bool arch_try_cmpxchg64(volatile u64 *ptr, u64 *oldp, u64 new)
+{
+ return __arch_try_cmpxchg64_emu(ptr, oldp, new);
+}
+#define arch_try_cmpxchg64 arch_try_cmpxchg64
+
#endif
#define system_has_cmpxchg64() boot_cpu_has(X86_FEATURE_CX8)
--
2.44.0
next prev parent reply other threads:[~2024-04-08 9:15 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-08 9:13 [PATCH 0/3] locking/atomic/x86: Improve arch_cmpxchg64() and friends for x86_32 Uros Bizjak
2024-04-08 9:13 ` [PATCH 1/3] locking/atomic/x86: Correct the definition of __arch_try_cmpxchg128() Uros Bizjak
2024-04-09 8:26 ` [tip: locking/core] " tip-bot2 for Uros Bizjak
2024-04-08 9:13 ` [PATCH 2/3] locking/atomic/x86: Modernize x86_32 arch_{,try_}_cmpxchg64{,_local}() Uros Bizjak
2024-04-09 8:26 ` [tip: locking/core] " tip-bot2 for Uros Bizjak
2024-04-08 9:13 ` Uros Bizjak [this message]
2024-04-09 7:50 ` [PATCH 3/3] locking/atomic/x86: Introduce arch_try_cmpxchg64() for !CONFIG_X86_CMPXCHG64 Ingo Molnar
2024-04-09 8:26 ` [tip: locking/core] " tip-bot2 for Uros Bizjak
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240408091547.90111-4-ubizjak@gmail.com \
--to=ubizjak@gmail.com \
--cc=bp@alien8.de \
--cc=dave.hansen@linux.intel.com \
--cc=hpa@zytor.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=peterz@infradead.org \
--cc=tglx@linutronix.de \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.