From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org, stable@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Will Deacon <will.deacon@arm.com>,
Ben Hutchings <ben@decadent.org.uk>
Subject: [PATCH 4.9 46/53] arm64: futex: Bound number of LDXR/STXR loops in FUTEX_WAKE_OP
Date: Mon, 29 Mar 2021 09:58:21 +0200 [thread overview]
Message-ID: <20210329075609.019809612@linuxfoundation.org> (raw)
In-Reply-To: <20210329075607.561619583@linuxfoundation.org>
From: Will Deacon <will.deacon@arm.com>
commit 03110a5cb2161690ae5ac04994d47ed0cd6cef75 upstream.
Our futex implementation makes use of LDXR/STXR loops to perform atomic
updates to user memory from atomic context. This can lead to latency
problems if we end up spinning around the LL/SC sequence at the expense
of doing something useful.
Rework our futex atomic operations so that we return -EAGAIN if we fail
to update the futex word after 128 attempts. The core futex code will
reschedule if necessary and we'll try again later.
Fixes: 6170a97460db ("arm64: Atomic operations")
Signed-off-by: Will Deacon <will.deacon@arm.com>
[bwh: Backported to 4.9: adjust context]
Signed-off-by: Ben Hutchings <ben@decadent.org.uk>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
arch/arm64/include/asm/futex.h | 59 +++++++++++++++++++++++++----------------
1 file changed, 37 insertions(+), 22 deletions(-)
--- a/arch/arm64/include/asm/futex.h
+++ b/arch/arm64/include/asm/futex.h
@@ -26,7 +26,12 @@
#include <asm/errno.h>
#include <asm/sysreg.h>
+#define FUTEX_MAX_LOOPS 128 /* What's the largest number you can think of? */
+
#define __futex_atomic_op(insn, ret, oldval, uaddr, tmp, oparg) \
+do { \
+ unsigned int loops = FUTEX_MAX_LOOPS; \
+ \
asm volatile( \
ALTERNATIVE("nop", SET_PSTATE_PAN(0), ARM64_HAS_PAN, \
CONFIG_ARM64_PAN) \
@@ -34,21 +39,26 @@
"1: ldxr %w1, %2\n" \
insn "\n" \
"2: stlxr %w0, %w3, %2\n" \
-" cbnz %w0, 1b\n" \
-" dmb ish\n" \
+" cbz %w0, 3f\n" \
+" sub %w4, %w4, %w0\n" \
+" cbnz %w4, 1b\n" \
+" mov %w0, %w7\n" \
"3:\n" \
+" dmb ish\n" \
" .pushsection .fixup,\"ax\"\n" \
" .align 2\n" \
-"4: mov %w0, %w5\n" \
+"4: mov %w0, %w6\n" \
" b 3b\n" \
" .popsection\n" \
_ASM_EXTABLE(1b, 4b) \
_ASM_EXTABLE(2b, 4b) \
ALTERNATIVE("nop", SET_PSTATE_PAN(1), ARM64_HAS_PAN, \
CONFIG_ARM64_PAN) \
- : "=&r" (ret), "=&r" (oldval), "+Q" (*uaddr), "=&r" (tmp) \
- : "r" (oparg), "Ir" (-EFAULT) \
- : "memory")
+ : "=&r" (ret), "=&r" (oldval), "+Q" (*uaddr), "=&r" (tmp), \
+ "+r" (loops) \
+ : "r" (oparg), "Ir" (-EFAULT), "Ir" (-EAGAIN) \
+ : "memory"); \
+} while (0)
static inline int
arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *uaddr)
@@ -59,23 +69,23 @@ arch_futex_atomic_op_inuser(int op, int
switch (op) {
case FUTEX_OP_SET:
- __futex_atomic_op("mov %w3, %w4",
+ __futex_atomic_op("mov %w3, %w5",
ret, oldval, uaddr, tmp, oparg);
break;
case FUTEX_OP_ADD:
- __futex_atomic_op("add %w3, %w1, %w4",
+ __futex_atomic_op("add %w3, %w1, %w5",
ret, oldval, uaddr, tmp, oparg);
break;
case FUTEX_OP_OR:
- __futex_atomic_op("orr %w3, %w1, %w4",
+ __futex_atomic_op("orr %w3, %w1, %w5",
ret, oldval, uaddr, tmp, oparg);
break;
case FUTEX_OP_ANDN:
- __futex_atomic_op("and %w3, %w1, %w4",
+ __futex_atomic_op("and %w3, %w1, %w5",
ret, oldval, uaddr, tmp, ~oparg);
break;
case FUTEX_OP_XOR:
- __futex_atomic_op("eor %w3, %w1, %w4",
+ __futex_atomic_op("eor %w3, %w1, %w5",
ret, oldval, uaddr, tmp, oparg);
break;
default:
@@ -95,6 +105,7 @@ futex_atomic_cmpxchg_inatomic(u32 *uval,
u32 oldval, u32 newval)
{
int ret = 0;
+ unsigned int loops = FUTEX_MAX_LOOPS;
u32 val, tmp;
u32 __user *uaddr;
@@ -106,21 +117,25 @@ futex_atomic_cmpxchg_inatomic(u32 *uval,
ALTERNATIVE("nop", SET_PSTATE_PAN(0), ARM64_HAS_PAN, CONFIG_ARM64_PAN)
" prfm pstl1strm, %2\n"
"1: ldxr %w1, %2\n"
-" sub %w3, %w1, %w4\n"
-" cbnz %w3, 3f\n"
-"2: stlxr %w3, %w5, %2\n"
-" cbnz %w3, 1b\n"
-" dmb ish\n"
+" sub %w3, %w1, %w5\n"
+" cbnz %w3, 4f\n"
+"2: stlxr %w3, %w6, %2\n"
+" cbz %w3, 3f\n"
+" sub %w4, %w4, %w3\n"
+" cbnz %w4, 1b\n"
+" mov %w0, %w8\n"
"3:\n"
+" dmb ish\n"
+"4:\n"
" .pushsection .fixup,\"ax\"\n"
-"4: mov %w0, %w6\n"
-" b 3b\n"
+"5: mov %w0, %w7\n"
+" b 4b\n"
" .popsection\n"
- _ASM_EXTABLE(1b, 4b)
- _ASM_EXTABLE(2b, 4b)
+ _ASM_EXTABLE(1b, 5b)
+ _ASM_EXTABLE(2b, 5b)
ALTERNATIVE("nop", SET_PSTATE_PAN(1), ARM64_HAS_PAN, CONFIG_ARM64_PAN)
- : "+r" (ret), "=&r" (val), "+Q" (*uaddr), "=&r" (tmp)
- : "r" (oldval), "r" (newval), "Ir" (-EFAULT)
+ : "+r" (ret), "=&r" (val), "+Q" (*uaddr), "=&r" (tmp), "+r" (loops)
+ : "r" (oldval), "r" (newval), "Ir" (-EFAULT), "Ir" (-EAGAIN)
: "memory");
*uval = val;
next prev parent reply other threads:[~2021-03-29 8:06 UTC|newest]
Thread overview: 58+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-29 7:57 [PATCH 4.9 00/53] 4.9.264-rc1 review Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 01/53] net: fec: ptp: avoid register access when ipg clock is disabled Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 02/53] powerpc/4xx: Fix build errors from mfdcr() Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 03/53] atm: eni: dont release is never initialized Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 04/53] atm: lanai: dont run lanai_dev_close if not open Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 05/53] ixgbe: Fix memleak in ixgbe_configure_clsu32 Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 06/53] net: tehuti: fix error return code in bdx_probe() Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 07/53] sun/niu: fix wrong RXMAC_BC_FRM_CNT_COUNT count Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 08/53] nfs: fix PNFS_FLEXFILE_LAYOUT Kconfig default Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 09/53] NFS: Correct size calculation for create reply length Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 10/53] net: wan: fix error return code of uhdlc_init() Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 11/53] atm: uPD98402: fix incorrect allocation Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 12/53] atm: idt77252: fix null-ptr-dereference Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 13/53] u64_stats,lockdep: Fix u64_stats_init() vs lockdep Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 14/53] nfs: we dont support removing system.nfs4_acl Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 15/53] ia64: fix ia64_syscall_get_set_arguments() for break-based syscalls Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 16/53] ia64: fix ptrace(PTRACE_SYSCALL_INFO_EXIT) sign Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 17/53] x86/tlb: Flush global mappings when KAISER is disabled Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 18/53] squashfs: fix inode lookup sanity checks Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 19/53] squashfs: fix xattr id and id " Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 20/53] arm64: dts: ls1043a: mark crypto engine dma coherent Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 21/53] bus: omap_l3_noc: mark l3 irqs as IRQF_NO_THREAD Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 22/53] macvlan: macvlan_count_rx() needs to be aware of preemption Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 23/53] net: dsa: bcm_sf2: Qualify phydev->dev_flags based on port Greg Kroah-Hartman
2021-03-29 7:57 ` [PATCH 4.9 24/53] e1000e: add rtnl_lock() to e1000_reset_task Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 25/53] e1000e: Fix error handling in e1000_set_d0_lplu_state_82571 Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 26/53] net/qlcnic: Fix a use after free in qlcnic_83xx_get_minidump_template Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 27/53] can: c_can_pci: c_can_pci_remove(): fix use-after-free Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 28/53] can: c_can: move runtime PM enable/disable to c_can_platform Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 29/53] can: m_can: m_can_do_rx_poll(): fix extraneous msg loss warning Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 30/53] mac80211: fix rate mask reset Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 31/53] net: cdc-phonet: fix data-interface release on probe failure Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 32/53] RDMA/cxgb4: Fix adapter LE hash errors while destroying ipv6 listening server Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 33/53] ACPI: scan: Rearrange memory allocation in acpi_device_add() Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 34/53] ACPI: scan: Use unique number for instance_no Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 35/53] perf auxtrace: Fix auxtrace queue conflict Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 36/53] idr: add ida_is_empty Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 37/53] futex: Use smp_store_release() in mark_wake_futex() Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 38/53] futex,rt_mutex: Introduce rt_mutex_init_waiter() Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 39/53] futex: Rework futex_lock_pi() to use rt_mutex_*_proxy_lock() Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 40/53] futex: Drop hb->lock before enqueueing on the rtmutex Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 41/53] futex: Avoid freeing an active timer Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 42/53] futex,rt_mutex: Fix rt_mutex_cleanup_proxy_lock() Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 43/53] futex: Handle early deadlock return correctly Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 44/53] futex: Fix (possible) missed wakeup Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 45/53] locking/futex: Allow low-level atomic operations to return -EAGAIN Greg Kroah-Hartman
2021-03-29 7:58 ` Greg Kroah-Hartman [this message]
2021-03-29 7:58 ` [PATCH 4.9 47/53] futex: Prevent robust futex exit race Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 48/53] futex: Fix incorrect should_fail_futex() handling Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 49/53] futex: Handle transient "ownerless" rtmutex state correctly Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 50/53] can: dev: Move device back to init netns on owning netns delete Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 51/53] net: sched: validate stab values Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 52/53] net: qrtr: fix a kernel-infoleak in qrtr_recvmsg() Greg Kroah-Hartman
2021-03-29 7:58 ` [PATCH 4.9 53/53] mac80211: fix double free in ibss_leave Greg Kroah-Hartman
2021-03-29 18:45 ` [PATCH 4.9 00/53] 4.9.264-rc1 review Florian Fainelli
2021-03-29 21:32 ` Guenter Roeck
2021-03-30 1:27 ` Shuah Khan
2021-03-30 7:05 ` Naresh Kamboju
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210329075609.019809612@linuxfoundation.org \
--to=gregkh@linuxfoundation.org \
--cc=ben@decadent.org.uk \
--cc=linux-kernel@vger.kernel.org \
--cc=stable@vger.kernel.org \
--cc=will.deacon@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).