From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
stable@vger.kernel.org, Andrea Righi <andrea.righi@canonical.com>,
Masami Hiramatsu <mhiramat@kernel.org>,
Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com>,
"David S. Miller" <davem@davemloft.net>,
Linus Torvalds <torvalds@linux-foundation.org>,
"Naveen N. Rao" <naveen.n.rao@linux.ibm.com>,
Peter Zijlstra <peterz@infradead.org>,
Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@kernel.org>, Sasha Levin <sashal@kernel.org>
Subject: [PATCH 4.19 38/57] kprobes: Fix potential deadlock in kprobe_optimizer()
Date: Sun, 8 Sep 2019 13:42:02 +0100 [thread overview]
Message-ID: <20190908121143.508859269@linuxfoundation.org> (raw)
In-Reply-To: <20190908121125.608195329@linuxfoundation.org>
[ Upstream commit f1c6ece23729257fb46562ff9224cf5f61b818da ]
lockdep reports the following deadlock scenario:
WARNING: possible circular locking dependency detected
kworker/1:1/48 is trying to acquire lock:
000000008d7a62b2 (text_mutex){+.+.}, at: kprobe_optimizer+0x163/0x290
but task is already holding lock:
00000000850b5e2d (module_mutex){+.+.}, at: kprobe_optimizer+0x31/0x290
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (module_mutex){+.+.}:
__mutex_lock+0xac/0x9f0
mutex_lock_nested+0x1b/0x20
set_all_modules_text_rw+0x22/0x90
ftrace_arch_code_modify_prepare+0x1c/0x20
ftrace_run_update_code+0xe/0x30
ftrace_startup_enable+0x2e/0x50
ftrace_startup+0xa7/0x100
register_ftrace_function+0x27/0x70
arm_kprobe+0xb3/0x130
enable_kprobe+0x83/0xa0
enable_trace_kprobe.part.0+0x2e/0x80
kprobe_register+0x6f/0xc0
perf_trace_event_init+0x16b/0x270
perf_kprobe_init+0xa7/0xe0
perf_kprobe_event_init+0x3e/0x70
perf_try_init_event+0x4a/0x140
perf_event_alloc+0x93a/0xde0
__do_sys_perf_event_open+0x19f/0xf30
__x64_sys_perf_event_open+0x20/0x30
do_syscall_64+0x65/0x1d0
entry_SYSCALL_64_after_hwframe+0x49/0xbe
-> #0 (text_mutex){+.+.}:
__lock_acquire+0xfcb/0x1b60
lock_acquire+0xca/0x1d0
__mutex_lock+0xac/0x9f0
mutex_lock_nested+0x1b/0x20
kprobe_optimizer+0x163/0x290
process_one_work+0x22b/0x560
worker_thread+0x50/0x3c0
kthread+0x112/0x150
ret_from_fork+0x3a/0x50
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(module_mutex);
lock(text_mutex);
lock(module_mutex);
lock(text_mutex);
*** DEADLOCK ***
As a reproducer I've been using bcc's funccount.py
(https://github.com/iovisor/bcc/blob/master/tools/funccount.py),
for example:
# ./funccount.py '*interrupt*'
That immediately triggers the lockdep splat.
Fix by acquiring text_mutex before module_mutex in kprobe_optimizer().
Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Naveen N. Rao <naveen.n.rao@linux.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Fixes: d5b844a2cf50 ("ftrace/x86: Remove possible deadlock between register_kprobe() and ftrace_run_update_code()")
Link: http://lkml.kernel.org/r/20190812184302.GA7010@xps-13
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
kernel/kprobes.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 29ff6635d2597..714d63f60460b 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -483,6 +483,7 @@ static DECLARE_DELAYED_WORK(optimizing_work, kprobe_optimizer);
*/
static void do_optimize_kprobes(void)
{
+ lockdep_assert_held(&text_mutex);
/*
* The optimization/unoptimization refers online_cpus via
* stop_machine() and cpu-hotplug modifies online_cpus.
@@ -500,9 +501,7 @@ static void do_optimize_kprobes(void)
list_empty(&optimizing_list))
return;
- mutex_lock(&text_mutex);
arch_optimize_kprobes(&optimizing_list);
- mutex_unlock(&text_mutex);
}
/*
@@ -513,6 +512,7 @@ static void do_unoptimize_kprobes(void)
{
struct optimized_kprobe *op, *tmp;
+ lockdep_assert_held(&text_mutex);
/* See comment in do_optimize_kprobes() */
lockdep_assert_cpus_held();
@@ -520,7 +520,6 @@ static void do_unoptimize_kprobes(void)
if (list_empty(&unoptimizing_list))
return;
- mutex_lock(&text_mutex);
arch_unoptimize_kprobes(&unoptimizing_list, &freeing_list);
/* Loop free_list for disarming */
list_for_each_entry_safe(op, tmp, &freeing_list, list) {
@@ -537,7 +536,6 @@ static void do_unoptimize_kprobes(void)
} else
list_del_init(&op->list);
}
- mutex_unlock(&text_mutex);
}
/* Reclaim all kprobes on the free_list */
@@ -563,6 +561,7 @@ static void kprobe_optimizer(struct work_struct *work)
{
mutex_lock(&kprobe_mutex);
cpus_read_lock();
+ mutex_lock(&text_mutex);
/* Lock modules while optimizing kprobes */
mutex_lock(&module_mutex);
@@ -590,6 +589,7 @@ static void kprobe_optimizer(struct work_struct *work)
do_free_cleaned_kprobes();
mutex_unlock(&module_mutex);
+ mutex_unlock(&text_mutex);
cpus_read_unlock();
mutex_unlock(&kprobe_mutex);
--
2.20.1
next prev parent reply other threads:[~2019-09-08 12:56 UTC|newest]
Thread overview: 69+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-09-08 12:41 [PATCH 4.19 00/57] 4.19.72-stable review Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 01/57] mld: fix memory leak in mld_del_delrec() Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 02/57] net: fix skb use after free in netpoll Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 03/57] net: sched: act_sample: fix psample group handling on overwrite Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 04/57] net_sched: fix a NULL pointer deref in ipt action Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 05/57] net: stmmac: dwmac-rk: Dont fail if phy regulator is absent Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 06/57] tcp: inherit timestamp on mtu probe Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 07/57] tcp: remove empty skb from write queue in error cases Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 08/57] net/rds: Fix info leak in rds6_inc_info_copy() Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 09/57] x86/boot: Preserve boot_params.secure_boot from sanitizing Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 10/57] spi: bcm2835aux: unifying code between polling and interrupt driven code Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 11/57] spi: bcm2835aux: remove dangerous uncontrolled read of fifo Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 12/57] spi: bcm2835aux: fix corruptions for longer spi transfers Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 13/57] net: tundra: tsi108: use spin_lock_irqsave instead of spin_lock_irq in IRQ context Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 14/57] netfilter: nf_tables: use-after-free in failing rule with bound set Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 15/57] tools: bpftool: fix error message (prog -> object) Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 16/57] hv_netvsc: Fix a warning of suspicious RCU usage Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 17/57] net: tc35815: Explicitly check NET_IP_ALIGN is not zero in tc35815_rx Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 18/57] Bluetooth: btqca: Add a short delay before downloading the NVM Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 19/57] Bluetooth: hidp: Let hidp_send_message return number of queued bytes Greg Kroah-Hartman
2019-09-09 12:15 ` Pavel Machek
2019-09-09 13:00 ` Fabian Henneke
2019-09-09 22:59 ` Greg Kroah-Hartman
2019-09-10 6:27 ` Fabian Henneke
2019-09-10 9:29 ` Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 20/57] ibmveth: Convert multicast list size for little-endian system Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 21/57] gpio: Fix build error of function redefinition Greg Kroah-Hartman
2019-09-09 12:23 ` Pavel Machek
2019-09-08 12:41 ` [PATCH 4.19 22/57] netfilter: nft_flow_offload: skip tcp rst and fin packets Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 23/57] drm/mediatek: use correct device to import PRIME buffers Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 24/57] drm/mediatek: set DMA max segment size Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 25/57] scsi: qla2xxx: Fix gnl.l memory leak on adapter init failure Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 26/57] scsi: target: tcmu: avoid use-after-free after command timeout Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 27/57] cxgb4: fix a memory leak bug Greg Kroah-Hartman
2019-09-09 13:06 ` Pavel Machek
2019-09-08 12:41 ` [PATCH 4.19 28/57] liquidio: add cleanup in octeon_setup_iq() Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 29/57] net: myri10ge: fix memory leaks Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 30/57] lan78xx: Fix " Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 31/57] vfs: fix page locking deadlocks when deduping files Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 32/57] cx82310_eth: fix a memory leak bug Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 33/57] net: kalmia: fix memory leaks Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 34/57] ibmvnic: Unmap DMA address of TX descriptor buffers after use Greg Kroah-Hartman
2019-09-08 12:41 ` [PATCH 4.19 35/57] net: cavium: fix driver name Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 36/57] wimax/i2400m: fix a memory leak bug Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 37/57] ravb: Fix use-after-free ravb_tstamp_skb Greg Kroah-Hartman
2019-09-08 12:42 ` Greg Kroah-Hartman [this message]
2019-09-08 12:42 ` [PATCH 4.19 39/57] HID: cp2112: prevent sleeping function called from invalid context Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 40/57] x86/boot/compressed/64: Fix boot on machines with broken E820 table Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 41/57] Input: hyperv-keyboard: Use in-place iterator API in the channel callback Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 42/57] Tools: hv: kvp: eliminate may be used uninitialized warning Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 43/57] nvme-multipath: fix possible I/O hang when paths are updated Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 44/57] IB/mlx4: Fix memory leaks Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 45/57] infiniband: hfi1: fix a memory leak bug Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 46/57] infiniband: hfi1: fix memory leaks Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 47/57] selftests: kvm: fix state save/load on processors without XSAVE Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 48/57] selftests/kvm: make platform_info_test pass on AMD Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 49/57] ceph: fix buffer free while holding i_ceph_lock in __ceph_setxattr() Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 50/57] ceph: fix buffer free while holding i_ceph_lock in __ceph_build_xattrs_blob() Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 51/57] ceph: fix buffer free while holding i_ceph_lock in fill_inode() Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 52/57] KVM: arm/arm64: Only skip MMIO insn once Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 53/57] afs: Fix leak in afs_lookup_cell_rcu() Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 54/57] KVM: arm/arm64: VGIC: Properly initialise private IRQ affinity Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 55/57] x86/boot/compressed/64: Fix missing initialization in find_trampoline_placement() Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 56/57] libceph: allow ceph_buffer_put() to receive a NULL ceph_buffer Greg Kroah-Hartman
2019-09-08 12:42 ` [PATCH 4.19 57/57] Revert "x86/apic: Include the LDR when clearing out APIC registers" Greg Kroah-Hartman
2019-09-09 3:12 ` [PATCH 4.19 00/57] 4.19.72-stable review Naresh Kamboju
2019-09-09 19:39 ` Guenter Roeck
2019-09-10 9:20 ` Jon Hunter
2019-09-10 9:27 ` Jack Wang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190908121143.508859269@linuxfoundation.org \
--to=gregkh@linuxfoundation.org \
--cc=andrea.righi@canonical.com \
--cc=anil.s.keshavamurthy@intel.com \
--cc=davem@davemloft.net \
--cc=linux-kernel@vger.kernel.org \
--cc=mhiramat@kernel.org \
--cc=mingo@kernel.org \
--cc=naveen.n.rao@linux.ibm.com \
--cc=peterz@infradead.org \
--cc=sashal@kernel.org \
--cc=stable@vger.kernel.org \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).