linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	stable@vger.kernel.org, Ingo Molnar <mingo@redhat.com>,
	Xunlei Pang <xlpang@linux.alibaba.com>,
	yinbinbin <yinbinbin@alibabacloud.com>,
	Wetp Zhang <wetp.zy@linux.alibaba.com>,
	James Wang <jnwang@linux.alibaba.com>,
	Liangyan <liangyan.peng@linux.alibaba.com>,
	"Steven Rostedt (VMware)" <rostedt@goodmis.org>
Subject: [PATCH 4.14 49/49] tracing: Correct the length check which causes memory corruption
Date: Mon, 14 Jun 2021 12:27:42 +0200	[thread overview]
Message-ID: <20210614102643.468107408@linuxfoundation.org> (raw)
In-Reply-To: <20210614102641.857724541@linuxfoundation.org>

From: Liangyan <liangyan.peng@linux.alibaba.com>

commit 3e08a9f9760f4a70d633c328a76408e62d6f80a3 upstream.

We've suffered from severe kernel crashes due to memory corruption on
our production environment, like,

Call Trace:
[1640542.554277] general protection fault: 0000 [#1] SMP PTI
[1640542.554856] CPU: 17 PID: 26996 Comm: python Kdump: loaded Tainted:G
[1640542.556629] RIP: 0010:kmem_cache_alloc+0x90/0x190
[1640542.559074] RSP: 0018:ffffb16faa597df8 EFLAGS: 00010286
[1640542.559587] RAX: 0000000000000000 RBX: 0000000000400200 RCX:
0000000006e931bf
[1640542.560323] RDX: 0000000006e931be RSI: 0000000000400200 RDI:
ffff9a45ff004300
[1640542.560996] RBP: 0000000000400200 R08: 0000000000023420 R09:
0000000000000000
[1640542.561670] R10: 0000000000000000 R11: 0000000000000000 R12:
ffffffff9a20608d
[1640542.562366] R13: ffff9a45ff004300 R14: ffff9a45ff004300 R15:
696c662f65636976
[1640542.563128] FS:  00007f45d7c6f740(0000) GS:ffff9a45ff840000(0000)
knlGS:0000000000000000
[1640542.563937] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1640542.564557] CR2: 00007f45d71311a0 CR3: 000000189d63e004 CR4:
00000000003606e0
[1640542.565279] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[1640542.566069] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[1640542.566742] Call Trace:
[1640542.567009]  anon_vma_clone+0x5d/0x170
[1640542.567417]  __split_vma+0x91/0x1a0
[1640542.567777]  do_munmap+0x2c6/0x320
[1640542.568128]  vm_munmap+0x54/0x70
[1640542.569990]  __x64_sys_munmap+0x22/0x30
[1640542.572005]  do_syscall_64+0x5b/0x1b0
[1640542.573724]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[1640542.575642] RIP: 0033:0x7f45d6e61e27

James Wang has reproduced it stably on the latest 4.19 LTS.
After some debugging, we finally proved that it's due to ftrace
buffer out-of-bound access using a debug tool as follows:
[   86.775200] BUG: Out-of-bounds write at addr 0xffff88aefe8b7000
[   86.780806]  no_context+0xdf/0x3c0
[   86.784327]  __do_page_fault+0x252/0x470
[   86.788367]  do_page_fault+0x32/0x140
[   86.792145]  page_fault+0x1e/0x30
[   86.795576]  strncpy_from_unsafe+0x66/0xb0
[   86.799789]  fetch_memory_string+0x25/0x40
[   86.804002]  fetch_deref_string+0x51/0x60
[   86.808134]  kprobe_trace_func+0x32d/0x3a0
[   86.812347]  kprobe_dispatcher+0x45/0x50
[   86.816385]  kprobe_ftrace_handler+0x90/0xf0
[   86.820779]  ftrace_ops_assist_func+0xa1/0x140
[   86.825340]  0xffffffffc00750bf
[   86.828603]  do_sys_open+0x5/0x1f0
[   86.832124]  do_syscall_64+0x5b/0x1b0
[   86.835900]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

commit b220c049d519 ("tracing: Check length before giving out
the filter buffer") adds length check to protect trace data
overflow introduced in 0fc1b09ff1ff, seems that this fix can't prevent
overflow entirely, the length check should also take the sizeof
entry->array[0] into account, since this array[0] is filled the
length of trace data and occupy addtional space and risk overflow.

Link: https://lkml.kernel.org/r/20210607125734.1770447-1-liangyan.peng@linux.alibaba.com

Cc: stable@vger.kernel.org
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Xunlei Pang <xlpang@linux.alibaba.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Fixes: b220c049d519 ("tracing: Check length before giving out the filter buffer")
Reviewed-by: Xunlei Pang <xlpang@linux.alibaba.com>
Reviewed-by: yinbinbin <yinbinbin@alibabacloud.com>
Reviewed-by: Wetp Zhang <wetp.zy@linux.alibaba.com>
Tested-by: James Wang <jnwang@linux.alibaba.com>
Signed-off-by: Liangyan <liangyan.peng@linux.alibaba.com>
Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
 kernel/trace/trace.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -2274,7 +2274,7 @@ trace_event_buffer_lock_reserve(struct r
 	    (entry = this_cpu_read(trace_buffered_event))) {
 		/* Try to use the per cpu buffer first */
 		val = this_cpu_inc_return(trace_buffered_event_cnt);
-		if ((len < (PAGE_SIZE - sizeof(*entry))) && val == 1) {
+		if ((len < (PAGE_SIZE - sizeof(*entry) - sizeof(entry->array[0]))) && val == 1) {
 			trace_event_setup(entry, type, flags, pc);
 			entry->array[0] = len;
 			return entry;



  parent reply	other threads:[~2021-06-14 10:43 UTC|newest]

Thread overview: 53+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-14 10:26 [PATCH 4.14 00/49] 4.14.237-rc1 review Greg Kroah-Hartman
2021-06-14 10:26 ` [PATCH 4.14 01/49] proc: Track /proc/$pid/attr/ opener mm_struct Greg Kroah-Hartman
2021-06-14 10:26 ` [PATCH 4.14 02/49] net/nfc/rawsock.c: fix a permission check bug Greg Kroah-Hartman
2021-06-14 10:26 ` [PATCH 4.14 03/49] ASoC: sti-sas: add missing MODULE_DEVICE_TABLE Greg Kroah-Hartman
2021-06-14 10:26 ` [PATCH 4.14 04/49] isdn: mISDN: netjet: Fix crash in nj_probe: Greg Kroah-Hartman
2021-06-14 10:26 ` [PATCH 4.14 05/49] bonding: init notify_work earlier to avoid uninitialized use Greg Kroah-Hartman
2021-06-14 10:26 ` [PATCH 4.14 06/49] netlink: disable IRQs for netlink_lock_table() Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 07/49] net: mdiobus: get rid of a BUG_ON() Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 08/49] cgroup: disable controllers at parse time Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 09/49] wq: handle VM suspension in stall detection Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 10/49] net/qla3xxx: fix schedule while atomic in ql_sem_spinlock Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 11/49] scsi: vmw_pvscsi: Set correct residual data length Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 12/49] scsi: target: qla2xxx: Wait for stop_phase1 at WWN removal Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 13/49] net: macb: ensure the device is available before accessing GEMGXL control registers Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 14/49] net: appletalk: cops: Fix data race in cops_probe1 Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 15/49] MIPS: Fix kernel hang under FUNCTION_GRAPH_TRACER and PREEMPT_TRACER Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 16/49] bnx2x: Fix missing error code in bnx2x_iov_init_one() Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 17/49] powerpc/fsl: set fsl,i2c-erratum-a004447 flag for P2041 i2c controllers Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 18/49] powerpc/fsl: set fsl,i2c-erratum-a004447 flag for P1010 " Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 19/49] i2c: mpc: Make use of i2c_recover_bus() Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 20/49] i2c: mpc: implement erratum A-004447 workaround Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 21/49] drm: Fix use-after-free read in drm_getunique() Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 22/49] drm: Lock pointer access in drm_master_release() Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 23/49] kvm: avoid speculation-based attacks from out-of-range memslot accesses Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 24/49] staging: rtl8723bs: Fix uninitialized variables Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 25/49] btrfs: return value from btrfs_mark_extent_written() in case of error Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 26/49] cgroup1: dont allow \n in renaming Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 27/49] USB: f_ncm: ncm_bitrate (speed) is unsigned Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 28/49] usb: dwc3: ep0: fix NULL pointer exception Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 29/49] usb: typec: ucsi: Clear PPM capability data in ucsi_init() error path Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 30/49] usb: gadget: f_fs: Ensure io_completion_wq is idle during unbind Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 31/49] USB: serial: ftdi_sio: add NovaTech OrionMX product ID Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 32/49] USB: serial: omninet: add device id for Zyxel Omni 56K Plus Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 33/49] USB: serial: quatech2: fix control-request directions Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 34/49] usb: gadget: eem: fix wrong eem header operation Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 35/49] usb: fix various gadgets null ptr deref on 10gbps cabling Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 36/49] usb: fix various gadget panics " Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 37/49] regulator: core: resolve supply for boot-on/always-on regulators Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 38/49] regulator: max77620: Use device_set_of_node_from_dev() Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 39/49] perf: Fix data race between pin_count increment/decrement Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 40/49] NFS: Fix a potential NULL dereference in nfs_get_client() Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 41/49] perf session: Correct buffer copying when peeking events Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 42/49] kvm: fix previous commit for 32-bit builds Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 43/49] NFS: Fix use-after-free in nfs4_init_client() Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 44/49] NFSv4: nfs4_proc_set_acl needs to restore NFS_CAP_UIDGID_NOMAP on error Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 45/49] scsi: core: Fix error handling of scsi_host_alloc() Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 46/49] scsi: core: Put .shost_dev in failure path if host state changes to RUNNING Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 47/49] scsi: core: Only put parent device if host state differs from SHOST_CREATED Greg Kroah-Hartman
2021-06-14 10:27 ` [PATCH 4.14 48/49] ftrace: Do not blindly read the ip address in ftrace_bug() Greg Kroah-Hartman
2021-06-14 10:27 ` Greg Kroah-Hartman [this message]
2021-06-15  1:03 ` [PATCH 4.14 00/49] 4.14.237-rc1 review Samuel Zou
2021-06-15  9:53 ` Naresh Kamboju
2021-06-15 14:20 ` Guenter Roeck

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210614102643.468107408@linuxfoundation.org \
    --to=gregkh@linuxfoundation.org \
    --cc=jnwang@linux.alibaba.com \
    --cc=liangyan.peng@linux.alibaba.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@redhat.com \
    --cc=rostedt@goodmis.org \
    --cc=stable@vger.kernel.org \
    --cc=wetp.zy@linux.alibaba.com \
    --cc=xlpang@linux.alibaba.com \
    --cc=yinbinbin@alibabacloud.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).