All of lore.kernel.org
 help / color / mirror / Atom feed
From: John Sperbeck <jsperbeck@google.com>
To: Bean Huo <beanhuo@micron.com>, Sagi Grimberg <sagi@grimberg.me>
Cc: khazhy@google.com, Jens Axboe <axboe@kernel.dk>, stable@vger.kernel.org
Subject: Crash in NVME tracing on 5.10LTS
Date: Tue,  9 Jan 2024 10:17:22 -0800	[thread overview]
Message-ID: <20240109181722.228783-1-jsperbeck@google.com> (raw)

With 5.10LTS (e.g., 5.10.206), on a machine using an NVME device, the
following tracing commands will trigger a crash due to a NULL pointer
dereference:

KDIR=/sys/kernel/debug/tracing
echo 1 > $KDIR/tracing_on
echo 1 > $KDIR/events/nvme/enable
echo "Waiting for trace events..."
cat $KDIR/trace_pipe

The backtrace looks something like this:

Call Trace:
 <IRQ>
 ? __die_body+0x6b/0xb0
 ? __die+0x9e/0xb0
 ? no_context+0x3eb/0x460
 ? ttwu_do_activate+0xf0/0x120
 ? __bad_area_nosemaphore+0x157/0x200
 ? select_idle_sibling+0x2f/0x410
 ? bad_area_nosemaphore+0x13/0x20
 ? do_user_addr_fault+0x2ab/0x360
 ? exc_page_fault+0x69/0x180
 ? asm_exc_page_fault+0x1e/0x30
 ? trace_event_raw_event_nvme_complete_rq+0xba/0x170
 ? trace_event_raw_event_nvme_complete_rq+0xa3/0x170
 nvme_complete_rq+0x168/0x170
 nvme_pci_complete_rq+0x16c/0x1f0
 nvme_handle_cqe+0xde/0x190
 nvme_irq+0x78/0x100
 __handle_irq_event_percpu+0x77/0x1e0
 handle_irq_event+0x54/0xb0
 handle_edge_irq+0xdf/0x230
 asm_call_irq_on_stack+0xf/0x20
 </IRQ>
 common_interrupt+0x9e/0x150
 asm_common_interrupt+0x1e/0x40

It looks to me like these two upstream commits were backported to 5.10:

679c54f2de67 ("nvme: use command_id instead of req->tag in trace_nvme_complete_rq()")
e7006de6c238 ("nvme: code command_id with a genctr for use-after-free validation")

But they depend on this upstream commit to initialize the 'cmd' field in
some cases:

f4b9e6c90c57 ("nvme: use driver pdu command for passthrough")

Does it sound like I'm on the right track?  The 5.15LTS and later seems to be okay.

             reply	other threads:[~2024-01-09 18:17 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-01-09 18:17 John Sperbeck [this message]
2024-01-11  9:46 ` Crash in NVME tracing on 5.10LTS Greg KH
2024-01-11 17:00   ` John Sperbeck
2024-01-11 19:38     ` Jens Axboe
2024-01-13  9:34       ` Greg KH
2024-01-13 16:11         ` Jens Axboe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20240109181722.228783-1-jsperbeck@google.com \
    --to=jsperbeck@google.com \
    --cc=axboe@kernel.dk \
    --cc=beanhuo@micron.com \
    --cc=khazhy@google.com \
    --cc=sagi@grimberg.me \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.