Stable Archive on lore.kernel.org
 help / color / Atom feed
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	stable@vger.kernel.org, Paolo Valente <paolo.valente@unimore.it>,
	Douglas Anderson <dianders@chromium.org>,
	Jens Axboe <axboe@kernel.dk>
Subject: [PATCH 5.2 12/61] block, bfq: NULL out the bic when its no longer valid
Date: Fri, 12 Jul 2019 14:19:25 +0200
Message-ID: <20190712121621.281677748@linuxfoundation.org> (raw)
In-Reply-To: <20190712121620.632595223@linuxfoundation.org>

From: Douglas Anderson <dianders@chromium.org>

commit dbc3117d4ca9e17819ac73501e914b8422686750 upstream.

In reboot tests on several devices we were seeing a "use after free"
when slub_debug or KASAN was enabled.  The kernel complained about:

  Unable to handle kernel paging request at virtual address 6b6b6c2b

...which is a classic sign of use after free under slub_debug.  The
stack crawl in kgdb looked like:

 0  test_bit (addr=<optimized out>, nr=<optimized out>)
 1  bfq_bfqq_busy (bfqq=<optimized out>)
 2  bfq_select_queue (bfqd=<optimized out>)
 3  __bfq_dispatch_request (hctx=<optimized out>)
 4  bfq_dispatch_request (hctx=<optimized out>)
 5  0xc056ef00 in blk_mq_do_dispatch_sched (hctx=0xed249440)
 6  0xc056f728 in blk_mq_sched_dispatch_requests (hctx=0xed249440)
 7  0xc0568d24 in __blk_mq_run_hw_queue (hctx=0xed249440)
 8  0xc0568d94 in blk_mq_run_work_fn (work=<optimized out>)
 9  0xc024c5c4 in process_one_work (worker=0xec6d4640, work=0xed249480)
 10 0xc024cff4 in worker_thread (__worker=0xec6d4640)

Digging in kgdb, it could be found that, though bfqq looked fine,
bfqq->bic had been freed.

Through further digging, I postulated that perhaps it is illegal to
access a "bic" (AKA an "icq") after bfq_exit_icq() had been called
because the "bic" can be freed at some point in time after this call
is made.  I confirmed that there certainly were cases where the exact
crashing code path would access the "bic" after bfq_exit_icq() had
been called.  Sspecifically I set the "bfqq->bic" to (void *)0x7 and
saw that the bic was 0x7 at the time of the crash.

To understand a bit more about why this crash was fairly uncommon (I
saw it only once in a few hundred reboots), you can see that much of
the time bfq_exit_icq_fbqq() fully frees the bfqq and thus it can't
access the ->bic anymore.  The only case it doesn't is if
bfq_put_queue() sees a reference still held.

However, even in the case when bfqq isn't freed, the crash is still
rare.  Why?  I tracked what happened to the "bic" after the exit
routine.  It doesn't get freed right away.  Rather,
put_io_context_active() eventually called put_io_context() which
queued up freeing on a workqueue.  The freeing then actually happened
later than that through call_rcu().  Despite all these delays, some
extra debugging showed that all the hoops could be jumped through in
time and the memory could be freed causing the original crash.  Phew!

To make a long story short, assuming it truly is illegal to access an
icq after the "exit_icq" callback is finished, this patch is needed.

Cc: stable@vger.kernel.org
Reviewed-by: Paolo Valente <paolo.valente@unimore.it>
Signed-off-by: Douglas Anderson <dianders@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

---
 block/bfq-iosched.c |    1 +
 1 file changed, 1 insertion(+)

--- a/block/bfq-iosched.c
+++ b/block/bfq-iosched.c
@@ -4584,6 +4584,7 @@ static void bfq_exit_icq_bfqq(struct bfq
 		unsigned long flags;
 
 		spin_lock_irqsave(&bfqd->lock, flags);
+		bfqq->bic = NULL;
 		bfq_exit_bfqq(bfqd, bfqq);
 		bic_set_bfqq(bic, NULL, is_sync);
 		spin_unlock_irqrestore(&bfqd->lock, flags);



  parent reply index

Thread overview: 80+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-07-12 12:19 [PATCH 5.2 00/61] 5.2.1-stable review Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 01/61] crypto: talitos - fix hash on SEC1 Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 02/61] crypto: lrw - use correct alignmask Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 03/61] crypto: talitos - rename alternative AEAD algos Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 04/61] fscrypt: dont set policy for a dead directory Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 05/61] udf: Fix incorrect final NOT_ALLOCATED (hole) extent length Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 06/61] media: stv0297: fix frequency range limit Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 07/61] ALSA: usb-audio: Fix parse of UAC2 Extension Units Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 08/61] ALSA: hda/realtek - Headphone Mic cant record after S3 Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 09/61] tpm: Actually fail on TPM errors during "get random" Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 10/61] tpm: Fix TPM 1.2 Shutdown sequence to prevent future TPM operations Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 11/61] block: fix .bi_size overflow Greg Kroah-Hartman
2019-07-12 12:19 ` Greg Kroah-Hartman [this message]
2019-07-12 12:19 ` [PATCH 5.2 13/61] perf intel-pt: Fix itrace defaults for perf script Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 14/61] perf auxtrace: " Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 15/61] perf intel-pt: Fix itrace defaults for perf script intel-pt documentation Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 16/61] perf pmu: Fix uncore PMU alias list for ARM64 Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 17/61] perf thread-stack: Fix thread stack return from kernel for kernel-only case Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 18/61] perf header: Assign proper ff->ph in perf_event__synthesize_features() Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 19/61] x86/ptrace: Fix possible spectre-v1 in ptrace_get_debugreg() Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 20/61] x86/tls: Fix possible spectre-v1 in do_get_thread_area() Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 21/61] Documentation: Add section about CPU vulnerabilities for Spectre Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 22/61] Documentation/admin: Remove the vsyscall=native documentation Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 23/61] mwifiex: Dont abort on small, spec-compliant vendor IEs Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 24/61] USB: serial: ftdi_sio: add ID for isodebug v1 Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 25/61] USB: serial: option: add support for GosunCn ME3630 RNDIS mode Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 26/61] Revert "serial: 8250: Dont service RX FIFO if interrupts are disabled" Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 27/61] p54usb: Fix race between disconnect and firmware loading Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 28/61] usb: gadget: f_fs: data_len used before properly set Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 29/61] usb: gadget: ether: Fix race between gether_disconnect and rx_submit Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 30/61] usb: dwc2: use a longer AHB idle timeout in dwc2_core_reset() Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 31/61] usb: renesas_usbhs: add a workaround for a race condition of workqueue Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 32/61] drivers/usb/typec/tps6598x.c: fix portinfo width Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 33/61] drivers/usb/typec/tps6598x.c: fix 4CC cmd write Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 34/61] p54: fix crash during initialization Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 35/61] staging: comedi: dt282x: fix a null pointer deref on interrupt Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 36/61] staging: wilc1000: fix error path cleanup in wilc_wlan_initialize() Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 37/61] staging: bcm2835-camera: Restore return behavior of ctrl_set_bitrate() Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 38/61] staging: comedi: amplc_pci230: fix null pointer deref on interrupt Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 39/61] staging: mt7621-pci: fix PCIE_FTS_NUM_LO macro Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 40/61] HID: Add another Primax PIXART OEM mouse quirk Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 41/61] lkdtm: support llvm-objcopy Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 42/61] binder: fix memory leak in error path Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 43/61] binder: return errors from buffer copy functions Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 44/61] iio: adc: stm32-adc: add missing vdda-supply Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 45/61] coresight: Potential uninitialized variable in probe() Greg Kroah-Hartman
2019-07-12 12:19 ` [PATCH 5.2 46/61] coresight: etb10: Do not call smp_processor_id from preemptible Greg Kroah-Hartman
2019-07-12 12:20 ` [PATCH 5.2 47/61] coresight: tmc-etr: Do not call smp_processor_id() " Greg Kroah-Hartman
2019-07-12 12:20 ` [PATCH 5.2 48/61] coresight: tmc-etr: alloc_perf_buf: Do not call smp_processor_id " Greg Kroah-Hartman
2019-07-12 12:20 ` [PATCH 5.2 49/61] coresight: tmc-etf: " Greg Kroah-Hartman
2019-07-12 12:20 ` [PATCH 5.2 50/61] carl9170: fix misuse of device driver API Greg Kroah-Hartman
2019-07-12 12:20 ` [PATCH 5.2 51/61] Revert "x86/build: Move _etext to actual end of .text" Greg Kroah-Hartman
2019-07-12 12:20 ` [PATCH 5.2 52/61] VMCI: Fix integer overflow in VMCI handle arrays Greg Kroah-Hartman
2019-07-12 12:20 ` [PATCH 5.2 53/61] staging: vchiq_2835_arm: revert "quit using custom down_interruptible()" Greg Kroah-Hartman
2019-07-12 12:20 ` [PATCH 5.2 54/61] staging: vchiq: make wait events interruptible Greg Kroah-Hartman
2019-07-12 12:20 ` [PATCH 5.2 55/61] staging: vchiq: revert "switch to wait_for_completion_killable" Greg Kroah-Hartman
2019-07-12 12:20 ` [PATCH 5.2 56/61] staging: fsl-dpaa2/ethsw: fix memory leak of switchdev_work Greg Kroah-Hartman
2019-07-12 12:20 ` [PATCH 5.2 57/61] staging: bcm2835-camera: Replace spinlock protecting context_map with mutex Greg Kroah-Hartman
2019-07-12 12:20 ` [PATCH 5.2 58/61] staging: bcm2835-camera: Ensure all buffers are returned on disable Greg Kroah-Hartman
2019-07-12 12:20 ` [PATCH 5.2 59/61] staging: bcm2835-camera: Remove check of the number of buffers supplied Greg Kroah-Hartman
2019-07-12 12:20 ` [PATCH 5.2 60/61] staging: bcm2835-camera: Handle empty EOS buffers whilst streaming Greg Kroah-Hartman
2019-07-12 12:20 ` [PATCH 5.2 61/61] staging: rtl8712: reduce stack usage, again Greg Kroah-Hartman
2019-07-12 13:36 ` [PATCH 5.2 00/61] 5.2.1-stable review Jon Hunter
2019-07-12 15:31   ` Greg Kroah-Hartman
2019-07-12 16:10     ` Jon Hunter
2019-07-12 16:51       ` Greg Kroah-Hartman
2019-07-12 17:00 ` Major Hayden
2019-07-12 17:17   ` Dan Rue
2019-07-12 22:07 ` shuah
2019-07-13  8:23   ` Greg Kroah-Hartman
2019-07-13  3:04 ` Naresh Kamboju
2019-07-13 14:32   ` Greg Kroah-Hartman
2019-07-13 10:15 ` Shreeya Patel
2019-07-13 15:21   ` Greg Kroah-Hartman
2019-07-13 20:37 ` Luke Nowakowski-Krijger
2019-07-14  6:02   ` Greg Kroah-Hartman
2019-07-13 22:04 ` Guenter Roeck
2019-07-14  6:02   ` Greg Kroah-Hartman
2019-07-14  5:35 ` Kelsey Skunberg
2019-07-14  6:02   ` Greg Kroah-Hartman

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190712121621.281677748@linuxfoundation.org \
    --to=gregkh@linuxfoundation.org \
    --cc=axboe@kernel.dk \
    --cc=dianders@chromium.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=paolo.valente@unimore.it \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Stable Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/stable/0 stable/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 stable stable/ https://lore.kernel.org/stable \
		stable@vger.kernel.org
	public-inbox-index stable

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.stable


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git