[PATCH] delayacct: Fix crash in delayacct_blkio_end() after delayacct init failure

[PATCH] delayacct: Fix crash in delayacct_blkio_end() after delayacct init failure

[Tejun Heo]

While forking, if delayacct init fails due to memory shortage, it
continues expecting all delayacct users to check task->delays pointer
against NULL before dereferencing it, which all of them used to do.

c96f5471ce7d ("delayacct: Account blkio completion on the correct
task"), while updating delayacct_blkio_end() to take the target task
instead of always using %current, made the function test NULL on
%current->delays and then continue to operated on @p->delays.  If
%current succeeded init while @p didn't, it leads to the following
crash.

 BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
 IP: __delayacct_blkio_end+0xc/0x40
 PGD 8000001fd07e1067 P4D 8000001fd07e1067 PUD 1fcffbb067 PMD 0 
 Oops: 0000 [#1] SMP PTI
 CPU: 4 PID: 25774 Comm: QIOThread0 Not tainted 4.16.0-9_fbk1_rc2_1180_g6b593215b4d7 #9
 Hardware name: Quanta Leopard ORv2-DDR4/Leopard ORv2-DDR4, BIOS F06_3B12 08/17/2017
 RIP: 0010:__delayacct_blkio_end+0xc/0x40
 RSP: 0000:ffff881fff703bf8 EFLAGS: 00010086
 RAX: ffff881f1ec8b800 RBX: ffff8804f735cd54 RCX: ffff881fff703cb0
 RDX: 0000000000000002 RSI: 0000000000000003 RDI: 0000000000000000
 RBP: 0000000000000000 R08: 0000000000000000 R09: ffff881fff703cc0
 R10: 0000000000001000 R11: ffff881fd3f73d00 R12: ffff8804f735c600
 R13: 0000000000000000 R14: 000000000000001d R15: ffff881fff703cb0
 FS:  00007f5003f7d700(0000) GS:ffff881fff700000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 0000000000000004 CR3: 0000001f401a6006 CR4: 00000000003606e0
 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
 Call Trace:
  <IRQ>
  try_to_wake_up+0x2c0/0x600
  autoremove_wake_function+0xe/0x30
  __wake_up_common+0x74/0x120
  wake_up_page_bit+0x9c/0xe0
  mpage_end_io+0x27/0x70
  blk_update_request+0x78/0x2c0
  scsi_end_request+0x2c/0x1e0
  scsi_io_completion+0x20b/0x5f0
  blk_mq_complete_request+0xa2/0x100
  ata_scsi_qc_complete+0x79/0x400
  ata_qc_complete_multiple+0x86/0xd0
  ahci_handle_port_interrupt+0xc9/0x5c0
  ahci_handle_port_intr+0x54/0xb0
  ahci_single_level_irq_intr+0x3b/0x60
  __handle_irq_event_percpu+0x43/0x190
  handle_irq_event_percpu+0x20/0x50
  handle_irq_event+0x2a/0x50
  handle_edge_irq+0x80/0x1c0
  handle_irq+0xaf/0x120
  do_IRQ+0x41/0xc0
  common_interrupt+0xf/0xf
  </IRQ>

Fix it by updating delayacct_blkio_end() check @p->delays instead.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-and-debugged-by: Dave Jones <dsj@fb.com>
Cc: Josh Snyder <joshs@netflix.com>
Fixes: c96f5471ce7d ("delayacct: Account blkio completion on the correct task")
Cc: stable@vger.kernel.org # v4.15+
---
 include/linux/delayacct.h |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/delayacct.h b/include/linux/delayacct.h
index e6c0448ebcc7..31c865d1842e 100644
--- a/include/linux/delayacct.h
+++ b/include/linux/delayacct.h
@@ -124,7 +124,7 @@ static inline void delayacct_blkio_start(void)
 
 static inline void delayacct_blkio_end(struct task_struct *p)
 {
-	if (current->delays)
+	if (p->delays)
 		__delayacct_blkio_end(p);
 	delayacct_clear_flag(DELAYACCT_PF_BLKIO);
 }

Re: [PATCH] delayacct: Fix crash in delayacct_blkio_end() after delayacct init failure

[Andrew Morton]

On Tue, 24 Jul 2018 10:55:42 -0700 Tejun Heo <tj@kernel.org> wrote:

> While forking, if delayacct init fails due to memory shortage, it
> continues expecting all delayacct users to check task->delays pointer
> against NULL before dereferencing it, which all of them used to do.
> 
> c96f5471ce7d ("delayacct: Account blkio completion on the correct
> task"), while updating delayacct_blkio_end() to take the target task
> instead of always using %current, made the function test NULL on
> %current->delays and then continue to operated on @p->delays.  If
> %current succeeded init while @p didn't, it leads to the following
> crash.
> 

lgtm.

How did you make this happen, btw?  Fault injection, or did a small
GFP_KERNEL allocation fail?

Re: [PATCH] delayacct: Fix crash in delayacct_blkio_end() after delayacct init failure

[Tejun Heo]

Hello, Andrew.

On Tue, Jul 24, 2018 at 12:29:13PM -0700, Andrew Morton wrote:
> How did you make this happen, btw?  Fault injection, or did a small
> GFP_KERNEL allocation fail?

We have a group of machines which are pushing memory really hard and
this actually triggered in prod on several of them.

Thanks.

-- 
tejun