Linux-mm Archive on lore.kernel.org
 help / color / Atom feed
From: Thibaut Sautereau <thibaut.sautereau@clip-os.org>
To: netdev@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org
Cc: "David S. Miller" <davem@davemloft.net>,
	Laura Abbott <labbott@redhat.com>,
	Kees Cook <keescook@chromium.org>,
	Alexander Potapenko <glider@google.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	clipos@ssi.gouv.fr
Subject: Double free of struct sk_buff reported by SLAB_CONSISTENCY_CHECKS with init_on_free
Date: Mon, 4 Nov 2019 18:03:03 +0100
Message-ID: <20191104170303.GA50361@gandi.net> (raw)

I ran into the following BUG on a 5.3.8 kernel:

	=============================================================================
	BUG skbuff_head_cache (Tainted: G                T): Object already free
	-----------------------------------------------------------------------------

	INFO: Slab 0x000000000d2d2f8f objects=16 used=3 fp=0x0000000064309071 flags=0x3fff00000000201
	BUG: kernel NULL pointer dereference, address: 0000000000000000
	#PF: supervisor read access in kernel mode
	#PF: error_code(0x0000) - not-present page
	PGD 0 P4D 0
	Oops: 0000 [#1] PREEMPT SMP PTI
	CPU: 0 PID: 0 Comm: swapper/0 Tainted: G    B           T 5.3.8 #1
	Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
	RIP: 0010:print_trailer+0x70/0x1d5
	Code: 28 4d 8b 4d 00 4d 8b 45 20 81 e2 ff 7f 00 00 e8 86 ce ef ff 8b 4b 20 48 89 ea 48 89 ee 4c 29 e2 48 c7 c7 90 6f d4 89 48 01 e9 <48> 33 09 48 33 8b 70 01 00 00 e8 61 ce ef ff f6 43 09 04 74 35 8b
	RSP: 0018:ffffbf7680003d58 EFLAGS: 00010046
	RAX: 000000000000005d RBX: ffffa3d2bb08e540 RCX: 0000000000000000
	RDX: 00005c2d8fdc2000 RSI: 0000000000000000 RDI: ffffffff89d46f90
	RBP: 0000000000000000 R08: 0000000000000242 R09: 000000000000006c
	R10: 0000000000000000 R11: 0000000000000030 R12: ffffa3d27023e000
	R13: fffff11080c08f80 R14: ffffa3d2bb047a80 R15: 0000000000000002
	FS:  0000000000000000(0000) GS:ffffa3d2be400000(0000) knlGS:0000000000000000
	CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
	CR2: 0000000000000000 CR3: 000000007a6c4000 CR4: 00000000000006f0
	Call Trace:
	 <IRQ>
	 free_debug_processing.cold.37+0xc9/0x149
	 ? __kfree_skb_flush+0x30/0x40
	 ? __kfree_skb_flush+0x30/0x40
	 __slab_free+0x22a/0x3d0
	 ? tcp_wfree+0x2a/0x140
	 ? __sock_wfree+0x1b/0x30
	 kmem_cache_free_bulk+0x415/0x420
	 ? __kfree_skb_flush+0x30/0x40
	 __kfree_skb_flush+0x30/0x40
	 net_rx_action+0x2dd/0x480
	 __do_softirq+0xf0/0x246
	 irq_exit+0x93/0xb0
	 do_IRQ+0xa0/0x110
	 common_interrupt+0xf/0xf
	 </IRQ>
	RIP: 0010:default_idle+0x13/0x20
	Code: 24 28 eb 9e 4c 89 ff e8 eb 9f c9 ff eb 9f e8 84 03 8c ff 90 90 90 90 e8 2b 96 bc ff e9 07 00 00 00 0f 00 2d 01 71 41 00 fb f4 <e9> 18 96 bc ff 0f 1f 84 00 00 00 00 00 53 65 48 8b 1c 25 00 5d 01
	RSP: 0018:ffffffff89e03eb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffffd6
	RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
	RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff89d8fe58
	RBP: ffffffff89e85d78 R08: 00000000000d2730 R09: 0000000000000000
	R10: 0000000000000222 R11: 0000000000000000 R12: 0000000000000000
	R13: 0000000000000000 R14: ffffffff8a2ea920 R15: 000000007efbbd98
	 ? default_idle+0x5/0x20
	 do_idle+0x1ad/0x220
	 cpu_startup_entry+0x14/0x20
	 start_kernel+0x62f/0x66e
	 secondary_startup_64+0xa4/0xb0
	Modules linked in: [...]
	CR2: 0000000000000000
	---[ end trace 0d08341f1cb4ff43 ]---
	RIP: 0010:print_trailer+0x70/0x1d5
	Code: 28 4d 8b 4d 00 4d 8b 45 20 81 e2 ff 7f 00 00 e8 86 ce ef ff 8b 4b 20 48 89 ea 48 89 ee 4c 29 e2 48 c7 c7 90 6f d4 89 48 01 e9 <48> 33 09 48 33 8b 70 01 00 00 e8 61 ce ef ff f6 43 09 04 74 35 8b
	RSP: 0018:ffffbf7680003d58 EFLAGS: 00010046
	RAX: 000000000000005d RBX: ffffa3d2bb08e540 RCX: 0000000000000000
	RDX: 00005c2d8fdc2000 RSI: 0000000000000000 RDI: ffffffff89d46f90
	RBP: 0000000000000000 R08: 0000000000000242 R09: 000000000000006c
	R10: 0000000000000000 R11: 0000000000000030 R12: ffffa3d27023e000
	R13: fffff11080c08f80 R14: ffffa3d2bb047a80 R15: 0000000000000002
	FS:  0000000000000000(0000) GS:ffffa3d2be400000(0000) knlGS:0000000000000000
	CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
	CR2: 0000000000000000 CR3: 000000007a6c4000 CR4: 00000000000006f0
	Kernel panic - not syncing: Fatal exception in interrupt
	Shutting down cpus with NMI
	Kernel Offset: 0x8000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
	---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

We first encountered this issue under huge network traffic (system image
download), and I was able to reproduce by simply sending a big packet
with `ping -s 65507 <ip>`, which crashes the kernel every single time.

The BUG only happens when using `slub_debug=F` on the command-line (to
enable SLAB_CONSISTENCY_CHECKS), otherwise the double free is not
reported and the system keeps running.

The code path is:
	net_rx_action
	  __kfree_skb_flush
		kmem_cache_free_bulk()  # skbuff_head_cache
		  slab_free()
			do_slab_free()
			  __slab_free()
				free_debug_processing()
				  free_consistency_check()
					object_err()    # "Object already free"
					  print_trailer()
						print_tracking()    # !(s->flags & SLAB_STORE_USER) => return;
						print_page_info()   # "INFO: Slab ..."
						pr_err("INFO: Object ...", ..., get_freepointer(s, p))
						  get_freepointer()
						    freelist_dereference() # NULL pointer dereference

Enabling KASAN shows less info because the NULL pointer dereference then
apparently happens before reaching free_debug_processing().

Bisection points to the following commit: 1b7e816fc80e ("mm: slub: Fix
slab walking for init_on_free"), and indeed the BUG is not triggered
when init_on_free is disabled.

-- 
Thibaut Sautereau
CLIP OS developer


             reply index

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-11-04 17:03 Thibaut Sautereau [this message]
2019-11-04 17:33 ` Eric Dumazet
2019-11-05  8:05   ` Thibaut Sautereau
2019-11-05 10:19     ` Alexander Potapenko
2019-11-05 15:04       ` Thibaut Sautereau
2019-11-05  9:00 ` Vlastimil Babka
2019-11-05 14:32   ` Thibaut Sautereau
2019-11-05 15:02     ` Vlastimil Babka
2019-11-05 17:01       ` Laura Abbott
2019-11-06  9:29         ` Thibaut Sautereau

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191104170303.GA50361@gandi.net \
    --to=thibaut.sautereau@clip-os.org \
    --cc=akpm@linux-foundation.org \
    --cc=clipos@ssi.gouv.fr \
    --cc=davem@davemloft.net \
    --cc=glider@google.com \
    --cc=keescook@chromium.org \
    --cc=labbott@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=netdev@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-mm Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-mm/0 linux-mm/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-mm linux-mm/ https://lore.kernel.org/linux-mm \
		linux-mm@kvack.org
	public-inbox-index linux-mm

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kvack.linux-mm


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git