From: Randy Dunlap <randy.dunlap@oracle.com>
To: "Torsten Kaiser" <just.for.lkml@googlemail.com>
Cc: "Herbert Xu" <herbert@gondor.apana.org.au>,
"Andrew Morton" <akpm@linux-foundation.org>,
linux-kernel@vger.kernel.org, "Neil Brown" <neilb@suse.de>,
"J. Bruce Fields" <bfields@fieldses.org>,
netdev@vger.kernel.org
Subject: Re: 2.6.24-rc6-mm1
Date: Sat, 29 Dec 2007 21:41:17 -0800 [thread overview]
Message-ID: <20071229214117.3797d9c2.randy.dunlap@oracle.com> (raw)
In-Reply-To: <64bb37e0712291934o77a3d365h56c9c31ac8437469@mail.gmail.com>
On Sun, 30 Dec 2007 04:34:36 +0100 Torsten Kaiser wrote:
> On Dec 30, 2007 2:30 AM, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> > On Sat, Dec 29, 2007 at 05:51:13PM +0100, Torsten Kaiser wrote:
> > >
> > > > > The cause, why I am resending this: I just got a crash with
> > > > > 2.6.24-rc6-mm1, again looking network related:
> > > > >
> > > > > [93436.933356] WARNING: at include/net/dst.h:165 dst_release()
> > > > > [93436.936685] Pid: 8079, comm: konqueror Not tainted 2.6.24-rc6-mm1 #11
> > > > > [93436.939292]
> > > > > [93436.939293] Call Trace:
> > > > > [93436.939304] [<ffffffff80531d2d>] skb_release_all+0xdd/0x110
> > > > > [93436.939307] [<ffffffff80531311>] __kfree_skb+0x11/0xa0
> > > > > [93436.939309] [<ffffffff805313b7>] kfree_skb+0x17/0x30
> > > > > [93436.939312] [<ffffffff805a0b48>] unix_release_sock+0x128/0x250
> > > > > [93436.939315] [<ffffffff805a0c91>] unix_release+0x21/0x30
> > > > > [93436.939318] [<ffffffff8052b144>] sock_release+0x24/0x90
> > > > > [93436.939320] [<ffffffff8052b656>] sock_close+0x26/0x50
> > > > > [93436.939324] [<ffffffff8029f921>] __fput+0xc1/0x230
> > > > > [93436.939327] [<ffffffff8029fe46>] fput+0x16/0x20
> > > > > [93436.939329] [<ffffffff8029c576>] filp_close+0x56/0x90
> > > > > [93436.939331] [<ffffffff8029de46>] sys_close+0xa6/0x110
> > > > > [93436.939335] [<ffffffff8020b57b>] system_call_after_swapgs+0x7b/0x80
> > >
> > > >From code inspection I would blame the patch "[SKBUFF]: Free old skb
> > > properly in skb_morph" from Herbert Xu. (CC added)
> >
> > I doubt it. skb_morph is only used on IP fragments so I don't see how
> > you could attribute an error from a Unix domain socket to this patch.
>
> That's why I wrote that I do not know much about the network core...
>
> > In any case, Unix socket packets should not have a dst at all so the
> > very fact that you're in that path means that you have some sort of
> > memory corruption.
>
> ... I did not know about the fact that there should not have been an dst.
>
> Its just that this warning was the first nice clue about the memory
> corruption related to networking that I see since 2.6.24-rc3-mm2.
> The time of the patch (Mon, 26 Nov 2007 15:11:19) even fits into the
> window between -rc3-mm1 and -rc3-mm2.
>
> I doubt that the memory corruption is a hardware problem, because the
> system in question is using ECC ram and I did not see any messages
> about corrected/detected errors.
>
> > Is this the very first OOPS/warning that you see? If not you should
> > ignore all but the very first one as that may have left your system
> > in an inconsistent state which may render all subsequent OOPSes and
> > warnings useless.
>
> I looked into the log in question and the only other warning was a
> circular locking dependency that lockdep detected around 1.5 hour
> before this warning.
>
> As reported in my original mail immeadeatly after the warning the
> system OOPSed and hang:
> [93436.947241] general protection fault: 0000 [1] SMP
> -> first OOPS ^
FYI, that's what this counter is... -----^
> [93436.947243] last sysfs file:
> /sys/devices/pci0000:00/0000:00:0f.0/0000:01:00.1/irq
> [93436.947245] CPU 1
> [93436.947246] Modules linked in: radeon drm nfsd exportfs w83792d
> ipv6 tuner tea5767 tda8290 tuner_xc2
> 028 tda9887 tuner_simple mt20xx tea5761 tvaudio msp3400 bttv ir_common
> compat_ioctl32 videobuf_dma_sg v
> ideobuf_core btcx_risc tveeprom usbhid videodev v4l2_common hid
> v4l1_compat pata_amd sg i2c_nforce2
> [93436.947257] Pid: 8079, comm: konqueror Not tainted 2.6.24-rc6-mm1 #11
> -> not tainted by a previous OOPS
> [93436.947259] RIP: 0010:[<ffffffff80531438>] [<ffffffff80531438>]
> skb_drop_list+0x18/0x30
> [93436.947262] RSP: 0018:ffff810005f4fda8 EFLAGS: 00010286
> [93436.947263] RAX: ab1ed5ca5b74e7de RBX: ab1ed5ca5b74e7de RCX: 000000000000d135
> [93436.947265] RDX: ffff81011d089a80 RSI: 0000000000000001 RDI: ffff81011d089a88
> [93436.947266] RBP: ffff810005f4fdb8 R08: 0000000000000001 R09: 0000000000000006
> [93436.947268] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8100de02c500
> [93436.947269] R13: ffff81011c188a00 R14: 0000000000000001 R15: ffff81011c189198
> [93436.947271] FS: 00007fb5bde0d700(0000) GS:ffff81007ff22000(0000)
> knlGS:0000000000000000
> [93436.947273] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [93436.947274] CR2: 00007fb5bdd76000 CR3: 00000000664d5000 CR4: 00000000000006e0
> [93436.947276] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [93436.947277] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [93436.947279] Process konqueror (pid: 8079, threadinfo
> ffff810005f4e000, task ffff8100a1dec000)
> [93436.947281] Stack: ffff810005f4fdd8 ffff810116c86140
> ffff810005f4fdd8 ffffffff805314ae
> [93436.947284] ffff810116c86140 ffff8100de02c500 ffff810005f4fdf8
> ffffffff80531cf0
> [93436.947286] ffff8100de02c500 ffff81011c188b48 ffff810005f4fe18
> ffffffff80531311
> [93436.947288] Call Trace:
> [93436.947290] [<ffffffff805314ae>] skb_release_data+0x5e/0xa0
> [93436.947293] [<ffffffff80531cf0>] skb_release_all+0xa0/0x110
> [93436.947295] [<ffffffff80531311>] __kfree_skb+0x11/0xa0
> [93436.947297] [<ffffffff805313b7>] kfree_skb+0x17/0x30
> [93436.947299] [<ffffffff805a0b48>] unix_release_sock+0x128/0x250
> [93436.947302] [<ffffffff805a0c91>] unix_release+0x21/0x30
> [93436.947304] [<ffffffff8052b144>] sock_release+0x24/0x90
> [93436.947307] [<ffffffff8052b656>] sock_close+0x26/0x50
> [93436.947309] [<ffffffff8029f921>] __fput+0xc1/0x230
> [93436.947312] [<ffffffff8029fe46>] fput+0x16/0x20
> [93436.947314] [<ffffffff8029c576>] filp_close+0x56/0x90
> [93436.947316] [<ffffffff8029de46>] sys_close+0xa6/0x110
> [93436.947319] [<ffffffff8020b57b>] system_call_after_swapgs+0x7b/0x80
> [93436.947322]
> [93436.947322]
> [93436.947323] Code: 48 8b 18 48 89 c7 e8 5d ff ff ff 48 85 db 75 ed 48 83 c4 08
> [93436.947328] RIP [<ffffffff80531438>] skb_drop_list+0x18/0x30
> [93436.947330] RSP <ffff810005f4fda8>
> [93436.947332] ---[ end trace befb7cc3528ab3b1 ]---
>
> Your patch just fit so "good" to my problems:
> * it had the correct time frame for 2.6.24-rc3-mm2
> * it looked guilty at changing the refcounting of __refcnt because of
> the added dst_release()
> * it added other release / freeing operations so that a use-after-free
> memory corruption seemed possible
>
> I just have no better idea to what caused this OOPS and the other
> hangs in -rc3-mm2.
---
~Randy
desserts: http://www.xenotime.net/linux/recipes/
next prev parent reply other threads:[~2007-12-30 5:44 UTC|newest]
Thread overview: 146+ messages / expand[flat|nested] mbox.gz Atom feed top
2007-12-23 7:30 2.6.24-rc6-mm1 Andrew Morton
2007-12-23 11:04 ` 2.6.24-rc6-mm1 Ingo Molnar
2007-12-23 11:10 ` 2.6.24-rc6-mm1 Ingo Molnar
2007-12-23 11:34 ` 2.6.24-rc6-mm1 Andrew Morton
2007-12-23 11:57 ` 2.6.24-rc6-mm1 Ingo Molnar
2007-12-23 12:12 ` 2.6.24-rc6-mm1 Christoph Hellwig
2007-12-23 12:35 ` 2.6.24-rc6-mm1 Rafael J. Wysocki
2007-12-23 13:00 ` 2.6.24-rc6-mm1 Ingo Molnar
2007-12-23 13:48 ` 2.6.24-rc6-mm1 Rafael J. Wysocki
2007-12-23 13:53 ` 2.6.24-rc6-mm1 Rafael J. Wysocki
2007-12-23 20:09 ` 2.6.24-rc6-mm1 Sam Ravnborg
2007-12-23 22:44 ` 2.6.24-rc6-mm1 Rafael J. Wysocki
2008-01-02 20:05 ` [patch] x86 compat_binfmt_elf, Makefile fixes (was: Re: 2.6.24-rc6-mm1) Ingo Molnar
2008-01-02 20:32 ` Rafael J. Wysocki
2008-01-02 20:43 ` Ingo Molnar
2007-12-23 23:09 ` 2.6.24-rc6-mm1 H. Peter Anvin
2007-12-23 16:27 ` 2.6.24-rc6-mm1 Torsten Kaiser
2007-12-23 20:39 ` 2.6.24-rc6-mm1 Andrew Morton
2007-12-27 11:42 ` 2.6.24-rc6-mm1 Torsten Kaiser
2007-12-27 14:30 ` 2.6.24-rc6-mm1 Torsten Kaiser
2007-12-28 22:53 ` 2.6.24-rc6-mm1 Torsten Kaiser
2007-12-28 23:07 ` 2.6.24-rc6-mm1 Andrew Morton
2007-12-29 16:51 ` 2.6.24-rc6-mm1 Torsten Kaiser
2007-12-30 1:30 ` 2.6.24-rc6-mm1 Herbert Xu
2007-12-30 3:34 ` 2.6.24-rc6-mm1 Torsten Kaiser
2007-12-30 5:41 ` Randy Dunlap [this message]
2007-12-31 20:15 ` 2.6.24-rc6-mm1 Torsten Kaiser
2008-01-01 12:04 ` 2.6.24-rc6-mm1 Herbert Xu
2008-01-01 12:59 ` 2.6.24-rc6-mm1 Torsten Kaiser
2008-01-01 18:29 ` 2.6.24-rc6-mm1 Torsten Kaiser
2008-01-02 18:29 ` 2.6.24-rc6-mm1 Torsten Kaiser
2008-01-02 21:51 ` 2.6.24-rc6-mm1 Herbert Xu
2008-01-02 21:57 ` 2.6.24-rc6-mm1 J. Bruce Fields
2008-01-03 5:02 ` 2.6.24-rc6-mm1 Torsten Kaiser
2008-01-03 15:37 ` 2.6.24-rc6-mm1 Torsten Kaiser
2008-01-03 18:52 ` 2.6.24-rc6-mm1 J. Bruce Fields
2008-01-04 10:23 ` 2.6.24-rc6-mm1 Torsten Kaiser
2008-01-04 13:30 ` 2.6.24-rc6-mm1 Jarek Poplawski
2008-01-04 15:21 ` 2.6.24-rc6-mm1 Torsten Kaiser
2008-01-04 21:24 ` 2.6.24-rc6-mm1 Torsten Kaiser
2008-01-05 0:07 ` 2.6.24-rc6-mm1 Jarek Poplawski
2008-01-05 8:01 ` 2.6.24-rc6-mm1 Torsten Kaiser
2008-01-05 10:13 ` 2.6.24-rc6-mm1 Jarek Poplawski
2008-01-05 14:52 ` 2.6.24-rc6-mm1 Torsten Kaiser
2008-01-05 22:10 ` 2.6.24-rc6-mm1 Torsten Kaiser
2008-01-06 1:25 ` 2.6.24-rc6-mm1 Andrew Morton
2008-01-06 3:28 ` 2.6.24-rc6-mm1 FUJITA Tomonori
2008-01-06 10:41 ` 2.6.24-rc6-mm1 Torsten Kaiser
2008-01-06 11:23 ` 2.6.24-rc6-mm1 FUJITA Tomonori
2008-01-06 11:35 ` 2.6.24-rc6-mm1 Torsten Kaiser
2008-01-06 13:33 ` 2.6.24-rc6-mm1 FUJITA Tomonori
2008-01-06 20:03 ` 2.6.24-rc6-mm1 Torsten Kaiser
2008-01-07 6:16 ` 2.6.24-rc6-mm1 FUJITA Tomonori
2008-01-08 15:59 ` 2.6.24-rc6-mm1 Ingo Molnar
2008-01-08 23:57 ` 2.6.24-rc6-mm1 FUJITA Tomonori
2008-01-09 0:27 ` 2.6.24-rc6-mm1 Andrew Morton
2008-01-09 0:54 ` 2.6.24-rc6-mm1 FUJITA Tomonori
2008-01-09 1:07 ` 2.6.24-rc6-mm1 Andrew Morton
2008-01-09 9:04 ` 2.6.24-rc6-mm1 Jarek Poplawski
2008-01-10 0:54 ` 2.6.24-rc6-mm1 FUJITA Tomonori
2008-01-25 21:06 ` 2.6.24-rc6-mm1 Torsten Kaiser
2008-01-06 3:16 ` 2.6.24-rc6-mm1 Torsten Kaiser
2008-01-06 8:27 ` 2.6.24-rc6-mm1 Jarek Poplawski
2008-01-06 10:30 ` 2.6.24-rc6-mm1 Torsten Kaiser
2008-01-06 14:52 ` 2.6.24-rc6-mm1 Jarek Poplawski
2007-12-30 21:24 ` 2.6.24-rc6-mm1 J. Bruce Fields
2007-12-30 21:35 ` 2.6.24-rc6-mm1 Torsten Kaiser
2007-12-31 13:17 ` 2.6.24-rc6-mm1 Torsten Kaiser
2007-12-23 22:54 ` 2.6.24-rc6-mm1: suspend broken on HP nx6325 due to cpufreq changes Rafael J. Wysocki
2007-12-23 22:50 ` Andrew Morton
2007-12-23 22:56 ` Dave Jones
2007-12-24 11:21 ` Takashi Iwai
2007-12-24 14:13 ` Rafael J. Wysocki
2007-12-25 21:51 ` 2.6.24-rc6-mm1 Andreas Mohr
2007-12-26 5:43 ` 2.6.24-rc6-mm1 - git-lblnet.patch and networking horkage Valdis.Kletnieks
2007-12-26 7:34 ` James Morris
2007-12-26 8:25 ` Valdis.Kletnieks
2007-12-26 8:52 ` James Morris
2007-12-26 14:16 ` James Morris
2007-12-26 22:46 ` Andrew Morton
2007-12-26 23:39 ` 2.6.24-rc6-mm1 - e1000 breakage James Morris
2007-12-26 16:44 ` 2.6.24-rc6-mm1 - git-lblnet.patch and networking horkage Valdis.Kletnieks
2007-12-26 8:37 ` 2.6.24-rc6-mm1 Dave Young
2007-12-26 12:29 ` 2.6.24-rc6-mm1: some section mismatches on sparc64 Mariusz Kozlowski
2007-12-27 3:05 ` David Miller
2007-12-28 23:22 ` Adrian Bunk
2007-12-29 8:14 ` David Miller
2007-12-29 8:27 ` David Miller
2007-12-29 8:48 ` Adrian Bunk
2007-12-29 8:54 ` David Miller
2007-12-29 9:06 ` Adrian Bunk
2007-12-29 9:18 ` David Miller
2007-12-29 9:53 ` Adrian Bunk
2007-12-29 9:15 ` David Miller
2007-12-27 2:21 ` 2.6.24-rc6-mm1: __raw_spin_is_contended undefined Joseph Fannin
2007-12-27 5:21 ` Nick Piggin
2007-12-27 6:04 ` 2.6.24-rc6-mm1 - drivers/char/tpm/tpm_bios.c oddness? Valdis.Kletnieks
2007-12-27 8:49 ` 2.6.24-rc6-mm1 Kernel panics at different functions () Kamalesh Babulal
2007-12-27 9:54 ` Andrew Morton
2007-12-27 10:25 ` Kamalesh Babulal
2007-12-28 9:11 ` Kamalesh Babulal
2007-12-28 13:03 ` Dhaval Giani
2007-12-27 9:16 ` 2.6.24-rc6-mm1 - power_supply driver dmesg weirdness? Valdis.Kletnieks
2007-12-27 17:54 ` 2.6.24-rc6-mm1 - crash in tick_sched_timer/update_process_times Valdis.Kletnieks
2007-12-28 7:34 ` Andrew Morton
2007-12-29 16:02 ` Valdis.Kletnieks
2007-12-31 18:05 ` [PATCH -mm] crypto: scatterwalk.h needs sched.h Randy Dunlap
2007-12-31 22:31 ` Herbert Xu
2007-12-31 18:05 ` [PATCH -mm] driver core: build with SYSFS=n Randy Dunlap
2007-12-31 18:18 ` 2.6.24-rc6-mm1 (build problem: v4l / i2c) Randy Dunlap
2007-12-31 19:01 ` [v4l-dvb-maintainer] " Michael Krufky
2007-12-31 19:08 ` Michael Krufky
2007-12-31 18:18 ` 2.6.24-rc6-mm1 (build problem: gpio_keys) Randy Dunlap
2007-12-31 18:40 ` David Brownell
2007-12-31 19:10 ` [PATCH -mm] gpio: fix x86 build problem: gpio_keys Randy Dunlap
2008-01-01 15:32 ` Ingo Molnar
2007-12-31 18:19 ` 2.6.24-rc6-mm1 (build problem: gpio/W1) Randy Dunlap
2008-01-05 15:29 ` Evgeniy Polyakov
2008-01-05 16:16 ` Ville Syrjälä
2008-01-05 17:18 ` Randy Dunlap
2007-12-31 20:11 ` 2.6.24-rc6-mm1 (driver core/sysfs) Randy Dunlap
2008-01-12 1:05 ` Greg KH
2008-01-14 16:11 ` [patch] auto-qa Kconfig Ingo Molnar
2008-01-15 22:13 ` Pavel Machek
2008-01-06 10:03 ` 2.6.24-rc6-mm1: sparc64: undefined reference to `vmemmap_table' Mariusz Kozlowski
2008-01-06 10:15 ` Andrew Morton
2008-01-07 1:18 ` David Miller
2008-01-07 13:50 ` [PATCH -mm/x86] revert i386: handle an initrd in highmem (Was Re: 2.6.24-rc6-mm1) Dhaval Giani
2008-01-07 13:53 ` Thomas Gleixner
2008-01-07 14:56 ` Ingo Molnar
2008-01-07 15:19 ` Dhaval Giani
2008-01-07 16:22 ` Randy Dunlap
2008-01-07 16:32 ` Dhaval Giani
2008-01-07 16:50 ` Thomas Gleixner
2008-01-07 22:15 ` Randy Dunlap
[not found] <9DtBq-2jD-3@gated-at.bofh.it>
[not found] ` <476EAF98.6040004@yahoo.fr>
[not found] ` <20071223121410.0d572e03.akpm@linux-foundation.org>
2007-12-24 1:25 ` 2.6.24-rc6-mm1 Herbert Xu
2007-12-30 13:10 ` 2.6.24-rc6-mm1 Ingo Molnar
2008-01-02 10:31 ` 2.6.24-rc6-mm1 Nick Piggin
2008-01-02 11:01 ` 2.6.24-rc6-mm1 Peter Zijlstra
2008-01-02 11:12 ` 2.6.24-rc6-mm1 Nick Piggin
2008-01-02 11:24 ` 2.6.24-rc6-mm1 Peter Zijlstra
2008-01-02 12:19 ` 2.6.24-rc6-mm1 Ingo Molnar
2008-01-02 13:26 ` 2.6.24-rc6-mm1 Alan Cox
2008-01-02 16:18 ` 2.6.24-rc6-mm1 Ingo Molnar
2008-01-02 22:49 ` 2.6.24-rc6-mm1 Alan Cox
2008-01-02 11:08 ` 2.6.24-rc6-mm1 Ingo Molnar
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20071229214117.3797d9c2.randy.dunlap@oracle.com \
--to=randy.dunlap@oracle.com \
--cc=akpm@linux-foundation.org \
--cc=bfields@fieldses.org \
--cc=herbert@gondor.apana.org.au \
--cc=just.for.lkml@googlemail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=neilb@suse.de \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).