* Kernel oops+crash on repeated auditd restarts
@ 2012-01-25 16:45 Valentin Avram
2012-01-25 16:53 ` Peter Moody
2012-01-25 19:20 ` Eric Paris
0 siblings, 2 replies; 24+ messages in thread
From: Valentin Avram @ 2012-01-25 16:45 UTC (permalink / raw)
To: linux-audit
[-- Attachment #1.1: Type: text/plain, Size: 1606 bytes --]
Hello.
Did anybody ever experience kernel oopses and even kernel crashes (after a
while), by just restarting repeatedly the auditd daemon?
I ask this because i had this problem on Dell R610 servers running Gentoo
Linux kernels gentoo-sources-3.0.6 and gentoo-sources-2.6.37-r4 (see this
bug: https://bugs.gentoo.org/show_bug.cgi?id=389405 ).
The kernels are nothing special, just the vanilla 2.6.37 and 3.0.6 with a
few gentoo patches (see https://lkml.org/lkml/2011/11/28/330 ).
The auditd version is 2.1.3 (latest). The audit.rules file contains
basically the following rules:
-D
-w /etc -p wa -k etc-directory
[snip: same for /sbin, /bin, /usr/sbin, /usr/bin]
-a exit,never -F dir=/lib/rc -k skip-lib-rc
-w /lib -p wa -k lib-directory
-w /usr/lib -p wa -k usr-lib-directory
-a exit,never -F arch=b32 -S read [snip: -S for write,open,fstat,mmap etc.]
-k excluded-syscalls
-b 8192
The bug seems to be somewhere in the fsnotify kernel part, however Gentoo
kernel devs and ppl on lkml did not seem too interested, so.. did anybody
notice a similar behaviour? Or better yet, is anybody willing to run on one
of your servers this simple test: start the minimum server services, use a
similar audit.rules configuration, then start auditd and run in a shell the
following one-liner:
while :; do /etc/init.d/auditd stop ; sleep 5 ; /etc/init.d/auditd start ;
sleep 5 ; done
This was enough to oops and crash the kernel in less than one hour on the
servers where i did the tests. If any similar behavior happens, i'd be very
interested to know the the kernel version and distro.
Thank you for your time.
[-- Attachment #1.2: Type: text/html, Size: 2036 bytes --]
[-- Attachment #2: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-01-25 16:45 Kernel oops+crash on repeated auditd restarts Valentin Avram
@ 2012-01-25 16:53 ` Peter Moody
2012-01-25 19:20 ` Eric Paris
1 sibling, 0 replies; 24+ messages in thread
From: Peter Moody @ 2012-01-25 16:53 UTC (permalink / raw)
To: Valentin Avram; +Cc: linux-audit
Just flushing the rules (auditctl -D) would cause my ubuntu machine
running a 2.6.38 kernel to oops fairly regularly, maybe one in five
times. This was especially painful when testing new rules.
On Wed, Jan 25, 2012 at 8:45 AM, Valentin Avram <aval13@gmail.com> wrote:
> Hello.
>
> Did anybody ever experience kernel oopses and even kernel crashes (after a
> while), by just restarting repeatedly the auditd daemon?
>
> I ask this because i had this problem on Dell R610 servers running Gentoo
> Linux kernels gentoo-sources-3.0.6 and gentoo-sources-2.6.37-r4 (see this
> bug: https://bugs.gentoo.org/show_bug.cgi?id=389405 ).
>
> The kernels are nothing special, just the vanilla 2.6.37 and 3.0.6 with a
> few gentoo patches (see https://lkml.org/lkml/2011/11/28/330 ).
>
> The auditd version is 2.1.3 (latest). The audit.rules file contains
> basically the following rules:
>
> -D
> -w /etc -p wa -k etc-directory
> [snip: same for /sbin, /bin, /usr/sbin, /usr/bin]
> -a exit,never -F dir=/lib/rc -k skip-lib-rc
> -w /lib -p wa -k lib-directory
> -w /usr/lib -p wa -k usr-lib-directory
> -a exit,never -F arch=b32 -S read [snip: -S for write,open,fstat,mmap etc.]
> -k excluded-syscalls
> -b 8192
>
> The bug seems to be somewhere in the fsnotify kernel part, however Gentoo
> kernel devs and ppl on lkml did not seem too interested, so.. did anybody
> notice a similar behaviour? Or better yet, is anybody willing to run on one
> of your servers this simple test: start the minimum server services, use a
> similar audit.rules configuration, then start auditd and run in a shell the
> following one-liner:
>
> while :; do /etc/init.d/auditd stop ; sleep 5 ; /etc/init.d/auditd start ;
> sleep 5 ; done
>
> This was enough to oops and crash the kernel in less than one hour on the
> servers where i did the tests. If any similar behavior happens, i'd be very
> interested to know the the kernel version and distro.
>
> Thank you for your time.
>
>
> --
> Linux-audit mailing list
> Linux-audit@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit
--
Peter Moody Google 1.650.253.7306
Security Engineer pgp:0xC3410038
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-01-25 16:45 Kernel oops+crash on repeated auditd restarts Valentin Avram
2012-01-25 16:53 ` Peter Moody
@ 2012-01-25 19:20 ` Eric Paris
2012-01-26 7:13 ` Valentin Avram
1 sibling, 1 reply; 24+ messages in thread
From: Eric Paris @ 2012-01-25 19:20 UTC (permalink / raw)
To: Valentin Avram; +Cc: linux-audit
On Wed, 2012-01-25 at 18:45 +0200, Valentin Avram wrote:
> Did anybody ever experience kernel oopses and even kernel crashes
> (after a while), by just restarting repeatedly the auditd daemon?
No, but I'll try to remember to take a look. We did have a BUG() that
was recently fixed when using -w rules (as I recall). But I've never
seen this particular NULL pointer bug. We did recently fix a race in
fsnotify mark destruction that could be this, but those symptoms weren't
exactly the same.
I'm both the upstream Audit and fsnotify maintainer so I'm grumbley at
Gentoo for never letting me know isn't working. Where else did you
report this? I'm wondering where all the information failure is
happening.
Can you send me any and all info you have?
I'll see if I can reproduce a problem here (but I'm a Fedora guy)
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-01-25 19:20 ` Eric Paris
@ 2012-01-26 7:13 ` Valentin Avram
2012-02-08 16:11 ` Valentin Avram
0 siblings, 1 reply; 24+ messages in thread
From: Valentin Avram @ 2012-01-26 7:13 UTC (permalink / raw)
To: Eric Paris; +Cc: linux-audit
[-- Attachment #1.1: Type: text/plain, Size: 2131 bytes --]
Please read below.
On Wed, Jan 25, 2012 at 9:20 PM, Eric Paris <eparis@redhat.com> wrote:
> On Wed, 2012-01-25 at 18:45 +0200, Valentin Avram wrote:
>
> > Did anybody ever experience kernel oopses and even kernel crashes
> > (after a while), by just restarting repeatedly the auditd daemon?
>
> No, but I'll try to remember to take a look. We did have a BUG() that
> was recently fixed when using -w rules (as I recall). But I've never
> seen this particular NULL pointer bug. We did recently fix a race in
> fsnotify mark destruction that could be this, but those symptoms weren't
> exactly the same.
>
> I'm both the upstream Audit and fsnotify maintainer so I'm grumbley at
> Gentoo for never letting me know isn't working. Where else did you
> report this? I'm wondering where all the information failure is
> happening.
>
I only reported the issue on Gentoo bugs and LKML (the two links i included
in the original email). The Gentoo guys at first did seem interested in the
bug and asked for a test with a kernel compiled with CONFIG_DEBUG_INFO and
CONFIG_DEBUG_LIST. After that test it looked like some list is getting
messed up somewhere (altough i'm part C programmer, my kernel insides
knowledge is limited). The LKML guys didn't even bother to answer.
> Can you send me any and all info you have?
>
>
All the information i had is posted on the Gentoo bug report. The two
machines i used to test the issue are now in production mode, so i can't do
any testing on them. However I'll soon have access to a new machine that
can stay in test mode for a while, where i plan to retest with Gentoo's
latest "stable-marked" kernel gentoo-sources-3.1.6.
> I'll see if I can reproduce a problem here (but I'm a Fedora guy)
>
At this moment i'm not extremely sure if it's a auditd issue or a kernel
issue or both. However, if you're running a kernel lower than 3.0.7 and
auditd 2.1.3, I'd be very interested if running the one-liner i posted
(audit start and stop on a loop with 5 seconds delay) will eventually (in 1
hour or something close) crash the kernel completely (or at least oops a
lot of times).
Thank you.
[-- Attachment #1.2: Type: text/html, Size: 2893 bytes --]
[-- Attachment #2: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-01-26 7:13 ` Valentin Avram
@ 2012-02-08 16:11 ` Valentin Avram
2012-03-05 8:35 ` Valentin Avram
0 siblings, 1 reply; 24+ messages in thread
From: Valentin Avram @ 2012-02-08 16:11 UTC (permalink / raw)
To: Eric Paris; +Cc: linux-audit
[-- Attachment #1.1: Type: text/plain, Size: 2193 bytes --]
Hello.
Fresh news: Gentoo's gentoo-sources-3.1.10-r1 with audit-2.1.3 still gives
oops using the simple "start ; sleep 5 ; stop ; sleep 5 ; repeat" one-liner.
Kernel oops after less than 5 minutes:
BUG: unable to handle kernel NULL pointer dereference at 00000004
IP: [<c10f2337>] fsnotify_mark_destroy+0x87/0x130
*pdpt = 0000000000000000 *pde = f000def8f000def8
Oops: 0002 [#1] SMP
Pid: 690, comm: fsnotify_mark Not tainted 3.1.10-gentoo-r1-drbd-version3 #1
Dell Inc. PowerEdge R610/0F0XJ6
EIP: 0060:[<c10f2337>] EFLAGS: 00010216 CPU: 3
EIP is at fsnotify_mark_destroy+0x87/0x130
EAX: f2e51708 EBX: f2415fa8 ECX: 00000000 EDX: f2e51744
ESI: f2f46c00 EDI: ffffffc4 EBP: c10ea000 ESP: f2415f90
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process fsnotify_mark (pid: 690, ti=f2414000 task=f2f46c00 task.ti=f2414000)
Stack:
f2f46c00 00000000 f2f46c00 c1050150 f2415fa0 f2415fa0 f2e51744 f2e51744
f2c47f68 00000000 c10f22b0 00000000 c104f854 00000000 00000000 00000000
00000000 f2415fd4 f2415fd4 00000000 c104f7e0 f2c47f68 c15820b6 00000000
Call Trace:
[<c1050150>] ? abort_exclusive_wait+0x90/0x90
[<c10f22b0>] ? fsnotify_put_mark+0x20/0x20
[<c104f854>] ? kthread+0x74/0x80
[<c104f7e0>] ? kthread_flush_work_fn+0x10/0x10
[<c15820b6>] ? kernel_thread_helper+0x6/0xd
Code: 34 1b 8b c1 e8 4b 2d f6 ff 8b 54 24 18 8d 42 c4 39 da 8b 48 3c 8d 79
c4 75 0e eb 2d 90 8d b4 26 00 00 00 00 89 f8 89 ef 8b 68 40
69 04 89 4d 00 89 50 3c 89 50 40 e8 48 ff ff ff 8b 4f 3c 8d
EIP: [<c10f2337>] fsnotify_mark_destroy+0x87/0x130 SS:ESP 0068:f2415f90
CR2: 0000000000000004
---[ end trace d10081cf0e5b936c ]---
So far only one oops occured, however the test server is doing quite
nothing right now. I'll install more services, retry and post back here the
results.
On Thu, Jan 26, 2012 at 9:13 AM, Valentin Avram <aval13@gmail.com> wrote:
>
> All the information i had is posted on the Gentoo bug report. The two
> machines i used to test the issue are now in production mode, so i can't do
> any testing on them. However I'll soon have access to a new machine that
> can stay in test mode for a while, where i plan to retest with Gentoo's
> latest "stable-marked" kernel gentoo-sources-3.1.6.
>
>
[-- Attachment #1.2: Type: text/html, Size: 2698 bytes --]
[-- Attachment #2: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-02-08 16:11 ` Valentin Avram
@ 2012-03-05 8:35 ` Valentin Avram
2012-03-28 20:51 ` Peter Moody
0 siblings, 1 reply; 24+ messages in thread
From: Valentin Avram @ 2012-03-05 8:35 UTC (permalink / raw)
To: Eric Paris; +Cc: linux-audit
[-- Attachment #1.1: Type: text/plain, Size: 3835 bytes --]
Finally i found some time and spare server to retest the oops and list_add
corruptions i was getting with the 3.x kernels and auditd 2.1.3.
I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and kernel.org's
3.2.9.
Both get the oops/BUG in the same way and after that, they keep pouring
list_add corruptions with audit_prune_tre(truncated?) and auditctl as comms.
Since this is not about Gentoo's kernel only, i'll post here the oops in
3.2.9 and also attach some list_add corruptions.
3.2.9 BUG:
kernel: [ 301.240011] BUG: unable to handle kernel NULL pointer
dereference at (null)
kernel: [ 301.240305] IP: [<c1238dd0>] __list_del_entry+0x20/0xe0
kernel: [ 301.240481] *pdpt = 0000000000000000 *pde = f000ddc8f000ddc8
kernel: [ 301.240698] Oops: 0000 [#1] SMP
kernel: [ 301.240910]
kernel: [ 301.241030] Pid: 642, comm: fsnotify_mark Not tainted
3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396
kernel: [ 301.241370] EIP: 0060:[<c1238dd0>] EFLAGS: 00010287 CPU: 6
kernel: [ 301.241498] EIP is at __list_del_entry+0x20/0xe0
kernel: [ 301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff EDX: 00000000
kernel: [ 301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c ESP: f47cff64
kernel: [ 301.241879] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
kernel: [ 301.242005] Process fsnotify_mark (pid: 642, ti=f47ce000
task=f4f47c00 task.ti=f47ce000)
kernel: [ 301.242207] Stack:
kernel: [ 301.242327] c10813c0 f47cffa4 f4f47c00 f4e70888 f47cff7c
f47cffa4 f47cffb8 c10f6976
kernel: [ 301.242882] ffffffc3 f4f47c00 f4f47c00 00000000 f4f47c00
c10530c0 f47cff9c f47cff9c
kernel: [ 301.243438] f4fae544 f4fae544 f4c47f58 00000000 c10f68f0
f47cffe4 c1052834 00000000
kernel: [ 301.243995] Call Trace:
kernel: [ 301.244119] [<c10813c0>] ? rcu_check_callbacks+0x110/0x110
kernel: [ 301.244248] [<c10f6976>] fsnotify_mark_destroy+0x86/0x120
kernel: [ 301.244377] [<c10530c0>] ? abort_exclusive_wait+0x80/0x80
kernel: [ 301.244504] [<c10f68f0>] ? fsnotify_put_mark+0x30/0x30
kernel: [ 301.244631] [<c1052834>] kthread+0x74/0x80
kernel: [ 301.244756] [<c10527c0>] ? kthread_flush_work_fn+0x10/0x10
kernel: [ 301.244885] [<c1582ab6>] kernel_thread_helper+0x6/0xd
kernel: [ 301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89 e5
53 83 ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00
0f 84 8e 00 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89
0a 83 c4 14
kernel: [ 301.248195] EIP: [<c1238dd0>] __list_del_entry+0x20/0xe0
SS:ESP 0068:f47cff64
kernel: [ 301.248414] CR2: 0000000000000000
kernel: [ 301.248538] ---[ end trace 15082dbfb353f84c ]---
The kernel was compiled with the following DEBUG support (the bolded one
were requested by Gentoo's Dev:
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_SLUB_DEBUG=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_PNP_DEBUG_MESSAGES=y
CONFIG_AIC94XX_DEBUG=y
CONFIG_USB_DEBUG=y
CONFIG_DEBUG_KERNEL=y
CONFIG_SCHED_DEBUG=y
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_PI_LIST=y
CONFIG_DEBUG_BUGVERBOSE=y
*CONFIG_DEBUG_INFO=y*
CONFIG_DEBUG_MEMORY_INIT=y
*CONFIG_DEBUG_LIST=y*
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_DEBUG_RODATA=y
CONFIG_DEBUG_RODATA_TEST=y
I attached the kernel config i used for 3.2.9 to generate this oops and
warnings.
>From the list_add warnings that come after, out of 805 warnings i
processed, after masking with XXXXX the PID and next= values that kept
changing in every one, i got 26 types of MD5. I also attached the files
relevant as an archive to this email.
The Gentoo bug i opened is sleeping, it seems nobody has the time to at
least test to confirm or not the problems i'm seeing (or everybody's
thinking that nobody would restart auditd so often, so the bug it's not
that serious).
Thank you for your time.
On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram <aval13@gmail.com> wrote:
[-- Attachment #1.2: Type: text/html, Size: 4292 bytes --]
[-- Attachment #2: parse_oops.tgz --]
[-- Type: application/x-gzip, Size: 1783 bytes --]
[-- Attachment #3: kernel_config.gz --]
[-- Type: application/x-gzip, Size: 15572 bytes --]
[-- Attachment #4: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-03-05 8:35 ` Valentin Avram
@ 2012-03-28 20:51 ` Peter Moody
2012-03-28 22:42 ` Peter Moody
0 siblings, 1 reply; 24+ messages in thread
From: Peter Moody @ 2012-03-28 20:51 UTC (permalink / raw)
To: Valentin Avram; +Cc: linux-audit
Are you still able to reliably reproduce this oops? I'm trying to
track this down because this bug (or a very similar bug) is causing
some significant headaches here at work, but I haven't had a lot of
luck. I'm using usermode linux, though, so that might be interfering
with things.
On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram <aval13@gmail.com> wrote:
> Finally i found some time and spare server to retest the oops and list_add
> corruptions i was getting with the 3.x kernels and auditd 2.1.3.
>
> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and kernel.org's
> 3.2.9.
>
> Both get the oops/BUG in the same way and after that, they keep pouring
> list_add corruptions with audit_prune_tre(truncated?) and auditctl as comms.
>
> Since this is not about Gentoo's kernel only, i'll post here the oops in
> 3.2.9 and also attach some list_add corruptions.
>
> 3.2.9 BUG:
>
> kernel: [ 301.240011] BUG: unable to handle kernel NULL pointer dereference
> at (null)
> kernel: [ 301.240305] IP: [<c1238dd0>] __list_del_entry+0x20/0xe0
> kernel: [ 301.240481] *pdpt = 0000000000000000 *pde = f000ddc8f000ddc8
> kernel: [ 301.240698] Oops: 0000 [#1] SMP
> kernel: [ 301.240910]
> kernel: [ 301.241030] Pid: 642, comm: fsnotify_mark Not tainted
> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396
> kernel: [ 301.241370] EIP: 0060:[<c1238dd0>] EFLAGS: 00010287 CPU: 6
> kernel: [ 301.241498] EIP is at __list_del_entry+0x20/0xe0
> kernel: [ 301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff EDX:
> 00000000
> kernel: [ 301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c ESP:
> f47cff64
> kernel: [ 301.241879] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> kernel: [ 301.242005] Process fsnotify_mark (pid: 642, ti=f47ce000
> task=f4f47c00 task.ti=f47ce000)
> kernel: [ 301.242207] Stack:
> kernel: [ 301.242327] c10813c0 f47cffa4 f4f47c00 f4e70888 f47cff7c
> f47cffa4 f47cffb8 c10f6976
> kernel: [ 301.242882] ffffffc3 f4f47c00 f4f47c00 00000000 f4f47c00
> c10530c0 f47cff9c f47cff9c
> kernel: [ 301.243438] f4fae544 f4fae544 f4c47f58 00000000 c10f68f0
> f47cffe4 c1052834 00000000
> kernel: [ 301.243995] Call Trace:
> kernel: [ 301.244119] [<c10813c0>] ? rcu_check_callbacks+0x110/0x110
> kernel: [ 301.244248] [<c10f6976>] fsnotify_mark_destroy+0x86/0x120
> kernel: [ 301.244377] [<c10530c0>] ? abort_exclusive_wait+0x80/0x80
> kernel: [ 301.244504] [<c10f68f0>] ? fsnotify_put_mark+0x30/0x30
> kernel: [ 301.244631] [<c1052834>] kthread+0x74/0x80
> kernel: [ 301.244756] [<c10527c0>] ? kthread_flush_work_fn+0x10/0x10
> kernel: [ 301.244885] [<c1582ab6>] kernel_thread_helper+0x6/0xd
> kernel: [ 301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89 e5 53 83
> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 0f 84 8e 00
> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 83 c4 14
> kernel: [ 301.248195] EIP: [<c1238dd0>] __list_del_entry+0x20/0xe0 SS:ESP
> 0068:f47cff64
> kernel: [ 301.248414] CR2: 0000000000000000
> kernel: [ 301.248538] ---[ end trace 15082dbfb353f84c ]---
>
> The kernel was compiled with the following DEBUG support (the bolded one
> were requested by Gentoo's Dev:
> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
> CONFIG_SLUB_DEBUG=y
> CONFIG_HAVE_DMA_API_DEBUG=y
> CONFIG_X86_DEBUGCTLMSR=y
> CONFIG_PNP_DEBUG_MESSAGES=y
> CONFIG_AIC94XX_DEBUG=y
> CONFIG_USB_DEBUG=y
> CONFIG_DEBUG_KERNEL=y
> CONFIG_SCHED_DEBUG=y
> CONFIG_DEBUG_RT_MUTEXES=y
> CONFIG_DEBUG_PI_LIST=y
> CONFIG_DEBUG_BUGVERBOSE=y
> CONFIG_DEBUG_INFO=y
> CONFIG_DEBUG_MEMORY_INIT=y
> CONFIG_DEBUG_LIST=y
> CONFIG_DEBUG_STACKOVERFLOW=y
> CONFIG_DEBUG_RODATA=y
> CONFIG_DEBUG_RODATA_TEST=y
>
> I attached the kernel config i used for 3.2.9 to generate this oops and
> warnings.
>
> From the list_add warnings that come after, out of 805 warnings i processed,
> after masking with XXXXX the PID and next= values that kept changing in
> every one, i got 26 types of MD5. I also attached the files relevant as an
> archive to this email.
>
> The Gentoo bug i opened is sleeping, it seems nobody has the time to at
> least test to confirm or not the problems i'm seeing (or everybody's
> thinking that nobody would restart auditd so often, so the bug it's not that
> serious).
>
>
> Thank you for your time.
>
> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram <aval13@gmail.com> wrote:
>
>
> --
> Linux-audit mailing list
> Linux-audit@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit
--
Peter Moody Google 1.650.253.7306
Security Engineer pgp:0xC3410038
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-03-28 20:51 ` Peter Moody
@ 2012-03-28 22:42 ` Peter Moody
2012-03-29 1:14 ` Eric Paris
0 siblings, 1 reply; 24+ messages in thread
From: Peter Moody @ 2012-03-28 22:42 UTC (permalink / raw)
To: Valentin Avram; +Cc: linux-audit
fyi: this patch [1] seems to fix the issue for me. The explanation in
the subject would reliably oops my machine.
[1] http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=fed474857efbed79cd390d0aee224231ca718f63
On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody <pmoody@google.com> wrote:
> Are you still able to reliably reproduce this oops? I'm trying to
> track this down because this bug (or a very similar bug) is causing
> some significant headaches here at work, but I haven't had a lot of
> luck. I'm using usermode linux, though, so that might be interfering
> with things.
>
> On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram <aval13@gmail.com> wrote:
>> Finally i found some time and spare server to retest the oops and list_add
>> corruptions i was getting with the 3.x kernels and auditd 2.1.3.
>>
>> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and kernel.org's
>> 3.2.9.
>>
>> Both get the oops/BUG in the same way and after that, they keep pouring
>> list_add corruptions with audit_prune_tre(truncated?) and auditctl as comms.
>>
>> Since this is not about Gentoo's kernel only, i'll post here the oops in
>> 3.2.9 and also attach some list_add corruptions.
>>
>> 3.2.9 BUG:
>>
>> kernel: [ 301.240011] BUG: unable to handle kernel NULL pointer dereference
>> at (null)
>> kernel: [ 301.240305] IP: [<c1238dd0>] __list_del_entry+0x20/0xe0
>> kernel: [ 301.240481] *pdpt = 0000000000000000 *pde = f000ddc8f000ddc8
>> kernel: [ 301.240698] Oops: 0000 [#1] SMP
>> kernel: [ 301.240910]
>> kernel: [ 301.241030] Pid: 642, comm: fsnotify_mark Not tainted
>> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396
>> kernel: [ 301.241370] EIP: 0060:[<c1238dd0>] EFLAGS: 00010287 CPU: 6
>> kernel: [ 301.241498] EIP is at __list_del_entry+0x20/0xe0
>> kernel: [ 301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff EDX:
>> 00000000
>> kernel: [ 301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c ESP:
>> f47cff64
>> kernel: [ 301.241879] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
>> kernel: [ 301.242005] Process fsnotify_mark (pid: 642, ti=f47ce000
>> task=f4f47c00 task.ti=f47ce000)
>> kernel: [ 301.242207] Stack:
>> kernel: [ 301.242327] c10813c0 f47cffa4 f4f47c00 f4e70888 f47cff7c
>> f47cffa4 f47cffb8 c10f6976
>> kernel: [ 301.242882] ffffffc3 f4f47c00 f4f47c00 00000000 f4f47c00
>> c10530c0 f47cff9c f47cff9c
>> kernel: [ 301.243438] f4fae544 f4fae544 f4c47f58 00000000 c10f68f0
>> f47cffe4 c1052834 00000000
>> kernel: [ 301.243995] Call Trace:
>> kernel: [ 301.244119] [<c10813c0>] ? rcu_check_callbacks+0x110/0x110
>> kernel: [ 301.244248] [<c10f6976>] fsnotify_mark_destroy+0x86/0x120
>> kernel: [ 301.244377] [<c10530c0>] ? abort_exclusive_wait+0x80/0x80
>> kernel: [ 301.244504] [<c10f68f0>] ? fsnotify_put_mark+0x30/0x30
>> kernel: [ 301.244631] [<c1052834>] kthread+0x74/0x80
>> kernel: [ 301.244756] [<c10527c0>] ? kthread_flush_work_fn+0x10/0x10
>> kernel: [ 301.244885] [<c1582ab6>] kernel_thread_helper+0x6/0xd
>> kernel: [ 301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89 e5 53 83
>> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 0f 84 8e 00
>> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 83 c4 14
>> kernel: [ 301.248195] EIP: [<c1238dd0>] __list_del_entry+0x20/0xe0 SS:ESP
>> 0068:f47cff64
>> kernel: [ 301.248414] CR2: 0000000000000000
>> kernel: [ 301.248538] ---[ end trace 15082dbfb353f84c ]---
>>
>> The kernel was compiled with the following DEBUG support (the bolded one
>> were requested by Gentoo's Dev:
>> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
>> CONFIG_SLUB_DEBUG=y
>> CONFIG_HAVE_DMA_API_DEBUG=y
>> CONFIG_X86_DEBUGCTLMSR=y
>> CONFIG_PNP_DEBUG_MESSAGES=y
>> CONFIG_AIC94XX_DEBUG=y
>> CONFIG_USB_DEBUG=y
>> CONFIG_DEBUG_KERNEL=y
>> CONFIG_SCHED_DEBUG=y
>> CONFIG_DEBUG_RT_MUTEXES=y
>> CONFIG_DEBUG_PI_LIST=y
>> CONFIG_DEBUG_BUGVERBOSE=y
>> CONFIG_DEBUG_INFO=y
>> CONFIG_DEBUG_MEMORY_INIT=y
>> CONFIG_DEBUG_LIST=y
>> CONFIG_DEBUG_STACKOVERFLOW=y
>> CONFIG_DEBUG_RODATA=y
>> CONFIG_DEBUG_RODATA_TEST=y
>>
>> I attached the kernel config i used for 3.2.9 to generate this oops and
>> warnings.
>>
>> From the list_add warnings that come after, out of 805 warnings i processed,
>> after masking with XXXXX the PID and next= values that kept changing in
>> every one, i got 26 types of MD5. I also attached the files relevant as an
>> archive to this email.
>>
>> The Gentoo bug i opened is sleeping, it seems nobody has the time to at
>> least test to confirm or not the problems i'm seeing (or everybody's
>> thinking that nobody would restart auditd so often, so the bug it's not that
>> serious).
>>
>>
>> Thank you for your time.
>>
>> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram <aval13@gmail.com> wrote:
>>
>>
>> --
>> Linux-audit mailing list
>> Linux-audit@redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-audit
>
>
>
> --
> Peter Moody Google 1.650.253.7306
> Security Engineer pgp:0xC3410038
--
Peter Moody Google 1.650.253.7306
Security Engineer pgp:0xC3410038
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-03-28 22:42 ` Peter Moody
@ 2012-03-29 1:14 ` Eric Paris
2012-03-29 6:44 ` Valentin Avram
0 siblings, 1 reply; 24+ messages in thread
From: Eric Paris @ 2012-03-29 1:14 UTC (permalink / raw)
To: Peter Moody; +Cc: linux-audit
That patch fixes a BUG() . The report has a NULL ptr deref and some
apparent list correuption.... Sadly they aren't the same....
On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote:
> fyi: this patch [1] seems to fix the issue for me. The explanation in
> the subject would reliably oops my machine.
>
> [1] http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=fed474857efbed79cd390d0aee224231ca718f63
>
> On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody <pmoody@google.com> wrote:
> > Are you still able to reliably reproduce this oops? I'm trying to
> > track this down because this bug (or a very similar bug) is causing
> > some significant headaches here at work, but I haven't had a lot of
> > luck. I'm using usermode linux, though, so that might be interfering
> > with things.
> >
> > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram <aval13@gmail.com> wrote:
> >> Finally i found some time and spare server to retest the oops and list_add
> >> corruptions i was getting with the 3.x kernels and auditd 2.1.3.
> >>
> >> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and kernel.org's
> >> 3.2.9.
> >>
> >> Both get the oops/BUG in the same way and after that, they keep pouring
> >> list_add corruptions with audit_prune_tre(truncated?) and auditctl as comms.
> >>
> >> Since this is not about Gentoo's kernel only, i'll post here the oops in
> >> 3.2.9 and also attach some list_add corruptions.
> >>
> >> 3.2.9 BUG:
> >>
> >> kernel: [ 301.240011] BUG: unable to handle kernel NULL pointer dereference
> >> at (null)
> >> kernel: [ 301.240305] IP: [<c1238dd0>] __list_del_entry+0x20/0xe0
> >> kernel: [ 301.240481] *pdpt = 0000000000000000 *pde = f000ddc8f000ddc8
> >> kernel: [ 301.240698] Oops: 0000 [#1] SMP
> >> kernel: [ 301.240910]
> >> kernel: [ 301.241030] Pid: 642, comm: fsnotify_mark Not tainted
> >> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396
> >> kernel: [ 301.241370] EIP: 0060:[<c1238dd0>] EFLAGS: 00010287 CPU: 6
> >> kernel: [ 301.241498] EIP is at __list_del_entry+0x20/0xe0
> >> kernel: [ 301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff EDX:
> >> 00000000
> >> kernel: [ 301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c ESP:
> >> f47cff64
> >> kernel: [ 301.241879] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> >> kernel: [ 301.242005] Process fsnotify_mark (pid: 642, ti=f47ce000
> >> task=f4f47c00 task.ti=f47ce000)
> >> kernel: [ 301.242207] Stack:
> >> kernel: [ 301.242327] c10813c0 f47cffa4 f4f47c00 f4e70888 f47cff7c
> >> f47cffa4 f47cffb8 c10f6976
> >> kernel: [ 301.242882] ffffffc3 f4f47c00 f4f47c00 00000000 f4f47c00
> >> c10530c0 f47cff9c f47cff9c
> >> kernel: [ 301.243438] f4fae544 f4fae544 f4c47f58 00000000 c10f68f0
> >> f47cffe4 c1052834 00000000
> >> kernel: [ 301.243995] Call Trace:
> >> kernel: [ 301.244119] [<c10813c0>] ? rcu_check_callbacks+0x110/0x110
> >> kernel: [ 301.244248] [<c10f6976>] fsnotify_mark_destroy+0x86/0x120
> >> kernel: [ 301.244377] [<c10530c0>] ? abort_exclusive_wait+0x80/0x80
> >> kernel: [ 301.244504] [<c10f68f0>] ? fsnotify_put_mark+0x30/0x30
> >> kernel: [ 301.244631] [<c1052834>] kthread+0x74/0x80
> >> kernel: [ 301.244756] [<c10527c0>] ? kthread_flush_work_fn+0x10/0x10
> >> kernel: [ 301.244885] [<c1582ab6>] kernel_thread_helper+0x6/0xd
> >> kernel: [ 301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89 e5 53 83
> >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 0f 84 8e 00
> >> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 83 c4 14
> >> kernel: [ 301.248195] EIP: [<c1238dd0>] __list_del_entry+0x20/0xe0 SS:ESP
> >> 0068:f47cff64
> >> kernel: [ 301.248414] CR2: 0000000000000000
> >> kernel: [ 301.248538] ---[ end trace 15082dbfb353f84c ]---
> >>
> >> The kernel was compiled with the following DEBUG support (the bolded one
> >> were requested by Gentoo's Dev:
> >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
> >> CONFIG_SLUB_DEBUG=y
> >> CONFIG_HAVE_DMA_API_DEBUG=y
> >> CONFIG_X86_DEBUGCTLMSR=y
> >> CONFIG_PNP_DEBUG_MESSAGES=y
> >> CONFIG_AIC94XX_DEBUG=y
> >> CONFIG_USB_DEBUG=y
> >> CONFIG_DEBUG_KERNEL=y
> >> CONFIG_SCHED_DEBUG=y
> >> CONFIG_DEBUG_RT_MUTEXES=y
> >> CONFIG_DEBUG_PI_LIST=y
> >> CONFIG_DEBUG_BUGVERBOSE=y
> >> CONFIG_DEBUG_INFO=y
> >> CONFIG_DEBUG_MEMORY_INIT=y
> >> CONFIG_DEBUG_LIST=y
> >> CONFIG_DEBUG_STACKOVERFLOW=y
> >> CONFIG_DEBUG_RODATA=y
> >> CONFIG_DEBUG_RODATA_TEST=y
> >>
> >> I attached the kernel config i used for 3.2.9 to generate this oops and
> >> warnings.
> >>
> >> From the list_add warnings that come after, out of 805 warnings i processed,
> >> after masking with XXXXX the PID and next= values that kept changing in
> >> every one, i got 26 types of MD5. I also attached the files relevant as an
> >> archive to this email.
> >>
> >> The Gentoo bug i opened is sleeping, it seems nobody has the time to at
> >> least test to confirm or not the problems i'm seeing (or everybody's
> >> thinking that nobody would restart auditd so often, so the bug it's not that
> >> serious).
> >>
> >>
> >> Thank you for your time.
> >>
> >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram <aval13@gmail.com> wrote:
> >>
> >>
> >> --
> >> Linux-audit mailing list
> >> Linux-audit@redhat.com
> >> https://www.redhat.com/mailman/listinfo/linux-audit
> >
> >
> >
> > --
> > Peter Moody Google 1.650.253.7306
> > Security Engineer pgp:0xC3410038
>
>
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-03-29 1:14 ` Eric Paris
@ 2012-03-29 6:44 ` Valentin Avram
2012-04-03 16:15 ` Peter Moody
0 siblings, 1 reply; 24+ messages in thread
From: Valentin Avram @ 2012-03-29 6:44 UTC (permalink / raw)
To: Eric Paris; +Cc: linux-audit
[-- Attachment #1.1: Type: text/plain, Size: 6852 bytes --]
Yes, i know that patch. It made it into kernel 3.2.2. I tested it
successfully (oops in 3.2.1, no oops in 3.2.9), but this oops i'm seeing is
also in 3.2.9.
I monitored changelogs since 3.2.1 to 3.2.12 but there were no fixes either
in audit subsystem or in fsnotify. I'll try to reproduce in latest 3.2.13
and repost the oops, but i'm 99% confident it will be the same.
Sadly nobody except you seems to pay attention to this problem, probably
because it requires special conditions to reproduce (really, who starts and
stops auditd every 5 seconds on a production server?). We only ran into it
because one of our servers would randomly oops and then freeze about each
month after stopping and then starting
auditd
every morning (and the stop-start sequence was needed to workaround a bug
somewhere that would hang a
gzip
running on a file outside a watched folder).
Anyway, as a last note, i have a feeling that the oops is not exactly
random, there is a pattern, just that i haven't figured it out completely
yet.
Will keep you
uptodate
with the things i find out.
V.
On Mar 29, 2012 4:14 AM, "Eric Paris" <eparis@redhat.com> wrote:
> That patch fixes a BUG() . The report has a NULL ptr deref and some
> apparent list correuption.... Sadly they aren't the same....
>
> On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote:
> > fyi: this patch [1] seems to fix the issue for me. The explanation in
> > the subject would reliably oops my machine.
> >
> > [1]
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=fed474857efbed79cd390d0aee224231ca718f63
> >
> > On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody <pmoody@google.com> wrote:
> > > Are you still able to reliably reproduce this oops? I'm trying to
> > > track this down because this bug (or a very similar bug) is causing
> > > some significant headaches here at work, but I haven't had a lot of
> > > luck. I'm using usermode linux, though, so that might be interfering
> > > with things.
> > >
> > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram <aval13@gmail.com>
> wrote:
> > >> Finally i found some time and spare server to retest the oops and
> list_add
> > >> corruptions i was getting with the 3.x kernels and auditd 2.1.3.
> > >>
> > >> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and
> kernel.org's
> > >> 3.2.9.
> > >>
> > >> Both get the oops/BUG in the same way and after that, they keep
> pouring
> > >> list_add corruptions with audit_prune_tre(truncated?) and auditctl as
> comms.
> > >>
> > >> Since this is not about Gentoo's kernel only, i'll post here the oops
> in
> > >> 3.2.9 and also attach some list_add corruptions.
> > >>
> > >> 3.2.9 BUG:
> > >>
> > >> kernel: [ 301.240011] BUG: unable to handle kernel NULL pointer
> dereference
> > >> at (null)
> > >> kernel: [ 301.240305] IP: [<c1238dd0>] __list_del_entry+0x20/0xe0
> > >> kernel: [ 301.240481] *pdpt = 0000000000000000 *pde =
> f000ddc8f000ddc8
> > >> kernel: [ 301.240698] Oops: 0000 [#1] SMP
> > >> kernel: [ 301.240910]
> > >> kernel: [ 301.241030] Pid: 642, comm: fsnotify_mark Not tainted
> > >> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396
> > >> kernel: [ 301.241370] EIP: 0060:[<c1238dd0>] EFLAGS: 00010287 CPU: 6
> > >> kernel: [ 301.241498] EIP is at __list_del_entry+0x20/0xe0
> > >> kernel: [ 301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff EDX:
> > >> 00000000
> > >> kernel: [ 301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c ESP:
> > >> f47cff64
> > >> kernel: [ 301.241879] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> > >> kernel: [ 301.242005] Process fsnotify_mark (pid: 642, ti=f47ce000
> > >> task=f4f47c00 task.ti=f47ce000)
> > >> kernel: [ 301.242207] Stack:
> > >> kernel: [ 301.242327] c10813c0 f47cffa4 f4f47c00 f4e70888 f47cff7c
> > >> f47cffa4 f47cffb8 c10f6976
> > >> kernel: [ 301.242882] ffffffc3 f4f47c00 f4f47c00 00000000 f4f47c00
> > >> c10530c0 f47cff9c f47cff9c
> > >> kernel: [ 301.243438] f4fae544 f4fae544 f4c47f58 00000000 c10f68f0
> > >> f47cffe4 c1052834 00000000
> > >> kernel: [ 301.243995] Call Trace:
> > >> kernel: [ 301.244119] [<c10813c0>] ? rcu_check_callbacks+0x110/0x110
> > >> kernel: [ 301.244248] [<c10f6976>] fsnotify_mark_destroy+0x86/0x120
> > >> kernel: [ 301.244377] [<c10530c0>] ? abort_exclusive_wait+0x80/0x80
> > >> kernel: [ 301.244504] [<c10f68f0>] ? fsnotify_put_mark+0x30/0x30
> > >> kernel: [ 301.244631] [<c1052834>] kthread+0x74/0x80
> > >> kernel: [ 301.244756] [<c10527c0>] ? kthread_flush_work_fn+0x10/0x10
> > >> kernel: [ 301.244885] [<c1582ab6>] kernel_thread_helper+0x6/0xd
> > >> kernel: [ 301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89
> e5 53 83
> > >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 0f 84
> 8e 00
> > >> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 83 c4 14
> > >> kernel: [ 301.248195] EIP: [<c1238dd0>] __list_del_entry+0x20/0xe0
> SS:ESP
> > >> 0068:f47cff64
> > >> kernel: [ 301.248414] CR2: 0000000000000000
> > >> kernel: [ 301.248538] ---[ end trace 15082dbfb353f84c ]---
> > >>
> > >> The kernel was compiled with the following DEBUG support (the bolded
> one
> > >> were requested by Gentoo's Dev:
> > >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
> > >> CONFIG_SLUB_DEBUG=y
> > >> CONFIG_HAVE_DMA_API_DEBUG=y
> > >> CONFIG_X86_DEBUGCTLMSR=y
> > >> CONFIG_PNP_DEBUG_MESSAGES=y
> > >> CONFIG_AIC94XX_DEBUG=y
> > >> CONFIG_USB_DEBUG=y
> > >> CONFIG_DEBUG_KERNEL=y
> > >> CONFIG_SCHED_DEBUG=y
> > >> CONFIG_DEBUG_RT_MUTEXES=y
> > >> CONFIG_DEBUG_PI_LIST=y
> > >> CONFIG_DEBUG_BUGVERBOSE=y
> > >> CONFIG_DEBUG_INFO=y
> > >> CONFIG_DEBUG_MEMORY_INIT=y
> > >> CONFIG_DEBUG_LIST=y
> > >> CONFIG_DEBUG_STACKOVERFLOW=y
> > >> CONFIG_DEBUG_RODATA=y
> > >> CONFIG_DEBUG_RODATA_TEST=y
> > >>
> > >> I attached the kernel config i used for 3.2.9 to generate this oops
> and
> > >> warnings.
> > >>
> > >> From the list_add warnings that come after, out of 805 warnings i
> processed,
> > >> after masking with XXXXX the PID and next= values that kept changing
> in
> > >> every one, i got 26 types of MD5. I also attached the files relevant
> as an
> > >> archive to this email.
> > >>
> > >> The Gentoo bug i opened is sleeping, it seems nobody has the time to
> at
> > >> least test to confirm or not the problems i'm seeing (or everybody's
> > >> thinking that nobody would restart auditd so often, so the bug it's
> not that
> > >> serious).
> > >>
> > >>
> > >> Thank you for your time.
> > >>
> > >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram <aval13@gmail.com>
> wrote:
> > >>
> > >>
> > >> --
> > >> Linux-audit mailing list
> > >> Linux-audit@redhat.com
> > >> https://www.redhat.com/mailman/listinfo/linux-audit
> > >
> > >
> > >
> > > --
> > > Peter Moody Google 1.650.253.7306
> > > Security Engineer pgp:0xC3410038
> >
> >
> >
>
>
>
[-- Attachment #1.2: Type: text/html, Size: 9139 bytes --]
[-- Attachment #2: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-03-29 6:44 ` Valentin Avram
@ 2012-04-03 16:15 ` Peter Moody
2012-04-05 21:03 ` Peter Moody
0 siblings, 1 reply; 24+ messages in thread
From: Peter Moody @ 2012-04-03 16:15 UTC (permalink / raw)
To: Valentin Avram; +Cc: linux-audit
This may already be known, but the issue seems to be limited to watch
rules. With any watch rules, I can reliably crash my machine while
freeing a watch rule after only starting/stopping auditd a few times.
With no watch rules, I have no issues.
Cheers,
peter
On Wed, Mar 28, 2012 at 11:44 PM, Valentin Avram <aval13@gmail.com> wrote:
> Yes, i know that patch. It made it into kernel 3.2.2. I tested it
> successfully (oops in 3.2.1, no oops in 3.2.9), but this oops i'm seeing is
> also in 3.2.9.
>
> I monitored changelogs since 3.2.1 to 3.2.12 but there were no fixes either
> in audit subsystem or in fsnotify. I'll try to reproduce in latest 3.2.13
> and repost the oops, but i'm 99% confident it will be the same.
>
> Sadly nobody except you seems to pay attention to this problem, probably
> because it requires special conditions to reproduce (really, who starts and
> stops auditd every 5 seconds on a production server?). We only ran into it
> because one of our servers would randomly oops and then freeze about each
> month after stopping and then starting
>
> auditd
>
> every morning (and the stop-start sequence was needed to workaround a bug
> somewhere that would hang a
>
> gzip
>
> running on a file outside a watched folder).
>
> Anyway, as a last note, i have a feeling that the oops is not exactly
> random, there is a pattern, just that i haven't figured it out completely
> yet.
>
> Will keep you
>
> uptodate
>
> with the things i find out.
>
> V.
>
> On Mar 29, 2012 4:14 AM, "Eric Paris" <eparis@redhat.com> wrote:
>>
>> That patch fixes a BUG() . The report has a NULL ptr deref and some
>> apparent list correuption.... Sadly they aren't the same....
>>
>> On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote:
>> > fyi: this patch [1] seems to fix the issue for me. The explanation in
>> > the subject would reliably oops my machine.
>> >
>> > [1]
>> > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=fed474857efbed79cd390d0aee224231ca718f63
>> >
>> > On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody <pmoody@google.com> wrote:
>> > > Are you still able to reliably reproduce this oops? I'm trying to
>> > > track this down because this bug (or a very similar bug) is causing
>> > > some significant headaches here at work, but I haven't had a lot of
>> > > luck. I'm using usermode linux, though, so that might be interfering
>> > > with things.
>> > >
>> > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram <aval13@gmail.com>
>> > > wrote:
>> > >> Finally i found some time and spare server to retest the oops and
>> > >> list_add
>> > >> corruptions i was getting with the 3.x kernels and auditd 2.1.3.
>> > >>
>> > >> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and
>> > >> kernel.org's
>> > >> 3.2.9.
>> > >>
>> > >> Both get the oops/BUG in the same way and after that, they keep
>> > >> pouring
>> > >> list_add corruptions with audit_prune_tre(truncated?) and auditctl as
>> > >> comms.
>> > >>
>> > >> Since this is not about Gentoo's kernel only, i'll post here the oops
>> > >> in
>> > >> 3.2.9 and also attach some list_add corruptions.
>> > >>
>> > >> 3.2.9 BUG:
>> > >>
>> > >> kernel: [ 301.240011] BUG: unable to handle kernel NULL pointer
>> > >> dereference
>> > >> at (null)
>> > >> kernel: [ 301.240305] IP: [<c1238dd0>] __list_del_entry+0x20/0xe0
>> > >> kernel: [ 301.240481] *pdpt = 0000000000000000 *pde =
>> > >> f000ddc8f000ddc8
>> > >> kernel: [ 301.240698] Oops: 0000 [#1] SMP
>> > >> kernel: [ 301.240910]
>> > >> kernel: [ 301.241030] Pid: 642, comm: fsnotify_mark Not tainted
>> > >> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396
>> > >> kernel: [ 301.241370] EIP: 0060:[<c1238dd0>] EFLAGS: 00010287 CPU: 6
>> > >> kernel: [ 301.241498] EIP is at __list_del_entry+0x20/0xe0
>> > >> kernel: [ 301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff EDX:
>> > >> 00000000
>> > >> kernel: [ 301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c ESP:
>> > >> f47cff64
>> > >> kernel: [ 301.241879] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
>> > >> kernel: [ 301.242005] Process fsnotify_mark (pid: 642, ti=f47ce000
>> > >> task=f4f47c00 task.ti=f47ce000)
>> > >> kernel: [ 301.242207] Stack:
>> > >> kernel: [ 301.242327] c10813c0 f47cffa4 f4f47c00 f4e70888 f47cff7c
>> > >> f47cffa4 f47cffb8 c10f6976
>> > >> kernel: [ 301.242882] ffffffc3 f4f47c00 f4f47c00 00000000 f4f47c00
>> > >> c10530c0 f47cff9c f47cff9c
>> > >> kernel: [ 301.243438] f4fae544 f4fae544 f4c47f58 00000000 c10f68f0
>> > >> f47cffe4 c1052834 00000000
>> > >> kernel: [ 301.243995] Call Trace:
>> > >> kernel: [ 301.244119] [<c10813c0>] ?
>> > >> rcu_check_callbacks+0x110/0x110
>> > >> kernel: [ 301.244248] [<c10f6976>] fsnotify_mark_destroy+0x86/0x120
>> > >> kernel: [ 301.244377] [<c10530c0>] ? abort_exclusive_wait+0x80/0x80
>> > >> kernel: [ 301.244504] [<c10f68f0>] ? fsnotify_put_mark+0x30/0x30
>> > >> kernel: [ 301.244631] [<c1052834>] kthread+0x74/0x80
>> > >> kernel: [ 301.244756] [<c10527c0>] ?
>> > >> kthread_flush_work_fn+0x10/0x10
>> > >> kernel: [ 301.244885] [<c1582ab6>] kernel_thread_helper+0x6/0xd
>> > >> kernel: [ 301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89
>> > >> e5 53 83
>> > >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 0f 84
>> > >> 8e 00
>> > >> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 83 c4
>> > >> 14
>> > >> kernel: [ 301.248195] EIP: [<c1238dd0>] __list_del_entry+0x20/0xe0
>> > >> SS:ESP
>> > >> 0068:f47cff64
>> > >> kernel: [ 301.248414] CR2: 0000000000000000
>> > >> kernel: [ 301.248538] ---[ end trace 15082dbfb353f84c ]---
>> > >>
>> > >> The kernel was compiled with the following DEBUG support (the bolded
>> > >> one
>> > >> were requested by Gentoo's Dev:
>> > >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
>> > >> CONFIG_SLUB_DEBUG=y
>> > >> CONFIG_HAVE_DMA_API_DEBUG=y
>> > >> CONFIG_X86_DEBUGCTLMSR=y
>> > >> CONFIG_PNP_DEBUG_MESSAGES=y
>> > >> CONFIG_AIC94XX_DEBUG=y
>> > >> CONFIG_USB_DEBUG=y
>> > >> CONFIG_DEBUG_KERNEL=y
>> > >> CONFIG_SCHED_DEBUG=y
>> > >> CONFIG_DEBUG_RT_MUTEXES=y
>> > >> CONFIG_DEBUG_PI_LIST=y
>> > >> CONFIG_DEBUG_BUGVERBOSE=y
>> > >> CONFIG_DEBUG_INFO=y
>> > >> CONFIG_DEBUG_MEMORY_INIT=y
>> > >> CONFIG_DEBUG_LIST=y
>> > >> CONFIG_DEBUG_STACKOVERFLOW=y
>> > >> CONFIG_DEBUG_RODATA=y
>> > >> CONFIG_DEBUG_RODATA_TEST=y
>> > >>
>> > >> I attached the kernel config i used for 3.2.9 to generate this oops
>> > >> and
>> > >> warnings.
>> > >>
>> > >> From the list_add warnings that come after, out of 805 warnings i
>> > >> processed,
>> > >> after masking with XXXXX the PID and next= values that kept changing
>> > >> in
>> > >> every one, i got 26 types of MD5. I also attached the files relevant
>> > >> as an
>> > >> archive to this email.
>> > >>
>> > >> The Gentoo bug i opened is sleeping, it seems nobody has the time to
>> > >> at
>> > >> least test to confirm or not the problems i'm seeing (or everybody's
>> > >> thinking that nobody would restart auditd so often, so the bug it's
>> > >> not that
>> > >> serious).
>> > >>
>> > >>
>> > >> Thank you for your time.
>> > >>
>> > >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram <aval13@gmail.com>
>> > >> wrote:
>> > >>
>> > >>
>> > >> --
>> > >> Linux-audit mailing list
>> > >> Linux-audit@redhat.com
>> > >> https://www.redhat.com/mailman/listinfo/linux-audit
>> > >
>> > >
>> > >
>> > > --
>> > > Peter Moody Google 1.650.253.7306
>> > > Security Engineer pgp:0xC3410038
>> >
>> >
>> >
>>
>>
>
--
Peter Moody Google 1.650.253.7306
Security Engineer pgp:0xC3410038
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-04-03 16:15 ` Peter Moody
@ 2012-04-05 21:03 ` Peter Moody
2012-04-05 21:07 ` Eric Paris
0 siblings, 1 reply; 24+ messages in thread
From: Peter Moody @ 2012-04-05 21:03 UTC (permalink / raw)
To: Valentin Avram; +Cc: linux-audit
(please let me know if I should take this off-list)
One other thing (again, maybe already known), but this seems to be
exacerbated by SMP. On my machine, I can't reproduce the crash if I
booth with maxcpus=1.
Still hunting.
Cheers,
peter
On Tue, Apr 3, 2012 at 9:15 AM, Peter Moody <pmoody@google.com> wrote:
> This may already be known, but the issue seems to be limited to watch
> rules. With any watch rules, I can reliably crash my machine while
> freeing a watch rule after only starting/stopping auditd a few times.
> With no watch rules, I have no issues.
>
> Cheers,
> peter
>
> On Wed, Mar 28, 2012 at 11:44 PM, Valentin Avram <aval13@gmail.com> wrote:
>> Yes, i know that patch. It made it into kernel 3.2.2. I tested it
>> successfully (oops in 3.2.1, no oops in 3.2.9), but this oops i'm seeing is
>> also in 3.2.9.
>>
>> I monitored changelogs since 3.2.1 to 3.2.12 but there were no fixes either
>> in audit subsystem or in fsnotify. I'll try to reproduce in latest 3.2.13
>> and repost the oops, but i'm 99% confident it will be the same.
>>
>> Sadly nobody except you seems to pay attention to this problem, probably
>> because it requires special conditions to reproduce (really, who starts and
>> stops auditd every 5 seconds on a production server?). We only ran into it
>> because one of our servers would randomly oops and then freeze about each
>> month after stopping and then starting
>>
>> auditd
>>
>> every morning (and the stop-start sequence was needed to workaround a bug
>> somewhere that would hang a
>>
>> gzip
>>
>> running on a file outside a watched folder).
>>
>> Anyway, as a last note, i have a feeling that the oops is not exactly
>> random, there is a pattern, just that i haven't figured it out completely
>> yet.
>>
>> Will keep you
>>
>> uptodate
>>
>> with the things i find out.
>>
>> V.
>>
>> On Mar 29, 2012 4:14 AM, "Eric Paris" <eparis@redhat.com> wrote:
>>>
>>> That patch fixes a BUG() . The report has a NULL ptr deref and some
>>> apparent list correuption.... Sadly they aren't the same....
>>>
>>> On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote:
>>> > fyi: this patch [1] seems to fix the issue for me. The explanation in
>>> > the subject would reliably oops my machine.
>>> >
>>> > [1]
>>> > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=fed474857efbed79cd390d0aee224231ca718f63
>>> >
>>> > On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody <pmoody@google.com> wrote:
>>> > > Are you still able to reliably reproduce this oops? I'm trying to
>>> > > track this down because this bug (or a very similar bug) is causing
>>> > > some significant headaches here at work, but I haven't had a lot of
>>> > > luck. I'm using usermode linux, though, so that might be interfering
>>> > > with things.
>>> > >
>>> > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram <aval13@gmail.com>
>>> > > wrote:
>>> > >> Finally i found some time and spare server to retest the oops and
>>> > >> list_add
>>> > >> corruptions i was getting with the 3.x kernels and auditd 2.1.3.
>>> > >>
>>> > >> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and
>>> > >> kernel.org's
>>> > >> 3.2.9.
>>> > >>
>>> > >> Both get the oops/BUG in the same way and after that, they keep
>>> > >> pouring
>>> > >> list_add corruptions with audit_prune_tre(truncated?) and auditctl as
>>> > >> comms.
>>> > >>
>>> > >> Since this is not about Gentoo's kernel only, i'll post here the oops
>>> > >> in
>>> > >> 3.2.9 and also attach some list_add corruptions.
>>> > >>
>>> > >> 3.2.9 BUG:
>>> > >>
>>> > >> kernel: [ 301.240011] BUG: unable to handle kernel NULL pointer
>>> > >> dereference
>>> > >> at (null)
>>> > >> kernel: [ 301.240305] IP: [<c1238dd0>] __list_del_entry+0x20/0xe0
>>> > >> kernel: [ 301.240481] *pdpt = 0000000000000000 *pde =
>>> > >> f000ddc8f000ddc8
>>> > >> kernel: [ 301.240698] Oops: 0000 [#1] SMP
>>> > >> kernel: [ 301.240910]
>>> > >> kernel: [ 301.241030] Pid: 642, comm: fsnotify_mark Not tainted
>>> > >> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396
>>> > >> kernel: [ 301.241370] EIP: 0060:[<c1238dd0>] EFLAGS: 00010287 CPU: 6
>>> > >> kernel: [ 301.241498] EIP is at __list_del_entry+0x20/0xe0
>>> > >> kernel: [ 301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff EDX:
>>> > >> 00000000
>>> > >> kernel: [ 301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c ESP:
>>> > >> f47cff64
>>> > >> kernel: [ 301.241879] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
>>> > >> kernel: [ 301.242005] Process fsnotify_mark (pid: 642, ti=f47ce000
>>> > >> task=f4f47c00 task.ti=f47ce000)
>>> > >> kernel: [ 301.242207] Stack:
>>> > >> kernel: [ 301.242327] c10813c0 f47cffa4 f4f47c00 f4e70888 f47cff7c
>>> > >> f47cffa4 f47cffb8 c10f6976
>>> > >> kernel: [ 301.242882] ffffffc3 f4f47c00 f4f47c00 00000000 f4f47c00
>>> > >> c10530c0 f47cff9c f47cff9c
>>> > >> kernel: [ 301.243438] f4fae544 f4fae544 f4c47f58 00000000 c10f68f0
>>> > >> f47cffe4 c1052834 00000000
>>> > >> kernel: [ 301.243995] Call Trace:
>>> > >> kernel: [ 301.244119] [<c10813c0>] ?
>>> > >> rcu_check_callbacks+0x110/0x110
>>> > >> kernel: [ 301.244248] [<c10f6976>] fsnotify_mark_destroy+0x86/0x120
>>> > >> kernel: [ 301.244377] [<c10530c0>] ? abort_exclusive_wait+0x80/0x80
>>> > >> kernel: [ 301.244504] [<c10f68f0>] ? fsnotify_put_mark+0x30/0x30
>>> > >> kernel: [ 301.244631] [<c1052834>] kthread+0x74/0x80
>>> > >> kernel: [ 301.244756] [<c10527c0>] ?
>>> > >> kthread_flush_work_fn+0x10/0x10
>>> > >> kernel: [ 301.244885] [<c1582ab6>] kernel_thread_helper+0x6/0xd
>>> > >> kernel: [ 301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89
>>> > >> e5 53 83
>>> > >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 0f 84
>>> > >> 8e 00
>>> > >> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 83 c4
>>> > >> 14
>>> > >> kernel: [ 301.248195] EIP: [<c1238dd0>] __list_del_entry+0x20/0xe0
>>> > >> SS:ESP
>>> > >> 0068:f47cff64
>>> > >> kernel: [ 301.248414] CR2: 0000000000000000
>>> > >> kernel: [ 301.248538] ---[ end trace 15082dbfb353f84c ]---
>>> > >>
>>> > >> The kernel was compiled with the following DEBUG support (the bolded
>>> > >> one
>>> > >> were requested by Gentoo's Dev:
>>> > >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
>>> > >> CONFIG_SLUB_DEBUG=y
>>> > >> CONFIG_HAVE_DMA_API_DEBUG=y
>>> > >> CONFIG_X86_DEBUGCTLMSR=y
>>> > >> CONFIG_PNP_DEBUG_MESSAGES=y
>>> > >> CONFIG_AIC94XX_DEBUG=y
>>> > >> CONFIG_USB_DEBUG=y
>>> > >> CONFIG_DEBUG_KERNEL=y
>>> > >> CONFIG_SCHED_DEBUG=y
>>> > >> CONFIG_DEBUG_RT_MUTEXES=y
>>> > >> CONFIG_DEBUG_PI_LIST=y
>>> > >> CONFIG_DEBUG_BUGVERBOSE=y
>>> > >> CONFIG_DEBUG_INFO=y
>>> > >> CONFIG_DEBUG_MEMORY_INIT=y
>>> > >> CONFIG_DEBUG_LIST=y
>>> > >> CONFIG_DEBUG_STACKOVERFLOW=y
>>> > >> CONFIG_DEBUG_RODATA=y
>>> > >> CONFIG_DEBUG_RODATA_TEST=y
>>> > >>
>>> > >> I attached the kernel config i used for 3.2.9 to generate this oops
>>> > >> and
>>> > >> warnings.
>>> > >>
>>> > >> From the list_add warnings that come after, out of 805 warnings i
>>> > >> processed,
>>> > >> after masking with XXXXX the PID and next= values that kept changing
>>> > >> in
>>> > >> every one, i got 26 types of MD5. I also attached the files relevant
>>> > >> as an
>>> > >> archive to this email.
>>> > >>
>>> > >> The Gentoo bug i opened is sleeping, it seems nobody has the time to
>>> > >> at
>>> > >> least test to confirm or not the problems i'm seeing (or everybody's
>>> > >> thinking that nobody would restart auditd so often, so the bug it's
>>> > >> not that
>>> > >> serious).
>>> > >>
>>> > >>
>>> > >> Thank you for your time.
>>> > >>
>>> > >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram <aval13@gmail.com>
>>> > >> wrote:
>>> > >>
>>> > >>
>>> > >> --
>>> > >> Linux-audit mailing list
>>> > >> Linux-audit@redhat.com
>>> > >> https://www.redhat.com/mailman/listinfo/linux-audit
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Peter Moody Google 1.650.253.7306
>>> > > Security Engineer pgp:0xC3410038
>>> >
>>> >
>>> >
>>>
>>>
>>
>
>
>
> --
> Peter Moody Google 1.650.253.7306
> Security Engineer pgp:0xC3410038
--
Peter Moody Google 1.650.253.7306
Security Engineer pgp:0xC3410038
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-04-05 21:03 ` Peter Moody
@ 2012-04-05 21:07 ` Eric Paris
2012-04-17 17:56 ` Peter Moody
0 siblings, 1 reply; 24+ messages in thread
From: Eric Paris @ 2012-04-05 21:07 UTC (permalink / raw)
To: Peter Moody; +Cc: linux-audit
please please please keep on list. Everything you say might help track
it down!
On Thu, 2012-04-05 at 14:03 -0700, Peter Moody wrote:
> (please let me know if I should take this off-list)
>
> One other thing (again, maybe already known), but this seems to be
> exacerbated by SMP. On my machine, I can't reproduce the crash if I
> booth with maxcpus=1.
>
> Still hunting.
>
> Cheers,
> peter
>
> On Tue, Apr 3, 2012 at 9:15 AM, Peter Moody <pmoody@google.com> wrote:
> > This may already be known, but the issue seems to be limited to watch
> > rules. With any watch rules, I can reliably crash my machine while
> > freeing a watch rule after only starting/stopping auditd a few times.
> > With no watch rules, I have no issues.
> >
> > Cheers,
> > peter
> >
> > On Wed, Mar 28, 2012 at 11:44 PM, Valentin Avram <aval13@gmail.com> wrote:
> >> Yes, i know that patch. It made it into kernel 3.2.2. I tested it
> >> successfully (oops in 3.2.1, no oops in 3.2.9), but this oops i'm seeing is
> >> also in 3.2.9.
> >>
> >> I monitored changelogs since 3.2.1 to 3.2.12 but there were no fixes either
> >> in audit subsystem or in fsnotify. I'll try to reproduce in latest 3.2.13
> >> and repost the oops, but i'm 99% confident it will be the same.
> >>
> >> Sadly nobody except you seems to pay attention to this problem, probably
> >> because it requires special conditions to reproduce (really, who starts and
> >> stops auditd every 5 seconds on a production server?). We only ran into it
> >> because one of our servers would randomly oops and then freeze about each
> >> month after stopping and then starting
> >>
> >> auditd
> >>
> >> every morning (and the stop-start sequence was needed to workaround a bug
> >> somewhere that would hang a
> >>
> >> gzip
> >>
> >> running on a file outside a watched folder).
> >>
> >> Anyway, as a last note, i have a feeling that the oops is not exactly
> >> random, there is a pattern, just that i haven't figured it out completely
> >> yet.
> >>
> >> Will keep you
> >>
> >> uptodate
> >>
> >> with the things i find out.
> >>
> >> V.
> >>
> >> On Mar 29, 2012 4:14 AM, "Eric Paris" <eparis@redhat.com> wrote:
> >>>
> >>> That patch fixes a BUG() . The report has a NULL ptr deref and some
> >>> apparent list correuption.... Sadly they aren't the same....
> >>>
> >>> On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote:
> >>> > fyi: this patch [1] seems to fix the issue for me. The explanation in
> >>> > the subject would reliably oops my machine.
> >>> >
> >>> > [1]
> >>> > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=fed474857efbed79cd390d0aee224231ca718f63
> >>> >
> >>> > On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody <pmoody@google.com> wrote:
> >>> > > Are you still able to reliably reproduce this oops? I'm trying to
> >>> > > track this down because this bug (or a very similar bug) is causing
> >>> > > some significant headaches here at work, but I haven't had a lot of
> >>> > > luck. I'm using usermode linux, though, so that might be interfering
> >>> > > with things.
> >>> > >
> >>> > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram <aval13@gmail.com>
> >>> > > wrote:
> >>> > >> Finally i found some time and spare server to retest the oops and
> >>> > >> list_add
> >>> > >> corruptions i was getting with the 3.x kernels and auditd 2.1.3.
> >>> > >>
> >>> > >> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and
> >>> > >> kernel.org's
> >>> > >> 3.2.9.
> >>> > >>
> >>> > >> Both get the oops/BUG in the same way and after that, they keep
> >>> > >> pouring
> >>> > >> list_add corruptions with audit_prune_tre(truncated?) and auditctl as
> >>> > >> comms.
> >>> > >>
> >>> > >> Since this is not about Gentoo's kernel only, i'll post here the oops
> >>> > >> in
> >>> > >> 3.2.9 and also attach some list_add corruptions.
> >>> > >>
> >>> > >> 3.2.9 BUG:
> >>> > >>
> >>> > >> kernel: [ 301.240011] BUG: unable to handle kernel NULL pointer
> >>> > >> dereference
> >>> > >> at (null)
> >>> > >> kernel: [ 301.240305] IP: [<c1238dd0>] __list_del_entry+0x20/0xe0
> >>> > >> kernel: [ 301.240481] *pdpt = 0000000000000000 *pde =
> >>> > >> f000ddc8f000ddc8
> >>> > >> kernel: [ 301.240698] Oops: 0000 [#1] SMP
> >>> > >> kernel: [ 301.240910]
> >>> > >> kernel: [ 301.241030] Pid: 642, comm: fsnotify_mark Not tainted
> >>> > >> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396
> >>> > >> kernel: [ 301.241370] EIP: 0060:[<c1238dd0>] EFLAGS: 00010287 CPU: 6
> >>> > >> kernel: [ 301.241498] EIP is at __list_del_entry+0x20/0xe0
> >>> > >> kernel: [ 301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff EDX:
> >>> > >> 00000000
> >>> > >> kernel: [ 301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c ESP:
> >>> > >> f47cff64
> >>> > >> kernel: [ 301.241879] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> >>> > >> kernel: [ 301.242005] Process fsnotify_mark (pid: 642, ti=f47ce000
> >>> > >> task=f4f47c00 task.ti=f47ce000)
> >>> > >> kernel: [ 301.242207] Stack:
> >>> > >> kernel: [ 301.242327] c10813c0 f47cffa4 f4f47c00 f4e70888 f47cff7c
> >>> > >> f47cffa4 f47cffb8 c10f6976
> >>> > >> kernel: [ 301.242882] ffffffc3 f4f47c00 f4f47c00 00000000 f4f47c00
> >>> > >> c10530c0 f47cff9c f47cff9c
> >>> > >> kernel: [ 301.243438] f4fae544 f4fae544 f4c47f58 00000000 c10f68f0
> >>> > >> f47cffe4 c1052834 00000000
> >>> > >> kernel: [ 301.243995] Call Trace:
> >>> > >> kernel: [ 301.244119] [<c10813c0>] ?
> >>> > >> rcu_check_callbacks+0x110/0x110
> >>> > >> kernel: [ 301.244248] [<c10f6976>] fsnotify_mark_destroy+0x86/0x120
> >>> > >> kernel: [ 301.244377] [<c10530c0>] ? abort_exclusive_wait+0x80/0x80
> >>> > >> kernel: [ 301.244504] [<c10f68f0>] ? fsnotify_put_mark+0x30/0x30
> >>> > >> kernel: [ 301.244631] [<c1052834>] kthread+0x74/0x80
> >>> > >> kernel: [ 301.244756] [<c10527c0>] ?
> >>> > >> kthread_flush_work_fn+0x10/0x10
> >>> > >> kernel: [ 301.244885] [<c1582ab6>] kernel_thread_helper+0x6/0xd
> >>> > >> kernel: [ 301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89
> >>> > >> e5 53 83
> >>> > >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 0f 84
> >>> > >> 8e 00
> >>> > >> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 83 c4
> >>> > >> 14
> >>> > >> kernel: [ 301.248195] EIP: [<c1238dd0>] __list_del_entry+0x20/0xe0
> >>> > >> SS:ESP
> >>> > >> 0068:f47cff64
> >>> > >> kernel: [ 301.248414] CR2: 0000000000000000
> >>> > >> kernel: [ 301.248538] ---[ end trace 15082dbfb353f84c ]---
> >>> > >>
> >>> > >> The kernel was compiled with the following DEBUG support (the bolded
> >>> > >> one
> >>> > >> were requested by Gentoo's Dev:
> >>> > >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
> >>> > >> CONFIG_SLUB_DEBUG=y
> >>> > >> CONFIG_HAVE_DMA_API_DEBUG=y
> >>> > >> CONFIG_X86_DEBUGCTLMSR=y
> >>> > >> CONFIG_PNP_DEBUG_MESSAGES=y
> >>> > >> CONFIG_AIC94XX_DEBUG=y
> >>> > >> CONFIG_USB_DEBUG=y
> >>> > >> CONFIG_DEBUG_KERNEL=y
> >>> > >> CONFIG_SCHED_DEBUG=y
> >>> > >> CONFIG_DEBUG_RT_MUTEXES=y
> >>> > >> CONFIG_DEBUG_PI_LIST=y
> >>> > >> CONFIG_DEBUG_BUGVERBOSE=y
> >>> > >> CONFIG_DEBUG_INFO=y
> >>> > >> CONFIG_DEBUG_MEMORY_INIT=y
> >>> > >> CONFIG_DEBUG_LIST=y
> >>> > >> CONFIG_DEBUG_STACKOVERFLOW=y
> >>> > >> CONFIG_DEBUG_RODATA=y
> >>> > >> CONFIG_DEBUG_RODATA_TEST=y
> >>> > >>
> >>> > >> I attached the kernel config i used for 3.2.9 to generate this oops
> >>> > >> and
> >>> > >> warnings.
> >>> > >>
> >>> > >> From the list_add warnings that come after, out of 805 warnings i
> >>> > >> processed,
> >>> > >> after masking with XXXXX the PID and next= values that kept changing
> >>> > >> in
> >>> > >> every one, i got 26 types of MD5. I also attached the files relevant
> >>> > >> as an
> >>> > >> archive to this email.
> >>> > >>
> >>> > >> The Gentoo bug i opened is sleeping, it seems nobody has the time to
> >>> > >> at
> >>> > >> least test to confirm or not the problems i'm seeing (or everybody's
> >>> > >> thinking that nobody would restart auditd so often, so the bug it's
> >>> > >> not that
> >>> > >> serious).
> >>> > >>
> >>> > >>
> >>> > >> Thank you for your time.
> >>> > >>
> >>> > >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram <aval13@gmail.com>
> >>> > >> wrote:
> >>> > >>
> >>> > >>
> >>> > >> --
> >>> > >> Linux-audit mailing list
> >>> > >> Linux-audit@redhat.com
> >>> > >> https://www.redhat.com/mailman/listinfo/linux-audit
> >>> > >
> >>> > >
> >>> > >
> >>> > > --
> >>> > > Peter Moody Google 1.650.253.7306
> >>> > > Security Engineer pgp:0xC3410038
> >>> >
> >>> >
> >>> >
> >>>
> >>>
> >>
> >
> >
> >
> > --
> > Peter Moody Google 1.650.253.7306
> > Security Engineer pgp:0xC3410038
>
>
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-04-05 21:07 ` Eric Paris
@ 2012-04-17 17:56 ` Peter Moody
2012-04-17 18:24 ` Peter Moody
0 siblings, 1 reply; 24+ messages in thread
From: Peter Moody @ 2012-04-17 17:56 UTC (permalink / raw)
To: Eric Paris; +Cc: linux-audit
[-- Attachment #1: Type: text/plain, Size: 9665 bytes --]
Here's a trace with debugging turned way up plus a few extra printk's
added to fs/notify/mark.c. I'm looping through private_destroy_list
before and after the call to synchronize_srcu.
I can reproduce this reliably with kvm with 2 virtual processors:
Linux desktop 3.4.0-rc3-oops1+ #1 SMP Tue Apr 17 09:59:44 PDT 2012
x86_64 GNU/Linux
Cheers,
peter
On Thu, Apr 5, 2012 at 2:07 PM, Eric Paris <eparis@redhat.com> wrote:
> please please please keep on list. Everything you say might help track
> it down!
>
> On Thu, 2012-04-05 at 14:03 -0700, Peter Moody wrote:
>> (please let me know if I should take this off-list)
>>
>> One other thing (again, maybe already known), but this seems to be
>> exacerbated by SMP. On my machine, I can't reproduce the crash if I
>> booth with maxcpus=1.
>>
>> Still hunting.
>>
>> Cheers,
>> peter
>>
>> On Tue, Apr 3, 2012 at 9:15 AM, Peter Moody <pmoody@google.com> wrote:
>> > This may already be known, but the issue seems to be limited to watch
>> > rules. With any watch rules, I can reliably crash my machine while
>> > freeing a watch rule after only starting/stopping auditd a few times.
>> > With no watch rules, I have no issues.
>> >
>> > Cheers,
>> > peter
>> >
>> > On Wed, Mar 28, 2012 at 11:44 PM, Valentin Avram <aval13@gmail.com> wrote:
>> >> Yes, i know that patch. It made it into kernel 3.2.2. I tested it
>> >> successfully (oops in 3.2.1, no oops in 3.2.9), but this oops i'm seeing is
>> >> also in 3.2.9.
>> >>
>> >> I monitored changelogs since 3.2.1 to 3.2.12 but there were no fixes either
>> >> in audit subsystem or in fsnotify. I'll try to reproduce in latest 3.2.13
>> >> and repost the oops, but i'm 99% confident it will be the same.
>> >>
>> >> Sadly nobody except you seems to pay attention to this problem, probably
>> >> because it requires special conditions to reproduce (really, who starts and
>> >> stops auditd every 5 seconds on a production server?). We only ran into it
>> >> because one of our servers would randomly oops and then freeze about each
>> >> month after stopping and then starting
>> >>
>> >> auditd
>> >>
>> >> every morning (and the stop-start sequence was needed to workaround a bug
>> >> somewhere that would hang a
>> >>
>> >> gzip
>> >>
>> >> running on a file outside a watched folder).
>> >>
>> >> Anyway, as a last note, i have a feeling that the oops is not exactly
>> >> random, there is a pattern, just that i haven't figured it out completely
>> >> yet.
>> >>
>> >> Will keep you
>> >>
>> >> uptodate
>> >>
>> >> with the things i find out.
>> >>
>> >> V.
>> >>
>> >> On Mar 29, 2012 4:14 AM, "Eric Paris" <eparis@redhat.com> wrote:
>> >>>
>> >>> That patch fixes a BUG() . The report has a NULL ptr deref and some
>> >>> apparent list correuption.... Sadly they aren't the same....
>> >>>
>> >>> On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote:
>> >>> > fyi: this patch [1] seems to fix the issue for me. The explanation in
>> >>> > the subject would reliably oops my machine.
>> >>> >
>> >>> > [1]
>> >>> > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=fed474857efbed79cd390d0aee224231ca718f63
>> >>> >
>> >>> > On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody <pmoody@google.com> wrote:
>> >>> > > Are you still able to reliably reproduce this oops? I'm trying to
>> >>> > > track this down because this bug (or a very similar bug) is causing
>> >>> > > some significant headaches here at work, but I haven't had a lot of
>> >>> > > luck. I'm using usermode linux, though, so that might be interfering
>> >>> > > with things.
>> >>> > >
>> >>> > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram <aval13@gmail.com>
>> >>> > > wrote:
>> >>> > >> Finally i found some time and spare server to retest the oops and
>> >>> > >> list_add
>> >>> > >> corruptions i was getting with the 3.x kernels and auditd 2.1.3.
>> >>> > >>
>> >>> > >> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and
>> >>> > >> kernel.org's
>> >>> > >> 3.2.9.
>> >>> > >>
>> >>> > >> Both get the oops/BUG in the same way and after that, they keep
>> >>> > >> pouring
>> >>> > >> list_add corruptions with audit_prune_tre(truncated?) and auditctl as
>> >>> > >> comms.
>> >>> > >>
>> >>> > >> Since this is not about Gentoo's kernel only, i'll post here the oops
>> >>> > >> in
>> >>> > >> 3.2.9 and also attach some list_add corruptions.
>> >>> > >>
>> >>> > >> 3.2.9 BUG:
>> >>> > >>
>> >>> > >> kernel: [ 301.240011] BUG: unable to handle kernel NULL pointer
>> >>> > >> dereference
>> >>> > >> at (null)
>> >>> > >> kernel: [ 301.240305] IP: [<c1238dd0>] __list_del_entry+0x20/0xe0
>> >>> > >> kernel: [ 301.240481] *pdpt = 0000000000000000 *pde =
>> >>> > >> f000ddc8f000ddc8
>> >>> > >> kernel: [ 301.240698] Oops: 0000 [#1] SMP
>> >>> > >> kernel: [ 301.240910]
>> >>> > >> kernel: [ 301.241030] Pid: 642, comm: fsnotify_mark Not tainted
>> >>> > >> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396
>> >>> > >> kernel: [ 301.241370] EIP: 0060:[<c1238dd0>] EFLAGS: 00010287 CPU: 6
>> >>> > >> kernel: [ 301.241498] EIP is at __list_del_entry+0x20/0xe0
>> >>> > >> kernel: [ 301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff EDX:
>> >>> > >> 00000000
>> >>> > >> kernel: [ 301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c ESP:
>> >>> > >> f47cff64
>> >>> > >> kernel: [ 301.241879] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
>> >>> > >> kernel: [ 301.242005] Process fsnotify_mark (pid: 642, ti=f47ce000
>> >>> > >> task=f4f47c00 task.ti=f47ce000)
>> >>> > >> kernel: [ 301.242207] Stack:
>> >>> > >> kernel: [ 301.242327] c10813c0 f47cffa4 f4f47c00 f4e70888 f47cff7c
>> >>> > >> f47cffa4 f47cffb8 c10f6976
>> >>> > >> kernel: [ 301.242882] ffffffc3 f4f47c00 f4f47c00 00000000 f4f47c00
>> >>> > >> c10530c0 f47cff9c f47cff9c
>> >>> > >> kernel: [ 301.243438] f4fae544 f4fae544 f4c47f58 00000000 c10f68f0
>> >>> > >> f47cffe4 c1052834 00000000
>> >>> > >> kernel: [ 301.243995] Call Trace:
>> >>> > >> kernel: [ 301.244119] [<c10813c0>] ?
>> >>> > >> rcu_check_callbacks+0x110/0x110
>> >>> > >> kernel: [ 301.244248] [<c10f6976>] fsnotify_mark_destroy+0x86/0x120
>> >>> > >> kernel: [ 301.244377] [<c10530c0>] ? abort_exclusive_wait+0x80/0x80
>> >>> > >> kernel: [ 301.244504] [<c10f68f0>] ? fsnotify_put_mark+0x30/0x30
>> >>> > >> kernel: [ 301.244631] [<c1052834>] kthread+0x74/0x80
>> >>> > >> kernel: [ 301.244756] [<c10527c0>] ?
>> >>> > >> kthread_flush_work_fn+0x10/0x10
>> >>> > >> kernel: [ 301.244885] [<c1582ab6>] kernel_thread_helper+0x6/0xd
>> >>> > >> kernel: [ 301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89
>> >>> > >> e5 53 83
>> >>> > >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 0f 84
>> >>> > >> 8e 00
>> >>> > >> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 83 c4
>> >>> > >> 14
>> >>> > >> kernel: [ 301.248195] EIP: [<c1238dd0>] __list_del_entry+0x20/0xe0
>> >>> > >> SS:ESP
>> >>> > >> 0068:f47cff64
>> >>> > >> kernel: [ 301.248414] CR2: 0000000000000000
>> >>> > >> kernel: [ 301.248538] ---[ end trace 15082dbfb353f84c ]---
>> >>> > >>
>> >>> > >> The kernel was compiled with the following DEBUG support (the bolded
>> >>> > >> one
>> >>> > >> were requested by Gentoo's Dev:
>> >>> > >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
>> >>> > >> CONFIG_SLUB_DEBUG=y
>> >>> > >> CONFIG_HAVE_DMA_API_DEBUG=y
>> >>> > >> CONFIG_X86_DEBUGCTLMSR=y
>> >>> > >> CONFIG_PNP_DEBUG_MESSAGES=y
>> >>> > >> CONFIG_AIC94XX_DEBUG=y
>> >>> > >> CONFIG_USB_DEBUG=y
>> >>> > >> CONFIG_DEBUG_KERNEL=y
>> >>> > >> CONFIG_SCHED_DEBUG=y
>> >>> > >> CONFIG_DEBUG_RT_MUTEXES=y
>> >>> > >> CONFIG_DEBUG_PI_LIST=y
>> >>> > >> CONFIG_DEBUG_BUGVERBOSE=y
>> >>> > >> CONFIG_DEBUG_INFO=y
>> >>> > >> CONFIG_DEBUG_MEMORY_INIT=y
>> >>> > >> CONFIG_DEBUG_LIST=y
>> >>> > >> CONFIG_DEBUG_STACKOVERFLOW=y
>> >>> > >> CONFIG_DEBUG_RODATA=y
>> >>> > >> CONFIG_DEBUG_RODATA_TEST=y
>> >>> > >>
>> >>> > >> I attached the kernel config i used for 3.2.9 to generate this oops
>> >>> > >> and
>> >>> > >> warnings.
>> >>> > >>
>> >>> > >> From the list_add warnings that come after, out of 805 warnings i
>> >>> > >> processed,
>> >>> > >> after masking with XXXXX the PID and next= values that kept changing
>> >>> > >> in
>> >>> > >> every one, i got 26 types of MD5. I also attached the files relevant
>> >>> > >> as an
>> >>> > >> archive to this email.
>> >>> > >>
>> >>> > >> The Gentoo bug i opened is sleeping, it seems nobody has the time to
>> >>> > >> at
>> >>> > >> least test to confirm or not the problems i'm seeing (or everybody's
>> >>> > >> thinking that nobody would restart auditd so often, so the bug it's
>> >>> > >> not that
>> >>> > >> serious).
>> >>> > >>
>> >>> > >>
>> >>> > >> Thank you for your time.
>> >>> > >>
>> >>> > >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram <aval13@gmail.com>
>> >>> > >> wrote:
>> >>> > >>
>> >>> > >>
>> >>> > >> --
>> >>> > >> Linux-audit mailing list
>> >>> > >> Linux-audit@redhat.com
>> >>> > >> https://www.redhat.com/mailman/listinfo/linux-audit
>> >>> > >
>> >>> > >
>> >>> > >
>> >>> > > --
>> >>> > > Peter Moody Google 1.650.253.7306
>> >>> > > Security Engineer pgp:0xC3410038
>> >>> >
>> >>> >
>> >>> >
>> >>>
>> >>>
>> >>
>> >
>> >
>> >
>> > --
>> > Peter Moody Google 1.650.253.7306
>> > Security Engineer pgp:0xC3410038
>>
>>
>>
>
>
--
Peter Moody Google 1.650.253.7306
Security Engineer pgp:0xC3410038
[-- Attachment #2: trace.gz --]
[-- Type: application/x-gzip, Size: 3060 bytes --]
[-- Attachment #3: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-04-17 17:56 ` Peter Moody
@ 2012-04-17 18:24 ` Peter Moody
2012-04-17 21:54 ` Peter Moody
0 siblings, 1 reply; 24+ messages in thread
From: Peter Moody @ 2012-04-17 18:24 UTC (permalink / raw)
To: Eric Paris; +Cc: linux-audit
[-- Attachment #1: Type: text/plain, Size: 10100 bytes --]
and my config.gz
On Tue, Apr 17, 2012 at 10:56 AM, Peter Moody <pmoody@google.com> wrote:
> Here's a trace with debugging turned way up plus a few extra printk's
> added to fs/notify/mark.c. I'm looping through private_destroy_list
> before and after the call to synchronize_srcu.
>
> I can reproduce this reliably with kvm with 2 virtual processors:
> Linux desktop 3.4.0-rc3-oops1+ #1 SMP Tue Apr 17 09:59:44 PDT 2012
> x86_64 GNU/Linux
>
> Cheers,
> peter
>
> On Thu, Apr 5, 2012 at 2:07 PM, Eric Paris <eparis@redhat.com> wrote:
>> please please please keep on list. Everything you say might help track
>> it down!
>>
>> On Thu, 2012-04-05 at 14:03 -0700, Peter Moody wrote:
>>> (please let me know if I should take this off-list)
>>>
>>> One other thing (again, maybe already known), but this seems to be
>>> exacerbated by SMP. On my machine, I can't reproduce the crash if I
>>> booth with maxcpus=1.
>>>
>>> Still hunting.
>>>
>>> Cheers,
>>> peter
>>>
>>> On Tue, Apr 3, 2012 at 9:15 AM, Peter Moody <pmoody@google.com> wrote:
>>> > This may already be known, but the issue seems to be limited to watch
>>> > rules. With any watch rules, I can reliably crash my machine while
>>> > freeing a watch rule after only starting/stopping auditd a few times.
>>> > With no watch rules, I have no issues.
>>> >
>>> > Cheers,
>>> > peter
>>> >
>>> > On Wed, Mar 28, 2012 at 11:44 PM, Valentin Avram <aval13@gmail.com> wrote:
>>> >> Yes, i know that patch. It made it into kernel 3.2.2. I tested it
>>> >> successfully (oops in 3.2.1, no oops in 3.2.9), but this oops i'm seeing is
>>> >> also in 3.2.9.
>>> >>
>>> >> I monitored changelogs since 3.2.1 to 3.2.12 but there were no fixes either
>>> >> in audit subsystem or in fsnotify. I'll try to reproduce in latest 3.2.13
>>> >> and repost the oops, but i'm 99% confident it will be the same.
>>> >>
>>> >> Sadly nobody except you seems to pay attention to this problem, probably
>>> >> because it requires special conditions to reproduce (really, who starts and
>>> >> stops auditd every 5 seconds on a production server?). We only ran into it
>>> >> because one of our servers would randomly oops and then freeze about each
>>> >> month after stopping and then starting
>>> >>
>>> >> auditd
>>> >>
>>> >> every morning (and the stop-start sequence was needed to workaround a bug
>>> >> somewhere that would hang a
>>> >>
>>> >> gzip
>>> >>
>>> >> running on a file outside a watched folder).
>>> >>
>>> >> Anyway, as a last note, i have a feeling that the oops is not exactly
>>> >> random, there is a pattern, just that i haven't figured it out completely
>>> >> yet.
>>> >>
>>> >> Will keep you
>>> >>
>>> >> uptodate
>>> >>
>>> >> with the things i find out.
>>> >>
>>> >> V.
>>> >>
>>> >> On Mar 29, 2012 4:14 AM, "Eric Paris" <eparis@redhat.com> wrote:
>>> >>>
>>> >>> That patch fixes a BUG() . The report has a NULL ptr deref and some
>>> >>> apparent list correuption.... Sadly they aren't the same....
>>> >>>
>>> >>> On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote:
>>> >>> > fyi: this patch [1] seems to fix the issue for me. The explanation in
>>> >>> > the subject would reliably oops my machine.
>>> >>> >
>>> >>> > [1]
>>> >>> > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=fed474857efbed79cd390d0aee224231ca718f63
>>> >>> >
>>> >>> > On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody <pmoody@google.com> wrote:
>>> >>> > > Are you still able to reliably reproduce this oops? I'm trying to
>>> >>> > > track this down because this bug (or a very similar bug) is causing
>>> >>> > > some significant headaches here at work, but I haven't had a lot of
>>> >>> > > luck. I'm using usermode linux, though, so that might be interfering
>>> >>> > > with things.
>>> >>> > >
>>> >>> > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram <aval13@gmail.com>
>>> >>> > > wrote:
>>> >>> > >> Finally i found some time and spare server to retest the oops and
>>> >>> > >> list_add
>>> >>> > >> corruptions i was getting with the 3.x kernels and auditd 2.1.3.
>>> >>> > >>
>>> >>> > >> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and
>>> >>> > >> kernel.org's
>>> >>> > >> 3.2.9.
>>> >>> > >>
>>> >>> > >> Both get the oops/BUG in the same way and after that, they keep
>>> >>> > >> pouring
>>> >>> > >> list_add corruptions with audit_prune_tre(truncated?) and auditctl as
>>> >>> > >> comms.
>>> >>> > >>
>>> >>> > >> Since this is not about Gentoo's kernel only, i'll post here the oops
>>> >>> > >> in
>>> >>> > >> 3.2.9 and also attach some list_add corruptions.
>>> >>> > >>
>>> >>> > >> 3.2.9 BUG:
>>> >>> > >>
>>> >>> > >> kernel: [ 301.240011] BUG: unable to handle kernel NULL pointer
>>> >>> > >> dereference
>>> >>> > >> at (null)
>>> >>> > >> kernel: [ 301.240305] IP: [<c1238dd0>] __list_del_entry+0x20/0xe0
>>> >>> > >> kernel: [ 301.240481] *pdpt = 0000000000000000 *pde =
>>> >>> > >> f000ddc8f000ddc8
>>> >>> > >> kernel: [ 301.240698] Oops: 0000 [#1] SMP
>>> >>> > >> kernel: [ 301.240910]
>>> >>> > >> kernel: [ 301.241030] Pid: 642, comm: fsnotify_mark Not tainted
>>> >>> > >> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396
>>> >>> > >> kernel: [ 301.241370] EIP: 0060:[<c1238dd0>] EFLAGS: 00010287 CPU: 6
>>> >>> > >> kernel: [ 301.241498] EIP is at __list_del_entry+0x20/0xe0
>>> >>> > >> kernel: [ 301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff EDX:
>>> >>> > >> 00000000
>>> >>> > >> kernel: [ 301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c ESP:
>>> >>> > >> f47cff64
>>> >>> > >> kernel: [ 301.241879] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
>>> >>> > >> kernel: [ 301.242005] Process fsnotify_mark (pid: 642, ti=f47ce000
>>> >>> > >> task=f4f47c00 task.ti=f47ce000)
>>> >>> > >> kernel: [ 301.242207] Stack:
>>> >>> > >> kernel: [ 301.242327] c10813c0 f47cffa4 f4f47c00 f4e70888 f47cff7c
>>> >>> > >> f47cffa4 f47cffb8 c10f6976
>>> >>> > >> kernel: [ 301.242882] ffffffc3 f4f47c00 f4f47c00 00000000 f4f47c00
>>> >>> > >> c10530c0 f47cff9c f47cff9c
>>> >>> > >> kernel: [ 301.243438] f4fae544 f4fae544 f4c47f58 00000000 c10f68f0
>>> >>> > >> f47cffe4 c1052834 00000000
>>> >>> > >> kernel: [ 301.243995] Call Trace:
>>> >>> > >> kernel: [ 301.244119] [<c10813c0>] ?
>>> >>> > >> rcu_check_callbacks+0x110/0x110
>>> >>> > >> kernel: [ 301.244248] [<c10f6976>] fsnotify_mark_destroy+0x86/0x120
>>> >>> > >> kernel: [ 301.244377] [<c10530c0>] ? abort_exclusive_wait+0x80/0x80
>>> >>> > >> kernel: [ 301.244504] [<c10f68f0>] ? fsnotify_put_mark+0x30/0x30
>>> >>> > >> kernel: [ 301.244631] [<c1052834>] kthread+0x74/0x80
>>> >>> > >> kernel: [ 301.244756] [<c10527c0>] ?
>>> >>> > >> kthread_flush_work_fn+0x10/0x10
>>> >>> > >> kernel: [ 301.244885] [<c1582ab6>] kernel_thread_helper+0x6/0xd
>>> >>> > >> kernel: [ 301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89
>>> >>> > >> e5 53 83
>>> >>> > >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 0f 84
>>> >>> > >> 8e 00
>>> >>> > >> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 83 c4
>>> >>> > >> 14
>>> >>> > >> kernel: [ 301.248195] EIP: [<c1238dd0>] __list_del_entry+0x20/0xe0
>>> >>> > >> SS:ESP
>>> >>> > >> 0068:f47cff64
>>> >>> > >> kernel: [ 301.248414] CR2: 0000000000000000
>>> >>> > >> kernel: [ 301.248538] ---[ end trace 15082dbfb353f84c ]---
>>> >>> > >>
>>> >>> > >> The kernel was compiled with the following DEBUG support (the bolded
>>> >>> > >> one
>>> >>> > >> were requested by Gentoo's Dev:
>>> >>> > >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
>>> >>> > >> CONFIG_SLUB_DEBUG=y
>>> >>> > >> CONFIG_HAVE_DMA_API_DEBUG=y
>>> >>> > >> CONFIG_X86_DEBUGCTLMSR=y
>>> >>> > >> CONFIG_PNP_DEBUG_MESSAGES=y
>>> >>> > >> CONFIG_AIC94XX_DEBUG=y
>>> >>> > >> CONFIG_USB_DEBUG=y
>>> >>> > >> CONFIG_DEBUG_KERNEL=y
>>> >>> > >> CONFIG_SCHED_DEBUG=y
>>> >>> > >> CONFIG_DEBUG_RT_MUTEXES=y
>>> >>> > >> CONFIG_DEBUG_PI_LIST=y
>>> >>> > >> CONFIG_DEBUG_BUGVERBOSE=y
>>> >>> > >> CONFIG_DEBUG_INFO=y
>>> >>> > >> CONFIG_DEBUG_MEMORY_INIT=y
>>> >>> > >> CONFIG_DEBUG_LIST=y
>>> >>> > >> CONFIG_DEBUG_STACKOVERFLOW=y
>>> >>> > >> CONFIG_DEBUG_RODATA=y
>>> >>> > >> CONFIG_DEBUG_RODATA_TEST=y
>>> >>> > >>
>>> >>> > >> I attached the kernel config i used for 3.2.9 to generate this oops
>>> >>> > >> and
>>> >>> > >> warnings.
>>> >>> > >>
>>> >>> > >> From the list_add warnings that come after, out of 805 warnings i
>>> >>> > >> processed,
>>> >>> > >> after masking with XXXXX the PID and next= values that kept changing
>>> >>> > >> in
>>> >>> > >> every one, i got 26 types of MD5. I also attached the files relevant
>>> >>> > >> as an
>>> >>> > >> archive to this email.
>>> >>> > >>
>>> >>> > >> The Gentoo bug i opened is sleeping, it seems nobody has the time to
>>> >>> > >> at
>>> >>> > >> least test to confirm or not the problems i'm seeing (or everybody's
>>> >>> > >> thinking that nobody would restart auditd so often, so the bug it's
>>> >>> > >> not that
>>> >>> > >> serious).
>>> >>> > >>
>>> >>> > >>
>>> >>> > >> Thank you for your time.
>>> >>> > >>
>>> >>> > >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram <aval13@gmail.com>
>>> >>> > >> wrote:
>>> >>> > >>
>>> >>> > >>
>>> >>> > >> --
>>> >>> > >> Linux-audit mailing list
>>> >>> > >> Linux-audit@redhat.com
>>> >>> > >> https://www.redhat.com/mailman/listinfo/linux-audit
>>> >>> > >
>>> >>> > >
>>> >>> > >
>>> >>> > > --
>>> >>> > > Peter Moody Google 1.650.253.7306
>>> >>> > > Security Engineer pgp:0xC3410038
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>>
>>> >>>
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Peter Moody Google 1.650.253.7306
>>> > Security Engineer pgp:0xC3410038
>>>
>>>
>>>
>>
>>
>
>
>
> --
> Peter Moody Google 1.650.253.7306
> Security Engineer pgp:0xC3410038
--
Peter Moody Google 1.650.253.7306
Security Engineer pgp:0xC3410038
[-- Attachment #2: config.gz --]
[-- Type: application/x-gzip, Size: 34351 bytes --]
[-- Attachment #3: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-04-17 18:24 ` Peter Moody
@ 2012-04-17 21:54 ` Peter Moody
2012-04-21 2:14 ` Marcelo Cerri
0 siblings, 1 reply; 24+ messages in thread
From: Peter Moody @ 2012-04-17 21:54 UTC (permalink / raw)
To: Eric Paris; +Cc: linux-audit
Last thing. moving synchronize_srcu(&fsnotify_mark_srcu) out of the
for(;;) loop in fs/notify/mark.c appears to solve the stability issues
for me. I don't know enough about kernel internals to determine if
this is doing lots of other bad things to my system or not.
Cheers,
peter
On Tue, Apr 17, 2012 at 11:24 AM, Peter Moody <pmoody@google.com> wrote:
> and my config.gz
>
> On Tue, Apr 17, 2012 at 10:56 AM, Peter Moody <pmoody@google.com> wrote:
>> Here's a trace with debugging turned way up plus a few extra printk's
>> added to fs/notify/mark.c. I'm looping through private_destroy_list
>> before and after the call to synchronize_srcu.
>>
>> I can reproduce this reliably with kvm with 2 virtual processors:
>> Linux desktop 3.4.0-rc3-oops1+ #1 SMP Tue Apr 17 09:59:44 PDT 2012
>> x86_64 GNU/Linux
>>
>> Cheers,
>> peter
>>
>> On Thu, Apr 5, 2012 at 2:07 PM, Eric Paris <eparis@redhat.com> wrote:
>>> please please please keep on list. Everything you say might help track
>>> it down!
>>>
>>> On Thu, 2012-04-05 at 14:03 -0700, Peter Moody wrote:
>>>> (please let me know if I should take this off-list)
>>>>
>>>> One other thing (again, maybe already known), but this seems to be
>>>> exacerbated by SMP. On my machine, I can't reproduce the crash if I
>>>> booth with maxcpus=1.
>>>>
>>>> Still hunting.
>>>>
>>>> Cheers,
>>>> peter
>>>>
>>>> On Tue, Apr 3, 2012 at 9:15 AM, Peter Moody <pmoody@google.com> wrote:
>>>> > This may already be known, but the issue seems to be limited to watch
>>>> > rules. With any watch rules, I can reliably crash my machine while
>>>> > freeing a watch rule after only starting/stopping auditd a few times.
>>>> > With no watch rules, I have no issues.
>>>> >
>>>> > Cheers,
>>>> > peter
>>>> >
>>>> > On Wed, Mar 28, 2012 at 11:44 PM, Valentin Avram <aval13@gmail.com> wrote:
>>>> >> Yes, i know that patch. It made it into kernel 3.2.2. I tested it
>>>> >> successfully (oops in 3.2.1, no oops in 3.2.9), but this oops i'm seeing is
>>>> >> also in 3.2.9.
>>>> >>
>>>> >> I monitored changelogs since 3.2.1 to 3.2.12 but there were no fixes either
>>>> >> in audit subsystem or in fsnotify. I'll try to reproduce in latest 3.2.13
>>>> >> and repost the oops, but i'm 99% confident it will be the same.
>>>> >>
>>>> >> Sadly nobody except you seems to pay attention to this problem, probably
>>>> >> because it requires special conditions to reproduce (really, who starts and
>>>> >> stops auditd every 5 seconds on a production server?). We only ran into it
>>>> >> because one of our servers would randomly oops and then freeze about each
>>>> >> month after stopping and then starting
>>>> >>
>>>> >> auditd
>>>> >>
>>>> >> every morning (and the stop-start sequence was needed to workaround a bug
>>>> >> somewhere that would hang a
>>>> >>
>>>> >> gzip
>>>> >>
>>>> >> running on a file outside a watched folder).
>>>> >>
>>>> >> Anyway, as a last note, i have a feeling that the oops is not exactly
>>>> >> random, there is a pattern, just that i haven't figured it out completely
>>>> >> yet.
>>>> >>
>>>> >> Will keep you
>>>> >>
>>>> >> uptodate
>>>> >>
>>>> >> with the things i find out.
>>>> >>
>>>> >> V.
>>>> >>
>>>> >> On Mar 29, 2012 4:14 AM, "Eric Paris" <eparis@redhat.com> wrote:
>>>> >>>
>>>> >>> That patch fixes a BUG() . The report has a NULL ptr deref and some
>>>> >>> apparent list correuption.... Sadly they aren't the same....
>>>> >>>
>>>> >>> On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote:
>>>> >>> > fyi: this patch [1] seems to fix the issue for me. The explanation in
>>>> >>> > the subject would reliably oops my machine.
>>>> >>> >
>>>> >>> > [1]
>>>> >>> > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=fed474857efbed79cd390d0aee224231ca718f63
>>>> >>> >
>>>> >>> > On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody <pmoody@google.com> wrote:
>>>> >>> > > Are you still able to reliably reproduce this oops? I'm trying to
>>>> >>> > > track this down because this bug (or a very similar bug) is causing
>>>> >>> > > some significant headaches here at work, but I haven't had a lot of
>>>> >>> > > luck. I'm using usermode linux, though, so that might be interfering
>>>> >>> > > with things.
>>>> >>> > >
>>>> >>> > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram <aval13@gmail.com>
>>>> >>> > > wrote:
>>>> >>> > >> Finally i found some time and spare server to retest the oops and
>>>> >>> > >> list_add
>>>> >>> > >> corruptions i was getting with the 3.x kernels and auditd 2.1.3.
>>>> >>> > >>
>>>> >>> > >> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and
>>>> >>> > >> kernel.org's
>>>> >>> > >> 3.2.9.
>>>> >>> > >>
>>>> >>> > >> Both get the oops/BUG in the same way and after that, they keep
>>>> >>> > >> pouring
>>>> >>> > >> list_add corruptions with audit_prune_tre(truncated?) and auditctl as
>>>> >>> > >> comms.
>>>> >>> > >>
>>>> >>> > >> Since this is not about Gentoo's kernel only, i'll post here the oops
>>>> >>> > >> in
>>>> >>> > >> 3.2.9 and also attach some list_add corruptions.
>>>> >>> > >>
>>>> >>> > >> 3.2.9 BUG:
>>>> >>> > >>
>>>> >>> > >> kernel: [ 301.240011] BUG: unable to handle kernel NULL pointer
>>>> >>> > >> dereference
>>>> >>> > >> at (null)
>>>> >>> > >> kernel: [ 301.240305] IP: [<c1238dd0>] __list_del_entry+0x20/0xe0
>>>> >>> > >> kernel: [ 301.240481] *pdpt = 0000000000000000 *pde =
>>>> >>> > >> f000ddc8f000ddc8
>>>> >>> > >> kernel: [ 301.240698] Oops: 0000 [#1] SMP
>>>> >>> > >> kernel: [ 301.240910]
>>>> >>> > >> kernel: [ 301.241030] Pid: 642, comm: fsnotify_mark Not tainted
>>>> >>> > >> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396
>>>> >>> > >> kernel: [ 301.241370] EIP: 0060:[<c1238dd0>] EFLAGS: 00010287 CPU: 6
>>>> >>> > >> kernel: [ 301.241498] EIP is at __list_del_entry+0x20/0xe0
>>>> >>> > >> kernel: [ 301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff EDX:
>>>> >>> > >> 00000000
>>>> >>> > >> kernel: [ 301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c ESP:
>>>> >>> > >> f47cff64
>>>> >>> > >> kernel: [ 301.241879] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
>>>> >>> > >> kernel: [ 301.242005] Process fsnotify_mark (pid: 642, ti=f47ce000
>>>> >>> > >> task=f4f47c00 task.ti=f47ce000)
>>>> >>> > >> kernel: [ 301.242207] Stack:
>>>> >>> > >> kernel: [ 301.242327] c10813c0 f47cffa4 f4f47c00 f4e70888 f47cff7c
>>>> >>> > >> f47cffa4 f47cffb8 c10f6976
>>>> >>> > >> kernel: [ 301.242882] ffffffc3 f4f47c00 f4f47c00 00000000 f4f47c00
>>>> >>> > >> c10530c0 f47cff9c f47cff9c
>>>> >>> > >> kernel: [ 301.243438] f4fae544 f4fae544 f4c47f58 00000000 c10f68f0
>>>> >>> > >> f47cffe4 c1052834 00000000
>>>> >>> > >> kernel: [ 301.243995] Call Trace:
>>>> >>> > >> kernel: [ 301.244119] [<c10813c0>] ?
>>>> >>> > >> rcu_check_callbacks+0x110/0x110
>>>> >>> > >> kernel: [ 301.244248] [<c10f6976>] fsnotify_mark_destroy+0x86/0x120
>>>> >>> > >> kernel: [ 301.244377] [<c10530c0>] ? abort_exclusive_wait+0x80/0x80
>>>> >>> > >> kernel: [ 301.244504] [<c10f68f0>] ? fsnotify_put_mark+0x30/0x30
>>>> >>> > >> kernel: [ 301.244631] [<c1052834>] kthread+0x74/0x80
>>>> >>> > >> kernel: [ 301.244756] [<c10527c0>] ?
>>>> >>> > >> kthread_flush_work_fn+0x10/0x10
>>>> >>> > >> kernel: [ 301.244885] [<c1582ab6>] kernel_thread_helper+0x6/0xd
>>>> >>> > >> kernel: [ 301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89
>>>> >>> > >> e5 53 83
>>>> >>> > >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 0f 84
>>>> >>> > >> 8e 00
>>>> >>> > >> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 83 c4
>>>> >>> > >> 14
>>>> >>> > >> kernel: [ 301.248195] EIP: [<c1238dd0>] __list_del_entry+0x20/0xe0
>>>> >>> > >> SS:ESP
>>>> >>> > >> 0068:f47cff64
>>>> >>> > >> kernel: [ 301.248414] CR2: 0000000000000000
>>>> >>> > >> kernel: [ 301.248538] ---[ end trace 15082dbfb353f84c ]---
>>>> >>> > >>
>>>> >>> > >> The kernel was compiled with the following DEBUG support (the bolded
>>>> >>> > >> one
>>>> >>> > >> were requested by Gentoo's Dev:
>>>> >>> > >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
>>>> >>> > >> CONFIG_SLUB_DEBUG=y
>>>> >>> > >> CONFIG_HAVE_DMA_API_DEBUG=y
>>>> >>> > >> CONFIG_X86_DEBUGCTLMSR=y
>>>> >>> > >> CONFIG_PNP_DEBUG_MESSAGES=y
>>>> >>> > >> CONFIG_AIC94XX_DEBUG=y
>>>> >>> > >> CONFIG_USB_DEBUG=y
>>>> >>> > >> CONFIG_DEBUG_KERNEL=y
>>>> >>> > >> CONFIG_SCHED_DEBUG=y
>>>> >>> > >> CONFIG_DEBUG_RT_MUTEXES=y
>>>> >>> > >> CONFIG_DEBUG_PI_LIST=y
>>>> >>> > >> CONFIG_DEBUG_BUGVERBOSE=y
>>>> >>> > >> CONFIG_DEBUG_INFO=y
>>>> >>> > >> CONFIG_DEBUG_MEMORY_INIT=y
>>>> >>> > >> CONFIG_DEBUG_LIST=y
>>>> >>> > >> CONFIG_DEBUG_STACKOVERFLOW=y
>>>> >>> > >> CONFIG_DEBUG_RODATA=y
>>>> >>> > >> CONFIG_DEBUG_RODATA_TEST=y
>>>> >>> > >>
>>>> >>> > >> I attached the kernel config i used for 3.2.9 to generate this oops
>>>> >>> > >> and
>>>> >>> > >> warnings.
>>>> >>> > >>
>>>> >>> > >> From the list_add warnings that come after, out of 805 warnings i
>>>> >>> > >> processed,
>>>> >>> > >> after masking with XXXXX the PID and next= values that kept changing
>>>> >>> > >> in
>>>> >>> > >> every one, i got 26 types of MD5. I also attached the files relevant
>>>> >>> > >> as an
>>>> >>> > >> archive to this email.
>>>> >>> > >>
>>>> >>> > >> The Gentoo bug i opened is sleeping, it seems nobody has the time to
>>>> >>> > >> at
>>>> >>> > >> least test to confirm or not the problems i'm seeing (or everybody's
>>>> >>> > >> thinking that nobody would restart auditd so often, so the bug it's
>>>> >>> > >> not that
>>>> >>> > >> serious).
>>>> >>> > >>
>>>> >>> > >>
>>>> >>> > >> Thank you for your time.
>>>> >>> > >>
>>>> >>> > >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram <aval13@gmail.com>
>>>> >>> > >> wrote:
>>>> >>> > >>
>>>> >>> > >>
>>>> >>> > >> --
>>>> >>> > >> Linux-audit mailing list
>>>> >>> > >> Linux-audit@redhat.com
>>>> >>> > >> https://www.redhat.com/mailman/listinfo/linux-audit
>>>> >>> > >
>>>> >>> > >
>>>> >>> > >
>>>> >>> > > --
>>>> >>> > > Peter Moody Google 1.650.253.7306
>>>> >>> > > Security Engineer pgp:0xC3410038
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>>
>>>> >>>
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Peter Moody Google 1.650.253.7306
>>>> > Security Engineer pgp:0xC3410038
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>> --
>> Peter Moody Google 1.650.253.7306
>> Security Engineer pgp:0xC3410038
>
>
>
> --
> Peter Moody Google 1.650.253.7306
> Security Engineer pgp:0xC3410038
--
Peter Moody Google 1.650.253.7306
Security Engineer pgp:0xC3410038
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-04-17 21:54 ` Peter Moody
@ 2012-04-21 2:14 ` Marcelo Cerri
2012-04-23 16:05 ` Peter Moody
2012-04-23 16:26 ` Eric Paris
0 siblings, 2 replies; 24+ messages in thread
From: Marcelo Cerri @ 2012-04-21 2:14 UTC (permalink / raw)
To: Peter Moody, Valentin Avram; +Cc: linux-audit
I took a look at the source code and made some tests. It seems to be a
problem with the reference count of the fsnotify_mark structure.
This error occurs because the fsnotify_mark_destroy function
(which runs in a separated kthread) is trying to iterate through a mark
that is already freed.
Looking at the fsnotify_destroy_mark function (not confuse with
fsnotify_mark_destroy), which adds a mark to destroy_list to be freed
later by fsnotify_mark_destroy, I noticed that it does not increment
the reference count for the reference added to the destroy_list and
usually the callers dispose the references they held after calling
fsnotify_destroy_mark.
The patch below increments the reference count of a mark when it is
added to the destroy list. It seems to solve the issue and it doesn't
seem to cause any memory leak. Please, can you make some tests in your
environments and let me know if there is any problem with this patch.
Regarding the synchronize_scru call, I don't think it's causing this
error. Probably it just make it more frequently because it forces all
the cpus to schedule, giving the chance to someone else to free the
mark.
---
fs/notify/mark.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/fs/notify/mark.c b/fs/notify/mark.c
index f104d56..2985fff 100644
--- a/fs/notify/mark.c
+++ b/fs/notify/mark.c
@@ -150,6 +150,7 @@ void fsnotify_destroy_mark(struct fsnotify_mark
*mark) spin_unlock(&group->mark_lock);
spin_unlock(&mark->lock);
+ fsnotify_get_mark(mark);
spin_lock(&destroy_lock);
list_add(&mark->destroy_list, &destroy_list);
spin_unlock(&destroy_lock);
--
1.7.9.4
On Tue, 17 Apr 2012 14:54:29 -0700
Peter Moody <pmoody@google.com> wrote:
> Last thing. moving synchronize_srcu(&fsnotify_mark_srcu) out of the
> for(;;) loop in fs/notify/mark.c appears to solve the stability issues
> for me. I don't know enough about kernel internals to determine if
> this is doing lots of other bad things to my system or not.
>
> Cheers,
> peter
>
> On Tue, Apr 17, 2012 at 11:24 AM, Peter Moody <pmoody@google.com>
> wrote:
> > and my config.gz
> >
> > On Tue, Apr 17, 2012 at 10:56 AM, Peter Moody <pmoody@google.com>
> > wrote:
> >> Here's a trace with debugging turned way up plus a few extra
> >> printk's added to fs/notify/mark.c. I'm looping through
> >> private_destroy_list before and after the call to synchronize_srcu.
> >>
> >> I can reproduce this reliably with kvm with 2 virtual processors:
> >> Linux desktop 3.4.0-rc3-oops1+ #1 SMP Tue Apr 17 09:59:44 PDT 2012
> >> x86_64 GNU/Linux
> >>
> >> Cheers,
> >> peter
> >>
> >> On Thu, Apr 5, 2012 at 2:07 PM, Eric Paris <eparis@redhat.com>
> >> wrote:
> >>> please please please keep on list. Everything you say might help
> >>> track it down!
> >>>
> >>> On Thu, 2012-04-05 at 14:03 -0700, Peter Moody wrote:
> >>>> (please let me know if I should take this off-list)
> >>>>
> >>>> One other thing (again, maybe already known), but this seems to
> >>>> be exacerbated by SMP. On my machine, I can't reproduce the
> >>>> crash if I booth with maxcpus=1.
> >>>>
> >>>> Still hunting.
> >>>>
> >>>> Cheers,
> >>>> peter
> >>>>
> >>>> On Tue, Apr 3, 2012 at 9:15 AM, Peter Moody <pmoody@google.com>
> >>>> wrote:
> >>>> > This may already be known, but the issue seems to be limited
> >>>> > to watch rules. With any watch rules, I can reliably crash my
> >>>> > machine while freeing a watch rule after only
> >>>> > starting/stopping auditd a few times. With no watch rules, I
> >>>> > have no issues.
> >>>> >
> >>>> > Cheers,
> >>>> > peter
> >>>> >
> >>>> > On Wed, Mar 28, 2012 at 11:44 PM, Valentin Avram
> >>>> > <aval13@gmail.com> wrote:
> >>>> >> Yes, i know that patch. It made it into kernel 3.2.2. I
> >>>> >> tested it successfully (oops in 3.2.1, no oops in 3.2.9), but
> >>>> >> this oops i'm seeing is also in 3.2.9.
> >>>> >>
> >>>> >> I monitored changelogs since 3.2.1 to 3.2.12 but there were
> >>>> >> no fixes either in audit subsystem or in fsnotify. I'll try
> >>>> >> to reproduce in latest 3.2.13 and repost the oops, but i'm
> >>>> >> 99% confident it will be the same.
> >>>> >>
> >>>> >> Sadly nobody except you seems to pay attention to this
> >>>> >> problem, probably because it requires special conditions to
> >>>> >> reproduce (really, who starts and stops auditd every 5
> >>>> >> seconds on a production server?). We only ran into it because
> >>>> >> one of our servers would randomly oops and then freeze about
> >>>> >> each month after stopping and then starting
> >>>> >>
> >>>> >> auditd
> >>>> >>
> >>>> >> every morning (and the stop-start sequence was needed to
> >>>> >> workaround a bug somewhere that would hang a
> >>>> >>
> >>>> >> gzip
> >>>> >>
> >>>> >> running on a file outside a watched folder).
> >>>> >>
> >>>> >> Anyway, as a last note, i have a feeling that the oops is not
> >>>> >> exactly random, there is a pattern, just that i haven't
> >>>> >> figured it out completely yet.
> >>>> >>
> >>>> >> Will keep you
> >>>> >>
> >>>> >> uptodate
> >>>> >>
> >>>> >> with the things i find out.
> >>>> >>
> >>>> >> V.
> >>>> >>
> >>>> >> On Mar 29, 2012 4:14 AM, "Eric Paris" <eparis@redhat.com>
> >>>> >> wrote:
> >>>> >>>
> >>>> >>> That patch fixes a BUG() . The report has a NULL ptr deref
> >>>> >>> and some apparent list correuption.... Sadly they aren't
> >>>> >>> the same....
> >>>> >>>
> >>>> >>> On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote:
> >>>> >>> > fyi: this patch [1] seems to fix the issue for me. The
> >>>> >>> > explanation in the subject would reliably oops my machine.
> >>>> >>> >
> >>>> >>> > [1]
> >>>> >>> > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=fed474857efbed79cd390d0aee224231ca718f63
> >>>> >>> >
> >>>> >>> > On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody
> >>>> >>> > <pmoody@google.com> wrote:
> >>>> >>> > > Are you still able to reliably reproduce this oops? I'm
> >>>> >>> > > trying to track this down because this bug (or a very
> >>>> >>> > > similar bug) is causing some significant headaches here
> >>>> >>> > > at work, but I haven't had a lot of luck. I'm using
> >>>> >>> > > usermode linux, though, so that might be interfering
> >>>> >>> > > with things.
> >>>> >>> > >
> >>>> >>> > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram
> >>>> >>> > > <aval13@gmail.com> wrote:
> >>>> >>> > >> Finally i found some time and spare server to retest
> >>>> >>> > >> the oops and list_add
> >>>> >>> > >> corruptions i was getting with the 3.x kernels and
> >>>> >>> > >> auditd 2.1.3.
> >>>> >>> > >>
> >>>> >>> > >> I tested now with gentoo's latest stable
> >>>> >>> > >> 3.2.1-gentoo-r2 and kernel.org's
> >>>> >>> > >> 3.2.9.
> >>>> >>> > >>
> >>>> >>> > >> Both get the oops/BUG in the same way and after that,
> >>>> >>> > >> they keep pouring
> >>>> >>> > >> list_add corruptions with audit_prune_tre(truncated?)
> >>>> >>> > >> and auditctl as comms.
> >>>> >>> > >>
> >>>> >>> > >> Since this is not about Gentoo's kernel only, i'll post
> >>>> >>> > >> here the oops in
> >>>> >>> > >> 3.2.9 and also attach some list_add corruptions.
> >>>> >>> > >>
> >>>> >>> > >> 3.2.9 BUG:
> >>>> >>> > >>
> >>>> >>> > >> kernel: [ 301.240011] BUG: unable to handle kernel
> >>>> >>> > >> NULL pointer dereference
> >>>> >>> > >> at (null)
> >>>> >>> > >> kernel: [ 301.240305] IP: [<c1238dd0>]
> >>>> >>> > >> __list_del_entry+0x20/0xe0 kernel: [ 301.240481] *pdpt
> >>>> >>> > >> = 0000000000000000 *pde = f000ddc8f000ddc8
> >>>> >>> > >> kernel: [ 301.240698] Oops: 0000 [#1] SMP
> >>>> >>> > >> kernel: [ 301.240910]
> >>>> >>> > >> kernel: [ 301.241030] Pid: 642, comm: fsnotify_mark
> >>>> >>> > >> Not tainted 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge
> >>>> >>> > >> 2950/0CX396 kernel: [ 301.241370] EIP:
> >>>> >>> > >> 0060:[<c1238dd0>] EFLAGS: 00010287 CPU: 6 kernel:
> >>>> >>> > >> [ 301.241498] EIP is at __list_del_entry+0x20/0xe0
> >>>> >>> > >> kernel: [ 301.241623] EAX: f4fae544 EBX: f47cffa4 ECX:
> >>>> >>> > >> ffffffff EDX: 00000000 kernel: [ 301.241751] ESI:
> >>>> >>> > >> f4fae544 EDI: f4fae508 EBP: f47cff7c ESP: f47cff64
> >>>> >>> > >> kernel: [ 301.241879] DS: 007b ES: 007b FS: 00d8 GS:
> >>>> >>> > >> 0000 SS: 0068 kernel: [ 301.242005] Process
> >>>> >>> > >> fsnotify_mark (pid: 642, ti=f47ce000 task=f4f47c00
> >>>> >>> > >> task.ti=f47ce000) kernel: [ 301.242207] Stack:
> >>>> >>> > >> kernel: [ 301.242327] c10813c0 f47cffa4 f4f47c00
> >>>> >>> > >> f4e70888 f47cff7c f47cffa4 f47cffb8 c10f6976
> >>>> >>> > >> kernel: [ 301.242882] ffffffc3 f4f47c00 f4f47c00
> >>>> >>> > >> 00000000 f4f47c00 c10530c0 f47cff9c f47cff9c
> >>>> >>> > >> kernel: [ 301.243438] f4fae544 f4fae544 f4c47f58
> >>>> >>> > >> 00000000 c10f68f0 f47cffe4 c1052834 00000000
> >>>> >>> > >> kernel: [ 301.243995] Call Trace:
> >>>> >>> > >> kernel: [ 301.244119] [<c10813c0>] ?
> >>>> >>> > >> rcu_check_callbacks+0x110/0x110
> >>>> >>> > >> kernel: [ 301.244248] [<c10f6976>]
> >>>> >>> > >> fsnotify_mark_destroy+0x86/0x120 kernel: [ 301.244377]
> >>>> >>> > >> [<c10530c0>] ? abort_exclusive_wait+0x80/0x80 kernel:
> >>>> >>> > >> [ 301.244504] [<c10f68f0>] ?
> >>>> >>> > >> fsnotify_put_mark+0x30/0x30 kernel: [ 301.244631]
> >>>> >>> > >> [<c1052834>] kthread+0x74/0x80 kernel: [ 301.244756]
> >>>> >>> > >> [<c10527c0>] ? kthread_flush_work_fn+0x10/0x10 kernel:
> >>>> >>> > >> [ 301.244885] [<c1582ab6>]
> >>>> >>> > >> kernel_thread_helper+0x6/0xd kernel: [ 301.245011]
> >>>> >>> > >> Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89 e5 53 83
> >>>> >>> > >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00
> >>>> >>> > >> 02 20 00 0f 84 8e 00 00 00 <8b> 1a 39 d8 75 62 8b 59 04
> >>>> >>> > >> 39 d8 75 35 89 51 04 89 0a 83 c4 14
> >>>> >>> > >> kernel: [ 301.248195] EIP: [<c1238dd0>]
> >>>> >>> > >> __list_del_entry+0x20/0xe0 SS:ESP
> >>>> >>> > >> 0068:f47cff64
> >>>> >>> > >> kernel: [ 301.248414] CR2: 0000000000000000
> >>>> >>> > >> kernel: [ 301.248538] ---[ end trace
> >>>> >>> > >> 15082dbfb353f84c ]---
> >>>> >>> > >>
> >>>> >>> > >> The kernel was compiled with the following DEBUG
> >>>> >>> > >> support (the bolded one
> >>>> >>> > >> were requested by Gentoo's Dev:
> >>>> >>> > >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
> >>>> >>> > >> CONFIG_SLUB_DEBUG=y
> >>>> >>> > >> CONFIG_HAVE_DMA_API_DEBUG=y
> >>>> >>> > >> CONFIG_X86_DEBUGCTLMSR=y
> >>>> >>> > >> CONFIG_PNP_DEBUG_MESSAGES=y
> >>>> >>> > >> CONFIG_AIC94XX_DEBUG=y
> >>>> >>> > >> CONFIG_USB_DEBUG=y
> >>>> >>> > >> CONFIG_DEBUG_KERNEL=y
> >>>> >>> > >> CONFIG_SCHED_DEBUG=y
> >>>> >>> > >> CONFIG_DEBUG_RT_MUTEXES=y
> >>>> >>> > >> CONFIG_DEBUG_PI_LIST=y
> >>>> >>> > >> CONFIG_DEBUG_BUGVERBOSE=y
> >>>> >>> > >> CONFIG_DEBUG_INFO=y
> >>>> >>> > >> CONFIG_DEBUG_MEMORY_INIT=y
> >>>> >>> > >> CONFIG_DEBUG_LIST=y
> >>>> >>> > >> CONFIG_DEBUG_STACKOVERFLOW=y
> >>>> >>> > >> CONFIG_DEBUG_RODATA=y
> >>>> >>> > >> CONFIG_DEBUG_RODATA_TEST=y
> >>>> >>> > >>
> >>>> >>> > >> I attached the kernel config i used for 3.2.9 to
> >>>> >>> > >> generate this oops and
> >>>> >>> > >> warnings.
> >>>> >>> > >>
> >>>> >>> > >> From the list_add warnings that come after, out of 805
> >>>> >>> > >> warnings i processed,
> >>>> >>> > >> after masking with XXXXX the PID and next= values that
> >>>> >>> > >> kept changing in
> >>>> >>> > >> every one, i got 26 types of MD5. I also attached the
> >>>> >>> > >> files relevant as an
> >>>> >>> > >> archive to this email.
> >>>> >>> > >>
> >>>> >>> > >> The Gentoo bug i opened is sleeping, it seems nobody
> >>>> >>> > >> has the time to at
> >>>> >>> > >> least test to confirm or not the problems i'm seeing
> >>>> >>> > >> (or everybody's thinking that nobody would restart
> >>>> >>> > >> auditd so often, so the bug it's not that
> >>>> >>> > >> serious).
> >>>> >>> > >>
> >>>> >>> > >>
> >>>> >>> > >> Thank you for your time.
> >>>> >>> > >>
> >>>> >>> > >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram
> >>>> >>> > >> <aval13@gmail.com> wrote:
> >>>> >>> > >>
> >>>> >>> > >>
> >>>> >>> > >> --
> >>>> >>> > >> Linux-audit mailing list
> >>>> >>> > >> Linux-audit@redhat.com
> >>>> >>> > >> https://www.redhat.com/mailman/listinfo/linux-audit
> >>>> >>> > >
> >>>> >>> > >
> >>>> >>> > >
> >>>> >>> > > --
> >>>> >>> > > Peter Moody Google 1.650.253.7306
> >>>> >>> > > Security Engineer pgp:0xC3410038
> >>>> >>> >
> >>>> >>> >
> >>>> >>> >
> >>>> >>>
> >>>> >>>
> >>>> >>
> >>>> >
> >>>> >
> >>>> >
> >>>> > --
> >>>> > Peter Moody Google 1.650.253.7306
> >>>> > Security Engineer pgp:0xC3410038
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Peter Moody Google 1.650.253.7306
> >> Security Engineer pgp:0xC3410038
> >
> >
> >
> > --
> > Peter Moody Google 1.650.253.7306
> > Security Engineer pgp:0xC3410038
>
>
>
^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-04-21 2:14 ` Marcelo Cerri
@ 2012-04-23 16:05 ` Peter Moody
2012-04-23 16:26 ` Eric Paris
1 sibling, 0 replies; 24+ messages in thread
From: Peter Moody @ 2012-04-23 16:05 UTC (permalink / raw)
To: Marcelo Cerri; +Cc: linux-audit
This works for me. Thanks, Marcelo!
Cheers,
peter
On Fri, Apr 20, 2012 at 7:14 PM, Marcelo Cerri
<mhcerri@linux.vnet.ibm.com> wrote:
>
> I took a look at the source code and made some tests. It seems to be a
> problem with the reference count of the fsnotify_mark structure.
>
> This error occurs because the fsnotify_mark_destroy function
> (which runs in a separated kthread) is trying to iterate through a mark
> that is already freed.
>
> Looking at the fsnotify_destroy_mark function (not confuse with
> fsnotify_mark_destroy), which adds a mark to destroy_list to be freed
> later by fsnotify_mark_destroy, I noticed that it does not increment
> the reference count for the reference added to the destroy_list and
> usually the callers dispose the references they held after calling
> fsnotify_destroy_mark.
>
> The patch below increments the reference count of a mark when it is
> added to the destroy list. It seems to solve the issue and it doesn't
> seem to cause any memory leak. Please, can you make some tests in your
> environments and let me know if there is any problem with this patch.
>
> Regarding the synchronize_scru call, I don't think it's causing this
> error. Probably it just make it more frequently because it forces all
> the cpus to schedule, giving the chance to someone else to free the
> mark.
>
> ---
> fs/notify/mark.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/fs/notify/mark.c b/fs/notify/mark.c
> index f104d56..2985fff 100644
> --- a/fs/notify/mark.c
> +++ b/fs/notify/mark.c
> @@ -150,6 +150,7 @@ void fsnotify_destroy_mark(struct fsnotify_mark
> *mark) spin_unlock(&group->mark_lock);
> spin_unlock(&mark->lock);
>
> + fsnotify_get_mark(mark);
> spin_lock(&destroy_lock);
> list_add(&mark->destroy_list, &destroy_list);
> spin_unlock(&destroy_lock);
> --
> 1.7.9.4
>
>
> On Tue, 17 Apr 2012 14:54:29 -0700
> Peter Moody <pmoody@google.com> wrote:
>
>> Last thing. moving synchronize_srcu(&fsnotify_mark_srcu) out of the
>> for(;;) loop in fs/notify/mark.c appears to solve the stability issues
>> for me. I don't know enough about kernel internals to determine if
>> this is doing lots of other bad things to my system or not.
>>
>> Cheers,
>> peter
>>
>> On Tue, Apr 17, 2012 at 11:24 AM, Peter Moody <pmoody@google.com>
>> wrote:
>> > and my config.gz
>> >
>> > On Tue, Apr 17, 2012 at 10:56 AM, Peter Moody <pmoody@google.com>
>> > wrote:
>> >> Here's a trace with debugging turned way up plus a few extra
>> >> printk's added to fs/notify/mark.c. I'm looping through
>> >> private_destroy_list before and after the call to synchronize_srcu.
>> >>
>> >> I can reproduce this reliably with kvm with 2 virtual processors:
>> >> Linux desktop 3.4.0-rc3-oops1+ #1 SMP Tue Apr 17 09:59:44 PDT 2012
>> >> x86_64 GNU/Linux
>> >>
>> >> Cheers,
>> >> peter
>> >>
>> >> On Thu, Apr 5, 2012 at 2:07 PM, Eric Paris <eparis@redhat.com>
>> >> wrote:
>> >>> please please please keep on list. Everything you say might help
>> >>> track it down!
>> >>>
>> >>> On Thu, 2012-04-05 at 14:03 -0700, Peter Moody wrote:
>> >>>> (please let me know if I should take this off-list)
>> >>>>
>> >>>> One other thing (again, maybe already known), but this seems to
>> >>>> be exacerbated by SMP. On my machine, I can't reproduce the
>> >>>> crash if I booth with maxcpus=1.
>> >>>>
>> >>>> Still hunting.
>> >>>>
>> >>>> Cheers,
>> >>>> peter
>> >>>>
>> >>>> On Tue, Apr 3, 2012 at 9:15 AM, Peter Moody <pmoody@google.com>
>> >>>> wrote:
>> >>>> > This may already be known, but the issue seems to be limited
>> >>>> > to watch rules. With any watch rules, I can reliably crash my
>> >>>> > machine while freeing a watch rule after only
>> >>>> > starting/stopping auditd a few times. With no watch rules, I
>> >>>> > have no issues.
>> >>>> >
>> >>>> > Cheers,
>> >>>> > peter
>> >>>> >
>> >>>> > On Wed, Mar 28, 2012 at 11:44 PM, Valentin Avram
>> >>>> > <aval13@gmail.com> wrote:
>> >>>> >> Yes, i know that patch. It made it into kernel 3.2.2. I
>> >>>> >> tested it successfully (oops in 3.2.1, no oops in 3.2.9), but
>> >>>> >> this oops i'm seeing is also in 3.2.9.
>> >>>> >>
>> >>>> >> I monitored changelogs since 3.2.1 to 3.2.12 but there were
>> >>>> >> no fixes either in audit subsystem or in fsnotify. I'll try
>> >>>> >> to reproduce in latest 3.2.13 and repost the oops, but i'm
>> >>>> >> 99% confident it will be the same.
>> >>>> >>
>> >>>> >> Sadly nobody except you seems to pay attention to this
>> >>>> >> problem, probably because it requires special conditions to
>> >>>> >> reproduce (really, who starts and stops auditd every 5
>> >>>> >> seconds on a production server?). We only ran into it because
>> >>>> >> one of our servers would randomly oops and then freeze about
>> >>>> >> each month after stopping and then starting
>> >>>> >>
>> >>>> >> auditd
>> >>>> >>
>> >>>> >> every morning (and the stop-start sequence was needed to
>> >>>> >> workaround a bug somewhere that would hang a
>> >>>> >>
>> >>>> >> gzip
>> >>>> >>
>> >>>> >> running on a file outside a watched folder).
>> >>>> >>
>> >>>> >> Anyway, as a last note, i have a feeling that the oops is not
>> >>>> >> exactly random, there is a pattern, just that i haven't
>> >>>> >> figured it out completely yet.
>> >>>> >>
>> >>>> >> Will keep you
>> >>>> >>
>> >>>> >> uptodate
>> >>>> >>
>> >>>> >> with the things i find out.
>> >>>> >>
>> >>>> >> V.
>> >>>> >>
>> >>>> >> On Mar 29, 2012 4:14 AM, "Eric Paris" <eparis@redhat.com>
>> >>>> >> wrote:
>> >>>> >>>
>> >>>> >>> That patch fixes a BUG() . The report has a NULL ptr deref
>> >>>> >>> and some apparent list correuption.... Sadly they aren't
>> >>>> >>> the same....
>> >>>> >>>
>> >>>> >>> On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote:
>> >>>> >>> > fyi: this patch [1] seems to fix the issue for me. The
>> >>>> >>> > explanation in the subject would reliably oops my machine.
>> >>>> >>> >
>> >>>> >>> > [1]
>> >>>> >>> > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=fed474857efbed79cd390d0aee224231ca718f63
>> >>>> >>> >
>> >>>> >>> > On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody
>> >>>> >>> > <pmoody@google.com> wrote:
>> >>>> >>> > > Are you still able to reliably reproduce this oops? I'm
>> >>>> >>> > > trying to track this down because this bug (or a very
>> >>>> >>> > > similar bug) is causing some significant headaches here
>> >>>> >>> > > at work, but I haven't had a lot of luck. I'm using
>> >>>> >>> > > usermode linux, though, so that might be interfering
>> >>>> >>> > > with things.
>> >>>> >>> > >
>> >>>> >>> > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram
>> >>>> >>> > > <aval13@gmail.com> wrote:
>> >>>> >>> > >> Finally i found some time and spare server to retest
>> >>>> >>> > >> the oops and list_add
>> >>>> >>> > >> corruptions i was getting with the 3.x kernels and
>> >>>> >>> > >> auditd 2.1.3.
>> >>>> >>> > >>
>> >>>> >>> > >> I tested now with gentoo's latest stable
>> >>>> >>> > >> 3.2.1-gentoo-r2 and kernel.org's
>> >>>> >>> > >> 3.2.9.
>> >>>> >>> > >>
>> >>>> >>> > >> Both get the oops/BUG in the same way and after that,
>> >>>> >>> > >> they keep pouring
>> >>>> >>> > >> list_add corruptions with audit_prune_tre(truncated?)
>> >>>> >>> > >> and auditctl as comms.
>> >>>> >>> > >>
>> >>>> >>> > >> Since this is not about Gentoo's kernel only, i'll post
>> >>>> >>> > >> here the oops in
>> >>>> >>> > >> 3.2.9 and also attach some list_add corruptions.
>> >>>> >>> > >>
>> >>>> >>> > >> 3.2.9 BUG:
>> >>>> >>> > >>
>> >>>> >>> > >> kernel: [ 301.240011] BUG: unable to handle kernel
>> >>>> >>> > >> NULL pointer dereference
>> >>>> >>> > >> at (null)
>> >>>> >>> > >> kernel: [ 301.240305] IP: [<c1238dd0>]
>> >>>> >>> > >> __list_del_entry+0x20/0xe0 kernel: [ 301.240481] *pdpt
>> >>>> >>> > >> = 0000000000000000 *pde = f000ddc8f000ddc8
>> >>>> >>> > >> kernel: [ 301.240698] Oops: 0000 [#1] SMP
>> >>>> >>> > >> kernel: [ 301.240910]
>> >>>> >>> > >> kernel: [ 301.241030] Pid: 642, comm: fsnotify_mark
>> >>>> >>> > >> Not tainted 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge
>> >>>> >>> > >> 2950/0CX396 kernel: [ 301.241370] EIP:
>> >>>> >>> > >> 0060:[<c1238dd0>] EFLAGS: 00010287 CPU: 6 kernel:
>> >>>> >>> > >> [ 301.241498] EIP is at __list_del_entry+0x20/0xe0
>> >>>> >>> > >> kernel: [ 301.241623] EAX: f4fae544 EBX: f47cffa4 ECX:
>> >>>> >>> > >> ffffffff EDX: 00000000 kernel: [ 301.241751] ESI:
>> >>>> >>> > >> f4fae544 EDI: f4fae508 EBP: f47cff7c ESP: f47cff64
>> >>>> >>> > >> kernel: [ 301.241879] DS: 007b ES: 007b FS: 00d8 GS:
>> >>>> >>> > >> 0000 SS: 0068 kernel: [ 301.242005] Process
>> >>>> >>> > >> fsnotify_mark (pid: 642, ti=f47ce000 task=f4f47c00
>> >>>> >>> > >> task.ti=f47ce000) kernel: [ 301.242207] Stack:
>> >>>> >>> > >> kernel: [ 301.242327] c10813c0 f47cffa4 f4f47c00
>> >>>> >>> > >> f4e70888 f47cff7c f47cffa4 f47cffb8 c10f6976
>> >>>> >>> > >> kernel: [ 301.242882] ffffffc3 f4f47c00 f4f47c00
>> >>>> >>> > >> 00000000 f4f47c00 c10530c0 f47cff9c f47cff9c
>> >>>> >>> > >> kernel: [ 301.243438] f4fae544 f4fae544 f4c47f58
>> >>>> >>> > >> 00000000 c10f68f0 f47cffe4 c1052834 00000000
>> >>>> >>> > >> kernel: [ 301.243995] Call Trace:
>> >>>> >>> > >> kernel: [ 301.244119] [<c10813c0>] ?
>> >>>> >>> > >> rcu_check_callbacks+0x110/0x110
>> >>>> >>> > >> kernel: [ 301.244248] [<c10f6976>]
>> >>>> >>> > >> fsnotify_mark_destroy+0x86/0x120 kernel: [ 301.244377]
>> >>>> >>> > >> [<c10530c0>] ? abort_exclusive_wait+0x80/0x80 kernel:
>> >>>> >>> > >> [ 301.244504] [<c10f68f0>] ?
>> >>>> >>> > >> fsnotify_put_mark+0x30/0x30 kernel: [ 301.244631]
>> >>>> >>> > >> [<c1052834>] kthread+0x74/0x80 kernel: [ 301.244756]
>> >>>> >>> > >> [<c10527c0>] ? kthread_flush_work_fn+0x10/0x10 kernel:
>> >>>> >>> > >> [ 301.244885] [<c1582ab6>]
>> >>>> >>> > >> kernel_thread_helper+0x6/0xd kernel: [ 301.245011]
>> >>>> >>> > >> Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89 e5 53 83
>> >>>> >>> > >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00
>> >>>> >>> > >> 02 20 00 0f 84 8e 00 00 00 <8b> 1a 39 d8 75 62 8b 59 04
>> >>>> >>> > >> 39 d8 75 35 89 51 04 89 0a 83 c4 14
>> >>>> >>> > >> kernel: [ 301.248195] EIP: [<c1238dd0>]
>> >>>> >>> > >> __list_del_entry+0x20/0xe0 SS:ESP
>> >>>> >>> > >> 0068:f47cff64
>> >>>> >>> > >> kernel: [ 301.248414] CR2: 0000000000000000
>> >>>> >>> > >> kernel: [ 301.248538] ---[ end trace
>> >>>> >>> > >> 15082dbfb353f84c ]---
>> >>>> >>> > >>
>> >>>> >>> > >> The kernel was compiled with the following DEBUG
>> >>>> >>> > >> support (the bolded one
>> >>>> >>> > >> were requested by Gentoo's Dev:
>> >>>> >>> > >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
>> >>>> >>> > >> CONFIG_SLUB_DEBUG=y
>> >>>> >>> > >> CONFIG_HAVE_DMA_API_DEBUG=y
>> >>>> >>> > >> CONFIG_X86_DEBUGCTLMSR=y
>> >>>> >>> > >> CONFIG_PNP_DEBUG_MESSAGES=y
>> >>>> >>> > >> CONFIG_AIC94XX_DEBUG=y
>> >>>> >>> > >> CONFIG_USB_DEBUG=y
>> >>>> >>> > >> CONFIG_DEBUG_KERNEL=y
>> >>>> >>> > >> CONFIG_SCHED_DEBUG=y
>> >>>> >>> > >> CONFIG_DEBUG_RT_MUTEXES=y
>> >>>> >>> > >> CONFIG_DEBUG_PI_LIST=y
>> >>>> >>> > >> CONFIG_DEBUG_BUGVERBOSE=y
>> >>>> >>> > >> CONFIG_DEBUG_INFO=y
>> >>>> >>> > >> CONFIG_DEBUG_MEMORY_INIT=y
>> >>>> >>> > >> CONFIG_DEBUG_LIST=y
>> >>>> >>> > >> CONFIG_DEBUG_STACKOVERFLOW=y
>> >>>> >>> > >> CONFIG_DEBUG_RODATA=y
>> >>>> >>> > >> CONFIG_DEBUG_RODATA_TEST=y
>> >>>> >>> > >>
>> >>>> >>> > >> I attached the kernel config i used for 3.2.9 to
>> >>>> >>> > >> generate this oops and
>> >>>> >>> > >> warnings.
>> >>>> >>> > >>
>> >>>> >>> > >> From the list_add warnings that come after, out of 805
>> >>>> >>> > >> warnings i processed,
>> >>>> >>> > >> after masking with XXXXX the PID and next= values that
>> >>>> >>> > >> kept changing in
>> >>>> >>> > >> every one, i got 26 types of MD5. I also attached the
>> >>>> >>> > >> files relevant as an
>> >>>> >>> > >> archive to this email.
>> >>>> >>> > >>
>> >>>> >>> > >> The Gentoo bug i opened is sleeping, it seems nobody
>> >>>> >>> > >> has the time to at
>> >>>> >>> > >> least test to confirm or not the problems i'm seeing
>> >>>> >>> > >> (or everybody's thinking that nobody would restart
>> >>>> >>> > >> auditd so often, so the bug it's not that
>> >>>> >>> > >> serious).
>> >>>> >>> > >>
>> >>>> >>> > >>
>> >>>> >>> > >> Thank you for your time.
>> >>>> >>> > >>
>> >>>> >>> > >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram
>> >>>> >>> > >> <aval13@gmail.com> wrote:
>> >>>> >>> > >>
>> >>>> >>> > >>
>> >>>> >>> > >> --
>> >>>> >>> > >> Linux-audit mailing list
>> >>>> >>> > >> Linux-audit@redhat.com
>> >>>> >>> > >> https://www.redhat.com/mailman/listinfo/linux-audit
>> >>>> >>> > >
>> >>>> >>> > >
>> >>>> >>> > >
>> >>>> >>> > > --
>> >>>> >>> > > Peter Moody Google 1.650.253.7306
>> >>>> >>> > > Security Engineer pgp:0xC3410038
>> >>>> >>> >
>> >>>> >>> >
>> >>>> >>> >
>> >>>> >>>
>> >>>> >>>
>> >>>> >>
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > --
>> >>>> > Peter Moody Google 1.650.253.7306
>> >>>> > Security Engineer pgp:0xC3410038
>> >>>>
>> >>>>
>> >>>>
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Peter Moody Google 1.650.253.7306
>> >> Security Engineer pgp:0xC3410038
>> >
>> >
>> >
>> > --
>> > Peter Moody Google 1.650.253.7306
>> > Security Engineer pgp:0xC3410038
>>
>>
>>
>
--
Peter Moody Google 1.650.253.7306
Security Engineer pgp:0xC3410038
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-04-21 2:14 ` Marcelo Cerri
2012-04-23 16:05 ` Peter Moody
@ 2012-04-23 16:26 ` Eric Paris
2012-04-24 1:27 ` Peter Moody
2012-04-24 5:12 ` Marcelo Cerri
1 sibling, 2 replies; 24+ messages in thread
From: Eric Paris @ 2012-04-23 16:26 UTC (permalink / raw)
To: Marcelo Cerri; +Cc: linux-audit
On Fri, 2012-04-20 at 23:14 -0300, Marcelo Cerri wrote:
> The patch below increments the reference count of a mark when it is
> added to the destroy list. It seems to solve the issue and it doesn't
> seem to cause any memory leak. Please, can you make some tests in your
> environments and let me know if there is any problem with this patch.
That is almost certainly the wrong thing to do. This test program
should show a memory leak with your patch. If it doesn't show a memory
leak then something is screwed up in inotify as well.
#include <errno.h>
#include <unistd.h>
#include <sys/inotify.h>
int main(void)
{
int fd;
int rc;
struct inotify_event event[10];
fd = inotify_init();
if (fd < 0)
return errno;
while(1) {
rc = inotify_add_watch(fd, "/tmp", IN_CLOSE_WRITE);
if (rc < 0)
return errno;
rc = inotify_rm_watch(fd, rc);
if (rc)
return errno;
rc = read(fd, event, sizeof(event));
if (rc < 0)
return errno;
}
return 0;
}
The lifetime of an object is supposed to be from fsnotify_init_mark()
until it's matching reference is dropped in fsnotify_mark_destroy(). It
sounds to me like we are calling put somewhere in the audit code when we
didn't previously call a get....
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-04-23 16:26 ` Eric Paris
@ 2012-04-24 1:27 ` Peter Moody
2012-04-24 5:12 ` Marcelo Cerri
1 sibling, 0 replies; 24+ messages in thread
From: Peter Moody @ 2012-04-24 1:27 UTC (permalink / raw)
To: Eric Paris; +Cc: linux-audit
On Mon, Apr 23, 2012 at 9:26 AM, Eric Paris <eparis@redhat.com> wrote:
> On Fri, 2012-04-20 at 23:14 -0300, Marcelo Cerri wrote:
>
>> The patch below increments the reference count of a mark when it is
>> added to the destroy list. It seems to solve the issue and it doesn't
>> seem to cause any memory leak. Please, can you make some tests in your
>> environments and let me know if there is any problem with this patch.
>
> That is almost certainly the wrong thing to do. This test program
> should show a memory leak with your patch. If it doesn't show a memory
> leak then something is screwed up in inotify as well.
>
> #include <errno.h>
> #include <unistd.h>
> #include <sys/inotify.h>
>
> int main(void)
> {
> int fd;
> int rc;
> struct inotify_event event[10];
>
> fd = inotify_init();
> if (fd < 0)
> return errno;
>
> while(1) {
> rc = inotify_add_watch(fd, "/tmp", IN_CLOSE_WRITE);
> if (rc < 0)
> return errno;
>
> rc = inotify_rm_watch(fd, rc);
> if (rc)
> return errno;
>
> rc = read(fd, event, sizeof(event));
> if (rc < 0)
> return errno;
> }
>
> return 0;
> }
>
> The lifetime of an object is supposed to be from fsnotify_init_mark()
> until it's matching reference is dropped in fsnotify_mark_destroy(). It
> sounds to me like we are calling put somewhere in the audit code when we
> didn't previously call a get....
>
FWIW, bisecting points me to 75c1be487a690db43da2c1234fcacd84c982803c
75c1be487a690db43da2c1234fcacd84c982803c is the first bad commit
commit 75c1be487a690db43da2c1234fcacd84c982803c
Author: Eric Paris <eparis@redhat.com>
Date: Wed Jul 28 10:18:38 2010 -0400
fsnotify: srcu to protect read side of inode and vfsmount locks
Currently reading the inode->i_fsnotify_marks or
vfsmount->mnt_fsnotify_marks lists are protected by a spinlock on both the
read and the write side. This patch protects the read side of those lists
with a new single srcu.
Signed-off-by: Eric Paris <eparis@redhat.com>
:040000 040000 4b5d9b446eefaca96f8a89b8e9c2ef18da88534e
1abcff76e285ae57f5855b60857ef1708e937a0c M fs
:040000 040000 a02d4ab5b164aa9282a342d73ebe3658f88b4539
3ca9f66ba26cc265d118e6c8558ff2214b9ed192 M include
Cheers,
peter
--
Peter Moody Google 1.650.253.7306
Security Engineer pgp:0xC3410038
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-04-23 16:26 ` Eric Paris
2012-04-24 1:27 ` Peter Moody
@ 2012-04-24 5:12 ` Marcelo Cerri
2012-04-24 18:31 ` Eric Paris
1 sibling, 1 reply; 24+ messages in thread
From: Marcelo Cerri @ 2012-04-24 5:12 UTC (permalink / raw)
To: Eric Paris; +Cc: linux-audit
On Mon, 23 Apr 2012 12:26:16 -0400
Eric Paris <eparis@redhat.com> wrote:
> On Fri, 2012-04-20 at 23:14 -0300, Marcelo Cerri wrote:
>
> > The patch below increments the reference count of a mark when it is
> > added to the destroy list. It seems to solve the issue and it
> > doesn't seem to cause any memory leak. Please, can you make some
> > tests in your environments and let me know if there is any problem
> > with this patch.
>
> That is almost certainly the wrong thing to do. This test program
> should show a memory leak with your patch. If it doesn't show a
> memory leak then something is screwed up in inotify as well.
Sorry, I should have tested the other features that also make use of
fsnotify. You're right, my patch adds a memory leak for inotify (and
probably for dnotify and fanotify too).
> ...
>
> The lifetime of an object is supposed to be from fsnotify_init_mark()
> until it's matching reference is dropped in fsnotify_mark_destroy().
> It sounds to me like we are calling put somewhere in the audit code
> when we didn't previously call a get....
>
Considering that the issue is specific to audit and it seems to occur
only with watches on directories, I investigated the audit_tree.c file
and found a probable cause. The untag_chunk() holds a reference to a
mark at the begging of the function and releases it at the end of it (on
the label out). However when it jumps to the "out" label, it calls
fsnotify_put_mark once more.
Peter and Valentin, can you test this new patch to check if it
solves the oops problem?
Eric, do you agree with this solution?
Regards,
Marcelo
---
kernel/audit_tree.c | 2 --
1 file changed, 2 deletions(-)
diff --git a/kernel/audit_tree.c b/kernel/audit_tree.c
index 5bf0790..b5bd9f9 100644
--- a/kernel/audit_tree.c
+++ b/kernel/audit_tree.c
@@ -250,7 +250,6 @@ static void untag_chunk(struct node *p)
spin_unlock(&hash_lock);
spin_unlock(&entry->lock);
fsnotify_destroy_mark(entry);
- fsnotify_put_mark(entry);
goto out;
}
@@ -293,7 +292,6 @@ static void untag_chunk(struct node *p)
spin_unlock(&hash_lock);
spin_unlock(&entry->lock);
fsnotify_destroy_mark(entry);
- fsnotify_put_mark(entry);
goto out;
Fallback:
--
1.7.9.4
^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-04-24 5:12 ` Marcelo Cerri
@ 2012-04-24 18:31 ` Eric Paris
2012-04-24 18:38 ` Peter Moody
0 siblings, 1 reply; 24+ messages in thread
From: Eric Paris @ 2012-04-24 18:31 UTC (permalink / raw)
To: Marcelo Cerri; +Cc: linux-audit
On Tue, 2012-04-24 at 02:12 -0300, Marcelo Cerri wrote:
> On Mon, 23 Apr 2012 12:26:16 -0400, Eric Paris <eparis@redhat.com> wrote:
> Considering that the issue is specific to audit and it seems to occur
> only with watches on directories, I investigated the audit_tree.c file
> and found a probable cause. The untag_chunk() holds a reference to a
> mark at the begging of the function and releases it at the end of it (on
> the label out). However when it jumps to the "out" label, it calls
> fsnotify_put_mark once more.
>
> Peter and Valentin, can you test this new patch to check if it
> solves the oops problem?
>
> Eric, do you agree with this solution?
>
> Regards,
> Marcelo
>
> ---
> kernel/audit_tree.c | 2 --
> 1 file changed, 2 deletions(-)
>
> diff --git a/kernel/audit_tree.c b/kernel/audit_tree.c
> index 5bf0790..b5bd9f9 100644
> --- a/kernel/audit_tree.c
> +++ b/kernel/audit_tree.c
> @@ -250,7 +250,6 @@ static void untag_chunk(struct node *p)
> spin_unlock(&hash_lock);
> spin_unlock(&entry->lock);
> fsnotify_destroy_mark(entry);
> - fsnotify_put_mark(entry);
> goto out;
> }
>
> @@ -293,7 +292,6 @@ static void untag_chunk(struct node *p)
> spin_unlock(&hash_lock);
> spin_unlock(&entry->lock);
> fsnotify_destroy_mark(entry);
> - fsnotify_put_mark(entry);
> goto out;
>
> Fallback:
This looks right to me. The old audit logic before the switch to
fsnotify was:
- inotify_evict_watch(&chunk->watch);
- mutex_unlock(&chunk->watch.inode->inotify_mutex);
- put_inotify_watch(&chunk->watch);
Which I changed to:
+ spin_unlock(&entry->lock);
+ fsnotify_destroy_mark_by_entry(entry);
+ fsnotify_put_mark(entry);
The difference being that inotify_evict_watch() took a reference on
chunk->watch, however fsnotify_destroy_mark_by_entry() does not. So the
fsnotify_put_mark() was incorrect.
I'd love to hear testing results, and I'm going to try to figure out if
I screwed that up other places....
-Eric
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-04-24 18:31 ` Eric Paris
@ 2012-04-24 18:38 ` Peter Moody
2012-04-24 19:06 ` Eric Paris
0 siblings, 1 reply; 24+ messages in thread
From: Peter Moody @ 2012-04-24 18:38 UTC (permalink / raw)
To: Eric Paris; +Cc: linux-audit
On Tue, Apr 24, 2012 at 11:31 AM, Eric Paris <eparis@redhat.com> wrote:
> On Tue, 2012-04-24 at 02:12 -0300, Marcelo Cerri wrote:
>> On Mon, 23 Apr 2012 12:26:16 -0400, Eric Paris <eparis@redhat.com> wrote:
>
>> Considering that the issue is specific to audit and it seems to occur
>> only with watches on directories, I investigated the audit_tree.c file
>> and found a probable cause. The untag_chunk() holds a reference to a
>> mark at the begging of the function and releases it at the end of it (on
>> the label out). However when it jumps to the "out" label, it calls
>> fsnotify_put_mark once more.
>>
>> Peter and Valentin, can you test this new patch to check if it
>> solves the oops problem?
>>
>> Eric, do you agree with this solution?
>>
>> Regards,
>> Marcelo
>>
>> ---
>> kernel/audit_tree.c | 2 --
>> 1 file changed, 2 deletions(-)
>>
>> diff --git a/kernel/audit_tree.c b/kernel/audit_tree.c
>> index 5bf0790..b5bd9f9 100644
>> --- a/kernel/audit_tree.c
>> +++ b/kernel/audit_tree.c
>> @@ -250,7 +250,6 @@ static void untag_chunk(struct node *p)
>> spin_unlock(&hash_lock);
>> spin_unlock(&entry->lock);
>> fsnotify_destroy_mark(entry);
>> - fsnotify_put_mark(entry);
>> goto out;
>> }
>>
>> @@ -293,7 +292,6 @@ static void untag_chunk(struct node *p)
>> spin_unlock(&hash_lock);
>> spin_unlock(&entry->lock);
>> fsnotify_destroy_mark(entry);
>> - fsnotify_put_mark(entry);
>> goto out;
>>
>> Fallback:
>
> This looks right to me. The old audit logic before the switch to
> fsnotify was:
> - inotify_evict_watch(&chunk->watch);
> - mutex_unlock(&chunk->watch.inode->inotify_mutex);
> - put_inotify_watch(&chunk->watch);
>
> Which I changed to:
> + spin_unlock(&entry->lock);
> + fsnotify_destroy_mark_by_entry(entry);
> + fsnotify_put_mark(entry);
>
> The difference being that inotify_evict_watch() took a reference on
> chunk->watch, however fsnotify_destroy_mark_by_entry() does not. So the
> fsnotify_put_mark() was incorrect.
>
> I'd love to hear testing results, and I'm going to try to figure out if
> I screwed that up other places....
I'm testing this now. It looks good WRT the crash. I need to spend
some more time testing be sure memory isn't leaking anywhere.
> -Eric
>
--
Peter Moody Google 1.650.253.7306
Security Engineer pgp:0xC3410038
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Kernel oops+crash on repeated auditd restarts
2012-04-24 18:38 ` Peter Moody
@ 2012-04-24 19:06 ` Eric Paris
0 siblings, 0 replies; 24+ messages in thread
From: Eric Paris @ 2012-04-24 19:06 UTC (permalink / raw)
To: Peter Moody; +Cc: linux-audit
On Tue, 2012-04-24 at 11:38 -0700, Peter Moody wrote:
> On Tue, Apr 24, 2012 at 11:31 AM, Eric Paris <eparis@redhat.com> wrote:
> > On Tue, 2012-04-24 at 02:12 -0300, Marcelo Cerri wrote:
> >> On Mon, 23 Apr 2012 12:26:16 -0400, Eric Paris <eparis@redhat.com> wrote:
> > I'd love to hear testing results, and I'm going to try to figure out if
> > I screwed that up other places....
>
> I'm testing this now. It looks good WRT the crash. I need to spend
> some more time testing be sure memory isn't leaking anywhere.
I just sent another version which fixed a couple of other places I
believe I was doing ref counting wrong in the audit_tree code.
Hopefully everyone can give that one a whirl....
-Eric
^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2012-04-24 19:06 UTC | newest]
Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-01-25 16:45 Kernel oops+crash on repeated auditd restarts Valentin Avram
2012-01-25 16:53 ` Peter Moody
2012-01-25 19:20 ` Eric Paris
2012-01-26 7:13 ` Valentin Avram
2012-02-08 16:11 ` Valentin Avram
2012-03-05 8:35 ` Valentin Avram
2012-03-28 20:51 ` Peter Moody
2012-03-28 22:42 ` Peter Moody
2012-03-29 1:14 ` Eric Paris
2012-03-29 6:44 ` Valentin Avram
2012-04-03 16:15 ` Peter Moody
2012-04-05 21:03 ` Peter Moody
2012-04-05 21:07 ` Eric Paris
2012-04-17 17:56 ` Peter Moody
2012-04-17 18:24 ` Peter Moody
2012-04-17 21:54 ` Peter Moody
2012-04-21 2:14 ` Marcelo Cerri
2012-04-23 16:05 ` Peter Moody
2012-04-23 16:26 ` Eric Paris
2012-04-24 1:27 ` Peter Moody
2012-04-24 5:12 ` Marcelo Cerri
2012-04-24 18:31 ` Eric Paris
2012-04-24 18:38 ` Peter Moody
2012-04-24 19:06 ` Eric Paris
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.