From mboxrd@z Thu Jan 1 00:00:00 1970 From: Valentin Avram Subject: Kernel oops+crash on repeated auditd restarts Date: Wed, 25 Jan 2012 18:45:03 +0200 Message-ID: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8680552689528749155==" Return-path: Received: from mx1.redhat.com (ext-mx14.extmail.prod.ext.phx2.redhat.com [10.5.110.19]) by int-mx02.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id q0PGj5E9027479 for ; Wed, 25 Jan 2012 11:45:05 -0500 Received: from mail-gy0-f174.google.com (mail-gy0-f174.google.com [209.85.160.174]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q0PGj39V019382 for ; Wed, 25 Jan 2012 11:45:04 -0500 Received: by ghrr11 with SMTP id r11so1733563ghr.33 for ; Wed, 25 Jan 2012 08:45:03 -0800 (PST) List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: linux-audit@redhat.com List-Id: linux-audit@redhat.com --===============8680552689528749155== Content-Type: multipart/alternative; boundary=001636c92b853c40fb04b75cfc6a --001636c92b853c40fb04b75cfc6a Content-Type: text/plain; charset=ISO-8859-1 Hello. Did anybody ever experience kernel oopses and even kernel crashes (after a while), by just restarting repeatedly the auditd daemon? I ask this because i had this problem on Dell R610 servers running Gentoo Linux kernels gentoo-sources-3.0.6 and gentoo-sources-2.6.37-r4 (see this bug: https://bugs.gentoo.org/show_bug.cgi?id=389405 ). The kernels are nothing special, just the vanilla 2.6.37 and 3.0.6 with a few gentoo patches (see https://lkml.org/lkml/2011/11/28/330 ). The auditd version is 2.1.3 (latest). The audit.rules file contains basically the following rules: -D -w /etc -p wa -k etc-directory [snip: same for /sbin, /bin, /usr/sbin, /usr/bin] -a exit,never -F dir=/lib/rc -k skip-lib-rc -w /lib -p wa -k lib-directory -w /usr/lib -p wa -k usr-lib-directory -a exit,never -F arch=b32 -S read [snip: -S for write,open,fstat,mmap etc.] -k excluded-syscalls -b 8192 The bug seems to be somewhere in the fsnotify kernel part, however Gentoo kernel devs and ppl on lkml did not seem too interested, so.. did anybody notice a similar behaviour? Or better yet, is anybody willing to run on one of your servers this simple test: start the minimum server services, use a similar audit.rules configuration, then start auditd and run in a shell the following one-liner: while :; do /etc/init.d/auditd stop ; sleep 5 ; /etc/init.d/auditd start ; sleep 5 ; done This was enough to oops and crash the kernel in less than one hour on the servers where i did the tests. If any similar behavior happens, i'd be very interested to know the the kernel version and distro. Thank you for your time. --001636c92b853c40fb04b75cfc6a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hello.

Did anybody ever experience kernel oopses and eve= n kernel crashes (after a while), by just restarting repeatedly the auditd = daemon?

I ask this because i had this problem on D= ell R610 servers running Gentoo Linux kernels gentoo-sources-3.0.6 and gent= oo-sources-2.6.37-r4 (see this bug:=A0https://bugs.gentoo.org/show_bug.cgi?id=3D389405=A0).

=
The auditd version is 2.1.3 (latest). The audit.rules file conta= ins basically the following rules:

-D
-w= /etc -p wa -k etc-directory
[snip: same for /sbin, /bin, /usr/sb= in, /usr/bin]
-a exit,never -F dir=3D/lib/rc -k skip-lib-rc
-w /lib -p wa = -k lib-directory
-w /usr/lib -p wa -k usr-lib-directory
-a exit,never -F arch=3Db32 -S read [snip: -S for write,open,fstat,mmap et= c.] -k excluded-syscalls
-b 8192

The bug seems to be somewhere in the = fsnotify kernel part, however Gentoo kernel devs and ppl on lkml did not se= em too interested, so.. did anybody notice a similar behaviour? Or better y= et, is anybody willing to run on one of your servers this simple test: star= t the minimum server services, use a similar audit.rules configuration, the= n start auditd and run in a shell the following one-liner:

while :; do /etc/init.d/auditd stop ; sleep 5 ; /etc/in= it.d/auditd start ; sleep 5 ; done

This was enough= to oops and crash the kernel in less than one hour on the servers where i = did the tests. If any similar behavior happens, i'd be very interested = to know the the kernel version and distro.

Thank you for your time.

--001636c92b853c40fb04b75cfc6a-- --===============8680552689528749155== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============8680552689528749155==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter Moody Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Wed, 25 Jan 2012 08:53:01 -0800 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Received: from mx1.redhat.com (ext-mx12.extmail.prod.ext.phx2.redhat.com [10.5.110.17]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id q0PGrWVj002306 for ; Wed, 25 Jan 2012 11:53:32 -0500 Received: from mail-qy0-f174.google.com (mail-qy0-f174.google.com [209.85.216.174]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q0PGrVLw020818 for ; Wed, 25 Jan 2012 11:53:31 -0500 Received: by qcsg15 with SMTP id g15so940455qcs.33 for ; Wed, 25 Jan 2012 08:53:31 -0800 (PST) In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Valentin Avram Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com Just flushing the rules (auditctl -D) would cause my ubuntu machine running a 2.6.38 kernel to oops fairly regularly, maybe one in five times. This was especially painful when testing new rules. On Wed, Jan 25, 2012 at 8:45 AM, Valentin Avram wrote: > Hello. > > Did anybody ever experience kernel oopses and even kernel crashes (after a > while), by just restarting repeatedly the auditd daemon? > > I ask this because i had this problem on Dell R610 servers running Gentoo > Linux kernels gentoo-sources-3.0.6 and gentoo-sources-2.6.37-r4 (see this > bug:=A0https://bugs.gentoo.org/show_bug.cgi?id=3D389405=A0). > > The kernels are nothing special, just the vanilla 2.6.37 and 3.0.6 with a > few gentoo patches (see=A0https://lkml.org/lkml/2011/11/28/330=A0). > > The auditd version is 2.1.3 (latest). The audit.rules file contains > basically the following rules: > > -D > -w /etc -p wa -k etc-directory > [snip: same for /sbin, /bin, /usr/sbin, /usr/bin] > -a exit,never -F dir=3D/lib/rc -k skip-lib-rc > -w /lib -p wa -k lib-directory > -w /usr/lib -p wa -k usr-lib-directory > -a exit,never -F arch=3Db32 -S read [snip: -S for write,open,fstat,mmap e= tc.] > -k excluded-syscalls > -b 8192 > > The bug seems to be somewhere in the fsnotify kernel part, however Gentoo > kernel devs and ppl on lkml did not seem too interested, so.. did anybody > notice a similar behaviour? Or better yet, is anybody willing to run on o= ne > of your servers this simple test: start the minimum server services, use a > similar audit.rules configuration, then start auditd and run in a shell t= he > following one-liner: > > while :; do /etc/init.d/auditd stop ; sleep 5 ; /etc/init.d/auditd start ; > sleep 5 ; done > > This was enough to oops and crash the kernel in less than one hour on the > servers where i did the tests. If any similar behavior happens, i'd be ve= ry > interested to know the the kernel version and distro. > > Thank you for your time. > > > -- > Linux-audit mailing list > Linux-audit@redhat.com > https://www.redhat.com/mailman/listinfo/linux-audit -- = Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 Security Engineer=A0 pgp:0xC3410038 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Paris Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Wed, 25 Jan 2012 14:20:03 -0500 Message-ID: <1327519203.4131.25.camel@localhost> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Valentin Avram Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com On Wed, 2012-01-25 at 18:45 +0200, Valentin Avram wrote: > Did anybody ever experience kernel oopses and even kernel crashes > (after a while), by just restarting repeatedly the auditd daemon? No, but I'll try to remember to take a look. We did have a BUG() that was recently fixed when using -w rules (as I recall). But I've never seen this particular NULL pointer bug. We did recently fix a race in fsnotify mark destruction that could be this, but those symptoms weren't exactly the same. I'm both the upstream Audit and fsnotify maintainer so I'm grumbley at Gentoo for never letting me know isn't working. Where else did you report this? I'm wondering where all the information failure is happening. Can you send me any and all info you have? I'll see if I can reproduce a problem here (but I'm a Fedora guy) From mboxrd@z Thu Jan 1 00:00:00 1970 From: Valentin Avram Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Thu, 26 Jan 2012 09:13:23 +0200 Message-ID: References: <1327519203.4131.25.camel@localhost> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4982181313618091715==" Return-path: In-Reply-To: <1327519203.4131.25.camel@localhost> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Eric Paris Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com --===============4982181313618091715== Content-Type: multipart/alternative; boundary=20cf30363d03a2425304b7691d52 --20cf30363d03a2425304b7691d52 Content-Type: text/plain; charset=ISO-8859-1 Please read below. On Wed, Jan 25, 2012 at 9:20 PM, Eric Paris wrote: > On Wed, 2012-01-25 at 18:45 +0200, Valentin Avram wrote: > > > Did anybody ever experience kernel oopses and even kernel crashes > > (after a while), by just restarting repeatedly the auditd daemon? > > No, but I'll try to remember to take a look. We did have a BUG() that > was recently fixed when using -w rules (as I recall). But I've never > seen this particular NULL pointer bug. We did recently fix a race in > fsnotify mark destruction that could be this, but those symptoms weren't > exactly the same. > > I'm both the upstream Audit and fsnotify maintainer so I'm grumbley at > Gentoo for never letting me know isn't working. Where else did you > report this? I'm wondering where all the information failure is > happening. > I only reported the issue on Gentoo bugs and LKML (the two links i included in the original email). The Gentoo guys at first did seem interested in the bug and asked for a test with a kernel compiled with CONFIG_DEBUG_INFO and CONFIG_DEBUG_LIST. After that test it looked like some list is getting messed up somewhere (altough i'm part C programmer, my kernel insides knowledge is limited). The LKML guys didn't even bother to answer. > Can you send me any and all info you have? > > All the information i had is posted on the Gentoo bug report. The two machines i used to test the issue are now in production mode, so i can't do any testing on them. However I'll soon have access to a new machine that can stay in test mode for a while, where i plan to retest with Gentoo's latest "stable-marked" kernel gentoo-sources-3.1.6. > I'll see if I can reproduce a problem here (but I'm a Fedora guy) > At this moment i'm not extremely sure if it's a auditd issue or a kernel issue or both. However, if you're running a kernel lower than 3.0.7 and auditd 2.1.3, I'd be very interested if running the one-liner i posted (audit start and stop on a loop with 5 seconds delay) will eventually (in 1 hour or something close) crash the kernel completely (or at least oops a lot of times). Thank you. --20cf30363d03a2425304b7691d52 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Please read below.

On Wed, Jan 25, 2012 a= t 9:20 PM, Eric Paris <eparis@redhat.com> wrote:
On Wed, 2012-01-25 at 18:45 +0200, Valentin Avram wrote:<= br>
> Did anybody ever experience kernel oopses and even kernel crashes
> (after a while), by just restarting repeatedly the auditd daemon?

No, but I'll try to remember to take a look. =A0We did have a BUG= () that
was recently fixed when using -w rules (as I recall). =A0 But I've neve= r
seen this particular NULL pointer bug. =A0We did recently fix a race in
fsnotify mark destruction that could be this, but those symptoms weren'= t
exactly the same.

I'm both the upstream Audit and fsnotify maintainer so I'm grumbley= at
Gentoo for never letting me know isn't working. =A0Where else did you report this? =A0I'm wondering where all the information failure is
happening.

I only reported the issue on= Gentoo bugs and LKML (the two links i included in the original email). The= Gentoo guys at first did seem interested in the bug and asked for a test w= ith a kernel compiled with CONFIG_DEBUG_INFO and CONFIG_DEBUG_LIST. After t= hat test it looked like some list is getting messed up somewhere (altough i= 'm part C programmer, my kernel insides knowledge is limited). The LKML= guys didn't even bother to answer.
=A0
Can you send me any and all info you have?


All the information i had is posted on= the Gentoo bug report. The two machines i used to test the issue are now i= n production mode, so i can't do any testing on them. However I'll = soon have access to a new machine that can stay in test mode for a while, w= here i plan to retest with Gentoo's latest "stable-marked" ke= rnel gentoo-sources-3.1.6.

=A0
I'll see if I can reproduce a problem here (but I'm a Fedora guy)

At this moment i'm not extremely sur= e if it's a auditd issue or a kernel issue or both. However, if you'= ;re running a kernel lower than 3.0.7 and auditd 2.1.3, I'd be very int= erested if running the one-liner i posted (audit start and stop on a loop w= ith 5 seconds delay) will eventually (in 1 hour or something close) crash t= he kernel completely (or at least oops a lot of times).=A0

Thank you.

--20cf30363d03a2425304b7691d52-- --===============4982181313618091715== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============4982181313618091715==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Valentin Avram Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Wed, 8 Feb 2012 18:11:03 +0200 Message-ID: References: <1327519203.4131.25.camel@localhost> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7474275148628762262==" Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Eric Paris Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com --===============7474275148628762262== Content-Type: multipart/alternative; boundary=001636c92c9a70425104b8762458 --001636c92c9a70425104b8762458 Content-Type: text/plain; charset=ISO-8859-1 Hello. Fresh news: Gentoo's gentoo-sources-3.1.10-r1 with audit-2.1.3 still gives oops using the simple "start ; sleep 5 ; stop ; sleep 5 ; repeat" one-liner. Kernel oops after less than 5 minutes: BUG: unable to handle kernel NULL pointer dereference at 00000004 IP: [] fsnotify_mark_destroy+0x87/0x130 *pdpt = 0000000000000000 *pde = f000def8f000def8 Oops: 0002 [#1] SMP Pid: 690, comm: fsnotify_mark Not tainted 3.1.10-gentoo-r1-drbd-version3 #1 Dell Inc. PowerEdge R610/0F0XJ6 EIP: 0060:[] EFLAGS: 00010216 CPU: 3 EIP is at fsnotify_mark_destroy+0x87/0x130 EAX: f2e51708 EBX: f2415fa8 ECX: 00000000 EDX: f2e51744 ESI: f2f46c00 EDI: ffffffc4 EBP: c10ea000 ESP: f2415f90 DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 Process fsnotify_mark (pid: 690, ti=f2414000 task=f2f46c00 task.ti=f2414000) Stack: f2f46c00 00000000 f2f46c00 c1050150 f2415fa0 f2415fa0 f2e51744 f2e51744 f2c47f68 00000000 c10f22b0 00000000 c104f854 00000000 00000000 00000000 00000000 f2415fd4 f2415fd4 00000000 c104f7e0 f2c47f68 c15820b6 00000000 Call Trace: [] ? abort_exclusive_wait+0x90/0x90 [] ? fsnotify_put_mark+0x20/0x20 [] ? kthread+0x74/0x80 [] ? kthread_flush_work_fn+0x10/0x10 [] ? kernel_thread_helper+0x6/0xd Code: 34 1b 8b c1 e8 4b 2d f6 ff 8b 54 24 18 8d 42 c4 39 da 8b 48 3c 8d 79 c4 75 0e eb 2d 90 8d b4 26 00 00 00 00 89 f8 89 ef 8b 68 40 69 04 89 4d 00 89 50 3c 89 50 40 e8 48 ff ff ff 8b 4f 3c 8d EIP: [] fsnotify_mark_destroy+0x87/0x130 SS:ESP 0068:f2415f90 CR2: 0000000000000004 ---[ end trace d10081cf0e5b936c ]--- So far only one oops occured, however the test server is doing quite nothing right now. I'll install more services, retry and post back here the results. On Thu, Jan 26, 2012 at 9:13 AM, Valentin Avram wrote: > > All the information i had is posted on the Gentoo bug report. The two > machines i used to test the issue are now in production mode, so i can't do > any testing on them. However I'll soon have access to a new machine that > can stay in test mode for a while, where i plan to retest with Gentoo's > latest "stable-marked" kernel gentoo-sources-3.1.6. > > --001636c92c9a70425104b8762458 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hello.

Fresh news: Gentoo's gentoo-sources-3.1.10-r1 with audit-= 2.1.3 still gives oops using the simple "start ; sleep 5 ; stop ; slee= p 5 ; repeat" one-liner.

Kernel oops after less than 5 minutes:=

BUG: unable to handle kernel NULL pointer dereference at 00000004
IP= : [<c10f2337>] fsnotify_mark_destroy+0x87/0x130
*pdpt =3D 00000000= 00000000 *pde =3D f000def8f000def8
Oops: 0002 [#1] SMP

Pid: 690= , comm: fsnotify_mark Not tainted 3.1.10-gentoo-r1-drbd-version3 #1 Dell In= c. PowerEdge R610/0F0XJ6
EIP: 0060:[<c10f2337>] EFLAGS: 00010216 CPU: 3
EIP is at fsnotify_= mark_destroy+0x87/0x130
EAX: f2e51708 EBX: f2415fa8 ECX: 00000000 EDX: f= 2e51744
ESI: f2f46c00 EDI: ffffffc4 EBP: c10ea000 ESP: f2415f90
=A0DS= : 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process fsnotify_mark (pid: 690, ti=3Df2414000 task=3Df2f46c00 task.ti=3Df2= 414000)
Stack:
=A0f2f46c00 00000000 f2f46c00 c1050150 f2415fa0 f2415f= a0 f2e51744 f2e51744
=A0f2c47f68 00000000 c10f22b0 00000000 c104f854 000= 00000 00000000 00000000
=A000000000 f2415fd4 f2415fd4 00000000 c104f7e0 f2c47f68 c15820b6 00000000<= br>Call Trace:
=A0[<c1050150>] ? abort_exclusive_wait+0x90/0x90=A0[<c10f22b0>] ? fsnotify_put_mark+0x20/0x20
=A0[<c104f854>= ;] ? kthread+0x74/0x80
=A0[<c104f7e0>] ? kthread_flush_work_fn+0x10/0x10
=A0[<c15820b6= >] ? kernel_thread_helper+0x6/0xd
Code: 34 1b 8b c1 e8 4b 2d f6 ff 8b= 54 24 18 8d 42 c4 39 da 8b 48 3c 8d 79 c4 75 0e eb 2d 90 8d b4 26 00 00 00= 00 89 f8 89 ef 8b 68 40
=A069 04 89 4d 00 89 50 3c 89 50 40 e8 48 ff ff ff 8b 4f 3c 8d
EIP: [&l= t;c10f2337>] fsnotify_mark_destroy+0x87/0x130 SS:ESP 0068:f2415f90
CR= 2: 0000000000000004
---[ end trace d10081cf0e5b936c ]---

So far o= nly one oops occured, however the test server is doing quite nothing right = now. I'll install more services, retry and post back here the results.<= br>
On Thu, Jan 26, 2012 at 9:13 AM, Valentin Av= ram <aval13@gmail.= com> wrote:

All = the information i had is posted on the Gentoo bug report. The two machines = i used to test the issue are now in production mode, so i can't do any = testing on them. However I'll soon have access to a new machine that ca= n stay in test mode for a while, where i plan to retest with Gentoo's l= atest "stable-marked" kernel gentoo-sources-3.1.6.


--001636c92c9a70425104b8762458-- --===============7474275148628762262== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============7474275148628762262==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Valentin Avram Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Mon, 5 Mar 2012 10:35:20 +0200 Message-ID: References: <1327519203.4131.25.camel@localhost> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary=20cf30563a018a72fa04ba7aceec Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Eric Paris Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com --20cf30563a018a72fa04ba7aceec Content-Type: multipart/alternative; boundary=20cf30563a018a72ee04ba7aceea --20cf30563a018a72ee04ba7aceea Content-Type: text/plain; charset=ISO-8859-1 Finally i found some time and spare server to retest the oops and list_add corruptions i was getting with the 3.x kernels and auditd 2.1.3. I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and kernel.org's 3.2.9. Both get the oops/BUG in the same way and after that, they keep pouring list_add corruptions with audit_prune_tre(truncated?) and auditctl as comms. Since this is not about Gentoo's kernel only, i'll post here the oops in 3.2.9 and also attach some list_add corruptions. 3.2.9 BUG: kernel: [ 301.240011] BUG: unable to handle kernel NULL pointer dereference at (null) kernel: [ 301.240305] IP: [] __list_del_entry+0x20/0xe0 kernel: [ 301.240481] *pdpt = 0000000000000000 *pde = f000ddc8f000ddc8 kernel: [ 301.240698] Oops: 0000 [#1] SMP kernel: [ 301.240910] kernel: [ 301.241030] Pid: 642, comm: fsnotify_mark Not tainted 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396 kernel: [ 301.241370] EIP: 0060:[] EFLAGS: 00010287 CPU: 6 kernel: [ 301.241498] EIP is at __list_del_entry+0x20/0xe0 kernel: [ 301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff EDX: 00000000 kernel: [ 301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c ESP: f47cff64 kernel: [ 301.241879] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 kernel: [ 301.242005] Process fsnotify_mark (pid: 642, ti=f47ce000 task=f4f47c00 task.ti=f47ce000) kernel: [ 301.242207] Stack: kernel: [ 301.242327] c10813c0 f47cffa4 f4f47c00 f4e70888 f47cff7c f47cffa4 f47cffb8 c10f6976 kernel: [ 301.242882] ffffffc3 f4f47c00 f4f47c00 00000000 f4f47c00 c10530c0 f47cff9c f47cff9c kernel: [ 301.243438] f4fae544 f4fae544 f4c47f58 00000000 c10f68f0 f47cffe4 c1052834 00000000 kernel: [ 301.243995] Call Trace: kernel: [ 301.244119] [] ? rcu_check_callbacks+0x110/0x110 kernel: [ 301.244248] [] fsnotify_mark_destroy+0x86/0x120 kernel: [ 301.244377] [] ? abort_exclusive_wait+0x80/0x80 kernel: [ 301.244504] [] ? fsnotify_put_mark+0x30/0x30 kernel: [ 301.244631] [] kthread+0x74/0x80 kernel: [ 301.244756] [] ? kthread_flush_work_fn+0x10/0x10 kernel: [ 301.244885] [] kernel_thread_helper+0x6/0xd kernel: [ 301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89 e5 53 83 ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 0f 84 8e 00 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 83 c4 14 kernel: [ 301.248195] EIP: [] __list_del_entry+0x20/0xe0 SS:ESP 0068:f47cff64 kernel: [ 301.248414] CR2: 0000000000000000 kernel: [ 301.248538] ---[ end trace 15082dbfb353f84c ]--- The kernel was compiled with the following DEBUG support (the bolded one were requested by Gentoo's Dev: CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y CONFIG_SLUB_DEBUG=y CONFIG_HAVE_DMA_API_DEBUG=y CONFIG_X86_DEBUGCTLMSR=y CONFIG_PNP_DEBUG_MESSAGES=y CONFIG_AIC94XX_DEBUG=y CONFIG_USB_DEBUG=y CONFIG_DEBUG_KERNEL=y CONFIG_SCHED_DEBUG=y CONFIG_DEBUG_RT_MUTEXES=y CONFIG_DEBUG_PI_LIST=y CONFIG_DEBUG_BUGVERBOSE=y *CONFIG_DEBUG_INFO=y* CONFIG_DEBUG_MEMORY_INIT=y *CONFIG_DEBUG_LIST=y* CONFIG_DEBUG_STACKOVERFLOW=y CONFIG_DEBUG_RODATA=y CONFIG_DEBUG_RODATA_TEST=y I attached the kernel config i used for 3.2.9 to generate this oops and warnings. >>From the list_add warnings that come after, out of 805 warnings i processed, after masking with XXXXX the PID and next= values that kept changing in every one, i got 26 types of MD5. I also attached the files relevant as an archive to this email. The Gentoo bug i opened is sleeping, it seems nobody has the time to at least test to confirm or not the problems i'm seeing (or everybody's thinking that nobody would restart auditd so often, so the bug it's not that serious). Thank you for your time. On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram wrote: --20cf30563a018a72ee04ba7aceea Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Finally i found some time and spare server to retest the oops and list_add = corruptions i was getting with the 3.x kernels and auditd 2.1.3.

I t= ested now with gentoo's latest stable 3.2.1-gentoo-r2 and kernel.org's 3.2.9.

Both get the oops/BUG in the same way and after that, they keep pouring= list_add corruptions with audit_prune_tre(truncated?) and auditctl as comm= s.

Since this is not about Gentoo's kernel only, i'll post h= ere the oops in 3.2.9 and also attach some list_add corruptions.

3.2.9 BUG:
ker=
nel: [  301.240011] BUG: unable to handle kernel NULL pointer dereference a=
t   (null)
kernel: [  301.240305] IP: [<c1238dd0>] __list_del_entry+0x20/0xe0
kernel: [  301.240481] *pdpt =3D 0000000000000000 *pde =3D f000ddc8f000ddc8=
=20
kernel: [  301.240698] Oops: 0000 [#1] SMP=20
kernel: [  301.240910]=20
kernel: [  301.241030] Pid: 642, comm: fsnotify_mark Not tainted 3.2.9-drbd=
-version3 #1 Dell Inc. PowerEdge 2950/0CX396
kernel: [  301.241370] EIP: 0060:[<c1238dd0>] EFLAGS: 00010287 CPU: 6
kernel: [  301.241498] EIP is at __list_del_entry+0x20/0xe0
kernel: [  301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff EDX: 00000=
000=20
kernel: [  301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c ESP: f47cf=
f64=20
kernel: [  301.241879]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
kernel: [  301.242005] Process fsnotify_mark (pid: 642, ti=3Df47ce000 task=
=3Df4f47c00 task.ti=3Df47ce000)
kernel: [  301.242207] Stack:
kernel: [  301.242327]  c10813c0 f47cffa4 f4f47c00 f4e70888 f47cff7c f47cff=
a4 f47cffb8 c10f6976
kernel: [  301.242882]  ffffffc3 f4f47c00 f4f47c00 00000000 f4f47c00 c10530=
c0 f47cff9c f47cff9c
kernel: [  301.243438]  f4fae544 f4fae544 f4c47f58 00000000 c10f68f0 f47cff=
e4 c1052834 00000000
kernel: [  301.243995] Call Trace:
kernel: [  301.244119]  [<c10813c0>] ? rcu_check_callbacks+0x110/0x11=
0
kernel: [  301.244248]  [<c10f6976>] fsnotify_mark_destroy+0x86/0x120
kernel: [  301.244377]  [<c10530c0>] ? abort_exclusive_wait+0x80/0x80
kernel: [  301.244504]  [<c10f68f0>] ? fsnotify_put_mark+0x30/0x30
kernel: [  301.244631]  [<c1052834>] kthread+0x74/0x80
kernel: [  301.244756]  [<c10527c0>] ? kthread_flush_work_fn+0x10/0x1=
0
kernel: [  301.244885]  [<c1582ab6>] kernel_thread_helper+0x6/0xd
kernel: [  301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89 e5 53 8=
3 ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 0f 84 8e 0=
0 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 83 c4=
 14
kernel: [  301.248195] EIP: [<c1238dd0>] __list_del_entry+0x20/0xe0 S=
S:ESP 0068:f47cff64
kernel: [  301.248414] CR2: 0000000000000000
kernel: [  301.248538] ---[ end trace 15082dbfb353f84c ]---
The kernel= was compiled with the following DEBUG support (the bolded one were request= ed by Gentoo's Dev:
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=3Dy
CONF= IG_SLUB_DEBUG=3Dy
CONFIG_HAVE_DMA_API_DEBUG=3Dy
CONFIG_X86_DEBUGCTLMSR=3Dy
CONFIG_PNP_D= EBUG_MESSAGES=3Dy
CONFIG_AIC94XX_DEBUG=3Dy
CONFIG_USB_DEBUG=3Dy
CO= NFIG_DEBUG_KERNEL=3Dy
CONFIG_SCHED_DEBUG=3Dy
CONFIG_DEBUG_RT_MUTEXES= =3Dy
CONFIG_DEBUG_PI_LIST=3Dy
CONFIG_DEBUG_BUGVERBOSE=3Dy
CONFIG_DEBUG_INFO=3Dy
CONFIG_DEBUG= _MEMORY_INIT=3Dy
CONFIG_DEBUG_LIST=3Dy
CONFIG_DEBUG_STACKOVERF= LOW=3Dy
CONFIG_DEBUG_RODATA=3Dy
CONFIG_DEBUG_RODATA_TEST=3Dy

I= attached the kernel config i used for 3.2.9 to generate this oops and warn= ings.

From the list_add warnings that come after, out of 805 warnings i proce= ssed, after masking with XXXXX the PID and next=3D values that kept changin= g in every one, i got 26 types of MD5. I also attached the files relevant a= s an archive to this email.

The Gentoo bug i opened is sleeping, it seems nobody has the time to at= least test to confirm or not the problems i'm seeing (or everybody'= ;s thinking that nobody would restart auditd so often, so the bug it's = not that serious).

Thank you for your time.

On Wed, Feb = 8, 2012 at 6:11 PM, Valentin Avram <aval13@gmail.com> wrote:

--20cf30563a018a72ee04ba7aceea-- --20cf30563a018a72fa04ba7aceec Content-Type: application/x-gzip; name="parse_oops.tgz" Content-Disposition: attachment; filename="parse_oops.tgz" Content-Transfer-Encoding: base64 X-Attachment-Id: f_gzf8t7kq0 H4sIALFyVE8AA+1dW4+bRhjd5/0VSH3JqskGhvuqTVW1VduXKKoipVJVoWEYbGQMFpfs7r/vN2Cc XbxNdi2DZ5NzItnx+pi54XN8mAE2vKplFDDmXOblIjLPJoBJ8BxHPVu+a3avrf51/1/PPrMsn7mm b1FFzkzLpr+eGZNUZoy2bnhlGGerIlssm8/xZFXPUaF58eHnv97++fb3K4M3Rp7Fr/OsbqJExu3i Ulwxz4ii7i88Sb43b1z3tXnDzRcX53/wKrnmlTQKvpZXxrvyWla/JQtpsNA1z4ePGKKsqnbTZGVx aRTypnn1ZlPJj0a9LNs8MWJpdC9fCMsXNo35xUsjbhvjmteGYbwo2jy/uKRn+uCPfytcXJ6/y5Ir o3vxkra+XlPF2yRrok3VFjJqqEbveVY0kli/Gx1+7Z/sS3YZvkqqOHn1kUaSqmQb31nnv/A8N95X XMir839+EJZp8zQO3vxLlaiKqM7L6w1vlpEqqiyoC7yg64KOy+wg8Vzi/vRgNz2aY9qxacu9MtN1 Q0QmiWiPNvb/m0o9zmNipHVRNll6S2NZN1V5G615tSJyGBPZYlu2z13pEbstGr6IxLItFMcSnirT 7UluYAaCbVuwbht5E+WlWO3qSR+w2f2N+sLs+P2glIVUtTRfq4cdJVTtuEtg7n1CGu4INKz0sKwk V+21QyL6W6LLAtsh4mr3tu/Q28Hu7aEuW0KU5m29jK7LahWlakAtVS9raCrjseqPlawKmW+LjJYy 38hKDT5Rk/NTf2WPis19/bemKOMg/WfQ/znwLej/B+g/9B/6/yDG+m96xy/jAP23iAb9nwG66n/q cs5CZkL/of/Q/+kw1n97AtH9gv4z2zP39N/B7/9ZoK/+Oyxx+GP1XzQ5hP9Jwi+6xirZX5ck/0oJ U9VEW24pibRCJcFZIyveyJ5WD/r7Saj5VmB7H+6kOqWHrFgMRVtBp7GDJvuxY6my+w9QU/oPVW2u bIBEmMjMHMiuHaQ7ciWFzD5SAVne9IrspsQOk57tcOmFflebQjZ5VqwiLlQNEv9eBWzG2Hibqsd9 5WN86ADbCXxzO4DXfCWjdkMk556PsTS+2/qkpB2mSDPlpbxYqI12bQ+G+iWhpeo31K4tMsHrbrRt VUX2iZha1h1iLYtkXS9U/3BLEcWWGNA/5Vh1Z8k7Vufg1q4TA99Kti3pK5oVZdL1tpmorXkDMTSD 5E571FZpgFRPO4oYDqW6Qah6sL6tu0Kbkhix6j057LA2Z7Lbkig3t1FTRkpAh93a2ZYnAkeKnrWJ CnkdkRg3nqNGTBAtHYrzLN8biqMqyUbQt7frDNVOFu/s23VEz5MkBVU3HlumqhxjX5d5HwFj/3fc 45dxiP8j/80DXf3/icf/4P/wf/g//B/+/0SM/d+bYAbwS/7vUNjfy/8+/H8OwP/h//D/Q/0/ThPB t5TVmvaoUlBjqMzOdroaWiZXZTv4xYBfDBpi7P9+cPwyDpn/9bD+cxZ8Jf6P+d8p539Dy/vs/G/g Yf73mWKk/8zyj18Gjv/qC131X63/MS3M/yL/6Zz/kOaQ5p43xv7vhMcv46D8B/+fBRr7v+fTD0Pk Pw3yH9b/fjP5zz3B+l/HfmD+z4X+zwFd9R/zf8h/+ue/WDphP/+3Wss1ZQ2xlFE3DUhMrib+YiRF JEWNsef/86//efj8T8z/zQJd/b87/stN5D/kP+S/CTHW/0CX8z+Q/2aB1vpPv9iR/5D/9M1/SHVI dc8bY/8PneOXgfk/faGx/3uBeLT/I/8h/yH/HYA9/dcl/3nQ/zmgsf6HkiP/If8h/yH/If9NhZH/ 26Z9/DIOyX84/38eaOz/3PQl8h/yH/LfhBjr/xQ3gPjy+k9/7/4PHvR/Fuiq/1j/ifyH/Dd3/mOS e0Gfx5aSNkhfT7neNF3zu9Z3u4glEBe/Foz9f4L4d8jxX+LD/+eArv7f5b/k0ef/wf/h//B/HP+F oT8Re/5/guu/Pnj/F6z/nQUa+z8Z2qPP/4D/w//h//B/+P8TMfZ/Z/7rP+D+nyeEtv7vhyKUuP8n 5n8x/zslRvrveLpc/w3Hf2eBtvpP+Y8iCPQf+g/9nxAj/Xc9Xe7/AP2fBdrqvx/G3HGg/9B/6P+E GOm/Z57k+p/O/vwPzv+bBbrqP9Z/Yv7nxPM/bhDYve/VYimTSHS2Rw9cTSjIzvMwV4S5omeNsf/b 3vHLwPVf9IWu/q/yn2DIf8h/yH9TYqz//vzzPw+u/8Pxv3mgrf4/7frPyH/If1j/h0yHTPdEjPzf P8H1X3D/3xNCY/8PJVkL/B/+D/+H/8P/p8HY/7VZ/4nr/8wCjf2fmyHWf+L4L47/AgAAAAAAAAAA AAAAAMCh+A8fQCAcAMgAAA== --20cf30563a018a72fa04ba7aceec Content-Type: application/x-gzip; name="kernel_config.gz" Content-Disposition: attachment; filename="kernel_config.gz" Content-Transfer-Encoding: base64 X-Attachment-Id: f_gzf965ch1 H4sICJh5VE8AA2tlcm5lbF9jb25maWcAlDxdc9s6ru/7Kzw592H3oc1nsz33Th5oirJ5LIk6JGU7 fdGkidpmNo27sXO2/fcXJPVByqDj7UynNQB+gQAIgKB++9tvE/K623y/2z3e3z09/Zp8bZ6bl7td 8zD58vjU/N8kEZNC6AlLuH4PxNnj8+vP08fLj9eTy/cX73+fLJqX5+ZpQjfPXx6/vkLLx83z334D SiqKlM/q66sp15PH7eR5s5tsm93fWvj643V9eXHza6A0kOsrnBTo2h+8UFpWVHNR1AmjImFyQIpK l5WuUyFzom9OmqcvlxfvzGRPOgoi6Rzape7nzcndy/23058fr0/v7QK2dmn1Q/PF/e7bzVjBJKc1 zYWqqzIhmg3D0kzQhRKVpKxeEU3niZgN2L6poWJLVmh1EFlPpSAJJUoPZAabsLJWVVkK6SGUJnSh JYGR93BzsmR1BjMt6K0WSOM8r4Yfn0TB6iQn/pYUjCUGVuekNENphmyPJVIzS5exYqbn+8vjirR9 jxECut4HTyuEgfMV47O5N3+7mTm5dSstaZ0m1J++XCmW981VyQvDR2QJjnBN5zOSJDXJZkJyPc9H I82JqmlZ1TzJzD5zjWwjyfhUAp9AxjJy60+mI9E8Z/VS3SqgzZC5BENJ6GWNTAMkmFSZtlPBZkno HLaew44q/omNREIxXZV1yaQdgkjm7YvdzA7F8in8SrlUuqbzqlhE6EoyYziZmxGfMlkQq7KlUIpP 9yatKlWyIvHRPeM6wby8iHGrFGVl5FzVBdiD2klU355UYLssYax9qxiqFiXsDjAsATMD3OPFbG+e LWXCQETtwmEXBR3InF2rVV6GMF9uW5qMfLqtZ2rMLSflNU0zAsiTd1+MYX63vfureXjX3P+chICH n72NWljRMnJFPJvYGzuQBzApJ6dPj59Pv28eXp+a7en/VAWBNiBmjCh2+n5k9ay4cPlnvRLS21UP Yk39zJ4ZT4avrz8A0pKxNQgITKnQJPMNOOwGK5awWjOlHNTo8qK3pBIEoKYiLzkIwUk/EWAxyZZM KpAhYMnDy+eHd381L1s4bC5RIth1LUZyvwApZFk9+8RLHDMFzAWOyj751svHrD/FWnjjh0P3oumP 6wvnmMCMfgi//nS4tUAkvzMhc6G0EYKbk78/b56bf/TsVCvfMoPBWvKS7gHMv1R7+wsqzNd1/mfF KoZD95pMFWi+FJTB3hNKtc+iMa5eXqJrTeekAGOILFQTtTBnl6doBuQsdDeej1gjMC4MpagKHRoF Y1tGP1vTPoI6x2AM1JKNrXN/2hGZgKKpTsng/5Pt6+ftr+2u+T4oGUIe2PqSSGV1GDl7QY2BtVOG o9RcrPYxxkwDC8IeDTk4XRQMp57DeZIETArmYNciaTVR+2sx3KgB5+9/CbC81AaM7jvAa+v/YMc6 4FJSgFdoTEzQxIHBRBGqo/2m4IPBkVLPcfXqphsdny9aN/PXGGJF2nfvQNCrFFjOU31z/jHYwgo8 XgJnIlj2OTDYeoqe7zmToiqVzzIHckcUMqsWncLsP1nneb/hkqPrgbMefAdPxMwq6pInLWY8KwMO dUmy1k22qgSmGhk+B1cM6GowERLle0sHh0saX55l1jBwSrisQ8zQXwo+N1iPFU/0HBcG7bdFSabZ oh0ac5HBvoIWgBEbZlSBF1F4v40t9X8DQ2UAMHz2fxdMu9/9HJyAmJNvbyIDza1KjfsCekXBZcIX Y9zO2+gyQT7sKS7xxpT2fpQxCtYDRXgyPgRAIQvoFhy4YE3WicACyIon59fB6m2HddyzXgBY3eZB 9x2sxpvMhS4zPxYpJS+0p35BoMKyFARb+sceVeVC1iV4pyYoHRnnKHYKzlidVv4hklaaeZEAK4WP VXxWkCz1BN74gtIH2ODSB6g5aJrHfy4CvhvfPwnlI5g79F730ay16m0qoGxevmxevt893zcT9lfz vNtOyPPDhG5en3fgsg3mPuxisPgG3NkHVMJa59vQgZbUy9z64MhMl7nr3zc3/c7w+s+Ky4UHU1k1 dZ0H5gG8UaIhKl/gCpWRKSbe0Fd4jomUQ0yBWWTLUOEogtDnjyovIUyYMlQ2TSuWppxys0Tw5DOQ AmMAqPGWRrLGQS5NFA+jjFMEC+sEjBssJNMoApQbhdvoxR6FcyEWI6RJDhCtZWjghtgc1jlnWRls kW0o2QyUE0JDm+lo11aTkmMDlLzfPh83X8HuMbIoBSjvCJfzNbBsQCs74ojI2mLgRyUL8Ac0T7mf dRrrQ13k49lZ3gy7uReyAwk0qhVJ4WjMS5OJ6IMrKpbvPt9tm4fJv5yC/XjZfHl8enz+OqhS4Dka VlAxZ5L5qzVyyovUC0rgVMuNVfF30loeZRTq5myQw1wkVcYUJuYmsZQTOueFZ/amoY+STROS+lh3 hkxVoGceOOOYSg1nj2YzyfVt7xqLibr/1piQ1rcvXLgDsRDCC2Q6aAJuahbMusPQ1HNtuzDJI/fs kMNBg4idcngzgQMRWDvuzclDc/cA+9r0IVhJEqKDjJzNSzCbywJdux3nsw7R1VPcufFJj+rvv+nM eP1HEyqQYsyN9kirohUttCOHHc0tTucikkM9AQW4ohps56FFmIjnmN2wdEcTvcHige4NFgeEh1ls SaMs9rAHWezT4SweURzB4hXoOzuGx47weKo3uOwRvsHmkPIwnx1tlNE++iCnA0Kc1WMShNedkTdn gZN+8MTFqvDPuCFatBa3fNncN9vt5mWy+/WjmdyBf/eludu9vjSe/dUchhQFU3Pf5bDxtHc88tnc hnvWYX3rZqTiWXKYJIfpk+SPyr892UvG5qVNSYTAKZ85wuBGiq01KxKT7G699MhdVp8ZBqdayFsT cGaVZOPuLi9gEzh6ktqzR+RcA7NNUta6I2GADrMmS26i0Qq2EQs0RT0VQjvHfji+Wa5H9APOXOWh iKsY4sMBhFY0isvzNY67jnVYwpbyKuf8DfRhfH4Qe4VjF5EpLf4ZgX/E4VRWSuCmLbfOOxMFjl3x AtyqkkYm0qIv8SA8hwg+0u+MQYw9W58fwNZZZKforeTrKL+XnNDLGs9kW+Q/MbNjIubgIjgnWuB7 ZjSoVfqIGlqVsbdA7g7MZdSufZLBv+4CPAdw9wf74ERUwYWV6SM7j/e/JkkSUq9Ke12iaj8scrNd ZuVsZIbCvLsBlQIcQaSxZpkNTagob0Ocia1KiNJqmCNdqCoP0aClIaBd6/XVGCyWI8vJC55Xub32 S0nOs9ubDz7ehl9UZ7nyDg9DDPbRzXgfbIXKFQOMMCRPEHJzoVQh3UPsWaicaYL2VeU0gM9Lpt2h M4Kx3F4mQnjkMSmxId1wuIo8r9qIFTPD9l5U3Zxfh9Zd5f6pZEF5cGNtEun2Fm+4bEC673LihQjj kQ6+FBm0JRJP4bVUEQWySmAi6ZGEmAuQPaBkUoDXYNN8UykWrLDHTx1eRFi5oWwP0MtDoN8GAfse 1X9SuHxHjqaoux7MNQT4HVmCdc+LPxiNMUBD4Azxb70EEuHJxjL/GKQbtQDdn+LXcvwj7n06nhkW pXxdlZgbkHMqhSlrGUbuQfsMG1Ajlu3hgRXOOKZkbyt8XbVqX1Y8YFwhjJsGbgS6qhZ3haa2HO76 ys+hmnIBkabmxuDsJz1zf0YtRhaQjCsGyvmtAkcvkbV2NUYjvM0HhehhOVV4neq3S0H3zQ0EKwhS pmDdxjiaZSA0nQeYA9v9e+8sYzOQq9atq5ckq9hNv+yDbbtJ5aSoSCAAw4wcDllW2zjsra2UsO38 W96+O3Nj6qu7S5axfJTaMXtpCxV6R9nvLANnttQ29LD28Ko3yianaou4fDGfSRKCDmxyVxViVjO7 Oe9zRGA1ffm23rIGr7gK0/4K8+O7hExuEqUukEjkzdXZ79fhNkdd/cHfjoQA8xVsv7IXHWMT5F0r wjEG+2CM3LyaMcNi/KIlY2AMjRsSUbxSjzTdegMQ6AhTYyFlVYbstu4HGCcTZuTdKgZC13x8AoD/ YcrMxOrm+irw4ebtScpDB7cj0DIIbczvWpGCa/4JPVOdGRirOTg6qi5nEOBaRiQjNLAyEZ7os9TL zCpGjRwGuX7q8swmS27tP8bZT/X52Vmwq5/qiw9nuG38VF+eRVHQzxk6ws35YBCdnzCXJpwOZJit GR5sUUkUWMAqx7KORqe4OeNhodIY4PPQ/kpmXABN9nxeW/JlsSrUUNubvXzwehuW6a7QjAHAJ+uu V5aJEjg+t3lXI5jRcjlnjTuJHS7tXK5i85/mZfL97vnua/O9ed7ZbAWhJZ9sfphq1a1frtoWokXO ub6ODXes8loany3H9NEM6Akn/LK356lCgXBKrMLQ3yIZdQ52inkOfXN3DTHql9ARYEo0HEO3Y2il tW8SLDAlY0gSHAL9yEwpsT/rHlOT2UzCIYjrlaVtva+9PmilICisE5VgfpubegaKa2rc6ltGpH99 4aY8LocYTZLyWmUi5hXC3naxwmhiAjxtkE+8VMGSqCl+jWmRsXoCt2klw0NtlQZwJ8GgYOlL8+/X 5vn+12R7f9deEQ2aZGI2yf5EltiVtYZhky10nYllncFByGQEmbMiqNhxcSkfFWPZiUxft53STf4O HJ80u/v3//BuhGkQZZk9mQlzXOH6ZtF57n4eIEm4xP19hyaFpwUGZEYMIa6HENYNPKIUOYiDGi+D FtOLM2CYvXLGZ8KMU+H8FA9IRhlAAIEZkLjhbxvEAxxLoMp83KWBRQuGRk6P4iM+qL1Ni+uaVTNd Ydd7cz1OYRtiLpbRjkqJa4fFEcWxiMj1GYayBja6l20L1tuL5kG2FR7w5RSHK/rzInL8w4G5xpid J3URVg5QSiKlNt3ZalQJw0uYfsLFnh6yn8396+7u81NjX3lMbL3Gbjs5nbDvr0933anY+dS8SHNt qls8CyAksx5G7zSb4pc5I8n+FT4xBXj+Ra/rz4DRWbf4nCssy+e6hMNg//WARWkIROD4XLAw7V/Y 8jS7+qLZ/Wfz8i+wjp4H0MczdOEXslUFXwehI9PgeN0i8+KF345DAAFc4eEzDoCSZAlOu/GkTOYk 0G3ApnwK/j+ouC1YxUZp+y3BllgfLaggc522FMR/iNHjlkxOhWKjccsCrxYzy+UlP4ScGUEATx/P GJv52oFxHZAlpqPqFmINIRY8LAUzHK5J7LrOZO4UPlHuZmo8zTh+nUo446uiCIttQqK38LYTE7y3 UZyQuIiPiY/udsoYZtctVSbFHrsSTjCTrmlpPJdZL45BZqtDTtE0e4+m1ZRTtOGKKb0SAjdaPdUc /vcGhXqb5Bb8vsMkS3A5cQ+iJ7GPJqYZLqU9VfbGXMDtxmOYngI8U1x+ewqeQbQj0BvCvtxkzPcO IUfjj9DdGDcn96+fH+9P/A3Nkw+Kz0KrsLxGOgOh6o5K3xCY+w0TUOckUibX0UDAaD1pMHB5iae3 gTTl2cg49sComzJQeELt/N/NS2OsPpx1OwgFx28V99qbNfJigQ/fImv7lOHgNDrKzH8NWJjaxqIw dXILHFobHgZj+0h3zY2z2KfreRwl1bYWUNQJpWjNskeiqC7HM+pwVWLSfLjmBDMiOSkSLO8aUKX7 I/W4+eXF5VvtuaTR9rAhNvdVHME/VRxgXU9VlvoIKkWKI/ijeGwbhk113Al24aCwmkrNNeatDCTr 3o2wurK2buF2cr/5/vnxuXmYtE/FMD0BR6uVVQxlpt2ig553dy9fm12QdQnaaSJn5qA3D2RipmSP urvnPL5BRpTiKX5FhjZol3N0g0RFzi6MeI6f/hipiaxtLuKNbe3oD2xRS1GkzkQdHLZI9yzeQXrz CjZWdILRA/WRC4LzKlfqrfkClSi1Am862IVAFCHouf/WxEUxN2+27C2Avi2PWLqjp1mldCQhhJGL 3LyRfGvxHXFRTG81U5EdHajc08q3qA4Ix0DUHVYHVxFJ7yKk4wPnIC1bxt+yYPRHKZ2jjYULCOmc qLll6NENoqUBCO3YmRoT8BKCicglEELuHuEfTT1eVpw095PIKP4NYXKBqfsGwaE5CZUeO31bp3gs cbnQ/40i/1kJjccWCPHR9q4lZyQ74rTqiOnRFqL11Q4QaPfxikMD2ttfpTmeZkQbyJEzf4jaWfCj qUc3Kodoq0u8/IyX5rMHMdRS7R0RvPzfI0KG1ARXktho6Srizh9A2dsadwE2SsgAWSzAcFhe9n7b qF17aKFlwx4Bo96ljo/Q/qs8h+jDsQDaHfTsjyA3HiCD2CfAVGOfI60LguY22rkVs2ycthp6a8/c SKYqIIWFv00ETDhAJMnqABbC4Mo8TzlAQmS7fTGaSFBmxHgUi5nfdTKd1bnC9a8nENM/aIHrgKWZ g323NWVvkKg5OUfmNhDkyQd/hjKJZF94idsXonHLmF1Eoq2p5En8fASnALfiy4wU9cezi/PY8x1a RGxGltGYncHzoESTLFITdvEBH4KUUxTB4N/ItFawHpdXxu9F5tGWnDFmOPEBL4c2XIy/fU4oPtMp bCQxVTRLRFxk6fmwMrVfd/BvYNZlH5O2CfvJrtnuRneaZmJwqs8YrtNzkksyugHpkTRSIc1lgsvL FOcdAYOwliWWKzX5fFkFWfoVN19b8d9nrpiNlv0KIgsKvxFA05nZoPPAZmYWVJvqfpMexpfZNjS+ NMuE+WjPikjzcRtcLXt6yWbxjwgMnbrk4ugTBB3a1WOTzHSWYLd+PaVZ8j6XTFmZe7Pud5/xqW2E 9Af2p2PTCGK/jCQpgpDUVHuBC+PLH4YNvuLjE/SVYwe7aaluTr4/Pm93L81T/W13MqyqJ82Zwt33 nsK9gDY377gd4DnB7ZBMFzyL5hrq33H7SgnHHXLKShNB4OpfpIFKWLVNmr8e75tJ8vL4l3urOXxA 6PG+BU/E+Faucq+4XVBV25utk9Pt58fn02+b3Y+n168eF0HKdV6iFTLg1RYJyYT/6LOUru+UyxzU go3fGKUr8GFIUP3Qk/Ki/cSOV2C2Bs+upzBfMPImZvYskXwZMc8tAVvKiF8AelDPIX6RS67QEhrj WY7KloeCoECDco2b8hI8E4F+b6EsvLACfvQyqBSZDfnD8mWz29xvnvw71aIMq5/ah7zBnXD7tjc9 8HWN8s86ZphbNOVgMQ7QmBESQn+/xq/kO5IqVuncEVCxsmkbtLKxI8qCZ8c+1NaW2vr+m49I5/K2 1CLD3wz3q5DT4M7O/K7b58S2+CNWpNZ1UEzjnDZ4tcYfNPU8OvRCW5J8f+WmptQtengW4ePsl+rO ry8/Xu2Pto449jSRIjcOAE2WkToqTWoBGlezSEqkG2J+mB8ywi/FCtBFZT6udpktzy4iz7G4ou13 dTDD5L7Q4Nf8ux9Oo56au20D5GA2N/evpoTRhqGnjw+N+ft+93NnKjgm35qnH6ePz182k83zBDqY PBhz6j+IN+8E9KikpX9VD0hFNHZBY1CzJJgc/K7HT9N7aMkPyAaQ0WRfOiy4q06omZQi/DiQRwcD 4J6OR2PrO1Hnzly3uy960ViMByTWbKZIWgDYev/t8QcAJtvXHz82L7vTz69fvzz+DNPIlhn7L9X2 poq9Kt23CHlyfXXYZMF4oxoOJzuU+1PednP2p9p1ccx0zYd9/p+xK2tuG1fWf0WPM1UnZ0Rqox7m VFEkJDHmZoLUkheVx3FOXOPYKdu5J/PvbzdAkQTZDeYhi9AfQRBrd6OXpUu7DjYr5VPfrHkA8UWw dE80h9Jg4shZnOhgbw0mCVfzsXrKKDoxYmO3f+21lEW0jYUdE8jFwrV/OEJmvwChRUIDQruFNrtZ Xs6WdshH5RxkPyVk4HD2a80MjiJ7t0Sl56xoabkDcR37UCuI/UWp9FZzx951eRi4U5h66A/1a8BU 0LqeposOxxuaXWsQEbDjjHqixcCYjnSBjIP1VIyMalkk7to+YofI99zgNLJuysBbBtOpfa3D3hKa vlRqZ5GBjCafNavf7jjXExOIaMzUkf79KOxHB0KU+csMIKxKyp2hN1Vl9X04qaCS0XVjvx6uqqV1 E3Usg98+P779/a/J+933h39NgvADMBgdC+VmLMwIcvtCl9IKgis5k6TTflNnMTwWZYFWRWGXo29e tiObENBsju51vH+5xBXFtioA/B+FJFNWUJQ42+1ocx1FlgEqeuQ5DYyeLV/vnt+wb9964y/Rur0e cfNF20AT+K+I1N8DkFE9iPrEhFLlIK7CPwRB8zP9UoyPeukFhtDEIrc3Is6OMapcTANiNW+ZS0pN VS60SufB90Fw2m1mGm8HzcdAm/Tk/grmBCPDBDLYCJev4DqTZ8cLbDkntdj5N+1zRlerqFDHmtu3 rgDJ2PrpScOaUmuyH9ib50fBytqAGsAeLw1ofaKsv6/kNXA17VysC4ah6fSiPlg/OTlUiWVowxwk RpdWkOrWoDEdzEYLoggSSakjErHz1d4OZyhwlx2l2JWQJFShH8Wb7ERQ0IM2OBOE4YJO8tJlS+EV J9g/Qdx0vX531Qhiv+vPVJCxmRibas1UEupg+OmDViDlB3bZwC61pZ9VFGYlWgmXbco0pz5eTzNn 7VjmrfCZ63C9cVdlhbbgygGGh+24IKvXvd0yWaPcdiyg76hlIgPddxhmVm/oJcPga+o5WcwCD5Yh zcvWDaRVe4p4q+bDBeacpRG3sX9hBr6hj+zWcW6rIAxm68VPO33K3RIifyVz5o5dP866/mj2KxnZ YfPE45jO66lrWXPbfucZTdv3Gcr9pQj9YLCjQvk+v0ia778iRGLr5D3Ir5VlsWQy1BOWc4Y0dSuo xUo1bxb2TrAOwtCedI4PIJkKWNTzQE/nrYH08/vryxOGT5z87/H9K1T+/EFut5Pnu/fH/3uYPGLk 1i939w8dLk7Vujcd9ppCu15DweDrA2fpMitOtxk+VVXHY2QUu1TOGEXbbhtmFL7lvv+R9z/e3l++ TZQA0/nA5gUgAsLpx0TeV2+/lb3R6zXuxDVtk2hhRjcOt1WyhQrW9rkatch0SFIvSmgfOUVLLTQQ hpJIMvf9dffaiMxurIgHZvkgsYotQwriqY1YCimHnqX5aB+2w6rmFtMCTWTCz2hi4UuQdQL6QqyG lAxnoMm8Rqqm595yRS8MBbDoqzSdV0Q1dEYL1dJpPUpLp9UQmn7OC8HYdymA2PqMhQJSLbqrhm7r HqSfXMZHvwHQuhZFt6ijWrqlATbNmgIkfgFyIb20FCAVZWAHROlHnzmHNWCoDOuSQT7p7yS6HLha bstTAK0Xs3U/bpuc+KMAaJHBCRQaEDLuzWpjGagkTep+8E0qQkmhIrBZqo3iJcOX5bbNThFtsaA0 wKI+zm2bniIeo3STmUEp9KYXZR9enp/+6W98g91OXyD03YSNGUnOBj2JLL2C08UyjoObAOO2/8vd 09Nfd/d/T/6YPD389+7+H31hRbSfvc1Bou3eQj2tpUZKPiXuobpliU6mEIpSmElpgIAxSXxS6A2V imNqVIMlzrBkCJovOrnutAcxelq3pG4bGj9f8tuT+kKY+vLmGtmIBBASMQC6RHUzTdQGJJn6eR3n tftEuY9S5DIOESZlovWHAEuimmPtPsvKCUDDhCkY+VAl1aDr/CSKTqxzfEnHK7pbV1MOshX3vhbD qHhV1/WSZrSk6hqRpYvfxn7Pk7yhNi5HZKDebSV12KZWulclqFSlhX9NJiWj66NdVUld1lWSmBSd uKNfP6Ey0XemQoiJM1vPJ79tH18fjvDnd+oichsVAu2+6G+oiSAGSdL93u2MNvy4xOgbh7H6zxcV ugo9KKQJQdsyP/RzWN4sAW8dumExkaz/S5XpMHuZkdgEqUZYD1VghlHDIvPGA0tqA8B2RftBlOJZ U1sRUZpn4BxqOwPDCaBnX4jHic7R1M7IKkno2ShuK2CXP/FuFpzCgg3c6weHXnjatjL4n8zivoNC XXoJz6mfRNREBlCdE6X7HBahLz7pWom2o0iH32UB/zE9jsuK5uAOnCWJX/TthfXsRuO59+tFSOeM u3Yvxjwygkdgw/Stz2UWZMYWfcgKTktVnvN9Rlokdeprp3vb7HqiYzywbUTuON0K4lKYmw/swCk3 Is1TiXFfBj89x3FYm5wcO5LM99itE/stMy6Q/DKmWWIg0DolJDCxKYAy8lU6Uas5QJs5JfPjBYpx W8dpYstol6XMNTBewow2KPBDY+1sUtLlun0m2ItYmrl/6qJLSfdZQ6bb2ZBpU/KWfKDSd3VbFsnA zEnE9VqYklExOnWF5pzXsTxi2uO681TfszqMXSYLUJWGGHPAXh8GRBQGm70R7mjbxck3NibpMmru w2k30oC9cY25zx1Skuo8MIg4KzgVumCtfhSFye6wo02HofxAK1iiE/cIEJiXIGVsskWeuzgZQ/OJ 9ZXI2U/9SMb+67ymlvyN4bxhbEPkzXlkC0ygKj/NjGYn8Wl+YdwhFI3hKrvVRkFhjvqN9Lw5k6IU SAsHqqaZ5xv5CR49MbJn96XnwtAl429nynTNVvhxOrIZpn4phRm2uy6i+RjpzTx3ZDl4s/XU3Efc m/FPSw9RGBn6fJ0ss3f8DR/Mbnqh1fYXbpJDXRnrrVmHtYJpAQz6yODrW6rua29jf8Zdet/G7FF2 G/MuWSeRXkaZBnSRLYWx/3ogRjC2E0gqM3pjLDxnuR55WS91+HI6H5kLUggjWh3+9qZTSuXWfSqK fYN9ksHanc4oVxbjKdNQJpJr7iYzkg5j/yVJAbD7lkQanSDyKGBvTAG7no+tF1kqVajR9jJBH8HR uQ8nqnFW+Xl+ToTP3EdUIN3Rl00BxvlImTUfVfZGlGJflcYa1CUjT5lPsKLOFX8wNwf4eSn2EeOH iVT04wo4D9NtGNJfu49yRkGg3Ps27KmW78+cj0+eM9exMWOxXxpSpBKQKrlpXPz8z3ff33sqQKDD KJb0LoPEG//ISRJIzjH+V0Wrb5FelLHnMPGbkQ5/OFYAyVG+595+7A29Dr74rAIvHh/RR+u3YTDC 3yfvLxO09n//ekURetEj47x4SE4oO9GMiwyZS4l9z+tX622ev/94Z804ozQ3wzqqgst2i2GY4971 jwFBmbvn/qMJOvr8TWJK63qOvD28PmHg5uam9q3XlEuSVVLoesnySy796sRSJfA8cCqd/nSm7tyO Of+5Wnr9xn/MzgBhP1ocyE8Wh56+tdPzA18548kbcd5kfmHI1dcykOnzxcKjPXh6IOpcbCHlzYZ+ w23pTFf0mulgXIdxtWow8c0N41nTQMrAX84d+kKwAcEyWc0W6xFQQO8CLSAvHMa9ocGk4lgyl3sN JstFinz2yOtkmR39I5NquUVV6WgfncoeZDiBO/cKmcoPJF2i6OLHXcfsthzZQvg3zymiPKd+XkYB /WS0xZwnNxRNxcG45mVtef+GjmmsSsHYNHfeL1BOZRjRztuyKtjfkOFuNEiKIjKjj+tyYD5ioZ62 vGETJIv1isknphAHCXy0z4QQ1g24duQlcumvaXYbDM/CJTlEiAowwYUfVQD8Hr2lWVD9MLxqg9rf vX7+393rwyT6I5v07buhDzu6ePUTxOvp3O0Xwt/9YECaEJSeG6yYi0cNAaELRpQYSE0GfkXP795j XOQOTUV5o/AvvYr7b5YuXgeRiJ2fiL7/rD61v9693t1jKBl9nnb669DR/Aa1mltFj9W5MzpL6lBe AW3Z/jgsA1xbjOGUa13/lWlJo9Pau+Tl2RAq4HzKMb73NbxwpMzNAmYP05eqqhK2r/yYM1prOc/s U8YJ5JedpFl6lWQaJB3SywM+40anu9HGVQ+vj3dPQ6173T7PXUzNaVkXDnt1G50uwi/ic4B3Y92E Ot3HdAh7itD1n+4S0uJS+UUp/5y5FLlOKVFj5hSESL3ToSZ+CuOEsdtputz76Mte3NJkffvN0wvZ GLKlL88fsAyGQ3W6uu8gLvrqx5NteNlLSgKrAeadVaewMzj9Sj8yc6YmQ1duRBH6TKjfGlXvAx9L f4fd/gvQMRiqbsYwJ8wPf4L9hUdGeRKBKJyGMRlcDlZ9k/GmFfuuhTqDRZT1vNsHsIHOtyVxlnEt QuWPHMH0jF2u6/2g3cZbHcxsvaRPUzyOo4BJEimz9MzIucmxl5a3o1rzVrPlz8suZ3TrqQx4Ipwq RGCH63fnwjjiVJLBpCfkdPQc6U6naRrE+Gz3vwD+5MzeKOKgn/24s8H3jlvo8Swj89AB+9G9jA4u is80E8hjcRO9rFsGO0pf2oHiXoz4DqUO7IHGNmZNfoz5P5rI/cgTNawHJhNpN5fa3XkiEyz/+vL2 3rGIogRoXX3kcJaODX3JCNNXOmNJqOhJuFrQgktNxmtQplOAZXLM3oi0dVunBK225v1+TpV2mYlP BXTla7rmvxroS8Y8tCavl0y0fyBzlmw1LS+G6SiUeRczRDJIhqlucB68/fP2/vBt8heMdv3o5Ldv MOxP/0wevv318Pnzw+fJHzXqAxxM6AL/e7/2UMhol2pfHMbMXHW8SmmTMPZIiMh4aU+NU+DbTeIV 6ORbWyGjpGQyfiFZHx8sWezc6ZAvFT+BJ32Gsxogf+i1c6dVcNyAhFGGYdMqTjzBD9FBZIAJ3+2Z cwxQpZ/Jizjw/VFGwLigcrDf6Oz9K7SvbXFnDhhshkq+cvGZ6GR6/DGGTf9GjIDgRjQC2TAqxp5V lt6oYM8lGKPc3Iob6H/RYvLu/eV1uOOV+eT+6eX+7z6h1jKaOqTJh/+o+joKxraCWv34746/ASb+ OYIAruw8MwzjhUcgyFbdLacuohWg1OamcpuphELx2bje6pTbUgiFvoYy558sLeQ669kllO6KMfc1 IEzcghqyuXWnn1ZMxIEWs/rJxVCoMYl/clZTLsSHCaJbhEfuTuD98slbT6ko+ftjYtrtqALYkjmO Dqn1+tgTMbRT7RpErbs6JI4PJ3e1qwo6RtkARR+jDSxczR2aGzQgtM6zhSTOlFHwmRh6UE0MfbKb GFojaWBmo+1Zu8zcaDElfPtYPYBZcpqNDoZR65qYkf6RwYqLE3PF3HilYHIsNBBnOorZ+omz2FuW fNumDXuZeYWUp9ze6FAuGc+aFuGMfXm0wGwY9I7ZfNbK8aYL2gami/HcLeNu2YAWs9WCU9LUGGAs GcnuCtmtllPGL+uKiBeOx0rfDcadkrl3rwg8i3HMTdG+7rfSW1mr/xgw22MbpSsVXCiWBpMwHH8L WI0C7MsDAPYPAYB9I4sTzom4BYw10htr5EhvxwlnaNACRoYjWY80sgzmzsK+mhDjMmF/DMz8FzDL kXcl64U7/xWM/V14li+ny18AOfYDRGGW9rkCmOVyNlrPcjmyeBSGuSe/YtIy0DlcI941toZqlYu9 OsTMXfs0zIOtt1jTY5InLFdePy335chubROXG0ySLEdGM0yEs5rZP0UkgTOf2pcEYFxnHLM8upwb fdNmGcxXya+BRhayhm1ma/v3wTGzWJ5OROxMGjoy8grDOKK23Ip0piO7A2CA0x/hHKFLvZGZUu6T YGR9lEkOTOEYZD4ydAgZaQuaCQV5Nco+AW7pLe0n+6F03BHm8lB67ggfe/RmK48LL9LBsCFIuhgm zqaBsa8RBbHPC4DEK2/BpFszUUsuVkyLgum8tzN0GiRGUOoyWibDW1lOj9Y8qBJdjzPK5c3UcSjL vjY7t1kwFC6PmEoizMh4YWiSlUkZ6Qzz+pbu5fnx/m0iH58e71+eJ5u7+7+/P909G+EYpCRDzgaJ P6hu8/py9/n+5dvk7fvD/eOXx/sJsNtGEAB8bNB/yY+n98cvP57vURFyNXwa3BzitVV9M9VUh2We lyfeckpr4BAgkwWzrvdloBIWB/SULXO5dKYLxu0HiIsp45BdHmM4T6a8hku1C7YTnloGs4W35r+q TJhLkRwz9Ib+ejpj/J6RrmyQGIkKa1f6DEYYb+jMqdPS6a0d6cfEmzNCYU2eOSfWFuMKWUzHIOs1 YxcCZGBgFtPljB57f3NaTEcGcBivoSbGedD3iEd8EQYzLoAB0j/66adLkGRc4gTEHE7egh+WY+y4 qxnf6kLsqrgf+qatQYRwetUZxwerdPd69/0r7hYDk0R/Z4iN8JON86FojO+6oiVkkB1N6fqDYtHg whQLtdE/+wIunoCi8bEKkMzdsCBNbLdRIEgvwMMOI5NsOmYiukD5+e7ySmJc7qYyJMpjBLu4KDIy q2xh+o5j9uRwS08qJBYOw2Epoh8KhiFEcuQxtkCKuCbDXCAp2fn9NnL9jjSuY9Vz/oFTHKhHk+OO +XhMdw8nVBXLDeO2tfNVTi6aGEHPXLKqRFssKENHVeJrt4ZKvslQAAuJmsfbzSU0/R2gRIWdPAhJ rrwWFsCfbRTHhZH0qSYEWX6G9/oDgoqAu4mjsvdSpBUqivBJxBIzKmB2Q7InAIkJEK7vtmGuzbBh mhZxoC1sXtEuvYgU1jE9N69Nypi4dEA/JuVlJy6FBSJhAnAXKFtcnrgVM5eLOG5+cDO4aes8jmnD 9QW7NIaljGL1/aU2DlN76/b17tvD5K8fX748vE6+Xm/aCZYShy4qCka4Bmqe0FIOPnjeiMLlnA0A 4Bf0WYokGcUwGuygRYksWSL0M2NGDMQKpz77pIUmtkzUhc0l5XTyQNvv+OnphA7rcYXV8kcLLqfo wNLYjRRosfCmC1Pt2J2EsPGceotXF14SWJEijZjkux0cmqzekjntWtDOmKN1oTZGoar0D4zxKnYE f6jgTCrP3Imkqezo0IwaUvgzAqnMyYPTix/NVGSwUUXsgrg5F/SpArQZdx7jK7MszDJ2doI8v3TZ Dy0LOJX4RchlSVbbAltp4BcJ5/QEZN6wDBvECSHY74kMKr4jqpB2KVPDUpQV43GGc1DAHEwzJjkl AjYem+YAN24MHyD3gjHSbvb1SxyELDccPr6BpPzP0E2nZU8imcf+ma0heHl+e3mCXf5ak07CNOCu kSMMBqbGOz/A8DHZFhPEYRCNjZHYnqbrXC/LxphVxR4hLG4LPxGbarsVxS8Rr7aqeQGHe2HYFlDo Iiv9fp6cKzxLu+ay+PPi/fQGJY4RkkkVLn9yez5SVz+ZS2xFzdG+GF/EQ6A7i9QOgVUUXeY/mZMO EbJKse1WgOP+dJkjHBHO9KdDnRVxtjPcPfE3XrtVsFnDYiEr7GAGR/QQEsRV6bqG1ZvMqpTic/dR s3A6FvNRaPwo/OOV/UFjo67vVTcFmdwMngSutVNQI7Tho9Zqofbp7mkCpUM1Ej7vz9G3xTDKwNKg IC0lFQ11f2YjNiK+iVKzDKW24twvi+BXv1CHauw3ATpll6UFFxsUISIBxpcKqqCIn27EuV/pPkN3 brZCeGLgYdMln0W/wirgI2wj/ejHXCBOJO/OxWD9G4DyGKV70nlXtzeVwDqXPasaDFMeDJSsXSq0 uR52ovQSfuwtH128pXXCSC+qZBOL3A9dG2q3nk9t9CMcRLFlSJUjN+7jZrthUZeiGA62ciPhPaYQ ArJsRCvNkYqBjvjJkoMMAHM8zhi/EoURpR+fGZNIvXyDDM5Wnl6weRSRfFtBI2jGDclFFgQ+fbSr fclP0Gudp8Na5Ym5EGE/zYmJqNI8ZsQzNfnR7QukKb79Es7Q8mN2tlYD60sKJi2sou9BRiwTX3I5 aRGEofqsb/l0Dv1+RNXOrjvY4pXHuLpjNMuUQSOmAd0HIUPJ9l0HHoMkeNKpTzJx18QN3UfTFA6t QKA76aWN2KbvIR7f7h+e8P7j5cebOpReBhkxVa34lJkZtS2Hc9JnYvAqkA6khsxCxiRbUbBjQBkT Iql344NFx5617oB4CTa+scE0x+4eLfbbOOQdr2ejkmC5Ok2nOH7se3As+oAOWdTkfttVeYE6sH0F AgXv76+AZYkDx6dERmB2qlxnus+trQXm3HGWp1HMamn/asTMlq7lwzPmw1X5JtrVii60iedDLfTB SU/w6kNRvwSstkabK6Bi2iNj7/8bu7LexnUd/FeCeT+YZm16L/rgNVHjrbKcLi9Gps1MC3RDmuJi /v0VKduRLNE9L9MJ+VmWJYqiJIqEZL0D38qX3mIxvzgfBN04xMAArG+8Pr+TxsZtOnjZfTpdy3EE B+4NDww5wTHxMV25kH5WOE52s1xE/xlh64icwybD4/5j//b4CakeMRvEL2m0npJTjV53f1tP8N3L 5/vo1370tt8/7h//OwLPb72k9f7lA/NHvr4fINT/73dTzTQ4q5sUecBRW0c1sXy+xYWe8GKPViIt LpbTF3UBTMexMqQS2Okw+X9iqtZRZRjyM7cfVR9GnI3psKsqxZC53wK9xKuIpLY6DHK7k/aWDtx4 nIjwraOalQGEWCYuUOjoKJON6C8mQ2FUzCxFGidhflVW6FdwWoa97v5AUJTTdWmjsDQMlgP9isZq T966onthOM3vwauYzsfMWZl4PkoZ4WLdcAmncVQoYSUqt7mpqrYtI3qkcZbPB1okiVa5IF0/EDGg K5OBaa4VlODuPCA8ZxUMnUXoCSzECAgkPxYhw3AQdPvB2rjZ7aJbkZXyz5bYfMdvpT8VruUH0rry OXl2jZ+S33hc9geNgFsqA1M57PrF7kAFALirOBEDFrgrL1w5wtDmYjWCADqH/cvuuIdAOr8Pu8/j 4evh+HUwHG2yvFCGVBAxd/KQ6satEVIiJ04apVa0ivZl0vyVvWYYsfDbNl+xiiCIj2byeCCJw/Of P73h6AXSpC6ZzxIqUhaT/2bM9zJ3j0dy0eGoMRdBra5AaQSsrUlaByIv79zExka//HE4PpxpOeIB Yk2o+FEc4l85Yh/BE9LWiFXCQPNlSG+CSPfJvfu5Or2uWIShSp2tglXkW0uEu4uZUFOHdmyfC8vx 9MztsKNDiDgqGmRBJJU9QaYTIkNGCwGPZiqQXovh5TyYfvMqVibjyZl7S9bEEFuqLehWQtyWQ4tA b2dCzRoYwv2/+/TZWBBXCFqIfz0lot22iHI6n14Q10JaTJxOx8Rt4q6Jb2V13ZvmGoS6k9VConR6 RjgMd6VsJeQimFhyC3c0TbltUq/vjtIyfu3xeqUGqRmFWpPRydI93WuQOXFcoEPmw30pIeeE737X U2IzPhfesIims6X4psIAIS7M6xAiEFcHKdPF5Jsa+9ezJeEp3XVnMQ8Ij8oWAh1uZyB5f/snKCp3 r4apd9qO6co7UQn9DElzrEOD7t44PK/HU0A6yz2h0oB1r4HwJDWZfwd2h1NX6AxJbWK7tiY04IKX 5/2bHpkJwjz2Xogpe2txS75U0vuavuUEZknV7aDtxex7/9vnw/H53TVZwMmrCvLkLKxh+16S5MT+ fQPp7090/FJ4ZH7P2+Wiu63f7s1Zle/iUJQfz294gdoSpSDZlDzA0NJTLcZTsom2ok/Fn7WZFUIi fcgn0UPmaVppe56aOaOkSrLJJnFljjhttd4VEXdbfAU2hFx1X1sNET8fXrEhHFt2Ueh0P2vd06Rd mHpaWKQoZvXW472wVElSc9/9SWEQ+s5lZZgyFpoDmA3kVFaJl5kfC1k5whyUVjkxRlZ5vkqi7rMc LfSybyJR6HkfbsWk1k22hlDfQlZpm1zkJbuVVq0Zjr1hllFQccrO7Qp2xruX3Glt7iIjKYwwfVBZ i7zOeRjxyNWXzdOOSk+HKj2lKn2CzPqNM9Pfoxc2M95ENcHs37TTzFbvDfPK144M5A8tmp0sN/UD L1hrJ7M8YmXE47LXsh0ZY/y4p64WgqnqIeLOMKxtFCfqCgFuLUezVnE5oXh5MMD0BadLzVhiP3pq Ql1UOkUhhxXoRCN+XVyqvD4nStgnsCbzD+SIMjw0PMVw1u+6yonjPOQEwuV4CzGL4tKU1RgiOZr9 HvS2NtqGVrnLmq9WmnP38GTud8UlypalVoLwH56nP8NtiBrGUjCszC8WizOjald5wvRUM/cSpPOr MK77v7OkO5kK8/Jn7Imfmei98rTULyWGkoCtfJaUDmFJjrLBP/dfj++j364vxBGiVxcJGzNuHNKk 6jeSMiGxgJ3LNM+Y0NPl9hbSIi2sny5ZVYyeGlxXq0gkvl5AQ8KXa1M9/rHUBQTfxFEgKyWilGi5 SMCFAQqnWnEnTS281iT+fpjiVXjSUgI/CJXqPXCqedWtHfRUcSUcpx/ttHH54+v4e/lD5wR5GGGT z6bn5jMd55zmnM8JzlKP4NjjTEgOXRpVg+WCfM9iTHLIGiymJGdGcshaLxYk54LgXEypZy7IFr2Y Ut9zMaPeszzvfY9UOcvl/KJeEg+MJ+T7JavX1F4ZMOYuf+wmT9zkqZtM1H3uJi/c5HM3+YKoN1GV MVGXca8ym5wta+6gVSatErHWBWFi3udJHMkgUW1s9oe3/cvoafcAUetPylhFZWX8Ok68VWk7ZeCZ qAprqL1VaQu5EihL6OAELOltlJwCnUYZxDGupWENR9MFjwK5INVMsYafVqVQRpU2D4PTJz55OTmb aRHb0aMaFD2/dpv6grNCilYqMYTSrTKICQ98P0+IK0UYPImY7taRJ03qITtQPS4NVtTNciZI4U6t a62DwA0kkTPM7KaANSO+EgyrqmicZonDCojkOIDr3oR+t+sqg9yV5cbu346F/Z9X4nI8OdPsxjwX cnLxUg1YeO4MMSRWLh2TKro86yZy8M7ofG9OnQtkuHDl7hgMu1nTfNWquX8lv5i4GJ5UfgsjghwA 4rtXbDCPgueO5AkAuZpPKxHd6qkeFacAx4vSjPTZgHHfnjgFbASmYFk/gmgP0rx2AAEF1LA9Q+Tl 5fm2s+zdTYS5IGoeVDWGhyeqjO+Blhyoiydy8HQqkyhyO4OeqixrU3uy9cooiaGhhj5RvlW+uyrJ a4GqF5WkDCDWbLV2h+Rt4wquthH38zLqd3MTjbVXHlDl6jCsAuLQVsGIwIsNc+Dmg0LccPBLzavM uTWopCRKc34n68NEv+Z96cSofUCsy5wIfdw0OpGcGJm4tmPU7lYTy062TJQJ5iWuZegp4qGaOxrR K/tzSkPXPwGvTzZznJUK+DQMpTxLHSoqHlknaBokKCqQryTp1OVC15ZS8JoY6FJO6cPkRn5wjEBK 2zDaDgExbGx9I1VODdF4YYecBjcIOZVSFxtR77PsSk1g7tErp/IsuOs5aZ/WYXLJhn+gIdq5vV8P OfwsPdPO0aXS5F1rWVcBPKmF4ipTUyyCOMVdca9Y/ytMXGCXDL6mTnHsuICtZ2aMwH4pxtPSGMp5 2IO0DYZIFNSyhwiaB1Up2hKWQ4LhlWtvWb2X1OUsjNDfD7I842kEDHrqPAE1LO02ptT1YtYpYTcK Dy4gR/iCHgCll0KoFIdoYDvgaN+sQuPuNPwefkDqNUsYO6LbpgD1tpFWpcipO+KcBThA3TMBnEg0 kwAqmkrfNIecBI3mMTZGNXod+ivXGNNmMjkh8zjJb/Qi8CBEhFVKP8tzcNnr63dF7Ul3mFfSUle7 A/qxhTrPENydMwobHo5xiDHMcqVuIf9xVJ/dLs9OFmCfJ9cNYzevwv9fTtzcLM+iy6nFw5cZX9Ix iKm3Q1T0FNFhstyZq7E1qY0qXmrTQ1B4A8MLQvmm7B7yeyRylBLmVysYKJaoR+FOvVLC9u5SuX/4 Ojwf/2p+6qeREZEZQtRWPERGK2E/W73NNRM0SMOKb2jE8qorvNkcGyhVTiFi7SpaTtRpfkccH7YY r5CGakqsnpj7tLbpwFMD6GfDfe7lj24T7RZSPMNsril0pQfNHU9FA41U3PWpt/pWpyIV132KUqsw SWz7LDgqZiUuuYtrWHE1J8oUCOpsoQJ+V2CofyUoh78fx/fRA7gfvx9GT/uXj/1BO0VFMETglvNG v4yGPLHpcn3tJNpQP9kEkMSP0xz7obVXrp1EG8r1w4sTzQYWkEzBTbXBqZdJ08euc0M3040rFoxj hzyaD3Y9h+sfq/hVPJ4s0yqxGFmV2MQC/1pkGJHXVVRFFgf/ODquEmtpKbYS430dn/Zvx+cH9GSM 3h5AgmBz+3/Px6eR9/n5/vCMrHB33FmSFASp/VkOWhlds+4up49H/K/vj7p/R1ugH9gfIuyuCRzt GQW+RUv4jaMlHS+5Fd3JzHr3+URUb234nrSPusrbKmRzsfzP/vNofysPphPH5yK5i4TUa1znYEnD mYM2t2ksWHvoRmO/lqehFEcnWd+tP5En84WLPJ3Y6HLtjV1EVxGSPB/bY1Ss+PjCJt8UCqy03/PH k+k10uoqW1gkrZ4vjTvnGidjqv3pIe5llc8cxfLA7go50dzEzNFvLcO6YdeKggdpyZml7iETr7C7 F6h2g4aOr4/dymSz9u4dur6UK3zP1a2K3rRjf8RHjoIiXkSZ/V4R2Z8obQ1nmzV0qslatqpTE5Xh 9eOw//yUWs0SDWkiwMrZKubecEtulcl9N9Hy3dvj++so+3r9tT+MViqphOsFXlYyaUaqics1i+AS Dka10+zpA8tmFqUFc33jeBHmbPVCci9Sg1nBQfTFw2ZrnGw0e1vsnr787bPM483ayb6umDz/OuwO f0eH96/j85vhjI12k25P+Uyua8DhSuut9tAfPI3AaUiuDX1myJc06AImTNJ4YSLE+CxksUljoqrN p6aG9gm0gygpK50u15o0cAV18KqQibbepyJA3mC70RRGpFoiKsUQc0U10QY0ahhp9NPS8x4YbscV ZNV+cOXs8bLGFGf6ngeQ+js/Zc8kxZxbkA3Ig805jVFUcobQvdSyxDzkT3hV97x/pNj178zBhot5 nvF/jXiLDn39AAA= --20cf30563a018a72fa04ba7aceec Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --20cf30563a018a72fa04ba7aceec-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter Moody Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Wed, 28 Mar 2012 13:51:10 -0700 Message-ID: References: <1327519203.4131.25.camel@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Received: from mx1.redhat.com (ext-mx16.extmail.prod.ext.phx2.redhat.com [10.5.110.21]) by int-mx02.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id q2SKpg0b016714 for ; Wed, 28 Mar 2012 16:51:42 -0400 Received: from mail-iy0-f174.google.com (mail-iy0-f174.google.com [209.85.210.174]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q2SKpekW026536 for ; Wed, 28 Mar 2012 16:51:41 -0400 Received: by iagz16 with SMTP id z16so2701503iag.33 for ; Wed, 28 Mar 2012 13:51:40 -0700 (PDT) In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Valentin Avram Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com Are you still able to reliably reproduce this oops? I'm trying to track this down because this bug (or a very similar bug) is causing some significant headaches here at work, but I haven't had a lot of luck. I'm using usermode linux, though, so that might be interfering with things. On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram wrote: > Finally i found some time and spare server to retest the oops and list_add > corruptions i was getting with the 3.x kernels and auditd 2.1.3. > > I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and kernel.org's > 3.2.9. > > Both get the oops/BUG in the same way and after that, they keep pouring > list_add corruptions with audit_prune_tre(truncated?) and auditctl as com= ms. > > Since this is not about Gentoo's kernel only, i'll post here the oops in > 3.2.9 and also attach some list_add corruptions. > > 3.2.9 BUG: > > kernel: [ 301.240011] BUG: unable to handle kernel NULL pointer derefere= nce > at (null) > kernel: [ 301.240305] IP: [] __list_del_entry+0x20/0xe0 > kernel: [ 301.240481] *pdpt =3D 0000000000000000 *pde =3D f000ddc8f000dd= c8 > kernel: [ 301.240698] Oops: 0000 [#1] SMP > kernel: [ 301.240910] > kernel: [ 301.241030] Pid: 642, comm: fsnotify_mark Not tainted > 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396 > kernel: [ 301.241370] EIP: 0060:[] EFLAGS: 00010287 CPU: 6 > kernel: [ 301.241498] EIP is at __list_del_entry+0x20/0xe0 > kernel: [ 301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff EDX: > 00000000 > kernel: [ 301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c ESP: > f47cff64 > kernel: [ 301.241879] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 > kernel: [ 301.242005] Process fsnotify_mark (pid: 642, ti=3Df47ce000 > task=3Df4f47c00 task.ti=3Df47ce000) > kernel: [ 301.242207] Stack: > kernel: [ 301.242327] c10813c0 f47cffa4 f4f47c00 f4e70888 f47cff7c > f47cffa4 f47cffb8 c10f6976 > kernel: [ 301.242882] ffffffc3 f4f47c00 f4f47c00 00000000 f4f47c00 > c10530c0 f47cff9c f47cff9c > kernel: [ 301.243438] f4fae544 f4fae544 f4c47f58 00000000 c10f68f0 > f47cffe4 c1052834 00000000 > kernel: [ 301.243995] Call Trace: > kernel: [ 301.244119] [] ? rcu_check_callbacks+0x110/0x110 > kernel: [ 301.244248] [] fsnotify_mark_destroy+0x86/0x120 > kernel: [ 301.244377] [] ? abort_exclusive_wait+0x80/0x80 > kernel: [ 301.244504] [] ? fsnotify_put_mark+0x30/0x30 > kernel: [ 301.244631] [] kthread+0x74/0x80 > kernel: [ 301.244756] [] ? kthread_flush_work_fn+0x10/0x10 > kernel: [ 301.244885] [] kernel_thread_helper+0x6/0xd > kernel: [ 301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89 e5 53= 83 > ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 0f 84 8e 00 > 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 83 c4 14 > kernel: [ 301.248195] EIP: [] __list_del_entry+0x20/0xe0 SS:ESP > 0068:f47cff64 > kernel: [ 301.248414] CR2: 0000000000000000 > kernel: [ 301.248538] ---[ end trace 15082dbfb353f84c ]--- > > The kernel was compiled with the following DEBUG support (the bolded one > were requested by Gentoo's Dev: > CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=3Dy > CONFIG_SLUB_DEBUG=3Dy > CONFIG_HAVE_DMA_API_DEBUG=3Dy > CONFIG_X86_DEBUGCTLMSR=3Dy > CONFIG_PNP_DEBUG_MESSAGES=3Dy > CONFIG_AIC94XX_DEBUG=3Dy > CONFIG_USB_DEBUG=3Dy > CONFIG_DEBUG_KERNEL=3Dy > CONFIG_SCHED_DEBUG=3Dy > CONFIG_DEBUG_RT_MUTEXES=3Dy > CONFIG_DEBUG_PI_LIST=3Dy > CONFIG_DEBUG_BUGVERBOSE=3Dy > CONFIG_DEBUG_INFO=3Dy > CONFIG_DEBUG_MEMORY_INIT=3Dy > CONFIG_DEBUG_LIST=3Dy > CONFIG_DEBUG_STACKOVERFLOW=3Dy > CONFIG_DEBUG_RODATA=3Dy > CONFIG_DEBUG_RODATA_TEST=3Dy > > I attached the kernel config i used for 3.2.9 to generate this oops and > warnings. > > From the list_add warnings that come after, out of 805 warnings i process= ed, > after masking with XXXXX the PID and next=3D values that kept changing in > every one, i got 26 types of MD5. I also attached the files relevant as an > archive to this email. > > The Gentoo bug i opened is sleeping, it seems nobody has the time to at > least test to confirm or not the problems i'm seeing (or everybody's > thinking that nobody would restart auditd so often, so the bug it's not t= hat > serious). > > > Thank you for your time. > > On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram wrote: > > > -- > Linux-audit mailing list > Linux-audit@redhat.com > https://www.redhat.com/mailman/listinfo/linux-audit -- = Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 Security Engineer=A0 pgp:0xC3410038 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter Moody Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Wed, 28 Mar 2012 15:42:37 -0700 Message-ID: References: <1327519203.4131.25.camel@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Received: from mx1.redhat.com (ext-mx16.extmail.prod.ext.phx2.redhat.com [10.5.110.21]) by int-mx02.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id q2SMh9Em017347 for ; Wed, 28 Mar 2012 18:43:09 -0400 Received: from mail-iy0-f174.google.com (mail-iy0-f174.google.com [209.85.210.174]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q2SMh7aq021130 for ; Wed, 28 Mar 2012 18:43:07 -0400 Received: by iagz16 with SMTP id z16so2878363iag.33 for ; Wed, 28 Mar 2012 15:43:07 -0700 (PDT) In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Valentin Avram Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com fyi: this patch [1] seems to fix the issue for me. The explanation in the subject would reliably oops my machine. [1] http://git.kernel.org/?p=3Dlinux/kernel/git/torvalds/linux-2.6.git;a=3D= commit;h=3Dfed474857efbed79cd390d0aee224231ca718f63 On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody wrote: > Are you still able to reliably reproduce this oops? I'm trying to > track this down because this bug (or a very similar bug) is causing > some significant headaches here at work, but I haven't had a lot of > luck. I'm using usermode linux, though, so that might be interfering > with things. > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram wrote: >> Finally i found some time and spare server to retest the oops and list_a= dd >> corruptions i was getting with the 3.x kernels and auditd 2.1.3. >> >> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and kernel.org's >> 3.2.9. >> >> Both get the oops/BUG in the same way and after that, they keep pouring >> list_add corruptions with audit_prune_tre(truncated?) and auditctl as co= mms. >> >> Since this is not about Gentoo's kernel only, i'll post here the oops in >> 3.2.9 and also attach some list_add corruptions. >> >> 3.2.9 BUG: >> >> kernel: [ =A0301.240011] BUG: unable to handle kernel NULL pointer deref= erence >> at =A0 (null) >> kernel: [ =A0301.240305] IP: [] __list_del_entry+0x20/0xe0 >> kernel: [ =A0301.240481] *pdpt =3D 0000000000000000 *pde =3D f000ddc8f00= 0ddc8 >> kernel: [ =A0301.240698] Oops: 0000 [#1] SMP >> kernel: [ =A0301.240910] >> kernel: [ =A0301.241030] Pid: 642, comm: fsnotify_mark Not tainted >> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396 >> kernel: [ =A0301.241370] EIP: 0060:[] EFLAGS: 00010287 CPU: 6 >> kernel: [ =A0301.241498] EIP is at __list_del_entry+0x20/0xe0 >> kernel: [ =A0301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff EDX: >> 00000000 >> kernel: [ =A0301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c ESP: >> f47cff64 >> kernel: [ =A0301.241879] =A0DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 >> kernel: [ =A0301.242005] Process fsnotify_mark (pid: 642, ti=3Df47ce000 >> task=3Df4f47c00 task.ti=3Df47ce000) >> kernel: [ =A0301.242207] Stack: >> kernel: [ =A0301.242327] =A0c10813c0 f47cffa4 f4f47c00 f4e70888 f47cff7c >> f47cffa4 f47cffb8 c10f6976 >> kernel: [ =A0301.242882] =A0ffffffc3 f4f47c00 f4f47c00 00000000 f4f47c00 >> c10530c0 f47cff9c f47cff9c >> kernel: [ =A0301.243438] =A0f4fae544 f4fae544 f4c47f58 00000000 c10f68f0 >> f47cffe4 c1052834 00000000 >> kernel: [ =A0301.243995] Call Trace: >> kernel: [ =A0301.244119] =A0[] ? rcu_check_callbacks+0x110/0x1= 10 >> kernel: [ =A0301.244248] =A0[] fsnotify_mark_destroy+0x86/0x120 >> kernel: [ =A0301.244377] =A0[] ? abort_exclusive_wait+0x80/0x80 >> kernel: [ =A0301.244504] =A0[] ? fsnotify_put_mark+0x30/0x30 >> kernel: [ =A0301.244631] =A0[] kthread+0x74/0x80 >> kernel: [ =A0301.244756] =A0[] ? kthread_flush_work_fn+0x10/0x= 10 >> kernel: [ =A0301.244885] =A0[] kernel_thread_helper+0x6/0xd >> kernel: [ =A0301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89 e5= 53 83 >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 0f 84 8e = 00 >> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 83 c4 14 >> kernel: [ =A0301.248195] EIP: [] __list_del_entry+0x20/0xe0 SS= :ESP >> 0068:f47cff64 >> kernel: [ =A0301.248414] CR2: 0000000000000000 >> kernel: [ =A0301.248538] ---[ end trace 15082dbfb353f84c ]--- >> >> The kernel was compiled with the following DEBUG support (the bolded one >> were requested by Gentoo's Dev: >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=3Dy >> CONFIG_SLUB_DEBUG=3Dy >> CONFIG_HAVE_DMA_API_DEBUG=3Dy >> CONFIG_X86_DEBUGCTLMSR=3Dy >> CONFIG_PNP_DEBUG_MESSAGES=3Dy >> CONFIG_AIC94XX_DEBUG=3Dy >> CONFIG_USB_DEBUG=3Dy >> CONFIG_DEBUG_KERNEL=3Dy >> CONFIG_SCHED_DEBUG=3Dy >> CONFIG_DEBUG_RT_MUTEXES=3Dy >> CONFIG_DEBUG_PI_LIST=3Dy >> CONFIG_DEBUG_BUGVERBOSE=3Dy >> CONFIG_DEBUG_INFO=3Dy >> CONFIG_DEBUG_MEMORY_INIT=3Dy >> CONFIG_DEBUG_LIST=3Dy >> CONFIG_DEBUG_STACKOVERFLOW=3Dy >> CONFIG_DEBUG_RODATA=3Dy >> CONFIG_DEBUG_RODATA_TEST=3Dy >> >> I attached the kernel config i used for 3.2.9 to generate this oops and >> warnings. >> >> From the list_add warnings that come after, out of 805 warnings i proces= sed, >> after masking with XXXXX the PID and next=3D values that kept changing in >> every one, i got 26 types of MD5. I also attached the files relevant as = an >> archive to this email. >> >> The Gentoo bug i opened is sleeping, it seems nobody has the time to at >> least test to confirm or not the problems i'm seeing (or everybody's >> thinking that nobody would restart auditd so often, so the bug it's not = that >> serious). >> >> >> Thank you for your time. >> >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram wrote: >> >> >> -- >> Linux-audit mailing list >> Linux-audit@redhat.com >> https://www.redhat.com/mailman/listinfo/linux-audit > > > > -- > Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 > Security Engineer=A0 pgp:0xC3410038 -- = Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 Security Engineer=A0 pgp:0xC3410038 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Paris Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Wed, 28 Mar 2012 21:14:03 -0400 Message-ID: <1332983643.384.8.camel@localhost> References: <1327519203.4131.25.camel@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Peter Moody Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com That patch fixes a BUG() . The report has a NULL ptr deref and some apparent list correuption.... Sadly they aren't the same.... On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote: > fyi: this patch [1] seems to fix the issue for me. The explanation in > the subject would reliably oops my machine. > > [1] http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=fed474857efbed79cd390d0aee224231ca718f63 > > On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody wrote: > > Are you still able to reliably reproduce this oops? I'm trying to > > track this down because this bug (or a very similar bug) is causing > > some significant headaches here at work, but I haven't had a lot of > > luck. I'm using usermode linux, though, so that might be interfering > > with things. > > > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram wrote: > >> Finally i found some time and spare server to retest the oops and list_add > >> corruptions i was getting with the 3.x kernels and auditd 2.1.3. > >> > >> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and kernel.org's > >> 3.2.9. > >> > >> Both get the oops/BUG in the same way and after that, they keep pouring > >> list_add corruptions with audit_prune_tre(truncated?) and auditctl as comms. > >> > >> Since this is not about Gentoo's kernel only, i'll post here the oops in > >> 3.2.9 and also attach some list_add corruptions. > >> > >> 3.2.9 BUG: > >> > >> kernel: [ 301.240011] BUG: unable to handle kernel NULL pointer dereference > >> at (null) > >> kernel: [ 301.240305] IP: [] __list_del_entry+0x20/0xe0 > >> kernel: [ 301.240481] *pdpt = 0000000000000000 *pde = f000ddc8f000ddc8 > >> kernel: [ 301.240698] Oops: 0000 [#1] SMP > >> kernel: [ 301.240910] > >> kernel: [ 301.241030] Pid: 642, comm: fsnotify_mark Not tainted > >> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396 > >> kernel: [ 301.241370] EIP: 0060:[] EFLAGS: 00010287 CPU: 6 > >> kernel: [ 301.241498] EIP is at __list_del_entry+0x20/0xe0 > >> kernel: [ 301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff EDX: > >> 00000000 > >> kernel: [ 301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c ESP: > >> f47cff64 > >> kernel: [ 301.241879] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 > >> kernel: [ 301.242005] Process fsnotify_mark (pid: 642, ti=f47ce000 > >> task=f4f47c00 task.ti=f47ce000) > >> kernel: [ 301.242207] Stack: > >> kernel: [ 301.242327] c10813c0 f47cffa4 f4f47c00 f4e70888 f47cff7c > >> f47cffa4 f47cffb8 c10f6976 > >> kernel: [ 301.242882] ffffffc3 f4f47c00 f4f47c00 00000000 f4f47c00 > >> c10530c0 f47cff9c f47cff9c > >> kernel: [ 301.243438] f4fae544 f4fae544 f4c47f58 00000000 c10f68f0 > >> f47cffe4 c1052834 00000000 > >> kernel: [ 301.243995] Call Trace: > >> kernel: [ 301.244119] [] ? rcu_check_callbacks+0x110/0x110 > >> kernel: [ 301.244248] [] fsnotify_mark_destroy+0x86/0x120 > >> kernel: [ 301.244377] [] ? abort_exclusive_wait+0x80/0x80 > >> kernel: [ 301.244504] [] ? fsnotify_put_mark+0x30/0x30 > >> kernel: [ 301.244631] [] kthread+0x74/0x80 > >> kernel: [ 301.244756] [] ? kthread_flush_work_fn+0x10/0x10 > >> kernel: [ 301.244885] [] kernel_thread_helper+0x6/0xd > >> kernel: [ 301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89 e5 53 83 > >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 0f 84 8e 00 > >> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 83 c4 14 > >> kernel: [ 301.248195] EIP: [] __list_del_entry+0x20/0xe0 SS:ESP > >> 0068:f47cff64 > >> kernel: [ 301.248414] CR2: 0000000000000000 > >> kernel: [ 301.248538] ---[ end trace 15082dbfb353f84c ]--- > >> > >> The kernel was compiled with the following DEBUG support (the bolded one > >> were requested by Gentoo's Dev: > >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y > >> CONFIG_SLUB_DEBUG=y > >> CONFIG_HAVE_DMA_API_DEBUG=y > >> CONFIG_X86_DEBUGCTLMSR=y > >> CONFIG_PNP_DEBUG_MESSAGES=y > >> CONFIG_AIC94XX_DEBUG=y > >> CONFIG_USB_DEBUG=y > >> CONFIG_DEBUG_KERNEL=y > >> CONFIG_SCHED_DEBUG=y > >> CONFIG_DEBUG_RT_MUTEXES=y > >> CONFIG_DEBUG_PI_LIST=y > >> CONFIG_DEBUG_BUGVERBOSE=y > >> CONFIG_DEBUG_INFO=y > >> CONFIG_DEBUG_MEMORY_INIT=y > >> CONFIG_DEBUG_LIST=y > >> CONFIG_DEBUG_STACKOVERFLOW=y > >> CONFIG_DEBUG_RODATA=y > >> CONFIG_DEBUG_RODATA_TEST=y > >> > >> I attached the kernel config i used for 3.2.9 to generate this oops and > >> warnings. > >> > >> From the list_add warnings that come after, out of 805 warnings i processed, > >> after masking with XXXXX the PID and next= values that kept changing in > >> every one, i got 26 types of MD5. I also attached the files relevant as an > >> archive to this email. > >> > >> The Gentoo bug i opened is sleeping, it seems nobody has the time to at > >> least test to confirm or not the problems i'm seeing (or everybody's > >> thinking that nobody would restart auditd so often, so the bug it's not that > >> serious). > >> > >> > >> Thank you for your time. > >> > >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram wrote: > >> > >> > >> -- > >> Linux-audit mailing list > >> Linux-audit@redhat.com > >> https://www.redhat.com/mailman/listinfo/linux-audit > > > > > > > > -- > > Peter Moody Google 1.650.253.7306 > > Security Engineer pgp:0xC3410038 > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Valentin Avram Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Thu, 29 Mar 2012 09:44:56 +0300 Message-ID: References: <1327519203.4131.25.camel@localhost> <1332983643.384.8.camel@localhost> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4201550010719350373==" Return-path: In-Reply-To: <1332983643.384.8.camel@localhost> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Eric Paris Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com --===============4201550010719350373== Content-Type: multipart/alternative; boundary=20cf305e2373e052b704bc5c0f0d --20cf305e2373e052b704bc5c0f0d Content-Type: text/plain; charset=ISO-8859-1 Yes, i know that patch. It made it into kernel 3.2.2. I tested it successfully (oops in 3.2.1, no oops in 3.2.9), but this oops i'm seeing is also in 3.2.9. I monitored changelogs since 3.2.1 to 3.2.12 but there were no fixes either in audit subsystem or in fsnotify. I'll try to reproduce in latest 3.2.13 and repost the oops, but i'm 99% confident it will be the same. Sadly nobody except you seems to pay attention to this problem, probably because it requires special conditions to reproduce (really, who starts and stops auditd every 5 seconds on a production server?). We only ran into it because one of our servers would randomly oops and then freeze about each month after stopping and then starting auditd every morning (and the stop-start sequence was needed to workaround a bug somewhere that would hang a gzip running on a file outside a watched folder). Anyway, as a last note, i have a feeling that the oops is not exactly random, there is a pattern, just that i haven't figured it out completely yet. Will keep you uptodate with the things i find out. V. On Mar 29, 2012 4:14 AM, "Eric Paris" wrote: > That patch fixes a BUG() . The report has a NULL ptr deref and some > apparent list correuption.... Sadly they aren't the same.... > > On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote: > > fyi: this patch [1] seems to fix the issue for me. The explanation in > > the subject would reliably oops my machine. > > > > [1] > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=fed474857efbed79cd390d0aee224231ca718f63 > > > > On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody wrote: > > > Are you still able to reliably reproduce this oops? I'm trying to > > > track this down because this bug (or a very similar bug) is causing > > > some significant headaches here at work, but I haven't had a lot of > > > luck. I'm using usermode linux, though, so that might be interfering > > > with things. > > > > > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram > wrote: > > >> Finally i found some time and spare server to retest the oops and > list_add > > >> corruptions i was getting with the 3.x kernels and auditd 2.1.3. > > >> > > >> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and > kernel.org's > > >> 3.2.9. > > >> > > >> Both get the oops/BUG in the same way and after that, they keep > pouring > > >> list_add corruptions with audit_prune_tre(truncated?) and auditctl as > comms. > > >> > > >> Since this is not about Gentoo's kernel only, i'll post here the oops > in > > >> 3.2.9 and also attach some list_add corruptions. > > >> > > >> 3.2.9 BUG: > > >> > > >> kernel: [ 301.240011] BUG: unable to handle kernel NULL pointer > dereference > > >> at (null) > > >> kernel: [ 301.240305] IP: [] __list_del_entry+0x20/0xe0 > > >> kernel: [ 301.240481] *pdpt = 0000000000000000 *pde = > f000ddc8f000ddc8 > > >> kernel: [ 301.240698] Oops: 0000 [#1] SMP > > >> kernel: [ 301.240910] > > >> kernel: [ 301.241030] Pid: 642, comm: fsnotify_mark Not tainted > > >> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396 > > >> kernel: [ 301.241370] EIP: 0060:[] EFLAGS: 00010287 CPU: 6 > > >> kernel: [ 301.241498] EIP is at __list_del_entry+0x20/0xe0 > > >> kernel: [ 301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff EDX: > > >> 00000000 > > >> kernel: [ 301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c ESP: > > >> f47cff64 > > >> kernel: [ 301.241879] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 > > >> kernel: [ 301.242005] Process fsnotify_mark (pid: 642, ti=f47ce000 > > >> task=f4f47c00 task.ti=f47ce000) > > >> kernel: [ 301.242207] Stack: > > >> kernel: [ 301.242327] c10813c0 f47cffa4 f4f47c00 f4e70888 f47cff7c > > >> f47cffa4 f47cffb8 c10f6976 > > >> kernel: [ 301.242882] ffffffc3 f4f47c00 f4f47c00 00000000 f4f47c00 > > >> c10530c0 f47cff9c f47cff9c > > >> kernel: [ 301.243438] f4fae544 f4fae544 f4c47f58 00000000 c10f68f0 > > >> f47cffe4 c1052834 00000000 > > >> kernel: [ 301.243995] Call Trace: > > >> kernel: [ 301.244119] [] ? rcu_check_callbacks+0x110/0x110 > > >> kernel: [ 301.244248] [] fsnotify_mark_destroy+0x86/0x120 > > >> kernel: [ 301.244377] [] ? abort_exclusive_wait+0x80/0x80 > > >> kernel: [ 301.244504] [] ? fsnotify_put_mark+0x30/0x30 > > >> kernel: [ 301.244631] [] kthread+0x74/0x80 > > >> kernel: [ 301.244756] [] ? kthread_flush_work_fn+0x10/0x10 > > >> kernel: [ 301.244885] [] kernel_thread_helper+0x6/0xd > > >> kernel: [ 301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89 > e5 53 83 > > >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 0f 84 > 8e 00 > > >> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 83 c4 14 > > >> kernel: [ 301.248195] EIP: [] __list_del_entry+0x20/0xe0 > SS:ESP > > >> 0068:f47cff64 > > >> kernel: [ 301.248414] CR2: 0000000000000000 > > >> kernel: [ 301.248538] ---[ end trace 15082dbfb353f84c ]--- > > >> > > >> The kernel was compiled with the following DEBUG support (the bolded > one > > >> were requested by Gentoo's Dev: > > >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y > > >> CONFIG_SLUB_DEBUG=y > > >> CONFIG_HAVE_DMA_API_DEBUG=y > > >> CONFIG_X86_DEBUGCTLMSR=y > > >> CONFIG_PNP_DEBUG_MESSAGES=y > > >> CONFIG_AIC94XX_DEBUG=y > > >> CONFIG_USB_DEBUG=y > > >> CONFIG_DEBUG_KERNEL=y > > >> CONFIG_SCHED_DEBUG=y > > >> CONFIG_DEBUG_RT_MUTEXES=y > > >> CONFIG_DEBUG_PI_LIST=y > > >> CONFIG_DEBUG_BUGVERBOSE=y > > >> CONFIG_DEBUG_INFO=y > > >> CONFIG_DEBUG_MEMORY_INIT=y > > >> CONFIG_DEBUG_LIST=y > > >> CONFIG_DEBUG_STACKOVERFLOW=y > > >> CONFIG_DEBUG_RODATA=y > > >> CONFIG_DEBUG_RODATA_TEST=y > > >> > > >> I attached the kernel config i used for 3.2.9 to generate this oops > and > > >> warnings. > > >> > > >> From the list_add warnings that come after, out of 805 warnings i > processed, > > >> after masking with XXXXX the PID and next= values that kept changing > in > > >> every one, i got 26 types of MD5. I also attached the files relevant > as an > > >> archive to this email. > > >> > > >> The Gentoo bug i opened is sleeping, it seems nobody has the time to > at > > >> least test to confirm or not the problems i'm seeing (or everybody's > > >> thinking that nobody would restart auditd so often, so the bug it's > not that > > >> serious). > > >> > > >> > > >> Thank you for your time. > > >> > > >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram > wrote: > > >> > > >> > > >> -- > > >> Linux-audit mailing list > > >> Linux-audit@redhat.com > > >> https://www.redhat.com/mailman/listinfo/linux-audit > > > > > > > > > > > > -- > > > Peter Moody Google 1.650.253.7306 > > > Security Engineer pgp:0xC3410038 > > > > > > > > > --20cf305e2373e052b704bc5c0f0d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Yes, i know that patch. It made it into kernel 3.2.2. I tested it succes= sfully (oops in 3.2.1, no oops in 3.2.9), but this oops i'm seeing is a= lso in 3.2.9.

I monitored changelogs since 3.2.1 to 3.2.12 but there were no fixes eit= her in audit subsystem or in fsnotify. I'll try to reproduce in latest = 3.2.13 and repost the oops, but i'm 99% confident it will be the same.<= /p>

Sadly nobody except you seems to pay attention to this problem, probably= because it requires special conditions to reproduce (really, who starts an= d stops auditd every 5 seconds on a production server?). We only ran into i= t because one of our servers would randomly oops and then freeze about each= month after stopping and then starting

auditd

every morning (and the stop-start sequence was needed to workaround a b= ug somewhere that would hang a

gzip

running on a file outside a watched folder).

Anyway, as a last note, i have a feeling that the oops is not exactly ra= ndom, there is a pattern, just that i haven't figured it out completely= yet.

Will keep you

uptodate

with the things i find out.

V.

On Mar 29, 2012 4:14 AM, "Eric Paris" = <eparis@redhat.com> wrote:
That patch fixes a BUG() . =A0The report has a NULL ptr deref and some
apparent list correuption.... =A0Sadly they aren't the same....

On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote:
> fyi: this patch [1] seems to fix the issue for me. The explanation in<= br> > the subject would reliably oops my machine.
>
> [1] http://git.kernel.org/?p=3Dlinux/kernel/git/torvalds/linux-2.6.= git;a=3Dcommit;h=3Dfed474857efbed79cd390d0aee224231ca718f63
>
> On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody <pmoody@google.com> wrote:
> > Are you still able to reliably reproduce this oops? I'm tryin= g to
> > track this down because this bug (or a very similar bug) is causi= ng
> > some significant headaches here at work, but I haven't had a = lot of
> > luck. I'm using usermode linux, though, so that might be inte= rfering
> > with things.
> >
> > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram <aval13@gmail.com> wrote:
> >> Finally i found some time and spare server to retest the oops= and list_add
> >> corruptions i was getting with the 3.x kernels and auditd 2.1= .3.
> >>
> >> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 = and kernel.org's > >> 3.2.9.
> >>
> >> Both get the oops/BUG in the same way and after that, they ke= ep pouring
> >> list_add corruptions with audit_prune_tre(truncated?) and aud= itctl as comms.
> >>
> >> Since this is not about Gentoo's kernel only, i'll po= st here the oops in
> >> 3.2.9 and also attach some list_add corruptions.
> >>
> >> 3.2.9 BUG:
> >>
> >> kernel: [ =A0301.240011] BUG: unable to handle kernel NULL po= inter dereference
> >> at =A0 (null)
> >> kernel: [ =A0301.240305] IP: [<c1238dd0>] __list_del_en= try+0x20/0xe0
> >> kernel: [ =A0301.240481] *pdpt =3D 0000000000000000 *pde =3D = f000ddc8f000ddc8
> >> kernel: [ =A0301.240698] Oops: 0000 [#1] SMP
> >> kernel: [ =A0301.240910]
> >> kernel: [ =A0301.241030] Pid: 642, comm: fsnotify_mark Not ta= inted
> >> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396
> >> kernel: [ =A0301.241370] EIP: 0060:[<c1238dd0>] EFLAGS:= 00010287 CPU: 6
> >> kernel: [ =A0301.241498] EIP is at __list_del_entry+0x20/0xe0=
> >> kernel: [ =A0301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: fff= fffff EDX:
> >> 00000000
> >> kernel: [ =A0301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47= cff7c ESP:
> >> f47cff64
> >> kernel: [ =A0301.241879] =A0DS: 007b ES: 007b FS: 00d8 GS: 00= 00 SS: 0068
> >> kernel: [ =A0301.242005] Process fsnotify_mark (pid: 642, ti= =3Df47ce000
> >> task=3Df4f47c00 task.ti=3Df47ce000)
> >> kernel: [ =A0301.242207] Stack:
> >> kernel: [ =A0301.242327] =A0c10813c0 f47cffa4 f4f47c00 f4e708= 88 f47cff7c
> >> f47cffa4 f47cffb8 c10f6976
> >> kernel: [ =A0301.242882] =A0ffffffc3 f4f47c00 f4f47c00 000000= 00 f4f47c00
> >> c10530c0 f47cff9c f47cff9c
> >> kernel: [ =A0301.243438] =A0f4fae544 f4fae544 f4c47f58 000000= 00 c10f68f0
> >> f47cffe4 c1052834 00000000
> >> kernel: [ =A0301.243995] Call Trace:
> >> kernel: [ =A0301.244119] =A0[<c10813c0>] ? rcu_check_ca= llbacks+0x110/0x110
> >> kernel: [ =A0301.244248] =A0[<c10f6976>] fsnotify_mark_= destroy+0x86/0x120
> >> kernel: [ =A0301.244377] =A0[<c10530c0>] ? abort_exclus= ive_wait+0x80/0x80
> >> kernel: [ =A0301.244504] =A0[<c10f68f0>] ? fsnotify_put= _mark+0x30/0x30
> >> kernel: [ =A0301.244631] =A0[<c1052834>] kthread+0x74/0= x80
> >> kernel: [ =A0301.244756] =A0[<c10527c0>] ? kthread_flus= h_work_fn+0x10/0x10
> >> kernel: [ =A0301.244885] =A0[<c1582ab6>] kernel_thread_= helper+0x6/0xd
> >> kernel: [ =A0301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff = 90 55 89 e5 53 83
> >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 0= 0 0f 84 8e 00
> >> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04= 89 0a 83 c4 14
> >> kernel: [ =A0301.248195] EIP: [<c1238dd0>] __list_del_e= ntry+0x20/0xe0 SS:ESP
> >> 0068:f47cff64
> >> kernel: [ =A0301.248414] CR2: 0000000000000000
> >> kernel: [ =A0301.248538] ---[ end trace 15082dbfb353f84c ]---=
> >>
> >> The kernel was compiled with the following DEBUG support (the= bolded one
> >> were requested by Gentoo's Dev:
> >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=3Dy
> >> CONFIG_SLUB_DEBUG=3Dy
> >> CONFIG_HAVE_DMA_API_DEBUG=3Dy
> >> CONFIG_X86_DEBUGCTLMSR=3Dy
> >> CONFIG_PNP_DEBUG_MESSAGES=3Dy
> >> CONFIG_AIC94XX_DEBUG=3Dy
> >> CONFIG_USB_DEBUG=3Dy
> >> CONFIG_DEBUG_KERNEL=3Dy
> >> CONFIG_SCHED_DEBUG=3Dy
> >> CONFIG_DEBUG_RT_MUTEXES=3Dy
> >> CONFIG_DEBUG_PI_LIST=3Dy
> >> CONFIG_DEBUG_BUGVERBOSE=3Dy
> >> CONFIG_DEBUG_INFO=3Dy
> >> CONFIG_DEBUG_MEMORY_INIT=3Dy
> >> CONFIG_DEBUG_LIST=3Dy
> >> CONFIG_DEBUG_STACKOVERFLOW=3Dy
> >> CONFIG_DEBUG_RODATA=3Dy
> >> CONFIG_DEBUG_RODATA_TEST=3Dy
> >>
> >> I attached the kernel config i used for 3.2.9 to generate thi= s oops and
> >> warnings.
> >>
> >> From the list_add warnings that come after, out of 805 warnin= gs i processed,
> >> after masking with XXXXX the PID and next=3D values that kept= changing in
> >> every one, i got 26 types of MD5. I also attached the files r= elevant as an
> >> archive to this email.
> >>
> >> The Gentoo bug i opened is sleeping, it seems nobody has the = time to at
> >> least test to confirm or not the problems i'm seeing (or = everybody's
> >> thinking that nobody would restart auditd so often, so the bu= g it's not that
> >> serious).
> >>
> >>
> >> Thank you for your time.
> >>
> >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram <aval13@gmail.com> wrote:
> >>
> >>
> >> --
> >> Linux-audit mailing list
> >> Linux-audit@redhat.= com
> >> https://www.redhat.com/mailman/listinfo/linux-audit
> >
> >
> >
> > --
> > Peter Moody =A0 =A0 =A0Google =A0 =A01.650.253.7306
> > Security Engineer =A0pgp:0xC3410038
>
>
>


--20cf305e2373e052b704bc5c0f0d-- --===============4201550010719350373== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============4201550010719350373==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter Moody Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Tue, 3 Apr 2012 09:15:37 -0700 Message-ID: References: <1327519203.4131.25.camel@localhost> <1332983643.384.8.camel@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Received: from mx1.redhat.com (ext-mx14.extmail.prod.ext.phx2.redhat.com [10.5.110.19]) by int-mx01.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id q33GG9aN019464 for ; Tue, 3 Apr 2012 12:16:10 -0400 Received: from mail-yw0-f46.google.com (mail-yw0-f46.google.com [209.85.213.46]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q33GG8ru021876 for ; Tue, 3 Apr 2012 12:16:08 -0400 Received: by yhmm54 with SMTP id m54so2367313yhm.33 for ; Tue, 03 Apr 2012 09:16:08 -0700 (PDT) In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Valentin Avram Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com This may already be known, but the issue seems to be limited to watch rules. With any watch rules, I can reliably crash my machine while freeing a watch rule after only starting/stopping auditd a few times. With no watch rules, I have no issues. Cheers, peter On Wed, Mar 28, 2012 at 11:44 PM, Valentin Avram wrote: > Yes, i know that patch. It made it into kernel 3.2.2. I tested it > successfully (oops in 3.2.1, no oops in 3.2.9), but this oops i'm seeing = is > also in 3.2.9. > > I monitored changelogs since 3.2.1 to 3.2.12 but there were no fixes eith= er > in audit subsystem or in fsnotify. I'll try to reproduce in latest 3.2.13 > and repost the oops, but i'm 99% confident it will be the same. > > Sadly nobody except you seems to pay attention to this problem, probably > because it requires special conditions to reproduce (really, who starts a= nd > stops auditd every 5 seconds on a production server?). We only ran into it > because one of our servers would randomly oops and then freeze about each > month after stopping and then starting > > auditd > > every morning (and the stop-start sequence was needed to workaround a bug > somewhere that would hang a > > gzip > > running on a file outside a watched folder). > > Anyway, as a last note, i have a feeling that the oops is not exactly > random, there is a pattern, just that i haven't figured it out completely > yet. > > Will keep you > > uptodate > > with the things i find out. > > V. > > On Mar 29, 2012 4:14 AM, "Eric Paris" wrote: >> >> That patch fixes a BUG() . =A0The report has a NULL ptr deref and some >> apparent list correuption.... =A0Sadly they aren't the same.... >> >> On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote: >> > fyi: this patch [1] seems to fix the issue for me. The explanation in >> > the subject would reliably oops my machine. >> > >> > [1] >> > http://git.kernel.org/?p=3Dlinux/kernel/git/torvalds/linux-2.6.git;a= =3Dcommit;h=3Dfed474857efbed79cd390d0aee224231ca718f63 >> > >> > On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody wrote: >> > > Are you still able to reliably reproduce this oops? I'm trying to >> > > track this down because this bug (or a very similar bug) is causing >> > > some significant headaches here at work, but I haven't had a lot of >> > > luck. I'm using usermode linux, though, so that might be interfering >> > > with things. >> > > >> > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram >> > > wrote: >> > >> Finally i found some time and spare server to retest the oops and >> > >> list_add >> > >> corruptions i was getting with the 3.x kernels and auditd 2.1.3. >> > >> >> > >> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and >> > >> kernel.org's >> > >> 3.2.9. >> > >> >> > >> Both get the oops/BUG in the same way and after that, they keep >> > >> pouring >> > >> list_add corruptions with audit_prune_tre(truncated?) and auditctl = as >> > >> comms. >> > >> >> > >> Since this is not about Gentoo's kernel only, i'll post here the oo= ps >> > >> in >> > >> 3.2.9 and also attach some list_add corruptions. >> > >> >> > >> 3.2.9 BUG: >> > >> >> > >> kernel: [ =A0301.240011] BUG: unable to handle kernel NULL pointer >> > >> dereference >> > >> at =A0 (null) >> > >> kernel: [ =A0301.240305] IP: [] __list_del_entry+0x20/0xe0 >> > >> kernel: [ =A0301.240481] *pdpt =3D 0000000000000000 *pde =3D >> > >> f000ddc8f000ddc8 >> > >> kernel: [ =A0301.240698] Oops: 0000 [#1] SMP >> > >> kernel: [ =A0301.240910] >> > >> kernel: [ =A0301.241030] Pid: 642, comm: fsnotify_mark Not tainted >> > >> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396 >> > >> kernel: [ =A0301.241370] EIP: 0060:[] EFLAGS: 00010287 CP= U: 6 >> > >> kernel: [ =A0301.241498] EIP is at __list_del_entry+0x20/0xe0 >> > >> kernel: [ =A0301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff = EDX: >> > >> 00000000 >> > >> kernel: [ =A0301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c = ESP: >> > >> f47cff64 >> > >> kernel: [ =A0301.241879] =A0DS: 007b ES: 007b FS: 00d8 GS: 0000 SS:= 0068 >> > >> kernel: [ =A0301.242005] Process fsnotify_mark (pid: 642, ti=3Df47c= e000 >> > >> task=3Df4f47c00 task.ti=3Df47ce000) >> > >> kernel: [ =A0301.242207] Stack: >> > >> kernel: [ =A0301.242327] =A0c10813c0 f47cffa4 f4f47c00 f4e70888 f47= cff7c >> > >> f47cffa4 f47cffb8 c10f6976 >> > >> kernel: [ =A0301.242882] =A0ffffffc3 f4f47c00 f4f47c00 00000000 f4f= 47c00 >> > >> c10530c0 f47cff9c f47cff9c >> > >> kernel: [ =A0301.243438] =A0f4fae544 f4fae544 f4c47f58 00000000 c10= f68f0 >> > >> f47cffe4 c1052834 00000000 >> > >> kernel: [ =A0301.243995] Call Trace: >> > >> kernel: [ =A0301.244119] =A0[] ? >> > >> rcu_check_callbacks+0x110/0x110 >> > >> kernel: [ =A0301.244248] =A0[] fsnotify_mark_destroy+0x86= /0x120 >> > >> kernel: [ =A0301.244377] =A0[] ? abort_exclusive_wait+0x8= 0/0x80 >> > >> kernel: [ =A0301.244504] =A0[] ? fsnotify_put_mark+0x30/0= x30 >> > >> kernel: [ =A0301.244631] =A0[] kthread+0x74/0x80 >> > >> kernel: [ =A0301.244756] =A0[] ? >> > >> kthread_flush_work_fn+0x10/0x10 >> > >> kernel: [ =A0301.244885] =A0[] kernel_thread_helper+0x6/0= xd >> > >> kernel: [ =A0301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 = 89 >> > >> e5 53 83 >> > >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 0f 84 >> > >> 8e 00 >> > >> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 83 c4 >> > >> 14 >> > >> kernel: [ =A0301.248195] EIP: [] __list_del_entry+0x20/0x= e0 >> > >> SS:ESP >> > >> 0068:f47cff64 >> > >> kernel: [ =A0301.248414] CR2: 0000000000000000 >> > >> kernel: [ =A0301.248538] ---[ end trace 15082dbfb353f84c ]--- >> > >> >> > >> The kernel was compiled with the following DEBUG support (the bolded >> > >> one >> > >> were requested by Gentoo's Dev: >> > >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=3Dy >> > >> CONFIG_SLUB_DEBUG=3Dy >> > >> CONFIG_HAVE_DMA_API_DEBUG=3Dy >> > >> CONFIG_X86_DEBUGCTLMSR=3Dy >> > >> CONFIG_PNP_DEBUG_MESSAGES=3Dy >> > >> CONFIG_AIC94XX_DEBUG=3Dy >> > >> CONFIG_USB_DEBUG=3Dy >> > >> CONFIG_DEBUG_KERNEL=3Dy >> > >> CONFIG_SCHED_DEBUG=3Dy >> > >> CONFIG_DEBUG_RT_MUTEXES=3Dy >> > >> CONFIG_DEBUG_PI_LIST=3Dy >> > >> CONFIG_DEBUG_BUGVERBOSE=3Dy >> > >> CONFIG_DEBUG_INFO=3Dy >> > >> CONFIG_DEBUG_MEMORY_INIT=3Dy >> > >> CONFIG_DEBUG_LIST=3Dy >> > >> CONFIG_DEBUG_STACKOVERFLOW=3Dy >> > >> CONFIG_DEBUG_RODATA=3Dy >> > >> CONFIG_DEBUG_RODATA_TEST=3Dy >> > >> >> > >> I attached the kernel config i used for 3.2.9 to generate this oops >> > >> and >> > >> warnings. >> > >> >> > >> From the list_add warnings that come after, out of 805 warnings i >> > >> processed, >> > >> after masking with XXXXX the PID and next=3D values that kept chang= ing >> > >> in >> > >> every one, i got 26 types of MD5. I also attached the files relevant >> > >> as an >> > >> archive to this email. >> > >> >> > >> The Gentoo bug i opened is sleeping, it seems nobody has the time to >> > >> at >> > >> least test to confirm or not the problems i'm seeing (or everybody's >> > >> thinking that nobody would restart auditd so often, so the bug it's >> > >> not that >> > >> serious). >> > >> >> > >> >> > >> Thank you for your time. >> > >> >> > >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram >> > >> wrote: >> > >> >> > >> >> > >> -- >> > >> Linux-audit mailing list >> > >> Linux-audit@redhat.com >> > >> https://www.redhat.com/mailman/listinfo/linux-audit >> > > >> > > >> > > >> > > -- >> > > Peter Moody =A0 =A0 =A0Google =A0 =A01.650.253.7306 >> > > Security Engineer =A0pgp:0xC3410038 >> > >> > >> > >> >> > -- = Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 Security Engineer=A0 pgp:0xC3410038 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter Moody Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Thu, 5 Apr 2012 14:03:57 -0700 Message-ID: References: <1327519203.4131.25.camel@localhost> <1332983643.384.8.camel@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Received: from mx1.redhat.com (ext-mx13.extmail.prod.ext.phx2.redhat.com [10.5.110.18]) by int-mx12.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id q35L4Umh013054 for ; Thu, 5 Apr 2012 17:04:30 -0400 Received: from mail-iy0-f174.google.com (mail-iy0-f174.google.com [209.85.210.174]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q35L4Sil009721 for ; Thu, 5 Apr 2012 17:04:28 -0400 Received: by iagz16 with SMTP id z16so3083881iag.33 for ; Thu, 05 Apr 2012 14:04:28 -0700 (PDT) In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Valentin Avram Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com (please let me know if I should take this off-list) One other thing (again, maybe already known), but this seems to be exacerbated by SMP. On my machine, I can't reproduce the crash if I booth with maxcpus=3D1. Still hunting. Cheers, peter On Tue, Apr 3, 2012 at 9:15 AM, Peter Moody wrote: > This may already be known, but the issue seems to be limited to watch > rules. With any watch rules, I can reliably crash my machine while > freeing a watch rule after only starting/stopping auditd a few times. > With no watch rules, I have no issues. > > Cheers, > peter > > On Wed, Mar 28, 2012 at 11:44 PM, Valentin Avram wrote: >> Yes, i know that patch. It made it into kernel 3.2.2. I tested it >> successfully (oops in 3.2.1, no oops in 3.2.9), but this oops i'm seeing= is >> also in 3.2.9. >> >> I monitored changelogs since 3.2.1 to 3.2.12 but there were no fixes eit= her >> in audit subsystem or in fsnotify. I'll try to reproduce in latest 3.2.13 >> and repost the oops, but i'm 99% confident it will be the same. >> >> Sadly nobody except you seems to pay attention to this problem, probably >> because it requires special conditions to reproduce (really, who starts = and >> stops auditd every 5 seconds on a production server?). We only ran into = it >> because one of our servers would randomly oops and then freeze about each >> month after stopping and then starting >> >> auditd >> >> every morning (and the stop-start sequence was needed to workaround a bug >> somewhere that would hang a >> >> gzip >> >> running on a file outside a watched folder). >> >> Anyway, as a last note, i have a feeling that the oops is not exactly >> random, there is a pattern, just that i haven't figured it out completely >> yet. >> >> Will keep you >> >> uptodate >> >> with the things i find out. >> >> V. >> >> On Mar 29, 2012 4:14 AM, "Eric Paris" wrote: >>> >>> That patch fixes a BUG() . =A0The report has a NULL ptr deref and some >>> apparent list correuption.... =A0Sadly they aren't the same.... >>> >>> On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote: >>> > fyi: this patch [1] seems to fix the issue for me. The explanation in >>> > the subject would reliably oops my machine. >>> > >>> > [1] >>> > http://git.kernel.org/?p=3Dlinux/kernel/git/torvalds/linux-2.6.git;a= =3Dcommit;h=3Dfed474857efbed79cd390d0aee224231ca718f63 >>> > >>> > On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody wrot= e: >>> > > Are you still able to reliably reproduce this oops? I'm trying to >>> > > track this down because this bug (or a very similar bug) is causing >>> > > some significant headaches here at work, but I haven't had a lot of >>> > > luck. I'm using usermode linux, though, so that might be interfering >>> > > with things. >>> > > >>> > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram >>> > > wrote: >>> > >> Finally i found some time and spare server to retest the oops and >>> > >> list_add >>> > >> corruptions i was getting with the 3.x kernels and auditd 2.1.3. >>> > >> >>> > >> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and >>> > >> kernel.org's >>> > >> 3.2.9. >>> > >> >>> > >> Both get the oops/BUG in the same way and after that, they keep >>> > >> pouring >>> > >> list_add corruptions with audit_prune_tre(truncated?) and auditctl= as >>> > >> comms. >>> > >> >>> > >> Since this is not about Gentoo's kernel only, i'll post here the o= ops >>> > >> in >>> > >> 3.2.9 and also attach some list_add corruptions. >>> > >> >>> > >> 3.2.9 BUG: >>> > >> >>> > >> kernel: [ =A0301.240011] BUG: unable to handle kernel NULL pointer >>> > >> dereference >>> > >> at =A0 (null) >>> > >> kernel: [ =A0301.240305] IP: [] __list_del_entry+0x20/0x= e0 >>> > >> kernel: [ =A0301.240481] *pdpt =3D 0000000000000000 *pde =3D >>> > >> f000ddc8f000ddc8 >>> > >> kernel: [ =A0301.240698] Oops: 0000 [#1] SMP >>> > >> kernel: [ =A0301.240910] >>> > >> kernel: [ =A0301.241030] Pid: 642, comm: fsnotify_mark Not tainted >>> > >> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396 >>> > >> kernel: [ =A0301.241370] EIP: 0060:[] EFLAGS: 00010287 C= PU: 6 >>> > >> kernel: [ =A0301.241498] EIP is at __list_del_entry+0x20/0xe0 >>> > >> kernel: [ =A0301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff= EDX: >>> > >> 00000000 >>> > >> kernel: [ =A0301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c= ESP: >>> > >> f47cff64 >>> > >> kernel: [ =A0301.241879] =A0DS: 007b ES: 007b FS: 00d8 GS: 0000 SS= : 0068 >>> > >> kernel: [ =A0301.242005] Process fsnotify_mark (pid: 642, ti=3Df47= ce000 >>> > >> task=3Df4f47c00 task.ti=3Df47ce000) >>> > >> kernel: [ =A0301.242207] Stack: >>> > >> kernel: [ =A0301.242327] =A0c10813c0 f47cffa4 f4f47c00 f4e70888 f4= 7cff7c >>> > >> f47cffa4 f47cffb8 c10f6976 >>> > >> kernel: [ =A0301.242882] =A0ffffffc3 f4f47c00 f4f47c00 00000000 f4= f47c00 >>> > >> c10530c0 f47cff9c f47cff9c >>> > >> kernel: [ =A0301.243438] =A0f4fae544 f4fae544 f4c47f58 00000000 c1= 0f68f0 >>> > >> f47cffe4 c1052834 00000000 >>> > >> kernel: [ =A0301.243995] Call Trace: >>> > >> kernel: [ =A0301.244119] =A0[] ? >>> > >> rcu_check_callbacks+0x110/0x110 >>> > >> kernel: [ =A0301.244248] =A0[] fsnotify_mark_destroy+0x8= 6/0x120 >>> > >> kernel: [ =A0301.244377] =A0[] ? abort_exclusive_wait+0x= 80/0x80 >>> > >> kernel: [ =A0301.244504] =A0[] ? fsnotify_put_mark+0x30/= 0x30 >>> > >> kernel: [ =A0301.244631] =A0[] kthread+0x74/0x80 >>> > >> kernel: [ =A0301.244756] =A0[] ? >>> > >> kthread_flush_work_fn+0x10/0x10 >>> > >> kernel: [ =A0301.244885] =A0[] kernel_thread_helper+0x6/= 0xd >>> > >> kernel: [ =A0301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55= 89 >>> > >> e5 53 83 >>> > >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 0f = 84 >>> > >> 8e 00 >>> > >> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 83 c4 >>> > >> 14 >>> > >> kernel: [ =A0301.248195] EIP: [] __list_del_entry+0x20/0= xe0 >>> > >> SS:ESP >>> > >> 0068:f47cff64 >>> > >> kernel: [ =A0301.248414] CR2: 0000000000000000 >>> > >> kernel: [ =A0301.248538] ---[ end trace 15082dbfb353f84c ]--- >>> > >> >>> > >> The kernel was compiled with the following DEBUG support (the bold= ed >>> > >> one >>> > >> were requested by Gentoo's Dev: >>> > >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=3Dy >>> > >> CONFIG_SLUB_DEBUG=3Dy >>> > >> CONFIG_HAVE_DMA_API_DEBUG=3Dy >>> > >> CONFIG_X86_DEBUGCTLMSR=3Dy >>> > >> CONFIG_PNP_DEBUG_MESSAGES=3Dy >>> > >> CONFIG_AIC94XX_DEBUG=3Dy >>> > >> CONFIG_USB_DEBUG=3Dy >>> > >> CONFIG_DEBUG_KERNEL=3Dy >>> > >> CONFIG_SCHED_DEBUG=3Dy >>> > >> CONFIG_DEBUG_RT_MUTEXES=3Dy >>> > >> CONFIG_DEBUG_PI_LIST=3Dy >>> > >> CONFIG_DEBUG_BUGVERBOSE=3Dy >>> > >> CONFIG_DEBUG_INFO=3Dy >>> > >> CONFIG_DEBUG_MEMORY_INIT=3Dy >>> > >> CONFIG_DEBUG_LIST=3Dy >>> > >> CONFIG_DEBUG_STACKOVERFLOW=3Dy >>> > >> CONFIG_DEBUG_RODATA=3Dy >>> > >> CONFIG_DEBUG_RODATA_TEST=3Dy >>> > >> >>> > >> I attached the kernel config i used for 3.2.9 to generate this oops >>> > >> and >>> > >> warnings. >>> > >> >>> > >> From the list_add warnings that come after, out of 805 warnings i >>> > >> processed, >>> > >> after masking with XXXXX the PID and next=3D values that kept chan= ging >>> > >> in >>> > >> every one, i got 26 types of MD5. I also attached the files releva= nt >>> > >> as an >>> > >> archive to this email. >>> > >> >>> > >> The Gentoo bug i opened is sleeping, it seems nobody has the time = to >>> > >> at >>> > >> least test to confirm or not the problems i'm seeing (or everybody= 's >>> > >> thinking that nobody would restart auditd so often, so the bug it's >>> > >> not that >>> > >> serious). >>> > >> >>> > >> >>> > >> Thank you for your time. >>> > >> >>> > >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram >>> > >> wrote: >>> > >> >>> > >> >>> > >> -- >>> > >> Linux-audit mailing list >>> > >> Linux-audit@redhat.com >>> > >> https://www.redhat.com/mailman/listinfo/linux-audit >>> > > >>> > > >>> > > >>> > > -- >>> > > Peter Moody =A0 =A0 =A0Google =A0 =A01.650.253.7306 >>> > > Security Engineer =A0pgp:0xC3410038 >>> > >>> > >>> > >>> >>> >> > > > > -- > Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 > Security Engineer=A0 pgp:0xC3410038 -- = Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 Security Engineer=A0 pgp:0xC3410038 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Paris Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Thu, 05 Apr 2012 17:07:01 -0400 Message-ID: <1333660021.2273.0.camel@localhost> References: <1327519203.4131.25.camel@localhost> <1332983643.384.8.camel@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Peter Moody Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com please please please keep on list. Everything you say might help track it down! On Thu, 2012-04-05 at 14:03 -0700, Peter Moody wrote: > (please let me know if I should take this off-list) > > One other thing (again, maybe already known), but this seems to be > exacerbated by SMP. On my machine, I can't reproduce the crash if I > booth with maxcpus=1. > > Still hunting. > > Cheers, > peter > > On Tue, Apr 3, 2012 at 9:15 AM, Peter Moody wrote: > > This may already be known, but the issue seems to be limited to watch > > rules. With any watch rules, I can reliably crash my machine while > > freeing a watch rule after only starting/stopping auditd a few times. > > With no watch rules, I have no issues. > > > > Cheers, > > peter > > > > On Wed, Mar 28, 2012 at 11:44 PM, Valentin Avram wrote: > >> Yes, i know that patch. It made it into kernel 3.2.2. I tested it > >> successfully (oops in 3.2.1, no oops in 3.2.9), but this oops i'm seeing is > >> also in 3.2.9. > >> > >> I monitored changelogs since 3.2.1 to 3.2.12 but there were no fixes either > >> in audit subsystem or in fsnotify. I'll try to reproduce in latest 3.2.13 > >> and repost the oops, but i'm 99% confident it will be the same. > >> > >> Sadly nobody except you seems to pay attention to this problem, probably > >> because it requires special conditions to reproduce (really, who starts and > >> stops auditd every 5 seconds on a production server?). We only ran into it > >> because one of our servers would randomly oops and then freeze about each > >> month after stopping and then starting > >> > >> auditd > >> > >> every morning (and the stop-start sequence was needed to workaround a bug > >> somewhere that would hang a > >> > >> gzip > >> > >> running on a file outside a watched folder). > >> > >> Anyway, as a last note, i have a feeling that the oops is not exactly > >> random, there is a pattern, just that i haven't figured it out completely > >> yet. > >> > >> Will keep you > >> > >> uptodate > >> > >> with the things i find out. > >> > >> V. > >> > >> On Mar 29, 2012 4:14 AM, "Eric Paris" wrote: > >>> > >>> That patch fixes a BUG() . The report has a NULL ptr deref and some > >>> apparent list correuption.... Sadly they aren't the same.... > >>> > >>> On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote: > >>> > fyi: this patch [1] seems to fix the issue for me. The explanation in > >>> > the subject would reliably oops my machine. > >>> > > >>> > [1] > >>> > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=fed474857efbed79cd390d0aee224231ca718f63 > >>> > > >>> > On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody wrote: > >>> > > Are you still able to reliably reproduce this oops? I'm trying to > >>> > > track this down because this bug (or a very similar bug) is causing > >>> > > some significant headaches here at work, but I haven't had a lot of > >>> > > luck. I'm using usermode linux, though, so that might be interfering > >>> > > with things. > >>> > > > >>> > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram > >>> > > wrote: > >>> > >> Finally i found some time and spare server to retest the oops and > >>> > >> list_add > >>> > >> corruptions i was getting with the 3.x kernels and auditd 2.1.3. > >>> > >> > >>> > >> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and > >>> > >> kernel.org's > >>> > >> 3.2.9. > >>> > >> > >>> > >> Both get the oops/BUG in the same way and after that, they keep > >>> > >> pouring > >>> > >> list_add corruptions with audit_prune_tre(truncated?) and auditctl as > >>> > >> comms. > >>> > >> > >>> > >> Since this is not about Gentoo's kernel only, i'll post here the oops > >>> > >> in > >>> > >> 3.2.9 and also attach some list_add corruptions. > >>> > >> > >>> > >> 3.2.9 BUG: > >>> > >> > >>> > >> kernel: [ 301.240011] BUG: unable to handle kernel NULL pointer > >>> > >> dereference > >>> > >> at (null) > >>> > >> kernel: [ 301.240305] IP: [] __list_del_entry+0x20/0xe0 > >>> > >> kernel: [ 301.240481] *pdpt = 0000000000000000 *pde = > >>> > >> f000ddc8f000ddc8 > >>> > >> kernel: [ 301.240698] Oops: 0000 [#1] SMP > >>> > >> kernel: [ 301.240910] > >>> > >> kernel: [ 301.241030] Pid: 642, comm: fsnotify_mark Not tainted > >>> > >> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396 > >>> > >> kernel: [ 301.241370] EIP: 0060:[] EFLAGS: 00010287 CPU: 6 > >>> > >> kernel: [ 301.241498] EIP is at __list_del_entry+0x20/0xe0 > >>> > >> kernel: [ 301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffffffff EDX: > >>> > >> 00000000 > >>> > >> kernel: [ 301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cff7c ESP: > >>> > >> f47cff64 > >>> > >> kernel: [ 301.241879] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 > >>> > >> kernel: [ 301.242005] Process fsnotify_mark (pid: 642, ti=f47ce000 > >>> > >> task=f4f47c00 task.ti=f47ce000) > >>> > >> kernel: [ 301.242207] Stack: > >>> > >> kernel: [ 301.242327] c10813c0 f47cffa4 f4f47c00 f4e70888 f47cff7c > >>> > >> f47cffa4 f47cffb8 c10f6976 > >>> > >> kernel: [ 301.242882] ffffffc3 f4f47c00 f4f47c00 00000000 f4f47c00 > >>> > >> c10530c0 f47cff9c f47cff9c > >>> > >> kernel: [ 301.243438] f4fae544 f4fae544 f4c47f58 00000000 c10f68f0 > >>> > >> f47cffe4 c1052834 00000000 > >>> > >> kernel: [ 301.243995] Call Trace: > >>> > >> kernel: [ 301.244119] [] ? > >>> > >> rcu_check_callbacks+0x110/0x110 > >>> > >> kernel: [ 301.244248] [] fsnotify_mark_destroy+0x86/0x120 > >>> > >> kernel: [ 301.244377] [] ? abort_exclusive_wait+0x80/0x80 > >>> > >> kernel: [ 301.244504] [] ? fsnotify_put_mark+0x30/0x30 > >>> > >> kernel: [ 301.244631] [] kthread+0x74/0x80 > >>> > >> kernel: [ 301.244756] [] ? > >>> > >> kthread_flush_work_fn+0x10/0x10 > >>> > >> kernel: [ 301.244885] [] kernel_thread_helper+0x6/0xd > >>> > >> kernel: [ 301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89 > >>> > >> e5 53 83 > >>> > >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 0f 84 > >>> > >> 8e 00 > >>> > >> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 83 c4 > >>> > >> 14 > >>> > >> kernel: [ 301.248195] EIP: [] __list_del_entry+0x20/0xe0 > >>> > >> SS:ESP > >>> > >> 0068:f47cff64 > >>> > >> kernel: [ 301.248414] CR2: 0000000000000000 > >>> > >> kernel: [ 301.248538] ---[ end trace 15082dbfb353f84c ]--- > >>> > >> > >>> > >> The kernel was compiled with the following DEBUG support (the bolded > >>> > >> one > >>> > >> were requested by Gentoo's Dev: > >>> > >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y > >>> > >> CONFIG_SLUB_DEBUG=y > >>> > >> CONFIG_HAVE_DMA_API_DEBUG=y > >>> > >> CONFIG_X86_DEBUGCTLMSR=y > >>> > >> CONFIG_PNP_DEBUG_MESSAGES=y > >>> > >> CONFIG_AIC94XX_DEBUG=y > >>> > >> CONFIG_USB_DEBUG=y > >>> > >> CONFIG_DEBUG_KERNEL=y > >>> > >> CONFIG_SCHED_DEBUG=y > >>> > >> CONFIG_DEBUG_RT_MUTEXES=y > >>> > >> CONFIG_DEBUG_PI_LIST=y > >>> > >> CONFIG_DEBUG_BUGVERBOSE=y > >>> > >> CONFIG_DEBUG_INFO=y > >>> > >> CONFIG_DEBUG_MEMORY_INIT=y > >>> > >> CONFIG_DEBUG_LIST=y > >>> > >> CONFIG_DEBUG_STACKOVERFLOW=y > >>> > >> CONFIG_DEBUG_RODATA=y > >>> > >> CONFIG_DEBUG_RODATA_TEST=y > >>> > >> > >>> > >> I attached the kernel config i used for 3.2.9 to generate this oops > >>> > >> and > >>> > >> warnings. > >>> > >> > >>> > >> From the list_add warnings that come after, out of 805 warnings i > >>> > >> processed, > >>> > >> after masking with XXXXX the PID and next= values that kept changing > >>> > >> in > >>> > >> every one, i got 26 types of MD5. I also attached the files relevant > >>> > >> as an > >>> > >> archive to this email. > >>> > >> > >>> > >> The Gentoo bug i opened is sleeping, it seems nobody has the time to > >>> > >> at > >>> > >> least test to confirm or not the problems i'm seeing (or everybody's > >>> > >> thinking that nobody would restart auditd so often, so the bug it's > >>> > >> not that > >>> > >> serious). > >>> > >> > >>> > >> > >>> > >> Thank you for your time. > >>> > >> > >>> > >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram > >>> > >> wrote: > >>> > >> > >>> > >> > >>> > >> -- > >>> > >> Linux-audit mailing list > >>> > >> Linux-audit@redhat.com > >>> > >> https://www.redhat.com/mailman/listinfo/linux-audit > >>> > > > >>> > > > >>> > > > >>> > > -- > >>> > > Peter Moody Google 1.650.253.7306 > >>> > > Security Engineer pgp:0xC3410038 > >>> > > >>> > > >>> > > >>> > >>> > >> > > > > > > > > -- > > Peter Moody Google 1.650.253.7306 > > Security Engineer pgp:0xC3410038 > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter Moody Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Tue, 17 Apr 2012 10:56:17 -0700 Message-ID: References: <1327519203.4131.25.camel@localhost> <1332983643.384.8.camel@localhost> <1333660021.2273.0.camel@localhost> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary=20cf300faff5f02d0904bde3a98f Return-path: Received: from mx1.redhat.com (ext-mx15.extmail.prod.ext.phx2.redhat.com [10.5.110.20]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id q3HHuto6000977 for ; Tue, 17 Apr 2012 13:56:55 -0400 Received: from mail-qc0-f174.google.com (mail-qc0-f174.google.com [209.85.216.174]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q3HHurZa022665 for ; Tue, 17 Apr 2012 13:56:53 -0400 Received: by qcro28 with SMTP id o28so5102648qcr.33 for ; Tue, 17 Apr 2012 10:56:53 -0700 (PDT) In-Reply-To: <1333660021.2273.0.camel@localhost> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Eric Paris Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com --20cf300faff5f02d0904bde3a98f Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Here's a trace with debugging turned way up plus a few extra printk's added to fs/notify/mark.c. I'm looping through private_destroy_list before and after the call to synchronize_srcu. I can reproduce this reliably with kvm with 2 virtual processors: Linux desktop 3.4.0-rc3-oops1+ #1 SMP Tue Apr 17 09:59:44 PDT 2012 x86_64 GNU/Linux Cheers, peter On Thu, Apr 5, 2012 at 2:07 PM, Eric Paris wrote: > please please please keep on list. =A0Everything you say might help track > it down! > > On Thu, 2012-04-05 at 14:03 -0700, Peter Moody wrote: >> (please let me know if I should take this off-list) >> >> One other thing (again, maybe already known), but this seems to be >> exacerbated by SMP. On my machine, I can't reproduce the crash if I >> booth with maxcpus=3D1. >> >> Still hunting. >> >> Cheers, >> peter >> >> On Tue, Apr 3, 2012 at 9:15 AM, Peter Moody wrote: >> > This may already be known, but the issue seems to be limited to watch >> > rules. With any watch rules, I can reliably crash my machine while >> > freeing a watch rule after only starting/stopping auditd a few times. >> > With no watch rules, I have no issues. >> > >> > Cheers, >> > peter >> > >> > On Wed, Mar 28, 2012 at 11:44 PM, Valentin Avram wr= ote: >> >> Yes, i know that patch. It made it into kernel 3.2.2. I tested it >> >> successfully (oops in 3.2.1, no oops in 3.2.9), but this oops i'm see= ing is >> >> also in 3.2.9. >> >> >> >> I monitored changelogs since 3.2.1 to 3.2.12 but there were no fixes = either >> >> in audit subsystem or in fsnotify. I'll try to reproduce in latest 3.= 2.13 >> >> and repost the oops, but i'm 99% confident it will be the same. >> >> >> >> Sadly nobody except you seems to pay attention to this problem, proba= bly >> >> because it requires special conditions to reproduce (really, who star= ts and >> >> stops auditd every 5 seconds on a production server?). We only ran in= to it >> >> because one of our servers would randomly oops and then freeze about = each >> >> month after stopping and then starting >> >> >> >> auditd >> >> >> >> every morning (and the stop-start sequence was needed to workaround a= bug >> >> somewhere that would hang a >> >> >> >> gzip >> >> >> >> running on a file outside a watched folder). >> >> >> >> Anyway, as a last note, i have a feeling that the oops is not exactly >> >> random, there is a pattern, just that i haven't figured it out comple= tely >> >> yet. >> >> >> >> Will keep you >> >> >> >> uptodate >> >> >> >> with the things i find out. >> >> >> >> V. >> >> >> >> On Mar 29, 2012 4:14 AM, "Eric Paris" wrote: >> >>> >> >>> That patch fixes a BUG() . =A0The report has a NULL ptr deref and so= me >> >>> apparent list correuption.... =A0Sadly they aren't the same.... >> >>> >> >>> On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote: >> >>> > fyi: this patch [1] seems to fix the issue for me. The explanation= in >> >>> > the subject would reliably oops my machine. >> >>> > >> >>> > [1] >> >>> > http://git.kernel.org/?p=3Dlinux/kernel/git/torvalds/linux-2.6.git= ;a=3Dcommit;h=3Dfed474857efbed79cd390d0aee224231ca718f63 >> >>> > >> >>> > On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody w= rote: >> >>> > > Are you still able to reliably reproduce this oops? I'm trying t= o >> >>> > > track this down because this bug (or a very similar bug) is caus= ing >> >>> > > some significant headaches here at work, but I haven't had a lot= of >> >>> > > luck. I'm using usermode linux, though, so that might be interfe= ring >> >>> > > with things. >> >>> > > >> >>> > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram >> >>> > > wrote: >> >>> > >> Finally i found some time and spare server to retest the oops a= nd >> >>> > >> list_add >> >>> > >> corruptions i was getting with the 3.x kernels and auditd 2.1.3= . >> >>> > >> >> >>> > >> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and >> >>> > >> kernel.org's >> >>> > >> 3.2.9. >> >>> > >> >> >>> > >> Both get the oops/BUG in the same way and after that, they keep >> >>> > >> pouring >> >>> > >> list_add corruptions with audit_prune_tre(truncated?) and audit= ctl as >> >>> > >> comms. >> >>> > >> >> >>> > >> Since this is not about Gentoo's kernel only, i'll post here th= e oops >> >>> > >> in >> >>> > >> 3.2.9 and also attach some list_add corruptions. >> >>> > >> >> >>> > >> 3.2.9 BUG: >> >>> > >> >> >>> > >> kernel: [ =A0301.240011] BUG: unable to handle kernel NULL poin= ter >> >>> > >> dereference >> >>> > >> at =A0 (null) >> >>> > >> kernel: [ =A0301.240305] IP: [] __list_del_entry+0x20= /0xe0 >> >>> > >> kernel: [ =A0301.240481] *pdpt =3D 0000000000000000 *pde =3D >> >>> > >> f000ddc8f000ddc8 >> >>> > >> kernel: [ =A0301.240698] Oops: 0000 [#1] SMP >> >>> > >> kernel: [ =A0301.240910] >> >>> > >> kernel: [ =A0301.241030] Pid: 642, comm: fsnotify_mark Not tain= ted >> >>> > >> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396 >> >>> > >> kernel: [ =A0301.241370] EIP: 0060:[] EFLAGS: 0001028= 7 CPU: 6 >> >>> > >> kernel: [ =A0301.241498] EIP is at __list_del_entry+0x20/0xe0 >> >>> > >> kernel: [ =A0301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: fffff= fff EDX: >> >>> > >> 00000000 >> >>> > >> kernel: [ =A0301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47cf= f7c ESP: >> >>> > >> f47cff64 >> >>> > >> kernel: [ =A0301.241879] =A0DS: 007b ES: 007b FS: 00d8 GS: 0000= SS: 0068 >> >>> > >> kernel: [ =A0301.242005] Process fsnotify_mark (pid: 642, ti=3D= f47ce000 >> >>> > >> task=3Df4f47c00 task.ti=3Df47ce000) >> >>> > >> kernel: [ =A0301.242207] Stack: >> >>> > >> kernel: [ =A0301.242327] =A0c10813c0 f47cffa4 f4f47c00 f4e70888= f47cff7c >> >>> > >> f47cffa4 f47cffb8 c10f6976 >> >>> > >> kernel: [ =A0301.242882] =A0ffffffc3 f4f47c00 f4f47c00 00000000= f4f47c00 >> >>> > >> c10530c0 f47cff9c f47cff9c >> >>> > >> kernel: [ =A0301.243438] =A0f4fae544 f4fae544 f4c47f58 00000000= c10f68f0 >> >>> > >> f47cffe4 c1052834 00000000 >> >>> > >> kernel: [ =A0301.243995] Call Trace: >> >>> > >> kernel: [ =A0301.244119] =A0[] ? >> >>> > >> rcu_check_callbacks+0x110/0x110 >> >>> > >> kernel: [ =A0301.244248] =A0[] fsnotify_mark_destroy+= 0x86/0x120 >> >>> > >> kernel: [ =A0301.244377] =A0[] ? abort_exclusive_wait= +0x80/0x80 >> >>> > >> kernel: [ =A0301.244504] =A0[] ? fsnotify_put_mark+0x= 30/0x30 >> >>> > >> kernel: [ =A0301.244631] =A0[] kthread+0x74/0x80 >> >>> > >> kernel: [ =A0301.244756] =A0[] ? >> >>> > >> kthread_flush_work_fn+0x10/0x10 >> >>> > >> kernel: [ =A0301.244885] =A0[] kernel_thread_helper+0= x6/0xd >> >>> > >> kernel: [ =A0301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90= 55 89 >> >>> > >> e5 53 83 >> >>> > >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00 = 0f 84 >> >>> > >> 8e 00 >> >>> > >> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a 8= 3 c4 >> >>> > >> 14 >> >>> > >> kernel: [ =A0301.248195] EIP: [] __list_del_entry+0x2= 0/0xe0 >> >>> > >> SS:ESP >> >>> > >> 0068:f47cff64 >> >>> > >> kernel: [ =A0301.248414] CR2: 0000000000000000 >> >>> > >> kernel: [ =A0301.248538] ---[ end trace 15082dbfb353f84c ]--- >> >>> > >> >> >>> > >> The kernel was compiled with the following DEBUG support (the b= olded >> >>> > >> one >> >>> > >> were requested by Gentoo's Dev: >> >>> > >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=3Dy >> >>> > >> CONFIG_SLUB_DEBUG=3Dy >> >>> > >> CONFIG_HAVE_DMA_API_DEBUG=3Dy >> >>> > >> CONFIG_X86_DEBUGCTLMSR=3Dy >> >>> > >> CONFIG_PNP_DEBUG_MESSAGES=3Dy >> >>> > >> CONFIG_AIC94XX_DEBUG=3Dy >> >>> > >> CONFIG_USB_DEBUG=3Dy >> >>> > >> CONFIG_DEBUG_KERNEL=3Dy >> >>> > >> CONFIG_SCHED_DEBUG=3Dy >> >>> > >> CONFIG_DEBUG_RT_MUTEXES=3Dy >> >>> > >> CONFIG_DEBUG_PI_LIST=3Dy >> >>> > >> CONFIG_DEBUG_BUGVERBOSE=3Dy >> >>> > >> CONFIG_DEBUG_INFO=3Dy >> >>> > >> CONFIG_DEBUG_MEMORY_INIT=3Dy >> >>> > >> CONFIG_DEBUG_LIST=3Dy >> >>> > >> CONFIG_DEBUG_STACKOVERFLOW=3Dy >> >>> > >> CONFIG_DEBUG_RODATA=3Dy >> >>> > >> CONFIG_DEBUG_RODATA_TEST=3Dy >> >>> > >> >> >>> > >> I attached the kernel config i used for 3.2.9 to generate this = oops >> >>> > >> and >> >>> > >> warnings. >> >>> > >> >> >>> > >> From the list_add warnings that come after, out of 805 warnings= i >> >>> > >> processed, >> >>> > >> after masking with XXXXX the PID and next=3D values that kept c= hanging >> >>> > >> in >> >>> > >> every one, i got 26 types of MD5. I also attached the files rel= evant >> >>> > >> as an >> >>> > >> archive to this email. >> >>> > >> >> >>> > >> The Gentoo bug i opened is sleeping, it seems nobody has the ti= me to >> >>> > >> at >> >>> > >> least test to confirm or not the problems i'm seeing (or everyb= ody's >> >>> > >> thinking that nobody would restart auditd so often, so the bug = it's >> >>> > >> not that >> >>> > >> serious). >> >>> > >> >> >>> > >> >> >>> > >> Thank you for your time. >> >>> > >> >> >>> > >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram >> >>> > >> wrote: >> >>> > >> >> >>> > >> >> >>> > >> -- >> >>> > >> Linux-audit mailing list >> >>> > >> Linux-audit@redhat.com >> >>> > >> https://www.redhat.com/mailman/listinfo/linux-audit >> >>> > > >> >>> > > >> >>> > > >> >>> > > -- >> >>> > > Peter Moody =A0 =A0 =A0Google =A0 =A01.650.253.7306 >> >>> > > Security Engineer =A0pgp:0xC3410038 >> >>> > >> >>> > >> >>> > >> >>> >> >>> >> >> >> > >> > >> > >> > -- >> > Peter Moody =A0 =A0 =A0Google =A0 =A01.650.253.7306 >> > Security Engineer =A0pgp:0xC3410038 >> >> >> > > --=20 Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 Security Engineer=A0 pgp:0xC3410038 --20cf300faff5f02d0904bde3a98f Content-Type: application/x-gzip; name="trace.gz" Content-Disposition: attachment; filename="trace.gz" Content-Transfer-Encoding: base64 X-Attachment-Id: f_h1597grv0 H4sICEyrjU8CA3RyYWNlAM1bbXPbNhL+nP4K3PSLPY0VACQIUFPnLu/NzDXN2O20d5kMBwRBWSeJ 5JCUX/rrbwFSkiWSVhH7OMcosg1Rz7O72F0shNWrokSEI4KnHp9igRJdLeq8QAtdZno5RV8QQj6b iJBjKr6itHqR5fU8vXuxkuVioqZeQNGJKtZTRE6RzuryDqVwCYExlyLF3BfPUVHq6+2wxyVONcPP UaZv63t3Sx3HCu4udaqyGuHvXv112TzqKtuGbU+2nch7su1EdpMtmGAfU94rG/d2sql8VSx1rZPn 6F/vLv/mBB70gQv2FycliPf/tYp3h1vFicCUYhIGnouQof8VzXSmS7kE2rzWqp7nGUrlellPEYYL ffmefEWXP39GTrig/JvPvyHi9i7+Ff2cJ+ulrtByni10gubZFMXzLF3V0WpeKVQUCRinqFb5utKo 0uU8j0p5g1ZzVeYqTzSaUxUV8/mtD++t9TKSs6L9bVbXqJBlkZd1VCi0LDZ/IU2MpukyL4o7B4E9 DGo63Q+B+nmeTJHHnhvfWk3BRRoPiYyHoE95jWppxE2QN/En+KxU3lmeFxX5AX1P0OtcXVXNswsv gSm8+PjZzCjc/uXHtL0EIZoIHrz8ilDv6J50ETDUZX73A74lafAC3/rEJdw8CtN7cdlIIaZ7IZyk GKF37//56sOldTsCOY06YYeA/eqPxmd3l5Do4jWMHoYNunjTuRc7KeNhIHzbAwIqfuyMErj3cJT6 gROhmcPXYL393BcCIRZ9hDjsjjoRekAIN/YYj5A+vQk9FI66mdQ3hN60k91h1N+N8pAK6VtCNu2k TydCCN/34HAdXU7M0ykCZ9zCJ/79VxbZEl58lP9AKnjTcBOM3l622fbd5pc3F3hnYxCAgTliJwKI iDcXtAHhaSI0D7TfQHs7aB6mTDej/sGkBtpJIx/85e19qbee8faC9I7Sx4WgHxhCrxc62I2aKYQn M8oP7vUdCSGBfYZlRlfVQd4+KTZZvb4qtUxgxcr33NiYGF6U1aLjx6dOIsCsXtZSLaYu72KQrLph vMn324HeO3aB6DkFFwN36MbsHoFUXnfuDt/ixMlazmYZA7f3BO5oMTzwTZwmkOVyiX4tpdJO0xIE B+tuIzCsu39HN3Kho3URxfMa1lsfm+XWSa4g7C7qQQu+dd5K183SPp9leakT+KNaRMtcQe0FtJ6h 9ZxoOenopJUytcSiCQ2AVaZ4SJxgBT2ADfyYct/A2pujBjy60stCl8ZiQOFWnoigQ0Fo3Bis1DUU ZVEJBRCYKZLlrDJVkGdIPCcS0WMe3JBE0Tyb11Frp+gmLxdWFW5mgTvpEpI+czU0sypSVzKb6W/T IISZeAOFNtTlAoUaCYIChkSMsEKUIcizSQJRbR5+gkSIEmKefR9RHwG0R5CCl5QZDBniJj7bhy/M IFHmTg1RGaPYg32khVKG4uD+H/3kpWWWhsoLkUoR940UDXMqUCyRAFqykUggxZEKkNCIBS61u09C W0OPUCj71MweVMrox8Mi+aUTDNRTZ2dnX2CjmaDaZCfk8TTEiknpa8EozN5XuGEfk5B+zHBCcQiO ZDG31xek1jW60mUDtb1cME0p/furi08fP32YIlnD1i9+sZxXNZgxXs9g10wFxIYdkYnJH0ECNpX4 5NSJBfz2J1kmNxJkzeQKHLhnH3UEA9a0jRywfyvLdWH2yxO7Nz97aTfu1VW+XiYo1s0+/mTnLMrT nsanz1EMJruRVWe9naATA3S+V9PiWJxOnIRk/3e76IcFNnWc3RUTSv3NvliuE8iFRbnOdFTDjP3a 7Iqn6AOy19vmR88m2Yn6ocX74beyzuLNKAlSyAXgYVlULfObQtZXkVEnz0waT8FrFXbiCLscNAg6 HDCxZskzqYY5EQSH6wRs0iVNgKAv4JyQWbf8SBTfLz/aPGmTJtBIblYj7sbDe3g0uZ+SD1gUcWfh 3uFE6DRONbCss1rOYEFdZwbbk9qke+EGzrrgsN20pmoCIM/0UB1wBJr3QGMG0PeBPeYOHPYAC7kF hpDV0bbg8yUQhE4E4lvryCOwwRPUkQ9ThPhJ6sgjJPSJ6sgjNP4j6sgj0OxYdZL2VCc07MUM+ITh IDQfQZw/5eVCbja8r3/7gBYrWFBydcYIRSfdZet3eD6dos/5vMozlF/r8qac17XOnLjYfhn26MuV 3Ol+SEIfP73/ZYrw7X5po/BZZ4hP0Pt5WdUovqu1eQeGwqOqwZNRnsLfQezCzfGG+5WZFFnbaghV sMON7DTBFjhLoqJaxODGlJv07QGlnOlzAlOKVLE+J6iYJ+fBYRo4wgxaP4tgjZZxw2TChJl0wj03 oNACtW4VZVCeRSZeFpGCERvZMgZYGjjBCmxhWyNY9bnsWxmPwEAienbMnE6Afg9gg0fMOkWd0Ewy frbO5rcR1ABarizcqpqZVc9+uuEzN7ydulAzm8i16S8UxnKxG5RxkCSPqrtMRSZTXzd41wCYmAkl jqqKBq8XymUOOJTxAHWdVjsU3ygYuIFYU93dA7Her9xAWANS65V19yiVVW1qXmP1wBrdCU5s0sF7 KFCaVLCIwWRLLSsdJbKWxmTYLvq9WUDgwIkw3GWBFCiNKZmt0z0nO5gTvGeLFoGEoWu8A4JnLDms rBOW36QkI88mNnXPFuEICtRiz1SeVevVBsUztbl2QxEHEV5qdd1EOLMFohBOeBTfj/C20iSSuAY4 INGDAAcgTV0DG3C8Nho3wnAb0qEbSBtIGxAbi27zRYMnjUW6jcVLiI62NtDSHFNq6UExi/L4P1rV FYQgWlc6MT/T4hzfou11kq2Xy1OULuWsOrduvDvacZz2cCPNL5a1U6uAPP/I07TS9TluxLj/uhcT JzrzmZfT/eBLrWCHYpkDWrcHWhxcTpL4A5KQ0SUJBiSho0vCByTxRpckHJDEH1sSHw9IwkaXZCh2 gtElGYodProkbEASMbokQ7ETji6JGJBEji0JG4qdeHRJhmJHgSSwT44JkhpxaQ7XhGjP5QYlmUwm f8L/yTdJ4g1Ikoxuk6HY0aNLMrQCpqNLMhA7ZPT6JBiIHTJ6fWIOV3olGb0+CQZih4xenwQDsUNG r0+Cgdgho9cnwVDsjF6fBAPVIxm9PuFDsTN6fcKHYmf0+oQPVEpk9PqED/nJ6PWJGPITNbokA5US Gb0+EQO7DDJ6fSKGsr1zfSJZR5KJkyTmuwc6+TPP9L4o1BQocdz/GLwm7eUkgvnyh0ySeTbbF8Ez ax+T/Y/B69/t5SKCOQ2xnTZ+sP0CShKvq7NE6lWeof7TyuNNNkdY6QNNNkfe2ukiETRg2rYZmIP0 upTz5ggsNZ8/E7ePxEPeRY+x6TFQV1otInMCWUUyS6JSmw4nQ2M/5/TdaDqduMJXYbqlaT7mtM2C pkWIupxIcYIPGyUYjoWv7RH9053LAY/X1SJNTMdHQ2Cb9aKiaY8HHzfGsueJwo2mO+OcEd92JD32 LJUT0ukqISJMUmusZL0q7MGqPWKx3VoedkP3x5kKEnR4NOO8bf64fyrjc9dP+QkR/0Nw2mlboUzI xv6VvAZk892G7SzQtKej7QhDj//EYWz952mP0IGrmz8EVY02i5U9JYEIj3akxrGwjXHqRhSO41he N5doGUtrvMc1CgC2N6TDk2pwOP1MSR0Ka6n+Q0KihfMhIfF7km6YsB5lvrFzASg69oI1g2DbcvhE 7QxA0jEXlrJtPbSarItE1jpSSwlJ3R4+MkNDYzeabk5hqRZbcz2ulYIThrtqhNJ7QI3Yc9eC0VGM xbqrCNNs09pniYz3RnVTmiQl1GT2BBY7V0Ck08RMwlhIky2frDGFk04rME0CGjcxKYtClqu8jFKo 46JCl6s52M72SxPhHDOd7zxRlTDa+oFWa9DiroeJej3tqQ8zdb7mREIVktZwj+/AIdzv0UToY5oI 7a5J1wWU5GZ2HtkARLjoIsexKeUe2RVERLfFNtQ0bJCfpj2BmP38+49/3G/nnMIW0jTumh3cX2pl PHdrVwRSz6ktgJitfkfIn2W5MCKaLVfbQmH7J777L0CIo8ZTRAAA --20cf300faff5f02d0904bde3a98f Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --20cf300faff5f02d0904bde3a98f-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter Moody Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Tue, 17 Apr 2012 11:24:53 -0700 Message-ID: References: <1327519203.4131.25.camel@localhost> <1332983643.384.8.camel@localhost> <1333660021.2273.0.camel@localhost> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary=20cf300faff5f177bb04bde40f38 Return-path: Received: from mx1.redhat.com (ext-mx13.extmail.prod.ext.phx2.redhat.com [10.5.110.18]) by int-mx12.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id q3HIPRcg027704 for ; Tue, 17 Apr 2012 14:25:27 -0400 Received: from mail-qa0-f45.google.com (mail-qa0-f45.google.com [209.85.216.45]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q3HIPOKO017961 for ; Tue, 17 Apr 2012 14:25:25 -0400 Received: by qaeb19 with SMTP id b19so669632qae.11 for ; Tue, 17 Apr 2012 11:25:24 -0700 (PDT) In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Eric Paris Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com --20cf300faff5f177bb04bde40f38 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable and my config.gz On Tue, Apr 17, 2012 at 10:56 AM, Peter Moody wrote: > Here's a trace with debugging turned way up plus a few extra printk's > added to fs/notify/mark.c. I'm looping through private_destroy_list > before and after the call to synchronize_srcu. > > I can reproduce this reliably with kvm with 2 virtual processors: > Linux desktop 3.4.0-rc3-oops1+ #1 SMP Tue Apr 17 09:59:44 PDT 2012 > x86_64 GNU/Linux > > Cheers, > peter > > On Thu, Apr 5, 2012 at 2:07 PM, Eric Paris wrote: >> please please please keep on list. =A0Everything you say might help trac= k >> it down! >> >> On Thu, 2012-04-05 at 14:03 -0700, Peter Moody wrote: >>> (please let me know if I should take this off-list) >>> >>> One other thing (again, maybe already known), but this seems to be >>> exacerbated by SMP. On my machine, I can't reproduce the crash if I >>> booth with maxcpus=3D1. >>> >>> Still hunting. >>> >>> Cheers, >>> peter >>> >>> On Tue, Apr 3, 2012 at 9:15 AM, Peter Moody wrote: >>> > This may already be known, but the issue seems to be limited to watch >>> > rules. With any watch rules, I can reliably crash my machine while >>> > freeing a watch rule after only starting/stopping auditd a few times. >>> > With no watch rules, I have no issues. >>> > >>> > Cheers, >>> > peter >>> > >>> > On Wed, Mar 28, 2012 at 11:44 PM, Valentin Avram w= rote: >>> >> Yes, i know that patch. It made it into kernel 3.2.2. I tested it >>> >> successfully (oops in 3.2.1, no oops in 3.2.9), but this oops i'm se= eing is >>> >> also in 3.2.9. >>> >> >>> >> I monitored changelogs since 3.2.1 to 3.2.12 but there were no fixes= either >>> >> in audit subsystem or in fsnotify. I'll try to reproduce in latest 3= .2.13 >>> >> and repost the oops, but i'm 99% confident it will be the same. >>> >> >>> >> Sadly nobody except you seems to pay attention to this problem, prob= ably >>> >> because it requires special conditions to reproduce (really, who sta= rts and >>> >> stops auditd every 5 seconds on a production server?). We only ran i= nto it >>> >> because one of our servers would randomly oops and then freeze about= each >>> >> month after stopping and then starting >>> >> >>> >> auditd >>> >> >>> >> every morning (and the stop-start sequence was needed to workaround = a bug >>> >> somewhere that would hang a >>> >> >>> >> gzip >>> >> >>> >> running on a file outside a watched folder). >>> >> >>> >> Anyway, as a last note, i have a feeling that the oops is not exactl= y >>> >> random, there is a pattern, just that i haven't figured it out compl= etely >>> >> yet. >>> >> >>> >> Will keep you >>> >> >>> >> uptodate >>> >> >>> >> with the things i find out. >>> >> >>> >> V. >>> >> >>> >> On Mar 29, 2012 4:14 AM, "Eric Paris" wrote: >>> >>> >>> >>> That patch fixes a BUG() . =A0The report has a NULL ptr deref and s= ome >>> >>> apparent list correuption.... =A0Sadly they aren't the same.... >>> >>> >>> >>> On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote: >>> >>> > fyi: this patch [1] seems to fix the issue for me. The explanatio= n in >>> >>> > the subject would reliably oops my machine. >>> >>> > >>> >>> > [1] >>> >>> > http://git.kernel.org/?p=3Dlinux/kernel/git/torvalds/linux-2.6.gi= t;a=3Dcommit;h=3Dfed474857efbed79cd390d0aee224231ca718f63 >>> >>> > >>> >>> > On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody = wrote: >>> >>> > > Are you still able to reliably reproduce this oops? I'm trying = to >>> >>> > > track this down because this bug (or a very similar bug) is cau= sing >>> >>> > > some significant headaches here at work, but I haven't had a lo= t of >>> >>> > > luck. I'm using usermode linux, though, so that might be interf= ering >>> >>> > > with things. >>> >>> > > >>> >>> > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram >>> >>> > > wrote: >>> >>> > >> Finally i found some time and spare server to retest the oops = and >>> >>> > >> list_add >>> >>> > >> corruptions i was getting with the 3.x kernels and auditd 2.1.= 3. >>> >>> > >> >>> >>> > >> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and >>> >>> > >> kernel.org's >>> >>> > >> 3.2.9. >>> >>> > >> >>> >>> > >> Both get the oops/BUG in the same way and after that, they kee= p >>> >>> > >> pouring >>> >>> > >> list_add corruptions with audit_prune_tre(truncated?) and audi= tctl as >>> >>> > >> comms. >>> >>> > >> >>> >>> > >> Since this is not about Gentoo's kernel only, i'll post here t= he oops >>> >>> > >> in >>> >>> > >> 3.2.9 and also attach some list_add corruptions. >>> >>> > >> >>> >>> > >> 3.2.9 BUG: >>> >>> > >> >>> >>> > >> kernel: [ =A0301.240011] BUG: unable to handle kernel NULL poi= nter >>> >>> > >> dereference >>> >>> > >> at =A0 (null) >>> >>> > >> kernel: [ =A0301.240305] IP: [] __list_del_entry+0x2= 0/0xe0 >>> >>> > >> kernel: [ =A0301.240481] *pdpt =3D 0000000000000000 *pde =3D >>> >>> > >> f000ddc8f000ddc8 >>> >>> > >> kernel: [ =A0301.240698] Oops: 0000 [#1] SMP >>> >>> > >> kernel: [ =A0301.240910] >>> >>> > >> kernel: [ =A0301.241030] Pid: 642, comm: fsnotify_mark Not tai= nted >>> >>> > >> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396 >>> >>> > >> kernel: [ =A0301.241370] EIP: 0060:[] EFLAGS: 000102= 87 CPU: 6 >>> >>> > >> kernel: [ =A0301.241498] EIP is at __list_del_entry+0x20/0xe0 >>> >>> > >> kernel: [ =A0301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: ffff= ffff EDX: >>> >>> > >> 00000000 >>> >>> > >> kernel: [ =A0301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47c= ff7c ESP: >>> >>> > >> f47cff64 >>> >>> > >> kernel: [ =A0301.241879] =A0DS: 007b ES: 007b FS: 00d8 GS: 000= 0 SS: 0068 >>> >>> > >> kernel: [ =A0301.242005] Process fsnotify_mark (pid: 642, ti= =3Df47ce000 >>> >>> > >> task=3Df4f47c00 task.ti=3Df47ce000) >>> >>> > >> kernel: [ =A0301.242207] Stack: >>> >>> > >> kernel: [ =A0301.242327] =A0c10813c0 f47cffa4 f4f47c00 f4e7088= 8 f47cff7c >>> >>> > >> f47cffa4 f47cffb8 c10f6976 >>> >>> > >> kernel: [ =A0301.242882] =A0ffffffc3 f4f47c00 f4f47c00 0000000= 0 f4f47c00 >>> >>> > >> c10530c0 f47cff9c f47cff9c >>> >>> > >> kernel: [ =A0301.243438] =A0f4fae544 f4fae544 f4c47f58 0000000= 0 c10f68f0 >>> >>> > >> f47cffe4 c1052834 00000000 >>> >>> > >> kernel: [ =A0301.243995] Call Trace: >>> >>> > >> kernel: [ =A0301.244119] =A0[] ? >>> >>> > >> rcu_check_callbacks+0x110/0x110 >>> >>> > >> kernel: [ =A0301.244248] =A0[] fsnotify_mark_destroy= +0x86/0x120 >>> >>> > >> kernel: [ =A0301.244377] =A0[] ? abort_exclusive_wai= t+0x80/0x80 >>> >>> > >> kernel: [ =A0301.244504] =A0[] ? fsnotify_put_mark+0= x30/0x30 >>> >>> > >> kernel: [ =A0301.244631] =A0[] kthread+0x74/0x80 >>> >>> > >> kernel: [ =A0301.244756] =A0[] ? >>> >>> > >> kthread_flush_work_fn+0x10/0x10 >>> >>> > >> kernel: [ =A0301.244885] =A0[] kernel_thread_helper+= 0x6/0xd >>> >>> > >> kernel: [ =A0301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff 9= 0 55 89 >>> >>> > >> e5 53 83 >>> >>> > >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 00= 0f 84 >>> >>> > >> 8e 00 >>> >>> > >> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a = 83 c4 >>> >>> > >> 14 >>> >>> > >> kernel: [ =A0301.248195] EIP: [] __list_del_entry+0x= 20/0xe0 >>> >>> > >> SS:ESP >>> >>> > >> 0068:f47cff64 >>> >>> > >> kernel: [ =A0301.248414] CR2: 0000000000000000 >>> >>> > >> kernel: [ =A0301.248538] ---[ end trace 15082dbfb353f84c ]--- >>> >>> > >> >>> >>> > >> The kernel was compiled with the following DEBUG support (the = bolded >>> >>> > >> one >>> >>> > >> were requested by Gentoo's Dev: >>> >>> > >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=3Dy >>> >>> > >> CONFIG_SLUB_DEBUG=3Dy >>> >>> > >> CONFIG_HAVE_DMA_API_DEBUG=3Dy >>> >>> > >> CONFIG_X86_DEBUGCTLMSR=3Dy >>> >>> > >> CONFIG_PNP_DEBUG_MESSAGES=3Dy >>> >>> > >> CONFIG_AIC94XX_DEBUG=3Dy >>> >>> > >> CONFIG_USB_DEBUG=3Dy >>> >>> > >> CONFIG_DEBUG_KERNEL=3Dy >>> >>> > >> CONFIG_SCHED_DEBUG=3Dy >>> >>> > >> CONFIG_DEBUG_RT_MUTEXES=3Dy >>> >>> > >> CONFIG_DEBUG_PI_LIST=3Dy >>> >>> > >> CONFIG_DEBUG_BUGVERBOSE=3Dy >>> >>> > >> CONFIG_DEBUG_INFO=3Dy >>> >>> > >> CONFIG_DEBUG_MEMORY_INIT=3Dy >>> >>> > >> CONFIG_DEBUG_LIST=3Dy >>> >>> > >> CONFIG_DEBUG_STACKOVERFLOW=3Dy >>> >>> > >> CONFIG_DEBUG_RODATA=3Dy >>> >>> > >> CONFIG_DEBUG_RODATA_TEST=3Dy >>> >>> > >> >>> >>> > >> I attached the kernel config i used for 3.2.9 to generate this= oops >>> >>> > >> and >>> >>> > >> warnings. >>> >>> > >> >>> >>> > >> From the list_add warnings that come after, out of 805 warning= s i >>> >>> > >> processed, >>> >>> > >> after masking with XXXXX the PID and next=3D values that kept = changing >>> >>> > >> in >>> >>> > >> every one, i got 26 types of MD5. I also attached the files re= levant >>> >>> > >> as an >>> >>> > >> archive to this email. >>> >>> > >> >>> >>> > >> The Gentoo bug i opened is sleeping, it seems nobody has the t= ime to >>> >>> > >> at >>> >>> > >> least test to confirm or not the problems i'm seeing (or every= body's >>> >>> > >> thinking that nobody would restart auditd so often, so the bug= it's >>> >>> > >> not that >>> >>> > >> serious). >>> >>> > >> >>> >>> > >> >>> >>> > >> Thank you for your time. >>> >>> > >> >>> >>> > >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram >>> >>> > >> wrote: >>> >>> > >> >>> >>> > >> >>> >>> > >> -- >>> >>> > >> Linux-audit mailing list >>> >>> > >> Linux-audit@redhat.com >>> >>> > >> https://www.redhat.com/mailman/listinfo/linux-audit >>> >>> > > >>> >>> > > >>> >>> > > >>> >>> > > -- >>> >>> > > Peter Moody =A0 =A0 =A0Google =A0 =A01.650.253.7306 >>> >>> > > Security Engineer =A0pgp:0xC3410038 >>> >>> > >>> >>> > >>> >>> > >>> >>> >>> >>> >>> >> >>> > >>> > >>> > >>> > -- >>> > Peter Moody =A0 =A0 =A0Google =A0 =A01.650.253.7306 >>> > Security Engineer =A0pgp:0xC3410038 >>> >>> >>> >> >> > > > > -- > Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 > Security Engineer=A0 pgp:0xC3410038 --=20 Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 Security Engineer=A0 pgp:0xC3410038 --20cf300faff5f177bb04bde40f38 Content-Type: application/x-gzip; name="config.gz" Content-Disposition: attachment; filename="config.gz" Content-Transfer-Encoding: base64 X-Attachment-Id: f_h15a8ag31 H4sICBu1jU8CA2NvbmZpZwCUPMt22ziy+/4KHfddzCzS8SPxpM89XkAgKKJFEgwAypY3PI6tdPtM bOfKdk/y97cK4AMAQcrTi5moqvAq1LtA//rLrwvy+vL0cPNyf3vz7dvPxZ+7x93+5mV3t/h6/233 v4tELEqhFyzh+jcgzu8fX3+8//HpvDn/sDj77cNvx+/2t2fvnp6+P58s1rv94+7bgj49fr3/8xVm uX96/OXXX6goU76CAUuuL37CJC3gCmY5O13cPy8en14Wz7uXXxzE+QcgHX4PP3iptKyp5qJsEkZF wuSAFLWuat2kQhZEXxztvn09//AOdvvu/MNRR0MkzWBkan9eHN3sb//CE72/Nft+bk/X3O2+Wkg/ csVKJjltaCFUU1cJ0WxYmuaCrpWoJWXNJdE0S8RqwPZDkYptWKnVgDQ7csfDzGR2bLOUgiSUKD2Q ITZhVaPqqhLSQShN6FpLAhOPcBnZsCaHg5R0q0VkcFHUw4+SsaRJCtIUpMJp3fMbnFoZdM7Klc7G J+CKIH6MWNarKLCRDDbHYY+V4KVmUo3JskvGV5keI1YVFwGTC7K1R65okybUlUZ5qVjRj1UVL5Gh Eem0hFc0W5EkaUi+EpLrrAhWyohqaFU3PMlRHnhkf5TkfCmBiSCNOdmOCTQvWLNRWwWU+cT8yKCr CA4EnNS5NuvHhhKawcXzEmSCX7OJyUkNMiHFkgUCo5iuq6Zi0lJJRgI56FCsWMKvlEulG5rV5XqC riIrFiez++FLJktidL4SSvHl6FCqVhUrkwj6WsAhQerOTp0hNdgzM3A0jZF/1YgKuA+cScDgAJt4 uZqiTBgKKp4AbklQz2yhHVFF5cNcWbXHsxLc0DQnK3Vx9O4r2uJ3zzd/7+7e7e/uFz7gOQTc/QgA tyHgU/D79+D3yXEIODkaVGNtBBGlkMiIQljzhbeIooISyYhyLqC3tSBvYLCO3n+7//L+4enu9dvu +f3/1CUp+kHvfwtMrhE3Lj83l0I6UuFAwMH8ulgZr/UNd/X6fXA57AoEDPZeapK7HgTunpUb2Dhu qQDdPDvtzbgEAWqoKCoOQnR05FhXkm/AAIEMIrhnjoswChPhkDnFGkSY5c3qmleBOrWYJWBO46j8 2rWaLubqemqEcK2bs3hkf8EGwlG4ujsqxF9dz2FhJ5ElO/uUCaVRAi6O/vH49Lj7Z89wdUkcPoER 3PCKjgD4/1Q7lwv6z6+a4nPNahaHjoYsVYKiSxlcPKFUT2OazZnLnjQjJZjXyOk0UWv0j467QpA1 9MEiBnEVgXGBlKIutW990HYFP8c+whg4E4eEQC0Zc09hoblY8bLmYO7A42uyjB7LCFjvd4lMQA9V p4Pw78Xz65fnn88vu4dBByPkYVQAyhx4GRelMnE5xqCxB7YgRXwYzVw9Q0giCsJL9+gD1FrxyJGR BMJJCo5AZ+DoEu8uVEWkYu0mDBckrRdqzAVkegM4d/UKYEWlERwLM2jdpKSEiPbi/IMTpvRgsG6E 6qjmWSJwpqVosuuYfLbbaUxY6LBp3UbFI4jRAzfYBO2oU7gcnuqLk0+ehNQQoqMEgcvIgG8mbnUC 5ZUUdaVcTljQ5B206BQ2fe1G+/2wDXcPAa4IAhRH0HDrTQWybTGjCQDsq59kbSButM+LOtshBYSA QNWAIZGzyMCOTVGAP0KOJRGueLRr+BW98pYY/F06zUFzH8NmUsJl42OG+VJIMsC8XfJEZzHx1E10 zmW+ble7eBhmszHSgIsfIWN0beJ8PK8WMmaC0FeA1oFJHtasIQornd/oF0pPwODaJYAi86FYuGNL poOxVojRs4+23udIjbgsjWQOo7YqxdgQlJxCiJ/EWOhH/cgdEGUTnsjED1ckKWA2K5ROUCKTIJwA QBBFAMQPHgDgxgwG74UKlPbBL1o+kx9Edh96UbBJJewV8nGPeyYE05Hx4GtOzgOOwYRN68piUQqA 1bZwLquDNJ7vy4SucjehrCTIlGOBvGyT5SmouXSDBaqqtWwqyDyxkBDEV5PYJcSvTVq7O0lrzZzk jFXCxSq+KkmeOleNcbZ0ASbbdwEqAwPgMJ17V2fSrSQqbHbvMHvTVx+Mv2prNtVu//Vp/3DzeLtb sL93jy/PC/J4t6BPr48vu/3z4Mj8KVygYysDvTdoUMFmU5gUKbK9TWEnjVjcivLmc83l2oGpvF7a yT3Jhaid6GYp11H7onKyjAkyzOX5ipTnnpM3PtLYJWcHwhIy18p1sPYoBcS3vMrZ1dR19HOMZm3K gltpcITdxEgee/+oi6qBU7GYuoAS9yPcVVmacspxg5B65SCDaN0oxrjOWpKNBhsbCvBalhBXaJ5y d3c2UQM9wtIQDA1rTOvoXmLrtKyJw02uaS4kE2IdILH0RLSW/qaHmg4wK2N55UmXGSjZCmxImdhS WcuNhlQ8tkDFe8lzcdklCB4j1oEFuIJfAZMHtDIrBkRv4K+jfCghMd4MIjGq6WysWCmSQmBTVFjB CmdoxcFWWE1NI6Box9msdwKXiHpcn7kkIHAiTzoltcmcTd6p2Lz7cvO8u1v821qj7/unr/ff7h// NPguuAeyNqeMqreXnOBNUZExCbyK6jxZ8jJ1yoMS9RWstStxxqIrtFkXx72vF0md+1poQTZLAMaQ mPltaeoS8ZODLTo+vC0xONtrhypJ+/KD516wnlsQyINK5kYZXjDeBR1LtYoCc76MRSiarSTX2+4G q5v9yz1W+xf65/ed6y2I1NyU7UiyISV1I0UCXrccKLyU1Ec1tC5IGS9BhKQMQveYwQ3pOFVzK5Ik VW9arxKXEN8w+oYlJVeUO+GAUKl3/G5YwVckitBE8hgCLjkKVolQMQSWNRKu1oGZKCDSvGpUvYwM USKHxZWpX0bQNYy8JJLFps2TIn7LiJhK+dQqelLwqBL2GNtgXcbAayKLKDNZGl0Ay0rnn6JMQ8Vp rVdf8hALdfvXDiuZbozEhc0ZSiHcEkQLTRhJck8nOwxNP7vhBPxsc6yWYKaM5kzqBF8Wh9uYGdpO fnF0t7u5A6O7O/olKFB03sjTGFMWZ6ZRAg55GzZL5uiaZXaQ9E3z/TeTYZHmzYQKfEksZyld4rps bWl0Kot9y+5ayrfsbyCNZMjhCKxWveVqDN2biQ6caKA7cB6PMOR3lLgu37TJNzHepXzLRv8rxl+C b2Rv4bwlfDvVgUM5hAfO5FMe4r6lPsx+l+5NW33bBYS0czdgYjirK+BxTVUmosdDJdGGL/un293z 89N+8QLhy+IGEt+vu5uX170bynRtRDdbh/0AVGVutmNKrk5kzFeZKd+ZbE4d6OrXPE/mScBJQ3Dy R+12/kcdxqIydWkfyK40KxNsug6lC+8txgbmmXiN0Tc7C1YIucXKYV67NRPrwETBNTAXG3ptv95J rIkkG45lw5op7x1IjzFtxqHlEdsLczwy/MDq/bEPqSTfQAa9gu10K7nYTbYpfFBBrromgD3dxcfj Y5dA2RTRCJ3HM9a2DZo0Vllcb4qw9I2gYFPd4aMM6V4hRAuXHRENvVCPmQquQESXQmhbSRpUhxUa thZXq/WnOLxSNI7AktppHEW0KCK76qW9qn3BNYKEhcX21YLtOZy7JKN0FoH5yfSAK5IkPvVlZTrK qnHLCojQKujpt4sFL5TwVdAmUEMIqIu6ML3xFML6fIutHIfAXBDVeaGk175AfTPHzsdgUiRjIMUW d+2WCyqmwwKSgbGixuc+kO86p0rcWsIKYl8ICu3DnyEYJTkgthYRC9gvudD50g1qgbCtt3hiRq7i dqY0T0GUwyJrVFShQztT0LCLN2M6Rv22Emx2TKFa9EbkMAmcNdKrmxlmlg4EEIskWDAKBAgbuyOg ZBDqa1tyX0qxBuOCStr4PVMjVpSNAKG0dGBPWjogNjLBaeVJbJo/GNWQhbgKkDFIovJmAxjhyBP/ tB4IC06lwNd4EVC4uQHhbW8AY53IaH1KRmcFVfF2BzLDkwFkik1VtlXgJRPZ6P7Zoff6D+t4UXTC JZyyWS2x9uW2hWrX88O2A0j71ofQigcY0xHBRjGEI8jJZtQiQXrUy6ilNoN9+2WfHxkPj71B2zkM Xyf16DbNC/Esx3O27ryAZcJ6oWmartGMNxBSOeN5Dt4VxKH18M2G5DW7OP6B6eOx81+vsnNLDfss SFmTGCZ4MtZV1Xs+eoVWuwjopGKuNjrcutIS/hFDbeB/ir4VG6Mw3YvG7rZqtFgxvNGZucbbC+pv HtgcqfGGWXHlihKZuMNdr40nbnte0Thhxcy7sj4AdPebQ8BWaVvaQOP7u/kvaKmYx7axqiRfSeJX SWY0rwugG9zNxUlfVwG76Sq5jQQhQqmV9wo1UhNfq8J7ZWbKGeYKbZScyIvzjx/PzuPv9qZC2RF8 qDBfgswr0+ZEKxl1MlqSEsUA+ytZvWLI/ni3O2ekNPHJRMxt2uHNkgt8jiZlXfmcRhJ0EBjtFd2m B0I7PHQxikl87ysuHS9baOnwFH9BxFtyzb23Fj681ZHeuBxPkBlOYLMDvXtHfOLuyVbT3AuqFZi8 CosthjtJgAb+QrTuFfK8FmjKG6XraKOPURRnJ3qituGD7arAt2XXzcnxsXf5183px+N45+G6OTue RME8x7EO4PXFiWMkTVyTScwpHQFnV8wJE6gkCqxBbc7gPLMDItP3iWk/6CPHKAQOCknG8Y9TzzZL hjGK9u1dP8a0g2DMiW/QW1vTBuph93WTKBEX+MJUKFFo86lXpK1Y+UbN7+ZBOIlRr3Gaxl7ypM/j n/6z2y8ebh5v/tw97B5fTCaPLnnx9B27E042P3pWnDHifUnQviceAcavkLpZMD3J8yXIkxojfSNZ wHUkTpF5eNGBqJyxyidGiJ/lA1RiqOt6sqrw8KP2OMC6Lsxkagg0mASOj08hfkFk+17MndR+WgAJ RbQH4jZRTXgUnM/AMNhIVRRoOywBCgTenADGPIzG2A5pMILQALAkGqKXbQittfZ6Lgjc8ISJYKGU BB0rbAvHPxjoN8aUEnI0ilcFnxo1oWr+jA1ZrcDEQlotp+ZpQ/jgXG2g6k9LawX5eZOoRE/NtsxB yPEtd7NlRILxD9gwIVt21xQlQASZEkqRn6zavQhIxHg5gqtlKCvoJx7iJykgSBPJRDMPT7OSehpL Kha7HWW8jm1PV/Ui3e/+73X3ePtz8Xx70zaunRQdbMZnP2lHSBPY3R6OL4cnwMBc4EiuIuguAFqJ jXkogN9Alb6yRmnx3YF5URe32rEhAsJImDx5+wjAYexhvuWJvU/sxow3H6VAk4BluQl8f6QJfLf/ CbS72e6Wf3w6Nzf9Nbzpxd3+/m+v/WcEmppClrl4L121ktlifB3AQ5Xisll/8hFgiVmiNLNlHslL EQz8YGuAkFrBWma3z3/d7Hd3Y7fnT4fN/QeXA/7HQv3nS8iTHMJpJieQBStrvw8G+b5Bu3dtdlbs Hp72Pxffjat+vvkbOPjsvvLg/4JQKj50+frcnWjxD7Aji93L7W//dB4bUO6/GrPJvA8rivCFM0K9 urIZagrCXpqOYFouT4/h0OY9Wsx9Us4wu/CSl86I4wRI4K3EiF8eQxC4Nxkvq7YDIgmIS6CqIpwS YdNOfyDoagXjwb3OTW/MkA3KO7ESGHs29S1VoYIrtADvDkan8LDSfoXVRZITH+QYLwRZgqOb2Byg HJ+3pBLcDgY/PTLT/ncGONz7uAMB3C3/mruWwWEqongSDvIrgQgL3k+0H036z9FMBcp9tkixQjDs uKIF5cR1iRZi3ic1lMc6CjiDFVwbTdN3tzf7u8WX/f3dn24nbIvF5mEp87MRzptjC5GciiwEah5C GKQEunafQ7SUQkHw7BTSquT8X6e/D+vyT6fHv5+6RzZFklLg1wfaZaqjgHGtTMnaqVu6GBO/TmIa viziWDq5FmKaa/3x48fj6aFdMyROobKKulcrQTQSLkYGk/3Y3b6+3Hz5tjMfcy/MC9+X58X7BXt4 /XYT+AZ8EFdofA/tWPn2RW0EZRR3QLR5OT4ar1wTZwqImLL24QAS22zLM6/2AaKoY1atXabgyuEI J2enbTsjeFhkMOFUXqvzKvrdef+4xz89VuRrLO5iOlz4dcX2K7Rw5Jpt1QhoToi1VLCla+bncyUb rwswSJjX4F2VaqvR5lrL3ct/nvb/xthj5OAh6Fm7U+HDLO9DCHzilXASba6m7nN2/GUCKVfSDFDV SzhBzuk2zl+ksaVBNk1gdFSBmqqJneALUazWPLhfZgBf3e20oNhqfSTicoNXtnzsfzgP0D4vNj0Y zyEDNuVLSHLBc5lP+mKrVENZ2gT13scodtKWgrjfxfe4DZNL4TbsewzNibJew91RVUY7aAzf1fKA Z7xaoQqCplxFEDHayB8XwBOaDQUbKdyz9aePs6jihSqazYk3awv0PjslskqiX5yUoBpizT324rZJ 5pgFBDBVBZBQmgzQyBk4nxL8rY+JAq1kY83JVnnxDyVMUsxPsGQsHJtLERwL9TQAaVp1YCfaBkSd VFN6jWPgn6vIG9wetXR9TQ+l9dL1oz38kil9KUQSGZLBv1wVHRAK/hkvmfck22VO5kk2bEXUPIn5 Tj34fnVMlVeHFirFPMWWkWyegudgv0U01uqcYcfh0aNNGawfoLs1Lo5uX7/c3x75qxfJR8WjX7NW G/dPqcAv816G0K2v1wC3lhBbaWn8cRYS2Q/F0Iw3SfThOsrm+Ug/z8cKej5oqL9EwavzyYknFfh8 AnpQhc8P6PD5WIk9RXTxhoXtt3Rkoodmzqi4Hp0bYM25TKaGlNgUNAVwva1YcKf9xgNWTlvpgNDs PO5hguwEIPhsBpsrBZFea7JDVdnWVPfAexZV/F0XkKY8DzxvD5x+yNRTjE3bUvJkxbyZbZ3hab/D EAoi4pfdfuqPFQ0zt1FYEHb4SPNF/+z+OkL7dwkepglysXLQ+FVdWWKBfR2HNi3TY6jYlbh4bMjG bJNHFFbmPWT/HWB8Abz0qSh8RGikY5JUm2/KRJNQWk1suSOxQU10tKL60GjwoznXbILbBKuGZAKZ 6moCk52dnk2guKQTmCEKi+NBYkxzuFThcQcJKIvq4AVXupqegESfSvk0fOrYesQRHVUnFzEvNL26 4NeBV9tZlbtqI/G2JHplEuHnxe3Tw5f7x93dov0LNDGdhxTNaM5DFIUHbNHezC83+z93L1410xun iVxhsIp/b+PAzjta08RXdTGxk47K5Ajp9gDVsO0DVJ3hmCfU8/hE0WqeIsvn8VgEtl2hWbLcb/1E ScTqjQw/fO4y9U10lGTSzDtEIjTrESLMapk6wGkgOUBgmrazJG+6cggtC6UOMRuoIOZRkDBXo4pU pyoPNy+3f+2mVaXAP1xjXvBgdHPo7iw1zWulJ4WlpRGFKeDM05TlcquZOkhl/3bUIaoZvg5Ec7LQ UlX1LB494+TVtCRsM/UnLGLU0wpsCRgt5/FqfnxGVHaYhe0L4lmSfBbNKwj1V+wAzWb+vts/aThL cvAsBaEH8AdExUbwNmmZu+kyPRiR9rRCpbNr2j9tMkdhi4HzJNlWgfjN06y1yWXmaD7Xwu09RCjm TWZLw0heHKCg1k7McXkqnIxQmr+V+XCAwiTSB6gk/mmKOZLeSM+Q4LOhOYL67NRPHvEvPsaLeq7a mJ9djcEdvVHTf1/LYCFUa5YcUomT0/bLWtDHxcv+5vH5+9P+Bf8gwMvT7f9TdmXLcePI9lcq5mlu xPh27aq6Ef0AgmQVWtxEsBb5haFRy2NFy7LDUm9/P0gALGZiqe7bEW67TiZBAMSaSJz8+jJ5+frw 8+TfDy8Pr49g/3779RvI0dVbnZxZyoMl6s+QQK3xwwJmBq2gLCpg+zAOreNyURiK8zZcFXaz27Zu HZ58qOCekg8VSRjzUkv3LiJ9JEtdqLobFr26RHIfL5Tcj191g555+Pbt5flRb7cnn59evvlPkl2D fW/OO6+WM7v1sGn/39/Y1edgQGuZNnMsYzuyuEh7l7nHOio3SslZ8RlQNJc9CMHt8mYfxs28ivvP RdQ2pquGO9JFresKN2lrBHHQYZmZ0VsfRHhwTRIV69zf2jUfW19ICiWTqnpalmaRV6ghCLzeY5mz fhzuO/F2PyQJbeVB6u5KAXO34YB1IcWABUC/sQnoett+AMl2efho1Y4yCZEKsAvbIHMnUSQLMiIJ NIiWnVxIbQIOmtbDwVkbacdsaI8BwVgq2z9/W/9/e+g63kPX+I3rYAdcx3rgOtYFkSA7iPUyIoPs RUSw8YmI9kVEAKWx3sdhhXJPm8b6b44HWK/z0g5swq3E/6Trq2PFmrRhCjutbO00M9000oy/Pr1f axzodIRrgr5cdXuWwNF/0Od1sP3mfZa439/KlAAsbAc8PiBR59UCEZIhEEk203m/CEpYWeOFG5a0 TRAXMXgdxJ3dB5LQXQUSeAtuJJNd+PXHglWxYrRZU9wHhWmswiBvfVjkj/s4e7EEyWSF8ENMMNhz 8NgbsTLDOph37i5bY5qsLMw6p6R9mgAdz+66Qp38xKsurrMHnh/38mpARe7ZLHR8flEo0xVhvk8j J6qiCbr0d2jrpH6o+UmQOhkwuGcueNACDSqqIWU0obKpGUWSdr7eLN3EDao+hz/4jeRC8+DejDTr XaDreW1E7Er12YFDx5ACkk+fShYiM1RFg6Fghnx/R6zfHdsG31uDIY2SpepBzhzABpIvCkzEW/A5 HmHPtCmf7e354DdgBRoXwPORNU2RUVg0ado4P/us4ozshgBU2WbkYtJ5vgp/GtYkQUGm/s7CHeCk ai9aIc2+dqpQZFkGlb1aRtxkBtZaPbnc/fr065PaTv5gyZyI/77V7nlyR31lANx3SQDMJfdR0m0G sGlF7aPayhN4W+tZujUs8yReSiUNpNRld0UATXIf3LV4GzigqfRsVRpXf2eBcqZtG8p5eQcVEDvq MdW+r2+zqxp3+d2V4qtO22ae0dpUvnNh1iw0Xh7e3p4/2R0qbQa8cIiCFQAkgY7RRsMdF1WanX2B bsVLH89PPubYYSwUJXm2YudIeHivPDaB3Ch0HcgMXBH1UMvrTJLOSsoxP2LmbiUKrYBE3HXss7g2 vAclpip8vMwce+Ag0JfWvSIwbEfQrmNw8gv2S+e1gO8YXn3smDkkTvwEStF6vQRwtfvwQbp9HrIA AZZ8WAq3ojR6m4TVuTkqJC0dSi2q9EoXyQUmwkw5KmBaSU2GDgEuMC+m2q5rSkf8shG94qJfN1l1 lCehvlCIEbtBHazNddAFwmpBaeMrII8VdYBTfMjSnctRmjR5/xMNQ2UhbySwbraT96e3d29GUEvn XVa5dd3xKwtBvV5ogZOzrkR437JnZctSzelseTUff3l6n7QPPz9/vdhA8S1WNcWiBZP61aesZEBv eiShmbq2RqNyC66mdvPFzv+rpulXW9afn357fnzyrzSVtwLblNcNOa1KmjtDhDAi7J7XZQ+Mtnl6 DuL7AN4wlMY9I22Zs6D7VKuNMcbE8D1loVoSLdmziLYivpctHAri3ynTPJPsYryFdD1Pa61nwyhB qIVCkiURSHUIBny5XqPDvVyT9uun73BV64M2c3vVr3WkaKMfRrRqC6eGifaymf76+p+XJ98wntZ6 fz5em5diwNDF1k7Ie+nhXXYLHAMeXItyMVcTjisAq5v25bgIEPvPejq1eLCX7ESbiOKaRsmb+Wx+ NRHgj0my4hbiesS1VHHn0+nVd6kUdsAsek1FpuzjR+BCuKazXW0DCvqL5Vc+70EmQxMfN3pqR8Lg fh08GBjVJQfJ+DVOokrqKrXgmEwJcVI4d5MZq6cQUdmxkCLy/qNg7qtKLiPKCVm1w6Y/h94YfCcY bKqsicl42UVle5H6vgjJy69P71+/vn+O1j48yUXSqa+AhioLyhQv3Q16YNjQNmL9HnGFITjh+FAJ CVi3X9wGJZjh2WaFl/Pp4uzlsGGzqY/mgcKkXTHzC7jgHlYcMnrN7FJDgYIf1R+Cle2x8IDeq0dT dtosTkDJH+aeZHnfntugpQLubbQHchvjJCDSmAwgPSE5OmXa0QvzwmiIRsHRkGzuPSWBrgLyfAdb UVTDVaEBzbQAjsI07orRhv14VtQQIO/EWog/F7yyN2gDxYrDSo9SMj7jeHGFhIZGIZQFYzpgBaSe Jtdef6KVh6rUBEJB9gqROLUxID1v7xv1qbMmKuNk3+AIu1tqhrqIY0fPJeNOVgZEB0BsOWHYtvXM gVtIdi3hMwtISZQ/rHDhKbqazHAT9x9fnl/f3r8/vfSf3/+Bp1GrWmZyf6Vo4BwnA28IfHUq01e6 AwmfRMnOeG7R7Jn6iQIa7I+bcfl+K/BgZX47GbKgqJoDWZNbXLW7qI0fltNbZ3O0bTR9lB5QyLpb Cdo0EsTOyqPBp5jI6TpU5FeVIUHj6kKfUQNvmC8ha8B3KIn4PnNv4krNdJXSVaIOgfj8aOFJ7a5X DyYoiXGn6vU9tJF2RuW3KxvcTwdE7VYIr6Ha1FcpK2p8YbdpTdpqji81C7zDY5uf9LVjvIW8qKol monpgJamZ7X4vmiQYIumpaVqfI2Yfa1CdmyDru2wut3fqwo4CoeKBbGP2oBiMhiyaUfo2czvXszR XGkxiS8Ow/1tuVelSSFqWk6IFOqqcsinRhIb3Hc7NO+qH3r2kBRS6WvOSGAWiojMmQqwKxkquQ8z NAS4SegIMZohKgtzffhPAFt2XRUh12xQZu3NJXe62R7eVGMtjSu2DjnUgdOPuZw8KR7+pNtQlQLf 400iIElxq76YdMEab1EvkNr8jmiOj8Ur71ffnvD1OyJv85Q+LiUEFCYLbFCI1ENdN8QmCRjEgfM6 u9p4/aA27z/kLw9vnyePn5+/BXbnUC25oKX9KUszbsIqEnynOZINTF6vUtDGm1rzx8lIxqFpJ0zt 7XRguH5GGqgrnV+VLsn048s30SbnZmL9dzWD176HwgunMBqbu5nU6DL6Qi3eXHsLcKMQ2+Sl8stU up2cazo7xnz00InC6eKsdIDaAVgijVejoWJ5+PYNEdgAPYBpYA+PQIfutK+6hFBWUKFwEEYHJu3e aTx4SGVY2LpMRSpF8tV8yvEhE6BqEtUCZ8yTq9XUwWTC+9357JS0TG/WZ68CBN/7YCaTuQfy2810 6etKnsyBwBJ7y9nsvj+9UKxYLqc7J18NFy7gEnWMqI6pd68m4FhnhAjnhueSllLT7xyBataRAPV7 S+1qOqcXd3tv/JFPL58+PH59fX/Ql2aUdtxICC8o+Wo184qjUYi7l4tztOtYrbjxGJRS1jFd//Eu P181m2msscluvnL6jSy8ntPsPUj9cTEwinV1B+yKEEN9Od2uHWnW6hBSIJ3NN95kNDeTulnVPb/9 8qF+/cChB3pLPFwDNQcSR5o7A86CYFMcvLlGiXaLSB0pWWDeUGjGeSAdE/V7uID5F2nqRgb77+kf U9J/IPpaIP0BB3NVJGlQCT6m1ogivzfhI6Kt5fKCJHgiodtHqe+N1FXwLeVBZoeyF6m8kj/VqCHW njNmjAI1rrTW73xnOvT0jzyfTTfT2cZ7RL0qkJCh2wnlEGIy1RXEQ75eC9AmltcKwVmeBd4M/5Oi DEgukSbo2FywQ+rMaGo/Yb+jD5IPSeedQWOkyyHFGsR10BcEa8zPsCjaQQ8PvsCUT3fUooFx8pNe j30xbGbBoVDr0QLdQcy5PtC5DonwgP5UIKZ2Z3DRCkmWDJy+I6nuINwVh0ynSmoEJFEaMNbS+/AW 6BvuY+qtgpEr5qO2PlEMM3SNOvKgtmGijucDlAI0Vla4k/zKo+y82dxs13621Si8JME3NFrVupAj XjXkx8UmQol4Gv+sSSlTylMbM88PopenJHcivRzJNQ/fH15enl4mCpt8fv7P5w8vT7+pn14rM4/1 jZeSKkwAy32o86FdMBuXCxjerVn7nNoiVl5iSYP3YBakx20WVCvf1gNz0c1D4MIDs4Z1TisZYL4J txMjF8x/QYt9by5gc/LAW8KaMoBdJzywrvBKdgTXxM5tWwaQ5EkJKw7RLObnUDRDHUm7uevxMDpg XEhJBBqQXIq+Y03mt8SU8e16GsrJIRZtfFDg9cnOjZFMglJBwuBhVPO0m8AZG1eujbq1fvaLl+M2 IVM+/O5tNDvNjCiuZqiiT1/gY5ldeaqWqZ8Ted4EKy65khCZZBBoa2K2Dsn0GtKw2XtvOwevBvC0 hSP1246nR8x0imFrjYIbOqM9kiicYhT1DEj/jkAFj69FAo2iscdgGkUSMsmKgSg7C/qlGBMu6IWq dp9ebZGqJfico89vj3bXggYutRWWdSv7QshFcZzOcRCRdDVfnfu0qbsgSA1+WGCsfqMfYlL2TJZB n0VWdXVJ6WNg21YKtW7qRCQmpqg5cYftRF7qc6owmQuX28VcLqchf+Cs4kUtD20Gc7k2eKIhQ+2z F6u+zHd4isDoGD6yvfvxxtHQdLQ2CrbEN2v2TS8KTKXdpHK7mc5Zga9NyGK+nU4XLoKH0OHjdUpC mBQHQbKf3Wwi+E0A1znZ4gPTfcnXixVxwEvlbL0JmZA6AaPozWqG5qujtYWDBbXGPvNlM92sMG8l XI/VXKQjTZ9MeuMl1eeSbZebcHQBPofli8/7mKklZun7gBhc9dw58j8cwZUHQqAXThj3rKBk5/Xm ZhVqVUZhu+DndeDB7eJ8Xob4nHhyM5uaE1d8cKLR6GnLKFW9TB5KY7kkl09PJV+oTXBX+6bV7umP h7eJgPO1X7/okPeWNnm8dwrhUCc/q+Hj+Rv8c6zKDsxyfiuCscQODsa7Cq6zPEzyZscmn56/f/ld pT/5+evvr/oeq1lL4fstDPzZGVj8miIWNV7QA2NYnXlGGzXZD2YarxXolQBwRY+nGEyNOSZw+hes hcNugbsQDmtk1hO7jj5xoXehevpAI7+sm3X2bL5MyMV/qjr+5V+T94dvT/+a8PSDaiqI2Pky2eJp bN8aDI9PFqslRi9Pk/OeEQWqtzR4xHh5xy7wXr53in4ZTx1ce3P3xaFyKoVrolpyxqPxot7tzFUD jEoIwMDkfcUdwYmJTkuHNqdrtxua8Jvz4SVw2vufus95DFYf2Yroi4X+f+ghyWQUL0Si/go+4LYu QPc1BKPATKlG1DbhNyg8lt+iPpkjasSsrBsxprkwkD7PMo4LNH1+3iULo0SOegbZ0siCt28g3eo8 jzwNorOq7jq4BM3mw1NO812c+rP6T3dhJ6/7RjIHUtrbMzaVD6j/SRh19DEY44H3MMFvSKIWgIAv cEu7teep4xJ/UADbBSwkCnbfl/LH2QoFmRl0zOjuxTgj0pLJ2x+nfur6ZBr8I5tWVMS/AGccvAVj HwxUtm7ZtoGyjV7ug4ZTOLds26tl214p2zZcNifb2+WZXAWyUDzguh6qjn5D0JgfTAbJOpXFIotd XNNqhzLaJ9IGltO126jAnKx6oAu3nAQALNUCRc9eVXbaYQ7/iwDbGkaQiSKpzwHJZcXjCgIV03Tz KKqjysqdMfs7tWE17Dgfr7ZDLvc82jT3sNhpnLcnB6mSxVYJuwptjnT8ME4QPcS0ZTviupvk3PlZ I7n/q88r/D67UjgvZtuZO1JkrGMBSDWgHYTrNTe4//TlsKLI9BEkXL6XIRWobpWMRLtnM38dOtjh mHAOzrt3abf3xmE1zkabqmjcry0g0Fntg2yGD2jMGqhxyy7K0n/9R9H0WdNEjrRHHfWdTz3v2lhe ZZe547y8L1cLvlGDwDwqGSJjZlKqtqlpTEfTsqs7cNEG6n3UunwZHNeVapSidtcWTetXTdMaJrMr NaNU0nAgNC2/050DjMDu17krWJ/zAOjMu2Yl0XiqAF2CQXvjZMoX29Uf0RFQSafuQtoMuE4TKkPz b1NuyMnfsBByh4bclpBmzRhbYlkb1kyDIf5PGl+iZ3s2W83PbtyJPnc7nMXvnNHJwqYtrLw+k+7d Lci+b1PmDTd7MCzIkw9npV9iBbPiwGJFrmVqujX4HTopguxQpAE01dOw3gdCh6Fv1AqxoFgdDcHV MRvBokpZkP0XNCxdfJ+1LfYbAxE9fZAANWUzkt++vn//+vIC7h6/P79/Vom/fpB5Pnl9eH/+7Wny /Kr2q58eHtFOVyfBiC/3BQoc8GmZqj8+W+N2YR7RAToCaUlRYJOEhvL8sl1U+Xt0M/7469v71y8T 7dTpZ7pJ1XaG7lch0Ttpvun4efSrzstIPSelScNkQyHhvGg1dCsL6lzouAv4NeXRKXXlAmAhETLz K8dDpIscTw5yKNxKPuKzDot0apwfz53+bgEb/TELcswBSJnSMxjAWibVxpXnwUMYrdDhVYzBOlV7 Pths1jdnB1Xr8PXSA+VqRc9bLLgIgmsv1/weIgjLWJ7VvNc6CanV2GK9DoBejgE8z6sQuvAyYuA+ 5rqsdUS3mc8WsbxqqZuHn3SobTcPJWvVUF84aJV1PICK6ie2mLuo3NwsZysHVfsr3R3cwoGjoVMy LFZ9eD6de/UHXbsu/JYGFyfVhiGWWptyJyHJZ/Pp1AP3LgLH6y0QmUtXIor1xktA0KBhbAxlFMta 14q8yNxyHunRpELsRa5LdxX1h6+vL3+6XRbbE8c+MoVjxVgOSvt9/G85DX616CcbjutJzX+cGXcl 4qb+6eHl5d8Pj79Mfpi8PP3n4fHP0PG2N58BMvh6uLVstm4hP4/UP24jcedTHSYszToSLU7BEK+W eBun2tAy9ZCZj/hKyxUZaUwkGgiPZIXBrKOwMTg9vfK+D5yMkshvaTzmmxLpE1bsow6sOo3c152T RrcX2mf4KKSoqzCDf1rqW/DUqV6BULawp34JO+Bw9MlR5WOG/bXhJX4UHYz2mNeCCCTlWCmNaSaW NXNbIibNC3abRZ8Fd6Uu1BAvFNvYRtzxsheOtzpguSgyvLUErHFNh3A8nmiqdZ10JByJSiwcTtAa QL1nh+39QZK4xeY3vdlgMbwhGtSwbcRi2CpCJRz70luMXNsesIt13NyfzrJsMltsl5N/5s/fn07q z//4Rx65aDO4lIZSs0hfk9XoBZZJMw/A5Ob4iNYSc+ZhY5b60Re8M8Hr73sdlx3IiHEEI6UCF+VY ypoOh/t2BLAKWC+p2May/OJj+g60qKFayRPJQVJ1p+kBRLe8gOhrj/ianNrs1BCLXN/EoTfMICYb +CxnSVcRDgJ7rcahJiF3LGF+IwS3ajdforrN7g5q3fnR4SOG1jfmQOC854nH0ZCxsE+LyjqQNIWv msFhTV1QjhCL9el9xUocJwl4KWp9941QVSgIDBixyF1aDhGPW/UPfEOow0dHR+JuYV0lSLNkLQ/8 7tViZ+aD05UPEs4+i3FMzzlgdbmd/vFHDMeD15CyUGNdSF8txPCC3RFYr3kUH4yKgzHkgChsbJ6E P0w3qsgj5DTI0pQxQaGsEi4lmYKuOK4PGhDnUvXAVnaRt8OQArdCmUOx9tGwR5EEP+qiRM0KpVqj c9mx1n3OwjDz9vJQiWiOsaJIu5sb1Voir9LiOXWWwPhf1M1FreVHtbEs/uItl6zjhWGp9p8Jk5Kl tVfkUXK1wvZ1Kz5SN28EX32UOqka5OoDuRr3VOvN3McGXJcxcLISUu3gQKhr75ERlsiNLXFKiupQ 7+2zSKWqEa6+XNaDe6zIO8Jbset7rh1el2oEzpwNJY17J1ZL7qvw6kRr7IOnqZpuhoR2LFOXSc0c 6/cLNUzQEMLa5Ljgq5tlJIrwoLDZhskv6rbLwtdYuvtmX9eROWTI0TjRj1/eTvHQ32AM+IsEii7D 6zK1eq6o87tB+roUasITO9V4xV8kWZIQk+rnZjabQT1H6kg9Sy/TufVXlTw6m17eCp+xdsbnYh5s 78WMdpViFmF1URJ+/bUmshJtFskyZASEU3nUmMlJl65XTPYKJ/jBJqjex1n6X8aubMltXMn+Sj3O PPQ0Fy3URPREQCQlocTNBFWi6oVRdtW0HeMt7Op7238/SIBLJgiw+qHdpZMJEASxJIDESWzmFGxG 08WyNk1Y1x6lHbJc/PiUZgIfmfRA1/jYF2bAOv9oUQ0tqisb9nCwvlMs1w6oCGYDTIq0WX6LhPYA HXQz49UbqXrvInThe+N5jqvgm8AhyQI7l4gc/hJH5HJUBog5nKLd3X0aEIMLq7YMGXIiwGPUQ3sk AyL87o+Q1DGXYyJEuR8u97wRF8vodsgf7v2oXU5+Imydp8r3vOUEykEVp0ntSVKf3E9TPzE19HFP fnRJjDbqJYCbHG+JtvyVGj+N5D20NyBHM+ZRsG5JhORHFwUphwqyve69EU1jyHvYT52+/5l+cfi9 YBkpMex2CWsIUXG+BTS3W7AUKXEsliwTK0rUgPOsXXX4+oYC6DaFggyGGZwpj2vaNs4iilaBgw4r itb+gkg+LXOIH2WurWMv0yhPCVfeSHMt4iC637zRyvNbjWwg+OV72EViQPp2N+Z+SFlWvNHlCtaI NOdk4NOQlckrCiO8LEIZReHOs/R71rpab5EGZ7PazNRqVn/jBR54wulrl3WcJoahME9Ynjk+PT51 R8pAIDMoXfOeDmouP9+RY9qRE5NT5YksxW6ybZfXA3/D7uiPocec3mUsJC5s7zI617/LjPhRcJGh MHlf+9whulCTnqlVY797GfnhLnaLmtJ2Ml9H/mZnf3JC5tR6463eaOwiTd8RdwX5O/Ks6zycisPW HPFy2AVe6L+VinpGcLGzjqdS4O+wKwDxtjjEigrlFwHiBFyaC4qKmpcP1tFK5CK29B6Rxztfvomd nLrise8wNSC/3Sp4q64bNSphLp1c7ZfhHZ0eQwfr01N6kaMT9+LBCWVRx+r9oGW1QXPew/GJyc6X vPGGF9IqTqyqbrnsDS5j6+i4XhZDPMvCOibyi7XlN+np0qARpv9tVSXDb/PmIuWBjnfAb1ifuDUu KsiAtS0mQVRQXq1iG7U0PRAElW2aPyQJx7RMB+rXqQDlQmU1EQ6Ek0POh5Vr4hR7arVVp1vG98M+ e875nUTm3A94yQhprGRkyg4CKXoR9sAV5SsG38FMSKEM+PQwEHO5lmIU6x0ZKAidkiLDuo+iPAaC BIr1QzgFCxXDmxkvIhpp37bIQR2oMdPG93x/ePy0xaysJEc1HbhcANK8YVMDImjtGb4PoFCg/rBk Le2n3W5tP1TPcESRqsJXnKuq24ukj7+OQNm6MhK5DsB5pG5A86qyL8iVEI7hTArHSV4a25sAuTKz XuWHZ+jLEaSkgND9KEEqQWQnfJ0CYlZAHYDvBw5mpQRAeN8YmIryDn9t0MiDFnxXVjR9uAazw/N9 bnXelHiJTsLCOFyjub7JRRh5+Kq0RAinDNvfxESuJfYjjTV7fvr+avRZuGYWs8Zmd4HozK5kbgKs So9MUD4RgOsmi/y158hI/kdWyIApzvI2inzs4TEI4iQeGKXJY3pZl6a541GDRhHn82yTfLfBpyAD DkPFdt22dsnOKjlmm8Bjc7yAvh95tpLDOLNfKHcei22EfZQGQV0kXN+FsVaVnEr3wqxfYBXJ15sQ cXApuAi2gUcxTZVs6NW57K+XlqJpJcoiiKIIt2jVVOJA2mwL7/bILjU+ehyL30ZB6Hu9ETSrsjPL cm53fhpU3skp43q1zuKgchLl/Km8aNZ+a7QEqMkxlAgmYK5Os24geFrXrDPCjoDkIdtYrdqrNpnp KFCzWyyIu31aN3ClQhoYQIBo8JlSoXOxP9ckpL+MY58KOfSm2Lsr5nNIry0pyprtJl577TzvLibD Q5/8moVrz77or2oucmuoFqjQx6Q/upx9ed0troYFNy1/YEyyMkFpyRax+WlgYCdF+/J5FPj+EsPq 6aq5YaaZ+JpFZ+sW9snCIwSo+5zotD4byuvz4lkayNXuYe3KTx2moiOe0+4s3wF/V0DGghJ038Rl 2gJ1iyKKobmYyvo5uGwSZKe9q+S780BLay+6lENs3+GIfJ73qXGdI9XZzt96rgdvzrZ91v0K7+2v wk4ItKcIgOoql6YUaUYcx3qZcu0icnTYEPbdws5lAE9LrBa9SkktPwWdbl3hUpeyrLKlODWOJJSk F5DTtabLQgBd7VbKzHtUI7RUa5PGvO5mWrMy9vhQ0pnA2LxEAuwnjoqBj1iIdqe4raQRbpJ3Uy2Q ur7v9IwFtTrOD/bTIDAFsEeG/j1x5v5yCLrigbCP9OIqa2d5Eauyx2IS3PIakFtOPTCEN6EhoLXI HQT6GgRmXgHJyxCoMCkNDUKmJdqJPr6UF2EWAcTvSuHY3dfyhTmVYwog/Xt+L5Jfsys/2BbZUmJ0 Comsdps1AcLdaj2Y75/+/Rl+3v0Of4HmXfLy/q8//wTSzonfGT8atXT7WxIVxyYRv+YLRoXFWiEW wIhaDRvK9j/BlDZlxN1vMqo4GNDl4iFPiR2iEW1sOBP0EV0OV3DVg2tAyGTL2inXcfWVzLAC3AOy GdybQQ5YVHKYq2n0prLmRRmXTuLwar1yky/VTdDiHiV/rzyPdGoJrWfQxjecraRWpLTsD5E5kM9f N9vQACC9hoxM9cIxDNvWYb8hpfU/UFq/ndOlOBfl1VFhUgfaFSl7q7HOrLkBN9+1XchjPjwjoea7 sopMqxSJlvpxr+buQ8rq3QWOG5W9VCxKk0VpfLEHc8z8wHH+ByLYzzSiD0yns3Ix4Pv11e7BCSMv 9cxR4+P1U87aO/AN/vzy8+fd/se3p+f3T1+f55xQOp4Bh66Sm1EONEo/OJH0Y9v0LlrYM9C4LDqS hTWSwhUvHOViJ08TfMnilGQx/UU9OAeEnjMoVDtxUOxQGwDsFPYTEsQK+12F7xzYW+T7PH/6+fT+ 88szurgRc/nC4obWD/IdWsoWGYfAymOrkQOr6VZeIuJ4ZfyEh1A2lBHuaoGGbfkKmIhf/gIv9Kl2 xZ56c8Lvcb/Stsmvjv6UN7OT4qwXL1Cc5XBQG87u49PIZCIpMFlJImekVUbldB93QLqHewPMiRrZ xTXTzjaClYRdEmwPKwzoJw6sHX3xJXb3vy9Pr3/9ANqh9zOqSJUoUS65vBz5agBdZZ++/vX33cen H8+aIonGtqggJue/Xu4+SPksP1ltJy5UVBJNpPzbh49PX7++fJ5IK/tCoaQqRZde8HEcsFiWZK9W axUl3OtNNAO6g0p41MyyeFnhnN4qZmsTWsNvarSZ3KfCzPQagjEDbtMCc59669Mn8fT30C1fns2q 6jPfdKGZE7DRC7InoHHh7bGPiAYPNW8eLcrsIe+YP7sl3tdyJmZYwtNTJpvCTCDSJNuzC26ZfSWk zT12+MRod5lXWRzfTHB/lqVczfIQcdO9u7AEtwUtObJHzBekwdMh7ixVcN1sdoFNV8xqMVVxlcqr uUXSV405X6Pv228awMeVoh/qLG7WzYyKJIb49EUscP8V5wLVRjROGtv7vqOiMhgvI5r1KvIXeoSs IZd9OyqsRCQWNFSjhLqsijnJG/vw6ho5YoYvMo2DiayFvT9esa9TJrBjASTSO2zGYKgEOU+SLKU2 3iSFf7A/D5XIZ09+1hW3DYa4DLKYxjOmwu99Yq7apA+rxdT0HqehAO2EbAVQcbP4dDyTqxdJeVya 8xyMDLMHAAZR2UnnQ6LKLYJ/bZ9bCeEKIk/sMmCxaCzvcuRHJujmfw+pl7IM74MYpvBf81TqTorV 4XPQ2DNMDTCgue+trahvK5trd+N0A1vjC/mp+we6V0VUcl0VojKhzC/5aDB+UTaBuyXrJLKL07gY A6rMMAtObHCNysalxgETlwvrNDngmGUah4ORIi1nb6SHcQPs5x4ziwozkfSYwPeFdXm1fd6He/3+ 16uTmXEIg4Z/doeDXA3lWYrj72oJXBjTgcYILCppP6dnHY5ljLH0GdY7I1PGT+OhnbqwR3jKKd5V guHDQEMq4jqVdm/7h+8Fq2Wd2x/bTURV7sub5T3SB0t50gdNi4Nq0xUrQyeQ5ta+BCq7MaMB6VhS rdcR4W82ZDtLV5lUmvOeXJoYJe8a33McNyCdwLd6oo4a2fm8R1QzI648P+ywahKpLVETs83K31jf VcqilR8tlSVnTY0ZAqZC5lEYhBaBNFC3xHVhktDjwQmvarm2XypFkV4b3GNHAUQVhyMDYZGJpryy K7vZRJfiTKnApxxl91ktlaW5ZisPn9WPkrYh3w31AjSYwk/Zp/BIM0Ady3CkzgkHp1X5f7ySm4Ry Lcyqhsf2lPyQ7svybJOBLXOuSkIiOEnTjEljNT5ZHwl2bEaZGFC+5SU+nbk16pNSGiNIEJRVsMiC tKZkH+fr3XZlwg+ibVsSIEvBtJf0TxzqqKfoNUo9ie1uGuNgJaQSqssB6VjB5DeanjoJwsSGYptj RONyXzMLfjwEZxtcY4cmAne5VXLhcozIMbP5KFMBX1lsEwmepFdeJPjq9ShscsyuMmWnfMOdArWV 9sslDLDnyiiUpnXNS1sZcnZUFxtsZQd+ibLeu0RA0W2TNbw42t/3yhP5wyJ5PKXF6WL7fsl+Z/sa LE/j0lbo5iIXIMeaHVpb0xFrDweuHQUwlV6s372tWOKAO8W1ZZXQU1b0GbKzbClylvPNka4BNlM0 lujf2qcoTmNcCCziFbjM20THBu9RIsGJFVeGyQKQ7LyXP6ySK4txULe+0DDgaAMFlXwCZV8V22iF CJ6ocBtttwuy3ZKMBhOwyI3DMaIBPjld3jqYP7DmRVoHvI2tYdix4v4SyMVDaC9RfIviJj/6lBeI ajSNqGYhCSyaCdt54cr+HKBUqLCXEhaeWF6JE6HzwOI0bZz1lR5Zxto3CjZcq3PkwTMua+itTI6X 4tFVwMzxYqppdteeztH6cGlW+X5kjedA1GKxJt7cRGhMr+TlirTljuLl5y2Ob4BF0v7ScYKtUvV3 zY+nJfkV00Vi6SXe+yvP2eD+UaO+5rstduI0ZXIN/csl84MFWWiXKXfCMq9KwRtHK1DulKo5urOo WHGPj6BNeZi7ZbxZEKZqdnHLdet1ipM87hoR+97C42vdytwKyXhNylUIIPpiWfdGRseywfGFTPE9 E4RVZ1YV2UI9pAF3Cx9vTV0WfCnvRk4t8WpNDB1TSfc4dx5M3BZqQP3N5WIydHUQ+aGUrV2+OUdI zcB+92uu5ZjuROMHoWOQaCqxWXvY3btfG3ERm+t8OfX5q5mmRumESSRm+B0l2+fM7preb6PEojrX 82R5LpfFa/syvteoLqG3qMFkD06zBYVjFbBFMdzB51nTr+ydL8GaDDZHm2K2S8Qa3tVg76fB/B3l olVawEWvsFCQc9vc79xVWF7TOtcXRYyEN9nLjXsfhkac+547636d3VXXen9pyPKfKjxwvW4ycpdi uAWpxc6nXPTWm7nBFh/W3iYMuyq/zHOW0sjgEJl/u7psWH2Dm8clWToNrbbNQlszV7C9nWsR4WAb 6pGFZM4nsC0zOKo+7xPjJFuf8Axbtvz38s6MIwKDCboJAz87HnmrwATlvzSipIbjJgrire+ZeMVq snXSozEnGyUazfgeUOOBxPFBQ31IHUsWEoINZrR7LhdjtLwD0hVivY4seLaygGl+8b2zb5Ec8kjZ dvqU5+PTj6cPry8/5n4oD2jSH2jGmpoVImNDiKNRc1CwYZ3I0hRNjqerVXuCuz3XDGzTtY+Ct7uo q5qbwEflD1UjenZFmYorxmpC7jZ4uJJ0E9gHu/PwJ5FzPeK+Rq5QJdz/pl8mvsUZS6iHX3x7hC2M 2rrl1DLNX5HRfaGW6ZtdhMX/VsSKZ2SG4JX1gHVHcgxUlI9lzh10i9bLl8MJwZRFdxT2i7GaGES4 BlT5Wc659WqUrNQMDhgHLuuXH5+eLBFF+68QBWuPdq0enLeeA2+liVJntxjoHdPcnoycyxEBjrRK HiRodx3wou4u8nOJP1aRTVzLRsXzdNCxqaQtDHTkfh+S5qyQ7bOsSVgmJBcncKOGoHfk1vykoGlk QcP2GXBRxUgtXnz7+htgcKYP30W5Vs0Dd+nEcgkY0vASGN/O8Vo0WuhbEh2S7iQuMxzqLyNrl15A +RoROG8avfBe5LNc5Dfap3XCsvkD+vH6vmFHKIRL/pYM3lh/xtWC0p5dErhU8ofvr6VRO/ui/NBu WusxTa8ATBZ9Ucy0LTiQtXKWUQq2S6u12m8ljs/VUI3WS67klOz0EPcEh2h8l5humwho8aZWD0wT /i+DS3LGdMmrnMNeW5LRsipcGo887mZkuVhF037qveUDi82cCdUoAFcIhJ7gfXT9JLAxywOi8ZGz lpwSkzIn85uGoIXBJA8j0jxBz2RkEWgO+xlMyLgxTCcOfbticv0Ndxsc1qCqMq43H7VXW+/M4zYD xikGj4bg9iVHqW5FjL0JXdF7FLxO6eFxftVMfGPwv+vQjBDBa6vx9EH8Eaw3U/hO4oFVpWqVhO0L VhzjUwrb11D/mL1Y/lfhHQkAuKBAIWKaxLhzCBA6uJ+mxTRTIUdtHNLSUiF31eQXKsuKkOASzt24 U8d4EKmdwrCfwxoDk7OB7pRTt5BwfrH7oYPslGYQNwwuN9l6jNQQuSbp1WfcQXzHPv/57cen149f 8Om2VGTZsdzzhryKAuXSxAYynOlo4e//+onmmr5l3slCSPzjt5+viFt+bjDozLm/DtdmJSh4Ezpe Uknb0ChmnmwpSfqEdmIVWWOv9irAV0hz45FnIoTeHxAgvl9RqFDnT4FZih6WxdhFa9eX43KZsFvT /CS4wQfFPbbbtOYjHjhzZCwlejtc+7TFFbd/ChHnHH/kn79+vr58uXsvP2Wvf/cfX+Q3/fzr7uXL +5fn55fnu997rd+kEfLh46fv/0nuFAXAKCP4sdBhxKzXQEApPQae0RLpEayq/yHCsfnmpTq9d2Qt G66VD0fJWrZQKMFz2Psz0uiJeR419285Bn+V1pfU+V03/yfNmWCjO1EVw8tMDrmXIHaVnBnrXwTK FazeECdZNqwUcjrNHRk2XFqn2vtJFaZ8/SiLNxUYfWizsKK57J2Dkr534I4jParAQPKGihy8bPZS RVxdgfjVGYoQHClh17YelyuyvedPP+FTTBGMkMcPyVbbd458BzKVLwS8NGCbZDcK9+2XZJ7lW6/L MjtlFyiU+iM55bLBwgUXe5sFhUYO1Bk/HMC0NJ8+b7mkOoduSl8juZr82T0KkcaN+CtjbVeD67yu 9lkly//kvOd6hSzdBC05sTFofPUsUwk08SCKF2FV/RMihzy9fvsxn66a6u7D528f/s8UpF/hZorh IXb32/+o/F6/yfxf7mQPQhl809cq/wtFc6qlia+IXMqq5/+BhqG5kYb9nRFAS3HoCyCwNgW1Oagi zNo8PtW1YHCg8sZbDfnLl28/ft19efr+XY7ZKt++o/83rjyVtI+QMg6arkfwJtpufOxUqkqF50iF PLTReo2nFfX4l7+/P3197gswL7lnZKJQTJCrHdZitluHcxS2Vsn8qIvW+mvryaveJzddshQ6efn1 pg1/o/g6hr2Rixznj1dC5tkX1KyrXC4ay9Os5NOC11X6OolDHalp7IT/pJp98/lxFYTCi2ZwGEaR WTfbVeurFUpvVuy2bzxzOIsdWMHK5JLBnLOc6p4Vj12DA3xcffx3px3AVZb+b//+1Bul09Azaep5 Qfkj4vitkyQRwWpHhh4qi+zsfljJt95/njT6eQEXV3x++tcLLakiPO+AxCgn5dS40AtT/HwtgDJ6 kf35kwZ2uSSC0HcJQufjwrCL6/iNR243niuDbeS56hTp+EsP2L8LtpR9Dlb7imgkIwxqGHcaEBXw w4EiWjmmojGxPYNp/2Z5OyxxvBxRsb0bUQhsuYu9WEgHVfI3rhIsaPsDnV8u4WgvzzSU240lW/As 2ZLNA0OCidBPEDKoHg/I5hIuKkhDdud7kcwukkWwvPqgMQ3ks8RZFW2D7UJa02ibnlqwI2XFmelI Gzpc2TI/XQm5qPop12RkGaPB3pA+8flNpEKH2LSY5xA1tKxFx6R1ejleavvV55lWuKyWbEN/9ZbK 6p+o/D9jV9bcOI6k/4repid2J5r38bAP4CGZbVJkExBN14vC7VJNO9ZlVdiu2ap/v0iQlHAk6Hqw I/R9iYM4E1dm8oFIPqI66YVvXMeTHdQqRGgjIhuRWggfTyP1Akfdnl0oxr/N/eDbuEzkfSwTO6sl ABLYh9I8jjwXy91twkrckfEi4DoggYXdksYNb6bhbj3rwujnuggbO3clGwWNPMf8Mj6LWj6sCm/B H8tKlNvYTZxwa0YKROJtdxgT+nFIEYJrs01h4js+4hMErkM3Uc8rLoTnoAQoOFuyxz4V1tMf1GKV NaREouV4pzjwXnCWxCb6Rx54WAaE/8JduZJ+3US+GR+sclEUlw1RNEbRBEMTNLUETS1BU0vQ1FI0 3tRDUR8tQZYHbrg+SICM54YfywS/IBO5q7WVhl6AdquJClcCi6ujLhoYqMiJPgoduak1dLQ+SXCZ KPLTj2UC72OZcH3I4ksg3/HW62zP8kk3r8Al9Mp3d3kS+xHSjoAIvBgj+OI1Vcq5ayybYnMQesNc pFlzGJs3r7vMxmfRpomi9VZWNKUb+/FKbsom54s8pPNxwnMdtJ9wKrrznLXGC6ZUg7jBPmhmUg// JmAzP41Xv4sP9GE0johFJFzU+4Xo/OiDmZ+6jhuuzvyUK/8JppdwIkY7I+FFmXhrJclumjxEdRrW dFxBWs00Fwkc90OR1QyAnfO8O8z6h0lGSYRMrgMD2z0YDiZgTfwu8fnSqsCJ1Ep4BVYyglrXmIVI +JFIHScho78gFaF+ciUZ3gRvtuhHcKbEKPE4jja55SDIbEp5V/2KGshuHdfFNFgxRBJp12sG9BXR 5czeQJYLI9ddgYXYt3fkvj0w01zTw/vj35/P/7a+7KXtliHXBJaboCZD8j8P4Kj1rpD9YhbD9LZR g+uqgeNME415Z5/Rq/cT0FUTETO6GR6C2UPlAdOuBKPhSuQ058qlBmX5cVuxLveQzwGzNWbGqyx2 HB1qCO1lK0xbvsxVRSLfcUp4F6WgPMsIcnHzoNl/Ba3U9bZ64QCsl4zkb+ryZVjJCefsahbEjOv6 Krgf1OKNHD4H6EXJe5QWFx+9tBoW7gL4qO+77mgyfpzF08coLrvArjFa80v3VmMCNLWgh52BJnFs iqYGCF5wPpnNp+z4ZOjLzUexUaXle9kc/9dfD2+nz9c+mKuGU+Aybm62SB7ZdMa0mJ3/IBougUVD 4c1LS2mViXtk00b4+eXp8W1Dn56fHs8vm+zh8X+/PT+8nKThQLYGAVFQsAEq3UMDE+dwPUS5tUyF TT8wbConabJaPLPd36yvip0RAO7z6DFeR1tFBGvzkPWialfytNB6vNajVODE5Z6LVV08YlVI2jXM G2JUiTCg93j+unn7dnp8+vL0uAEHq/IEBMGM1tV8f35/+vL95RGO2ExPIktr3hbGnAFYknRNEqGn P0DTJlRfzV1A4xBuGYAYnMXTKlcUWkArL+9q7K3FfPyxuEa73j/qaOQ6IfomUDx6kW9eSs9glMwK 3HNxtfQikETjukDqOh8JePCFHwtR1I7NMsvqdbTgDSlySx0tIpotOfFEQzzxaJTJXLzbwJoC5Qpq ac9/7odJulJKuK+crqnyaXN81NMDNDTKAxOxlauIXJxEevbaERvjXrjOp/EHfGLladjY3uUCfdck 2jPLGfXdcbXBzCJrxQMiofNRLGkaWPJmHjRc3uYQ2fTABdWPAUT55C54ecPNwGdjOHds3PvXJ7iz gfpuFwOWdmYs2hlyx7nucvV8HQSvJ78SKM5M86ZV7CUDsRzGy0VXu17sG6PSpTM7qb3REcK8JF75 cpHDPGRhgh3Z9OXuUBPN3PcFXLM9e5XRvbEjMpNvpKGtGb6deZWEK7kHcSV9Tw/KFdSrDBj8EoYf ZCkkVcvEcRUgReinCZbEPJAgTEVrrhm6BONgJ42P8BYqiYLUSslbVCqVhh5G1R1XJV0rg4aRuiFS XNDWPB+9La8KhY58fq1z8WjlvADn5vPCnyZ1GdSW1gy2cy8LmK+SYvL19PnpYfN4fj1hl5KmcLPr iRr1rjFURdkelSv6EzQEtWc41CiGTPOwCoji0nwKvVj71XNqKE6TOCUkdmVDFQp8HKTc9bmWg7rq c4VuZB8DVc+zB5f89KeMPYwPFjxC8T8GPB7a7u+tCdBPeBiyv29RpuEd/DYrUA4+9QiwcjuUY9fn svg1x/5YolaBOVEpFTxlQLN7DlLgwqTqbbFv4Y7sLZ7A/jC0TL35DmVTwhMWHw9CWV+S5pNWj3fV Pmv3BeRDqe+6bTswWqylMF2LQo0sTP2CMEYUG1ECZYe9OrJK8DE/UAaOGkprpJMcrcDqrt6BJ44V JPFSZ4WUxxKNdDnrWtk0kU+UFLIkYRy5a6QlZMM8ZxztXORYOS+KrJzrWzIDZuhcS5xj7jnyvrTK qbZDVC6wcs1Y84DyKazJxszC5kFAE/nQQWHJ6LmyowyzLt1klfWwQVnDssMWrk4iKHhzoTuEGBpS 122OBRkyPcmM5aOOsaEsxW6Fjs99R4vW00bsK97wMUt+1XJl8DyCdU926Mt55U9VF6cgQIphRX+b ZCa9rKn24HyY7HfoHf5JdH73AldjqHj9ok90MDZ6163t75+fzpvPp8cz3Cj8b7Dl/Xh6e4Pbv2DJ 8uvTD2TyY8O0tNQLsyBx4HsInCbyhacZLsFUYpijuGeIN7TzFRVjLl4a+rXvER3nCkksH5hfUT81 2mjnxbTpLndC+4JeSgSZ9yPpmuvw9Pl0tgpnLHFTs3kmIa43uA4OI625C93QMb9j9D1RclLeoBYf lEpGVZnYSCQfvXCqtkkl6srd5j8iytMLHhEP4QfXq7XDqrBIVT7XleDEqDjeSXgeHRy+3JnOH76e Xh94n3jhDRh57T81JJYOk1E3EWb7/PD2tyQrZf3pK+8N/zl9Pb28b+DFjhHVoSsirhG7RuubCHGx Yiq8p7fHE+ymnuGx1+n52+kVj7EJvTg1K7YapNebfEidrANdRhTZLeJUEzWhmhNIicANvwobP3R6 vSZBTRg5nuaUkQ1OpPvq3NVJpLnUFLEds3L/J4ZzoBxRoquIixEl12QoRsBLxa5CI/ujhLccf+hu DWeyBjs8WV5YHV9Ocrc8/pxZ/CbOImChiGA5aEiPZrrp09h3HYJnbX+XoHuwV4l2COXBRSFke28a cUTD8FW6J79pV5jY11uARLku/gG0DNBrCpLEPtW0I51Da5QvXqoxszJ/oAz/FzqOnXLtVGinIjuV WKnImpYbWgrjz9SSCyByC+Nbig8OoQNLnUHHTpz1Wjvswb8uFjWLVOtYEtNy5Ws1WjYkoY+2siF3 pvcsSLwD73PNarxj1U8vuyuGRf4p90dj5uuGnkt46gmkRJgPiixyXDm162kFl9JTLhsvMfMzoUcC /j1wCtGCZ6bPdf2P1TvPRxQpPnm742iBkXgWBsvWwk350kqHNZGDuu6d506aDdXsuEZ2icxbF9dI tM4D44Srj07dXa43Fw5ZD+8WHp0V5+/hU5KWyKfej4LRaJe8e92Vmb1VUs+Tt5VhLodzTmMun/Wb g6Nca7pokAEOigI3Vhj5iFodv1ZVkoTm/sFsvwN9QClpiOrtY4UQTWMldJKY7SZJLA2KE1lN8tus 6guURfqAwIfOd8IApUCn1YmbcqwODV+g8tVWZSHbXmme87pqYMaW5DZD4aa8Lw1szFBd2A8sMFpO C2d214VBiklQkZHOp7YnexQ8Frnv4Ax0C2WHRSazwydbfK4lSN3U8nGUQfWqiVyZJwONeEFbBmre 8SpKLq/E5w5olV0e+nwo2PCBl/9hchfLGIV4L0k6pnT5CadVEDuaO/gZ00wbqJjQ1ftEV3IKmsl7 jyKJhvBZkRS61QCIUTZIfM2POa3flvj27BRm2p4LjKh4O4ud6EbHWbmNksgzYORgb2Kmg7hlccVO Px7eNtXL2/vrd1irvW2AT35sts3iN+k3yjbipoxidOEaWYKp2nBQABZaLjWVyAyYTobTZrEpI5ts gwMHY89numJz8Ro2HxNMeRJvZclkAkFaEDK46nPUt60gejKoewVX7NjSQs1Kdih24CzHOAKZCAw7 Ko78rjAZULiD60WX57SXTwTb/uZHQThtr36GrNPzwhOlbV9CVZlo+XiUC3nckkPN7ldiX0RzNAkn xtI+NOzouAiRj4rnnAVuUmVJs8CwcnZNmByiyEW/eFexwcSHLnaCEMc9LNku6eitie/bgavjx1kD 1kjGcLxgjC+lDybRdlyJcLHKIdtUM4JriHQ5G4LQK9fqDTxeIxnK+ZK9390fGcKR/T0tS+wjhtDF Khqy6obYR+RlxJXplfxt+0q9IbEQZe5a3nJcUv3E9eR4rUdwAXkHbcHrpuSKsoOlCtdZ93zem+p3 NfmKpa4/roo0Y+26Lt2uCvWshpXIwfIhokWho9HETEeH1yuifEIiez6HFezixo0+PbycZ994m8fz 12/nF5gGpJuLDZ13zAYtLpgAJIdw1+Fr+3z6kZ+/VdXmt8zLvX9axjLgjtu6HPO200ZHiVG9Ghss MiAqPO6p8JpXrvrHiS2DbFpQakG65+/vZ88Shq/vWeuZgWjR5Jui4fNMaAk5kUhpPr2ewC/w5req LMuN66eBrUS3VV8W+vg2gxfPXVr05cPr+99v5y/vm465to9irpmv5uHlnZerGaIhe1ZRZNqcCBW7 abtOnCJpsb/sTi8nS3b2YDzEDPL581+vT5//zQvqH+3j+/nb97d/KAWlNJKimK7HrjQN2IHYvp5f 3k8vn43w2xI/Hr5egXh7f3j5DE4wf3t7eD89Pz+9n/65+SLFJ8VGWeYkaaoWDQcjY0iFbabU+WGA EW88P5QrGNf0H4WFmf/a8HJ4PXFN7+nh2ZqToh+1eW05h8694mJ6meP/or/yYXw1Gxj6lgDl6zWi cTDf1eabTzX/fD/CQL2owhtXOfdaisqT18lLoTpYoXpm8YtCxYrfMUoocRLfLDbHSSJT1NM1lqGk 7pjq4cXRcMUK18jufDtBFK2PxT/q8sRsSFNwLXu08pR9LNEiqG/koMmSiLgR9sWxKzcStvntV1od 7ZIk1osasNHItBcjX8JBre3wdlyoSB0FceJiWQ60VPYjM9sIb58h0j79UKsBrvFCeTUZDucGDEZF GhTt0MxqDVroV77ZSv0o1oup8BLP6eXqyefBwVox0DQTz8Ey4qFl6flY14ovR+aM8jT3fID9e0O+ nl6fHh9efr89v54eXvha9NJQfs8no1hssOaMVxLXmrWaa/tQfb+3gK6v1V2WN36o9+56VzDf1yOd 0RAZchxtzCCHJPQ8RPB6ZlrR4te7RaoXMm8ZCd4bPedij7N4+vfT+8OzPN5vzi/PPzfv31/4VPh7 V9e60lJjIwws3pzYsVLp5YCblvlignM5Dt58Ob9O84SaVr3POv2r4L5poBewAFFJvY3BDStfrzOa 7PSRjLCsSRLfbM1RFGozajV6oRNqmhQP3FNvKv5p9+R8fn4DI238m0/P52+bl9P/KXWpqhxwiZgr Doa6sHt9+PY3PCAyzGmTnWS4lv8AI6JRoEKanWCAqKxlAaDYBh525Eh6yTTcDIgNm113oP/jRtK+ HCfpXcXym7JvsXOoQjYexX8cb/niYLpJo2jmnKlbUhzLoipAKW3ADCAeH1++aHGyQvYxB0jvypv7 AiGF4iz3ikGWpK0dju/bw1CSgyo8g0e4XjhZvsTCTEuKRZGG5wLzNY0Nb/D4/QQIPZni5cN6pCY6 7R7Wyh6ywBNldAIklR/gL4j5bc2OqGJKcwBAaQ5CggxkV+rVNTR3u61lFcvpXQPnwlb6UNSokf8D vJ871DRrlSZJGWHGxn57YMI/yHSn+cpuM9m6/9SURMOSYNDxlbMsjmRty2BOu1yvRnLIxfItLJvq ui9zpsQIBF9V3vPkiEFU4N4xqyvl5dvM9eCeuBrLGi6QHLN7VuIp03t6TfmrRlxS1olryurnbvkS rNrtj+W+4Av/lRSna3ly0LuGHXflsW87iocrym3Z85XlUX4mxfGbMj9kWhZ59YLpSTWFhsDTkdIS PdIHIQy8S57GFqoQrKpFAbDJL8h0O+n14etp89f3L194j7T794YaqvpePpbnUNd4+m9eMduWz3yU o3ujZdxnZe8pU7OMirYoyxP5Gjv85mMAryM10qqhTEV4ScpqN0cO0Jy1ogUIL9ZyWynx7QNZW4Lq 22l15xbiRZIC7nnvrAgCzQbf5LzMhG2H+iqB13hfDWpCACDJCHglEcHLSciBqzhw8GB1mTihbBAI KpX0vIeCI5+9/ExKNGnV5OMFOjbg0HZfHRq1Oc8keCz981Bi3A4DFT9JUjxkkK8Gw1drU+IFUi3z XWFbCc30SgETdq/MyBfIUrGc1IWPuSFysRNb54XJjQaEp0V9tU37c3eUv3CaAi1jZKUODvz30Xcc NQ3A3FDtrZXRM6ErVDC6H7u+zS03YWbBcbbnX2V8bEBPYKD3lC0f/Su1Qdzey45JOeBP+pOcAkBH kucW32+LRIVaDocctm3RturoMbAk8tTCZj2fyffqZEb6Wy0zXeNbpmLSN9W+1DsgYFzLIM2xHITt j6uhRJmctuhs3yfMXODJwgNc9TMaOHeG0qrVatffdAuM5oftiMfM1SKjTYjngIi4cIKSab7PpG5f gnPHtlFLB1aDnjZgz5i4LrPTZqKF00eErOfaOr0pS0WrKMf7fUsn9Qy1DDr3P+iwphEOAKdrtJPH FeldF2fqYOs4XuAx+Y2FIBrqJf5uKy8RBc4GP3T+HFSUd5bUl/csAGRF6wWNig27nRf4HglUWLLj L6E0KiO/UQ6GRBaK1AlwJRho0lA/Src7J1otLKVMfpr8PArKdk8Xig8ju5L0GGWZqK4Clxegkt3X hRN2fFYzTckNkd3AS2GLLkl0c7UKidqilD4KecQpNa0uCcPRYvJ2EWphe+gDmQ5c5PSmFYj8/PJ2 fua64tPbt+eHn8tNCGN5Dmvjq0speckMnsaECSCaw2NMSBJb6oodAdNfGR+6+ES7BUM4hqsjhFz8 eXU9Xwj096sRCW+PYCNemf744ILq4Rw/Jj8SKcYJ4TqoFLxud9hYQNuDPGCJn8eWUt0/l4Lzjyh5 b6hk2yhKLPtC99sDUJc3KnBzV5SdCvXkruGapgr+AU8J5aSOtOSq2D7XU+DwVF8qzPNdNodajaLh K70eKCOTVhAODneV7DVRyYsIp1A3PVIMkMmZgAUaXNxQQ9lu3ojA02h9bGs+bnWVSg5ln7W0nLUW xbaLwlZ7hvv9E6nbTMBAFBd3GhJ4m7MaqaLprRtv2Co8VzCUglZJXe2DN86ZUfLEuWDhrBmnGbkr dQmJn71pmik33SFwXN0TnJQlFR1GEwOT95dnwHIhmI/ipvZEO0sukeZL4DmtloeqN7tJwzr5CtME UXnPamp+k2M+NwrlF1bXgtCaPm9tDdl7Y4B88WxFHfyA/VwhL83cUTKSGY/ZJ9iNjgXVPljcuAtH vVtqHYAUbpKkeulR5Tb4jKnPTyewCoNQ+3y+5L/ptHLmQ3M1dhgm9j60QY4cEmWXcME8BPN17M7T gE/M9xU7kByE6xGj3mUEeGx567N5NAOpnDiuvNkpsKZSrnOInjDec/XGbPcTroWngZe4BqY8qLpi fG10J6r7p8qFoV4Yk20U7WGmINi41fJbkL4megnzodvAanJvCk6hAyR0gIXWQN6i/5+xK2luHFfS f8XxTnOYjieuImeiDxBISyxxM0FKdF0Y7mp1teO57Arb9WZqfv0gAVLCkpDr0NXW9yVA7GsikxhI YQA53TXBVsfAX+62wbACRbNPuOyICxuw5VpYAU3RmnnBeoWBZnjmpUFiY3Fitk6JSnUX54A+Cwkt Vkf7Xfwgm9AyXsBhsjHv7zJiXscsMF/WiCfOEys+57/HocnL9YoxlgPBl0MCcKRSflK4kNe8KNuc GEB+93CBeY61vi4UVMW9DWpPZBET09MEWezzvZ2IvCYbWHpKfgJDl3tmi8lYSM0npa2LZcUWnLiX Lh6OXBzULqsKFzefALtYeG6pHc4aPAHbpNfYwHey0niKM7viAYbFLhtCi3BNmzqHz5o7a2oExJhz +A7VW3s+ApqDlegJybjCUSPafdNtPWkDQWuEZVMSR8MrxziMw9xYT1ckZ33XBGZEC24PDPrC3Vqp 1ZUfxebiYNwZK++uaPsiyw2wygPfgtIYgSJDDt5j0kOxMbNnHafJJRtJfHMOnEFs3SBOnBrWmItP 3zdScV/dWvM1M7cGoIJpAhMylwq1b+KZg6qA2ejf2zAlBblzwGd9Rq2WZWSe75fOoR9EYlCKdI37 nAKPtGbUC84LICd71zMW0cQZnC44opfmNY3N1dgK9/FGK8tEIap+UmXLoBYgHYzOlWORfOKRwwmz uWM17YYNQsB6N4hRPA1GDF8HgedhBBn4fB6gzCFZoXExT7OqcCb2obcKUVyzbafgceg5cB/H15h8 SSPtYFsjAiQmWKmWkecgsNIAwlwKnXEsotFfhVgmPn0O18IqpbVlBLZEPp2Rte8hn85uNxVlgY9Q FRnTZJUkLiaNHEyERMb7BN9HIXF1PS9dJO9s25eRtcsSTJsVtwjMKl+zxXAmeqJ5/FvwvjzwfJOC YrV+IeMrZGBtbHXa+dWM0AD7quk7UCFizSvWQgwZ8YJwdBAJEuL8OswaJjw/Wk35wcM4H60Kjvs4 rr8UueC6HTwFjzwH7gc4nuDya8/Dcc1JhoonOB444gkc8YQO+ciRzgjP13rtkF/HOJ74KJ5ggyhY N3WUf7oKHLgjntARTxTheOyQjz0HjucrXTvk1450rvH6TRNHfhNH+hNHPKkjX6kjnanju6njuyla 7+kq8XAcS09ZhUm0xofrdewa4ddIiL4lsRes1Of+8nQfbiZy5CpAW6QLsO8IzYu66F0XCnD425q3 BwfjxHKOSjVrI7FKO36XkHkyKVNMLMFWDct/yGN+MymaoXeB3FdB4llFAkfCsT4bCpzkfM+l6vgw GseqroH8PX1q7i2MZvDeNcTQDWG5stsTn9qTXTkQ1QVFugbX2BdkVyCXuLsi0yQ6clxUruDhavH8 /cf77BdDN6dvhZxaVWNvlpAmJKRaMdhuf3i64aittwXhSQiP8vQ4Ce0G7cjyDE63t5iNdaBbzYAj QJu83Be1joEObHdvYgX/ZYBt12TFPr83ipLet13ODJCX37apu4Kpr6XOGE+zVvJTXjHAtCjyz/xb ZpZ3TWlY6dRoHqJvBrpzlMj+3iiPgZaNpmYC4JGUvXp/ANj2vpMXjBraH4t6Rwxwn9esqLeGvVBg Smq5hFFZnhK74hd0Uo8vz6BajgB2Q7Up85ZkvqTU28xiyycwvLUAe9zloMhp1kxV0K6BS189WRVo bnS50UKqoewLUQE6XvNRa6tDvPPle6N9kRq83JSNas9DAa20tXlPyvt6NLsfbUCxCgM1HVsVR1Ss VFpT1NIIPrCp5Sz7ScEnE0ctdw2lpDdbBiMFbnxWkhUbaqP8mOyeeixtnmdg3MgVkWmpSbTsLs9r wtQT9zNkNCLRAe/bvDvw+h9QC7LiMxWfBfhIrn9LRZF4++LQOOLjnZHxjBldb9cNrJ+vVy/OCxTU ai6f867R07Qgtug93650psUh4ZF+R9jU7NSLNI3K3dRoUdYsNFzu+PA4dlSfW9RiRG0maCypaz5H 0lzeIQnFHGa/8xSmCRfbhDD5vXwHxyHa0w/5LYjAoe13EZB6QG4h7fLeZSRp4G27zszcNr07r9mR Bo7YrOI9inLfkFvTqsmZcDy5htLZvby9g3INvBN6Or3aXrNEHLHwDr2jmfkJaBWAO9Kaz7SeYIF2 oPe/G+BdPML2PdQy47O7EbYZB99b7do5MQoDvqO9eMRSKdxKx6srCQWJIPbtWJtLDvTKm/Gzapqz KvNflhRxbkCjTzwO4LXG6C8LV7h/krMoaOKX+Sytl+mA1tLgBUh5sDLxPKyMzwQvS1dCpAw1rO11 Cbz24utcJNZKrECzzl0OXV7njAiDCDuGNvHZHxx9enh7s9esYmRRVZWEpQ5QmlGHZtGdMkOqv7jU q5s+/68baTGt6UBr+c/Td3hxdvPyfMMoK27+4CvwTbmHYWVi2c23h5+LddaHp7eXmz9ON8+n05+n P/+bJ/6kxbQ7PX0XT/e+gZeBx+e/XvTUz3K26TYBfzCoLlLzRaCr4pa4SE9uyca0nyrJWz7x0qbC yYJlvmnMaeH436THKZZl3Sp1c1GEc5+GqmW7xhErKcmQEZxr6txYA6rsnnSVI+C8k5h4EdGNqzJ4 Y52GTexHLquUA2Fm2BJs+RjayJet3beHr4/PX23rwKL3ZNSyoCWWxHL1eo4EVCjxnjEwtvbNahMq RYY5w8/aldNFzL4WVThSdBSuobH4QSM90F7DK5y5E1QougtCD2WOO75M3eVWU5Ms2CGS2uT57OQD ibvlc8+IU3PtVwlK57ptPoW57UE9Tr2YU8mh1ncvCnfg42yHMkVL7nACl8+zrTvPC8nXuFajnhOZ eH7gu0aOuRWIJwCO1B5xfBhQHPbwfF8Fd3HX+Kthq7ZDG+TCD4z4yccS4y+IkF+Q2Xwk46UfSnyc GC89fixy9ysyxUcy4cef4iIlPirsS1bgtddsinJiFG+oFe2nwQ98RysVLzOuN9KqYWs52qERAOtF i4nFj6PSnARoHJ8MmhJvvTU5VI7G25Z+oB76K1TTF3ES4UPPHSUDPmbd8UkF9msoyVraJmOEcrs2 xGNk5DZ3FB9QvOyyLM+uFx0r8iqv+Uja4q2A811H4Nq/zE2r8IvIfbVpSpTqC8fwvcm7T9KHjc2O fJ5q8GI6Hh211bS9ZfR3oaq6qPPeGYw6wh2KQ9OSe3QCnkU+OyYSxgZvhTdH+XZeWQ3oW2h0WZBX RWwYEOaQb0zWJBt6u+kdmDkTdkVjWTMv8y3vJkdzDV6a+5EyN4BlIqb3axoHJiccURv7xkwcB+mg mJXz0qxbcSid8c1iSe6NfBUMLD1vLfv3ZwJU1hwtvzTy0MPzoPxQbDrhI09PbnMkHS8xA+5z1pv7 YTgRvDUm1/uhM90UkN5s2nAShyyC6Qhn9jo25GRb5lYUo1jTG+CWzFbxdNPnAv1gm6IKwWv8nP2i qOtMZpY68KX1JK5LfIRdVu31UE3yzQ7jcvr2c968OpNTHaYhc29fKwjtu8+8qiyKgvhaDLQQr8FB Pd4pw0cb31/71/lk5S7TZj+4Dzi2/qp3n2SJB1Xukxc4xTTa46yndo5mro2BUJcHhrzfmebVd1NX 855qntsAkedOI/tTLY4CkAZ15aSQr9IpDPbOxgYN1rVNVznRi13HfvAc8RKHlujzhKA3LXh4IxPv zC560EEzKp45N6zorW/Vzb6wnXzAEcpUOTubvFsxoxIgXk0LSVGvwFJCu7qUULbZtnbaAEXeQdky cznqPmQmaQB/mSHh1BacEL2enh7AJOHj81+vD2Cr+cv7j9eTPk1y2Wnoi9JwaiMc4x20W93Fx7Fm aHfxcKyBtbSwOY00Lw5aXo+Yd4aq0l7V859XRlpg+Y4TVmp8Dh2q3ClFy/2W9IiZxYr+k2X/BEOk wnm7fZ4MoYVbdcXx2gItb+ASm9mIo3/leh1Mnc5vuLSUmV3IStaHJ94QC8t2qpLtGZpNn2pf7PiC bQd/4cW/BCz720qP8bhhmZX84raC80FHVK31dZbJ71PmCEM3a82dXyU08nm4qqJmZIcBTOI5IhrY jqre8irGl6p73dOCxBz3DdXp28vrT/b++OVfyrLSDu1sgosga7vm3IisqN0VbH9KFHeFD+1noU/i zKyegmS8LtgZU/nSZfPjctW6XFnzX7NXq1mvgiNYmQjBsgqiwPGgHfiWkjRCT2AEPTu11mJsgzQM LTCKVONuF1DT3J/Tnh/A8n5Rur4qEhWNVkjA4yBwBTM9IM8g9fyQrXS/HoI6++x1RbjJ+MLGNyIU Lx74pijUzqRlhvsgUlW+ZALysoRF0qZp9sxKQ08JOCB2V5D18F0L3RXbrWJWH1pC//r49as2KKmi 5mtrjdvlfFDcaKecGn/RFcB5pLUs1PxCZRLbU5HWx+/vYBXy7eZdJhjy+OMZDK/Vp/e/Hp/ewfDa y/Nfj19v/gPy9f7w+vX0rnktgCNXxtz2VAr+b11siPri/IIJp9y8HV4h5QeuBM6Vh+0KSbKsEzsx htJVv1MdpJnM2SSdzRctUZerCnOnPrhUk1KxxtP27wh55Yt03G60PmxyzmfghqAyZFTlGOqVg8Ve o8bEtMJoHJkWzEQrRxkCueQY+7IiIa6trieDda0jB5zpP8gCUztjzgerpSt3J7hZgeuZtx9/vP18 ez9900Z3kHUW/UzCmwFwCa1/QXiplo4ANbSS39dRaaHym/FdgU5514F3+fpTLrYwrnTk60hVyRdY kfip9khborNpYv1jRYCPgIIc1VekUjzS1M4lttbNl8+CK1tQ12WfAwdIqphtcHzxpdJT8fz5pwos E7YC7WjfsHscXIzH/OP1/cvqH4qflp46Fkic4fsJPmj+9aDd5UEIPmXdSk9t+scEDiYgEFhz/aWi 01Dkk24kQ6SqOywHSWdLmpAm6wxwEc6YF6h+FlVc1e9W8HjtY3jgBwhekTHWFOAXomMRDbCYClZ6 vvpuRSd8JMjI8ciGW3qbROoJokYkCFGFXp+sXPh0zHq7uDd3gb+3g1gPSxaC8aVguiI2cVsFHhag G3liPRRf+dp6amHyKlj5+GrmHPjARVLq26b628JoMVZYKI00cPS5swDeeAKsogBPV67Gg1R4l65X nqNNhYmjdYZoQ/Cl0XyRz/bp4f2vl9dv17sMrRpmNwOeB181TK/gkefheBSg+DpEssz6vbfuSYI1 zqTHvgx4EOF4lCI4q2If+/TmLkzQWmgjilUDNK0VWtiL/SlR2i/Pv9F2wMs64wtCqR+oLg4uqGP4 5QK2krzcKlSFeOVA1ZUK4EVD+kzVCAEvX5OGtKD+rAJgKievt2BC7+flw/TpUffyUsMhdkXUHBB2 X9OpHwHHjfzCQ3z0CDFTF6tkGJdLjHNL3LOVp46b8rcwCPP76n+DdWIQfEvEg59Pzekt2UITDpXF 3AWbOrAk7J8fOwzaNVXRTFR9IQhAqzm3WRDQJim6O50g6ptKAPh6jDaqgUkRHOzsmTbbgODlrLkJ 7XrdFTz8nrqi3l4cgfM13Yvd6mZR7XX2jG3A0ISuUj8zptrerMT65fVFOIDZ/fx+ev3tcPP1x+nt 3bZ2JvWZFd1iqd8szxzn5I6nZ6e1NLBmuKTtpwoy2oFRaLLNmU7A4iE/9HSn3IzJWOhe6rhewFum y0jTmTOjf46vlmTahVKLxvH/4JLmbG5RI7d1D4sc7TNbvmfrRUKFeQ9FiftYNH25ASE9Fl5lIL7k QYut7YoDrQyQ0LYQiy6+s1Rv4lhPttLc8bLM7L1As7BSFtR4ZDWwTdEKQ2tolz4G0VBj54E8Yazy 9cNhsDSj9L8GDFmo35KIc8sx01XB6NmWlRHbtJf/12wDzdSmUQtvBvWLyRmcdSgsXB7A862Cb1N7 1YbLEo/ia9bk+P6d/wMSK3Ues1hvFfjXaO1qGqFVFTWEVg1y2bRmJsmi/dXVyP3rKdef+lo03xB5 12jPvxo6uh56RDMmvAPH2vpc59Z+sHJzaezkQrQoBJd4aB0ILvU87wqHpROWIoWnnambnH5gqrMT bt5oEarakoIIWneCbKkfxG5O3bEYXOH74RUSKXmw8p/TJdkWnfXmZn8h7msiimI1ju7MDnXRjKPd N7d8+Nm16uvRmShoa94kLowYC/zVntqJJHebhnSZ/iJ+Jj91gT5hL6Nczv8a9KvRpUw2Yr4gDGlV Z87FqDqLOoMUr2Qqd3QVFl+Vh1hOK9DevrPgupjiSH1qr+JIHwY8XuH4GscduZNMhTBdn0XI4MMi 3273LBag2QAZ43uhunW3PWnzxTl58PLlxe7F2mOFc4NifP2LELVobNMaHvg6WejboYMvyaalDg6M XSPM3UDkA0ty12K89BOJZzLrU+0V9CVINowO+JawHilvSQpLYO4yP1T7ZIW3KdM05HkloiogXMR7 vD2J8rOIRHNRr1aihd4qoQnjq85G0y9rSa0uQsTP81pmZcBdA6odv0eXspIE3ZF6m/P+yBhu3b5L vGStKqR1fZl4qT/ohzUlX4KhodeeKcqmlhlXZCobeUGKxAR3H5/1ZRpfJk/tjq/Rlaoq+UY2Np1+ nsEr2gaKzFDxxtt35a/IgVrLr8jxHv6BWEeOH0gUVTuUTFxFt02NvhnV9WmhL/ouZVvRUWeHw5d9 YB/HUYRELAhlwDvwEVzqjMhdaJXfbKRzTUuDAUQpST06qgcygPas8MQR1zmO+XUUGsfAdIdRgLV8 /LI2rCIxL6i+x6EqINg60ZyJjbxYlEfqhlPhAjbx/bQ7VuiNBNBSHU+5G4Igx2WfrsW0F05ieABH VPP1ph4X2OYeKj5qTi1cuM4i/uwy78uX09Pp9eXbSfeESrKCzxm+ujJeoMCGQhtKLUidB2dIVSvf jyxL1b7IgnLlZ3cIBCUgyoeXk2qmggSK+zTy/PD08lX4T5s91n15eeYVauaTr8W1TPLfmtVPjsSo gX1OpOmoBeW/ZbrMOuVcqFrR4b+TZG18Zu1QWwQKPWaGWLzYiMVDFTdI1q8DT6uBfh3qrqgh7wmq MkNGL1BX4zNg5FUt9qXM/3j87c/H19MXeLjsqABIWKwnjAO6sbkFlL5p5Fn8w/eHL/wbz7zLf1zJ XqTlnf/2zdIPY/sYdfF1+HLzIPOlRa8Ej1Zx6Ki8KIhXTspZ5VHoe2g9RrzbdGbpRGFotIMoWjkT tHandZ16LipOYlxniIyRt0bKDqr9XITs5/P736e3R61aojTQqiVN1L2v+B3qvyNDPtLlU72Jp2nk mX4rv/58fXn78vL9xJP6/PZiD3qr2BqoVokFeYE9nMVGdxo7FurPI+UL29P7/7y8/kt0lZ//d3r9 z5vi2/fTn6IlU1f7SnVFI3mp+cxFwYTNt9PD24/X07fT8/vNj+fHdytP4WVcLB+//v1u553lNWv4 HFGw0k9X6nHBwvSckbeI53QJKLETVj18fT69y0nF/SmyT9J1ZH9oV9EoCc/XI4QPHv8+3Zx4Zr/+ vBFxgh5CQdVM5mvNqpkEQhNITCDVgcQMwgHdH9YCyv43q0a8vTzBmfSHo5DPUqOF+MzDtQkk5V2c gc5qVDe/3UjX208vz4qKLszzfHYvmqyg4p5pmebVdYnU1r0w+iIF7uMXEu3nQoovI8ElCGjCI6n+ 3BFN5xx+O49oP7PKONU+lmTLwnRS150XLNK9ZFVrbf/V3bOelDv11HbsvVzznsQ2fNFXURpONezS cBdK41YdWeF0/G4o6J7Hrtnu5VH5fZAHpo42vOGfX7uj7mFoNR0dZmlue90bt/xt7kdmVO767Vvr 76eHf/34Dg1QOJl5+346fflb6XtgXnb4/8aObDluHPcrrjztPkyN+4hjP8wDJVHdSuuyRHW3/aLK Ot6Jayexy3ZqK3+/AKiD4KHZB5erAYgnCIIkDjMQswbA/JdqD/0tVSuWsOap1MLWFUxnENsltWpC 2KhsQ6hExio/LGDlWS1gw+1NFoo9yLvwh/nCh9zH2cLVh6oLYtW5bsId4ck1yKOpjTE8ENqvwrFd JL0qQGB88hNBKQLd8VqK5dsU2ZbbiNHDsPQfRoeXoJ5CiTsMJ358fX1++sp2qzJpqsz70LOvzGxr 9LPPzp89oDGUsR0IBn1u/S445+urKTBJPz+aTysZDj+ngq1t2eyT1LomyWGcKmZBRxazGux7jUas Lpd/AbBeeC/I0+5zptrOUxXHLFw4jIQKYw74PZZEkWF2rvSQ5f7LiH3tJuibUWyoijabWztfAQmK I+5g6HE0d8AYKcsLrLPhPdU4tyZSIE/b5GqflQdEoGAOgDFJinBDOnEaGtpUxGirm3HjCg+hd/w4 3WDpj1bGvgsoRkv57UIt21cKpE+PwtQ4246qUSLq1rbcKGSZVydmEihlHXt41jC/AFaHKfZpHtiI k5nMAGefsQMGAoPt0Jkd+lRV7T6L2MIdQH2klviRvo6LOvbaqJKJCut7VODVoVnPuVp97CVatvvi e+robU6ji3Nhr19dWyUOqrHcAmahOJR2uwoc4NC9tt8V3dmRmPsxWXD78vTjr2fmQTKuoPwgjwrN UkGpnRcF/uzJV+SXQRnliUMJ2k3H02xpkLMqRJFolAeEkl+xpTGjjn4PzNGUCEj8+Oa2b2Qhatdg 9PH78/vjy+vzwzgcF/+QZ9Rpgb1Bs/vnGGOoefn+9meYhlmljO8F3rbUtEukDc+7pZM7P71+pyly 7gBlYphtwI++MkPWTdnCC1mwQLIyhVUvzJxotEc0kZkkPk4i830sKbIs4VZfmVbTfQsEIzPBmkgV NMI0LZkRzA+1qnbAn2N7PQOAuuNk5j1bXsV72Z8wBuTgizD38KzWfeoC+rNQqnHB6JKJGWJzF9XK uGsydWf2fSot823CgN3YlW9GJw841VQ9NFk2ZgS24RNP8zbh5m2s5s2YrV391l/4Nlz4dqHv2+Ap 6nPEeAR+LpGuLdp1OOly2xcRzbdh4inRQQAwKTP7mcBAHPtjwk4kaGmEHgc+vcgo3h45E+UZPRPt Ts/nscXGb08hnwPDj/DgkOI3mPISveZMSzCrSvx921VKcBJPKxDcsMwj54X6d2m7ZhUNANolUGlP csPrBJMrcvIR0lfrOPKA8dmkrQUFj+x4iM+JZtwlGFx7qxSiPeh0S1NvTLRXkkWqsQZvhLDhmk/U I5a4j3agXWM5YE3EZZbrVvs53rswccM0m0Nu7sP4GqK/LSuVpQbbJTYg0wD9FmUmSBUa4WmTxTP0 E808MbgsdbUBHZYlkyHn8IEQRHsJjfSOhKYI8ZXGKjg7zv2+TQvVmykaNMC4q6Gv4LRuNRjTbNoK vuhUlbZbNs9phxGpDVaKWTCTMVm7np7hDeDhm2nhnLZaYplHKg3SXOo/Sw0Ue1jE1a4RxSLVwpls oKgi9EDC1yqf9TLRUNgPZnswQRcqMIi8bdVDkvzWVMXvyTGhTdzZw7O2urm6uuQCscozaUzOPRCZ 89AlKaPH32U+uXkmVft7KtTvpfJXCTj2edHCFwxytEnw9+ijiW5jaL77x3bzyYfPqngPqhV04MPT 2/P19ceb31YfJtNz5WxXBAqPMqGbkzOw9dvjz6/PF//29ZB2NXPECHDgqYgJdiw8QFDq2KohIHYZ I+RmyjQjttymVFE7P31STCOsTXXf7UCSRGYBA4gqN9Yh/bOkMpnWEjfewcZQ8HNCQsT+k1Eaxsm4 uatVCLt3PpwROno0m+dIhiuKFtoQquVzau+eI2Tw3rx04CfYiOT0PmpoFCMeH7LhgGIJf4uw7QrM vL1AMc5tsNUofjGWVFaVQziv1lR8kOSeOQlqWH5f2SDK/+x2pumirFxoYYyJ4vqyKuXfEtUYEiq0 gZuEmOwy3GMiScWx6hrWDWioxckjZFof843jiBHJEeNYJXogfbYzI6W3rnFw3YIXbIqqnZcNY5D6 Zuvb2060e876IyzIFROB3bAJAf3wX3xMRd8v47cUIBjjBNvz5NLKIpKBcHoTTdqIHR7z+2E/x0yn m2kDOTvSqQQZyFTdwiLZ1xbgtjxvnd0CgFchidA4ZWoIeV8kfXQ3RPYwxtcmKFTiPzDZBVVqH2wB LGqnIjKT82YhvGuPnP2dPmuIll7+UFELRxJ5rpwCNSz0BfOFBuX2VDUHa2cZkdZw4+/jhuOPG771 EWxrDo2G9H5DB10F8VgQj3pqLncivgMl38cYI9FBNmgnmZS8DwnrQQI6tNPkxO5X4utY4vYs6Qet aMEPHnklMbOYwE9dktF+qGu6LmQDbudFaLuyYY9l9LvfsWRIGoZPXH3DXAgBDIdupO4PTcS8eIeP FhRiWe/9KzPOOAPib/cIYCJPUhz6+oTJHfaG5oOoro6FeRlPQEuXIhgpXuZ8EJTavw7Vm5gV8A+X 2JAIYIp8zBfXfJHEpBSiKFGY4bhk51aNBR0aDmnOoVYjMbOtC0VWYHxN0Aq23pLpgkMZ+NYPar+/ vVhU7pQlz6oR5nGwSoQlWERINN/UFiUBwuNJ6Flz9jWTNTBvpyzHHx5e4GTywUSNR5Z+a6bsY5hP YYxpE8MwzPrJwqyDmHBpoRZcXwXrMVPmWZhgC0zLQQuzDWKCrb66CmJuApibTeibm+CI3mxC/bnZ huq5/mT1B87ReB7trwMfYNrLMMoaatHGWeYvf+UHr/3gjR8caPtHP/jKD/7kB98E2h1oyirQlpXV mEOVXfeNB9ZxWKdSYwrwTtR8rc+LgBf94fH1x+NfF9++PPzn6cef83Gf0gjiW1aKxknuo5oO0NgP J6XpwUdvzIO3BOr3uTzK/I+toSfpVPZ4bwf0cGKLhZJ+BXEgLbpWuffu4y0JHBV0aX+sLtfb2Xp3 l9GlQ3PLVH/VZDVwWYFBp73xL0vQaZIhJDV7ytKXuSyVlBQJmpyHnwT0V61WUvA2oRDKmzBOE2pV ij2H6QL2MA+BA1SMBjSJVFBHwGVkL5pkgW6qCVH9Hh98lWgP7qROKJp0tPJZrS+N64aqUrVAY7iZ sBZl5nvqDtL2R5F3ZppFTClkh8oioHUjTzHW7Gt6PXr6rjIEhtnJU/J6DuAxT0oIR7UGS0YdP4Rr 4o74J4SvZRPX3eTVGKAaFsi4S69mv/VuDKbFtFgEe8foUMgiB+0wBO+laPI7XM/6ULq9vLSZdCIN hbTgVFN0OvNtWdOA/lx0Sp6l08w6oytnMyCLJqVK3XFq66zktgTDm42/eApdqu01zXvKo+cpRINh Fj2gvpG1RKGWmwlGa7y/JXRdZXxS6XAiaw6gifK1kFFyMW2TC1VhurM2l+Y3c1H4eibqbFoDRgFQ e3yg0p1BRQzmhNg5qINmTBsMf0fZRFXrfICvpK68QygMYtLF3usSTXQs7MLQGqIzAwJrMB3xaSE5 bCCLqrnrMVmuMzqMzXBsCNS3bBMcBmRnQ+itK/OIhhh6JUuVCXNvoYR7J4yCoTezgT8MCgZnD4OY lm3YiinEi/+iBZgOpL7qmmDMaiRBgYMGxvkk4K8u/fhh2sziOaccYDFEEoQrCHV11w/ydW72yFvE ecsSQwcUhnMSXrU5Y59WDZSiD7awAUATHZmhwaAR1Gz3zg+J8plHI+04fyzeHn/O/Nv7BxAAZXyH ZlgOf6PyS2LG66Co97FpgGzFCyNKE8p85RAgeVL9QVlksC8oFz2EPQ98PGJ3jaj3/xdNWvd83/RU 0xe08HyEYy7G1GIdsy/D16AjYp5WToIPSsSrSElLo7Uo4uHDwlr8Xu+18OiBAos9ADSPFUNgSrI7 Ig19lj0sKeTe9pSB7ueMrNEY2npOQCgsAc/qHZ+I7YLGNjgMYw9wkBFCPGBatrWwXw8Y/3U3KWwL BMM4D7PnNY7AdBr70SOb3lP447+GY35RhUM9fBDIMTGRA6ssEuptzW352KohGSB5l3hkneYHQ/gE 2Hsa5KFhLgvaTO8MHXppdHVIamKIqrFOI1r4AHS+mjcJgxEjEF37QjSH0JCidS8lqlxtbrYUIs3O OD+3Omskpv/ReS0b2HKVtCOiWWPmBIrFCsSiobLWdK62kyrjs/qleGyNyJKrMc6Raeta597otyRK aIs+7BL2joS/lz8ABYMOh8z0GnWJAxxEVeVzkMETakx7nY50PJtIXV/1gxZF237nNRIkNV0rBMy8 zoBT8oTgdkvLABTZJkV7a6v+WqGvfvDbpsLQve5+p+Fh1huPyaDMV0kHi7WpyvMCZXkOl6VNd8OM Mlr94ktxaPLm1TILUzYOOs5vs8RsICdIIevVXS37y/P15XyqtXGwXaz8uM6K28exJBo3Zt8GLFa3 1Caq8pfnwy6sRE409gO3dU3Amgj9suZOa6x49veL4bgOW2zhq34Bx08QNnk2XPHbTKSXD2lRcVXf 6cuZdomZisw7j9rV7PHh5+vT+y8jLfW4+KUZR1hhKnDYigg6x3crydbDgeOvURUkIxpd1iyMBgtN kHuyRctP3SuftHAMQUeIafEylTc8PHrrGh8lz2lTLNTUUzh0XwEwiFnZ+dftTFSIwF3ZRAKSvLqr lpogamCggpkM2SjjiklfKa08Nd2JQgQcMXa2hewEpOclgWcp36eFYckIP+CIKVo8dtUxHCOS8x+r SxOLuZOUe9Pm66jBtRZRIuKF5ThxyIcvLy9fXr8/v06vOGccJjxdtaZ/GO6RPHS3huFeVt/Z0LMZ T1GD6lsbordcVGiONgqjo2Yt3WLVtzhVQxDVEBG22aGiVVZNBpOvv17eny8eMP3y8+vFt8e/Xh5f Db8TIgZ9cMd8Rxh47cKlmbTAALqkoCnGWb03NTsb435kPcnOQJe0KXdOSwDmEtZoyOOQEtQlLkQp dp42D3BmVD+gUMh6jXfMD6eZs+6XBqpdulpfY2Rzu5lllzO/p6n1+D9cK4qw20520qmJ/nnmsFN7 kNMO3AysLn6+f3v88f70QCmk5I8H5DAM9vHfp/dvF+Lt7fnhiVDJl/cvDqfFZrrysdseWCtvKUUU VRmR09T356+m6e9YYBS7vTP9jidY68BkHHlmMue2oPaQR7FnJs6q9bh9vX0LNHpfiNhT8xkK9+sB Gn8suHQbw1w8vr2749LEm7U7kxqs3bbciaCFZ0OLZOuBfXQXRxbvBcWbdqttigRY2ws2n55n8NoM HzWDN2uXut2LlWcwEdy3bSs34dkEGl9FAP7IgssNq2DXrG5c8KnWxHYDaDT7OIcFjQHeaMhd6+2n l2/cE3CUpq1HwrakcF9tvaixDhdZdlHmKa6J3YJgCzylmYcLRsQc7s7u70Sh2xge9VgUMs+5q7eF +j/KaNVHl7sB6k5n4hnLlP474MNe3IvE07AWfYzXl0urcyD526a3UiYeedfUeGXiqVljgJXl+m/L VlK4bHuqvDM6wN34hRw9cZzdME7Qb078rKTZ+/n7y+vj2xvsDw6Lg1KG98JOxfc8PvMglO+rKfzK lx9fn79flD+//+vx9WKHYWK+vPsqEGWbwRGq4WHVjP2arlFwlS5N60TYDvpKePT3J0co4i1ZLRLL h9fBede7iQeRZBiy4rn8YL75jBC8lYz3We3B0JVcatzFIhD6FdP1DL4KoRrP0Xj2tz7QWq3xTDk8 NGX3gpJGm9TGIQ+bxoUTlVawtaar7PRJwDsnx33VKjyd+S4ls1I0w71OOmUke/rX65fXXxevzz/f n36wxDSkh5v6eZSpRqL3sHQvmNHHFm/voPbIPKWMeAwznFXMBXhEBcEmQ8AxI84UE0jx6opTqNVl kqUclqmu519tLO0UAEtXMwNBnsUyurv2fKox28ASIRLRnETAcFhTRFmg6k9msL9o0lnMz699kYSA GN/dSH78YlBHqoDkmLwhjEfl+4ourmz4+R7B9m+UcQ6MXjtqlzYTV1sHCCdXH0ztuyJyEPg87pYb xZ/Z1ZuGBp8Lx771u/uMvXdPiAgQay8mvzdPlAbifB+gr1wGp0c9wZyZGonmRFVesQ3HhGKpJt/j a1ErUUL5YP3BPNMZ8KjwgtO2MKUomsGyFHoaxF+MEMYP2OR2Ol18sCVTd+gBi3YcdHUceFFtmEBI bk0b75ybGedN148ujuM1AIgobW5gRDJuEm5b9D84MoeC5xgCAA== --20cf300faff5f177bb04bde40f38 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --20cf300faff5f177bb04bde40f38-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter Moody Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Tue, 17 Apr 2012 14:54:29 -0700 Message-ID: References: <1327519203.4131.25.camel@localhost> <1332983643.384.8.camel@localhost> <1333660021.2273.0.camel@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Received: from mx1.redhat.com (ext-mx13.extmail.prod.ext.phx2.redhat.com [10.5.110.18]) by int-mx01.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id q3HLt2p8018494 for ; Tue, 17 Apr 2012 17:55:02 -0400 Received: from mail-qc0-f174.google.com (mail-qc0-f174.google.com [209.85.216.174]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q3HLsxrh015684 for ; Tue, 17 Apr 2012 17:55:00 -0400 Received: by qcro28 with SMTP id o28so5302083qcr.33 for ; Tue, 17 Apr 2012 14:54:59 -0700 (PDT) In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Eric Paris Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com Last thing. moving synchronize_srcu(&fsnotify_mark_srcu) out of the for(;;) loop in fs/notify/mark.c appears to solve the stability issues for me. I don't know enough about kernel internals to determine if this is doing lots of other bad things to my system or not. Cheers, peter On Tue, Apr 17, 2012 at 11:24 AM, Peter Moody wrote: > and my config.gz > > On Tue, Apr 17, 2012 at 10:56 AM, Peter Moody wrote: >> Here's a trace with debugging turned way up plus a few extra printk's >> added to fs/notify/mark.c. I'm looping through private_destroy_list >> before and after the call to synchronize_srcu. >> >> I can reproduce this reliably with kvm with 2 virtual processors: >> Linux desktop 3.4.0-rc3-oops1+ #1 SMP Tue Apr 17 09:59:44 PDT 2012 >> x86_64 GNU/Linux >> >> Cheers, >> peter >> >> On Thu, Apr 5, 2012 at 2:07 PM, Eric Paris wrote: >>> please please please keep on list. =A0Everything you say might help tra= ck >>> it down! >>> >>> On Thu, 2012-04-05 at 14:03 -0700, Peter Moody wrote: >>>> (please let me know if I should take this off-list) >>>> >>>> One other thing (again, maybe already known), but this seems to be >>>> exacerbated by SMP. On my machine, I can't reproduce the crash if I >>>> booth with maxcpus=3D1. >>>> >>>> Still hunting. >>>> >>>> Cheers, >>>> peter >>>> >>>> On Tue, Apr 3, 2012 at 9:15 AM, Peter Moody wrote: >>>> > This may already be known, but the issue seems to be limited to watch >>>> > rules. With any watch rules, I can reliably crash my machine while >>>> > freeing a watch rule after only starting/stopping auditd a few times. >>>> > With no watch rules, I have no issues. >>>> > >>>> > Cheers, >>>> > peter >>>> > >>>> > On Wed, Mar 28, 2012 at 11:44 PM, Valentin Avram = wrote: >>>> >> Yes, i know that patch. It made it into kernel 3.2.2. I tested it >>>> >> successfully (oops in 3.2.1, no oops in 3.2.9), but this oops i'm s= eeing is >>>> >> also in 3.2.9. >>>> >> >>>> >> I monitored changelogs since 3.2.1 to 3.2.12 but there were no fixe= s either >>>> >> in audit subsystem or in fsnotify. I'll try to reproduce in latest = 3.2.13 >>>> >> and repost the oops, but i'm 99% confident it will be the same. >>>> >> >>>> >> Sadly nobody except you seems to pay attention to this problem, pro= bably >>>> >> because it requires special conditions to reproduce (really, who st= arts and >>>> >> stops auditd every 5 seconds on a production server?). We only ran = into it >>>> >> because one of our servers would randomly oops and then freeze abou= t each >>>> >> month after stopping and then starting >>>> >> >>>> >> auditd >>>> >> >>>> >> every morning (and the stop-start sequence was needed to workaround= a bug >>>> >> somewhere that would hang a >>>> >> >>>> >> gzip >>>> >> >>>> >> running on a file outside a watched folder). >>>> >> >>>> >> Anyway, as a last note, i have a feeling that the oops is not exact= ly >>>> >> random, there is a pattern, just that i haven't figured it out comp= letely >>>> >> yet. >>>> >> >>>> >> Will keep you >>>> >> >>>> >> uptodate >>>> >> >>>> >> with the things i find out. >>>> >> >>>> >> V. >>>> >> >>>> >> On Mar 29, 2012 4:14 AM, "Eric Paris" wrote: >>>> >>> >>>> >>> That patch fixes a BUG() . =A0The report has a NULL ptr deref and = some >>>> >>> apparent list correuption.... =A0Sadly they aren't the same.... >>>> >>> >>>> >>> On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote: >>>> >>> > fyi: this patch [1] seems to fix the issue for me. The explanati= on in >>>> >>> > the subject would reliably oops my machine. >>>> >>> > >>>> >>> > [1] >>>> >>> > http://git.kernel.org/?p=3Dlinux/kernel/git/torvalds/linux-2.6.g= it;a=3Dcommit;h=3Dfed474857efbed79cd390d0aee224231ca718f63 >>>> >>> > >>>> >>> > On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody = wrote: >>>> >>> > > Are you still able to reliably reproduce this oops? I'm trying= to >>>> >>> > > track this down because this bug (or a very similar bug) is ca= using >>>> >>> > > some significant headaches here at work, but I haven't had a l= ot of >>>> >>> > > luck. I'm using usermode linux, though, so that might be inter= fering >>>> >>> > > with things. >>>> >>> > > >>>> >>> > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram >>>> >>> > > wrote: >>>> >>> > >> Finally i found some time and spare server to retest the oops= and >>>> >>> > >> list_add >>>> >>> > >> corruptions i was getting with the 3.x kernels and auditd 2.1= .3. >>>> >>> > >> >>>> >>> > >> I tested now with gentoo's latest stable 3.2.1-gentoo-r2 and >>>> >>> > >> kernel.org's >>>> >>> > >> 3.2.9. >>>> >>> > >> >>>> >>> > >> Both get the oops/BUG in the same way and after that, they ke= ep >>>> >>> > >> pouring >>>> >>> > >> list_add corruptions with audit_prune_tre(truncated?) and aud= itctl as >>>> >>> > >> comms. >>>> >>> > >> >>>> >>> > >> Since this is not about Gentoo's kernel only, i'll post here = the oops >>>> >>> > >> in >>>> >>> > >> 3.2.9 and also attach some list_add corruptions. >>>> >>> > >> >>>> >>> > >> 3.2.9 BUG: >>>> >>> > >> >>>> >>> > >> kernel: [ =A0301.240011] BUG: unable to handle kernel NULL po= inter >>>> >>> > >> dereference >>>> >>> > >> at =A0 (null) >>>> >>> > >> kernel: [ =A0301.240305] IP: [] __list_del_entry+0x= 20/0xe0 >>>> >>> > >> kernel: [ =A0301.240481] *pdpt =3D 0000000000000000 *pde =3D >>>> >>> > >> f000ddc8f000ddc8 >>>> >>> > >> kernel: [ =A0301.240698] Oops: 0000 [#1] SMP >>>> >>> > >> kernel: [ =A0301.240910] >>>> >>> > >> kernel: [ =A0301.241030] Pid: 642, comm: fsnotify_mark Not ta= inted >>>> >>> > >> 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge 2950/0CX396 >>>> >>> > >> kernel: [ =A0301.241370] EIP: 0060:[] EFLAGS: 00010= 287 CPU: 6 >>>> >>> > >> kernel: [ =A0301.241498] EIP is at __list_del_entry+0x20/0xe0 >>>> >>> > >> kernel: [ =A0301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: fff= fffff EDX: >>>> >>> > >> 00000000 >>>> >>> > >> kernel: [ =A0301.241751] ESI: f4fae544 EDI: f4fae508 EBP: f47= cff7c ESP: >>>> >>> > >> f47cff64 >>>> >>> > >> kernel: [ =A0301.241879] =A0DS: 007b ES: 007b FS: 00d8 GS: 00= 00 SS: 0068 >>>> >>> > >> kernel: [ =A0301.242005] Process fsnotify_mark (pid: 642, ti= =3Df47ce000 >>>> >>> > >> task=3Df4f47c00 task.ti=3Df47ce000) >>>> >>> > >> kernel: [ =A0301.242207] Stack: >>>> >>> > >> kernel: [ =A0301.242327] =A0c10813c0 f47cffa4 f4f47c00 f4e708= 88 f47cff7c >>>> >>> > >> f47cffa4 f47cffb8 c10f6976 >>>> >>> > >> kernel: [ =A0301.242882] =A0ffffffc3 f4f47c00 f4f47c00 000000= 00 f4f47c00 >>>> >>> > >> c10530c0 f47cff9c f47cff9c >>>> >>> > >> kernel: [ =A0301.243438] =A0f4fae544 f4fae544 f4c47f58 000000= 00 c10f68f0 >>>> >>> > >> f47cffe4 c1052834 00000000 >>>> >>> > >> kernel: [ =A0301.243995] Call Trace: >>>> >>> > >> kernel: [ =A0301.244119] =A0[] ? >>>> >>> > >> rcu_check_callbacks+0x110/0x110 >>>> >>> > >> kernel: [ =A0301.244248] =A0[] fsnotify_mark_destro= y+0x86/0x120 >>>> >>> > >> kernel: [ =A0301.244377] =A0[] ? abort_exclusive_wa= it+0x80/0x80 >>>> >>> > >> kernel: [ =A0301.244504] =A0[] ? fsnotify_put_mark+= 0x30/0x30 >>>> >>> > >> kernel: [ =A0301.244631] =A0[] kthread+0x74/0x80 >>>> >>> > >> kernel: [ =A0301.244756] =A0[] ? >>>> >>> > >> kthread_flush_work_fn+0x10/0x10 >>>> >>> > >> kernel: [ =A0301.244885] =A0[] kernel_thread_helper= +0x6/0xd >>>> >>> > >> kernel: [ =A0301.245011] Code: 55 f4 8b 45 f8 e9 75 ff ff ff = 90 55 89 >>>> >>> > >> e5 53 83 >>>> >>> > >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 02 20 0= 0 0f 84 >>>> >>> > >> 8e 00 >>>> >>> > >> 00 00 <8b> 1a 39 d8 75 62 8b 59 04 39 d8 75 35 89 51 04 89 0a= 83 c4 >>>> >>> > >> 14 >>>> >>> > >> kernel: [ =A0301.248195] EIP: [] __list_del_entry+0= x20/0xe0 >>>> >>> > >> SS:ESP >>>> >>> > >> 0068:f47cff64 >>>> >>> > >> kernel: [ =A0301.248414] CR2: 0000000000000000 >>>> >>> > >> kernel: [ =A0301.248538] ---[ end trace 15082dbfb353f84c ]--- >>>> >>> > >> >>>> >>> > >> The kernel was compiled with the following DEBUG support (the= bolded >>>> >>> > >> one >>>> >>> > >> were requested by Gentoo's Dev: >>>> >>> > >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=3Dy >>>> >>> > >> CONFIG_SLUB_DEBUG=3Dy >>>> >>> > >> CONFIG_HAVE_DMA_API_DEBUG=3Dy >>>> >>> > >> CONFIG_X86_DEBUGCTLMSR=3Dy >>>> >>> > >> CONFIG_PNP_DEBUG_MESSAGES=3Dy >>>> >>> > >> CONFIG_AIC94XX_DEBUG=3Dy >>>> >>> > >> CONFIG_USB_DEBUG=3Dy >>>> >>> > >> CONFIG_DEBUG_KERNEL=3Dy >>>> >>> > >> CONFIG_SCHED_DEBUG=3Dy >>>> >>> > >> CONFIG_DEBUG_RT_MUTEXES=3Dy >>>> >>> > >> CONFIG_DEBUG_PI_LIST=3Dy >>>> >>> > >> CONFIG_DEBUG_BUGVERBOSE=3Dy >>>> >>> > >> CONFIG_DEBUG_INFO=3Dy >>>> >>> > >> CONFIG_DEBUG_MEMORY_INIT=3Dy >>>> >>> > >> CONFIG_DEBUG_LIST=3Dy >>>> >>> > >> CONFIG_DEBUG_STACKOVERFLOW=3Dy >>>> >>> > >> CONFIG_DEBUG_RODATA=3Dy >>>> >>> > >> CONFIG_DEBUG_RODATA_TEST=3Dy >>>> >>> > >> >>>> >>> > >> I attached the kernel config i used for 3.2.9 to generate thi= s oops >>>> >>> > >> and >>>> >>> > >> warnings. >>>> >>> > >> >>>> >>> > >> From the list_add warnings that come after, out of 805 warnin= gs i >>>> >>> > >> processed, >>>> >>> > >> after masking with XXXXX the PID and next=3D values that kept= changing >>>> >>> > >> in >>>> >>> > >> every one, i got 26 types of MD5. I also attached the files r= elevant >>>> >>> > >> as an >>>> >>> > >> archive to this email. >>>> >>> > >> >>>> >>> > >> The Gentoo bug i opened is sleeping, it seems nobody has the = time to >>>> >>> > >> at >>>> >>> > >> least test to confirm or not the problems i'm seeing (or ever= ybody's >>>> >>> > >> thinking that nobody would restart auditd so often, so the bu= g it's >>>> >>> > >> not that >>>> >>> > >> serious). >>>> >>> > >> >>>> >>> > >> >>>> >>> > >> Thank you for your time. >>>> >>> > >> >>>> >>> > >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram >>>> >>> > >> wrote: >>>> >>> > >> >>>> >>> > >> >>>> >>> > >> -- >>>> >>> > >> Linux-audit mailing list >>>> >>> > >> Linux-audit@redhat.com >>>> >>> > >> https://www.redhat.com/mailman/listinfo/linux-audit >>>> >>> > > >>>> >>> > > >>>> >>> > > >>>> >>> > > -- >>>> >>> > > Peter Moody =A0 =A0 =A0Google =A0 =A01.650.253.7306 >>>> >>> > > Security Engineer =A0pgp:0xC3410038 >>>> >>> > >>>> >>> > >>>> >>> > >>>> >>> >>>> >>> >>>> >> >>>> > >>>> > >>>> > >>>> > -- >>>> > Peter Moody =A0 =A0 =A0Google =A0 =A01.650.253.7306 >>>> > Security Engineer =A0pgp:0xC3410038 >>>> >>>> >>>> >>> >>> >> >> >> >> -- >> Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 >> Security Engineer=A0 pgp:0xC3410038 > > > > -- > Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 > Security Engineer=A0 pgp:0xC3410038 -- = Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 Security Engineer=A0 pgp:0xC3410038 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Marcelo Cerri Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Fri, 20 Apr 2012 23:14:24 -0300 Message-ID: <20120420231424.1836e56b@oc8526070481.ibm.com> References: <1327519203.4131.25.camel@localhost> <1332983643.384.8.camel@localhost> <1333660021.2273.0.camel@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Received: from mx1.redhat.com (ext-mx16.extmail.prod.ext.phx2.redhat.com [10.5.110.21]) by int-mx02.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id q3L2EYZq011431 for ; Fri, 20 Apr 2012 22:14:34 -0400 Received: from e24smtp04.br.ibm.com (e24smtp04.br.ibm.com [32.104.18.25]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q3L2EUug008613 for ; Fri, 20 Apr 2012 22:14:31 -0400 Received: from /spool/local by e24smtp04.br.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 20 Apr 2012 23:14:29 -0300 In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Peter Moody , Valentin Avram Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com I took a look at the source code and made some tests. It seems to be a problem with the reference count of the fsnotify_mark structure. This error occurs because the fsnotify_mark_destroy function (which runs in a separated kthread) is trying to iterate through a mark that is already freed. Looking at the fsnotify_destroy_mark function (not confuse with fsnotify_mark_destroy), which adds a mark to destroy_list to be freed later by fsnotify_mark_destroy, I noticed that it does not increment the reference count for the reference added to the destroy_list and usually the callers dispose the references they held after calling fsnotify_destroy_mark. The patch below increments the reference count of a mark when it is added to the destroy list. It seems to solve the issue and it doesn't seem to cause any memory leak. Please, can you make some tests in your environments and let me know if there is any problem with this patch. Regarding the synchronize_scru call, I don't think it's causing this error. Probably it just make it more frequently because it forces all the cpus to schedule, giving the chance to someone else to free the mark. --- fs/notify/mark.c | 1 + 1 file changed, 1 insertion(+) diff --git a/fs/notify/mark.c b/fs/notify/mark.c index f104d56..2985fff 100644 --- a/fs/notify/mark.c +++ b/fs/notify/mark.c @@ -150,6 +150,7 @@ void fsnotify_destroy_mark(struct fsnotify_mark *mark) spin_unlock(&group->mark_lock); spin_unlock(&mark->lock); = + fsnotify_get_mark(mark); spin_lock(&destroy_lock); list_add(&mark->destroy_list, &destroy_list); spin_unlock(&destroy_lock); -- = 1.7.9.4 On Tue, 17 Apr 2012 14:54:29 -0700 Peter Moody wrote: > Last thing. moving synchronize_srcu(&fsnotify_mark_srcu) out of the > for(;;) loop in fs/notify/mark.c appears to solve the stability issues > for me. I don't know enough about kernel internals to determine if > this is doing lots of other bad things to my system or not. > = > Cheers, > peter > = > On Tue, Apr 17, 2012 at 11:24 AM, Peter Moody > wrote: > > and my config.gz > > > > On Tue, Apr 17, 2012 at 10:56 AM, Peter Moody > > wrote: > >> Here's a trace with debugging turned way up plus a few extra > >> printk's added to fs/notify/mark.c. I'm looping through > >> private_destroy_list before and after the call to synchronize_srcu. > >> > >> I can reproduce this reliably with kvm with 2 virtual processors: > >> Linux desktop 3.4.0-rc3-oops1+ #1 SMP Tue Apr 17 09:59:44 PDT 2012 > >> x86_64 GNU/Linux > >> > >> Cheers, > >> peter > >> > >> On Thu, Apr 5, 2012 at 2:07 PM, Eric Paris > >> wrote: > >>> please please please keep on list. =A0Everything you say might help > >>> track it down! > >>> > >>> On Thu, 2012-04-05 at 14:03 -0700, Peter Moody wrote: > >>>> (please let me know if I should take this off-list) > >>>> > >>>> One other thing (again, maybe already known), but this seems to > >>>> be exacerbated by SMP. On my machine, I can't reproduce the > >>>> crash if I booth with maxcpus=3D1. > >>>> > >>>> Still hunting. > >>>> > >>>> Cheers, > >>>> peter > >>>> > >>>> On Tue, Apr 3, 2012 at 9:15 AM, Peter Moody > >>>> wrote: > >>>> > This may already be known, but the issue seems to be limited > >>>> > to watch rules. With any watch rules, I can reliably crash my > >>>> > machine while freeing a watch rule after only > >>>> > starting/stopping auditd a few times. With no watch rules, I > >>>> > have no issues. > >>>> > > >>>> > Cheers, > >>>> > peter > >>>> > > >>>> > On Wed, Mar 28, 2012 at 11:44 PM, Valentin Avram > >>>> > wrote: > >>>> >> Yes, i know that patch. It made it into kernel 3.2.2. I > >>>> >> tested it successfully (oops in 3.2.1, no oops in 3.2.9), but > >>>> >> this oops i'm seeing is also in 3.2.9. > >>>> >> > >>>> >> I monitored changelogs since 3.2.1 to 3.2.12 but there were > >>>> >> no fixes either in audit subsystem or in fsnotify. I'll try > >>>> >> to reproduce in latest 3.2.13 and repost the oops, but i'm > >>>> >> 99% confident it will be the same. > >>>> >> > >>>> >> Sadly nobody except you seems to pay attention to this > >>>> >> problem, probably because it requires special conditions to > >>>> >> reproduce (really, who starts and stops auditd every 5 > >>>> >> seconds on a production server?). We only ran into it because > >>>> >> one of our servers would randomly oops and then freeze about > >>>> >> each month after stopping and then starting > >>>> >> > >>>> >> auditd > >>>> >> > >>>> >> every morning (and the stop-start sequence was needed to > >>>> >> workaround a bug somewhere that would hang a > >>>> >> > >>>> >> gzip > >>>> >> > >>>> >> running on a file outside a watched folder). > >>>> >> > >>>> >> Anyway, as a last note, i have a feeling that the oops is not > >>>> >> exactly random, there is a pattern, just that i haven't > >>>> >> figured it out completely yet. > >>>> >> > >>>> >> Will keep you > >>>> >> > >>>> >> uptodate > >>>> >> > >>>> >> with the things i find out. > >>>> >> > >>>> >> V. > >>>> >> > >>>> >> On Mar 29, 2012 4:14 AM, "Eric Paris" > >>>> >> wrote: > >>>> >>> > >>>> >>> That patch fixes a BUG() . =A0The report has a NULL ptr deref > >>>> >>> and some apparent list correuption.... =A0Sadly they aren't > >>>> >>> the same.... > >>>> >>> > >>>> >>> On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote: > >>>> >>> > fyi: this patch [1] seems to fix the issue for me. The > >>>> >>> > explanation in the subject would reliably oops my machine. > >>>> >>> > > >>>> >>> > [1] > >>>> >>> > http://git.kernel.org/?p=3Dlinux/kernel/git/torvalds/linux-2.6= .git;a=3Dcommit;h=3Dfed474857efbed79cd390d0aee224231ca718f63 > >>>> >>> > > >>>> >>> > On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody > >>>> >>> > wrote: > >>>> >>> > > Are you still able to reliably reproduce this oops? I'm > >>>> >>> > > trying to track this down because this bug (or a very > >>>> >>> > > similar bug) is causing some significant headaches here > >>>> >>> > > at work, but I haven't had a lot of luck. I'm using > >>>> >>> > > usermode linux, though, so that might be interfering > >>>> >>> > > with things. > >>>> >>> > > > >>>> >>> > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram > >>>> >>> > > wrote: > >>>> >>> > >> Finally i found some time and spare server to retest > >>>> >>> > >> the oops and list_add > >>>> >>> > >> corruptions i was getting with the 3.x kernels and > >>>> >>> > >> auditd 2.1.3. > >>>> >>> > >> > >>>> >>> > >> I tested now with gentoo's latest stable > >>>> >>> > >> 3.2.1-gentoo-r2 and kernel.org's > >>>> >>> > >> 3.2.9. > >>>> >>> > >> > >>>> >>> > >> Both get the oops/BUG in the same way and after that, > >>>> >>> > >> they keep pouring > >>>> >>> > >> list_add corruptions with audit_prune_tre(truncated?) > >>>> >>> > >> and auditctl as comms. > >>>> >>> > >> > >>>> >>> > >> Since this is not about Gentoo's kernel only, i'll post > >>>> >>> > >> here the oops in > >>>> >>> > >> 3.2.9 and also attach some list_add corruptions. > >>>> >>> > >> > >>>> >>> > >> 3.2.9 BUG: > >>>> >>> > >> > >>>> >>> > >> kernel: [ =A0301.240011] BUG: unable to handle kernel > >>>> >>> > >> NULL pointer dereference > >>>> >>> > >> at =A0 (null) > >>>> >>> > >> kernel: [ =A0301.240305] IP: [] > >>>> >>> > >> __list_del_entry+0x20/0xe0 kernel: [ =A0301.240481] *pdpt > >>>> >>> > >> =3D 0000000000000000 *pde =3D f000ddc8f000ddc8 > >>>> >>> > >> kernel: [ =A0301.240698] Oops: 0000 [#1] SMP > >>>> >>> > >> kernel: [ =A0301.240910] > >>>> >>> > >> kernel: [ =A0301.241030] Pid: 642, comm: fsnotify_mark > >>>> >>> > >> Not tainted 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge > >>>> >>> > >> 2950/0CX396 kernel: [ =A0301.241370] EIP: > >>>> >>> > >> 0060:[] EFLAGS: 00010287 CPU: 6 kernel: > >>>> >>> > >> [ =A0301.241498] EIP is at __list_del_entry+0x20/0xe0 > >>>> >>> > >> kernel: [ =A0301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: > >>>> >>> > >> ffffffff EDX: 00000000 kernel: [ =A0301.241751] ESI: > >>>> >>> > >> f4fae544 EDI: f4fae508 EBP: f47cff7c ESP: f47cff64 > >>>> >>> > >> kernel: [ =A0301.241879] =A0DS: 007b ES: 007b FS: 00d8 GS: > >>>> >>> > >> 0000 SS: 0068 kernel: [ =A0301.242005] Process > >>>> >>> > >> fsnotify_mark (pid: 642, ti=3Df47ce000 task=3Df4f47c00 > >>>> >>> > >> task.ti=3Df47ce000) kernel: [ =A0301.242207] Stack: > >>>> >>> > >> kernel: [ =A0301.242327] =A0c10813c0 f47cffa4 f4f47c00 > >>>> >>> > >> f4e70888 f47cff7c f47cffa4 f47cffb8 c10f6976 > >>>> >>> > >> kernel: [ =A0301.242882] =A0ffffffc3 f4f47c00 f4f47c00 > >>>> >>> > >> 00000000 f4f47c00 c10530c0 f47cff9c f47cff9c > >>>> >>> > >> kernel: [ =A0301.243438] =A0f4fae544 f4fae544 f4c47f58 > >>>> >>> > >> 00000000 c10f68f0 f47cffe4 c1052834 00000000 > >>>> >>> > >> kernel: [ =A0301.243995] Call Trace: > >>>> >>> > >> kernel: [ =A0301.244119] =A0[] ? > >>>> >>> > >> rcu_check_callbacks+0x110/0x110 > >>>> >>> > >> kernel: [ =A0301.244248] =A0[] > >>>> >>> > >> fsnotify_mark_destroy+0x86/0x120 kernel: [ =A0301.244377] > >>>> >>> > >> =A0[] ? abort_exclusive_wait+0x80/0x80 kernel: > >>>> >>> > >> [ =A0301.244504] =A0[] ? > >>>> >>> > >> fsnotify_put_mark+0x30/0x30 kernel: [ =A0301.244631] > >>>> >>> > >> =A0[] kthread+0x74/0x80 kernel: [ =A0301.244756] > >>>> >>> > >> =A0[] ? kthread_flush_work_fn+0x10/0x10 kernel: > >>>> >>> > >> [ =A0301.244885] =A0[] > >>>> >>> > >> kernel_thread_helper+0x6/0xd kernel: [ =A0301.245011] > >>>> >>> > >> Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89 e5 53 83 > >>>> >>> > >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 > >>>> >>> > >> 02 20 00 0f 84 8e 00 00 00 <8b> 1a 39 d8 75 62 8b 59 04 > >>>> >>> > >> 39 d8 75 35 89 51 04 89 0a 83 c4 14 > >>>> >>> > >> kernel: [ =A0301.248195] EIP: [] > >>>> >>> > >> __list_del_entry+0x20/0xe0 SS:ESP > >>>> >>> > >> 0068:f47cff64 > >>>> >>> > >> kernel: [ =A0301.248414] CR2: 0000000000000000 > >>>> >>> > >> kernel: [ =A0301.248538] ---[ end trace > >>>> >>> > >> 15082dbfb353f84c ]--- > >>>> >>> > >> > >>>> >>> > >> The kernel was compiled with the following DEBUG > >>>> >>> > >> support (the bolded one > >>>> >>> > >> were requested by Gentoo's Dev: > >>>> >>> > >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=3Dy > >>>> >>> > >> CONFIG_SLUB_DEBUG=3Dy > >>>> >>> > >> CONFIG_HAVE_DMA_API_DEBUG=3Dy > >>>> >>> > >> CONFIG_X86_DEBUGCTLMSR=3Dy > >>>> >>> > >> CONFIG_PNP_DEBUG_MESSAGES=3Dy > >>>> >>> > >> CONFIG_AIC94XX_DEBUG=3Dy > >>>> >>> > >> CONFIG_USB_DEBUG=3Dy > >>>> >>> > >> CONFIG_DEBUG_KERNEL=3Dy > >>>> >>> > >> CONFIG_SCHED_DEBUG=3Dy > >>>> >>> > >> CONFIG_DEBUG_RT_MUTEXES=3Dy > >>>> >>> > >> CONFIG_DEBUG_PI_LIST=3Dy > >>>> >>> > >> CONFIG_DEBUG_BUGVERBOSE=3Dy > >>>> >>> > >> CONFIG_DEBUG_INFO=3Dy > >>>> >>> > >> CONFIG_DEBUG_MEMORY_INIT=3Dy > >>>> >>> > >> CONFIG_DEBUG_LIST=3Dy > >>>> >>> > >> CONFIG_DEBUG_STACKOVERFLOW=3Dy > >>>> >>> > >> CONFIG_DEBUG_RODATA=3Dy > >>>> >>> > >> CONFIG_DEBUG_RODATA_TEST=3Dy > >>>> >>> > >> > >>>> >>> > >> I attached the kernel config i used for 3.2.9 to > >>>> >>> > >> generate this oops and > >>>> >>> > >> warnings. > >>>> >>> > >> > >>>> >>> > >> From the list_add warnings that come after, out of 805 > >>>> >>> > >> warnings i processed, > >>>> >>> > >> after masking with XXXXX the PID and next=3D values that > >>>> >>> > >> kept changing in > >>>> >>> > >> every one, i got 26 types of MD5. I also attached the > >>>> >>> > >> files relevant as an > >>>> >>> > >> archive to this email. > >>>> >>> > >> > >>>> >>> > >> The Gentoo bug i opened is sleeping, it seems nobody > >>>> >>> > >> has the time to at > >>>> >>> > >> least test to confirm or not the problems i'm seeing > >>>> >>> > >> (or everybody's thinking that nobody would restart > >>>> >>> > >> auditd so often, so the bug it's not that > >>>> >>> > >> serious). > >>>> >>> > >> > >>>> >>> > >> > >>>> >>> > >> Thank you for your time. > >>>> >>> > >> > >>>> >>> > >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram > >>>> >>> > >> wrote: > >>>> >>> > >> > >>>> >>> > >> > >>>> >>> > >> -- > >>>> >>> > >> Linux-audit mailing list > >>>> >>> > >> Linux-audit@redhat.com > >>>> >>> > >> https://www.redhat.com/mailman/listinfo/linux-audit > >>>> >>> > > > >>>> >>> > > > >>>> >>> > > > >>>> >>> > > -- > >>>> >>> > > Peter Moody =A0 =A0 =A0Google =A0 =A01.650.253.7306 > >>>> >>> > > Security Engineer =A0pgp:0xC3410038 > >>>> >>> > > >>>> >>> > > >>>> >>> > > >>>> >>> > >>>> >>> > >>>> >> > >>>> > > >>>> > > >>>> > > >>>> > -- > >>>> > Peter Moody =A0 =A0 =A0Google =A0 =A01.650.253.7306 > >>>> > Security Engineer =A0pgp:0xC3410038 > >>>> > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> -- > >> Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 > >> Security Engineer=A0 pgp:0xC3410038 > > > > > > > > -- > > Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 > > Security Engineer=A0 pgp:0xC3410038 > = > = > = From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter Moody Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Mon, 23 Apr 2012 09:05:27 -0700 Message-ID: References: <1327519203.4131.25.camel@localhost> <1332983643.384.8.camel@localhost> <1333660021.2273.0.camel@localhost> <20120420231424.1836e56b@oc8526070481.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Received: from mx1.redhat.com (ext-mx11.extmail.prod.ext.phx2.redhat.com [10.5.110.16]) by int-mx02.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id q3NG64ad009809 for ; Mon, 23 Apr 2012 12:06:04 -0400 Received: from mail-qc0-f174.google.com (mail-qc0-f174.google.com [209.85.216.174]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q3NG5xN6018129 for ; Mon, 23 Apr 2012 12:05:59 -0400 Received: by qcro28 with SMTP id o28so9064489qcr.33 for ; Mon, 23 Apr 2012 09:05:59 -0700 (PDT) In-Reply-To: <20120420231424.1836e56b@oc8526070481.ibm.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Marcelo Cerri Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com This works for me. Thanks, Marcelo! Cheers, peter On Fri, Apr 20, 2012 at 7:14 PM, Marcelo Cerri wrote: > > I took a look at the source code and made some tests. It seems to be a > problem with the reference count of the fsnotify_mark structure. > > This error occurs because the fsnotify_mark_destroy function > (which runs in a separated kthread) is trying to iterate through a mark > that is already freed. > > Looking at the fsnotify_destroy_mark function (not confuse with > fsnotify_mark_destroy), which adds a mark to destroy_list to be freed > later by fsnotify_mark_destroy, I noticed that it does not increment > the reference count for the reference added to the destroy_list and > usually the callers dispose the references they held after calling > fsnotify_destroy_mark. > > The patch below increments the reference count of a mark when it is > added to the destroy list. It seems to solve the issue and it doesn't > seem to cause any memory leak. Please, can you make some tests in your > environments and let me know if there is any problem with this patch. > > Regarding the synchronize_scru call, I don't think it's causing this > error. Probably it just make it more frequently because it forces all > the cpus to schedule, giving the chance to someone else to free the > mark. > > --- > =A0fs/notify/mark.c | =A0 =A01 + > =A01 file changed, 1 insertion(+) > > diff --git a/fs/notify/mark.c b/fs/notify/mark.c > index f104d56..2985fff 100644 > --- a/fs/notify/mark.c > +++ b/fs/notify/mark.c > @@ -150,6 +150,7 @@ void fsnotify_destroy_mark(struct fsnotify_mark > *mark) spin_unlock(&group->mark_lock); > =A0 =A0spin_unlock(&mark->lock); > > + =A0 fsnotify_get_mark(mark); > =A0 =A0spin_lock(&destroy_lock); > =A0 =A0list_add(&mark->destroy_list, &destroy_list); > =A0 =A0spin_unlock(&destroy_lock); > -- > 1.7.9.4 > > > On Tue, 17 Apr 2012 14:54:29 -0700 > Peter Moody wrote: > >> Last thing. moving synchronize_srcu(&fsnotify_mark_srcu) out of the >> for(;;) loop in fs/notify/mark.c appears to solve the stability issues >> for me. I don't know enough about kernel internals to determine if >> this is doing lots of other bad things to my system or not. >> >> Cheers, >> peter >> >> On Tue, Apr 17, 2012 at 11:24 AM, Peter Moody >> wrote: >> > and my config.gz >> > >> > On Tue, Apr 17, 2012 at 10:56 AM, Peter Moody >> > wrote: >> >> Here's a trace with debugging turned way up plus a few extra >> >> printk's added to fs/notify/mark.c. I'm looping through >> >> private_destroy_list before and after the call to synchronize_srcu. >> >> >> >> I can reproduce this reliably with kvm with 2 virtual processors: >> >> Linux desktop 3.4.0-rc3-oops1+ #1 SMP Tue Apr 17 09:59:44 PDT 2012 >> >> x86_64 GNU/Linux >> >> >> >> Cheers, >> >> peter >> >> >> >> On Thu, Apr 5, 2012 at 2:07 PM, Eric Paris >> >> wrote: >> >>> please please please keep on list. =A0Everything you say might help >> >>> track it down! >> >>> >> >>> On Thu, 2012-04-05 at 14:03 -0700, Peter Moody wrote: >> >>>> (please let me know if I should take this off-list) >> >>>> >> >>>> One other thing (again, maybe already known), but this seems to >> >>>> be exacerbated by SMP. On my machine, I can't reproduce the >> >>>> crash if I booth with maxcpus=3D1. >> >>>> >> >>>> Still hunting. >> >>>> >> >>>> Cheers, >> >>>> peter >> >>>> >> >>>> On Tue, Apr 3, 2012 at 9:15 AM, Peter Moody >> >>>> wrote: >> >>>> > This may already be known, but the issue seems to be limited >> >>>> > to watch rules. With any watch rules, I can reliably crash my >> >>>> > machine while freeing a watch rule after only >> >>>> > starting/stopping auditd a few times. With no watch rules, I >> >>>> > have no issues. >> >>>> > >> >>>> > Cheers, >> >>>> > peter >> >>>> > >> >>>> > On Wed, Mar 28, 2012 at 11:44 PM, Valentin Avram >> >>>> > wrote: >> >>>> >> Yes, i know that patch. It made it into kernel 3.2.2. I >> >>>> >> tested it successfully (oops in 3.2.1, no oops in 3.2.9), but >> >>>> >> this oops i'm seeing is also in 3.2.9. >> >>>> >> >> >>>> >> I monitored changelogs since 3.2.1 to 3.2.12 but there were >> >>>> >> no fixes either in audit subsystem or in fsnotify. I'll try >> >>>> >> to reproduce in latest 3.2.13 and repost the oops, but i'm >> >>>> >> 99% confident it will be the same. >> >>>> >> >> >>>> >> Sadly nobody except you seems to pay attention to this >> >>>> >> problem, probably because it requires special conditions to >> >>>> >> reproduce (really, who starts and stops auditd every 5 >> >>>> >> seconds on a production server?). We only ran into it because >> >>>> >> one of our servers would randomly oops and then freeze about >> >>>> >> each month after stopping and then starting >> >>>> >> >> >>>> >> auditd >> >>>> >> >> >>>> >> every morning (and the stop-start sequence was needed to >> >>>> >> workaround a bug somewhere that would hang a >> >>>> >> >> >>>> >> gzip >> >>>> >> >> >>>> >> running on a file outside a watched folder). >> >>>> >> >> >>>> >> Anyway, as a last note, i have a feeling that the oops is not >> >>>> >> exactly random, there is a pattern, just that i haven't >> >>>> >> figured it out completely yet. >> >>>> >> >> >>>> >> Will keep you >> >>>> >> >> >>>> >> uptodate >> >>>> >> >> >>>> >> with the things i find out. >> >>>> >> >> >>>> >> V. >> >>>> >> >> >>>> >> On Mar 29, 2012 4:14 AM, "Eric Paris" >> >>>> >> wrote: >> >>>> >>> >> >>>> >>> That patch fixes a BUG() . =A0The report has a NULL ptr deref >> >>>> >>> and some apparent list correuption.... =A0Sadly they aren't >> >>>> >>> the same.... >> >>>> >>> >> >>>> >>> On Wed, 2012-03-28 at 15:42 -0700, Peter Moody wrote: >> >>>> >>> > fyi: this patch [1] seems to fix the issue for me. The >> >>>> >>> > explanation in the subject would reliably oops my machine. >> >>>> >>> > >> >>>> >>> > [1] >> >>>> >>> > http://git.kernel.org/?p=3Dlinux/kernel/git/torvalds/linux-2.= 6.git;a=3Dcommit;h=3Dfed474857efbed79cd390d0aee224231ca718f63 >> >>>> >>> > >> >>>> >>> > On Wed, Mar 28, 2012 at 1:51 PM, Peter Moody >> >>>> >>> > wrote: >> >>>> >>> > > Are you still able to reliably reproduce this oops? I'm >> >>>> >>> > > trying to track this down because this bug (or a very >> >>>> >>> > > similar bug) is causing some significant headaches here >> >>>> >>> > > at work, but I haven't had a lot of luck. I'm using >> >>>> >>> > > usermode linux, though, so that might be interfering >> >>>> >>> > > with things. >> >>>> >>> > > >> >>>> >>> > > On Mon, Mar 5, 2012 at 12:35 AM, Valentin Avram >> >>>> >>> > > wrote: >> >>>> >>> > >> Finally i found some time and spare server to retest >> >>>> >>> > >> the oops and list_add >> >>>> >>> > >> corruptions i was getting with the 3.x kernels and >> >>>> >>> > >> auditd 2.1.3. >> >>>> >>> > >> >> >>>> >>> > >> I tested now with gentoo's latest stable >> >>>> >>> > >> 3.2.1-gentoo-r2 and kernel.org's >> >>>> >>> > >> 3.2.9. >> >>>> >>> > >> >> >>>> >>> > >> Both get the oops/BUG in the same way and after that, >> >>>> >>> > >> they keep pouring >> >>>> >>> > >> list_add corruptions with audit_prune_tre(truncated?) >> >>>> >>> > >> and auditctl as comms. >> >>>> >>> > >> >> >>>> >>> > >> Since this is not about Gentoo's kernel only, i'll post >> >>>> >>> > >> here the oops in >> >>>> >>> > >> 3.2.9 and also attach some list_add corruptions. >> >>>> >>> > >> >> >>>> >>> > >> 3.2.9 BUG: >> >>>> >>> > >> >> >>>> >>> > >> kernel: [ =A0301.240011] BUG: unable to handle kernel >> >>>> >>> > >> NULL pointer dereference >> >>>> >>> > >> at =A0 (null) >> >>>> >>> > >> kernel: [ =A0301.240305] IP: [] >> >>>> >>> > >> __list_del_entry+0x20/0xe0 kernel: [ =A0301.240481] *pdpt >> >>>> >>> > >> =3D 0000000000000000 *pde =3D f000ddc8f000ddc8 >> >>>> >>> > >> kernel: [ =A0301.240698] Oops: 0000 [#1] SMP >> >>>> >>> > >> kernel: [ =A0301.240910] >> >>>> >>> > >> kernel: [ =A0301.241030] Pid: 642, comm: fsnotify_mark >> >>>> >>> > >> Not tainted 3.2.9-drbd-version3 #1 Dell Inc. PowerEdge >> >>>> >>> > >> 2950/0CX396 kernel: [ =A0301.241370] EIP: >> >>>> >>> > >> 0060:[] EFLAGS: 00010287 CPU: 6 kernel: >> >>>> >>> > >> [ =A0301.241498] EIP is at __list_del_entry+0x20/0xe0 >> >>>> >>> > >> kernel: [ =A0301.241623] EAX: f4fae544 EBX: f47cffa4 ECX: >> >>>> >>> > >> ffffffff EDX: 00000000 kernel: [ =A0301.241751] ESI: >> >>>> >>> > >> f4fae544 EDI: f4fae508 EBP: f47cff7c ESP: f47cff64 >> >>>> >>> > >> kernel: [ =A0301.241879] =A0DS: 007b ES: 007b FS: 00d8 GS: >> >>>> >>> > >> 0000 SS: 0068 kernel: [ =A0301.242005] Process >> >>>> >>> > >> fsnotify_mark (pid: 642, ti=3Df47ce000 task=3Df4f47c00 >> >>>> >>> > >> task.ti=3Df47ce000) kernel: [ =A0301.242207] Stack: >> >>>> >>> > >> kernel: [ =A0301.242327] =A0c10813c0 f47cffa4 f4f47c00 >> >>>> >>> > >> f4e70888 f47cff7c f47cffa4 f47cffb8 c10f6976 >> >>>> >>> > >> kernel: [ =A0301.242882] =A0ffffffc3 f4f47c00 f4f47c00 >> >>>> >>> > >> 00000000 f4f47c00 c10530c0 f47cff9c f47cff9c >> >>>> >>> > >> kernel: [ =A0301.243438] =A0f4fae544 f4fae544 f4c47f58 >> >>>> >>> > >> 00000000 c10f68f0 f47cffe4 c1052834 00000000 >> >>>> >>> > >> kernel: [ =A0301.243995] Call Trace: >> >>>> >>> > >> kernel: [ =A0301.244119] =A0[] ? >> >>>> >>> > >> rcu_check_callbacks+0x110/0x110 >> >>>> >>> > >> kernel: [ =A0301.244248] =A0[] >> >>>> >>> > >> fsnotify_mark_destroy+0x86/0x120 kernel: [ =A0301.244377] >> >>>> >>> > >> =A0[] ? abort_exclusive_wait+0x80/0x80 kernel: >> >>>> >>> > >> [ =A0301.244504] =A0[] ? >> >>>> >>> > >> fsnotify_put_mark+0x30/0x30 kernel: [ =A0301.244631] >> >>>> >>> > >> =A0[] kthread+0x74/0x80 kernel: [ =A0301.244756] >> >>>> >>> > >> =A0[] ? kthread_flush_work_fn+0x10/0x10 kernel: >> >>>> >>> > >> [ =A0301.244885] =A0[] >> >>>> >>> > >> kernel_thread_helper+0x6/0xd kernel: [ =A0301.245011] >> >>>> >>> > >> Code: 55 f4 8b 45 f8 e9 75 ff ff ff 90 55 89 e5 53 83 >> >>>> >>> > >> ec 14 8b 08 8b 50 04 81 f9 00 01 10 00 74 24 81 fa 00 >> >>>> >>> > >> 02 20 00 0f 84 8e 00 00 00 <8b> 1a 39 d8 75 62 8b 59 04 >> >>>> >>> > >> 39 d8 75 35 89 51 04 89 0a 83 c4 14 >> >>>> >>> > >> kernel: [ =A0301.248195] EIP: [] >> >>>> >>> > >> __list_del_entry+0x20/0xe0 SS:ESP >> >>>> >>> > >> 0068:f47cff64 >> >>>> >>> > >> kernel: [ =A0301.248414] CR2: 0000000000000000 >> >>>> >>> > >> kernel: [ =A0301.248538] ---[ end trace >> >>>> >>> > >> 15082dbfb353f84c ]--- >> >>>> >>> > >> >> >>>> >>> > >> The kernel was compiled with the following DEBUG >> >>>> >>> > >> support (the bolded one >> >>>> >>> > >> were requested by Gentoo's Dev: >> >>>> >>> > >> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=3Dy >> >>>> >>> > >> CONFIG_SLUB_DEBUG=3Dy >> >>>> >>> > >> CONFIG_HAVE_DMA_API_DEBUG=3Dy >> >>>> >>> > >> CONFIG_X86_DEBUGCTLMSR=3Dy >> >>>> >>> > >> CONFIG_PNP_DEBUG_MESSAGES=3Dy >> >>>> >>> > >> CONFIG_AIC94XX_DEBUG=3Dy >> >>>> >>> > >> CONFIG_USB_DEBUG=3Dy >> >>>> >>> > >> CONFIG_DEBUG_KERNEL=3Dy >> >>>> >>> > >> CONFIG_SCHED_DEBUG=3Dy >> >>>> >>> > >> CONFIG_DEBUG_RT_MUTEXES=3Dy >> >>>> >>> > >> CONFIG_DEBUG_PI_LIST=3Dy >> >>>> >>> > >> CONFIG_DEBUG_BUGVERBOSE=3Dy >> >>>> >>> > >> CONFIG_DEBUG_INFO=3Dy >> >>>> >>> > >> CONFIG_DEBUG_MEMORY_INIT=3Dy >> >>>> >>> > >> CONFIG_DEBUG_LIST=3Dy >> >>>> >>> > >> CONFIG_DEBUG_STACKOVERFLOW=3Dy >> >>>> >>> > >> CONFIG_DEBUG_RODATA=3Dy >> >>>> >>> > >> CONFIG_DEBUG_RODATA_TEST=3Dy >> >>>> >>> > >> >> >>>> >>> > >> I attached the kernel config i used for 3.2.9 to >> >>>> >>> > >> generate this oops and >> >>>> >>> > >> warnings. >> >>>> >>> > >> >> >>>> >>> > >> From the list_add warnings that come after, out of 805 >> >>>> >>> > >> warnings i processed, >> >>>> >>> > >> after masking with XXXXX the PID and next=3D values that >> >>>> >>> > >> kept changing in >> >>>> >>> > >> every one, i got 26 types of MD5. I also attached the >> >>>> >>> > >> files relevant as an >> >>>> >>> > >> archive to this email. >> >>>> >>> > >> >> >>>> >>> > >> The Gentoo bug i opened is sleeping, it seems nobody >> >>>> >>> > >> has the time to at >> >>>> >>> > >> least test to confirm or not the problems i'm seeing >> >>>> >>> > >> (or everybody's thinking that nobody would restart >> >>>> >>> > >> auditd so often, so the bug it's not that >> >>>> >>> > >> serious). >> >>>> >>> > >> >> >>>> >>> > >> >> >>>> >>> > >> Thank you for your time. >> >>>> >>> > >> >> >>>> >>> > >> On Wed, Feb 8, 2012 at 6:11 PM, Valentin Avram >> >>>> >>> > >> wrote: >> >>>> >>> > >> >> >>>> >>> > >> >> >>>> >>> > >> -- >> >>>> >>> > >> Linux-audit mailing list >> >>>> >>> > >> Linux-audit@redhat.com >> >>>> >>> > >> https://www.redhat.com/mailman/listinfo/linux-audit >> >>>> >>> > > >> >>>> >>> > > >> >>>> >>> > > >> >>>> >>> > > -- >> >>>> >>> > > Peter Moody =A0 =A0 =A0Google =A0 =A01.650.253.7306 >> >>>> >>> > > Security Engineer =A0pgp:0xC3410038 >> >>>> >>> > >> >>>> >>> > >> >>>> >>> > >> >>>> >>> >> >>>> >>> >> >>>> >> >> >>>> > >> >>>> > >> >>>> > >> >>>> > -- >> >>>> > Peter Moody =A0 =A0 =A0Google =A0 =A01.650.253.7306 >> >>>> > Security Engineer =A0pgp:0xC3410038 >> >>>> >> >>>> >> >>>> >> >>> >> >>> >> >> >> >> >> >> >> >> -- >> >> Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 >> >> Security Engineer=A0 pgp:0xC3410038 >> > >> > >> > >> > -- >> > Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 >> > Security Engineer=A0 pgp:0xC3410038 >> >> >> > -- = Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 Security Engineer=A0 pgp:0xC3410038 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Paris Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Mon, 23 Apr 2012 12:26:16 -0400 Message-ID: <1335198376.8224.4.camel@localhost> References: <1327519203.4131.25.camel@localhost> <1332983643.384.8.camel@localhost> <1333660021.2273.0.camel@localhost> <20120420231424.1836e56b@oc8526070481.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20120420231424.1836e56b@oc8526070481.ibm.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Marcelo Cerri Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com On Fri, 2012-04-20 at 23:14 -0300, Marcelo Cerri wrote: > The patch below increments the reference count of a mark when it is > added to the destroy list. It seems to solve the issue and it doesn't > seem to cause any memory leak. Please, can you make some tests in your > environments and let me know if there is any problem with this patch. That is almost certainly the wrong thing to do. This test program should show a memory leak with your patch. If it doesn't show a memory leak then something is screwed up in inotify as well. #include #include #include int main(void) { int fd; int rc; struct inotify_event event[10]; fd = inotify_init(); if (fd < 0) return errno; while(1) { rc = inotify_add_watch(fd, "/tmp", IN_CLOSE_WRITE); if (rc < 0) return errno; rc = inotify_rm_watch(fd, rc); if (rc) return errno; rc = read(fd, event, sizeof(event)); if (rc < 0) return errno; } return 0; } The lifetime of an object is supposed to be from fsnotify_init_mark() until it's matching reference is dropped in fsnotify_mark_destroy(). It sounds to me like we are calling put somewhere in the audit code when we didn't previously call a get.... From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter Moody Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Mon, 23 Apr 2012 18:27:21 -0700 Message-ID: References: <1327519203.4131.25.camel@localhost> <1332983643.384.8.camel@localhost> <1333660021.2273.0.camel@localhost> <20120420231424.1836e56b@oc8526070481.ibm.com> <1335198376.8224.4.camel@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Received: from mx1.redhat.com (ext-mx12.extmail.prod.ext.phx2.redhat.com [10.5.110.17]) by int-mx01.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id q3O1Rqrn008493 for ; Mon, 23 Apr 2012 21:27:52 -0400 Received: from mail-iy0-f174.google.com (mail-iy0-f174.google.com [209.85.210.174]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q3O1Rp3u010728 for ; Mon, 23 Apr 2012 21:27:52 -0400 Received: by iadi9 with SMTP id i9so342397iad.33 for ; Mon, 23 Apr 2012 18:27:51 -0700 (PDT) In-Reply-To: <1335198376.8224.4.camel@localhost> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Eric Paris Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com On Mon, Apr 23, 2012 at 9:26 AM, Eric Paris wrote: > On Fri, 2012-04-20 at 23:14 -0300, Marcelo Cerri wrote: > >> The patch below increments the reference count of a mark when it is >> added to the destroy list. It seems to solve the issue and it doesn't >> seem to cause any memory leak. Please, can you make some tests in your >> environments and let me know if there is any problem with this patch. > > That is almost certainly the wrong thing to do. =A0This test program > should show a memory leak with your patch. =A0If it doesn't show a memory > leak then something is screwed up in inotify as well. > > #include > #include > #include > > int main(void) > { > =A0 =A0 =A0 =A0int fd; > =A0 =A0 =A0 =A0int rc; > =A0 =A0 =A0 =A0struct inotify_event event[10]; > > =A0 =A0 =A0 =A0fd =3D inotify_init(); > =A0 =A0 =A0 =A0if (fd < 0) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return errno; > > =A0 =A0 =A0 =A0while(1) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0rc =3D inotify_add_watch(fd, "/tmp", IN_CL= OSE_WRITE); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (rc < 0) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return errno; > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0rc =3D inotify_rm_watch(fd, rc); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (rc) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return errno; > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0rc =3D read(fd, event, sizeof(event)); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (rc < 0) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return errno; > =A0 =A0 =A0 =A0} > > =A0 =A0 =A0 =A0return 0; > } > > The lifetime of an object is supposed to be from fsnotify_init_mark() > until it's matching reference is dropped in fsnotify_mark_destroy(). =A0It > sounds to me like we are calling put somewhere in the audit code when we > didn't previously call a get.... > FWIW, bisecting points me to 75c1be487a690db43da2c1234fcacd84c982803c 75c1be487a690db43da2c1234fcacd84c982803c is the first bad commit commit 75c1be487a690db43da2c1234fcacd84c982803c Author: Eric Paris Date: Wed Jul 28 10:18:38 2010 -0400 fsnotify: srcu to protect read side of inode and vfsmount locks Currently reading the inode->i_fsnotify_marks or vfsmount->mnt_fsnotify_marks lists are protected by a spinlock on both = the read and the write side. This patch protects the read side of those li= sts with a new single srcu. Signed-off-by: Eric Paris :040000 040000 4b5d9b446eefaca96f8a89b8e9c2ef18da88534e 1abcff76e285ae57f5855b60857ef1708e937a0c M fs :040000 040000 a02d4ab5b164aa9282a342d73ebe3658f88b4539 3ca9f66ba26cc265d118e6c8558ff2214b9ed192 M include Cheers, peter -- = Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 Security Engineer=A0 pgp:0xC3410038 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Marcelo Cerri Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Tue, 24 Apr 2012 02:12:10 -0300 Message-ID: <20120424021210.283cd4cd@oc8526070481.ibm.com> References: <1332983643.384.8.camel@localhost> <1333660021.2273.0.camel@localhost> <20120420231424.1836e56b@oc8526070481.ibm.com> <1335198376.8224.4.camel@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from mx1.redhat.com (ext-mx16.extmail.prod.ext.phx2.redhat.com [10.5.110.21]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id q3O5CKRA028443 for ; Tue, 24 Apr 2012 01:12:20 -0400 Received: from e24smtp02.br.ibm.com (e24smtp02.br.ibm.com [32.104.18.86]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q3O5CIHn018074 for ; Tue, 24 Apr 2012 01:12:19 -0400 Received: from /spool/local by e24smtp02.br.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 24 Apr 2012 02:12:17 -0300 In-Reply-To: <1335198376.8224.4.camel@localhost> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Eric Paris Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com On Mon, 23 Apr 2012 12:26:16 -0400 Eric Paris wrote: > On Fri, 2012-04-20 at 23:14 -0300, Marcelo Cerri wrote: > > > The patch below increments the reference count of a mark when it is > > added to the destroy list. It seems to solve the issue and it > > doesn't seem to cause any memory leak. Please, can you make some > > tests in your environments and let me know if there is any problem > > with this patch. > > That is almost certainly the wrong thing to do. This test program > should show a memory leak with your patch. If it doesn't show a > memory leak then something is screwed up in inotify as well. Sorry, I should have tested the other features that also make use of fsnotify. You're right, my patch adds a memory leak for inotify (and probably for dnotify and fanotify too). > ... > > The lifetime of an object is supposed to be from fsnotify_init_mark() > until it's matching reference is dropped in fsnotify_mark_destroy(). > It sounds to me like we are calling put somewhere in the audit code > when we didn't previously call a get.... > Considering that the issue is specific to audit and it seems to occur only with watches on directories, I investigated the audit_tree.c file and found a probable cause. The untag_chunk() holds a reference to a mark at the begging of the function and releases it at the end of it (on the label out). However when it jumps to the "out" label, it calls fsnotify_put_mark once more. Peter and Valentin, can you test this new patch to check if it solves the oops problem? Eric, do you agree with this solution? Regards, Marcelo --- kernel/audit_tree.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/kernel/audit_tree.c b/kernel/audit_tree.c index 5bf0790..b5bd9f9 100644 --- a/kernel/audit_tree.c +++ b/kernel/audit_tree.c @@ -250,7 +250,6 @@ static void untag_chunk(struct node *p) spin_unlock(&hash_lock); spin_unlock(&entry->lock); fsnotify_destroy_mark(entry); - fsnotify_put_mark(entry); goto out; } @@ -293,7 +292,6 @@ static void untag_chunk(struct node *p) spin_unlock(&hash_lock); spin_unlock(&entry->lock); fsnotify_destroy_mark(entry); - fsnotify_put_mark(entry); goto out; Fallback: -- 1.7.9.4 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Paris Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Tue, 24 Apr 2012 14:31:39 -0400 Message-ID: <1335292299.10352.3.camel@localhost> References: <1332983643.384.8.camel@localhost> <1333660021.2273.0.camel@localhost> <20120420231424.1836e56b@oc8526070481.ibm.com> <1335198376.8224.4.camel@localhost> <20120424021210.283cd4cd@oc8526070481.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20120424021210.283cd4cd@oc8526070481.ibm.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Marcelo Cerri Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com On Tue, 2012-04-24 at 02:12 -0300, Marcelo Cerri wrote: > On Mon, 23 Apr 2012 12:26:16 -0400, Eric Paris wrote: > Considering that the issue is specific to audit and it seems to occur > only with watches on directories, I investigated the audit_tree.c file > and found a probable cause. The untag_chunk() holds a reference to a > mark at the begging of the function and releases it at the end of it (on > the label out). However when it jumps to the "out" label, it calls > fsnotify_put_mark once more. > > Peter and Valentin, can you test this new patch to check if it > solves the oops problem? > > Eric, do you agree with this solution? > > Regards, > Marcelo > > --- > kernel/audit_tree.c | 2 -- > 1 file changed, 2 deletions(-) > > diff --git a/kernel/audit_tree.c b/kernel/audit_tree.c > index 5bf0790..b5bd9f9 100644 > --- a/kernel/audit_tree.c > +++ b/kernel/audit_tree.c > @@ -250,7 +250,6 @@ static void untag_chunk(struct node *p) > spin_unlock(&hash_lock); > spin_unlock(&entry->lock); > fsnotify_destroy_mark(entry); > - fsnotify_put_mark(entry); > goto out; > } > > @@ -293,7 +292,6 @@ static void untag_chunk(struct node *p) > spin_unlock(&hash_lock); > spin_unlock(&entry->lock); > fsnotify_destroy_mark(entry); > - fsnotify_put_mark(entry); > goto out; > > Fallback: This looks right to me. The old audit logic before the switch to fsnotify was: - inotify_evict_watch(&chunk->watch); - mutex_unlock(&chunk->watch.inode->inotify_mutex); - put_inotify_watch(&chunk->watch); Which I changed to: + spin_unlock(&entry->lock); + fsnotify_destroy_mark_by_entry(entry); + fsnotify_put_mark(entry); The difference being that inotify_evict_watch() took a reference on chunk->watch, however fsnotify_destroy_mark_by_entry() does not. So the fsnotify_put_mark() was incorrect. I'd love to hear testing results, and I'm going to try to figure out if I screwed that up other places.... -Eric From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter Moody Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Tue, 24 Apr 2012 11:38:11 -0700 Message-ID: References: <1332983643.384.8.camel@localhost> <1333660021.2273.0.camel@localhost> <20120420231424.1836e56b@oc8526070481.ibm.com> <1335198376.8224.4.camel@localhost> <20120424021210.283cd4cd@oc8526070481.ibm.com> <1335292299.10352.3.camel@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Received: from mx1.redhat.com (ext-mx16.extmail.prod.ext.phx2.redhat.com [10.5.110.21]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id q3OIcgnT030243 for ; Tue, 24 Apr 2012 14:38:42 -0400 Received: from mail-iy0-f174.google.com (mail-iy0-f174.google.com [209.85.210.174]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q3OIcfsg007515 for ; Tue, 24 Apr 2012 14:38:41 -0400 Received: by iadi9 with SMTP id i9so1774948iad.33 for ; Tue, 24 Apr 2012 11:38:41 -0700 (PDT) In-Reply-To: <1335292299.10352.3.camel@localhost> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Eric Paris Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com On Tue, Apr 24, 2012 at 11:31 AM, Eric Paris wrote: > On Tue, 2012-04-24 at 02:12 -0300, Marcelo Cerri wrote: >> On Mon, 23 Apr 2012 12:26:16 -0400, Eric Paris wrote: > >> Considering that the issue is specific to audit and it seems to occur >> only with watches on directories, I investigated the audit_tree.c file >> and found a probable cause. The untag_chunk() holds a reference to a >> mark at the begging of the function and releases it at the end of it (on >> the label out). However when it jumps to the "out" label, it calls >> fsnotify_put_mark once more. >> >> Peter and Valentin, can you test this new patch to check if it >> solves the oops problem? >> >> Eric, do you agree with this solution? >> >> Regards, >> Marcelo >> >> --- >> =A0kernel/audit_tree.c | =A0 =A02 -- >> =A01 file changed, 2 deletions(-) >> >> diff --git a/kernel/audit_tree.c b/kernel/audit_tree.c >> index 5bf0790..b5bd9f9 100644 >> --- a/kernel/audit_tree.c >> +++ b/kernel/audit_tree.c >> @@ -250,7 +250,6 @@ static void untag_chunk(struct node *p) >> =A0 =A0 =A0 =A0 spin_unlock(&hash_lock); >> =A0 =A0 =A0 =A0 spin_unlock(&entry->lock); >> =A0 =A0 =A0 =A0 fsnotify_destroy_mark(entry); >> - =A0 =A0 =A0 fsnotify_put_mark(entry); >> =A0 =A0 =A0 =A0 goto out; >> =A0 =A0 } >> >> @@ -293,7 +292,6 @@ static void untag_chunk(struct node *p) >> =A0 =A0 spin_unlock(&hash_lock); >> =A0 =A0 spin_unlock(&entry->lock); >> =A0 =A0 fsnotify_destroy_mark(entry); >> - =A0 fsnotify_put_mark(entry); >> =A0 =A0 goto out; >> >> =A0Fallback: > > This looks right to me. =A0The old audit logic before the switch to > fsnotify was: > - =A0 =A0 =A0 inotify_evict_watch(&chunk->watch); > - =A0 =A0 =A0 mutex_unlock(&chunk->watch.inode->inotify_mutex); > - =A0 =A0 =A0 put_inotify_watch(&chunk->watch); > > Which I changed to: > + =A0 =A0 =A0 spin_unlock(&entry->lock); > + =A0 =A0 =A0 fsnotify_destroy_mark_by_entry(entry); > + =A0 =A0 =A0 fsnotify_put_mark(entry); > > The difference being that inotify_evict_watch() took a reference on > chunk->watch, however fsnotify_destroy_mark_by_entry() does not. =A0So the > fsnotify_put_mark() was incorrect. > > I'd love to hear testing results, and I'm going to try to figure out if > I screwed that up other places.... I'm testing this now. It looks good WRT the crash. I need to spend some more time testing be sure memory isn't leaking anywhere. > -Eric > -- = Peter Moody=A0 =A0 =A0 Google=A0 =A0 1.650.253.7306 Security Engineer=A0 pgp:0xC3410038 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Paris Subject: Re: Kernel oops+crash on repeated auditd restarts Date: Tue, 24 Apr 2012 15:06:45 -0400 Message-ID: <1335294405.10352.4.camel@localhost> References: <1332983643.384.8.camel@localhost> <1333660021.2273.0.camel@localhost> <20120420231424.1836e56b@oc8526070481.ibm.com> <1335198376.8224.4.camel@localhost> <20120424021210.283cd4cd@oc8526070481.ibm.com> <1335292299.10352.3.camel@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Peter Moody Cc: linux-audit@redhat.com List-Id: linux-audit@redhat.com On Tue, 2012-04-24 at 11:38 -0700, Peter Moody wrote: > On Tue, Apr 24, 2012 at 11:31 AM, Eric Paris wrote: > > On Tue, 2012-04-24 at 02:12 -0300, Marcelo Cerri wrote: > >> On Mon, 23 Apr 2012 12:26:16 -0400, Eric Paris wrote: > > I'd love to hear testing results, and I'm going to try to figure out if > > I screwed that up other places.... > > I'm testing this now. It looks good WRT the crash. I need to spend > some more time testing be sure memory isn't leaking anywhere. I just sent another version which fixed a couple of other places I believe I was doing ref counting wrong in the audit_tree code. Hopefully everyone can give that one a whirl.... -Eric