From mboxrd@z Thu Jan 1 00:00:00 1970 From: Satish Chandra Kilaru Subject: Re: Linux audit performance impact Date: Wed, 18 Feb 2015 16:21:20 -0500 Message-ID: References: <9DBA79E0CE64AA42B07DEDAAD0F7DB914165C5B7@G4W3222.americas.hpqcorp.net> <20150212182526.GX29998@madcap2.tricolour.ca> <9DBA79E0CE64AA42B07DEDAAD0F7DB914FA921C0@G9W0755.americas.hpqcorp.net> <2039734.h1hbe1jZF6@x2> <9DBA79E0CE64AA42B07DEDAAD0F7DB914FA92535@G9W0755.americas.hpqcorp.net> <20150218211341.GT18752@madcap2.tricolour.ca> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8288199939898505197==" Return-path: In-Reply-To: <20150218211341.GT18752@madcap2.tricolour.ca> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Richard Guy Briggs Cc: "linux-audit@redhat.com" List-Id: linux-audit@redhat.com --===============8288199939898505197== Content-Type: multipart/alternative; boundary=001a11354c26938f45050f6366e0 --001a11354c26938f45050f6366e0 Content-Type: text/plain; charset=UTF-8 HI Why/How will the user space tools switch over if the kernel does not support raw mode? Isn't it a chicken&egg issue? --Satish On Wed, Feb 18, 2015 at 4:13 PM, Richard Guy Briggs wrote: > On 15/02/17, Viswanath, Logeswari P (MCOU OSTL) wrote: > > I agree that changing the formatting of the records could break the > existing applications > > that consume them, and I didn't mean changing or eliminating of the > formatting completely. > > We agree that formatting is required for logging the records(as buffers) > into the log files. > > We are wondering if these records can be made available as RAW records > so that the > > analytical programs which are capable of reading them for processing can > perform better. > > There are tools that completely ignore any of the audit userspace suite > including libaudit, so changing the formatting in the kernel and > deferring to userspace to later do that formatting is not currently an > option. > > > This option of RAW mode for the events can be an additional option > > where, kauditd delivers the audit buffer without formatting. Any > > comments on this? > > For a transition period if we were to consider it, it would mean > rewriting *all* places in the kernel that generate audit messages and > provide two paths switched on this RAW mode for each one of them, then > copying all that duplication to userspace libaudit. > According to Linus' decree, it would need to remain that way until we > were certain that all tools including ones we don't know about had > switched over. > > > >On Monday, February 16, 2015 11:25:57 AM Viswanath, Logeswari P wrote: > > >> I configured the system to audit open system call alone instead of all > > > >the system calls (our loader program executes) and hence I saw the > > >> massive improvement in performance. My fix is not causing any change > > > >in the performance. I wrongly communicated that the fix is causing > > > >performance improvement. Sorry for that. > > > > > > >> As per the perf data, the format_decode is the function where most of > > >> the time is spent i.e. formatting the record in the buffer before > > > >delivering the data to user space. We need to eliminate formatting > > > >records to increase the performance. Any idea why we need to format > > > >the record and whether can we add an option (RAW) to deliver the > > > >record without formatting to user space? > > > > >Introducing any changes to the format of the record can cause all > analytical programs, both open source and proprietary, to stop working > correctly. This cannot be changed. > > > > > >I think there is room for improvement however. There are times when > strings are being glued together and a stpcpy works just fine. There are > times when a numeric hex conversion is being done and %x is very slow. Same > with %d. > > > > > >The other issue is that the audit system's philosophy has not been to > optimize the formatting of the event, because events _should_ be rare. > Meaning that if you are getting hundred of events per second, something is > seriously wrong with the rules. > > > > > >It has been optimized to provide as little impact as possible when > _not_ generating events. Meaning that we want it as fast as possible in > letting the system operate normally. > > > > > >Again, there is room for improvement in both cases of triggering and > not triggering events. But the format of events can't really change without > a lot of coordination. I have a test suite here: > > > > > >http://people.redhat.com/sgrubb/audit/ausearch-test-0.5.tar.gz > > > > > >That can check that events are searchable by the main audit utility. If > changes cause that to fail, then its a sign you'll break the whole world. > > > > > >-Steve > > > > > > - RGB > > -- > Richard Guy Briggs > Senior Software Engineer, Kernel Security, AMER ENG Base Operating > Systems, Red Hat > Remote, Ottawa, Canada > Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545 > > -- > Linux-audit mailing list > Linux-audit@redhat.com > https://www.redhat.com/mailman/listinfo/linux-audit > -- Please Donate to www.wikipedia.org --001a11354c26938f45050f6366e0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
HI

Why/How will the user space tools sw= itch over if the kernel does not support raw mode?
Isn't it a= chicken&egg issue?

--Satish

On Wed, Feb 18, 2015 at= 4:13 PM, Richard Guy Briggs <rgb@redhat.com> wrote:
On 15/02/17, Viswanath, Logeswari = P (MCOU OSTL) wrote:
> I agree that changing the formatting of the records could break the ex= isting applications
> that consume them, and I didn't mean changing or eliminating of th= e formatting completely.
> We agree that formatting is required for logging the records(as buffer= s) into the log files.
> We are wondering if these records can be made available as RAW records= so that the
> analytical programs which are capable of reading them for processing c= an perform better.

There are tools that completely ignore any of the audit userspace su= ite
including libaudit, so changing the formatting in the kernel and
deferring to userspace to later do that formatting is not currently an
option.

> This option of RAW mode for the events can be an additional option
> where, kauditd delivers the audit buffer without formatting. Any
> comments on this?

For a transition period if we were to consider it, it would mean
rewriting *all* places in the kernel that generate audit messages and
provide two paths switched on this RAW mode for each one of them, then
copying all that duplication to userspace libaudit.
According to Linus' decree, it would need to remain that way until we were certain that all tools including ones we don't know about had
switched over.

> >On Monday, February 16, 2015 11:25:57 AM Viswanath, Logeswari P wr= ote:
> >> I configured the system to audit open system call alone inste= ad of all
> > >the system calls (our loader program executes) and hence I sa= w the
> >> massive improvement in performance. My fix is not causing any= change
> > >in the performance. I wrongly communicated that the fix is ca= using
> > >performance improvement. Sorry for that.
> > >
> >> As per the perf data, the format_decode is the function where= most of
> >> the time is spent i.e. formatting the record in the buffer be= fore
> > >delivering the data to user space. We need to eliminate forma= tting
> > >records to increase the performance. Any idea why we need to = format
> > >the record and whether can we add an option (RAW) to deliver = the
> > >record without formatting to user space?
>
> >Introducing any changes to the format of the record can cause all = analytical programs, both open source and proprietary, to stop working corr= ectly. This cannot be changed.
> >
> >I think there is room for improvement however. There are times whe= n strings are being glued together and a stpcpy works just fine. There are = times when a numeric hex conversion is being done and %x is very slow. Same= with %d.
> >
> >The other issue is that the audit system's philosophy has not = been to optimize the formatting of the event, because events _should_ be ra= re. Meaning that if you are getting hundred of events per second, something= is seriously wrong with the rules.
> >
> >It has been optimized to provide as little impact as possible when= _not_ generating events. Meaning that we want it as fast as possible in le= tting the system operate normally.
> >
> >Again, there is room for improvement in both cases of triggering a= nd not triggering events. But the format of events can't really change = without a lot of coordination. I have a test suite here:
> >
> >http://people.redhat.com/sgrubb/audit/ausearch-t= est-0.5.tar.gz
> >
> >That can check that events are searchable by the main audit utilit= y. If changes cause that to fail, then its a sign you'll break the whol= e world.
> >
> >-Steve
>
>

- RGB

--
Richard Guy Briggs <rbriggs@redhat= .com>
Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems,= Red Hat
Remote, Ottawa, Canada
Voice: +1.647.777.= 2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit



--
=
Please Donate to www.wikipedia.org
--001a11354c26938f45050f6366e0-- --===============8288199939898505197== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============8288199939898505197==--