Maximum Value for q_depth

Maximum Value for q_depth

[Amjad Gabbar]

[-- Attachment #1.1: Type: text/plain, Size: 375 bytes --]

I am currently seeing a lot of auditd dispatch error issues. It is related
to a particular keyed rule that from the looks of it is generating close to
a million events /day. I have seen previous answers where it was advised to
increase the q_depth value to a suitable number.

Based on this, I would like to confirm what is the maximum advisable value
q_depth can have/take?

[-- Attachment #1.2: Type: text/html, Size: 421 bytes --]

[-- Attachment #2: Type: text/plain, Size: 106 bytes --]

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Maximum Value for q_depth

[Steve Grubb]

Hello,

On Tuesday, November 30, 2021 6:04:28 PM EST Amjad Gabbar wrote:
> I am currently seeing a lot of auditd dispatch error issues.

What version of auditd and what plugins do you have?

> It is related to a particular keyed rule that from the looks of it is
> generating close to a million events /day. I have seen previous answers
> where it was advised to increase the q_depth value to a suitable number.
> 
> Based on this, I would like to confirm what is the maximum advisable value
> q_depth can have/take?

Depends on what you are willing to set it to. You can easily go to 64k, but 
you really ought to look at the plugins to see why they can't keep up. And of 
course, are the rules really designed right and you need the million events/
day?

-Steve


--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Fwd: Maximum Value for q_depth

[Amjad Gabbar]

[-- Attachment #1.1: Type: text/plain, Size: 1444 bytes --]

1. The version of auditd is 1:2.8.4-3 and the plugins are af_unix.conf and
syslog.conf for audisp. The q_depth is currently set to 80 and I think it
calls for an increase but not sure if there is a way to figure out what the
proper number would be?

2. Another thing I would like to follow up on is the difference between
q_depth and backlog_limit. My assumption was if there is any drop due to a
burst of events it would be addressed by the backlog limit. Just would like
some clarification on this and how this is an event dispatcher issue?

Thanks
Amjad

On Wed, Dec 1, 2021 at 10:00 AM Steve Grubb <sgrubb@redhat.com> wrote:

> Hello,
>
> On Tuesday, November 30, 2021 6:04:28 PM EST Amjad Gabbar wrote:
> > I am currently seeing a lot of auditd dispatch error issues.
>
> What version of auditd and what plugins do you have?




>
> > It is related to a particular keyed rule that from the looks of it is
> > generating close to a million events /day. I have seen previous answers
> > where it was advised to increase the q_depth value to a suitable number.
> >
> > Based on this, I would like to confirm what is the maximum advisable
> value
> > q_depth can have/take?
>
> Depends on what you are willing to set it to. You can easily go to 64k,
> but
> you really ought to look at the plugins to see why they can't keep up. And
> of
> course, are the rules really designed right and you need the million
> events/
> day?
>
> -Steve
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 2289 bytes --]

[-- Attachment #2: Type: text/plain, Size: 106 bytes --]

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Maximum Value for q_depth

[Steve Grubb]

Hello,

On Wednesday, December 8, 2021 4:54:52 PM EST Amjad Gabbar wrote:
> 1. The version of auditd is 1:2.8.4-3 and the plugins are af_unix.conf and
> syslog.conf for audisp. The q_depth is currently set to 80 and I think it
> calls for an increase but not sure if there is a way to figure out what the
> proper number would be?

There is no good calculation that I can give you. It depends on the average 
rate of incoming events and the rate that they can be offloaded to the plugins 
+ some margin in case there is a burst. Looking at the 2.8.5 code, the 
default is 250.

https://github.com/linux-audit/audit-userspace/blob/2.8_maintenance/init.d/
audispd.conf

So, you should at least set it that high. Maybe a bit higher.


> 2. Another thing I would like to follow up on is the difference between
> q_depth and backlog_limit. My assumption was if there is any drop due to a
> burst of events it would be addressed by the backlog limit. Just would like
> some clarification on this and how this is an event dispatcher issue?

The backlog limit is inside the kernel. This is the buffer that holds events 
that are waiting for the audit daemon to offload them. Once the audit daemon 
has them, it sends it to the dispatcher which also buffers events because not 
all plugins are able to receive the events as soon as they arrive at the 
dispatcher.

So, for brief bursts, the kernel backlog will handle the load. But once they 
are pulled out of the kernel, the q_depth controls how much to hold waiting 
for plugins. If this number needs to increase much, then the plugins are 
having problems. The syslog plugin should be fine. I'd look more at the 
af_unix plugin. The client that attaches to it needs to unload events 
quickly. I'd investigate the af_unix client to see if it's the problem.

Cheers,
-Steve
 

> On Wed, Dec 1, 2021 at 10:00 AM Steve Grubb <sgrubb@redhat.com> wrote:
> > Hello,
> > 
> > On Tuesday, November 30, 2021 6:04:28 PM EST Amjad Gabbar wrote:
> > > I am currently seeing a lot of auditd dispatch error issues.
> > 
> > What version of auditd and what plugins do you have?
> > 
> > > It is related to a particular keyed rule that from the looks of it is
> > > generating close to a million events /day. I have seen previous answers
> > > where it was advised to increase the q_depth value to a suitable
> > > number.
> > > 
> > > Based on this, I would like to confirm what is the maximum advisable
> > 
> > value
> > 
> > > q_depth can have/take?
> > 
> > Depends on what you are willing to set it to. You can easily go to 64k,
> > but
> > you really ought to look at the plugins to see why they can't keep up.
> > And
> > of
> > course, are the rules really designed right and you need the million
> > events/
> > day?
> > 
> > -Steve




--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Maximum Value for q_depth

[Amjad Gabbar]

[-- Attachment #1.1: Type: text/plain, Size: 2539 bytes --]

Got it. Makes sense to me. Thanks for the explanation Steve.

One last thing though based on the discussion we had, if the kernel is able
to offload events even during bursts, wouldn’t setting q_depth
=backlog_limit be enough?

The reason being if there was an overflow on the kernel side, a different
message would be printed in the logs but because it is all dispatch errors,
I assume the kernel is able to handle the burst which is why the reasoning
of increasing q_depth to backlog_limit.

Thanks
Amjad

On Wed, Dec 8, 2021 at 4:38 PM Steve Grubb <sgrubb@redhat.com> wrote:

> On Wednesday, December 8, 2021 4:54:18 PM EST Amjad Gabbar wrote:
> > 1. The version of auditd is 1:2.8.4-3 and the plugins are af_unix.conf
> and
> > syslog.conf for audisp. The q_depth is currently set to 80 and I think it
> > calls for an increase but not sure if there is a way to figure out what
> the
> > proper number would be?
>
> There is no good calculation that I can give you. It depends on the
> average
> rate of incoming events and the rate that they can be offloaded to the
> plugins
> + some margin in case there is a burst. Looking at the 2.8.5 code, the
> default is 250.
>
> https://github.com/linux-audit/audit-userspace/blob/2.8_maintenance/init.d/
> audispd.conf
>
> So, you should at least set it that high. Maybe a bit higher.
>
>
> > 2. Another thing I would like to follow up on is the difference between
> > q_depth and backlog_limit. My assumption was if there is any drop due to
> a
> > burst of events it would be addressed by the backlog limit. Just would
> like
> > some clarification on this and how this is an event dispatcher issue?
>
> The backlog limit is inside the kernel. This is the buffer that holds
> events
> that are waiting for the audit daemon to offload them. Once the audit
> daemon
> has them, it sends it to the dispatcher which also buffers events because
> not
> all plugins are able to receive the events as soon as they arrive at the
> dispatcher.
>
> So, for brief bursts, the kernel backlog will handle the load. But once
> they
> are pulled out of the kernel, the q_depth controls how much to hold
> waiting
> for plugins. If this number needs to increase much, then the plugins are
> having problems. The syslog plugin should be fine. I'd look more at the
> af_unix plugin. The client that attaches to it needs to unload events
> quickly. I'd investigate the af_unix client to see if it's the problem.
>
> Cheers,
> -Steve
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 3177 bytes --]

[-- Attachment #2: Type: text/plain, Size: 106 bytes --]

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Maximum Value for q_depth

[Steve Grubb]

Hello,

On Wednesday, December 8, 2021 11:00:50 PM EST Amjad Gabbar wrote:
> Got it. Makes sense to me. Thanks for the explanation Steve.
> 
> One last thing though based on the discussion we had, if the kernel is able
> to offload events even during bursts, wouldn’t setting q_depth
> =backlog_limit be enough?

Depends on the client attached to the af_unix socket. The kernel could have a 
burst and then another. If the client isn't reading the af_unix socket fast 
enough, there could still be  overflows. That said, it is likely to be enough. 
But I'd look into the client app that reads the af_unix socket to see why 
it's taking so long.

> The reason being if there was an overflow on the kernel side, a different
> message would be printed in the logs but because it is all dispatch errors,
> I assume the kernel is able to handle the burst which is why the reasoning
> of increasing q_depth to backlog_limit.

The q_depth being 90 is your likely problem. Realistically, that only allows 
20 - 30 events to be enqueued.

-Steve

> On Wed, Dec 8, 2021 at 4:38 PM Steve Grubb <sgrubb@redhat.com> wrote:
> > On Wednesday, December 8, 2021 4:54:18 PM EST Amjad Gabbar wrote:
> > > 1. The version of auditd is 1:2.8.4-3 and the plugins are af_unix.conf
> > 
> > and
> > 
> > > syslog.conf for audisp. The q_depth is currently set to 80 and I think
> > > it
> > > calls for an increase but not sure if there is a way to figure out what
> > 
> > the
> > 
> > > proper number would be?
> > 
> > There is no good calculation that I can give you. It depends on the
> > average
> > rate of incoming events and the rate that they can be offloaded to the
> > plugins
> > + some margin in case there is a burst. Looking at the 2.8.5 code, the
> > default is 250.
> > 
> > https://github.com/linux-audit/audit-userspace/blob/2.8_maintenance/init.
> > d/ audispd.conf
> > 
> > So, you should at least set it that high. Maybe a bit higher.
> > 
> > > 2. Another thing I would like to follow up on is the difference between
> > > q_depth and backlog_limit. My assumption was if there is any drop due
> > > to
> > 
> > a
> > 
> > > burst of events it would be addressed by the backlog limit. Just would
> > 
> > like
> > 
> > > some clarification on this and how this is an event dispatcher issue?
> > 
> > The backlog limit is inside the kernel. This is the buffer that holds
> > events
> > that are waiting for the audit daemon to offload them. Once the audit
> > daemon
> > has them, it sends it to the dispatcher which also buffers events because
> > not
> > all plugins are able to receive the events as soon as they arrive at the
> > dispatcher.
> > 
> > So, for brief bursts, the kernel backlog will handle the load. But once
> > they
> > are pulled out of the kernel, the q_depth controls how much to hold
> > waiting
> > for plugins. If this number needs to increase much, then the plugins are
> > having problems. The syslog plugin should be fine. I'd look more at the
> > af_unix plugin. The client that attaches to it needs to unload events
> > quickly. I'd investigate the af_unix client to see if it's the problem.
> > 
> > Cheers,
> > -Steve





--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Maximum Value for q_depth

[Amjad Gabbar]

[-- Attachment #1.1: Type: text/plain, Size: 5402 bytes --]

Hi Steve,

Based on our discussion above, I performed some analysis as to why we were
seeing so many events. The reason seems to be due to the default rules
being triggered every time a cron job runs. We have numerous cron jobs
running per minute as a result of which multiple different events(LOGIN,
USER_END,CRED_DISP etc) are generated each time a cron job runs. As we do
not enable SELinux, disabling these thing use subj_type=crond_t is not a
viable option.

1. I have tried the following way to exclude using msg_type and exe
together and it seems to work.

-a exclude,always -F msgtype=MAC_IPSEC_EVENT -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=USER_AUTH -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=USER_ACCT -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=CRED_REFR -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=CRED_DISP -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=CRED_ACQ -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=USER_START -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=USER_END -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=SERVICE_START -F exe=/usr/sbin/cron

Just want to make sure there is nothing I am missing here and that this
only excludes the msg types for the cron executable.

2. Apart from these messages, there is a LOGIN message that gets generated
each time a cron runs. Eventhough, the LOGIN message in auditd does not
have an exe field, the following statement surprisingly seems to be working.

-a exclude,always -F msgtype=LOGIN -F exe=/usr/sbin/cron

I can still see LOGIN messages for other users but the cron LOGIN messages
seem to be suppressed. Could you provide some detail as to how this is
happening and is the expected result.

3. Is there a better way to suppress these cron messages that I am not
considering apart from the SELinux option mentioned.

Thank You.

On Thu, Dec 9, 2021 at 8:19 AM Steve Grubb <sgrubb@redhat.com> wrote:

> Hello,
>
> On Wednesday, December 8, 2021 11:00:50 PM EST Amjad Gabbar wrote:
> > Got it. Makes sense to me. Thanks for the explanation Steve.
> >
> > One last thing though based on the discussion we had, if the kernel is
> able
> > to offload events even during bursts, wouldn’t setting q_depth
> > =backlog_limit be enough?
>
> Depends on the client attached to the af_unix socket. The kernel could
> have a
> burst and then another. If the client isn't reading the af_unix socket
> fast
> enough, there could still be  overflows. That said, it is likely to be
> enough.
> But I'd look into the client app that reads the af_unix socket to see why
> it's taking so long.
>
> > The reason being if there was an overflow on the kernel side, a different
> > message would be printed in the logs but because it is all dispatch
> errors,
> > I assume the kernel is able to handle the burst which is why the
> reasoning
> > of increasing q_depth to backlog_limit.
>
> The q_depth being 90 is your likely problem. Realistically, that only
> allows
> 20 - 30 events to be enqueued.
>
> -Steve
>
> > On Wed, Dec 8, 2021 at 4:38 PM Steve Grubb <sgrubb@redhat.com> wrote:
> > > On Wednesday, December 8, 2021 4:54:18 PM EST Amjad Gabbar wrote:
> > > > 1. The version of auditd is 1:2.8.4-3 and the plugins are
> af_unix.conf
> > >
> > > and
> > >
> > > > syslog.conf for audisp. The q_depth is currently set to 80 and I
> think
> > > > it
> > > > calls for an increase but not sure if there is a way to figure out
> what
> > >
> > > the
> > >
> > > > proper number would be?
> > >
> > > There is no good calculation that I can give you. It depends on the
> > > average
> > > rate of incoming events and the rate that they can be offloaded to the
> > > plugins
> > > + some margin in case there is a burst. Looking at the 2.8.5 code, the
> > > default is 250.
> > >
> > >
> https://github.com/linux-audit/audit-userspace/blob/2.8_maintenance/init.
> > > d/ audispd.conf
> > >
> > > So, you should at least set it that high. Maybe a bit higher.
> > >
> > > > 2. Another thing I would like to follow up on is the difference
> between
> > > > q_depth and backlog_limit. My assumption was if there is any drop due
> > > > to
> > >
> > > a
> > >
> > > > burst of events it would be addressed by the backlog limit. Just
> would
> > >
> > > like
> > >
> > > > some clarification on this and how this is an event dispatcher issue?
> > >
> > > The backlog limit is inside the kernel. This is the buffer that holds
> > > events
> > > that are waiting for the audit daemon to offload them. Once the audit
> > > daemon
> > > has them, it sends it to the dispatcher which also buffers events
> because
> > > not
> > > all plugins are able to receive the events as soon as they arrive at
> the
> > > dispatcher.
> > >
> > > So, for brief bursts, the kernel backlog will handle the load. But once
> > > they
> > > are pulled out of the kernel, the q_depth controls how much to hold
> > > waiting
> > > for plugins. If this number needs to increase much, then the plugins
> are
> > > having problems. The syslog plugin should be fine. I'd look more at the
> > > af_unix plugin. The client that attaches to it needs to unload events
> > > quickly. I'd investigate the af_unix client to see if it's the problem.
> > >
> > > Cheers,
> > > -Steve
>
>
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 10668 bytes --]

[-- Attachment #2: Type: text/plain, Size: 106 bytes --]

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Maximum Value for q_depth

[Steve Grubb]

Hello,

On Tuesday, December 21, 2021 12:55:47 AM EST Amjad Gabbar wrote:
> Based on our discussion above, I performed some analysis as to why we were
> seeing so many events. The reason seems to be due to the default rules
> being triggered every time a cron job runs. We have numerous cron jobs
> running per minute as a result of which multiple different events(LOGIN,
> USER_END,CRED_DISP etc) are generated each time a cron job runs. As we do
> not enable SELinux, disabling these thing use subj_type=crond_t is not a
> viable option.
> 
> 1. I have tried the following way to exclude using msg_type and exe
> together and it seems to work.
> 
> -a exclude,always -F msgtype=MAC_IPSEC_EVENT -F exe=/usr/sbin/cron
> -a exclude,always -F msgtype=USER_AUTH -F exe=/usr/sbin/cron
> -a exclude,always -F msgtype=USER_ACCT -F exe=/usr/sbin/cron
> -a exclude,always -F msgtype=CRED_REFR -F exe=/usr/sbin/cron
> -a exclude,always -F msgtype=CRED_DISP -F exe=/usr/sbin/cron
> -a exclude,always -F msgtype=CRED_ACQ -F exe=/usr/sbin/cron
> -a exclude,always -F msgtype=USER_START -F exe=/usr/sbin/cron
> -a exclude,always -F msgtype=USER_END -F exe=/usr/sbin/cron
> -a exclude,always -F msgtype=SERVICE_START -F exe=/usr/sbin/cron
> 
> Just want to make sure there is nothing I am missing here and that this
> only excludes the msg types for the cron executable.

I think so. But it's easy enough to test. Just login and see if you get any 
USER_START events from something other than cron.

> 2. Apart from these messages, there is a LOGIN message that gets generated
> each time a cron runs. Eventhough, the LOGIN message in auditd does not
> have an exe field, the following statement surprisingly seems to be
> working.
> 
> -a exclude,always -F msgtype=LOGIN -F exe=/usr/sbin/cron
> 
> I can still see LOGIN messages for other users but the cron LOGIN messages
> seem to be suppressed. Could you provide some detail as to how this is
> happening and is the expected result.

It doesn't match against the text in the event. It matches against the 
process's attributes.

> 3. Is there a better way to suppress these cron messages that I am not
> considering apart from the SELinux option mentioned.

I think you found the best way for a non-selinux system. Back when it was 
documented that it could be supressed by selinux type, audit by executable 
did not exist. But as you found, that is an effective way to get rid of the 
events.

I also think the cronie program might be a little more audit friendly. It 
does not call PAM for the system crontabs run under the root user. PAM is run 
only for the local crontab (i.e. the one edited by the crontab command) and 
in case of the system crontabs only for jobs that are run under non-root 
user.

-Steve


--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Maximum Value for q_depth

[Amjad Gabbar]

[-- Attachment #1.1: Type: text/plain, Size: 5225 bytes --]

Hey Steve,

Reaching out regarding the same issue of syslog containing *"auditd
dispatch error (pipe full) event lost messages". *

Post excluding the default events(LOGIN, USER_START etc) mentioned in our
previous chat, there has been a significant drop in the log volume and
hence I was expecting these error messages to be resolved.

But unfortunately, even after increasing the dispatcher queue size(q_depth)
and changing disp_qos to become lossless , I am still seeing mentions of
these pipe full errors in my syslog.
The surprising thing is if I try to take a look at the events/keys causing
this issue, there doesn't seem to be a lot of events for messages to be
dropped.

Ex- Using the command *"aureport --summary -ts <start time of dropped
messages start reported in syslog > -te < end time  of dropped messages
end reported in syslog > -i (-x/-u/--key)"*, the total events are around
2000 during this time period. The dispatcher queue size is close to 25,000,
So I am not really sure why the dispatcher is unable to handle these
messages. The queue size is sufficient enough to handle 10x the total
events being seen.

Some other theoretical questions I had surrounding this are:


   - The audit daemon picks events from the kernel buffer and sends it to
   the dispatcher buffer. Who writes these logs to /var/log/audit.log - is it
   the daemon or the dispatcher? And also, are the total events reported in
   /var/log/audit.log inclusive of the dropped events reported in syslog or
   exclusive? i.e is it possible that all the events have been recorded in
   audit.log but syslog has an issue in keeping up with the events as it is
   the only plugin that is being used by the dispatcher.
   - Is there a way to find out what is the total number of events dropped
   by the dispatcher?
   - In auditd v3+, the daemon itself handles dispatching capabilities. So,
   what does q_depth refer to in this scenario?
   - In the man pages for different distros for disp_qos the following
   statement is common - " There is a 128k buffer between the audit daemon
   and dispatcher." But different distros seem to have different default
   values for q_depth ranging from 80 to 1200. How is it possible that these
   numbers vary but the size of the buffer remains 128k.




On Tue, Dec 21, 2021 at 2:39 PM Steve Grubb <sgrubb@redhat.com> wrote:

> Hello,
>
> On Tuesday, December 21, 2021 12:55:47 AM EST Amjad Gabbar wrote:
> > Based on our discussion above, I performed some analysis as to why we
> were
> > seeing so many events. The reason seems to be due to the default rules
> > being triggered every time a cron job runs. We have numerous cron jobs
> > running per minute as a result of which multiple different events(LOGIN,
> > USER_END,CRED_DISP etc) are generated each time a cron job runs. As we do
> > not enable SELinux, disabling these thing use subj_type=crond_t is not a
> > viable option.
> >
> > 1. I have tried the following way to exclude using msg_type and exe
> > together and it seems to work.
> >
> > -a exclude,always -F msgtype=MAC_IPSEC_EVENT -F exe=/usr/sbin/cron
> > -a exclude,always -F msgtype=USER_AUTH -F exe=/usr/sbin/cron
> > -a exclude,always -F msgtype=USER_ACCT -F exe=/usr/sbin/cron
> > -a exclude,always -F msgtype=CRED_REFR -F exe=/usr/sbin/cron
> > -a exclude,always -F msgtype=CRED_DISP -F exe=/usr/sbin/cron
> > -a exclude,always -F msgtype=CRED_ACQ -F exe=/usr/sbin/cron
> > -a exclude,always -F msgtype=USER_START -F exe=/usr/sbin/cron
> > -a exclude,always -F msgtype=USER_END -F exe=/usr/sbin/cron
> > -a exclude,always -F msgtype=SERVICE_START -F exe=/usr/sbin/cron
> >
> > Just want to make sure there is nothing I am missing here and that this
> > only excludes the msg types for the cron executable.
>
> I think so. But it's easy enough to test. Just login and see if you get
> any
> USER_START events from something other than cron.
>
> > 2. Apart from these messages, there is a LOGIN message that gets
> generated
> > each time a cron runs. Eventhough, the LOGIN message in auditd does not
> > have an exe field, the following statement surprisingly seems to be
> > working.
> >
> > -a exclude,always -F msgtype=LOGIN -F exe=/usr/sbin/cron
> >
> > I can still see LOGIN messages for other users but the cron LOGIN
> messages
> > seem to be suppressed. Could you provide some detail as to how this is
> > happening and is the expected result.
>
> It doesn't match against the text in the event. It matches against the
> process's attributes.
>
> > 3. Is there a better way to suppress these cron messages that I am not
> > considering apart from the SELinux option mentioned.
>
> I think you found the best way for a non-selinux system. Back when it was
> documented that it could be supressed by selinux type, audit by executable
> did not exist. But as you found, that is an effective way to get rid of
> the
> events.
>
> I also think the cronie program might be a little more audit friendly. It
> does not call PAM for the system crontabs run under the root user. PAM is
> run
> only for the local crontab (i.e. the one edited by the crontab command)
> and
> in case of the system crontabs only for jobs that are run under non-root
> user.
>
> -Steve
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 6234 bytes --]

[-- Attachment #2: Type: text/plain, Size: 106 bytes --]

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Maximum Value for q_depth

[Steve Grubb]

Hello,

On Tuesday, January 18, 2022 1:36:40 AM EST Amjad Gabbar wrote:
> Reaching out regarding the same issue of syslog containing *"auditd
> dispatch error (pipe full) event lost messages". *
> 
> Post excluding the default events(LOGIN, USER_START etc) mentioned in our
> previous chat, there has been a significant drop in the log volume and
> hence I was expecting these error messages to be resolved.
> 
> But unfortunately, even after increasing the dispatcher queue size(q_depth)
> and changing disp_qos to become lossless , I am still seeing mentions of
> these pipe full errors in my syslog.
> The surprising thing is if I try to take a look at the events/keys causing
> this issue, there doesn't seem to be a lot of events for messages to be
> dropped.

Dispatcher plugins can also cause things to get backed up. You mention that 
you have the af_unix plugin. Whatever reads from that needs to unload events 
quickly. It's recommended for the plugin to have 2 threads, one to dequeue 
and one to process the event. It should have storage to hold events if it's 
processing gets behind.


> Ex- Using the command *"aureport --summary -ts <start time of dropped
> messages start reported in syslog > -te < end time  of dropped messages
> end reported in syslog > -i (-x/-u/--key)"*, the total events are around
> 2000 during this time period. The dispatcher queue size is close to 25,000,
> So I am not really sure why the dispatcher is unable to handle these
> messages. The queue size is sufficient enough to handle 10x the total
> events being seen.

Its entirely depending on the plugin to grab it's event.

> Some other theoretical questions I had surrounding this are:
> 
> 
>    - The audit daemon picks events from the kernel buffer and sends it to
>    the dispatcher buffer. Who writes these logs to /var/log/audit.log - is
> it the daemon or the dispatcher?

The daemon

> And also, are the total events reported
> in /var/log/audit.log inclusive of the dropped events reported in syslog
> or exclusive?

It writes to the log and then dispatches the event.

> i.e is it possible that all the events have been recorded in
> audit.log but syslog has an issue in keeping up with the events as it is
> the only plugin that is being used by the dispatcher.

You previously mentioned an af_unix plugin. That would be my guess. You can 
disable the syslog plugin and find out if it's the cause.

>    - Is there a way to find out what is the total number of events dropped
>    by the dispatcher?

I don't think anything keeps metrics on that.

>    - In auditd v3+, the daemon itself handles dispatching capabilities. So,
> what does q_depth refer to in this scenario?

The internal queue between the logging thread and the dispatching thread.

>    - In the man pages for different distros for disp_qos the following
>    statement is common - " There is a 128k buffer between the audit daemon
>    and dispatcher." 

This buffer is the kernel inter-process pipe size. There are up to 4 buffers 
in the 2.8 series. The backlog, the inter-process kernel buffer between 
auditd and audispd, a buffer in audispd, and the inter-process kernel buffer 
with the plugin. Only 2 of those have config options.

>    But different distros seem to have different default
>    values for q_depth ranging from 80 to 1200. How is it possible that
> these numbers vary but the size of the buffer remains 128k.

It's a different buffer. The 128k refers to the inter-process pipe buffer.

-Steve

> On Tue, Dec 21, 2021 at 2:39 PM Steve Grubb <sgrubb@redhat.com> wrote:
> > Hello,
> > 
> > On Tuesday, December 21, 2021 12:55:47 AM EST Amjad Gabbar wrote:
> > > Based on our discussion above, I performed some analysis as to why we
> > 
> > were
> > 
> > > seeing so many events. The reason seems to be due to the default rules
> > > being triggered every time a cron job runs. We have numerous cron jobs
> > > running per minute as a result of which multiple different
> > > events(LOGIN,
> > > USER_END,CRED_DISP etc) are generated each time a cron job runs. As we
> > > do
> > > not enable SELinux, disabling these thing use subj_type=crond_t is not
> > > a
> > > viable option.
> > > 
> > > 1. I have tried the following way to exclude using msg_type and exe
> > > together and it seems to work.
> > > 
> > > -a exclude,always -F msgtype=MAC_IPSEC_EVENT -F exe=/usr/sbin/cron
> > > -a exclude,always -F msgtype=USER_AUTH -F exe=/usr/sbin/cron
> > > -a exclude,always -F msgtype=USER_ACCT -F exe=/usr/sbin/cron
> > > -a exclude,always -F msgtype=CRED_REFR -F exe=/usr/sbin/cron
> > > -a exclude,always -F msgtype=CRED_DISP -F exe=/usr/sbin/cron
> > > -a exclude,always -F msgtype=CRED_ACQ -F exe=/usr/sbin/cron
> > > -a exclude,always -F msgtype=USER_START -F exe=/usr/sbin/cron
> > > -a exclude,always -F msgtype=USER_END -F exe=/usr/sbin/cron
> > > -a exclude,always -F msgtype=SERVICE_START -F exe=/usr/sbin/cron
> > > 
> > > Just want to make sure there is nothing I am missing here and that this
> > > only excludes the msg types for the cron executable.
> > 
> > I think so. But it's easy enough to test. Just login and see if you get
> > any
> > USER_START events from something other than cron.
> > 
> > > 2. Apart from these messages, there is a LOGIN message that gets
> > 
> > generated
> > 
> > > each time a cron runs. Eventhough, the LOGIN message in auditd does not
> > > have an exe field, the following statement surprisingly seems to be
> > > working.
> > > 
> > > -a exclude,always -F msgtype=LOGIN -F exe=/usr/sbin/cron
> > > 
> > > I can still see LOGIN messages for other users but the cron LOGIN
> > 
> > messages
> > 
> > > seem to be suppressed. Could you provide some detail as to how this is
> > > happening and is the expected result.
> > 
> > It doesn't match against the text in the event. It matches against the
> > process's attributes.
> > 
> > > 3. Is there a better way to suppress these cron messages that I am not
> > > considering apart from the SELinux option mentioned.
> > 
> > I think you found the best way for a non-selinux system. Back when it was
> > documented that it could be supressed by selinux type, audit by
> > executable
> > did not exist. But as you found, that is an effective way to get rid of
> > the
> > events.
> > 
> > I also think the cronie program might be a little more audit friendly. It
> > does not call PAM for the system crontabs run under the root user. PAM is
> > run
> > only for the local crontab (i.e. the one edited by the crontab command)
> > and
> > in case of the system crontabs only for jobs that are run under non-root
> > user.
> > 
> > -Steve




--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit