Key based rate limiter (audit_set_rate_limit)

Key based rate limiter (audit_set_rate_limit)

[Anurag Aggarwal]

[-- Attachment #1.1: Type: text/plain, Size: 322 bytes --]

Hello All,

The current rate limiter, audit_set_rate_limit limits all types of events.
In our case, we want to limit auditd events with a specific key, as they
are very noisy and consume very high CPU.

From my understanding, this support is currently missing in AuditD.

Is my understanding correct?

-- 
Anurag Aggarwal

[-- Attachment #1.2: Type: text/html, Size: 686 bytes --]

[-- Attachment #2: Type: text/plain, Size: 107 bytes --]

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Key based rate limiter (audit_set_rate_limit)

[Paul Moore]

On Tue, Feb 28, 2023 at 5:53 AM Anurag Aggarwal
<anurag19aggarwal@gmail.com> wrote:
> Hello All,
>
> The current rate limiter, audit_set_rate_limit limits all types of events. In our case, we want to limit auditd events with a specific key, as they are very noisy and consume very high CPU.
>
> From my understanding, this support is currently missing in AuditD.
>
> Is my understanding correct?

Hello.

Limiting of audit records is actually done in the kernel, and
currently the rate limit applies equally[1] to all records, there is
no ability to enforce limits per-key.  If you have a particular audit
rule which is too verbose *and* you are willing to lose audit records
from that filter rule (which is what would happen if they were rate
limited), you might want to consider making that audit filter rule
more targeted to the event you are interested in logging.  Generating
more audit records than you want to see can be a sign of an overly
general audit rule.

Good luck!

[1] Audit records generated by auditd/auditctl are exempt from rate
limiting to help prevent lockups/contention.

-- 
paul-moore.com

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Key based rate limiter (audit_set_rate_limit)

[Anurag Aggarwal]

[-- Attachment #1.1: Type: text/plain, Size: 636 bytes --]

Hello Paul,

Thank you for your information.


> If you have a particular audit
> rule which is too verbose *and* you are willing to lose audit records
> from that filter rule (which is what would happen if they were rate
> limited), you might want to consider making that audit filter rule
> more targeted to the event you are interested in logging.  Generating
> more audit records than you want to see can be a sign of an overly
> general audit rule.
>

I agree that having rules which are too verbose is not a very good idea.

Beside this, is there any other mechanism which we can use to get a similar
effect?

-- 
Anurag Aggarwal

[-- Attachment #1.2: Type: text/html, Size: 1024 bytes --]

[-- Attachment #2: Type: text/plain, Size: 107 bytes --]

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Key based rate limiter (audit_set_rate_limit)

[Paul Moore]

On Tue, Feb 28, 2023 at 10:35 AM Anurag Aggarwal
<anurag19aggarwal@gmail.com> wrote:
>
> Hello Paul,
>
> Thank you for your information.
>
>> If you have a particular audit
>> rule which is too verbose *and* you are willing to lose audit records
>> from that filter rule (which is what would happen if they were rate
>> limited), you might want to consider making that audit filter rule
>> more targeted to the event you are interested in logging.  Generating
>> more audit records than you want to see can be a sign of an overly
>> general audit rule.
>
> I agree that having rules which are too verbose is not a very good idea.
>
> Beside this, is there any other mechanism which we can use to get a similar effect?

Nothing comes quickly to mind, perhaps others on the mailing list
might have some ideas ... ?

-- 
paul-moore.com

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Key based rate limiter (audit_set_rate_limit)

[Lenny Bruzenak]

On 2/28/23 09:31, Paul Moore wrote:

> On Tue, Feb 28, 2023 at 10:35 AM Anurag Aggarwal
> <anurag19aggarwal@gmail.com>  wrote:
>> Hello Paul,
>>
>> Thank you for your information.
>>
>>> If you have a particular audit
>>> rule which is too verbose *and* you are willing to lose audit records
>>> from that filter rule (which is what would happen if they were rate
>>> limited), you might want to consider making that audit filter rule
>>> more targeted to the event you are interested in logging.  Generating
>>> more audit records than you want to see can be a sign of an overly
>>> general audit rule.
>> I agree that having rules which are too verbose is not a very good idea.
>>
>> Beside this, is there any other mechanism which we can use to get a similar effect?
> Nothing comes quickly to mind, perhaps others on the mailing list
> might have some ideas ... ?

Not much else to offer above what Paul already replied. Maybe if we saw 
your rule we could offer more.

What we do not know is - do you have any filtering criteria in mind not 
covered by the available auditctl exclusions or do you just want to 
"sample" randomly?

If the latter, why bother auditing this with a rule at all? You might be 
able to remove the rule causing the events and do something in userspace 
to audit only what you really want.

Without a bit more context on the events, rule and intent it is hard to 
suggest alternatives. But in general, it is preferable to exclude as 
much noise as possible in your collection to ensure you get only what is 
required/desired in your audit logs.

LCB

-- 
Lenny Bruzenak
MagitekLTD

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Key based rate limiter (audit_set_rate_limit)

[Anurag Aggarwal]

[-- Attachment #1.1: Type: text/plain, Size: 721 bytes --]

>
>
> What we do not know is - do you have any filtering criteria in mind not
> covered by the available auditctl exclusions or do you just want to
> "sample" randomly?
>
> If the latter, why bother auditing this with a rule at all? You might be
> able to remove the rule causing the events and do something in userspace
> to audit only what you really want.
>
>
We want to sample system calls like rename.
In many cases, we have seen this overburden and increase auditd cpu
consumption.
In such cases, we want to drop some events randomly, so as to keep cpu
consumption under control.

There are other rules also, for example monitoring login/logout.
For such rules we do not want to drop any event.

--
Anurag Aggarwal

[-- Attachment #1.2: Type: text/html, Size: 1106 bytes --]

[-- Attachment #2: Type: text/plain, Size: 107 bytes --]

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Key based rate limiter (audit_set_rate_limit)

[Lenny Bruzenak]

[-- Attachment #1.1: Type: text/plain, Size: 2074 bytes --]

On 3/1/23 08:31, Anurag Aggarwal wrote:

>
>     What we do not know is - do you have any filtering criteria in
>     mind not
>     covered by the available auditctl exclusions or do you just want to
>     "sample" randomly?
>
>     If the latter, why bother auditing this with a rule at all? You
>     might be
>     able to remove the rule causing the events and do something in
>     userspace
>     to audit only what you really want.
>
>
> We want to sample system calls like rename.
> In many cases, we have seen this overburden and increase auditd cpu 
> consumption.
> In such cases, we want to drop some events randomly, so as to keep cpu 
> consumption under control.
>
> There are other rules also, for example monitoring login/logout.
> For such rules we do not want to drop any event.

OK; that makes sense.The concern is that when some of the events that 
can overflow (rename syscalls in this case) are flooding your auditd, 
the ones you care about (login/logout) could be dropped.

But _some_ renames are ok. Maybe you can decide which directories are OK 
to have renames under them and exclude those in your key rule? Or only 
watch for renames inside certain dirs?


Or if selinux is in force, create policy for the events you definitely 
want, then look for those types (either subject or object) in your rule. 
This is something I've seen before, where renames that are desired to be 
audited use the provided system tools, but for locally developed 
application code, they are made to run inside a certain type of a custom 
executable and then that type is excluded from the rename syscall rule. 
Ideally, the code which is written would self-audit a 1-liner like "I am 
going to rename every file under dir /opt/special/stuff/" using 
audit_log_user_message so you still have some idea what is happening (if 
you care).

Then your "my-rename" program subject type of my_rename_t can be used as 
an exclude on the rule. Of course, the caller must then know to use this 
rather than the standard utilities.


HTH,

LCB

-- 
Lenny Bruzenak
MagitekLTD

[-- Attachment #1.2: Type: text/html, Size: 3249 bytes --]

[-- Attachment #2: Type: text/plain, Size: 107 bytes --]

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Key based rate limiter (audit_set_rate_limit)

[Anurag Aggarwal]

[-- Attachment #1.1: Type: text/plain, Size: 1004 bytes --]

>
> Or if selinux is in force, create policy for the events you definitely
> want, then look for those types (either subject or object) in your rule.
> This is something I've seen before, where renames that are desired to be
> audited use the provided system tools, but for locally developed
> application code, they are made to run inside a certain type of a custom
> executable and then that type is excluded from the rename syscall rule.
> Ideally, the code which is written would self-audit a 1-liner like "I am
> going to rename every file under dir /opt/special/stuff/" using
> audit_log_user_message so you still have some idea what is happening (if
> you care).
>
> Then your "my-rename" program subject type of my_rename_t can be used as
> an exclude on the rule. Of course, the caller must then know to use this
> rather than the standard utilities.
>

This sounds useful and might solve our problem, will it be possible to
share some examples on how this can be achieved?

-- 
Anurag Aggarwal

[-- Attachment #1.2: Type: text/html, Size: 1394 bytes --]

[-- Attachment #2: Type: text/plain, Size: 107 bytes --]

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Key based rate limiter (audit_set_rate_limit)

[Lenny Bruzenak]

[-- Attachment #1.1: Type: text/plain, Size: 1212 bytes --]

On 3/1/23 22:13, Anurag Aggarwal wrote:

>     Or if selinux is in force, create policy for the events you
>     definitely want, then look for those types (either subject or
>     object) in your rule. This is something I've seen before, where
>     renames that are desired to be audited use the provided system
>     tools, but for locally developed application code, they are made
>     to run inside a certain type of a custom executable and then that
>     type is excluded from the rename syscall rule. Ideally, the code
>     which is written would self-audit a 1-liner like "I am going to
>     rename every file under dir /opt/special/stuff/" using
>     audit_log_user_message so you still have some idea what is
>     happening (if you care).
>
>     Then your "my-rename" program subject type of my_rename_t can be
>     used as an exclude on the rule. Of course, the caller must then
>     know to use this rather than the standard utilities.
>
>
> This sounds useful and might solve our problem, will it be possible to 
> share some examples on how this can be achieved?


Replying off-list as it is not specifically audit-focused. See Paul, I 
CAN learn. 😁

LCB

-- 
Lenny Bruzenak
MagitekLTD

[-- Attachment #1.2: Type: text/html, Size: 2232 bytes --]

[-- Attachment #2: Type: text/plain, Size: 107 bytes --]

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Key based rate limiter (audit_set_rate_limit)

[Paul Moore]

On Thu, Mar 2, 2023 at 12:24 PM Lenny Bruzenak <lenny@magitekltd.com> wrote:
> On 3/1/23 22:13, Anurag Aggarwal wrote:
>>
>> Or if selinux is in force, create policy for the events you definitely want, then look for those types (either subject or object) in your rule. This is something I've seen before, where renames that are desired to be audited use the provided system tools, but for locally developed application code, they are made to run inside a certain type of a custom executable and then that type is excluded from the rename syscall rule. Ideally, the code which is written would self-audit a 1-liner like "I am going to rename every file under dir /opt/special/stuff/" using audit_log_user_message so you still have some idea what is happening (if you care).
>>
>> Then your "my-rename" program subject type of my_rename_t can be used as an exclude on the rule. Of course, the caller must then know to use this rather than the standard utilities.
>
>
> This sounds useful and might solve our problem, will it be possible to share some examples on how this can be achieved?
>
> Replying off-list as it is not specifically audit-focused. See Paul, I CAN learn. 😁

 ;)

-- 
paul-moore.com

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Key based rate limiter (audit_set_rate_limit)

[Anurag Aggarwal]

[-- Attachment #1.1: Type: text/plain, Size: 61 bytes --]

Thank you, Lenny & Paul

Thank you & Regards
Anurag Aggarwal

[-- Attachment #1.2: Type: text/html, Size: 203 bytes --]

[-- Attachment #2: Type: text/plain, Size: 107 bytes --]

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Key based rate limiter (audit_set_rate_limit)

[Anurag Aggarwal]

[-- Attachment #1.1: Type: text/plain, Size: 305 bytes --]

>
>
> Limiting of audit records is actually done in the kernel, and
> currently the rate limit applies equally[1] to all records, there is
> no ability to enforce limits per-key.


One question Paul, will it be ok, if we contribute something similar to the
Auditd Kernel repository?


-- 
Anurag Aggarwal

[-- Attachment #1.2: Type: text/html, Size: 649 bytes --]

[-- Attachment #2: Type: text/plain, Size: 107 bytes --]

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Key based rate limiter (audit_set_rate_limit)

[Steve Grubb]

On Wednesday, March 8, 2023 6:53:39 AM EST Anurag Aggarwal wrote:
> > Limiting of audit records is actually done in the kernel, and
> > currently the rate limit applies equally[1] to all records, there is
> > no ability to enforce limits per-key.
> 
> One question Paul, will it be ok, if we contribute something similar to the
> Auditd Kernel repository?

I'm not Paul...but I think what you are proposing is a per rule service
class. Always and best effort where best effort gets discarded when the
backlog is above some heuristic. And rules not saying anything are assumed 
always for backwards compatibility. The main issue is that rules are defined
here:

https://github.com/linux-audit/audit-kernel/blob/main/include/uapi/linux/
audit.h#L510

There just really isn't room to add more thinkgs without some userspace API 
problem. (This would definitely need a feaure bitmap so user space can make 
sense of it.)

I suppose we could declare some bits in flags to carry this meaning? Anyways, 
maybe others might chime in to say if they want/need such a feature.

-Steve


--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Key based rate limiter (audit_set_rate_limit)

[Paul Moore]

On Wed, Mar 8, 2023 at 6:53 AM Anurag Aggarwal
<anurag19aggarwal@gmail.com> wrote:
>> Limiting of audit records is actually done in the kernel, and
>> currently the rate limit applies equally[1] to all records, there is
>> no ability to enforce limits per-key.
>
> One question Paul, will it be ok, if we contribute something similar to the Auditd Kernel repository?

I don't like telling people *not* to work on improvements to the
kernel, I'm happy to see more contributors, especially in the audit
space :)

However, I am fairly skeptical that we could add per-key rate limiting
without introducing a non-trivial amount of overhead to record
generation, which would be a show stopper for this feature given its
expected limited appeal.

-- 
paul-moore.com

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Key based rate limiter (audit_set_rate_limit)

[Anurag Aggarwal]

>
> However, I am fairly skeptical that we could add per-key rate limiting
> without introducing a non-trivial amount of overhead to record
> generation, which would be a show stopper for this feature given its
> expected limited appeal.
>

I understand the reservation. I will spend some time to analyze it impact

Steve, about your comment:

> There just really isn't room to add more thinkgs without some userspace API
> problem. (This would definitely need a feaure bitmap so user space can make
> sense of it.)

I was not aware that this could cause problems in the userspace API.

Key based filtering for rate limiting could be a useful feature. It is
something that would
help us a lot.

-- 
Anurag Aggarwal

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit