ausearch on the fly

ausearch on the fly

[MAUPERTUIS, PHILIPPE]

[-- Attachment #1.1: Type: text/plain, Size: 1254 bytes --]

Hi,
We are centralizing  the audit logs with rsyslog.
The SIEM behind the central log server is unable to process the raw logs.
We would like to push the ausearch  result in CSV format in real time or near real time.
Is there a way to have ausearch working from a pipe and and waiting when no logs are received

Regards
Philippe

equensWorldline is a registered trade mark and trading name owned by the Worldline Group through its holding company.
This e-mail and the documents attached are confidential and intended solely for the addressee. If you receive this e-mail in error, you are not authorized to copy, disclose, use or retain it. Please notify the sender immediately and delete this email from your systems. As emails may be intercepted, amended or lost, they are not secure. EquensWorldline and the Worldline Group therefore can accept no liability for any errors or their content. Although equensWorldline and the Worldline Group endeavours to maintain a virus-free network, we do not warrant that this transmission is virus-free and can accept no liability for any damages resulting from any virus transmitted. The risks are deemed to be accepted by everyone who communicates with equensWorldline and the Worldline Group by email

[-- Attachment #1.2: Type: text/html, Size: 6274 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: ausearch on the fly

[Steve Grubb]

On Friday, December 20, 2019 8:33:11 AM EST MAUPERTUIS, PHILIPPE wrote:
> We are centralizing  the audit logs with rsyslog.
> The SIEM behind the central log server is unable to process the raw logs.
> We would like to push the ausearch  result in CSV format in real time or
> near real time. Is there a way to have ausearch working from a pipe and
> and waiting when no logs are received

I think that I've seen others who setup a cron job and use the checkpointing
feature so that they do not miss anything. You can pipe its output into
logger. You probably also want to cut the first line which has the column
headers.

ausearch --start today --checkpoint /root/last-ausearch .chpt --format csv | tail -n +2 | logger

Also, the latest syslog plugin can now do interpretation. I think its in
alpha-9 which dates back to Nov 04, 2019. 

It really shouldn't be hard to copy and paste the code from ausearch into the
syslog plugin to log directly in that format. I wonder if anyone else would
find that useful?

-Steve

RE: ausearch on the fly

[MAUPERTUIS, PHILIPPE]

Thank you steve,
I will have a look at it.
Philippe

-----Message d'origine-----
De : Steve Grubb [mailto:sgrubb@redhat.com]
Envoyé : vendredi 20 décembre 2019 20:24
À : linux-audit@redhat.com
Cc : MAUPERTUIS, PHILIPPE
Objet : Re: ausearch on the fly

On Friday, December 20, 2019 8:33:11 AM EST MAUPERTUIS, PHILIPPE wrote:
> We are centralizing  the audit logs with rsyslog.
> The SIEM behind the central log server is unable to process the raw logs.
> We would like to push the ausearch  result in CSV format in real time or
> near real time. Is there a way to have ausearch working from a pipe and
> and waiting when no logs are received

I think that I've seen others who setup a cron job and use the checkpointing
feature so that they do not miss anything. You can pipe its output into
logger. You probably also want to cut the first line which has the column
headers.

ausearch --start today --checkpoint /root/last-ausearch .chpt --format csv | tail -n +2 | logger

Also, the latest syslog plugin can now do interpretation. I think its in
alpha-9 which dates back to Nov 04, 2019.

It really shouldn't be hard to copy and paste the code from ausearch into the
syslog plugin to log directly in that format. I wonder if anyone else would
find that useful?

-Steve


equensWorldline is a registered trade mark and trading name owned by the Worldline Group through its holding company.
This e-mail and the documents attached are confidential and intended solely for the addressee. If you receive this e-mail in error, you are not authorized to copy, disclose, use or retain it. Please notify the sender immediately and delete this email from your systems. As emails may be intercepted, amended or lost, they are not secure. EquensWorldline and the Worldline Group therefore can accept no liability for any errors or their content. Although equensWorldline and the Worldline Group endeavours to maintain a virus-free network, we do not warrant that this transmission is virus-free and can accept no liability for any damages resulting from any virus transmitted. The risks are deemed to be accepted by everyone who communicates with equensWorldline and the Worldline Group by email

Re: ausearch on the fly

[warron.french]

[-- Attachment #1.1: Type: text/plain, Size: 2691 bytes --]

I have seen it done in exactly this manner too.  Where I work we do this.

--------------------------
Warron French



On Fri, Dec 20, 2019 at 2:26 PM MAUPERTUIS, PHILIPPE <
philippe.maupertuis@equensworldline.com> wrote:

> Thank you steve,
> I will have a look at it.
> Philippe
>
> -----Message d'origine-----
> De : Steve Grubb [mailto:sgrubb@redhat.com]
> Envoyé : vendredi 20 décembre 2019 20:24
> À : linux-audit@redhat.com
> Cc : MAUPERTUIS, PHILIPPE
> Objet : Re: ausearch on the fly
>
> On Friday, December 20, 2019 8:33:11 AM EST MAUPERTUIS, PHILIPPE wrote:
> > We are centralizing  the audit logs with rsyslog.
> > The SIEM behind the central log server is unable to process the raw logs.
> > We would like to push the ausearch  result in CSV format in real time or
> > near real time. Is there a way to have ausearch working from a pipe and
> > and waiting when no logs are received
>
> I think that I've seen others who setup a cron job and use the
> checkpointing
> feature so that they do not miss anything. You can pipe its output into
> logger. You probably also want to cut the first line which has the column
> headers.
>
> ausearch --start today --checkpoint /root/last-ausearch .chpt --format csv
> | tail -n +2 | logger
>
> Also, the latest syslog plugin can now do interpretation. I think its in
> alpha-9 which dates back to Nov 04, 2019.
>
> It really shouldn't be hard to copy and paste the code from ausearch into
> the
> syslog plugin to log directly in that format. I wonder if anyone else would
> find that useful?
>
> -Steve
>
>
> equensWorldline is a registered trade mark and trading name owned by the
> Worldline Group through its holding company.
> This e-mail and the documents attached are confidential and intended
> solely for the addressee. If you receive this e-mail in error, you are not
> authorized to copy, disclose, use or retain it. Please notify the sender
> immediately and delete this email from your systems. As emails may be
> intercepted, amended or lost, they are not secure. EquensWorldline and the
> Worldline Group therefore can accept no liability for any errors or their
> content. Although equensWorldline and the Worldline Group endeavours to
> maintain a virus-free network, we do not warrant that this transmission is
> virus-free and can accept no liability for any damages resulting from any
> virus transmitted. The risks are deemed to be accepted by everyone who
> communicates with equensWorldline and the Worldline Group by email
>
>
> --
> Linux-audit mailing list
> Linux-audit@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit
>
>

[-- Attachment #1.2: Type: text/html, Size: 3590 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

RE: ausearch on the fly

[MAUPERTUIS, PHILIPPE]

> On Friday, December 20, 2019 8:33:11 AM EST MAUPERTUIS, PHILIPPE wrote:
> > We are centralizing  the audit logs with rsyslog.
> > The SIEM behind the central log server is unable to process the raw logs.
> > We would like to push the ausearch  result in CSV format in real time or
> > near real time. Is there a way to have ausearch working from a pipe and
> > and waiting when no logs are received
>
> I think that I've seen others who setup a cron job and use the checkpointing
> feature so that they do not miss anything. You can pipe its output into
> logger. You probably also want to cut the first line which has the column
> headers.
>
> ausearch --start today --checkpoint /root/last-ausearch .chpt --format csv |
> tail -n +2 | logger
On a central log server the input file can grow very big and very fast.
Probably logrotate is needed to keep it in check.
What happen to the checkpointing feature when the file is rotated ?
How not to miss the last events from the old file and get the new events from the new file ?

>
> Also, the latest syslog plugin can now do interpretation. I think its in
> alpha-9 which dates back to Nov 04, 2019.
>
> It really shouldn't be hard to copy and paste the code from ausearch into the
> syslog plugin to log directly in that format. I wonder if anyone else would
> find that useful?
>
It would be very useful indeed

Philippe

equensWorldline is a registered trade mark and trading name owned by the Worldline Group through its holding company.
This e-mail and the documents attached are confidential and intended solely for the addressee. If you receive this e-mail in error, you are not authorized to copy, disclose, use or retain it. Please notify the sender immediately and delete this email from your systems. As emails may be intercepted, amended or lost, they are not secure. EquensWorldline and the Worldline Group therefore can accept no liability for any errors or their content. Although equensWorldline and the Worldline Group endeavours to maintain a virus-free network, we do not warrant that this transmission is virus-free and can accept no liability for any damages resulting from any virus transmitted. The risks are deemed to be accepted by everyone who communicates with equensWorldline and the Worldline Group by email

Re: ausearch on the fly

[Burn Alting]

[-- Attachment #1.1: Type: text/plain, Size: 2363 bytes --]

Philippe,
On Fri, 2020-02-07 at 08:13 +0000, MAUPERTUIS, PHILIPPE wrote:
> > On Friday, December 20, 2019 8:33:11 AM EST MAUPERTUIS, PHILIPPE wrote:
> > > We are centralizing  the audit logs with rsyslog.The SIEM behind the central
> > > log server is unable to process the raw logs.We would like to push the
> > > ausearch  result in CSV format in real time ornear real time. Is there a way
> > > to have ausearch working from a pipe andand waiting when no logs are received
> > 
> > I think that I've seen others who setup a cron job and use the
> > checkpointingfeature so that they do not miss anything. You can pipe its output
> > intologger. You probably also want to cut the first line which has the
> > columnheaders.
> > ausearch --start today --checkpoint /root/last-ausearch .chpt --format csv |tail
> > -n +2 | logger
> 
> On a central log server the input file can grow very big and very fast.Probably
> logrotate is needed to keep it in check.What happen to the checkpointing feature
> when the file is rotated ?How not to miss the last events from the old file and
> get the new events from the new file ?

The above performs a checkpoint on the local machine and then sends it's output to
the local syslog service via the logger program. Ausearchis independent of the
syslog service. The checkpoint function of ausearch takes into account the audit.log
log file roll-over feature built into auditd so,providing your auditd log file
rotation is set appropriately, checkpoint works no matter how many audit.log files
are in the audit log directory.For information, a 9 file 32MB per log file
configuration works well for a very heavy processing host where exec's are logged.
Further, if the generation of logsis such that the checkpoint does miss logs, then
the checkpoint documentation shows one how to address this. If this is noted, then
include the size of or numberof local log files. 
> > Also, the latest syslog plugin can now do interpretation. I think its inalpha-9
> > which dates back to Nov 04, 2019.
> > It really shouldn't be hard to copy and paste the code from ausearch into
> > thesyslog plugin to log directly in that format. I wonder if anyone else
> > wouldfind that useful?
> 
> 
> --Linux-audit mailing listLinux-audit@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit
> 

[-- Attachment #1.2: Type: text/html, Size: 3598 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: ausearch on the fly

[MAUPERTUIS, PHILIPPE]

[-- Attachment #1.1: Type: text/plain, Size: 3850 bytes --]

Hi,
I should have been more precise in my question.
I want to run this ausearch on the central log server, not on the original server.
Very likely i need to specify the input file in that.
And this input file would need to be rotated outside auditd
Philippe

Envoyé depuis mon mobile Orange

------ Message d'origine------
De: Burn Alting
Date: ven. 7 févr. 2020 12:49
À: MAUPERTUIS, PHILIPPE;Steve Grubb;linux-audit@redhat.com;
Cc:
Objet :Re: ausearch on the fly

Philippe,

On Fri, 2020-02-07 at 08:13 +0000, MAUPERTUIS, PHILIPPE wrote:


On Friday, December 20, 2019 8:33:11 AM EST MAUPERTUIS, PHILIPPE wrote:


We are centralizing  the audit logs with rsyslog.

The SIEM behind the central log server is unable to process the raw logs.

We would like to push the ausearch  result in CSV format in real time or

near real time. Is there a way to have ausearch working from a pipe and

and waiting when no logs are received



I think that I've seen others who setup a cron job and use the checkpointing

feature so that they do not miss anything. You can pipe its output into

logger. You probably also want to cut the first line which has the column

headers.


ausearch --start today --checkpoint /root/last-ausearch .chpt --format csv |

tail -n +2 | logger


On a central log server the input file can grow very big and very fast.

Probably logrotate is needed to keep it in check.

What happen to the checkpointing feature when the file is rotated ?

How not to miss the last events from the old file and get the new events from the new file ?

The above performs a checkpoint on the local machine and then sends it's output to the local syslog service via the logger program. Ausearch
is independent of the syslog service. The checkpoint function of ausearch takes into account the audit.log log file roll-over feature built into auditd so,
providing your auditd log file rotation is set appropriately, checkpoint works no matter how many audit.log files are in the audit log directory.
For information, a 9 file 32MB per log file configuration works well for a very heavy processing host where exec's are logged. Further, if the generation of logs
is such that the checkpoint does miss logs, then the checkpoint documentation shows one how to address this. If this is noted, then include the size of or number
of local log files.



Also, the latest syslog plugin can now do interpretation. I think its in

alpha-9 which dates back to Nov 04, 2019.


It really shouldn't be hard to copy and paste the code from ausearch into the

syslog plugin to log directly in that format. I wonder if anyone else would

find that useful?





--

Linux-audit mailing list

<mailto:Linux-audit@redhat.com>

Linux-audit@redhat.com


<https://www.redhat.com/mailman/listinfo/linux-audit>

https://www.redhat.com/mailman/listinfo/linux-audit



equensWorldline is a registered trade mark and trading name owned by the Worldline Group through its holding company.
This e-mail and the documents attached are confidential and intended solely for the addressee. If you receive this e-mail in error, you are not authorized to copy, disclose, use or retain it. Please notify the sender immediately and delete this email from your systems. As emails may be intercepted, amended or lost, they are not secure. EquensWorldline and the Worldline Group therefore can accept no liability for any errors or their content. Although equensWorldline and the Worldline Group endeavours to maintain a virus-free network, we do not warrant that this transmission is virus-free and can accept no liability for any damages resulting from any virus transmitted. The risks are deemed to be accepted by everyone who communicates with equensWorldline and the Worldline Group by email

[-- Attachment #1.2: Type: text/html, Size: 5772 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: ausearch on the fly

[Burn Alting]

[-- Attachment #1.1: Type: text/plain, Size: 1051 bytes --]

Phillipe,

To close this off, yes, you will need to investigate making use of the --input
option.  You
may also want to consider having a central syslog directory structure that allows
you to simulate the local directory structure (/var/log/audit)
by having a revolving set of audit.log, audit.log.1, etc files and age them off to
your raw archive once you have processed them.

My preference is to enrich the logs via ausearch on the local host as this mitigates
a number of risks (lost logs as per this discussion, local uids)
and then send the results to a central log management capability. 

Regards
Burn

On Fri, 2020-02-07 at 12:44 +0000, MAUPERTUIS, PHILIPPE wrote:
> Hi,
> I should have been more precise in my question.
> I want to run this ausearch on the central log server, not on the original
> server. 
> Very likely i need to specify the input file in that.
> And this input file would need to be rotated outside auditd
> Philippe
> 
> 
> 
> 
> Envoyé depuis mon mobile Orange
> 
> 
> 
> 
> 

Snip

[-- Attachment #1.2: Type: text/html, Size: 1747 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

RE: ausearch on the fly

[MAUPERTUIS, PHILIPPE]

[-- Attachment #1.1: Type: text/plain, Size: 2301 bytes --]

Hi,
I should have read the man more carefully.
I missed the point that --input can be a directory and not only a single file as I wrongly assumed.
That solves any problems I had.

Philippe

De : Burn Alting [mailto:burn.alting@iinet.net.au]
Envoyé : samedi 8 février 2020 01:39
À : MAUPERTUIS, PHILIPPE; Steve Grubb; linux-audit@redhat.com
Objet : Re: ausearch on the fly

Phillipe,

To close this off, yes, you will need to investigate making use of the --input option.  You
may also want to consider having a central syslog directory structure that allows you to simulate the local directory structure (/var/log/audit)
by having a revolving set of audit.log, audit.log.1, etc files and age them off to your raw archive once you have processed them.

My preference is to enrich the logs via ausearch on the local host as this mitigates a number of risks (lost logs as per this discussion, local uids)
and then send the results to a central log management capability.

Regards
Burn

On Fri, 2020-02-07 at 12:44 +0000, MAUPERTUIS, PHILIPPE wrote:
Hi,
I should have been more precise in my question.
I want to run this ausearch on the central log server, not on the original server.
Very likely i need to specify the input file in that.
And this input file would need to be rotated outside auditd
Philippe

Envoyé depuis mon mobile Orange


Snip

equensWorldline is a registered trade mark and trading name owned by the Worldline Group through its holding company.
This e-mail and the documents attached are confidential and intended solely for the addressee. If you receive this e-mail in error, you are not authorized to copy, disclose, use or retain it. Please notify the sender immediately and delete this email from your systems. As emails may be intercepted, amended or lost, they are not secure. EquensWorldline and the Worldline Group therefore can accept no liability for any errors or their content. Although equensWorldline and the Worldline Group endeavours to maintain a virus-free network, we do not warrant that this transmission is virus-free and can accept no liability for any damages resulting from any virus transmitted. The risks are deemed to be accepted by everyone who communicates with equensWorldline and the Worldline Group by email

[-- Attachment #1.2: Type: text/html, Size: 7304 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]