Increasing audit netlink buffer size

Increasing audit netlink buffer size

[Seyeong Kim]

Hello all

Recently I've seen some people who faced below error msg while booting
or while the machine is working.

Error receiving audit netlink packet (No buffer space available)
Error setting audit daemon pid (No buffer space available)
Unable to set audit pid, exiting

increasing q_depth=75000 and -b 8192 didn't help for them.

There is no stable reproducer but I suspect this is because the
default netlink buffer is not big enough. Below were my test steps to
see the above msg.

1. launch instance
2. enable audit with kernel parameters
3. run for i in {1..100000}; do auditctl --reset-lost; done
4. while running #3, keep restarting systemctl restart auditd

I wasn't able to let them test this test pkg but could you please give
me any advice related to this if it makes sense or not?

Thanks in advance. Regards

Index: audit-3.0.7/lib/netlink.c
===================================================================
--- audit-3.0.7.orig/lib/netlink.c
+++ audit-3.0.7/lib/netlink.c
@@ -34,6 +34,9 @@
 #ifndef NETLINK_AUDIT
 #define NETLINK_AUDIT 9
 #endif
+#ifndef SO_RCVBUFFORCE
+#define SO_RCVBUFFORCE 33
+#endif

 static int adjust_reply(struct audit_reply *rep, int len);
 static int check_ack(int fd);
@@ -47,6 +50,7 @@ static int check_ack(int fd);
 int audit_open(void)
 {
        int saved_errno;
+       int rcvbuf;
        int fd = socket(PF_NETLINK, SOCK_RAW, NETLINK_AUDIT);

        if (fd < 0) {
@@ -62,6 +66,19 @@ int audit_open(void)
                errno = saved_errno;
                return fd;
        }
+
+       rcvbuf = 10*1024*1024;  // size is temp value for now.
+       if (setsockopt(fd, SOL_SOCKET, SO_RCVBUFFORCE,
+                      &rcvbuf, sizeof(rcvbuf))) {
+               saved_errno = errno;
+               audit_msg(LOG_ERR,
+                       "Error setting netlink sock buffer size (%s)",
+                       strerror(errno));
+               close(fd);
+               errno = saved_errno;
+               return -1;
+       }
+
        if (fcntl(fd, F_SETFD, FD_CLOEXEC) == -1) {
                saved_errno = errno;
                audit_msg(LOG_ERR,

--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit

Re: Increasing audit netlink buffer size

[Steve Grubb]

Hello,

Thanks for reporting the issue.

On Friday, September 15, 2023 1:33:42 AM EDT Seyeong Kim wrote:
> Recently I've seen some people who faced below error msg while booting
> or while the machine is working.
> 
> Error receiving audit netlink packet (No buffer space available)
> Error setting audit daemon pid (No buffer space available)
> Unable to set audit pid, exiting
> 
> increasing q_depth=75000 and -b 8192 didn't help for them.
> 
> There is no stable reproducer but I suspect this is because the
> default netlink buffer is not big enough.

The default netlink buffer is set by this sysctl:

# sysctl net.core.rmem_default
net.core.rmem_default = 212992

200k should be plenty to hold a 9k netlink packet at the most.

> Below were my test steps to
> see the above msg.
> 
> 1. launch instance
> 2. enable audit with kernel parameters
> 3. run for i in {1..100000}; do auditctl --reset-lost; done
> 4. while running #3, keep restarting systemctl restart auditd

Hmm. restarting auditd via systemctl can be problematic. It has to wait for 
auditd to terminate or you can have 2 active at once. This is one of the 
reasons why we disallow the direct use of systemctl to 

> I wasn't able to let them test this test pkg but could you please give
> me any advice related to this if it makes sense or not?

This is the only report of this I've heard of. Which kernel? Has the sysctl 
been modified from the default? What are the audit parameters given at the 
boot prompt? Which version of the audit package?

I don't think the code in this area has changed for a long time. Also, 
recvfrom man page does not mention ENOBUFS. The netlink(7) man page seems to 
indicate something about acks possibly causing this. However, loading rules 
is done one at at time. I don't really understand how it gets backed up like 
this unless 2 auditd are stepping on each other somehow.

-Steve

> Thanks in advance. Regards
> 
> Index: audit-3.0.7/lib/netlink.c
> ===================================================================
> --- audit-3.0.7.orig/lib/netlink.c
> +++ audit-3.0.7/lib/netlink.c
> @@ -34,6 +34,9 @@
>  #ifndef NETLINK_AUDIT
>  #define NETLINK_AUDIT 9
>  #endif
> +#ifndef SO_RCVBUFFORCE
> +#define SO_RCVBUFFORCE 33
> +#endif
> 
>  static int adjust_reply(struct audit_reply *rep, int len);
>  static int check_ack(int fd);
> @@ -47,6 +50,7 @@ static int check_ack(int fd);
>  int audit_open(void)
>  {
>         int saved_errno;
> +       int rcvbuf;
>         int fd = socket(PF_NETLINK, SOCK_RAW, NETLINK_AUDIT);
> 
>         if (fd < 0) {
> @@ -62,6 +66,19 @@ int audit_open(void)
>                 errno = saved_errno;
>                 return fd;
>         }
> +
> +       rcvbuf = 10*1024*1024;  // size is temp value for now.
> +       if (setsockopt(fd, SOL_SOCKET, SO_RCVBUFFORCE,
> +                      &rcvbuf, sizeof(rcvbuf))) {
> +               saved_errno = errno;
> +               audit_msg(LOG_ERR,
> +                       "Error setting netlink sock buffer size (%s)",
> +                       strerror(errno));
> +               close(fd);
> +               errno = saved_errno;
> +               return -1;
> +       }
> +
>         if (fcntl(fd, F_SETFD, FD_CLOEXEC) == -1) {
>                 saved_errno = errno;
>                 audit_msg(LOG_ERR,
> 
> --
> Linux-audit mailing list
> Linux-audit@redhat.com
> https://listman.redhat.com/mailman/listinfo/linux-audit




--
Linux-audit mailing list
Linux-audit@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-audit