NFQUEUE/iptables and kernel warning messages for net/ipv4/tcp_output.c

NFQUEUE/iptables and kernel warning messages for net/ipv4/tcp_output.c

[Vieri Di Paola]

Hi,

Whenever I use NFQUEUE/iptables to send traffic to an IDS/IPS (eg.
Suricata), I get an ugly kernel warning which can sometimes and on the
long run turn into a system freeze.

I'm using NFQUEUE 0:5, and I'm running Suricata with -q 0 -q 1 -q 2 -q
3 -q 4 -q 5 as arguments.

I've already reported the issue on the LKML here:

https://lkml.org/lkml/2020/2/13/1255

However, I've been told by the Suricata ML to try and post here too.

The message "WARNING: CPU: * at net/ipv4/tcp_output.c:915" does not
appear when I stop using Suricata with NFQUEUE.

Regards,

Vieri

Re: NFQUEUE/iptables and kernel warning messages for net/ipv4/tcp_output.c

[Florian Westphal]

Vieri Di Paola <vieridipaola@gmail.com> wrote:
> Hi,
> 
> Whenever I use NFQUEUE/iptables to send traffic to an IDS/IPS (eg.
> Suricata), I get an ugly kernel warning which can sometimes and on the
> long run turn into a system freeze.
> 
> I'm using NFQUEUE 0:5, and I'm running Suricata with -q 0 -q 1 -q 2 -q
> 3 -q 4 -q 5 as arguments.
> 
> I've already reported the issue on the LKML here:
> 
> https://lkml.org/lkml/2020/2/13/1255

No idea.  Suricata forces software-side segmentation for each packet,
could be related.

Can you post to suricata ML and get this patch working (untested):
If the problem doesn't occur with segmentation off we've at least
narrowed it down:

diff --git a/src/source-nfq.c b/src/source-nfq.c
--- a/src/source-nfq.c
+++ b/src/source-nfq.c
@@ -154,6 +154,7 @@ typedef enum NFQMode_ {
 } NFQMode;
 
 #define NFQ_FLAG_FAIL_OPEN  (1 << 0)
+#define NFQ_FLAG_GSO        (1 << 2)
 
 typedef struct NFQCnf_ {
     NFQMode mode;
@@ -242,6 +243,10 @@ void NFQInitConfig(char quiet)
 #endif
     }
 
+#ifdef HAVE_NFQ_SET_QUEUE_FLAGS
+        nfq_config.flags |= NFQ_FLAG_GSO;
+#endif
+
     if ((ConfGetInt("nfq.repeat-mark", &value)) == 1) {
         nfq_config.mark = (uint32_t)value;
     }
@@ -389,6 +394,16 @@ static inline void NFQMutexInit(NFQQueueVars *nq)
     }
 }
 
+/* Ugly Hack */
+struct nfq_data {
+	void **data;
+};
+
+static uint32_t nfq_get_pktinfo(struct nfq_data *nfad)
+{
+        return ntohl(nfnl_get_data(nfad->data, NFQA_SKB_INFO, uint32_t));
+}
+
 #define NFQMutexLock(nq) do {           \
     if ((nq)->use_mutex)                \
         SCMutexLock(&(nq)->mutex_qh);   \
@@ -412,6 +427,7 @@ static int NFQSetupPkt (Packet *p, struct nfq_q_handle *qh, void *data)
     int ret;
     char *pktdata;
     struct nfqnl_msg_packet_hdr *ph;
+    uint32_t pktinfo;
 
     ph = nfq_get_msg_packet_hdr(tb);
     if (ph != NULL) {
@@ -474,6 +490,11 @@ static int NFQSetupPkt (Packet *p, struct nfq_q_handle *qh, void *data)
         gettimeofday(&p->ts, NULL);
     }
 
+    pktinfo = nfq_get_pktinfo(tb);
+    /* kernel/nic will compute checksum on output */
+    if (pktinfo & NFQA_SKB_CSUMNOTREADY)
+       p->flags |= PKT_IGNORE_CHECKSUM;
+
     p->datalink = DLT_RAW;
     return 0;
 }
@@ -674,16 +695,14 @@ static TmEcode NFQInitThread(NFQThreadVars *t, uint32_t queue_maxlen)
 #endif
 
 #ifdef HAVE_NFQ_SET_QUEUE_FLAGS
-    if (nfq_config.flags & NFQ_FLAG_FAIL_OPEN) {
-        uint32_t flags = NFQA_CFG_F_FAIL_OPEN;
-        uint32_t mask = NFQA_CFG_F_FAIL_OPEN;
-        int r = nfq_set_queue_flags(q->qh, mask, flags);
+    if (nfq_config.flags) {
+        int r = nfq_set_queue_flags(q->qh, nfq_config.flags, nfq_config.flags);
 
         if (r == -1) {
-            SCLogWarning(SC_ERR_NFQ_SET_MODE, "can't set fail-open mode: %s",
-                         strerror(errno));
+            SCLogWarning(SC_ERR_NFQ_SET_MODE, "can't set nfq flags 0x%x: %s",
+                         nfq_config.flags, strerror(errno));
         } else {
-            SCLogInfo("fail-open mode should be set on queue");
+            SCLogInfo("Set flag modes 0x%x on queue", nfq_config.flags);
         }
     }
 #endif

Re: NFQUEUE/iptables and kernel warning messages for net/ipv4/tcp_output.c

[Vieri Di Paola]

On Tue, Feb 18, 2020 at 1:39 PM Florian Westphal <fw@strlen.de> wrote:
>
> get this patch working (untested)

> +static uint32_t nfq_get_pktinfo(struct nfq_data *nfad)
> +{
> +        return ntohl(nfnl_get_data(nfad->data, NFQA_SKB_INFO, uint32_t));

I applied the patch, but I get this compilation error:

In file included from suricata-common.h:180,
                 from source-nfq.c:28:
source-nfq.c: In function 'nfq_get_pktinfo':
source-nfq.c:404:48: error: 'NFQA_SKB_INFO' undeclared (first use in
this function)
  404 |         return ntohl(nfnl_get_data(nfad->data, NFQA_SKB_INFO,
uint32_t));
      |                                                ^~~~~~~~~~~~~

I'll post to the suricata ML asap.

Thanks,

Vieri

Re: NFQUEUE/iptables and kernel warning messages for net/ipv4/tcp_output.c

[Florian Westphal]

Vieri Di Paola <vieridipaola@gmail.com> wrote:
> On Tue, Feb 18, 2020 at 1:39 PM Florian Westphal <fw@strlen.de> wrote:
> >
> > get this patch working (untested)
> 
> > +static uint32_t nfq_get_pktinfo(struct nfq_data *nfad)
> > +{
> > +        return ntohl(nfnl_get_data(nfad->data, NFQA_SKB_INFO, uint32_t));
> 
> I applied the patch, but I get this compilation error:
> 
> In file included from suricata-common.h:180,
>                  from source-nfq.c:28:
> source-nfq.c: In function 'nfq_get_pktinfo':
> source-nfq.c:404:48: error: 'NFQA_SKB_INFO' undeclared (first use in
> this function)
>   404 |         return ntohl(nfnl_get_data(nfad->data, NFQA_SKB_INFO,
> uint32_t));
>       |                                                ^~~~~~~~~~~~~

This means your kernel headers are older than 3.10.
It should be part of /usr/include/linux/netfilter/nfnetlink_queue.h .


You can substitute 14 instead, or add

#define NFQA_SKB_INFO 14

> I'll post to the suricata ML asap.

Thanks.

Re: NFQUEUE/iptables and kernel warning messages for net/ipv4/tcp_output.c

[Vieri Di Paola]

On Tue, Feb 18, 2020 at 2:21 PM Florian Westphal <fw@strlen.de> wrote:
>
>
> This means your kernel headers are older than 3.10.
> It should be part of /usr/include/linux/netfilter/nfnetlink_queue.h .

Actually I have 4.19. I also had to define another constant.
In any case, it compiled OK, and the problem was not seen again for a
test period of at least 15 hours (I usually had several of these
kernel warnings almost each hour, so 15 hours in a weekday is
significant).
I then removed your patch, recompiled and ran suricata in nfq "accept"
mode (default). It was set as "repeat mode" before. It hasn't been
running long enough yet (7 hours), but for now I haven't seen any
kernel warnings in "accept mode".

So all I can say for now is that it seems I'm getting these kernel
warnings when using Suricata in nfq repeat mode.

I'll ask the Suricata ML what they think about that.

Thanks,

Vieri