nftables rules not matching after upgrading from 0.7 to 0.8

nftables rules not matching after upgrading from 0.7 to 0.8

[Anders K. Pedersen | Cohaesio]

Hello,

After upgrading to nftables 0.8 (from 0.7) on one of my systems, I've
experiences several cases where rules that used to work fine with 0.7
sometimes doesn't match anymore with 0.8 (it's not consistent -
sometimes the rules do match with 0.8).

The rule chains end with a log statement before rejecting or dropping
the packets, and I can see in the log that everything is as expected
and the rules should match. After downgrading to nftables 0.7
everything works again.

I tried loading my rule set with "--debug=netlink" and diff the output
from 0.7 vs. 0.8. Apart from syntax differences the only thing that
changed is

 ip filter input 
-  [ payload load 1b @ network header + 9 => reg 1 ]
+  [ meta load l4proto => reg 1 ]

and

 ip6 filter input 
-  [ payload load 1b @ network header + 6 => reg 1 ]
+  [ meta load l4proto => reg 1 ]

for many of the rules. I believe this is intentional and correct by
itself, so I guess the problem is in the kernel (the system is
currently running 4.13.8), where "meta l4proto" sometimes isn't what it
should be.

Does this sound right and does it ring any bells?

Regards,
Anders K. Pedersen

Re: nftables rules not matching after upgrading from 0.7 to 0.8

[Florian Westphal]

Anders K. Pedersen | Cohaesio <akp@cohaesio.com> wrote:
> After upgrading to nftables 0.8 (from 0.7) on one of my systems, I've
> experiences several cases where rules that used to work fine with 0.7
> sometimes doesn't match anymore with 0.8 (it's not consistent -
> sometimes the rules do match with 0.8).
> 
> The rule chains end with a log statement before rejecting or dropping
> the packets, and I can see in the log that everything is as expected
> and the rules should match. After downgrading to nftables 0.7
> everything works again.

Are those errors restricted to a particular table family, chain or protocol?

At least for ip family all packets will have pkt->tprot_set true,
so I don't see how meta would break there.

>  ip filter input 
> -  [ payload load 1b @ network header + 9 => reg 1 ]
> +  [ meta load l4proto => reg 1 ]
> 
> and
> 
>  ip6 filter input 
> -  [ payload load 1b @ network header + 6 => reg 1 ]
> +  [ meta load l4proto => reg 1 ]
> 
> for many of the rules. I believe this is intentional and correct by
> itself, so I guess the problem is in the kernel (the system is
> currently running 4.13.8), where "meta l4proto" sometimes isn't what it
> should be.

Weird.  We changed the implicit l4 dependencies to meta l4proto because
in ipv6 case this will skip extension headers.

IIRC ipv4 was only changed to keep it more simple.
(also nft_set_pktinfo_ipv4 sets
         pkt->tprot_set = true;
         pkt->tprot = iph->protocol;

so I fail to see why it should not be identical in all cases).

Re: nftables rules not matching after upgrading from 0.7 to 0.8

[Anders K. Pedersen | Cohaesio]

On ons, 2017-10-25 at 19:57 +0200, Florian Westphal wrote:
> Anders K. Pedersen | Cohaesio <akp@cohaesio.com> wrote:
> > After upgrading to nftables 0.8 (from 0.7) on one of my systems,
> > I've
> > experiences several cases where rules that used to work fine with
> > 0.7
> > sometimes doesn't match anymore with 0.8 (it's not consistent -
> > sometimes the rules do match with 0.8).
> > 
> > The rule chains end with a log statement before rejecting or
> > dropping
> > the packets, and I can see in the log that everything is as
> > expected
> > and the rules should match. After downgrading to nftables 0.7
> > everything works again.
> 
> Are those errors restricted to a particular table family, chain or
> protocol?

So far I've only registered it for IPv4 input filtering for TCP, but
that's also most of the traffic on this system, so I'm not sure that
it's limited to that.

As mentioned, it's not consistent. A rule that has worked fine could
suddenly stop working without any rule set changes for days. Some times
it has helped to just reload the exact same rule set. Other times
changing

tcp dport { imap2, imaps } flow table imap \
                { ip saddr & 255.255.255.240 \
                  timeout 5m limit rate 10/minute } \
        counter accept

to

tcp dport { imap2, imaps } flow table imap \
        counter accept

or

tcp dport { domain, http, https, 8080, 8443, 9091 } \
        meta iif eth1 accept

to

tcp dport { domain, http, https, 8080, 8443, 9091 } \
        meta iif eth1 counter accept

has resolved it, but it feels like it wasn't really due to the changes,
but more random luck.

> At least for ip family all packets will have pkt->tprot_set true,
> so I don't see how meta would break there.
> 
> >  ip filter input 
> > -  [ payload load 1b @ network header + 9 => reg 1 ]
> > +  [ meta load l4proto => reg 1 ]
> > 
> > and
> > 
> >  ip6 filter input 
> > -  [ payload load 1b @ network header + 6 => reg 1 ]
> > +  [ meta load l4proto => reg 1 ]
> > 
> > for many of the rules. I believe this is intentional and correct by
> > itself, so I guess the problem is in the kernel (the system is
> > currently running 4.13.8), where "meta l4proto" sometimes isn't
> > what it
> > should be.
> 
> Weird.  We changed the implicit l4 dependencies to meta l4proto
> because
> in ipv6 case this will skip extension headers.
> 
> IIRC ipv4 was only changed to keep it more simple.
> (also nft_set_pktinfo_ipv4 sets
>          pkt->tprot_set = true;
>          pkt->tprot = iph->protocol;
> 
> so I fail to see why it should not be identical in all cases).

Any testing, I can do?

Regards,
Anders

Re: nftables rules not matching after upgrading from 0.7 to 0.8

[Anders K. Pedersen | Cohaesio]

On ons, 2017-10-25 at 20:20 +0200, Anders K. Pedersen | Cohaesio wrote:
> On ons, 2017-10-25 at 19:57 +0200, Florian Westphal wrote:
> > Anders K. Pedersen | Cohaesio <akp@cohaesio.com> wrote:
> > > After upgrading to nftables 0.8 (from 0.7) on one of my systems,
> > > I've
> > > experiences several cases where rules that used to work fine with
> > > 0.7
> > > sometimes doesn't match anymore with 0.8 (it's not consistent -
> > > sometimes the rules do match with 0.8).
> > > 
> > > The rule chains end with a log statement before rejecting or
> > > dropping
> > > the packets, and I can see in the log that everything is as
> > > expected
> > > and the rules should match. After downgrading to nftables 0.7
> > > everything works again.
> > 
> > Are those errors restricted to a particular table family, chain or
> > protocol?
> 
> So far I've only registered it for IPv4 input filtering for TCP, but
> that's also most of the traffic on this system, so I'm not sure that
> it's limited to that.
> 
> As mentioned, it's not consistent. A rule that has worked fine could
> suddenly stop working without any rule set changes for days. Some
> times
> it has helped to just reload the exact same rule set. Other times
> changing
> 
> tcp dport { imap2, imaps } flow table imap \
>                 { ip saddr & 255.255.255.240 \
>                   timeout 5m limit rate 10/minute } \
>         counter accept
> 
> to
> 
> tcp dport { imap2, imaps } flow table imap \
>         counter accept
> 
> or
> 
> tcp dport { domain, http, https, 8080, 8443, 9091 } \
>         meta iif eth1 accept
> 
> to
> 
> tcp dport { domain, http, https, 8080, 8443, 9091 } \
>         meta iif eth1 counter accept
> 
> has resolved it, but it feels like it wasn't really due to the
> changes,
> but more random luck.
> 
> > At least for ip family all packets will have pkt->tprot_set true,
> > so I don't see how meta would break there.
> > 
> > >  ip filter input 
> > > -  [ payload load 1b @ network header + 9 => reg 1 ]
> > > +  [ meta load l4proto => reg 1 ]
> > > 
> > > and
> > > 
> > >  ip6 filter input 
> > > -  [ payload load 1b @ network header + 6 => reg 1 ]
> > > +  [ meta load l4proto => reg 1 ]
> > > 
> > > for many of the rules. I believe this is intentional and correct
> > > by
> > > itself, so I guess the problem is in the kernel (the system is
> > > currently running 4.13.8), where "meta l4proto" sometimes isn't
> > > what it
> > > should be.
> > 
> > Weird.  We changed the implicit l4 dependencies to meta l4proto
> > because
> > in ipv6 case this will skip extension headers.
> > 
> > IIRC ipv4 was only changed to keep it more simple.
> > (also nft_set_pktinfo_ipv4 sets
> >          pkt->tprot_set = true;
> >          pkt->tprot = iph->protocol;
> > 
> > so I fail to see why it should not be identical in all cases).
> 
> Any testing, I can do?

One more thing, I just noticed is that if I try to use nftables 0.7 to
dump the rule set that was loaded with 0.7, I get the following error:

# nft list ruleset
nft: payload.c:550: payload_expr_expand: Assertion `desc->base == expr->payload.base' failed.
Aborted

If I use 0.8 to dump the rule set that was loaded with 0.7, I get the
correct rule set except for a difference with regards to sets and maps
that use interfaces like:

--- nft-0.7-0.8 rule set loaded with 0.7 and dumped with 0.8
+++ nft-0.8     same rule set loaded and dumped with 0.8
@@ -9,12 +9,12 @@
        chain prerouting {
                type filter hook prerouting priority -100; policy accept;
                iif "lo" accept
-               ct mark set iif map { 33554432 : 0x00000001, 67108864 : 0x00000002 }
+               ct mark set iif map { "eth0" : 0x00000001, "eth2" : 0x00000002 }
                iif "eth1" jump prerouting_internal
-               iif { 33554432, 67108864 } ip saddr { 0.0.0.0/8, 10.0.0.0/8, 127.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 224.0.0.0-255.255.255.255 } counter packets 6 bytes 705 drop
+               iif { "eth0", "eth2" } ip saddr { 0.0.0.0/8, 10.0.0.0/8, 127.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 224.0.0.0-255.255.255.255 } counter packets 0 bytes 0 drop

--
Regards,
Anders

Re: nftables rules not matching after upgrading from 0.7 to 0.8

[Florian Westphal]

Anders K. Pedersen | Cohaesio <akp@cohaesio.com> wrote:
> On ons, 2017-10-25 at 20:20 +0200, Anders K. Pedersen | Cohaesio wrote:
> > On ons, 2017-10-25 at 19:57 +0200, Florian Westphal wrote:
> > > Anders K. Pedersen | Cohaesio <akp@cohaesio.com> wrote:
> > > > After upgrading to nftables 0.8 (from 0.7) on one of my systems,
> > > > I've
> > > > experiences several cases where rules that used to work fine with
> > > > 0.7
> > > > sometimes doesn't match anymore with 0.8 (it's not consistent -
> > > > sometimes the rules do match with 0.8).
> > > > 
> > > > The rule chains end with a log statement before rejecting or
> > > > dropping
> > > > the packets, and I can see in the log that everything is as
> > > > expected
> > > > and the rules should match. After downgrading to nftables 0.7
> > > > everything works again.
> > > 
> > > Are those errors restricted to a particular table family, chain or
> > > protocol?
> > 
> > So far I've only registered it for IPv4 input filtering for TCP, but
> > that's also most of the traffic on this system, so I'm not sure that
> > it's limited to that.
> > 
> > As mentioned, it's not consistent. A rule that has worked fine could
> > suddenly stop working without any rule set changes for days. Some
> > times
> > it has helped to just reload the exact same rule set. Other times
> > changing
> > 
> > tcp dport { imap2, imaps } flow table imap \
> >                 { ip saddr & 255.255.255.240 \
> >                   timeout 5m limit rate 10/minute } \
> >         counter accept
> > 
> > to
> > 
> > tcp dport { imap2, imaps } flow table imap \
> >         counter accept
> > 
> > or
> > 
> > tcp dport { domain, http, https, 8080, 8443, 9091 } \
> >         meta iif eth1 accept
> > 
> > to
> > 
> > tcp dport { domain, http, https, 8080, 8443, 9091 } \
> >         meta iif eth1 counter accept
> > 
> > has resolved it, but it feels like it wasn't really due to the
> > changes,
> > but more random luck.

note that sets are broken with 16bit elements at the moment in 4.13, see

https://patchwork.ozlabs.org/patch/821080/
or
https://patchwork.ozlabs.org/patch/830236/

> One more thing, I just noticed is that if I try to use nftables 0.7 to
> dump the rule set that was loaded with 0.7, I get the following error:
> 
> # nft list ruleset
> nft: payload.c:550: payload_expr_expand: Assertion `desc->base == expr->payload.base' failed.
> Aborted

Crap, I'll fix this, thanks for reporting.

> If I use 0.8 to dump the rule set that was loaded with 0.7, I get the
> correct rule set except for a difference with regards to sets and maps
> that use interfaces like:
> 
> --- nft-0.7-0.8 rule set loaded with 0.7 and dumped with 0.8
> +++ nft-0.8     same rule set loaded and dumped with 0.8
> @@ -9,12 +9,12 @@
>         chain prerouting {
>                 type filter hook prerouting priority -100; policy accept;
>                 iif "lo" accept
> -               ct mark set iif map { 33554432 : 0x00000001, 67108864 : 0x00000002 }
> +               ct mark set iif map { "eth0" : 0x00000001, "eth2" : 0x00000002 }
>                 iif "eth1" jump prerouting_internal
> -               iif { 33554432, 67108864 } ip saddr { 0.0.0.0/8, 10.0.0.0/8, 127.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 224.0.0.0-255.255.255.255 } counter packets 6 bytes 705 drop
> +               iif { "eth0", "eth2" } ip saddr { 0.0.0.0/8, 10.0.0.0/8, 127.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 224.0.0.0-255.255.255.255 } counter packets 0 bytes 0 drop

I will look at this too.

Re: nftables rules not matching after upgrading from 0.7 to 0.8

[Pablo Neira Ayuso]

On Thu, Oct 26, 2017 at 12:45:36AM +0200, Florian Westphal wrote:
> Anders K. Pedersen | Cohaesio <akp@cohaesio.com> wrote:
> > On ons, 2017-10-25 at 20:20 +0200, Anders K. Pedersen | Cohaesio wrote:
> > > On ons, 2017-10-25 at 19:57 +0200, Florian Westphal wrote:
> > > > Anders K. Pedersen | Cohaesio <akp@cohaesio.com> wrote:
[...]
> > If I use 0.8 to dump the rule set that was loaded with 0.7, I get the
> > correct rule set except for a difference with regards to sets and maps
> > that use interfaces like:
> > 
> > --- nft-0.7-0.8 rule set loaded with 0.7 and dumped with 0.8
> > +++ nft-0.8     same rule set loaded and dumped with 0.8
> > @@ -9,12 +9,12 @@
> >         chain prerouting {
> >                 type filter hook prerouting priority -100; policy accept;
> >                 iif "lo" accept
> > -               ct mark set iif map { 33554432 : 0x00000001, 67108864 : 0x00000002 }
> > +               ct mark set iif map { "eth0" : 0x00000001, "eth2" : 0x00000002 }
> >                 iif "eth1" jump prerouting_internal
> > -               iif { 33554432, 67108864 } ip saddr { 0.0.0.0/8, 10.0.0.0/8, 127.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 224.0.0.0-255.255.255.255 } counter packets 6 bytes 705 drop
> > +               iif { "eth0", "eth2" } ip saddr { 0.0.0.0/8, 10.0.0.0/8, 127.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 224.0.0.0-255.255.255.255 } counter packets 0 bytes 0 drop
> 
> I will look at this too.

This difference in the way things are dump is related to this:

commit b9b6092304aef17fea704c25b3d9d7dcdb3995a5
Author: Pablo Neira Ayuso <pablo@netfilter.org>
Date:   Fri Feb 24 20:47:49 2017 +0100

    evaluate: store byteorder for set keys

To my understanding, byteorder was not OK in 0.7.

Are these maps not matching anymore or is just the dump theat is
different?

Thanks!

Re: nftables rules not matching after upgrading from 0.7 to 0.8

[Pablo Neira Ayuso]

On Thu, Oct 26, 2017 at 12:45:36AM +0200, Florian Westphal wrote:
> Anders K. Pedersen | Cohaesio <akp@cohaesio.com> wrote:
> > On ons, 2017-10-25 at 20:20 +0200, Anders K. Pedersen | Cohaesio wrote:
> > > On ons, 2017-10-25 at 19:57 +0200, Florian Westphal wrote:
> > > > Anders K. Pedersen | Cohaesio <akp@cohaesio.com> wrote:
[...]
> > > As mentioned, it's not consistent. A rule that has worked fine could
> > > suddenly stop working without any rule set changes for days. Some
> > > times
> > > it has helped to just reload the exact same rule set. Other times
> > > changing
> > > 
> > > tcp dport { imap2, imaps } flow table imap \
> > >                 { ip saddr & 255.255.255.240 \
> > >                   timeout 5m limit rate 10/minute } \
> > >         counter accept
> > > 
> > > to
> > > 
> > > tcp dport { imap2, imaps } flow table imap \
> > >         counter accept
> > > 
> > > or
> > > 
> > > tcp dport { domain, http, https, 8080, 8443, 9091 } \
> > >         meta iif eth1 accept
> > > 
> > > to
> > > 
> > > tcp dport { domain, http, https, 8080, 8443, 9091 } \
> > >         meta iif eth1 counter accept
> > > 
> > > has resolved it, but it feels like it wasn't really due to the
> > > changes,
> > > but more random luck.
> 
> note that sets are broken with 16bit elements at the moment in 4.13, see
> 
> https://patchwork.ozlabs.org/patch/821080/
> or
> https://patchwork.ozlabs.org/patch/830236/

JFYI, the fix for sets is now in nf.git.

https://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf.git/commit/

I swear this was included in the previous batch, but it was not. Sorry
about this. Will send this upstream asap.

Re: nftables rules not matching after upgrading from 0.7 to 0.8

[Anders K. Pedersen | Cohaesio]

On tor, 2017-10-26 at 00:45 +0200, Florian Westphal wrote:
> Anders K. Pedersen | Cohaesio <akp@cohaesio.com> wrote:
> > On ons, 2017-10-25 at 20:20 +0200, Anders K. Pedersen | Cohaesio
> > wrote:
> > > On ons, 2017-10-25 at 19:57 +0200, Florian Westphal wrote:
> > > > Anders K. Pedersen | Cohaesio <akp@cohaesio.com> wrote:
> > > > > After upgrading to nftables 0.8 (from 0.7) on one of my
> > > > > systems,
> > > > > I've
> > > > > experiences several cases where rules that used to work fine
> > > > > with
> > > > > 0.7
> > > > > sometimes doesn't match anymore with 0.8 (it's not consistent
> > > > > -
> > > > > sometimes the rules do match with 0.8).
> > > > > 
> > > > > The rule chains end with a log statement before rejecting or
> > > > > dropping
> > > > > the packets, and I can see in the log that everything is as
> > > > > expected
> > > > > and the rules should match. After downgrading to nftables 0.7
> > > > > everything works again.
> > > > 
> > > > Are those errors restricted to a particular table family, chain
> > > > or
> > > > protocol?
> > > 
> > > So far I've only registered it for IPv4 input filtering for TCP,
> > > but
> > > that's also most of the traffic on this system, so I'm not sure
> > > that
> > > it's limited to that.
> > > 
> > > As mentioned, it's not consistent. A rule that has worked fine
> > > could
> > > suddenly stop working without any rule set changes for days. Some
> > > times
> > > it has helped to just reload the exact same rule set. Other times
> > > changing
> > > 
> > > tcp dport { imap2, imaps } flow table imap \
> > >                 { ip saddr & 255.255.255.240 \
> > >                   timeout 5m limit rate 10/minute } \
> > >         counter accept
> > > 
> > > to
> > > 
> > > tcp dport { imap2, imaps } flow table imap \
> > >         counter accept
> > > 
> > > or
> > > 
> > > tcp dport { domain, http, https, 8080, 8443, 9091 } \
> > >         meta iif eth1 accept
> > > 
> > > to
> > > 
> > > tcp dport { domain, http, https, 8080, 8443, 9091 } \
> > >         meta iif eth1 counter accept
> > > 
> > > has resolved it, but it feels like it wasn't really due to the
> > > changes,
> > > but more random luck.
> 
> note that sets are broken with 16bit elements at the moment in 4.13,
> see
> 
> https://patchwork.ozlabs.org/patch/821080/

I have just updated to 4.13.9 plus this patch, and will let you know,
if this changes anything.

> or
> https://patchwork.ozlabs.org/patch/830236/
> 
> > One more thing, I just noticed is that if I try to use nftables 0.7
> > to
> > dump the rule set that was loaded with 0.7, I get the following
> > error:

This should have said load with 0.8, dump with 0.7:

> > # nft list ruleset
> > nft: payload.c:550: payload_expr_expand: Assertion `desc->base ==
> > expr->payload.base' failed.
> > Aborted
> 
> Crap, I'll fix this, thanks for reporting.
> 
> > If I use 0.8 to dump the rule set that was loaded with 0.7, I get
> > the
> > correct rule set except for a difference with regards to sets and
> > maps
> > that use interfaces like:
> > 
> > --- nft-0.7-0.8 rule set loaded with 0.7 and dumped with 0.8
> > +++ nft-0.8     same rule set loaded and dumped with 0.8
> > @@ -9,12 +9,12 @@
> >         chain prerouting {
> >                 type filter hook prerouting priority -100; policy
> > accept;
> >                 iif "lo" accept
> > -               ct mark set iif map { 33554432 : 0x00000001,
> > 67108864 : 0x00000002 }
> > +               ct mark set iif map { "eth0" : 0x00000001, "eth2" :
> > 0x00000002 }
> >                 iif "eth1" jump prerouting_internal
> > -               iif { 33554432, 67108864 } ip saddr { 0.0.0.0/8,
> > 10.0.0.0/8, 127.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 224.0.0.0-
> > 255.255.255.255 } counter packets 6 bytes 705 drop
> > +               iif { "eth0", "eth2" } ip saddr { 0.0.0.0/8,
> > 10.0.0.0/8, 127.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 224.0.0.0-
> > 255.255.255.255 } counter packets 0 bytes 0 drop
> 
> I will look at this too.

Thanks.

--
Regards,
Anders

Re: nftables rules not matching after upgrading from 0.7 to 0.8

[Anders K. Pedersen | Cohaesio]

On tor, 2017-10-26 at 01:44 +0200, Pablo Neira Ayuso wrote:
> On Thu, Oct 26, 2017 at 12:45:36AM +0200, Florian Westphal wrote:
> > Anders K. Pedersen | Cohaesio <akp@cohaesio.com> wrote:
> > > On ons, 2017-10-25 at 20:20 +0200, Anders K. Pedersen | Cohaesio
> > > wrote:
> > > > On ons, 2017-10-25 at 19:57 +0200, Florian Westphal wrote:
> > > > > Anders K. Pedersen | Cohaesio <akp@cohaesio.com> wrote:
> 
> [...]
> > > If I use 0.8 to dump the rule set that was loaded with 0.7, I get
> > > the
> > > correct rule set except for a difference with regards to sets and
> > > maps
> > > that use interfaces like:
> > > 
> > > --- nft-0.7-0.8 rule set loaded with 0.7 and dumped with 0.8
> > > +++ nft-0.8     same rule set loaded and dumped with 0.8
> > > @@ -9,12 +9,12 @@
> > >         chain prerouting {
> > >                 type filter hook prerouting priority -100; policy
> > > accept;
> > >                 iif "lo" accept
> > > -               ct mark set iif map { 33554432 : 0x00000001,
> > > 67108864 : 0x00000002 }
> > > +               ct mark set iif map { "eth0" : 0x00000001, "eth2"
> > > : 0x00000002 }
> > >                 iif "eth1" jump prerouting_internal
> > > -               iif { 33554432, 67108864 } ip saddr { 0.0.0.0/8,
> > > 10.0.0.0/8, 127.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16,
> > > 224.0.0.0-255.255.255.255 } counter packets 6 bytes 705 drop
> > > +               iif { "eth0", "eth2" } ip saddr { 0.0.0.0/8,
> > > 10.0.0.0/8, 127.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16,
> > > 224.0.0.0-255.255.255.255 } counter packets 0 bytes 0 drop
> > 
> > I will look at this too.
> 
> This difference in the way things are dump is related to this:
> 
> commit b9b6092304aef17fea704c25b3d9d7dcdb3995a5
> Author: Pablo Neira Ayuso <pablo@netfilter.org>
> Date:   Fri Feb 24 20:47:49 2017 +0100
> 
>     evaluate: store byteorder for set keys
> 
> To my understanding, byteorder was not OK in 0.7.
> 
> Are these maps not matching anymore or is just the dump theat is
> different?

The maps are matching and sets the intended marks, so it's just the
dump that is weird, when the rule set has been loaded with 0.7 and is
dumped with 0.8.

-- 
Regards,
Anders

Re: nftables rules not matching after upgrading from 0.7 to 0.8

[Anders K. Pedersen | Cohaesio]

On tor, 2017-10-26 at 08:47 +0200, Anders K. Pedersen | Cohaesio wrote:
> On tor, 2017-10-26 at 00:45 +0200, Florian Westphal wrote:
> > Anders K. Pedersen | Cohaesio <akp@cohaesio.com> wrote:
> > > On ons, 2017-10-25 at 20:20 +0200, Anders K. Pedersen | Cohaesio
> > > wrote:
> > > > On ons, 2017-10-25 at 19:57 +0200, Florian Westphal wrote:
> > > > > Anders K. Pedersen | Cohaesio <akp@cohaesio.com> wrote:
> > > > > > After upgrading to nftables 0.8 (from 0.7) on one of my
> > > > > > systems,
> > > > > > I've
> > > > > > experiences several cases where rules that used to work
> > > > > > fine
> > > > > > with
> > > > > > 0.7
> > > > > > sometimes doesn't match anymore with 0.8 (it's not
> > > > > > consistent
> > > > > > -
> > > > > > sometimes the rules do match with 0.8).
> > > > > > 
> > > > > > The rule chains end with a log statement before rejecting
> > > > > > or
> > > > > > dropping
> > > > > > the packets, and I can see in the log that everything is as
> > > > > > expected
> > > > > > and the rules should match. After downgrading to nftables
> > > > > > 0.7
> > > > > > everything works again.
> > > > > 
> > > > > Are those errors restricted to a particular table family,
> > > > > chain
> > > > > or
> > > > > protocol?
> > > > 
> > > > So far I've only registered it for IPv4 input filtering for
> > > > TCP,
> > > > but
> > > > that's also most of the traffic on this system, so I'm not sure
> > > > that
> > > > it's limited to that.
> > > > 
> > > > As mentioned, it's not consistent. A rule that has worked fine
> > > > could
> > > > suddenly stop working without any rule set changes for days.
> > > > Some
> > > > times
> > > > it has helped to just reload the exact same rule set. Other
> > > > times
> > > > changing
> > > > 
> > > > tcp dport { imap2, imaps } flow table imap \
> > > >                 { ip saddr & 255.255.255.240 \
> > > >                   timeout 5m limit rate 10/minute } \
> > > >         counter accept
> > > > 
> > > > to
> > > > 
> > > > tcp dport { imap2, imaps } flow table imap \
> > > >         counter accept
> > > > 
> > > > or
> > > > 
> > > > tcp dport { domain, http, https, 8080, 8443, 9091 } \
> > > >         meta iif eth1 accept
> > > > 
> > > > to
> > > > 
> > > > tcp dport { domain, http, https, 8080, 8443, 9091 } \
> > > >         meta iif eth1 counter accept
> > > > 
> > > > has resolved it, but it feels like it wasn't really due to the
> > > > changes,
> > > > but more random luck.
> > 
> > note that sets are broken with 16bit elements at the moment in
> > 4.13,
> > see
> > 
> > https://patchwork.ozlabs.org/patch/821080/
> 
> I have just updated to 4.13.9 plus this patch, and will let you know,
> if this changes anything.

Since this patch was applied, I've not had any further problems with
nftables 0.8. Does it make sense that nftables 0.7 worked fine without
the patch, or was this just random luck?

Regards,
Anders

> > or
> > https://patchwork.ozlabs.org/patch/830236/
> > 
> > > One more thing, I just noticed is that if I try to use nftables
> > > 0.7
> > > to
> > > dump the rule set that was loaded with 0.7, I get the following
> > > error:
> 
> This should have said load with 0.8, dump with 0.7:
> 
> > > # nft list ruleset
> > > nft: payload.c:550: payload_expr_expand: Assertion `desc->base ==
> > > expr->payload.base' failed.
> > > Aborted
> > 
> > Crap, I'll fix this, thanks for reporting.
> > 
> > > If I use 0.8 to dump the rule set that was loaded with 0.7, I get
> > > the
> > > correct rule set except for a difference with regards to sets and
> > > maps
> > > that use interfaces like:
> > > 
> > > --- nft-0.7-0.8 rule set loaded with 0.7 and dumped with 0.8
> > > +++ nft-0.8     same rule set loaded and dumped with 0.8
> > > @@ -9,12 +9,12 @@
> > >         chain prerouting {
> > >                 type filter hook prerouting priority -100; policy
> > > accept;
> > >                 iif "lo" accept
> > > -               ct mark set iif map { 33554432 : 0x00000001,
> > > 67108864 : 0x00000002 }
> > > +               ct mark set iif map { "eth0" : 0x00000001, "eth2"
> > > :
> > > 0x00000002 }
> > >                 iif "eth1" jump prerouting_internal
> > > -               iif { 33554432, 67108864 } ip saddr { 0.0.0.0/8,
> > > 10.0.0.0/8, 127.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16,
> > > 224.0.0.0-
> > > 255.255.255.255 } counter packets 6 bytes 705 drop
> > > +               iif { "eth0", "eth2" } ip saddr { 0.0.0.0/8,
> > > 10.0.0.0/8, 127.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16,
> > > 224.0.0.0-
> > > 255.255.255.255 } counter packets 0 bytes 0 drop
> > 
> > I will look at this too.
> 
> Thanks.
> 
> --
> Regards,
> Anders

Re: nftables rules not matching after upgrading from 0.7 to 0.8

[Pablo Neira Ayuso]

On Mon, Nov 06, 2017 at 04:49:04PM +0000, Anders K. Pedersen | Cohaesio wrote:
[...]
> Since this patch was applied, I've not had any further problems with
> nftables 0.8. Does it make sense that nftables 0.7 worked fine without
> the patch, or was this just random luck?

Yes. nftables 0.7 doesn't not provide indications to the kernel on the
number of elements in the set descriptions, hence it selects the
slowier hashtable representation, which was not buggy :-)