* Re: [BUG / question] in routing rules, some options (e.g. ipproto, sport) cause rules to be ignored in presence of packet marks
2021-03-29 20:52 ` Ido Schimmel
@ 2021-03-29 21:18 ` Ido Schimmel
2021-03-30 11:46 ` Michal Soltys
2021-04-09 12:11 ` Michal Soltys
2 siblings, 0 replies; 8+ messages in thread
From: Ido Schimmel @ 2021-03-29 21:18 UTC (permalink / raw)
To: Michal Soltys; +Cc: Linux Netdev List, David Miller
On Mon, Mar 29, 2021 at 11:52:10PM +0300, Ido Schimmel wrote:
> On Sun, Mar 28, 2021 at 04:05:29PM +0200, Michal Soltys wrote:
> > Hi,
> >
> > I'm not sure how it behaved in earlier kernels (can check later), but it is
> > / looks bugged in at least recent 5.x+ ones (tests were done with 5.11.8 and
> > 5.10.25).
> >
> > Consider following setup:
> >
> > # ip -o ad sh
> > 1: lo inet 127.0.0.1/8 scope host lo
> > 2: right1 inet 10.0.10.2/24 scope global
> > 3: right2 inet 10.0.20.2/24 scope global
> >
> > # ip ro sh tab main
> > default via 10.0.10.1 dev right1
> > 10.0.10.0/24 dev right1 proto kernel scope link src 10.0.10.2
> > 10.0.20.0/24 dev right2 proto kernel scope link src 10.0.20.2
> >
> > # ip ro sh tab 123
> > default via 10.0.20.1 dev right2 src 10.0.20.2
> >
> > And routing rules:
> >
> > 0: from all lookup local
> > 9: from all fwmark 0x1 ipproto udp sport 1194 lookup 123
> > 10: from all ipproto udp sport 1194 lookup 123
> > 32766: from all lookup main
> > 32767: from all lookup default
> >
> > This - without any mangling via ipt/nft or by other means - works correctly,
> > for example:
> >
> > nc -u -p 1194 1.2.3.4 12345
> >
> > will be routed out correctly via 'right2' using 10.0.20.2
> >
> > But if we add mark to locally outgoing packets:
> >
> > iptables -t mangle -A OUTPUT -j MARK --set-mark 1
> >
> > Then *both* rule 9 and rule 10 will be ignored during reroute check. tcpdump
> > on interface 'right1' will show:
> >
> > # tcpdump -nvi right1 udp
> > tcpdump: listening on right1, link-type EN10MB (Ethernet), snapshot length
> > 262144 bytes
> > 13:21:59.684928 IP (tos 0x0, ttl 64, id 8801, offset 0, flags [DF], proto
> > UDP (17), length 33)
> > 10.0.20.2.1194 > 1.2.3.4.12345: UDP, length 5
> >
> > Initial routing decision in rule 10 will set the address correctly, but the
> > packet goes out via interface right1, ignoring both 9 and 10.
> >
> > If I add another routing roule:
> >
> > 8: from all fwmark 0x1 lookup 123
> >
> > Then the packects will flow correctly - but I *cannot* use (from the ones I
> > tested): sport, dport, ipproto, uidrange - as they will cause the rule to be
> > ignored. For example, this setup of routing rules will fail, if there is any
> > mark set on a packet (nc had uid 1120):
> >
> > # ip ru sh
> > 0: from all lookup local
> > 10: from all ipproto udp lookup 123
> > 10: from all sport 1194 lookup 123
> > 10: from all dport 12345 lookup 123
> > 10: from all uidrange 1120-1120 lookup 123
> > 32766: from all lookup main
> > 32767: from all lookup default
> >
> > Adding correct fwmark to the above rules will have *no* effect either. Only
> > fwmark *alone* will work (or in combination with: iif, from, to - from the
> > ones I tested).
> >
> > I peeked at fib_rule_match() in net/core/fib_rules.c - but it doesn't look
> > like there is anything wrong there. I initially suspected lack of
> > 'rule->mark &&' in mark related line - but considering that rules such as
> > 'from all fwmark 1 sport 1194 lookup main' also fail, it doesn't look like
> > it's the culprit (and mark_mask covers that test either way).
> >
> > OTOH, perhaps nf_ip_reroute() / ip_route_me_harder() are somehow the culprit
> > here - but I haven't analyzed them yet. Perhaps it's just an issue of
> > changing output interface incorrectly after ip_route_me_harder() ?
>
> ip_route_me_harder() does not set source / destination port in the
> flow key, so it explains why fib rules that use them are not hit after
> mangling the packet. These keys were added in 4.17, but I
> don't think this use case every worked. You have a different experience?
It's already tomorrow here, but I think that if you record the
'fib:fib_table_lookup' tracepoint before and after adding the mangling
rules you will see that there is a second lookup for the packet with
zero source / destination port. Something like:
# perf record -a -e fib:fib_table_lookup -- sleep 5
# perf script --stdio
>
> >
> > Is this a bug ? Or am I misinterpreting how 'reroute check' works after
> > initial routing decision ? One would expect routing rules during post-mangle
> > check to not be ignored out of the blue, only because packet mark changed on
> > the packet. Not mentioning both marks and routing rules can be used for
> > separate purposes (e.g. marks for shaping).
> >
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [BUG / question] in routing rules, some options (e.g. ipproto, sport) cause rules to be ignored in presence of packet marks
2021-03-29 20:52 ` Ido Schimmel
2021-03-29 21:18 ` Ido Schimmel
@ 2021-03-30 11:46 ` Michal Soltys
2021-04-09 12:11 ` Michal Soltys
2 siblings, 0 replies; 8+ messages in thread
From: Michal Soltys @ 2021-03-30 11:46 UTC (permalink / raw)
To: Ido Schimmel; +Cc: Linux Netdev List, David Miller
On 3/29/21 10:52 PM, Ido Schimmel wrote:
> On Sun, Mar 28, 2021 at 04:05:29PM +0200, Michal Soltys wrote:
>> Hi,
>>
>> I'm not sure how it behaved in earlier kernels (can check later), but it is
>> / looks bugged in at least recent 5.x+ ones (tests were done with 5.11.8 and
>> 5.10.25).
>>
>> Consider following setup:
>>
>> # ip -o ad sh
>> 1: lo inet 127.0.0.1/8 scope host lo
>> 2: right1 inet 10.0.10.2/24 scope global
>> 3: right2 inet 10.0.20.2/24 scope global
>>
>> # ip ro sh tab main
>> default via 10.0.10.1 dev right1
>> 10.0.10.0/24 dev right1 proto kernel scope link src 10.0.10.2
>> 10.0.20.0/24 dev right2 proto kernel scope link src 10.0.20.2
>>
>> # ip ro sh tab 123
>> default via 10.0.20.1 dev right2 src 10.0.20.2
>>
>> And routing rules:
>>
>> 0: from all lookup local
>> 9: from all fwmark 0x1 ipproto udp sport 1194 lookup 123
>> 10: from all ipproto udp sport 1194 lookup 123
>> 32766: from all lookup main
>> 32767: from all lookup default
>>
>> This - without any mangling via ipt/nft or by other means - works correctly,
>> for example:
>>
>> nc -u -p 1194 1.2.3.4 12345
>>
>> will be routed out correctly via 'right2' using 10.0.20.2
>>
>> But if we add mark to locally outgoing packets:
>>
>> iptables -t mangle -A OUTPUT -j MARK --set-mark 1
>>
>> Then *both* rule 9 and rule 10 will be ignored during reroute check. tcpdump
>> on interface 'right1' will show:
>>
>> # tcpdump -nvi right1 udp
>> tcpdump: listening on right1, link-type EN10MB (Ethernet), snapshot length
>> 262144 bytes
>> 13:21:59.684928 IP (tos 0x0, ttl 64, id 8801, offset 0, flags [DF], proto
>> UDP (17), length 33)
>> 10.0.20.2.1194 > 1.2.3.4.12345: UDP, length 5
>>
>> Initial routing decision in rule 10 will set the address correctly, but the
>> packet goes out via interface right1, ignoring both 9 and 10.
>>
>> If I add another routing roule:
>>
>> 8: from all fwmark 0x1 lookup 123
>>
>> Then the packects will flow correctly - but I *cannot* use (from the ones I
>> tested): sport, dport, ipproto, uidrange - as they will cause the rule to be
>> ignored. For example, this setup of routing rules will fail, if there is any
>> mark set on a packet (nc had uid 1120):
>>
>> # ip ru sh
>> 0: from all lookup local
>> 10: from all ipproto udp lookup 123
>> 10: from all sport 1194 lookup 123
>> 10: from all dport 12345 lookup 123
>> 10: from all uidrange 1120-1120 lookup 123
>> 32766: from all lookup main
>> 32767: from all lookup default
>>
>> Adding correct fwmark to the above rules will have *no* effect either. Only
>> fwmark *alone* will work (or in combination with: iif, from, to - from the
>> ones I tested).
>>
>> I peeked at fib_rule_match() in net/core/fib_rules.c - but it doesn't look
>> like there is anything wrong there. I initially suspected lack of
>> 'rule->mark &&' in mark related line - but considering that rules such as
>> 'from all fwmark 1 sport 1194 lookup main' also fail, it doesn't look like
>> it's the culprit (and mark_mask covers that test either way).
>>
>> OTOH, perhaps nf_ip_reroute() / ip_route_me_harder() are somehow the culprit
>> here - but I haven't analyzed them yet. Perhaps it's just an issue of
>> changing output interface incorrectly after ip_route_me_harder() ?
>
> ip_route_me_harder() does not set source / destination port in the
> flow key, so it explains why fib rules that use them are not hit after
> mangling the packet. These keys were added in 4.17, but I
> don't think this use case every worked. You have a different experience?
>
Nah, started using ipproto/sport just recently - so only tested with
5.10+ kernels.
But - I did quick check just a moment ago with 4.19.132 kernel - it's as
you say, not working there either.
>>
>> Is this a bug ? Or am I misinterpreting how 'reroute check' works after
>> initial routing decision ? One would expect routing rules during post-mangle
>> check to not be ignored out of the blue, only because packet mark changed on
>> the packet. Not mentioning both marks and routing rules can be used for
>> separate purposes (e.g. marks for shaping).
>>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [BUG / question] in routing rules, some options (e.g. ipproto, sport) cause rules to be ignored in presence of packet marks
2021-03-29 20:52 ` Ido Schimmel
2021-03-29 21:18 ` Ido Schimmel
2021-03-30 11:46 ` Michal Soltys
@ 2021-04-09 12:11 ` Michal Soltys
2021-04-09 13:02 ` Florian Westphal
2 siblings, 1 reply; 8+ messages in thread
From: Michal Soltys @ 2021-04-09 12:11 UTC (permalink / raw)
To: Ido Schimmel; +Cc: Linux Netdev List, David Miller
On 3/29/21 10:52 PM, Ido Schimmel wrote:
>
> ip_route_me_harder() does not set source / destination port in the
> flow key, so it explains why fib rules that use them are not hit after
> mangling the packet. These keys were added in 4.17, but I
> don't think this use case every worked. You have a different experience?
>
So all the more recent additions to routing rules - src port, dst port,
uid range and ipproto - are not functioning correctly with the second
routing check.
Are there plans to eventually fix that ?
While I just adjusted/rearranged my stuff to not rely on those, it
should probably be at least documented otherwise (presumably in ip-rule
manpage and perhaps in `ip rule help` as well).
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [BUG / question] in routing rules, some options (e.g. ipproto, sport) cause rules to be ignored in presence of packet marks
2021-04-09 12:11 ` Michal Soltys
@ 2021-04-09 13:02 ` Florian Westphal
2021-04-09 13:20 ` Ido Schimmel
0 siblings, 1 reply; 8+ messages in thread
From: Florian Westphal @ 2021-04-09 13:02 UTC (permalink / raw)
To: Michal Soltys; +Cc: Ido Schimmel, Linux Netdev List, David Miller
Michal Soltys <msoltyspl@yandex.pl> wrote:
> On 3/29/21 10:52 PM, Ido Schimmel wrote:
> >
> > ip_route_me_harder() does not set source / destination port in the
> > flow key, so it explains why fib rules that use them are not hit after
> > mangling the packet. These keys were added in 4.17, but I
> > don't think this use case every worked. You have a different experience?
> >
>
> So all the more recent additions to routing rules - src port, dst port, uid
> range and ipproto - are not functioning correctly with the second routing
> check.
>
> Are there plans to eventually fix that ?
>
> While I just adjusted/rearranged my stuff to not rely on those, it should
> probably be at least documented otherwise (presumably in ip-rule manpage and
> perhaps in `ip rule help` as well).
Fixing this would be better. As Ido implies it should be enough to fully
populate the flow keys in ip(6)_route_me_harder.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [BUG / question] in routing rules, some options (e.g. ipproto, sport) cause rules to be ignored in presence of packet marks
2021-04-09 13:02 ` Florian Westphal
@ 2021-04-09 13:20 ` Ido Schimmel
0 siblings, 0 replies; 8+ messages in thread
From: Ido Schimmel @ 2021-04-09 13:20 UTC (permalink / raw)
To: Florian Westphal; +Cc: Michal Soltys, Linux Netdev List, David Miller
On Fri, Apr 09, 2021 at 03:02:41PM +0200, Florian Westphal wrote:
> Michal Soltys <msoltyspl@yandex.pl> wrote:
> > On 3/29/21 10:52 PM, Ido Schimmel wrote:
> > >
> > > ip_route_me_harder() does not set source / destination port in the
> > > flow key, so it explains why fib rules that use them are not hit after
> > > mangling the packet. These keys were added in 4.17, but I
> > > don't think this use case every worked. You have a different experience?
> > >
> >
> > So all the more recent additions to routing rules - src port, dst port, uid
> > range and ipproto - are not functioning correctly with the second routing
> > check.
> >
> > Are there plans to eventually fix that ?
> >
> > While I just adjusted/rearranged my stuff to not rely on those, it should
> > probably be at least documented otherwise (presumably in ip-rule manpage and
> > perhaps in `ip rule help` as well).
>
> Fixing this would be better.
Yep.
> As Ido implies it should be enough to fully populate the flow keys in
> ip(6)_route_me_harder.
Will try to patch this.
^ permalink raw reply [flat|nested] 8+ messages in thread