packets trickling out of STP-blocked ports

packets trickling out of STP-blocked ports

[Colin Foster]

Hi all,

I'm not sure who all to include in this email, but I'm starting with
this list to start.

Probably obvious to those in this email list, I'm testing a VSC7512 dev
board controlled via SPI. The patches are still out-of-tree, but I
figured I'll report these findings, since they seem real.

My setup is port 0 of the 7512 is tied to a Beaglebone Black. Port 1 is
tied to my development PC. Ports 2 and 3 are tied together to test STP.

I run the commands:

ip link set eth0 up
ip link set swp[1-3] up
ip link add name br0 type bridge stp_state 1
ip link set dev swp[1-3] master br0
ip addr add 10.100.3.1/16 dev br0
ip link set dev br0 up

After running this, the STP blocks swp3, and swp1/2 are forwarding.

Periodically I see messages saying that swp2 is receiving packets with
own address as source address.

I can confirm that via ethtool that TX packets are increasing on swp3. I
believe I captured the event via tshark. A 4 minute capture showed three
non-STP packets on swp2. All three of these packets are ICMPv6 Router
Solicitation packets. 

I would expect no packets at all to egress swp3. Is this an issue that
is unique to me and my in-development configuration? Or is this an issue
with all Ocelot / Felix devices?

If this is an Ocelot thing, I can try to come up with a different test 
setup to capture more data... printing the packet when it is received,
capturing the traffic externally, capturing eth0 traffic to see if it is
coming from the kernel or being hardware-forwarded...

(side note - if there's a place where a parser for Ocelot NPI traffic is
hidden, that might eventually save me a lot of debugging in Lua)


An idea of how frequently this happens - my system has been currently up
for 3700 seconds. Eight "own address as source address" events have
happened at 66, 96, 156, 279, 509, 996, 1897, and 3699 seconds. 

Thanks, 

Colin Foster

Re: packets trickling out of STP-blocked ports

[Vladimir Oltean]

Hi Colin,

On Thu, Dec 30, 2021 at 03:07:40PM -0800, Colin Foster wrote:
> After running this, the STP blocks swp3, and swp1/2 are forwarding.
>
> Periodically I see messages saying that swp2 is receiving packets with
> own address as source address.
>
> I can confirm that via ethtool that TX packets are increasing on swp3. I
> believe I captured the event via tshark. A 4 minute capture showed three
> non-STP packets on swp2. All three of these packets are ICMPv6 Router
> Solicitation packets.
>
> I would expect no packets at all to egress swp3. Is this an issue that
> is unique to me and my in-development configuration? Or is this an issue
> with all Ocelot / Felix devices?

I don't remember noticing these (or maybe I did and I forgot), but
reasoning about it, it's a pretty logical consequence of some of the
design decisions that were made.

One would think that when a network interface is under a bridge, it is
unavailable for direct IP termination by itself - you do the IP
termination through the br0 interface. But that isn't really enforced
anywhere - it's just that the bridge breaks IP termination by default on
its individual member ports by stealing all their traffic with its RX handler.
That RX handler can be taught what to steal and what not to steal using
netfilter ebtables rules. With some carefully designed rules, you could
still have some IP termination through the individual bridge ports.

Hardware isn't carved out according to your expectation that no packets
should egress a blocked port, either. Switches in general, and Ocelot in
particular, have a way to send "control" packets that bypass the
analyzer block and STP state (the bridging service, basically) and are
sent towards a precise set of destination ports. This is done by setting
the BYPASS bit from the injection frame header. Currently, Linux sends
"control" packets to the switch all the time, and that is fine, because
although those packets have the ability to go where they don't belong,
the OS (the bridge driver) is supposed to know that, and just not send
packets there. As a side note, there was some work to allow switch
drivers to send "data" packets to the switch, and these correspond to
traffic that originates from a bridge device, but I am just mentioning
this to clarify that it is irrelevant for the purpose of the discussion here.

Even considering an Intel card with no bridging offload at all, if you
put it in the same situation (eth0 under br0, and eth0 is blocked), you
can still put an IP address on eth0 and ping away just fine (you won't
get back the reply as mentioned above, but that's separate really).
Nobody will prevent packets from eth0 from being sent, since the bridge
driver code path isn't invoked on TX unless the socket is bound to br0.

The key point is that the direct xmit data path through swp3, as well as
the data path br0 -> swp3, both exist, in hardware and in software. And
while in hardware they're a bit more clearly separated (in IEEE 802.1Q
there's even a block diagram to clarify that both exist), in software
they're entangled in a bit of a mess, and there are parts of the network
stack and of user space that aren't aware that swp3 is under a bridge,
so IPv6 Router Solicitation messages being sent through swp3 shouldn't
be much of a surprise.



With that out of the way.

Traditionally, DSA has made a design decision that all switch ports
inherit the single MAC address of the DSA master. IOW, if you have 1 DSA
master and 4 switch ports, you have 5 interfaces in the system with the
same MAC address. It was like this for a long time, and relatively
recently, Xiaofei Shen added the ability for individual DSA interfaces
to have their own MAC address stored in the device tree.

As an argument in favor of the status quo, Florian explained that:

| By default, DSA switch need to come up in a configuration where all
| ports (except CPU/management) must be strictly separate from every other
| port such that we can achieve what a standalone Ethernet NIC would do.
| This works because all ports are isolated from one another, so there is
| no cross talk and so having the same MAC address (the one from the CPU)
| on the DSA slave network devices just works, each port is a separate
| broadcast domain.
| 
| Once you start bridging one or ore ports, the bridge root port will have
| a MAC address, most likely the one the CPU/management Ethernet MAC, but
| similarly, this is not an issue and that's exactly how a software bridge
| would work as well.

https://patchwork.kernel.org/project/linux-arm-msm/patch/20190222125815.12866-1-vkoul@kernel.org/

Although yes, that does make some level of sense, it kind of omits the
fact that two DSA ports can be used for communication in loopback too
(either through a direct cable, or through an externally switched network).
In that case, having a MAC SA != MAC DA in the Ethernet packets is kind
of important (I found that out while trying to compose some selftests
for DSA).


If my intuition is correct, you are using the default configuration
where all DSA interfaces have the MAC address inherited from the DSA
master. Corrolary, swp2 and swp3 have the same MAC address.

swp3 is a bridged port, and a blocked port at that, but not all parts of
the network stack know that. So from time to time, you get these IPv6
Router Solicitation messages. They could be anything else, in fact.

swp2 is a bridged port, and in the forwarding state. So packets it
receives are eligible for learning.

When br0 receives a packet via swp2 that originated from swp3, it just
complains: "hey, learning the route for this packet's MAC SA to go
towards swp2 would mean that I would no longer terminate packets with
this MAC DA locally, which is kinda weird, since that MAC address is
also marked as non-forwarded." Which is fair.


So IMHO, this behavior is neither good nor bad, it is just the way it is,
nothing to worry about if that's what concerns you. To prove or disprove
what I said you could try to configure individual MAC addresses and see
whether that fixes the problem.

> (side note - if there's a place where a parser for Ocelot NPI traffic is
> hidden, that might eventually save me a lot of debugging in Lua)

Nope, there isn't, although it would certainly be great if you could
teach tcpdump about it, similar to what Vivien has done for Marvell:
https://github.com/the-tcpdump-group/tcpdump/blob/master/print-dsa.c

I've wanted to do that for a long time, but I've had lots of other
priorities, and it's tricky for various reasons (there isn't exactly a
single on-the-wire format, but it depends on whether you configure the
NPI port to have no prefix, a short prefix or a long prefix; this
configuration is independent for the RX and TX directions; currently we
use short prefix on RX and TX, but in older kernels we used to use no
prefix on TX, and long prefix on RX on some older kernels, all while the
tagging protocol was still "ocelot"; I'm not sure whether the presence
or absence of a prefix, and what kind, can be deduced by looking at the
packet alone).

Re: packets trickling out of STP-blocked ports

[Alexandre Belloni]

Hi,

On 30/12/2021 15:07:40-0800, Colin Foster wrote:
> Hi all,
> 
> I'm not sure who all to include in this email, but I'm starting with
> this list to start.
> 
> Probably obvious to those in this email list, I'm testing a VSC7512 dev
> board controlled via SPI. The patches are still out-of-tree, but I
> figured I'll report these findings, since they seem real.
> 
> My setup is port 0 of the 7512 is tied to a Beaglebone Black. Port 1 is
> tied to my development PC. Ports 2 and 3 are tied together to test STP.
> 
> I run the commands:
> 
> ip link set eth0 up
> ip link set swp[1-3] up
> ip link add name br0 type bridge stp_state 1
> ip link set dev swp[1-3] master br0
> ip addr add 10.100.3.1/16 dev br0
> ip link set dev br0 up
> 
> After running this, the STP blocks swp3, and swp1/2 are forwarding.
> 
> Periodically I see messages saying that swp2 is receiving packets with
> own address as source address.
> 
> I can confirm that via ethtool that TX packets are increasing on swp3. I
> believe I captured the event via tshark. A 4 minute capture showed three
> non-STP packets on swp2. All three of these packets are ICMPv6 Router
> Solicitation packets. 
> 
> I would expect no packets at all to egress swp3. Is this an issue that
> is unique to me and my in-development configuration? Or is this an issue
> with all Ocelot / Felix devices?
> 
> If this is an Ocelot thing, I can try to come up with a different test 
> setup to capture more data... printing the packet when it is received,
> capturing the traffic externally, capturing eth0 traffic to see if it is
> coming from the kernel or being hardware-forwarded...
> 
> (side note - if there's a place where a parser for Ocelot NPI traffic is
> hidden, that might eventually save me a lot of debugging in Lua)
> 
> 
> An idea of how frequently this happens - my system has been currently up
> for 3700 seconds. Eight "own address as source address" events have
> happened at 66, 96, 156, 279, 509, 996, 1897, and 3699 seconds. 
> 

This is something I solved back in 2017. I can exactly remember how, you
can try:

sysctl -w net.ipv6.conf.swp3.autoconf=0


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: packets trickling out of STP-blocked ports

[Colin Foster]

Hi Alexandre

On Fri, Dec 31, 2021 at 11:27:16AM +0100, Alexandre Belloni wrote:
> Hi,
> 
> On 30/12/2021 15:07:40-0800, Colin Foster wrote:
> > Hi all,
> > 
> > An idea of how frequently this happens - my system has been currently up
> > for 3700 seconds. Eight "own address as source address" events have
> > happened at 66, 96, 156, 279, 509, 996, 1897, and 3699 seconds. 
> > 
> 
> This is something I solved back in 2017. I can exactly remember how, you
> can try:
> 
> sysctl -w net.ipv6.conf.swp3.autoconf=0

That sounds very promising! Sorry you had to fix my system config, but
glad that this all makes perfect sense. 

> 
> 
> -- 
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

Re: packets trickling out of STP-blocked ports

[Alexandre Belloni]

On 31/12/2021 07:06:51-0800, Colin Foster wrote:
> Hi Alexandre
> 
> On Fri, Dec 31, 2021 at 11:27:16AM +0100, Alexandre Belloni wrote:
> > Hi,
> > 
> > On 30/12/2021 15:07:40-0800, Colin Foster wrote:
> > > Hi all,
> > > 
> > > An idea of how frequently this happens - my system has been currently up
> > > for 3700 seconds. Eight "own address as source address" events have
> > > happened at 66, 96, 156, 279, 509, 996, 1897, and 3699 seconds. 
> > > 
> > 
> > This is something I solved back in 2017. I can exactly remember how, you

Sorry, I meant "I can't exactly" ;)

> > can try:
> > 
> > sysctl -w net.ipv6.conf.swp3.autoconf=0
> 
> That sounds very promising! Sorry you had to fix my system config, but
> glad that this all makes perfect sense. 
> 

Let me know if this works ;) The bottom line being that you should
probably disable ipv6 autoconf on individual interfaces and then enable
it on the bridge.


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: packets trickling out of STP-blocked ports

[Andrew Lunn]

> > > sysctl -w net.ipv6.conf.swp3.autoconf=0
> > 
> > That sounds very promising! Sorry you had to fix my system config, but
> > glad that this all makes perfect sense. 
> > 

Hi Alexandre

> 
> Let me know if this works ;) The bottom line being that you should
> probably disable ipv6 autoconf on individual interfaces and then enable
> it on the bridge.

Does this also stop the interface getting a link local IPv6 address
based on its MAC address?

e.g. my wifi interface has MAC address b8:ae:ed:78:ef:9d and gets an
IPv6 address

inet6 fe80::baae:edff:fe78:ef9d/64 scope link 

It will also perform duplicate address detection, DAD, when the
interface is brought up. That is probably hard to see with tcpdump on
the host, since it happens very quickly, but a link peer should see
the packets.

    Andrew

Re: packets trickling out of STP-blocked ports

[Colin Foster]

On Fri, Dec 31, 2021 at 04:17:44PM +0100, Alexandre Belloni wrote:
> On 31/12/2021 07:06:51-0800, Colin Foster wrote:
> > Hi Alexandre
> > 
> > On Fri, Dec 31, 2021 at 11:27:16AM +0100, Alexandre Belloni wrote:
> > > Hi,
> > > 
> > > On 30/12/2021 15:07:40-0800, Colin Foster wrote:
> > > > Hi all,
> > > > 
> > > > An idea of how frequently this happens - my system has been currently up
> > > > for 3700 seconds. Eight "own address as source address" events have
> > > > happened at 66, 96, 156, 279, 509, 996, 1897, and 3699 seconds. 
> > > > 
> > > 
> > > This is something I solved back in 2017. I can exactly remember how, you
> 
> Sorry, I meant "I can't exactly" ;)
> 
> > > can try:
> > > 
> > > sysctl -w net.ipv6.conf.swp3.autoconf=0
> > 
> > That sounds very promising! Sorry you had to fix my system config, but
> > glad that this all makes perfect sense. 
> > 
> 
> Let me know if this works ;) The bottom line being that you should
> probably disable ipv6 autoconf on individual interfaces and then enable
> it on the bridge.

Just gave it a shot. No luck.

But poking around sysctl there's
net.ipv6.conf.swp3.router_solicitation{s,_delay,_interval,_max_interval}

As Andrew hints at, there might be some unintended consequences. It
seems that writing -1 to net.ipv6.conf.swp3.router_solicitation_delay
"fixed it." I don't know how that'll affect an IPv6 network in
production.

> 
> 
> -- 
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

Re: packets trickling out of STP-blocked ports

[Vladimir Oltean]

On Fri, Dec 31, 2021 at 02:28:23AM +0200, Vladimir Oltean wrote:
> Traditionally, DSA has made a design decision that all switch ports
> inherit the single MAC address of the DSA master. IOW, if you have 1 DSA
> master and 4 switch ports, you have 5 interfaces in the system with the
> same MAC address. It was like this for a long time, and relatively
> recently, Xiaofei Shen added the ability for individual DSA interfaces
> to have their own MAC address stored in the device tree.

I thought a bit more in the back of my head and I need to make a
correction to what I said. It doesn't matter that DSA interfaces have
the same MAC address, because swp2 and swp3 are both bridge ports, and
therefore, their MAC addresses are both local FDB entries, even if
unique. So the bridge would still complain that it receives packets with
a MAC SA equal to a local FDB entry.

ip link add veth0 type veth peer name veth1
ip link add br0 type bridge
ip link set veth0 master br0
[   84.987666] br0: port 1(veth0) entered blocking state
[   84.992857] br0: port 1(veth0) entered disabled state
[   84.998172] device veth0 entered promiscuous mode
ip link set veth1 master br0
[   87.083140] br0: port 2(veth1) entered blocking state
[   87.088280] br0: port 2(veth1) entered disabled state
[   87.093625] device veth1 entered promiscuous mode
ip link set br0 type bridge stp_state 1
ip link set br0 up
ip link set veth0 up
ip link set veth1 up
[  116.758260] IPv6: ADDRCONF(NETDEV_CHANGE): veth1: link becomes ready
[  116.764899] br0: port 2(veth1) entered blocking state
[  116.771353] br0: port 2(veth1) entered listening state
[  116.778272] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[  116.785703] br0: port 1(veth0) entered blocking state
[  116.790776] br0: port 1(veth0) entered listening state
[  117.112892] br0: port 2(veth1) entered blocking state
[  132.312686] br0: port 1(veth0) entered learning state
[  133.740183] br0: received packet on veth0 with own address as source address (addr:c2:fc:33:30:4c:bf, vlan:0)
[  145.752889] br0: received packet on veth0 with own address as source address (addr:c2:fc:33:30:4c:bf, vlan:0)
[  147.672675] br0: port 1(veth0) entered forwarding state
[  147.677978] br0: topology change detected, propagating
[  147.683388] IPv6: ADDRCONF(NETDEV_CHANGE): br0: link becomes ready
[  149.741219] br0: received packet on veth0 with own address as source address (addr:c2:fc:33:30:4c:bf, vlan:0)
[  176.472805] br0: received packet on veth0 with own address as source address (addr:c2:fc:33:30:4c:bf, vlan:0)
[  181.742095] br0: received packet on veth0 with own address as source address (addr:c2:fc:33:30:4c:bf, vlan:0)
[  238.552814] br0: received packet on veth0 with own address as source address (addr:c2:fc:33:30:4c:bf, vlan:0)
[  245.742659] br0: received packet on veth0 with own address as source address (addr:c2:fc:33:30:4c:bf, vlan:0)
bridge link
6: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state blocking priority 32 cost 2
7: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 2

Looking at br_fdb_update(), the print is pretty harmless - the bridge is
smart enough to not actually relearn the local FDB entry towards an
external port. No FDB update is being done. The print is also
rate-limited. So it's just that - a warning. I am not sure whether it's
worth disabling IPv6 Router Solicitations, given that, as mentioned,
basically any other traffic sent through the plain bridge port will
trigger this. I consider it a non-problem.

ip addr add 192.168.100.1/24 dev veth1
ping 192.168.100.2
PING 192.168.100.[ 1434.033119] br0: received packet on veth0 with own address as source address (addr:c2:fc:33:30:4c:bf, vlan:0)
2 (192.168.100.2) 56(84) bytes of data.
[ 1435.112977] br0: received packet on veth0 with own address as source address (addr:c2:fc:33:30:4c:bf, vlan:0)
[ 1436.152991] br0: received packet on veth0 with own address as source address (addr:c2:fc:33:30:4c:bf, vlan:0)
[ 1437.193428] br0: received packet on veth0 with own address as source address (addr:c2:fc:33:30:4c:bf, vlan:0)
From 192.168.100.1 icmp_seq=1 Destination Host Unreachable
From 192.168.100.1 icmp_seq=2 Destination Host Unreachable
From 192.168.100.1 icmp_seq=3 Destination Host Unreachable
[ 1438.232784] br0: received packet on veth0 with own address as source address (addr:c2:fc:33:30:4c:bf, vlan:0)
[ 1438.260075] br0: received packet on veth0 with own address as source address (addr:c2:fc:33:30:4c:bf, vlan:0)
[ 1439.272769] br0: received packet on veth0 with own address as source address (addr:c2:fc:33:30:4c:bf, vlan:0)
[ 1440.312904] br0: received packet on veth0 with own address as source address (addr:c2:fc:33:30:4c:bf, vlan:0)
From 192.168.100.1 icmp_seq=4 Destination Host Unreachable
^C
--- 192.168.100.2 ping statistics ---
7 packets transmitted, 0 received, +4 errors, 100% packet loss, time 6280ms
pipe 4