mv88e6321, dual cpu port

mv88e6321, dual cpu port

[Angelo Dureghello]

Hi All,

using kernel 5.4.70,
just looking for confirmation this layout
can work:

    eth0 -> cpu port (port5, mii)  bridging port3 and 4
    eth1 -> cpu port (port6, rgmii)  bridging port0, 1, 2

My devicetree actaully defines 2 cpu ports, it seems
to work, but please let me know if you see any
possible issue.

Regards,
Angelo Dureghello

Re: mv88e6321, dual cpu port

[Andrew Lunn]

On Mon, Jan 09, 2023 at 09:40:06PM +0100, Angelo Dureghello wrote:
> Hi All,
> 
> using kernel 5.4.70,
> just looking for confirmation this layout
> can work:
> 
>    eth0 -> cpu port (port5, mii)  bridging port3 and 4
>    eth1 -> cpu port (port6, rgmii)  bridging port0, 1, 2
> 
> My devicetree actaully defines 2 cpu ports, it seems
> to work, but please let me know if you see any
> possible issue.

Dual CPU ports is not supported with 5.4.70. Everything will go over
the first cpu port in DT.

    Andrew

Re: mv88e6321, dual cpu port

[Angelo Dureghello]

Hi Andrew,

thanks a lot for the prompt reply,

actually, seems i can use both cpu ports by
some brctl lines:

ip link add name br0 type bridge
ip link add name br1 type bridge
brctl addif br0 eth0
brctl addif br1 eth1

I verified looking the tx_packets statistics (ethtool -S)
and both eth0 and eth1 ports are used.

Is it something that may work, eventually as a workaround ?

Also, could you help me to know the kernel version
implementing dual cpu port properly ?

I cannot actually upgrade the kernel, due to cpu producer
customizations that are not mainlined, so would try to
downgrade the driver.

Thanks a lot,
regards,
angelo

On Mon, 9 Jan 2023, Andrew Lunn wrote:

> On Mon, Jan 09, 2023 at 09:40:06PM +0100, Angelo Dureghello wrote:
>> Hi All,
>>
>> using kernel 5.4.70,
>> just looking for confirmation this layout
>> can work:
>>
>>    eth0 -> cpu port (port5, mii)  bridging port3 and 4
>>    eth1 -> cpu port (port6, rgmii)  bridging port0, 1, 2
>>
>> My devicetree actaully defines 2 cpu ports, it seems
>> to work, but please let me know if you see any
>> possible issue.
>
> Dual CPU ports is not supported with 5.4.70. Everything will go over
> the first cpu port in DT.
>
>    Andrew
>

Re: mv88e6321, dual cpu port

[Vladimir Oltean]

Hi Angelo,

On Tue, Jan 10, 2023 at 11:23:58AM +0100, Angelo Dureghello wrote:
> Hi Andrew,
> 
> thanks a lot for the prompt reply,
> 
> actually, seems i can use both cpu ports by
> some brctl lines:
> 
> ip link add name br0 type bridge
> ip link add name br1 type bridge
> brctl addif br0 eth0
> brctl addif br1 eth1
> 
> I verified looking the tx_packets statistics (ethtool -S)
> and both eth0 and eth1 ports are used.
> 
> Is it something that may work, eventually as a workaround ?

I am afraid that these are DSA extensions added by your kernel vendor.
I would be curious to look at the kernel code, for my amusement, if you
can share it. Generally, DSA masters (eth0, eth1 in this case) cannot be
added to a bridge, because the bridge's RX handler steals packets from
the DSA data path. So I'd also like to see what was done there.

> Also, could you help me to know the kernel version
> implementing dual cpu port properly ?

I would recommend starting with the DSA documentation on this topic:
https://www.kernel.org/doc/html/latest/networking/dsa/configuration.html#affinity-of-user-ports-to-cpu-ports
then with "git blame" on the documentation file, and "git log" to see
the surrounding commits.

For mv88e6xxx, the API for multiple CPU ports is not yet implemented.
On the other hand, there have been proposed RFC patch sets from Marek
Behun and Christian Marangi which can be picked up and used as a base
(they need to be adapted to the API that was merged).

> I cannot actually upgrade the kernel, due to cpu producer
> customizations that are not mainlined, so would try to
> downgrade the driver.

The delta between v5.4 and the kernel where support for multiple CPU
ports was added is ~600 patches to the net/dsa/ folder alone. There are
some non-trivial interdependencies with phylib, phylink, devlink, switchdev.
You might also need support for the end result, if you end up cherry picking
only what you think is useful. Hopefully that will help you reconsider.

Re: mv88e6321, dual cpu port

[Andrew Lunn]

> > I cannot actually upgrade the kernel, due to cpu producer
> > customizations that are not mainlined, so would try to
> > downgrade the driver.
> 
> The delta between v5.4 and the kernel where support for multiple CPU
> ports was added is ~600 patches to the net/dsa/ folder alone. There are
> some non-trivial interdependencies with phylib, phylink, devlink, switchdev.
> You might also need support for the end result, if you end up cherry picking
> only what you think is useful. Hopefully that will help you reconsider.

v5.4 is really old, v5.10, v5.15 and v6.1 are also LTS kernels. I
suggest you ask your CPU vendor for a v6.1, or v5.15 port of their
vendor tree. v5.4 has a projected End Of Life December 2025, so if you
are going to invest a few months effort trying to get this working, do
you have enough time left to get some return on your investment? And a
plan what to do once it is EOL?

       Andrew

Re: mv88e6321, dual cpu port

[Angelo Dureghello]

Hi Andrew and Vladimir,

thanks for the kind support.

On Wed, 11 Jan 2023, Andrew Lunn wrote:

>>> I cannot actually upgrade the kernel, due to cpu producer
>>> customizations that are not mainlined, so would try to
>>> downgrade the driver.
>>
>> The delta between v5.4 and the kernel where support for multiple CPU
>> ports was added is ~600 patches to the net/dsa/ folder alone. There are
>> some non-trivial interdependencies with phylib, phylink, devlink, switchdev.
>> You might also need support for the end result, if you end up cherry picking
>> only what you think is useful. Hopefully that will help you reconsider.
>
> v5.4 is really old, v5.10, v5.15 and v6.1 are also LTS kernels. I
> suggest you ask your CPU vendor for a v6.1, or v5.15 port of their
> vendor tree. v5.4 has a projected End Of Life December 2025, so if you
> are going to invest a few months effort trying to get this working, do
> you have enough time left to get some return on your investment? And a
> plan what to do once it is EOL?
>

I am now stuck to 5.4.70 due to freescale stuff in it.
I tried shortly a downgrade of net/dsa part from 5.5,
but, even if i have no errors, i can't ping out, something
seems has gone wrong, i don't like too much this approach.

But of course, i could upgrade to 5.10 that's available
from freescale, with some effort.
Then, i still should apply a patch on the driver to have dual
cpu running, if i understand properly, correct ?

I am now trying this way on mv88e6321,
- one vlan using dsa kernel driver,
- other vlan using dsdt userspace driver.

Do you think i may succeed, or better avoid any attempt,
and go for kernel upgrade ?

>       Andrew
>

Thanks a lot,
angelo

Re: mv88e6321, dual cpu port

[Vladimir Oltean]

On Mon, Jan 23, 2023 at 09:52:15AM +0100, Angelo Dureghello wrote:
> I am now stuck to 5.4.70 due to freescale stuff in it.
> I tried shortly a downgrade of net/dsa part from 5.5,
> but, even if i have no errors, i can't ping out, something
> seems has gone wrong, i don't like too much this approach.
> 
> But of course, i could upgrade to 5.10 that's available
> from freescale, with some effort.
> Then, i still should apply a patch on the driver to have dual
> cpu running, if i understand properly, correct ?

What is the Freescale chip stuck on 5.4 (5.10 with some effort) if I may
ask? I got the impression that the vast majority of SoCs have good
enough mainline support to boot a development kernel and work on that,
the exception being some S32 platforms.

Re: mv88e6321, dual cpu port

[Angelo Dureghello]

Hi Vladimir,

On Mon, 23 Jan 2023, Vladimir Oltean wrote:

> On Mon, Jan 23, 2023 at 09:52:15AM +0100, Angelo Dureghello wrote:
>> I am now stuck to 5.4.70 due to freescale stuff in it.
>> I tried shortly a downgrade of net/dsa part from 5.5,
>> but, even if i have no errors, i can't ping out, something
>> seems has gone wrong, i don't like too much this approach.
>>
>> But of course, i could upgrade to 5.10 that's available
>> from freescale, with some effort.
>> Then, i still should apply a patch on the driver to have dual
>> cpu running, if i understand properly, correct ?
>
> What is the Freescale chip stuck on 5.4 (5.10 with some effort) if I may
> ask? I got the impression that the vast majority of SoCs have good
> enough mainline support to boot a development kernel and work on that,
> the exception being some S32 platforms.
>

I think i tested mainline kernel, but the rpmsg part to
communicate with M4 core and scu firmware seems not there.
So i am stuck with freescale stuff now. Anyway, freescale
released their 5.10 branch, so eventually i'll move
to that.

Regards,
angelo

Re: mv88e6321, dual cpu port

[Vladimir Oltean]

On Mon, Jan 23, 2023 at 02:26:55PM +0100, Angelo Dureghello wrote:
> Hi Vladimir,
> 
> On Mon, 23 Jan 2023, Vladimir Oltean wrote:
> 
> > On Mon, Jan 23, 2023 at 09:52:15AM +0100, Angelo Dureghello wrote:
> > > I am now stuck to 5.4.70 due to freescale stuff in it.
> > > I tried shortly a downgrade of net/dsa part from 5.5,
> > > but, even if i have no errors, i can't ping out, something
> > > seems has gone wrong, i don't like too much this approach.
> > > 
> > > But of course, i could upgrade to 5.10 that's available
> > > from freescale, with some effort.
> > > Then, i still should apply a patch on the driver to have dual
> > > cpu running, if i understand properly, correct ?
> > 
> > What is the Freescale chip stuck on 5.4 (5.10 with some effort) if I may
> > ask? I got the impression that the vast majority of SoCs have good
> > enough mainline support to boot a development kernel and work on that,
> > the exception being some S32 platforms.
> > 
> 
> I think i tested mainline kernel, but the rpmsg part to
> communicate with M4 core and scu firmware seems not there.
> So i am stuck with freescale stuff now. Anyway, freescale
> released their 5.10 branch, so eventually i'll move
> to that.

Ok, SC firmware isn't something that I'd be able to assist with.
Isn't there a 5.15 kernel currently available too?
https://github.com/nxp-imx/linux-imx/tree/lf-5.15.y
And I guess a kernel based on 6.1 should become publicly available
soon too (your FAE should probably be able to say more).

The point is that multi-CPU-port support for mv88e6xxx isn't present in
the mainline kernel, no matter what kernel version we're talking about.
And even if you plan on adding this support yourself, doing so on a very
old kernel is a waste of time, because you won't be able to backport DSA
in a way in which the backported code base is similar enough to what's
upstream (and not to mention, behaves similarly enough).

I don't know what this means:

| I am now trying this way on mv88e6321,
| - one vlan using dsa kernel driver,
| - other vlan using dsdt userspace driver.

specifically what is "dsdt userspace driver".

Re: mv88e6321, dual cpu port

[Andrew Lunn]

> I don't know what this means:
> 
> | I am now trying this way on mv88e6321,
> | - one vlan using dsa kernel driver,
> | - other vlan using dsdt userspace driver.
> 
> specifically what is "dsdt userspace driver".

I think DSDT is Marvells vendor crap code.

Having two drivers for the same hardware is a recipe for disaster.

  Andrew

Re: mv88e6321, dual cpu port

[Angelo Dureghello]

Hi Andrew and Vladimir,

On Mon, 23 Jan 2023, Andrew Lunn wrote:

>> I don't know what this means:
>>
>> | I am now trying this way on mv88e6321,
>> | - one vlan using dsa kernel driver,
>> | - other vlan using dsdt userspace driver.
>>
>> specifically what is "dsdt userspace driver".
>
> I think DSDT is Marvells vendor crap code.
>
Yes, i have seen someone succeeding using it, why do you think it's 
crap ?

> Having two drivers for the same hardware is a recipe for disaster.
>
>  Andrew
>

What i need is something as

         eth0 ->  vlan1 -> port5(rmii)  ->  port 0,1,2
         eth1 ->  vlan2 -> port6(rgmii) ->  port 3,4

The custom board i have here is already designed in this way
(2 fixed-link mac-to-mac connecitons) and trying my best to have
the above layout working.

If you have any suggestion on how to proceed, really appreciated.

Thanks,
angelo

Re: mv88e6321, dual cpu port

[Vladimir Oltean]

On Tue, Jan 24, 2023 at 08:21:35AM +0100, Angelo Dureghello wrote:
> What i need is something as
> 
>         eth0 ->  vlan1 -> port5(rmii)  ->  port 0,1,2
>         eth1 ->  vlan2 -> port6(rgmii) ->  port 3,4
> 
> The custom board i have here is already designed in this way
> (2 fixed-link mac-to-mac connecitons) and trying my best to have
> the above layout working.
> 
> If you have any suggestion on how to proceed, really appreciated.

The way this would have traditionally worked prior to having UAPI for
fine-grained user port to CPU port assignment was to declare in the
device tree that port6(rgmii) is a user port. This makes eth1 see
DSA-untagged traffic. Then you put port6 in a bridge with the other
ports, and isolate just ports 6, 3 and 4 in the VLAN that you desire,
and eth1's packets will be forwarded just to ports 3 and 4. Control
packts (STP, PTP) received over ports 3 and 4 are still trapped to the
CPU port (port5).

For some use cases, on LS1028A we still need to declare one of the CPU
ports as a user port. To achieve that, we take the existing upstream
device tree (which declares both CPU ports as CPU ports) and remove the
"ethernet" property from one of them, dynamically, from the U-Boot shell:

=> tftp $fdt_addr_r fsl-ls1028a-rdb.dtb
=> fdt addr $fdt_addr_r
=> fdt rm /soc/pcie@1f0000000/ethernet-switch@0,5/ports/port@4 ethernet
=> tftp $kernel_addr_r uImage
=> bootm $kernel_addr_r - $fdt_addr_r

Re: mv88e6321, dual cpu port

[Andrew Lunn]

On Tue, Jan 24, 2023 at 08:21:35AM +0100, Angelo Dureghello wrote:
> 
> Hi Andrew and Vladimir,
> 
> On Mon, 23 Jan 2023, Andrew Lunn wrote:
> 
> > > I don't know what this means:
> > > 
> > > | I am now trying this way on mv88e6321,
> > > | - one vlan using dsa kernel driver,
> > > | - other vlan using dsdt userspace driver.
> > > 
> > > specifically what is "dsdt userspace driver".
> > 
> > I think DSDT is Marvells vendor crap code.
> > 
> Yes, i have seen someone succeeding using it, why do you think it's crap ?

In the Linux kernel community, that is the name given to vendor code,
because in general, that is the quality level. The quality does vary
from vendor to vendor and SDK to SDK, some are actually O.K.

> 
> > Having two drivers for the same hardware is a recipe for disaster.
> > 
> >  Andrew
> > 
> 
> What i need is something as
> 
>         eth0 ->  vlan1 -> port5(rmii)  ->  port 0,1,2
>         eth1 ->  vlan2 -> port6(rgmii) ->  port 3,4
> 
> The custom board i have here is already designed in this way
> (2 fixed-link mac-to-mac connecitons) and trying my best to have
> the above layout working.

With todays mainline i would do:

So set eth0 as DSA master port.

Create a bridge br0 with ports 0, 1, 2.
Create a bridge br1 with ports 3, 4, 6.

You don't actually make use of the br1 interface in Linux, it just
needs to be up. You can think of eth1 being connected to an external
managed switch.

	Andrew

Re: mv88e6321, dual cpu port

[Angelo Dureghello]

Hi Andrew and Vladimir,

thanks a lot for both your suggestions, will study and reason
on them, and likely try them.

On Tue, 24 Jan 2023, Andrew Lunn wrote:

> On Tue, Jan 24, 2023 at 08:21:35AM +0100, Angelo Dureghello wrote:
>>
>> Hi Andrew and Vladimir,
>>
>> On Mon, 23 Jan 2023, Andrew Lunn wrote:
>>
>>>> I don't know what this means:
>>>>
>>>> | I am now trying this way on mv88e6321,
>>>> | - one vlan using dsa kernel driver,
>>>> | - other vlan using dsdt userspace driver.
>>>>
>>>> specifically what is "dsdt userspace driver".
>>>
>>> I think DSDT is Marvells vendor crap code.
>>>
>> Yes, i have seen someone succeeding using it, why do you think it's crap ?
>
> In the Linux kernel community, that is the name given to vendor code,
> because in general, that is the quality level. The quality does vary
> from vendor to vendor and SDK to SDK, some are actually O.K.
>
>>
>>> Having two drivers for the same hardware is a recipe for disaster.
>>>
>>>  Andrew
>>>
>>
>> What i need is something as
>>
>>         eth0 ->  vlan1 -> port5(rmii)  ->  port 0,1,2
>>         eth1 ->  vlan2 -> port6(rgmii) ->  port 3,4
>>
>> The custom board i have here is already designed in this way
>> (2 fixed-link mac-to-mac connecitons) and trying my best to have
>> the above layout working.
>
> With todays mainline i would do:
>
> So set eth0 as DSA master port.
>
> Create a bridge br0 with ports 0, 1, 2.
> Create a bridge br1 with ports 3, 4, 6.
>
> You don't actually make use of the br1 interface in Linux, it just
> needs to be up. You can think of eth1 being connected to an external
> managed switch.
>
> 	Andrew
>

Thanks,
angelo

Re: mv88e6321, dual cpu port

[Angelo Dureghello]

Hi Andrew,

still thanks a lot,

On 24/01/23 14:57, Andrew Lunn wrote:
> On Tue, Jan 24, 2023 at 08:21:35AM +0100, Angelo Dureghello wrote:
>>
>> Hi Andrew and Vladimir,
>>
>> On Mon, 23 Jan 2023, Andrew Lunn wrote:
>>
>>>> I don't know what this means:
>>>>
>>>> | I am now trying this way on mv88e6321,
>>>> | - one vlan using dsa kernel driver,
>>>> | - other vlan using dsdt userspace driver.
>>>>
>>>> specifically what is "dsdt userspace driver".
>>>
>>> I think DSDT is Marvells vendor crap code.
>>>
>> Yes, i have seen someone succeeding using it, why do you think it's crap ?
> 
> In the Linux kernel community, that is the name given to vendor code,
> because in general, that is the quality level. The quality does vary
> from vendor to vendor and SDK to SDK, some are actually O.K.
> 
>>
>>> Having two drivers for the same hardware is a recipe for disaster.
>>>
>>>   Andrew
>>>
>>
>> What i need is something as
>>
>>          eth0 ->  vlan1 -> port5(rmii)  ->  port 0,1,2
>>          eth1 ->  vlan2 -> port6(rgmii) ->  port 3,4
>>
>> The custom board i have here is already designed in this way
>> (2 fixed-link mac-to-mac connecitons) and trying my best to have
>> the above layout working.
> 
> With todays mainline i would do:
> 
> So set eth0 as DSA master port.
> 
> Create a bridge br0 with ports 0, 1, 2.
> Create a bridge br1 with ports 3, 4, 6.
> 

This is what i am testing now, a bit different,
swapped ports 5 and 6.

#
# Configuration:
#                                   cpu      +---- port0
#              br0 eth0  <-> rgmii  port 6  -+---- port1
#                                            |
#                                            +---- port2
#
#                                  user      +---- port3
#              br1 eth1  <-> rmii  port 5   -+-----port4
#
#

mdio {
		#address-cells = <1>;
		#size-cells = <0>;

		switch1: switch1@1d {
			compatible = "marvell,mv88e6085";
			reg = <0x1d>;
			interrupt-parent = <&lsio_gpio3>;
			interrupts = <4 IRQ_TYPE_LEVEL_LOW>;
			interrupt-controller;
			#interrupt-cells = <2>;

			ports {
				#address-cells = <1>;
				#size-cells = <0>;

				port@0 {
					reg = <0>;
					label = "port0";
					phy-mode = "1000base-x";
					managed = "in-band-status";
					sfp = <&sfp_0>;
				};
				port@1 {
					reg = <1>;
					label = "port1";
					phy-mode = "1000base-x";
					managed = "in-band-status";
					sfp = <&sfp_1>;
				};
				/* This is phyenet0 now */
				port@2 {
					reg = <2>;
					label = "port2";
					phy-handle = <&switchphy2>;
				};
				port@6 {
					/* wired to cpu fec1 */
					reg = <6>;
					label = "cpu";
					ethernet = <&fec1>;
					fixed-link = <0 1 1000 0 0>;
				};
				port@3 {
					/* phy is internal to the switch */
					reg = <3>;
					label = "port3";
					phy-handle = <&switchphy3>;
				};
				port@4 {
					/* phy is internal to the switch */
					reg = <4>;
					label = "port4";
					phy-handle = <&switchphy4>;
				};
				port@5 {
					/* wired to cpu fec2 */
					reg = <5>;
					label = "port5";
					ethernet = <&fec2>;
					fixed-link = <1 1 100 0 0>;
				};
			};

All seems to work properly, but on ports 0, 1, 2 i cannot go
over 100Mbit even if master port (6) is rgmii
(testing by iperf3).

What could the reason of this limitation ?

> You don't actually make use of the br1 interface in Linux, it just
> needs to be up. You can think of eth1 being connected to an external
> managed switch.
> 
> 	Andrew

Thanks a lot,
angelo

Re: mv88e6321, dual cpu port

[Andrew Lunn]

> This is what i am testing now, a bit different,
> swapped ports 5 and 6.
> 
> #
> # Configuration:
> #                                   cpu      +---- port0
> #              br0 eth0  <-> rgmii  port 6  -+---- port1
> #                                            |
> #                                            +---- port2
> #
> #                                  user      +---- port3
> #              br1 eth1  <-> rmii  port 5   -+-----port4
> #
> #
> 
> mdio {
> 		#address-cells = <1>;
> 		#size-cells = <0>;
> 
> 		switch1: switch1@1d {
> 			compatible = "marvell,mv88e6085";
> 			reg = <0x1d>;
> 			interrupt-parent = <&lsio_gpio3>;
> 			interrupts = <4 IRQ_TYPE_LEVEL_LOW>;
> 			interrupt-controller;
> 			#interrupt-cells = <2>;
> 
> 			ports {
> 				#address-cells = <1>;
> 				#size-cells = <0>;
> 
> 				port@0 {
> 					reg = <0>;
> 					label = "port0";
> 					phy-mode = "1000base-x";
> 					managed = "in-band-status";
> 					sfp = <&sfp_0>;
> 				};
> 				port@1 {
> 					reg = <1>;
> 					label = "port1";
> 					phy-mode = "1000base-x";
> 					managed = "in-band-status";
> 					sfp = <&sfp_1>;
> 				};
> 				/* This is phyenet0 now */
> 				port@2 {
> 					reg = <2>;
> 					label = "port2";
> 					phy-handle = <&switchphy2>;
> 				};
> 				port@6 {
> 					/* wired to cpu fec1 */
> 					reg = <6>;
> 					label = "cpu";
> 					ethernet = <&fec1>;
> 					fixed-link = <0 1 1000 0 0>;

This is the deprecated way to do fixed link. Use

                fixed-link {
                        speed = <1000>;
                        full-duplex;
                };


> 				};
> 				port@3 {
> 					/* phy is internal to the switch */
> 					reg = <3>;
> 					label = "port3";
> 					phy-handle = <&switchphy3>;
> 				};
> 				port@4 {
> 					/* phy is internal to the switch */
> 					reg = <4>;
> 					label = "port4";
> 					phy-handle = <&switchphy4>;
> 				};
> 				port@5 {
> 					/* wired to cpu fec2 */
> 					reg = <5>;
> 					label = "port5";
> 					ethernet = <&fec2>;

This is wrong. As far as the switch is concerned, this port is nothing
special. It is just a regular user port. So it should not have an
ethernet property.



> 					fixed-link = <1 1 100 0 0>;
> 				};
> 			};
> 
> All seems to work properly, but on ports 0, 1, 2 i cannot go
> over 100Mbit even if master port (6) is rgmii
> (testing by iperf3).

What SoC is this? Some FECs are only Fast ethernet.

What does ethtool show for eth0?

Do you also have a fixed link in the fec node?

   Andrew

Re: mv88e6321, dual cpu port

[Angelo Dureghello]

Hi Andrew,

On 24/01/23 2:57 PM, Andrew Lunn wrote:

 > With todays mainline i would do:
 >
 > So set eth0 as DSA master port.
 >
 > Create a bridge br0 with ports 0, 1, 2.
 > Create a bridge br1 with ports 3, 4, 6.
 >
 > You don't actually make use of the br1 interface in Linux, it just
 > needs to be up. You can think of eth1 being connected to an external
 > managed switch.
 >
 > 	Andrew

i upgraded to kernel 5.15.32, tried your option above.

In my initial request i inverted port 5
and 6 but i think this shouldn't matter.

Still data passes all trough port6, even when i ping from
host PC to port4. I was expecting instead to see port5
statistics increasing.

This is the script and the scheme i need:

#!/bin/sh
#
# Configuration:
#                                       +---- port0
#              br0 eth0  <->   port 6  -+---- port1
#                                       +---- port2
#
#                                       +---- port3
#              br1 eth1  <-> --------- -+-----port4
#                                       +---- port5
#
# tested, port4 ping, data passes always from port 6
#

ip link set eth0 up
ip link set eth1 up

# bring up the slave interfaces
ip link set port0 up
ip link set port1 up
ip link set port2 up
ip link set port3 up
ip link set port4 up
ip link set port5 up

# create bridge
ip link add name br0 type bridge
ip link add name br1 type bridge

# add ports to bridge
ip link set dev port0 master br0
ip link set dev port1 master br0
ip link set dev port2 master br0

ip link set dev port3 master br1
ip link set dev port4 master br1
ip link set dev port5 master br1

# configure the bridge
ip addr add 192.0.2.1/25 dev br0
ip addr add 192.0.2.129/25 dev br1

# bring up the bridge
ip link set dev br0 up
ip link set dev br1 up

And device tree:

&fec1 {
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_fec1>;
	phy-mode = "rgmii";
	/* fsl,magic-packet; */
	tx-internal-delay-ps = <2000>;
	rx-internal-delay-ps = <2000>;

	status = "okay";

	fixed-link {
		speed = <1000>;
		full-duplex;
	};

	mdio {
		#address-cells = <1>;
		#size-cells = <0>;

		switch1: switch1@1d {
			compatible = "marvell,mv88e6085";
			reg = <0x1d>;
			interrupt-parent = <&lsio_gpio3>;
			interrupts = <4 IRQ_TYPE_LEVEL_LOW>;
			interrupt-controller;
			#interrupt-cells = <2>;

			ports {
				#address-cells = <1>;
				#size-cells = <0>;

				port@0 {
					reg = <0>;
					label = "port0";
					phy-mode = "1000base-x";
					managed = "in-band-status";
					sfp = <&sfp_0>;
				};
				port@1 {
					reg = <1>;
					label = "port1";
					phy-mode = "1000base-x";
					managed = "in-band-status";
					sfp = <&sfp_1>;
				};
				/* This is phyenet0 now */
				port@2 {
					reg = <2>;
					label = "port2";
					phy-handle = <&switchphy2>;
				};
				port@6 {
					/* wired to cpu fec1 */
					reg = <6>;
					label = "cpu";
					ethernet = <&fec1>;
					phy-mode = "rgmii";
					fixed-link {
						speed = <1000>;
						full-duplex;
					};
				};
				port@3 {
					/* phy is internal to the switch */
					reg = <3>;
					label = "port3";
					phy-handle = <&switchphy3>;
				};
				port@4 {
					/* phy is internal to the switch */
					reg = <4>;
					label = "port4";
					phy-handle = <&switchphy4>;
				};
				port@5 {
					/* wired to cpu fec2 */
					reg = <5>;
					label = "port5";
					phy-mode = "rmii";
					fixed-link {
						speed = <100>;
						full-duplex;
					};
				};
			};

			mdio {
				#address-cells = <1>;
				#size-cells = <0>;

				switchphy2: switchphy@2 {
					reg = <0x2>;
				};

				switchphy3: switchphy@3 {
					reg = <0x3>;
				};

				switchphy4: switchphy@4 {
					reg = <0x4>;
				};
			};
		};
	};
};

In any hint, welcome,

Thanks a lot,
-- 
Angelo Dureghello
+++ kernelspace +++
+E: angelo AT kernel-space.org
+W: www.kernel-space.org

Re: mv88e6321, dual cpu port

[Vladimir Oltean]

On Thu, Feb 16, 2023 at 12:20:24PM +0100, Angelo Dureghello wrote:
> Still data passes all trough port6, even when i ping from
> host PC to port4. I was expecting instead to see port5
> statistics increasing.

> # configure the bridge
> ip addr add 192.0.2.1/25 dev br0
> ip addr add 192.0.2.129/25 dev br1

In this configuration you're supposed to put an IP address on the fec2
interface (eth1), not on br1.

br1 will handle offloaded forwarding between port5 and the external
ports (port3, port4). It doesn't need an IP address. In fact, if you
give it an IP address, you will make the sent packets go through the br1
interface, which does dev_queue_xmit() to the bridge ports (port3, port4,
port5), ports which are DSA, so they do dev_queue_xmit() through their
DSA master - eth0. So the system behaves as instructed.

Re: mv88e6321, dual cpu port

[Andrew Lunn]

On Thu, Feb 16, 2023 at 02:50:40PM +0200, Vladimir Oltean wrote:
> On Thu, Feb 16, 2023 at 12:20:24PM +0100, Angelo Dureghello wrote:
> > Still data passes all trough port6, even when i ping from
> > host PC to port4. I was expecting instead to see port5
> > statistics increasing.
> 
> > # configure the bridge
> > ip addr add 192.0.2.1/25 dev br0
> > ip addr add 192.0.2.129/25 dev br1
> 
> In this configuration you're supposed to put an IP address on the fec2
> interface (eth1), not on br1.
> 
> br1 will handle offloaded forwarding between port5 and the external
> ports (port3, port4). It doesn't need an IP address. In fact, if you
> give it an IP address, you will make the sent packets go through the br1
> interface, which does dev_queue_xmit() to the bridge ports (port3, port4,
> port5), ports which are DSA, so they do dev_queue_xmit() through their
> DSA master - eth0. So the system behaves as instructed.

Yep. As i said in another email, consider eth1 being connected to an
external managed switch. br1 is how you manage that switch, but that
is all you use br1 for. eth1 is where you do networking.

   Andrew

Re: mv88e6321, dual cpu port

[Angelo Dureghello]

Hi Andrew and Vladimir,

thanks, it works.

Attaching the final script (dts as avobe) if you want to
double check, and in case of any interest.

On 16/02/23 1:50 PM, Vladimir Oltean wrote:
 > On Thu, Feb 16, 2023 at 12:20:24PM +0100, Angelo Dureghello wrote:
 >> Still data passes all trough port6, even when i ping from
 >> host PC to port4. I was expecting instead to see port5
 >> statistics increasing.
 >
 >> # configure the bridge
 >> ip addr add 192.0.2.1/25 dev br0
 >> ip addr add 192.0.2.129/25 dev br1
 >
 > In this configuration you're supposed to put an IP address on the fec2
 > interface (eth1), not on br1.
 >
 > br1 will handle offloaded forwarding between port5 and the external
 > ports (port3, port4). It doesn't need an IP address. In fact, if you
 > give it an IP address, you will make the sent packets go through the br1
 > interface, which does dev_queue_xmit() to the bridge ports (port3, port4,
 > port5), ports which are DSA, so they do dev_queue_xmit() through their
 > DSA master - eth0. So the system behaves as instructed.
#!/bin/sh
#
# Configuration:
#                                       +---- port0
#              br0 eth0  <->   port 6  -+---- port1
#                                       +---- port2
#
#                                       +---- port3
#              br1 eth1  <-> --------- -+-----port4
#                                       +---- port5
#
# tested, port4 ping, data passes always from port 6
#

ip link set eth0 up
ip link set eth1 up

# bring up the slave interfaces
ip link set port0 up
ip link set port1 up
ip link set port2 up
ip link set port3 up
ip link set port4 up
ip link set port5 up

# create bridge
ip link add name br0 type bridge
ip link add name br1 type bridge

# add ports to bridge
ip link set dev port0 master br0
ip link set dev port1 master br0
ip link set dev port2 master br0

ip link set dev port3 master br1
ip link set dev port4 master br1
ip link set dev port5 master br1

# configure the bridge
ip addr add 192.0.2.1/25 dev br0
ip addr add 192.0.2.129/25 dev eth1

# bring up the bridge
ip link set dev br0 up
ip link set dev br1 up

Many thanks !
-- 
Angelo Dureghello

Re: mv88e6321, dual cpu port

[Vladimir Oltean]

On Thu, Feb 16, 2023 at 01:59:22PM +0100, Andrew Lunn wrote:
> On Thu, Feb 16, 2023 at 02:50:40PM +0200, Vladimir Oltean wrote:
> > On Thu, Feb 16, 2023 at 12:20:24PM +0100, Angelo Dureghello wrote:
> > > Still data passes all trough port6, even when i ping from
> > > host PC to port4. I was expecting instead to see port5
> > > statistics increasing.
> > 
> > > # configure the bridge
> > > ip addr add 192.0.2.1/25 dev br0
> > > ip addr add 192.0.2.129/25 dev br1
> > 
> > In this configuration you're supposed to put an IP address on the fec2
> > interface (eth1), not on br1.
> > 
> > br1 will handle offloaded forwarding between port5 and the external
> > ports (port3, port4). It doesn't need an IP address. In fact, if you
> > give it an IP address, you will make the sent packets go through the br1
> > interface, which does dev_queue_xmit() to the bridge ports (port3, port4,
> > port5), ports which are DSA, so they do dev_queue_xmit() through their
> > DSA master - eth0. So the system behaves as instructed.
> 
> Yep. As i said in another email, consider eth1 being connected to an
> external managed switch. br1 is how you manage that switch, but that
> is all you use br1 for. eth1 is where you do networking.

It would have been good to have support for subtractive device tree
overlays, such that when there are multiple CPU ports in the device
tree, the stable device tree has both CPU ports marked with the
"ethernet" phandle, but the user has the option of deleting that
property from one of the CPU ports, turning it into a user port.
Currently for LS1028A I am doing this device tree post-processing
from the U-Boot command line:

=> tftp $fdt_addr_r ls1028/fsl-ls1028a-rdb.dtb
=> fdt addr $fdt_addr_r
=> fdt rm /soc/pcie@1f0000000/ethernet-switch@0,5/ports/port@4 ethernet

but it has the disadvantage that you can only operate with the
configuration that you booted with.

I analyzed the possibility for DSA to dynamically switch a port between
operating as a CPU port or a user port, but it is simply insanely complicated.

Re: mv88e6321, dual cpu port

[Angelo Dureghello]

Hi,

On 16/02/23 4:31 PM, Vladimir Oltean wrote:
> On Thu, Feb 16, 2023 at 01:59:22PM +0100, Andrew Lunn wrote:
>> On Thu, Feb 16, 2023 at 02:50:40PM +0200, Vladimir Oltean wrote:
>>> On Thu, Feb 16, 2023 at 12:20:24PM +0100, Angelo Dureghello wrote:
>>>> Still data passes all trough port6, even when i ping from
>>>> host PC to port4. I was expecting instead to see port5
>>>> statistics increasing.
>>>
>>>> # configure the bridge
>>>> ip addr add 192.0.2.1/25 dev br0
>>>> ip addr add 192.0.2.129/25 dev br1
>>>
>>> In this configuration you're supposed to put an IP address on the fec2
>>> interface (eth1), not on br1.
>>>
>>> br1 will handle offloaded forwarding between port5 and the external
>>> ports (port3, port4). It doesn't need an IP address. In fact, if you
>>> give it an IP address, you will make the sent packets go through the br1
>>> interface, which does dev_queue_xmit() to the bridge ports (port3, port4,
>>> port5), ports which are DSA, so they do dev_queue_xmit() through their
>>> DSA master - eth0. So the system behaves as instructed.
>>
>> Yep. As i said in another email, consider eth1 being connected to an
>> external managed switch. br1 is how you manage that switch, but that
>> is all you use br1 for. eth1 is where you do networking.
> 
> It would have been good to have support for subtractive device tree
> overlays, such that when there are multiple CPU ports in the device
> tree, the stable device tree has both CPU ports marked with the
> "ethernet" phandle, but the user has the option of deleting that
> property from one of the CPU ports, turning it into a user port.
> Currently for LS1028A I am doing this device tree post-processing
> from the U-Boot command line:
> 
> => tftp $fdt_addr_r ls1028/fsl-ls1028a-rdb.dtb
> => fdt addr $fdt_addr_r
> => fdt rm /soc/pcie@1f0000000/ethernet-switch@0,5/ports/port@4 ethernet
> 
> but it has the disadvantage that you can only operate with the
> configuration that you booted with.
> 
> I analyzed the possibility for DSA to dynamically switch a port between
> operating as a CPU port or a user port, but it is simply insanely complicated.

thanks really, nice to know, i should not need
dynamic devicetree changes, but interesting that can be
done from u-boot this way.

I have a last issue, migrating from 5.4.70,
in 5.15.32 i have this error for both sfp cages:

# [   45.860784] mv88e6085 5b040000.ethernet-1:1d: p0: 
phylink_mac_link_state() failed: -95
[   45.860814] mv88e6085 5b040000.ethernet-1:1d: p0: 
phylink_mac_link_state() failed: -95
[   49.093371] mv88e6085 5b040000.ethernet-1:1d: p1: 
phylink_mac_link_state() failed: -95
[   49.093400] mv88e6085 5b040000.ethernet-1:1d: p1: 
phylink_mac_link_state() failed: -95


Is seems related to the fact that i am in in-band-status,
but 6321 has not serdes_pcs_get_state() op.

How can i fix this ?

Thanks !
-- 
Angelo Dureghello

Re: mv88e6321, dual cpu port

[Vladimir Oltean]

On Thu, Feb 16, 2023 at 07:39:11PM +0100, Angelo Dureghello wrote:
> I have a last issue, migrating from 5.4.70,
> in 5.15.32 i have this error for both sfp cages:
> 
> # [   45.860784] mv88e6085 5b040000.ethernet-1:1d: p0: phylink_mac_link_state() failed: -95
> [   45.860814] mv88e6085 5b040000.ethernet-1:1d: p0: phylink_mac_link_state() failed: -95
> [   49.093371] mv88e6085 5b040000.ethernet-1:1d: p1: phylink_mac_link_state() failed: -95
> [   49.093400] mv88e6085 5b040000.ethernet-1:1d: p1: phylink_mac_link_state() failed: -95
> 
> Is seems related to the fact that i am in in-band-status,
> but 6321 has not serdes_pcs_get_state() op.
> 
> How can i fix this ?
> 
> Thanks !
> -- 
> Angelo Dureghello

Looking at mv88e6321_ops in the latest net-next and in 5.4, I see no
serdes ops implemented in net-next. OTOH, in 5.4, the equivalent of the
current .serdes_pcs_get_state() which is now missing was .port_link_state().
In 5.4, mv88e6321_ops had .port_link_state() set to mv88e6352_port_link_state(),
but this got deleted with commit dc745ece3bd5 ("net: dsa: mv88e6xxx:
remove port_link_state functions") and seemingly was not replaced with
anything for 6321.

I don't actually know how this is supposed to work. Maybe Russell King can help?

Re: mv88e6321, dual cpu port

[Russell King (Oracle)]

On Sat, Feb 18, 2023 at 03:30:36PM +0200, Vladimir Oltean wrote:
> On Thu, Feb 16, 2023 at 07:39:11PM +0100, Angelo Dureghello wrote:
> > I have a last issue, migrating from 5.4.70,
> > in 5.15.32 i have this error for both sfp cages:
> > 
> > # [   45.860784] mv88e6085 5b040000.ethernet-1:1d: p0: phylink_mac_link_state() failed: -95
> > [   45.860814] mv88e6085 5b040000.ethernet-1:1d: p0: phylink_mac_link_state() failed: -95
> > [   49.093371] mv88e6085 5b040000.ethernet-1:1d: p1: phylink_mac_link_state() failed: -95
> > [   49.093400] mv88e6085 5b040000.ethernet-1:1d: p1: phylink_mac_link_state() failed: -95
> > 
> > Is seems related to the fact that i am in in-band-status,
> > but 6321 has not serdes_pcs_get_state() op.
> > 
> > How can i fix this ?
> > 
> > Thanks !
> > -- 
> > Angelo Dureghello
> 
> Looking at mv88e6321_ops in the latest net-next and in 5.4, I see no
> serdes ops implemented in net-next. OTOH, in 5.4, the equivalent of the
> current .serdes_pcs_get_state() which is now missing was .port_link_state().
> In 5.4, mv88e6321_ops had .port_link_state() set to mv88e6352_port_link_state(),
> but this got deleted with commit dc745ece3bd5 ("net: dsa: mv88e6xxx:
> remove port_link_state functions") and seemingly was not replaced with
> anything for 6321.
> 
> I don't actually know how this is supposed to work. Maybe Russell King can help?

I've no idea offhand, and as today I feel utterly shit due to post-
Covid, I'm not going to be able to sort through and work out what
happened for a few days. Sorry. I'll try to look at it next week
provided I feel better.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!