[Query]: DSA Understanding

[Query]: DSA Understanding

[Lad, Prabhakar]

Hi,

We are trying to integrate a MAC to an external switch as following:

+-------------------+            +------------------+
|                    | RGMII   |                    |
|                    +------------+  KSZ9897   +------ 1000baseT MDI ("LAN1")
|                    | PORT 6 |  7-port         +------ 1000baseT MDI ("LAN2")
|    AM572x    |              |  Ethernet     +------ 1000baseT MDI ("LAN3")
|     CPSW     |MIImgmt |  switch        +------ 1000baseT MDI ("LAN4")
|                    +-------------+                  +------
1000baseT MDI ("LAN5")
|                    |    SPI    |                    |
|                    |              |                   |
|                    |              |                   +------
1000baseT MII ("LAN7")
|                    |              |                   |
+------------------+             +------------------+

I have done all the configuration required for the MAC controller and
enabled the KSZ driver in the kernel, I can see the following:
~$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state
DOWN mode DEFAULT group default qlen 1000
    link/ether c4:f3:12:08:fe:7e brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,DYNAMIC,UP,LOWER_UP> mtu 1500 qdisc mq
state UP mode DEFAULT group default qlen 1000
    link/ether c4:f3:12:08:fe:7f brd ff:ff:ff:ff:ff:ff
4: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT
group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
5: lan1@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
mode DEFAULT group default qlen 1000
    link/ether c4:f3:12:08:fe:7e brd ff:ff:ff:ff:ff:ff
6: lan2@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
mode DEFAULT group default qlen 1000
    link/ether c4:f3:12:08:fe:7e brd ff:ff:ff:ff:ff:ff
7: lan3@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
mode DEFAULT group default qlen 1000
    link/ether c4:f3:12:08:fe:7e brd ff:ff:ff:ff:ff:ff
8: lan4@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
mode DEFAULT group default qlen 1000
    link/ether c4:f3:12:08:fe:7e brd ff:ff:ff:ff:ff:ff
9: lan5@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
mode DEFAULT group default qlen 1000
    link/ether c4:f3:12:08:fe:7e brd ff:ff:ff:ff:ff:ff
10: lan7@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
mode DEFAULT group default qlen 1000
    link/ether c4:f3:12:08:fe:7e brd ff:ff:ff:ff:ff:ff

When I connect port 1 and port 5 to two different devices I can ping
across them. But I am not able to ping the host/cpu port (eth0) from
other ports.
Do I need to create a bridge and then talk to host port ?  Should CPU
port come up as a different interface as cpu5 or should it remain eth0
itself ?

When I get the port 1 up (ifconfig lan0 up), the device connected on
other end of port1 says link is down, whereas the are connected.
>> Log on host machine where switch is connected
~$ ifconfig lan0 up
  [ 3720.547985] ksz9477-switch spi3.0 lan0: Link is Down

>> Log on device connected to port 1
[ 3629.045242] cpsw 48484000.ethernet eth0: Link is Down
[ 3629.112376] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Is the above behaviour expected ?

Basically I want all the ports to talk to each other, should I be
using DSA or look into swicthdev or something else.

Could someone point me in the right direction.

Cheers,
--Prabhakar Lad

Re: [Query]: DSA Understanding

[Andrew Lunn]

On Wed, Jul 25, 2018 at 09:39:50AM +0100, Lad, Prabhakar wrote:
> Hi,
> 
> We are trying to integrate a MAC to an external switch as following:
> 
> +-------------------+            +------------------+
> |                    | RGMII   |                    |
> |                    +------------+  KSZ9897   +------ 1000baseT MDI ("LAN1")
> |                    | PORT 6 |  7-port         +------ 1000baseT MDI ("LAN2")
> |    AM572x    |              |  Ethernet     +------ 1000baseT MDI ("LAN3")
> |     CPSW     |MIImgmt |  switch        +------ 1000baseT MDI ("LAN4")
> |                    +-------------+                  +------
> 1000baseT MDI ("LAN5")
> |                    |    SPI    |                    |
> |                    |              |                   |
> |                    |              |                   +------
> 1000baseT MII ("LAN7")
> |                    |              |                   |
> +------------------+             +------------------+

Please use a fixed size font, otherwise this is unreadable.

> 
> I have done all the configuration required for the MAC controller and
> enabled the KSZ driver in the kernel, I can see the following:
> ~$ ip link show
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
> mode DEFAULT group default qlen 1000
>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> 2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state
> DOWN mode DEFAULT group default qlen 1000

eth0 is you master device. NO-CARRIER is bad, since that means you
cannot send/receive from from the switch. Are you using a fixed phy on
the CPSW? What does your device tree look like?

    Andrew

Re: [Query]: DSA Understanding

[Lad, Prabhakar]

Hi Andrew,

Thanks for the reply.

On Wed, Jul 25, 2018 at 5:19 PM, Andrew Lunn <andrew@lunn.ch> wrote:
> On Wed, Jul 25, 2018 at 09:39:50AM +0100, Lad, Prabhakar wrote:
>> Hi,
>>
>> We are trying to integrate a MAC to an external switch as following:
>>
>> +-------------------+            +------------------+
>> |                    | RGMII   |                    |
>> |                    +------------+  KSZ9897   +------ 1000baseT MDI ("LAN1")
>> |                    | PORT 6 |  7-port         +------ 1000baseT MDI ("LAN2")
>> |    AM572x    |              |  Ethernet     +------ 1000baseT MDI ("LAN3")
>> |     CPSW     |MIImgmt |  switch        +------ 1000baseT MDI ("LAN4")
>> |                    +-------------+                  +------
>> 1000baseT MDI ("LAN5")
>> |                    |    SPI    |                    |
>> |                    |              |                   |
>> |                    |              |                   +------
>> 1000baseT MII ("LAN7")
>> |                    |              |                   |
>> +------------------+             +------------------+
>
> Please use a fixed size font, otherwise this is unreadable.
>
sorry about that, Ill take care of it henceforth.

>>
>> I have done all the configuration required for the MAC controller and
>> enabled the KSZ driver in the kernel, I can see the following:
>> ~$ ip link show
>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
>> mode DEFAULT group default qlen 1000
>>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>> 2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state
>> DOWN mode DEFAULT group default qlen 1000
>
> eth0 is you master device. NO-CARRIER is bad, since that means you
> cannot send/receive from from the switch. Are you using a fixed phy on
> the CPSW? What does your device tree look like?
>

With my earlier post, the dsa was linking to eth0, which is incorrect
as the switch is connected to eth1.
I had to patch the dsa to pickup eth1 instead of eth0.

CPSW can support two ports internally one port is connected to KSZ9031
phy chip which is eth0 and the
other port is connected to KSZ9897 chip eth1, following is the ip link
dump with my patch applied.

~$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state
DOWN mode DEFAULT group default qlen 1000
    link/ether c4:f3:12:08:fe:7e brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
mode DEFAULT group default qlen 1000
    link/ether c4:f3:12:08:fe:7f brd ff:ff:ff:ff:ff:ff
4: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT
group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
5: lan0@eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
mode DEFAULT group default qlen 1000
    link/ether c4:f3:12:08:fe:7f brd ff:ff:ff:ff:ff:ff
6: lan1@eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
mode DEFAULT group default qlen 1000
    link/ether c4:f3:12:08:fe:7f brd ff:ff:ff:ff:ff:ff
7: lan2@eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
mode DEFAULT group default qlen 1000
    link/ether c4:f3:12:08:fe:7f brd ff:ff:ff:ff:ff:ff
8: lan3@eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
mode DEFAULT group default qlen 1000
    link/ether c4:f3:12:08:fe:7f brd ff:ff:ff:ff:ff:ff
9: lan4@eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
mode DEFAULT group default qlen 1000
    link/ether c4:f3:12:08:fe:7f brd ff:ff:ff:ff:ff:ff
10: lan6@eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
mode DEFAULT group default qlen 1000
    link/ether c4:f3:12:08:fe:7f brd ff:ff:ff:ff:ff:ff
~$

Yes I am using fixed phy on slave1, following is my dts:

        ethernet@48484000 {
            compatible = "ti,dra7-cpsw", "ti,cpsw";
            ti,hwmods = "gmac";
            clocks = <0x124 0x125>;
            clock-names = "fck", "cpts";
            cpdma_channels = <0x8>;
            ale_entries = <0x400>;
            bd_ram_size = <0x2000>;
            mac_control = <0x20>;
            slaves = <0x2>;
            active_slave = <0x0>;
            cpts_clock_mult = <0x784cfe14>;
            cpts_clock_shift = <0x1d>;
            reg = <0x48484000 0x1000 0x48485200 0x2e00>;
            #address-cells = <0x1>;
            #size-cells = <0x1>;
            ti,no-idle;
            interrupts = <0x0 0x14e 0x4 0x0 0x14f 0x4 0x0 0x150 0x4
0x0 0x151 0x4>;
            ranges;
            syscon = <0x8>;
            status = "okay";
            pinctrl-names = "default", "sleep";
            pinctrl-0 = <0x126 0x127>;
            pinctrl-1 = <0x128 0x129>;
            dual_emac;
            linux,phandle = <0x500>;
            phandle = <0x500>;

            mdio@48485000 {
                compatible = "ti,cpsw-mdio", "ti,davinci_mdio";
                #address-cells = <0x1>;
                #size-cells = <0x0>;
                ti,hwmods = "davinci_mdio";
                bus_freq = <0xf4240>;
                reg = <0x48485000 0x100>;
                status = "okay";
                pinctrl-names = "default", "sleep";
                pinctrl-0 = <0x12a>;
                pinctrl-1 = <0x12b>;

                ethernet-phy@1 {
                    reg = <0x1>;
                    linux,phandle = <0x12c>;
                    phandle = <0x12c>;
                };
            };

            slave@48480200 {
                mac-address = [00 00 00 00 00 00];
                status = "okay";
                phy-handle = <0x12c>;
                phy-mode = "rgmii";
                dual_emac_res_vlan = <0x1>;
            };

            slave@48480300 {
                mac-address = [00 00 00 00 00 00];
                status = "okay";
                phy-mode = "rgmii";
                dual_emac_res_vlan = <0x2>;
                linux,phandle = <0xf3>;
                phandle = <0xf3>;

                fixed-link {
                    speed = <0x3e8>;
                    full-duplex;
                };
            };

            cpsw-phy-sel@4a002554 {
                compatible = "ti,dra7xx-cpsw-phy-sel";
                reg = <0x4a002554 0x4>;
                reg-names = "gmii-sel";
            };
        };

        spi@480ba000 {
            compatible = "ti,omap4-mcspi";
            reg = <0x480ba000 0x200>;
            interrupts = <0x0 0x2b 0x4>;
            #address-cells = <0x1>;
            #size-cells = <0x0>;
            ti,hwmods = "mcspi4";
            ti,spi-num-cs = <0x1>;
            dmas = <0xb2 0x46 0xb2 0x47>;
            dma-names = "tx0", "rx0";
            status = "okay";
            ti,pindir-d0-out-d1-in;

            ksz9477@0 {
                compatible = "microchip,ksz9477";
                reg = <0x0>;
                spi-max-frequency = <0x2dc6c00>;
                spi-cpha;
                spi-cpol;

                ports {
                    #address-cells = <0x1>;
                    #size-cells = <0x0>;

                    port@0 {
                        reg = <0x0>;
                        label = "lan0";
                    };

                    port@1 {
                        reg = <0x1>;
                        label = "lan1";
                    };

                    port@2 {
                        reg = <0x2>;
                        label = "lan2";
                    };

                    port@3 {
                        reg = <0x3>;
                        label = "lan3";
                    };

                    port@4 {
                        reg = <0x4>;
                        label = "lan4";
                    };

                    port@5 {
                        reg = <0x5>;
                        label = "cpu";
                        ethernet = <0x500>;
                    };

                    port@6 {
                        reg = <0x6>;
                        label = "lan6";
                    };
                };
            };
        };

Where port 6 of ksz9897 chip is connected to slave 1 (eth1) of cpsw
via rgmii interface. Following is the dmesg log:

[    2.596958] Microchip KSZ9477 dsa-0.0:00: attached PHY driver
[Microchip KSZ9477] (mii_bus:phy_addr=dsa-0.0:00, irq=POLL)
[    2.714959] Microchip KSZ9477 dsa-0.0:01: attached PHY driver
[Microchip KSZ9477] (mii_bus:phy_addr=dsa-0.0:01, irq=POLL)
[    2.832958] Microchip KSZ9477 dsa-0.0:02: attached PHY driver
[Microchip KSZ9477] (mii_bus:phy_addr=dsa-0.0:02, irq=POLL)
[    2.950960] Microchip KSZ9477 dsa-0.0:03: attached PHY driver
[Microchip KSZ9477] (mii_bus:phy_addr=dsa-0.0:03, irq=POLL)
[    3.068958] Microchip KSZ9477 dsa-0.0:04: attached PHY driver
[Microchip KSZ9477] (mii_bus:phy_addr=dsa-0.0:04, irq=POLL)
[    3.080953] Generic PHY dsa-0.0:06: attached PHY driver [Generic
PHY] (mii_bus:phy_addr=dsa-0.0:06, irq=POLL)


Cheers,
--Prabhakar

Re: [Query]: DSA Understanding

[Andrew Lunn]

> Yes I am using fixed phy on slave1, following is my dts:

Posting the original DTS file is better, not the decompiled version.

> 
>         ethernet@48484000 {
>             compatible = "ti,dra7-cpsw", "ti,cpsw";
>             ti,hwmods = "gmac";
>             clocks = <0x124 0x125>;
>             clock-names = "fck", "cpts";
>             cpdma_channels = <0x8>;
>             ale_entries = <0x400>;
>             bd_ram_size = <0x2000>;
>             mac_control = <0x20>;
>             slaves = <0x2>;
>             active_slave = <0x0>;
>             cpts_clock_mult = <0x784cfe14>;
>             cpts_clock_shift = <0x1d>;
>             reg = <0x48484000 0x1000 0x48485200 0x2e00>;
>             #address-cells = <0x1>;
>             #size-cells = <0x1>;
>             ti,no-idle;
>             interrupts = <0x0 0x14e 0x4 0x0 0x14f 0x4 0x0 0x150 0x4
> 0x0 0x151 0x4>;
>             ranges;
>             syscon = <0x8>;
>             status = "okay";
>             pinctrl-names = "default", "sleep";
>             pinctrl-0 = <0x126 0x127>;
>             pinctrl-1 = <0x128 0x129>;
>             dual_emac;
>             linux,phandle = <0x500>;
>             phandle = <0x500>;

So here is 0x500

> 
>             mdio@48485000 {
>                 compatible = "ti,cpsw-mdio", "ti,davinci_mdio";
>                 #address-cells = <0x1>;
>                 #size-cells = <0x0>;
>                 ti,hwmods = "davinci_mdio";
>                 bus_freq = <0xf4240>;
>                 reg = <0x48485000 0x100>;
>                 status = "okay";
>                 pinctrl-names = "default", "sleep";
>                 pinctrl-0 = <0x12a>;
>                 pinctrl-1 = <0x12b>;
> 
>                 ethernet-phy@1 {
>                     reg = <0x1>;
>                     linux,phandle = <0x12c>;
>                     phandle = <0x12c>;
>                 };
>             };
> 
>             slave@48480200 {
>                 mac-address = [00 00 00 00 00 00];
>                 status = "okay";
>                 phy-handle = <0x12c>;
>                 phy-mode = "rgmii";
>                 dual_emac_res_vlan = <0x1>;
>             };
> 
>             slave@48480300 {
>                 mac-address = [00 00 00 00 00 00];
>                 status = "okay";
>                 phy-mode = "rgmii";
>                 dual_emac_res_vlan = <0x2>;
>                 linux,phandle = <0xf3>;
>                 phandle = <0xf3>;

This is the actual interface you are using and it has a phandle of
0xf3.

> 
>                 fixed-link {
>                     speed = <0x3e8>;
>                     full-duplex;
>                 };
>             };
> 
>             cpsw-phy-sel@4a002554 {
>                 compatible = "ti,dra7xx-cpsw-phy-sel";
>                 reg = <0x4a002554 0x4>;
>                 reg-names = "gmii-sel";
>             };
>         };
> 
>         spi@480ba000 {
>             compatible = "ti,omap4-mcspi";
>             reg = <0x480ba000 0x200>;
>             interrupts = <0x0 0x2b 0x4>;
>             #address-cells = <0x1>;
>             #size-cells = <0x0>;
>             ti,hwmods = "mcspi4";
>             ti,spi-num-cs = <0x1>;
>             dmas = <0xb2 0x46 0xb2 0x47>;
>             dma-names = "tx0", "rx0";
>             status = "okay";
>             ti,pindir-d0-out-d1-in;
> 
>             ksz9477@0 {
>                 compatible = "microchip,ksz9477";
>                 reg = <0x0>;
>                 spi-max-frequency = <0x2dc6c00>;
>                 spi-cpha;
>                 spi-cpol;
> 
>                 ports {
>                     #address-cells = <0x1>;
>                     #size-cells = <0x0>;
> 
>                     port@0 {
>                         reg = <0x0>;
>                         label = "lan0";
>                     };
> 
>                     port@1 {
>                         reg = <0x1>;
>                         label = "lan1";
>                     };
> 
>                     port@2 {
>                         reg = <0x2>;
>                         label = "lan2";
>                     };
> 
>                     port@3 {
>                         reg = <0x3>;
>                         label = "lan3";
>                     };
> 
>                     port@4 {
>                         reg = <0x4>;
>                         label = "lan4";
>                     };
> 
>                     port@5 {
>                         reg = <0x5>;
>                         label = "cpu";
>                         ethernet = <0x500>;

So here you are pointing to the container node, not the actual
interface. You talk of having to patch DSA. That i would say is
wrong. You should be using the phandle of the actual interface.

>                     };
> 
>                     port@6 {
>                         reg = <0x6>;
>                         label = "lan6";
>                     };
>                 };
>             };
>         };
> 
> Where port 6 of ksz9897 chip is connected to slave 1 (eth1) of cpsw
> via rgmii interface. Following is the dmesg log:
> 
> [    2.596958] Microchip KSZ9477 dsa-0.0:00: attached PHY driver
> [Microchip KSZ9477] (mii_bus:phy_addr=dsa-0.0:00, irq=POLL)
> [    2.714959] Microchip KSZ9477 dsa-0.0:01: attached PHY driver
> [Microchip KSZ9477] (mii_bus:phy_addr=dsa-0.0:01, irq=POLL)
> [    2.832958] Microchip KSZ9477 dsa-0.0:02: attached PHY driver
> [Microchip KSZ9477] (mii_bus:phy_addr=dsa-0.0:02, irq=POLL)
> [    2.950960] Microchip KSZ9477 dsa-0.0:03: attached PHY driver
> [Microchip KSZ9477] (mii_bus:phy_addr=dsa-0.0:03, irq=POLL)
> [    3.068958] Microchip KSZ9477 dsa-0.0:04: attached PHY driver
> [Microchip KSZ9477] (mii_bus:phy_addr=dsa-0.0:04, irq=POLL)
> [    3.080953] Generic PHY dsa-0.0:06: attached PHY driver [Generic
> PHY] (mii_bus:phy_addr=dsa-0.0:06, irq=POLL)

You might also need a fixed-link for the cpu port of the switch.

What do the statistics counters show? Is it trying to send/receiver
frames out eth1? Are bad frames received? Maybe you need to set rgmii
delays?

	Andrew

Re: [Query]: DSA Understanding

[Lad, Prabhakar]

On Thu, Jul 26, 2018 at 3:08 PM, Andrew Lunn <andrew@lunn.ch> wrote:
>> Yes I am using fixed phy on slave1, following is my dts:
>
> Posting the original DTS file is better, not the decompiled version.
>
My bad will take care of it next time.

>>
>>         ethernet@48484000 {
>>             compatible = "ti,dra7-cpsw", "ti,cpsw";
>>             ti,hwmods = "gmac";
>>             clocks = <0x124 0x125>;
>>             clock-names = "fck", "cpts";
>>             cpdma_channels = <0x8>;
>>             ale_entries = <0x400>;
>>             bd_ram_size = <0x2000>;
>>             mac_control = <0x20>;
>>             slaves = <0x2>;
>>             active_slave = <0x0>;
>>             cpts_clock_mult = <0x784cfe14>;
>>             cpts_clock_shift = <0x1d>;
>>             reg = <0x48484000 0x1000 0x48485200 0x2e00>;
>>             #address-cells = <0x1>;
>>             #size-cells = <0x1>;
>>             ti,no-idle;
>>             interrupts = <0x0 0x14e 0x4 0x0 0x14f 0x4 0x0 0x150 0x4
>> 0x0 0x151 0x4>;
>>             ranges;
>>             syscon = <0x8>;
>>             status = "okay";
>>             pinctrl-names = "default", "sleep";
>>             pinctrl-0 = <0x126 0x127>;
>>             pinctrl-1 = <0x128 0x129>;
>>             dual_emac;
>>             linux,phandle = <0x500>;
>>             phandle = <0x500>;
>
> So here is 0x500
>
>>
>>             mdio@48485000 {
>>                 compatible = "ti,cpsw-mdio", "ti,davinci_mdio";
>>                 #address-cells = <0x1>;
>>                 #size-cells = <0x0>;
>>                 ti,hwmods = "davinci_mdio";
>>                 bus_freq = <0xf4240>;
>>                 reg = <0x48485000 0x100>;
>>                 status = "okay";
>>                 pinctrl-names = "default", "sleep";
>>                 pinctrl-0 = <0x12a>;
>>                 pinctrl-1 = <0x12b>;
>>
>>                 ethernet-phy@1 {
>>                     reg = <0x1>;
>>                     linux,phandle = <0x12c>;
>>                     phandle = <0x12c>;
>>                 };
>>             };
>>
>>             slave@48480200 {
>>                 mac-address = [00 00 00 00 00 00];
>>                 status = "okay";
>>                 phy-handle = <0x12c>;
>>                 phy-mode = "rgmii";
>>                 dual_emac_res_vlan = <0x1>;
>>             };
>>
>>             slave@48480300 {
>>                 mac-address = [00 00 00 00 00 00];
>>                 status = "okay";
>>                 phy-mode = "rgmii";
>>                 dual_emac_res_vlan = <0x2>;
>>                 linux,phandle = <0xf3>;
>>                 phandle = <0xf3>;
>
> This is the actual interface you are using and it has a phandle of
> 0xf3.
>
The dsa of_find_net_device_by_node() looks for ethernet device node,
but the above node is for the cpsw slave. I tried adding 0xf3 and got
probe defer,
that is the reason I added 0x500 (mdio) reference to cpu switch and
patched to pick
up eth1 instead of eth0.

>>
>>                 fixed-link {
>>                     speed = <0x3e8>;
>>                     full-duplex;
>>                 };
>>             };
>>
>>             cpsw-phy-sel@4a002554 {
>>                 compatible = "ti,dra7xx-cpsw-phy-sel";
>>                 reg = <0x4a002554 0x4>;
>>                 reg-names = "gmii-sel";
>>             };
>>         };
>>
>>         spi@480ba000 {
>>             compatible = "ti,omap4-mcspi";
>>             reg = <0x480ba000 0x200>;
>>             interrupts = <0x0 0x2b 0x4>;
>>             #address-cells = <0x1>;
>>             #size-cells = <0x0>;
>>             ti,hwmods = "mcspi4";
>>             ti,spi-num-cs = <0x1>;
>>             dmas = <0xb2 0x46 0xb2 0x47>;
>>             dma-names = "tx0", "rx0";
>>             status = "okay";
>>             ti,pindir-d0-out-d1-in;
>>
>>             ksz9477@0 {
>>                 compatible = "microchip,ksz9477";
>>                 reg = <0x0>;
>>                 spi-max-frequency = <0x2dc6c00>;
>>                 spi-cpha;
>>                 spi-cpol;
>>
>>                 ports {
>>                     #address-cells = <0x1>;
>>                     #size-cells = <0x0>;
>>
>>                     port@0 {
>>                         reg = <0x0>;
>>                         label = "lan0";
>>                     };
>>
>>                     port@1 {
>>                         reg = <0x1>;
>>                         label = "lan1";
>>                     };
>>
>>                     port@2 {
>>                         reg = <0x2>;
>>                         label = "lan2";
>>                     };
>>
>>                     port@3 {
>>                         reg = <0x3>;
>>                         label = "lan3";
>>                     };
>>
>>                     port@4 {
>>                         reg = <0x4>;
>>                         label = "lan4";
>>                     };
>>
>>                     port@5 {
>>                         reg = <0x5>;
>>                         label = "cpu";
>>                         ethernet = <0x500>;
>
> So here you are pointing to the container node, not the actual
> interface. You talk of having to patch DSA. That i would say is
> wrong. You should be using the phandle of the actual interface.
>
As mentioned due to above reason. agreed  the patch is wrong.

>>                     };
>>
>>                     port@6 {
>>                         reg = <0x6>;
>>                         label = "lan6";
>>                     };
>>                 };
>>             };
>>         };
>>
>> Where port 6 of ksz9897 chip is connected to slave 1 (eth1) of cpsw
>> via rgmii interface. Following is the dmesg log:
>>
>> [    2.596958] Microchip KSZ9477 dsa-0.0:00: attached PHY driver
>> [Microchip KSZ9477] (mii_bus:phy_addr=dsa-0.0:00, irq=POLL)
>> [    2.714959] Microchip KSZ9477 dsa-0.0:01: attached PHY driver
>> [Microchip KSZ9477] (mii_bus:phy_addr=dsa-0.0:01, irq=POLL)
>> [    2.832958] Microchip KSZ9477 dsa-0.0:02: attached PHY driver
>> [Microchip KSZ9477] (mii_bus:phy_addr=dsa-0.0:02, irq=POLL)
>> [    2.950960] Microchip KSZ9477 dsa-0.0:03: attached PHY driver
>> [Microchip KSZ9477] (mii_bus:phy_addr=dsa-0.0:03, irq=POLL)
>> [    3.068958] Microchip KSZ9477 dsa-0.0:04: attached PHY driver
>> [Microchip KSZ9477] (mii_bus:phy_addr=dsa-0.0:04, irq=POLL)
>> [    3.080953] Generic PHY dsa-0.0:06: attached PHY driver [Generic
>> PHY] (mii_bus:phy_addr=dsa-0.0:06, irq=POLL)
>
> You might also need a fixed-link for the cpu port of the switch.
>
> What do the statistics counters show? Is it trying to send/receiver
> frames out eth1? Are bad frames received? Maybe you need to set rgmii
> delays?
>
Yes I can see the statistics, I was sending dhcp requests from the
devices connected on lan0 and lan4 ports.

~$ ifconfig
eth0      Link encap:Ethernet  HWaddr C4:F3:12:08:FE:7E
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:102

eth1      Link encap:Ethernet  HWaddr C4:F3:12:08:FE:7F
          inet6 addr: fe80::c6f3:12ff:fe08:fe7f%lo/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:879 errors:0 dropped:0 overruns:0 frame:0
          TX packets:171 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:389836 (380.6 KiB)  TX bytes:23128 (22.5 KiB)

lan0      Link encap:Ethernet  HWaddr C4:F3:12:08:FE:7F
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:660 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:336530 (328.6 KiB)  TX bytes:0 (0.0 B)

lan3      Link encap:Ethernet  HWaddr C4:F3:12:08:FE:7F
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lan4      Link encap:Ethernet  HWaddr C4:F3:12:08:FE:7F
          inet6 addr: fe80::c6f3:12ff:fe08:fe7f%lo/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:219 errors:0 dropped:0 overruns:0 frame:0
          TX packets:85 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:36605 (35.7 KiB)  TX bytes:11732 (11.4 KiB)

~$

I am bit confused on how dsa needs to be actually working,
Q's
1] should I be running a dhcp server on eth1 (where switch is connected)
    so that devices connected on lan* devices get an ip ?

2] From the device where switch is connected if the cpu port wants to send
   any data to any other user ports lan* how do i do it (just open
socket on eth1 or lan*) ?
   I wrote a simple C code to dump some data onto port with lan0 but
couldn’t see
   anything on the device where lan0 is connected (using tcpdump).

  I also tried sending data from lan0 to lan4 but I couldnt see
anything on tcpdump
  but I can ping lan0 and lan4. (I have set lan0 and lan4 as local link only)

3] What is the ideal setup for dsa to work

Cheers,
--Prabhakar

Re: [Query]: DSA Understanding

[Andrew Lunn]

> I am bit confused on how dsa needs to be actually working,
> Q's
> 1] should I be running a dhcp server on eth1 (where switch is connected)
>     so that devices connected on lan* devices get an ip ?

Nope. You need eth1 up, but otherwise you do not use it. Use the lanX
interfaces like normal Linux interfaces. Run your dhclient on lanX, etc.
> 
> 2] From the device where switch is connected if the cpu port wants to send
>    any data to any other user ports lan* how do i do it (just open
> socket on eth1 or lan*) ?

Just treat the lanX interfaces as normal Linux interfaces.

     Andrew

Re: [Query]: DSA Understanding

[Lad, Prabhakar]

Hi Andrew,

Thank for your reply.

On Thu, Jul 26, 2018 at 4:39 PM, Andrew Lunn <andrew@lunn.ch> wrote:
>> I am bit confused on how dsa needs to be actually working,
>> Q's
>> 1] should I be running a dhcp server on eth1 (where switch is connected)
>>     so that devices connected on lan* devices get an ip ?
>
> Nope. You need eth1 up, but otherwise you do not use it. Use the lanX
> interfaces like normal Linux interfaces. Run your dhclient on lanX, etc.
>>
>> 2] From the device where switch is connected if the cpu port wants to send
>>    any data to any other user ports lan* how do i do it (just open
>> socket on eth1 or lan*) ?
>
> Just treat the lanX interfaces as normal Linux interfaces.
>
I have some more query’s on DSA.

I have manged to get the TI's cpsw slave1 connected to ksz9897
Ethernet switch chip partially working,

I have PC connected to lan4(ip = 169.254..126.126) and the PC ip is
169.254.78.251,
but when I ping from PC to lan4 I get Destination Host Unreachable,
but where as I can see
that in the tcpdump log for lan4 it does reply back, but it doesn’t
reach the PC, Is there I am missing
something here ?

Log from the device on which switch is present:
===================================

~$ ifconfig
eth0      Link encap:Ethernet  HWaddr C4:F3:12:08:FE:7E
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:102

eth1      Link encap:Ethernet  HWaddr C4:F3:12:08:FE:7F
          inet6 addr: fe80::c6f3:12ff:fe08:fe7f%lo/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:436 errors:0 dropped:0 overruns:0 frame:0
          TX packets:516 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:37254 (36.3 KiB)  TX bytes:51585 (50.3 KiB)

lan4      Link encap:Ethernet  HWaddr C4:F3:12:08:FE:7F
          inet addr:169.254.126.126  Bcast:169.254.255.255  Mask:255.255.0.0
          inet6 addr: fe80::c6f3:12ff:fe08:fe7f%lo/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:436 errors:0 dropped:0 overruns:0 frame:0
          TX packets:444 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:28970 (28.2 KiB)  TX bytes:35214 (34.3 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1%1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:67 errors:0 dropped:0 overruns:0 frame:0
          TX packets:67 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:6618 (6.4 KiB)  TX bytes:6618 (6.4 KiB)

~$ tcpdump -i lan4 -v
[  661.057166] device lan4 entered promiscuous mode
[  661.061814] device eth1 entered promiscuous mode
tcpdump: listening on lan4, link-type EN10MB (Ethernet), capture size
262144 bytes
07:40:20.255355 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
VB4-SN00000000 tell tango-charlie.local, length 46
07:40:20.255393 ARP, Ethernet (len 6), IPv4 (len 4), Reply
VB4-SN00000000 is-at c4:f3:12:08:fe:7f (oui Unknown), length 28
07:40:20.360936 IP6 (flowlabel 0x970aa, hlim 255, next-header UDP (17)
payload length: 53) VB4-SN00000000.mdns > ff02::fb.mdns: [udp sum o)
07:40:20.361085 IP (tos 0x0, ttl 255, id 17259, offset 0, flags [DF],
proto UDP (17), length 73)
    VB4-SN00000000.mdns > 224.0.0.251.mdns: 0 PTR (QM)?
251.78.254.169.in-addr.arpa. (45)
07:40:20.361848 IP (tos 0x0, ttl 255, id 1808, offset 0, flags [DF],
proto UDP (17), length 100)
    tango-charlie.local.mdns > 224.0.0.251.mdns: 0*- [0q] 1/0/0
251.78.254.169.in-addr.arpa. (Cache flush) PTR tango-charlie.local.
(72)
07:40:20.465933 IP6 (flowlabel 0x970aa, hlim 255, next-header UDP (17)
payload length: 98) VB4-SN00000000.mdns > ff02::fb.mdns: [udp sum o)
07:40:20.466124 IP (tos 0x0, ttl 255, id 17288, offset 0, flags [DF],
proto UDP (17), length 118)
    VB4-SN00000000.mdns > 224.0.0.251.mdns: 0 PTR (QM)?
b.f.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.f.f.ip6.arpa.
(90)
07:40:21.254161 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
VB4-SN00000000 tell tango-charlie.local, length 46
07:40:21.254181 ARP, Ethernet (len 6), IPv4 (len 4), Reply
VB4-SN00000000 is-at c4:f3:12:08:fe:7f (oui Unknown), length 28
07:40:25.470288 IP6 (flowlabel 0x970aa, hlim 255, next-header UDP (17)
payload length: 50) VB4-SN00000000.mdns > ff02::fb.mdns: [udp sum o)
07:40:31.301929 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
VB4-SN00000000 tell tango-charlie.local, length 46
07:40:31.301957 ARP, Ethernet (len 6), IPv4 (len 4), Reply
VB4-SN00000000 is-at c4:f3:12:08:fe:7f (oui Unknown), length 28
07:40:32.319104 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
VB4-SN00000000 tell tango-charlie.local, length 46
07:40:32.319131 ARP, Ethernet (len 6), IPv4 (len 4), Reply
VB4-SN00000000 is-at c4:f3:12:08:fe:7f (oui Unknown), length 28
07:40:33.317874 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
VB4-SN00000000 tell tango-charlie.local, length 46
07:40:33.317900 ARP, Ethernet (len 6), IPv4 (len 4), Reply
VB4-SN00000000 is-at c4:f3:12:08:fe:7f (oui Unknown), length 28
07:40:34.317840 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
VB4-SN00000000 tell tango-charlie.local, length 46
07:40:34.317866 ARP, Ethernet (len 6), IPv4 (len 4), Reply
VB4-SN00000000 is-at c4:f3:12:08:fe:7f (oui Unknown), length 28

~$ ethtool -S eth1
NIC statistics:
     Good Rx Frames: 715
     Broadcast Rx Frames: 553
     Multicast Rx Frames: 162
     Pause Rx Frames: 0
     Rx CRC Errors: 0
     Rx Align/Code Errors: 0
     Oversize Rx Frames: 0
     Rx Jabbers: 0
     Undersize (Short) Rx Frames: 0
     Rx Fragments: 0
     Rx Octets: 63055
     Good Tx Frames: 812
     Broadcast Tx Frames: 34
     Multicast Tx Frames: 283
     Pause Tx Frames: 0
     Deferred Tx Frames: 0
     Collisions: 0
     Single Collision Tx Frames: 0
     Multiple Collision Tx Frames: 0
     Excessive Collisions: 0
     Late Collisions: 0
     Tx Underrun: 0
     Carrier Sense Errors: 0
     Tx Octets: 85250
     Rx + Tx 64 Octet Frames: 0
     Rx + Tx 65-127 Octet Frames: 1287
     Rx + Tx 128-255 Octet Frames: 168
     Rx + Tx 256-511 Octet Frames: 72
     Rx + Tx 512-1023 Octet Frames: 0
     Rx + Tx 1024-Up Octet Frames: 0
     Net Octets: 148305
     Rx Start of Frame Overruns: 0
     Rx Middle of Frame Overruns: 0
     Rx DMA Overruns: 0
     Rx DMA chan 0: head_enqueue: 1
     Rx DMA chan 0: tail_enqueue: 813
     Rx DMA chan 0: pad_enqueue: 0
     Rx DMA chan 0: misqueued: 0
     Rx DMA chan 0: desc_alloc_fail: 0
     Rx DMA chan 0: pad_alloc_fail: 0
     Rx DMA chan 0: runt_receive_buf: 0
     Rx DMA chan 0: runt_transmit_bu: 0
     Rx DMA chan 0: empty_dequeue: 0
     Rx DMA chan 0: busy_dequeue: 668
     Rx DMA chan 0: good_dequeue: 686
     Rx DMA chan 0: requeue: 0
     Rx DMA chan 0: teardown_dequeue: 0
     Tx DMA chan 0: head_enqueue: 812
     Tx DMA chan 0: tail_enqueue: 0
     Tx DMA chan 0: pad_enqueue: 0
     Tx DMA chan 0: misqueued: 0
     Tx DMA chan 0: desc_alloc_fail: 0
     Tx DMA chan 0: pad_alloc_fail: 0
     Tx DMA chan 0: runt_receive_buf: 0
     Tx DMA chan 0: runt_transmit_bu: 502
     Tx DMA chan 0: empty_dequeue: 812
     Tx DMA chan 0: busy_dequeue: 0
     Tx DMA chan 0: good_dequeue: 812
     Tx DMA chan 0: requeue: 0
     Tx DMA chan 0: teardown_dequeue: 0
     p05_rx_hi: 0
     p05_rx_undersize: 0
     p05_rx_fragments: 0
     p05_rx_oversize: 0
     p05_rx_jabbers: 0
     p05_rx_symbol_err: 0
     p05_rx_crc_err: 0
     p05_rx_align_err: 0
     p05_rx_mac_ctrl: 0
     p05_rx_pause: 0
     p05_rx_bcast: 34
     p05_rx_mcast: 283
     p05_rx_ucast: 495
     p05_rx_64_or_less: 0
     p05_rx_65_127: 675
     p05_rx_128_255: 87
     p05_rx_256_511: 50
     p05_rx_512_1023: 0
     p05_rx_1024_1522: 0
     p05_rx_1523_2000: 0
     p05_rx_2001: 0
     p05_tx_hi: 0
     p05_tx_late_col: 0
     p05_tx_pause: 0
     p05_tx_bcast: 553
     p05_tx_mcast: 165
     p05_tx_ucast: 0
     p05_tx_deferred: 0
     p05_tx_total_col: 0
     p05_tx_exc_col: 0
     p05_tx_single_col: 0
     p05_tx_mult_col: 0
     p05_rx_total: 85250
     p05_tx_total: 63473
     p05_rx_discards: 505
     p05_tx_discards: 0
~$

~$ ethtool -S lan4
NIC statistics:
     tx_packets: 736
     tx_bytes: 59572
     rx_packets: 701
     rx_bytes: 45521
     rx_hi: 0
     rx_undersize: 0
     rx_fragments: 0
     rx_oversize: 0
     rx_jabbers: 0
     rx_symbol_err: 0
     rx_crc_err: 0
     rx_align_err: 0
     rx_mac_ctrl: 0
     rx_pause: 0
     rx_bcast: 602
     rx_mcast: 448
     rx_ucast: 495
     rx_64_or_less: 562
     rx_65_127: 742
     rx_128_255: 169
     rx_256_511: 72
     rx_512_1023: 0
     rx_1024_1522: 0
     rx_1523_2000: 0
     rx_2001: 0
     tx_hi: 0
     tx_late_col: 0
     tx_pause: 0
     tx_bcast: 582
     tx_mcast: 376
     tx_ucast: 0
     tx_deferred: 0
     tx_total_col: 0
     tx_exc_col: 0
     tx_single_col: 0
     tx_mult_col: 0
     rx_total: 148965
     tx_total: 104929
     rx_discards: 505
     tx_discards: 0
~$



Logs from the PC:
===================================

prabhakar@tango-charlie:~/Desktop/test$ ifconfig  enx00e04c68c229
enx00e04c68c229 Link encap:Ethernet  HWaddr 00:e0:4c:68:c2:29
          inet addr:169.254.78.251  Bcast:169.254.255.255  Mask:255.255.0.0
          inet6 addr: fe80::7cd0:12d6:d4bb:fc8b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:209 errors:0 dropped:0 overruns:0 frame:0
          TX packets:842 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:38308 (38.3 KB)  TX bytes:60751 (60.7 KB)

prabhakar@tango-charlie:~/Desktop/test$ ping -I enx00e04c68c229 169.254.126.126
PING 169.254.126.126 (169.254.126.126) from 169.254.78.251
enx00e04c68c229: 56(84) bytes of data.
>From 169.254.78.251 icmp_seq=1 Destination Host Unreachable
>From 169.254.78.251 icmp_seq=2 Destination Host Unreachable
>From 169.254.78.251 icmp_seq=3 Destination Host Unreachable
>From 169.254.78.251 icmp_seq=4 Destination Host Unreachable
>From 169.254.78.251 icmp_seq=5 Destination Host Unreachable
>From 169.254.78.251 icmp_seq=6 Destination Host Unreachable
^C
--- 169.254.126.126 ping statistics ---
8 packets transmitted, 0 received, +6 errors, 100% packet loss, time 6999ms
pipe 4

prabhakar@tango-charlie:~/Desktop/test$ sudo tcpdump -i enx00e04c68c229 -v
[sudo] password for prabhakar:
tcpdump: listening on enx00e04c68c229, link-type EN10MB (Ethernet),
capture size 262144 bytes
09:09:39.692909 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
VB4-SN00000000.local tell tango-charlie, length 28
09:09:39.794135 IP6 (flowlabel 0xaf44f, hlim 255, next-header UDP (17)
payload length: 54) tango-charlie.mdns > ff02::fb.mdns: [bad udp cksum
0x5fb4 -> 0x8657!] 0 PTR (QM)? 126.126.254.169.in-addr.arpa. (46)
09:09:39.794196 IP (tos 0x0, ttl 255, id 32415, offset 0, flags [DF],
proto UDP (17), length 74)
    tango-charlie.mdns > 224.0.0.251.mdns: 0 PTR (QM)?
126.126.254.169.in-addr.arpa. (46)
09:09:39.795280 IP (tos 0x0, ttl 255, id 37104, offset 0, flags [DF],
proto UDP (17), length 102)
    VB4-SN00000000.local.mdns > 224.0.0.251.mdns: 0*- [0q] 1/0/0
126.126.254.169.in-addr.arpa. (Cache flush) PTR VB4-SN00000000.local.
(74)
09:09:40.692890 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
VB4-SN00000000.local tell tango-charlie, length 28
09:09:41.710153 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
VB4-SN00000000.local tell tango-charlie, length 28
09:09:42.708884 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
VB4-SN00000000.local tell tango-charlie, length 28
09:09:43.708884 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
VB4-SN00000000.local tell tango-charlie, length 28
09:09:44.726124 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
VB4-SN00000000.local tell tango-charlie, length 28
09:09:45.724892 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
VB4-SN00000000.local tell tango-charlie, length 28
09:09:46.724887 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
VB4-SN00000000.local tell tango-charlie, length 28
^C
11 packets captured
11 packets received by filter
0 packets dropped by kernel

Cheers,
--Prabhakar

Re: [Query]: DSA Understanding

[Andrew Lunn]

> I have PC connected to lan4(ip = 169.254..126.126) and the PC ip is
> 169.254.78.251,
> but when I ping from PC to lan4 I get Destination Host Unreachable,
> but where as I can see
> that in the tcpdump log for lan4 it does reply back, but it doesn’t
> reach the PC, Is there I am missing
> something here ?
> 
> ~$ tcpdump -i lan4 -v
> [  661.057166] device lan4 entered promiscuous mode
> [  661.061814] device eth1 entered promiscuous mode

> 07:40:21.254161 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
> VB4-SN00000000 tell tango-charlie.local, length 46
> 07:40:21.254181 ARP, Ethernet (len 6), IPv4 (len 4), Reply
> VB4-SN00000000 is-at c4:f3:12:08:fe:7f (oui Unknown), length 28

Having names here does not help when you gave IP addresses above.

Am i reading this correct? The PC is ARPing the switch device. The
switch device is replying?

What does tcpdump on the PC show? Are the ARP replies getting to it?
Is the PC dropping the ARP replies?

If the PC is dropping the ARP replies, take a look at the
checksums. Wireshark is good at that.

If the ARP replies are not making it to the PC, look at the switch
statistics. ethtool -S lan4. Are the TX counts going up? Any error
counts going up?

       Andrew

Re: [Query]: DSA Understanding

[Lad, Prabhakar]

Hi Andrew,

On Thu, Aug 2, 2018 at 3:45 PM, Andrew Lunn <andrew@lunn.ch> wrote:
>> I have PC connected to lan4(ip = 169.254..126.126) and the PC ip is
>> 169.254.78.251,
>> but when I ping from PC to lan4 I get Destination Host Unreachable,
>> but where as I can see
>> that in the tcpdump log for lan4 it does reply back, but it doesn’t
>> reach the PC, Is there I am missing
>> something here ?
>>
>> ~$ tcpdump -i lan4 -v
>> [  661.057166] device lan4 entered promiscuous mode
>> [  661.061814] device eth1 entered promiscuous mode
>
>> 07:40:21.254161 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
>> VB4-SN00000000 tell tango-charlie.local, length 46
>> 07:40:21.254181 ARP, Ethernet (len 6), IPv4 (len 4), Reply
>> VB4-SN00000000 is-at c4:f3:12:08:fe:7f (oui Unknown), length 28
>
> Having names here does not help when you gave IP addresses above.
>
Sorry about that. VB4-SN00000000 is the switch device and
tango-charlie.local is the PC.

> Am i reading this correct? The PC is ARPing the switch device. The
> switch device is replying?
>
Yes the PC is ARPing to the switch device and from the tcpdump on
swicth device lan4
I can see that its trying to send Reply

~$ tcpdump -i lan4 -v
[ 1675.526326] device lan4 entered promiscuous mode
[ 1675.531503] device eth1 entered promiscuous mode
tcpdump: listening on lan4, link-type EN10MB (Ethernet), capture size
262144 bytes
07:57:06.133853 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
VB4-SN00000000 tell tango-charlie.local, length 46
07:57:06.133893 ARP, Ethernet (len 6), IPv4 (len 4), Reply
VB4-SN00000000 is-at c4:f3:12:08:fe:7f (oui Unknown), length 28
07:57:07.151100 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
VB4-SN00000000 tell tango-charlie.local, length 46
07:57:07.151133 ARP, Ethernet (len 6), IPv4 (len 4), Reply
VB4-SN00000000 is-at c4:f3:12:08:fe:7f (oui Unknown), length 28
07:57:07.703063 IP (tos 0x0, ttl 64, id 41824, offset 0, flags [DF],
proto UDP (17), length 224)
    tango-charlie.local.netbios-dgm > 169.254.255.255.netbios-dgm: NBT
UDP PACKET(138)
07:57:07.804780 IP6 (flowlabel 0x42aaa, hlim 255, next-header UDP (17)
payload length: 54) VB4-SN00000000.mdns > ff02::fb.mdns: [udp sum o)
07:57:07.804961 IP (tos 0x0, ttl 255, id 13070, offset 0, flags [DF],
proto UDP (17), length 74)
^C    VB4-SN00000000.mdns > 224.0.0.251.mdns: 0 PTR (QM)? 255.255.[
1689.046815] device lan4 left promiscuous mode
254.169.in-addr.arpa. (46)

7 packets captured
53 packets rec[ 1689.055592] device eth1 left promiscuous mode
eived by filter
40 packets dropped by kernel

> What does tcpdump on the PC show? Are the ARP replies getting to it?
> Is the PC dropping the ARP replies?
>
I dont see any Reply's on the PC with tcpdump on PC

> If the PC is dropping the ARP replies, take a look at the
> checksums. Wireshark is good at that.
>
Nor do I see anything in the wireshark.

> If the ARP replies are not making it to the PC, look at the switch
> statistics. ethtool -S lan4. Are the TX counts going up? Any error
> counts going up?
>
Yes the Tx counts are going up, without any errors, following is the log:

~$ ethtool -S lan4
NIC statistics:
     tx_packets: 499
     tx_bytes: 37105
     rx_packets: 462
     rx_bytes: 34138
     rx_hi: 0
     rx_undersize: 0
     rx_fragments: 0
     rx_oversize: 0
     rx_jabbers: 0
     rx_symbol_err: 0
     rx_crc_err: 0
     rx_align_err: 0
     rx_mac_ctrl: 0
     rx_pause: 0
     rx_bcast: 402
     rx_mcast: 69
     rx_ucast: 0
     rx_64_or_less: 361
     rx_65_127: 45
     rx_128_255: 34
     rx_256_511: 31
     rx_512_1023: 0
     rx_1024_1522: 0
     rx_1523_2000: 0
     rx_2001: 0
     tx_hi: 0
     tx_late_col: 0
     tx_pause: 0
     tx_bcast: 14
     tx_mcast: 196
     tx_ucast: 0
     tx_deferred: 0
     tx_total_col: 0
     tx_exc_col: 0
     tx_single_col: 0
     tx_mult_col: 0
     rx_total: 43248
     tx_total: 33759
     rx_discards: 0
     tx_discards: 0

Cheers,
--Prabhakar

Re: [Query]: DSA Understanding

[Andrew Lunn]

> I dont see any Reply's on the PC with tcpdump on PC

So try ethool -S on the PC. Any packets dropped because of errors?

Try turning off hardware checksums on the switch. ethtool -K.

    Andrew

Re: [Query]: DSA Understanding

[Florian Fainelli]

On 08/02/2018 09:05 AM, Andrew Lunn wrote:
>> I dont see any Reply's on the PC with tcpdump on PC
> 
> So try ethool -S on the PC. Any packets dropped because of errors?
> 
> Try turning off hardware checksums on the switch. ethtool -K.

Also make sure that cpsw is properly sending 64 bytes (including FCS)
packets to the switch, some switches will just discard packets when
packets are RUNT
-- 
Florian

Re: [Query]: DSA Understanding

[Lad, Prabhakar]

Hi Florain,

Thanks for your reply.

On Thu, Aug 2, 2018 at 5:24 PM Florian Fainelli <f.fainelli@gmail.com> wrote:
>
>
>
> On 08/02/2018 09:05 AM, Andrew Lunn wrote:
> >> I dont see any Reply's on the PC with tcpdump on PC
> >
> > So try ethool -S on the PC. Any packets dropped because of errors?
> >
> > Try turning off hardware checksums on the switch. ethtool -K.
>
> Also make sure that cpsw is properly sending 64 bytes (including FCS)
> packets to the switch, some switches will just discard packets when
> packets are RUNT

how can I dump the FCS from the switch (can I use tshark ?)

Cheers,
--Prabhakar Lad

Re: [Query]: DSA Understanding

[Andrew Lunn]

On Thu, Aug 09, 2018 at 12:33:30PM +0100, Lad, Prabhakar wrote:
> Hi Florain,
> 
> Thanks for your reply.
> 
> On Thu, Aug 2, 2018 at 5:24 PM Florian Fainelli <f.fainelli@gmail.com> wrote:
> >
> >
> >
> > On 08/02/2018 09:05 AM, Andrew Lunn wrote:
> > >> I dont see any Reply's on the PC with tcpdump on PC
> > >
> > > So try ethool -S on the PC. Any packets dropped because of errors?
> > >
> > > Try turning off hardware checksums on the switch. ethtool -K.
> >
> > Also make sure that cpsw is properly sending 64 bytes (including FCS)
> > packets to the switch, some switches will just discard packets when
> > packets are RUNT
> 
> how can I dump the FCS from the switch (can I use tshark ?)

tskark running on the target will not show you the FCS. The Ethernet
hardware should calculate that as it sends the packet. Wireshark on
the PC might, but some Ethernet cards strip off the FCS before passing
it to the stack.

What is important here is the idea of a runt packet. You appear to be
receiving some packets at the PC, if we can trust the statistics you
showed. Are those received packets all big? Are the small ARP replies
missing?

	Andrew

Re: [Query]: DSA Understanding

[Lad, Prabhakar]

Hi Andrew,

On Thu, Aug 2, 2018 at 5:05 PM Andrew Lunn <andrew@lunn.ch> wrote:
>
> > I dont see any Reply's on the PC with tcpdump on PC
>
> So try ethool -S on the PC. Any packets dropped because of errors?
>
I dont see any drops/errors on the PC, following is the dump from PC:

sudo ethtool -S enx00e04c68c229
[sudo] password for prabhakar:
NIC statistics:
     tx_packets: 1659
     rx_packets: 485
     tx_errors: 0
     rx_errors: 0
     rx_missed: 0
     align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     rx_unicast: 18
     rx_broadcast: 295
     rx_multicast: 172
     tx_aborted: 0
     tx_underrun: 0

> Try turning off hardware checksums on the switch. ethtool -K.
>
Following is the dump from the switch, the checksums are off

~$ ethtool -k eth1
Features for eth1:
Cannot get device udp-fragmentation-offload settings: Operation not supported
rx-checksumming: off [fixed]
tx-checksumming: off
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: off
        tx-scatter-gather: off [fixed]
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [fixed]
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off [fixed]
        tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]

Tshark dump on the switch:

~$ tshark -i lan4
Running as user "root" and group "root". This could be dangerous.
Capturing on 'lan4'
[ 1482.987520] device lan4 entered promiscuous mode
[ 1482.992169] device eth1 entered promiscuous mode
    1 0.000000000 RealtekS_68:c2:29 ��→ Broadcast    ARP 60 Who has
169.254.126.126? Tell 169.254.78.251
    2 0.000062952 c4:f3:12:08:fe:7f ��→ RealtekS_68:c2:29 ARP 42
169.254.126.126 is at c4:f3:12:08:fe:7f
    3 0.997115432 RealtekS_68:c2:29 ��→ Broadcast    ARP 60 Who has
169.254.126.126? Tell 169.254.78.251
    4 0.997142272 c4:f3:12:08:fe:7f ��→ RealtekS_68:c2:29 ARP 42
169.254.126.126 is at c4:f3:12:08:fe:7f
    5 1.997036539 RealtekS_68:c2:29 ��→ Broadcast    ARP 60 Who has
169.254.126.126? Tell 169.254.78.251
    6 1.997063379 c4:f3:12:08:fe:7f ��→ RealtekS_68:c2:29 ARP 42
169.254.126.126 is at c4:f3:12:08:fe:7f
    7 3.014232032 RealtekS_68:c2:29 ��→ Broadcast    ARP 60 Who has
169.254.126.126? Tell 169.254.78.251
    8 3.014252528 c4:f3:12:08:fe:7f ��→ RealtekS_68:c2:29 ARP 42
169.254.126.126 is at c4:f3:12:08:fe:7f
    9 4.013008290 RealtekS_68:c2:29 ��→ Broadcast    ARP 60 Who has
169.254.126.126? Tell 169.254.78.251
   10 4.013031064 c4:f3:12:08:fe:7f ��→ RealtekS_68:c2:29 ARP 42
169.254.126.126 is at c4:f3:12:08:fe:7f
   11 5.012951194 RealtekS_68:c2:29 ��→ Broadcast    ARP 60 Who has
169.254.126.126? Tell 169.254.78.251
   12 5.012970552 c4:f3:12:08:fe:7f ��→ RealtekS_68:c2:29 ARP 42
169.254.126.126 is at c4:f3:12:08:fe:7f
   13 6.030173853 RealtekS_68:c2:29 ��→ Broadcast    ARP 60 Who has
169.254.126.126? Tell 169.254.78.251
   14 6.030192234 c4:f3:12:08:fe:7f ��→ RealtekS_68:c2:29 ARP 42
169.254.126.126 is at c4:f3:12:08:fe:7f
   15 7.028911559 RealtekS_68:c2:29 ��→ Broadcast    ARP 60 Who has
169.254.126.126? Tell 169.254.78.251
   16 7.028947183 c4:f3:12:08:fe:7f ��→ RealtekS_68:c2:29 ARP 42
169.254.126.126 is at c4:f3:12:08:fe:7f
^C[ 1494.020087] device lan4 left promiscuous mode
[ 1494.024475] device eth1 left promiscuous mode
16 packets captured

Seems like the packet is not being transmitted from the switch at all
? (as ping from switch lan4 to PC fails)

~$ ping -I lan4 169.254.78.251
PING 169.254.78.251 (169.254.78.251): 56 data bytes
^C
--- 169.254.78.251 ping statistics ---
24 packets transmitted, 0 packets received, 100% packet loss

Cheers,
--Prabhakar

Re: [Query]: DSA Understanding

[Andrew Lunn]

On Thu, Aug 09, 2018 at 12:31:31PM +0100, Lad, Prabhakar wrote:
> Hi Andrew,
> 
> On Thu, Aug 2, 2018 at 5:05 PM Andrew Lunn <andrew@lunn.ch> wrote:
> >
> > > I dont see any Reply's on the PC with tcpdump on PC
> >
> > So try ethool -S on the PC. Any packets dropped because of errors?
> >
> I dont see any drops/errors on the PC, following is the dump from PC:
> 
> sudo ethtool -S enx00e04c68c229
> [sudo] password for prabhakar:
> NIC statistics:
>      tx_packets: 1659
>      rx_packets: 485
>      tx_errors: 0
>      rx_errors: 0
>      rx_missed: 0
>      align_errors: 0
>      tx_single_collisions: 0
>      tx_multi_collisions: 0
>      rx_unicast: 18
>      rx_broadcast: 295
>      rx_multicast: 172
>      tx_aborted: 0
>      tx_underrun: 0

So there are received packets at the PC. Not many unicast, mostly
broadcast, which fits with ARP. What does wireshark tell you about
these received packets? Are they ARP replies? Are they something else?
If they are ARP replies, why are they being ignored?  I don't know if
tshark will show you CRC problems. Wireshark does, when you unfold a
packet, and look at the fields in detail.

> Seems like the packet is not being transmitted from the switch at all
> ? (as ping from switch lan4 to PC fails)

I don't think you can make that conclusion yet. The PC is receiving
something, rx_packets=485. What are those packets?

Look at the statistics along the chain, from the target to the PC.
Look at the master device, lan4 and the PC. You should see about one
packet per second transmitted on the master device. One packet per
second transmitted on lan4, and one packet per second received on the
PC. Where does this break down.

	   Andrew

Re: [Query]: DSA Understanding

[Lad, Prabhakar]

On Thu, Aug 9, 2018 at 1:02 PM Andrew Lunn <andrew@lunn.ch> wrote:
>
> On Thu, Aug 09, 2018 at 12:31:31PM +0100, Lad, Prabhakar wrote:
> > Hi Andrew,
> >
> > On Thu, Aug 2, 2018 at 5:05 PM Andrew Lunn <andrew@lunn.ch> wrote:
> > >
> > > > I dont see any Reply's on the PC with tcpdump on PC
> > >
> > > So try ethool -S on the PC. Any packets dropped because of errors?
> > >
> > I dont see any drops/errors on the PC, following is the dump from PC:
> >
> > sudo ethtool -S enx00e04c68c229
> > [sudo] password for prabhakar:
> > NIC statistics:
> >      tx_packets: 1659
> >      rx_packets: 485
> >      tx_errors: 0
> >      rx_errors: 0
> >      rx_missed: 0
> >      align_errors: 0
> >      tx_single_collisions: 0
> >      tx_multi_collisions: 0
> >      rx_unicast: 18
> >      rx_broadcast: 295
> >      rx_multicast: 172
> >      tx_aborted: 0
> >      tx_underrun: 0
>
> So there are received packets at the PC. Not many unicast, mostly
> broadcast, which fits with ARP. What does wireshark tell you about
> these received packets? Are they ARP replies? Are they something else?
> If they are ARP replies, why are they being ignored?  I don't know if
> tshark will show you CRC problems. Wireshark does, when you unfold a
> packet, and look at the fields in detail.
>
> > Seems like the packet is not being transmitted from the switch at all
> > ? (as ping from switch lan4 to PC fails)
>
> I don't think you can make that conclusion yet. The PC is receiving
> something, rx_packets=485. What are those packets?
>
The received packets captured on the PC are MDNS and DHPC, these MDNS
are causing the rx
packet counter go up:

685    682.956266963    0.0.0.0    255.255.255.255    DHCP    422
DHCP Discover - Transaction ID 0x9d832fac
550    555.543322200    0.0.0.0    255.255.255.255    DHCP    422
DHCP Discover - Transaction ID 0x9d832fac
572    576.738682378    fe80::2f12:3d45:7cca:57fa    ff02::fb    MDNS
  136    Standard query 0x0000 AAAA VB4-SN00000000.local, "QM"
question SRV VB4-SN00000000._sftp-ssh._tcp.local, "QM" question
630    630.706578680    169.254.126.126    224.0.0.251    MDNS    168
  Standard query response 0x0000 AAAA, cache flush
fe80::c6f3:12ff:fe08:fe7f SRV, cache flush 0 0 22 VB4-SN00000000.local
A, cache flush 169.254.126.126
732    728.982449369    169.254.78.251    224.0.0.251    MDNS    122
 Standard query 0x0000 AAAA VB4-SN00000000.local, "QM" question A
VB4-SN00000000.local, "QM" question SRV
VB4-SN00000000._sftp-ssh._tcp.local, "QM" question
733    728.983534948    169.254.126.126    224.0.0.251    MDNS    168
  Standard query response 0x0000 SRV, cache flush 0 0 22
VB4-SN00000000.local AAAA, cache flush fe80::c6f3:12ff:fe08:fe7f A,
cache flush 169.254.126.126

> Look at the statistics along the chain, from the target to the PC.
> Look at the master device, lan4 and the PC. You should see about one
> packet per second transmitted on the master device. One packet per
> second transmitted on lan4, and one packet per second received on the
> PC. Where does this break down.
>
I don’t see any packets reaching the PC for the ping request. I can see the
RX and TX on the switch for lan4 increasing every second. seems like the
switch itself is consuming it and not forwarding(but then lan4 TX
shouldn’t have incremented ?).

any thoughts where I could focusing on the switch or cpsw ?

Cheers,
--Prabhakar Lad

Re: [Query]: DSA Understanding

[Andrew Lunn]

On Thu, Aug 09, 2018 at 01:45:52PM +0100, Lad, Prabhakar wrote:
> On Thu, Aug 9, 2018 at 1:02 PM Andrew Lunn <andrew@lunn.ch> wrote:
> >
> > On Thu, Aug 09, 2018 at 12:31:31PM +0100, Lad, Prabhakar wrote:
> > > Hi Andrew,
> > >
> > > On Thu, Aug 2, 2018 at 5:05 PM Andrew Lunn <andrew@lunn.ch> wrote:
> > > >
> > > > > I dont see any Reply's on the PC with tcpdump on PC
> > > >
> > > > So try ethool -S on the PC. Any packets dropped because of errors?
> > > >
> > > I dont see any drops/errors on the PC, following is the dump from PC:
> > >
> > > sudo ethtool -S enx00e04c68c229
> > > [sudo] password for prabhakar:
> > > NIC statistics:
> > >      tx_packets: 1659
> > >      rx_packets: 485
> > >      tx_errors: 0
> > >      rx_errors: 0
> > >      rx_missed: 0
> > >      align_errors: 0
> > >      tx_single_collisions: 0
> > >      tx_multi_collisions: 0
> > >      rx_unicast: 18
> > >      rx_broadcast: 295
> > >      rx_multicast: 172
> > >      tx_aborted: 0
> > >      tx_underrun: 0
> >
> > So there are received packets at the PC. Not many unicast, mostly
> > broadcast, which fits with ARP. What does wireshark tell you about
> > these received packets? Are they ARP replies? Are they something else?
> > If they are ARP replies, why are they being ignored?  I don't know if
> > tshark will show you CRC problems. Wireshark does, when you unfold a
> > packet, and look at the fields in detail.
> >
> > > Seems like the packet is not being transmitted from the switch at all
> > > ? (as ping from switch lan4 to PC fails)
> >
> > I don't think you can make that conclusion yet. The PC is receiving
> > something, rx_packets=485. What are those packets?
> >
> The received packets captured on the PC are MDNS and DHPC, these MDNS
> are causing the rx
> packet counter go up:

And where are these packets coming from? The target device? Or some
other device on your network?

> I don’t see any packets reaching the PC for the ping request. I can see the
> RX and TX on the switch for lan4 increasing every second. seems like the
> switch itself is consuming it and not forwarding(but then lan4 TX
> shouldn’t have incremented ?).

Which lan4 counters are going up? tx_packets, rx_packets, tx_errors,
rx_errors are software counters, and are incremented by the DSA
core. Other counters are hardware counters, and the DSA driver will
read them from the actual switch port. If the hardware counters show
packets are being transmitted, then the packets are probably on the
cable.

	Andrew

Re: [Query]: DSA Understanding

[Lad, Prabhakar]

Hi,

On Thu, Aug 9, 2018 at 1:56 PM Andrew Lunn <andrew@lunn.ch> wrote:
>
> On Thu, Aug 09, 2018 at 01:45:52PM +0100, Lad, Prabhakar wrote:
> > On Thu, Aug 9, 2018 at 1:02 PM Andrew Lunn <andrew@lunn.ch> wrote:
> > >
> > > On Thu, Aug 09, 2018 at 12:31:31PM +0100, Lad, Prabhakar wrote:
> > > > Hi Andrew,
> > > >
> > > > On Thu, Aug 2, 2018 at 5:05 PM Andrew Lunn <andrew@lunn.ch> wrote:
> > > > >
> > > > > > I dont see any Reply's on the PC with tcpdump on PC
> > > > >
> > > > > So try ethool -S on the PC. Any packets dropped because of errors?
> > > > >
> > > > I dont see any drops/errors on the PC, following is the dump from PC:
> > > >
> > > > sudo ethtool -S enx00e04c68c229
> > > > [sudo] password for prabhakar:
> > > > NIC statistics:
> > > >      tx_packets: 1659
> > > >      rx_packets: 485
> > > >      tx_errors: 0
> > > >      rx_errors: 0
> > > >      rx_missed: 0
> > > >      align_errors: 0
> > > >      tx_single_collisions: 0
> > > >      tx_multi_collisions: 0
> > > >      rx_unicast: 18
> > > >      rx_broadcast: 295
> > > >      rx_multicast: 172
> > > >      tx_aborted: 0
> > > >      tx_underrun: 0
> > >
> > > So there are received packets at the PC. Not many unicast, mostly
> > > broadcast, which fits with ARP. What does wireshark tell you about
> > > these received packets? Are they ARP replies? Are they something else?
> > > If they are ARP replies, why are they being ignored?  I don't know if
> > > tshark will show you CRC problems. Wireshark does, when you unfold a
> > > packet, and look at the fields in detail.
> > >
> > > > Seems like the packet is not being transmitted from the switch at all
> > > > ? (as ping from switch lan4 to PC fails)
> > >
> > > I don't think you can make that conclusion yet. The PC is receiving
> > > something, rx_packets=485. What are those packets?
> > >
> > The received packets captured on the PC are MDNS and DHPC, these MDNS
> > are causing the rx
> > packet counter go up:
>
> And where are these packets coming from? The target device? Or some
> other device on your network?
>
AFIK, MDNS is also kind of a bcast its sending MDNS requests and
receiving itself,
that’s the reason rx packets are incrementing (correct me if I am wrong)

On the PC where lan4 is connected , tx has high count because of ping requests
prabhakar@tango-charlie:~/Desktop/test$ ifconfig  enx00e04c68c229
enx00e04c68c229 Link encap:Ethernet  HWaddr 00:e0:4c:68:c2:29
          inet addr:169.254.78.251  Bcast:169.254.255.255  Mask:255.255.0.0
          inet6 addr: fe80::2f12:3d45:7cca:57fa/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:502 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4811 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:79843 (79.8 KB)  TX bytes:252647 (252.6 KB)

> > I don’t see any packets reaching the PC for the ping request. I can see the
> > RX and TX on the switch for lan4 increasing every second. seems like the
> > switch itself is consuming it and not forwarding(but then lan4 TX
> > shouldn’t have incremented ?).
>
> Which lan4 counters are going up? tx_packets, rx_packets, tx_errors,
> rx_errors are software counters, and are incremented by the DSA
> core. Other counters are hardware counters, and the DSA driver will
> read them from the actual switch port. If the hardware counters show
> packets are being transmitted, then the packets are probably on the
> cable.
>

I can see the RX and TX incrementing every second, no errors counters go up

$ watch -n1 ifconfig lan4
lan4      Link encap:Ethernet  HWaddr C4:F3:12:08:FE:7F
          inet addr:169.254.126.126  Bcast:169.254.255.255  Mask:255.255.0.0
          inet6 addr: fe80::18b9:d16c:7ff:ab73%3201178264/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2168 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1724 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:117816 (115.0 KiB)  TX bytes:99940 (97.5 KiB)

~$ ethtool -S eth1
NIC statistics:
     Good Rx Frames: 800
     Broadcast Rx Frames: 729
     Multicast Rx Frames: 71
     Pause Rx Frames: 0
     Rx CRC Errors: 0
     Rx Align/Code Errors: 0
     Oversize Rx Frames: 0
     Rx Jabbers: 0
     Undersize (Short) Rx Frames: 0
     Rx Fragments: 0
     Rx Octets: 65472
     Good Tx Frames: 369
     Broadcast Tx Frames: 16
     Multicast Tx Frames: 139
     Pause Tx Frames: 0
     Deferred Tx Frames: 0
     Collisions: 0
     Single Collision Tx Frames: 0
     Multiple Collision Tx Frames: 0
     Excessive Collisions: 0
     Late Collisions: 0
     Tx Underrun: 0
     Carrier Sense Errors: 0
     Tx Octets: 40990
     Rx + Tx 64 Octet Frames: 0
     Rx + Tx 65-127 Octet Frames: 1031
     Rx + Tx 128-255 Octet Frames: 73
     Rx + Tx 256-511 Octet Frames: 65
     Rx + Tx 512-1023 Octet Frames: 0
     Rx + Tx 1024-Up Octet Frames: 0
     Net Octets: 106462
     Rx Start of Frame Overruns: 0
     Rx Middle of Frame Overruns: 0
     Rx DMA Overruns: 0
     Rx DMA chan 0: head_enqueue: 1
     Rx DMA chan 0: tail_enqueue: 918
     Rx DMA chan 0: pad_enqueue: 0
     Rx DMA chan 0: misqueued: 0
     Rx DMA chan 0: desc_alloc_fail: 0
     Rx DMA chan 0: pad_alloc_fail: 0
     Rx DMA chan 0: runt_receive_buf: 0
     Rx DMA chan 0: runt_transmit_bu: 0
     Rx DMA chan 0: empty_dequeue: 0
     Rx DMA chan 0: busy_dequeue: 772
     Rx DMA chan 0: good_dequeue: 791
     Rx DMA chan 0: requeue: 0
     Rx DMA chan 0: teardown_dequeue: 0
     Tx DMA chan 0: head_enqueue: 369
     Tx DMA chan 0: tail_enqueue: 0
     Tx DMA chan 0: pad_enqueue: 0
     Tx DMA chan 0: misqueued: 0
     Tx DMA chan 0: desc_alloc_fail: 0
     Tx DMA chan 0: pad_alloc_fail: 0
     Tx DMA chan 0: runt_receive_buf: 0
     Tx DMA chan 0: runt_transmit_bu: 221
     Tx DMA chan 0: empty_dequeue: 369
     Tx DMA chan 0: busy_dequeue: 0
     Tx DMA chan 0: good_dequeue: 369
     Tx DMA chan 0: requeue: 0
     Tx DMA chan 0: teardown_dequeue: 0
     p05_rx_hi: 0
     p05_rx_undersize: 0
     p05_rx_fragments: 0
     p05_rx_oversize: 0
     p05_rx_jabbers: 0
     p05_rx_symbol_err: 0
     p05_rx_crc_err: 0
     p05_rx_align_err: 0
     p05_rx_mac_ctrl: 0
     p05_rx_pause: 0
     p05_rx_bcast: 651
     p05_rx_mcast: 208
     p05_rx_ucast: 214
     p05_rx_64_or_less: 593
     p05_rx_65_127: 343
     p05_rx_128_255: 73
     p05_rx_256_511: 64
     p05_rx_512_1023: 0
     p05_rx_1024_1522: 0
     p05_rx_1523_2000: 0
     p05_rx_2001: 0
     p05_tx_hi: 0
     p05_tx_late_col: 0
     p05_tx_pause: 0
     p05_tx_bcast: 744
     p05_tx_mcast: 177
     p05_tx_ucast: 0
     p05_tx_deferred: 0
     p05_tx_total_col: 0
     p05_tx_exc_col: 0
     p05_tx_single_col: 0
     p05_tx_mult_col: 0
     p05_rx_total: 99380
     p05_tx_total: 86102
     p05_rx_discards: 224
     p05_tx_discards: 0


~$ ethtool -S lan4
NIC statistics:
     tx_packets: 563
     tx_bytes: 41522
     rx_packets: 1010
     rx_bytes: 61155
     rx_hi: 0
     rx_undersize: 0
     rx_fragments: 0
     rx_oversize: 0
     rx_jabbers: 0
     rx_symbol_err: 0
     rx_crc_err: 0
     rx_align_err: 0
     rx_mac_ctrl: 0
     rx_pause: 0
     rx_bcast: 958
     rx_mcast: 222
     rx_ucast: 214
     rx_64_or_less: 898
     rx_65_127: 350
     rx_128_255: 80
     rx_256_511: 66
     rx_512_1023: 0
     rx_1024_1522: 0
     rx_1523_2000: 0
     rx_2001: 0
     tx_hi: 0
     tx_late_col: 0
     tx_pause: 0
     tx_bcast: 749
     tx_mcast: 212
     tx_ucast: 0
     tx_deferred: 0
     tx_total_col: 0
     tx_exc_col: 0
     tx_single_col: 0
     tx_mult_col: 0
     rx_total: 121503
     tx_total: 92530
     rx_discards: 224
     tx_discards: 0

Only weird thing I notice on target, when its replying for ping
requests ( (oui Unknown) is that something which is causing issues ?

08:11:20.230704 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
VB4-SN00000000 tell tango-charlie.local, length 46
08:11:20.230749 ARP, Ethernet (len 6), IPv4 (len 4), Reply
VB4-SN00000000 is-at c4:f3:12:08:fe:7f (oui Unknown), length 28
08:11:21.230629 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
VB4-SN00000000 tell tango-charlie.local, length 46
08:11:21.230657 ARP, Ethernet (len 6), IPv4 (len 4), Reply
VB4-SN00000000 is-at c4:f3:12:08:fe:7f (oui Unknown), length 28
08:11:22.247831 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
VB4-SN00000000 tell tango-charlie.local, length 46
08:11:22.247857 ARP, Ethernet (len 6), IPv4 (len 4), Reply
VB4-SN00000000 is-at c4:f3:12:08:fe:7f (oui Unknown), length 28


Cheers,
--Prabhakar

Re: [Query]: DSA Understanding

[Andrew Lunn]

> > > The received packets captured on the PC are MDNS and DHPC, these MDNS
> > > are causing the rx
> > > packet counter go up:
> >
> > And where are these packets coming from? The target device? Or some
> > other device on your network?
> >
> AFIK, MDNS is also kind of a bcast its sending MDNS requests and
> receiving itself,
> that’s the reason rx packets are incrementing (correct me if I am wrong)

Your Ethernet device should not be receiving its own
transmissions. Looping back for broadcast and multicast packets
happens higher up in the network stack.

Look at the source MAC address for these packets. Where are the coming
from?

> ~$ ethtool -S lan4
> NIC statistics:
...
>      tx_hi: 0
>      tx_late_col: 0
>      tx_pause: 0
>      tx_bcast: 749
>      tx_mcast: 212

This suggest the switch is putting packets onto the wire.

> Only weird thing I notice on target, when its replying for ping
> requests ( (oui Unknown) is that something which is causing issues ?

These are not ping requests. These are ARP requests/replies.

> 08:11:20.230704 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
> VB4-SN00000000 tell tango-charlie.local, length 46
> 08:11:20.230749 ARP, Ethernet (len 6), IPv4 (len 4), Reply
> VB4-SN00000000 is-at c4:f3:12:08:fe:7f (oui Unknown), length 28
> 08:11:21.230629 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
> VB4-SN00000000 tell tango-charlie.local, length 46
> 08:11:21.230657 ARP, Ethernet (len 6), IPv4 (len 4), Reply
> VB4-SN00000000 is-at c4:f3:12:08:fe:7f (oui Unknown), length 28
> 08:11:22.247831 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
> VB4-SN00000000 tell tango-charlie.local, length 46
> 08:11:22.247857 ARP, Ethernet (len 6), IPv4 (len 4), Reply
> VB4-SN00000000 is-at c4:f3:12:08:fe:7f (oui Unknown), length 28

c4:f3:12 is the OUI. It is actually registers to TI:

https://aruljohn.com/mac/C4F312

But tcpdump probably does not know this, or the build you have has the
oui table removed to keep the binary small.

    Andrew

Re: [Query]: DSA Understanding

[Lad, Prabhakar]

[-- Attachment #1: Type: text/plain, Size: 2567 bytes --]

Hi Andrew,

Thanks for your support.

On Thu, Aug 9, 2018 at 4:35 PM Andrew Lunn <andrew@lunn.ch> wrote:
>
> > > > The received packets captured on the PC are MDNS and DHPC, these MDNS
> > > > are causing the rx
> > > > packet counter go up:
> > >
> > > And where are these packets coming from? The target device? Or some
> > > other device on your network?
> > >
> > AFIK, MDNS is also kind of a bcast its sending MDNS requests and
> > receiving itself,
> > that’s the reason rx packets are incrementing (correct me if I am wrong)
>
> Your Ethernet device should not be receiving its own
> transmissions. Looping back for broadcast and multicast packets
> happens higher up in the network stack.
>
> Look at the source MAC address for these packets. Where are the coming
> from?
>
Its coming from the switch lan4 I have attached the png, where
C4:F3:12:08:FE:7F is
the mac of lan4, which is broadcast to ff:ff:ff:ff:ff:ff, which is
causing rx counter on
PC to go up.

> > ~$ ethtool -S lan4
> > NIC statistics:
> ...
> >      tx_hi: 0
> >      tx_late_col: 0
> >      tx_pause: 0
> >      tx_bcast: 749
> >      tx_mcast: 212
>
> This suggest the switch is putting packets onto the wire.
>
> > Only weird thing I notice on target, when its replying for ping
> > requests ( (oui Unknown) is that something which is causing issues ?
>
> These are not ping requests. These are ARP requests/replies.
>
Yes my bad.

> > 08:11:20.230704 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
> > VB4-SN00000000 tell tango-charlie.local, length 46
> > 08:11:20.230749 ARP, Ethernet (len 6), IPv4 (len 4), Reply
> > VB4-SN00000000 is-at c4:f3:12:08:fe:7f (oui Unknown), length 28
> > 08:11:21.230629 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
> > VB4-SN00000000 tell tango-charlie.local, length 46
> > 08:11:21.230657 ARP, Ethernet (len 6), IPv4 (len 4), Reply
> > VB4-SN00000000 is-at c4:f3:12:08:fe:7f (oui Unknown), length 28
> > 08:11:22.247831 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
> > VB4-SN00000000 tell tango-charlie.local, length 46
> > 08:11:22.247857 ARP, Ethernet (len 6), IPv4 (len 4), Reply
> > VB4-SN00000000 is-at c4:f3:12:08:fe:7f (oui Unknown), length 28
>
> c4:f3:12 is the OUI. It is actually registers to TI:
>
> https://aruljohn.com/mac/C4F312
>
> But tcpdump probably does not know this, or the build you have has the
> oui table removed to keep the binary small.
>
Yes I built it on yocto, probably to reduce they have small build.

Cheers,
--Prabhakar

[-- Attachment #2: frame.png --]
[-- Type: image/png, Size: 1281475 bytes --]

Re: [Query]: DSA Understanding

[Andrew Lunn]

> Its coming from the switch lan4 I have attached the png, where
> C4:F3:12:08:FE:7F is
> the mac of lan4, which is broadcast to ff:ff:ff:ff:ff:ff, which is
> causing rx counter on
> PC to go up.

So, big packets are making it from the switch to the PC. But the small
ARP packets are not.

This is what Florian was suggesting.

ARP packets are smaller than 64 bytes, which is the minimum packet
size for Ethernet. Any packets smaller than 64 bytes are called runt
packets. They have to be padded upto 64 bytes in order to make them
valid. Otherwise the destination, or any switch along the path, might
throw them away.

What could be happening is that the CSPW driver or hardware is padding
the packet to 64 bytes. But that packet has a DSA header in it. The
switch removes the header, recalculate the checksum and sends the
packet. It is now either 4 or 8 bytes smaller, depending on what DSA
header was used. It then becomes a runt packet.

Florian had to fix this problem recently.

http://patchwork.ozlabs.org/patch/836534/

You probably need something similar for the cpsw.

    Andrew

Re: [Query]: DSA Understanding

[Lad, Prabhakar]

Hi Andrew,

On Thu, Aug 9, 2018 at 6:23 PM Andrew Lunn <andrew@lunn.ch> wrote:
>
> > Its coming from the switch lan4 I have attached the png, where
> > C4:F3:12:08:FE:7F is
> > the mac of lan4, which is broadcast to ff:ff:ff:ff:ff:ff, which is
> > causing rx counter on
> > PC to go up.
>
> So, big packets are making it from the switch to the PC. But the small
> ARP packets are not.
>
> This is what Florian was suggesting.
>
> ARP packets are smaller than 64 bytes, which is the minimum packet
> size for Ethernet. Any packets smaller than 64 bytes are called runt
> packets. They have to be padded upto 64 bytes in order to make them
> valid. Otherwise the destination, or any switch along the path, might
> throw them away.
>
> What could be happening is that the CSPW driver or hardware is padding
> the packet to 64 bytes. But that packet has a DSA header in it. The
> switch removes the header, recalculate the checksum and sends the
> packet. It is now either 4 or 8 bytes smaller, depending on what DSA
> header was used. It then becomes a runt packet.
>
Thank you for the clarification, this really helped me out.

> Florian had to fix this problem recently.
>
> http://patchwork.ozlabs.org/patch/836534/
>
But seems like this patch was never accepted, instead
brcm_tag_xmit_ll() does it if I am understanding it correctly.
similarly to this ksz_xmit() is taking care of padding.

> You probably need something similar for the cpsw.
>
looking at the tag_ksz.c in xmit function this is taken care of

/* For Ingress (Host -> KSZ), 2 bytes are added before FCS.
 * ---------------------------------------------------------------------------
 * DA(6bytes)|SA(6bytes)|....|Data(nbytes)|tag0(1byte)|tag1(1byte)|FCS(4bytes)
 * ---------------------------------------------------------------------------
 * tag0 : Prioritization (not used now)
 * tag1 : each bit represents port (eg, 0x01=port1, 0x02=port2, 0x10=port5)
 *
 * For Egress (KSZ -> Host), 1 byte is added before FCS.
 * ---------------------------------------------------------------------------
 * DA(6bytes)|SA(6bytes)|....|Data(nbytes)|tag0(1byte)|FCS(4bytes)
 * ---------------------------------------------------------------------------
 * tag0 : zero-based value represents port
 *      (eg, 0x00=port1, 0x02=port3, 0x06=port7)
 */

#define    KSZ_INGRESS_TAG_LEN    2
#define    KSZ_EGRESS_TAG_LEN    1

static struct sk_buff *ksz_xmit(struct sk_buff *skb, struct net_device *dev)
{
    struct dsa_slave_priv *p = netdev_priv(dev);
    struct sk_buff *nskb;
    int padlen;
    u8 *tag;

    padlen = (skb->len >= ETH_ZLEN) ? 0 : ETH_ZLEN - skb->len;

    if (skb_tailroom(skb) >= padlen + KSZ_INGRESS_TAG_LEN) {
        /* Let dsa_slave_xmit() free skb */
        if (__skb_put_padto(skb, skb->len + padlen, false))
            return NULL;

        nskb = skb;
    } else {
        nskb = alloc_skb(NET_IP_ALIGN + skb->len +
                 padlen + KSZ_INGRESS_TAG_LEN, GFP_ATOMIC);
        if (!nskb)
            return NULL;
        skb_reserve(nskb, NET_IP_ALIGN);

        skb_reset_mac_header(nskb);
        skb_set_network_header(nskb,
                       skb_network_header(skb) - skb->head);
        skb_set_transport_header(nskb,
                     skb_transport_header(skb) - skb->head);
        skb_copy_and_csum_dev(skb, skb_put(nskb, skb->len));

        /* Let skb_put_padto() free nskb, and let dsa_slave_xmit() free
         * skb
         */
        if (skb_put_padto(nskb, nskb->len + padlen))
            return NULL;

        consume_skb(skb);
    }

    tag = skb_put(nskb, KSZ_INGRESS_TAG_LEN);
    tag[0] = 0;
    tag[1] = 1 << p->dp->index; /* destination port */

    return nskb;
}

Cheers,
--Prabhakar Lad

Re: [Query]: DSA Understanding

[Florian Fainelli]

On 08/10/2018 04:26 AM, Lad, Prabhakar wrote:
> Hi Andrew,
> 
> On Thu, Aug 9, 2018 at 6:23 PM Andrew Lunn <andrew@lunn.ch> wrote:
>>
>>> Its coming from the switch lan4 I have attached the png, where
>>> C4:F3:12:08:FE:7F is
>>> the mac of lan4, which is broadcast to ff:ff:ff:ff:ff:ff, which is
>>> causing rx counter on
>>> PC to go up.
>>
>> So, big packets are making it from the switch to the PC. But the small
>> ARP packets are not.
>>
>> This is what Florian was suggesting.
>>
>> ARP packets are smaller than 64 bytes, which is the minimum packet
>> size for Ethernet. Any packets smaller than 64 bytes are called runt
>> packets. They have to be padded upto 64 bytes in order to make them
>> valid. Otherwise the destination, or any switch along the path, might
>> throw them away.
>>
>> What could be happening is that the CSPW driver or hardware is padding
>> the packet to 64 bytes. But that packet has a DSA header in it. The
>> switch removes the header, recalculate the checksum and sends the
>> packet. It is now either 4 or 8 bytes smaller, depending on what DSA
>> header was used. It then becomes a runt packet.
>>
> Thank you for the clarification, this really helped me out.
> 
>> Florian had to fix this problem recently.
>>
>> http://patchwork.ozlabs.org/patch/836534/
>>
> But seems like this patch was never accepted, instead
> brcm_tag_xmit_ll() does it if I am understanding it correctly.
> similarly to this ksz_xmit() is taking care of padding.

net/dsa/tag_brcm.c ended up doing the padding because that was a more
generic and central location:

https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/tree/net/dsa/tag_brcm.c#n73

> 
>> You probably need something similar for the cpsw.
>>
> looking at the tag_ksz.c in xmit function this is taken care of

I agree, this should be padding packets correctly, can you still
instrument cpsw to make sure that what comes to its ndo_start_xmit() is
ETH_ZLEN + tag_len or more?

> 
> /* For Ingress (Host -> KSZ), 2 bytes are added before FCS.
>  * ---------------------------------------------------------------------------
>  * DA(6bytes)|SA(6bytes)|....|Data(nbytes)|tag0(1byte)|tag1(1byte)|FCS(4bytes)
>  * ---------------------------------------------------------------------------
>  * tag0 : Prioritization (not used now)
>  * tag1 : each bit represents port (eg, 0x01=port1, 0x02=port2, 0x10=port5)
>  *
>  * For Egress (KSZ -> Host), 1 byte is added before FCS.
>  * ---------------------------------------------------------------------------
>  * DA(6bytes)|SA(6bytes)|....|Data(nbytes)|tag0(1byte)|FCS(4bytes)
>  * ---------------------------------------------------------------------------
>  * tag0 : zero-based value represents port
>  *      (eg, 0x00=port1, 0x02=port3, 0x06=port7)
>  */
> 
> #define    KSZ_INGRESS_TAG_LEN    2
> #define    KSZ_EGRESS_TAG_LEN    1
> 
> static struct sk_buff *ksz_xmit(struct sk_buff *skb, struct net_device *dev)
> {
>     struct dsa_slave_priv *p = netdev_priv(dev);
>     struct sk_buff *nskb;
>     int padlen;
>     u8 *tag;
> 
>     padlen = (skb->len >= ETH_ZLEN) ? 0 : ETH_ZLEN - skb->len;
> 
>     if (skb_tailroom(skb) >= padlen + KSZ_INGRESS_TAG_LEN) {
>         /* Let dsa_slave_xmit() free skb */
>         if (__skb_put_padto(skb, skb->len + padlen, false))
>             return NULL;
> 
>         nskb = skb;
>     } else {
>         nskb = alloc_skb(NET_IP_ALIGN + skb->len +
>                  padlen + KSZ_INGRESS_TAG_LEN, GFP_ATOMIC);
>         if (!nskb)
>             return NULL;
>         skb_reserve(nskb, NET_IP_ALIGN);
> 
>         skb_reset_mac_header(nskb);
>         skb_set_network_header(nskb,
>                        skb_network_header(skb) - skb->head);
>         skb_set_transport_header(nskb,
>                      skb_transport_header(skb) - skb->head);
>         skb_copy_and_csum_dev(skb, skb_put(nskb, skb->len));
> 
>         /* Let skb_put_padto() free nskb, and let dsa_slave_xmit() free
>          * skb
>          */
>         if (skb_put_padto(nskb, nskb->len + padlen))
>             return NULL;
> 
>         consume_skb(skb);
>     }
> 
>     tag = skb_put(nskb, KSZ_INGRESS_TAG_LEN);
>     tag[0] = 0;
>     tag[1] = 1 << p->dp->index; /* destination port */
> 
>     return nskb;
> }
> 
> Cheers,
> --Prabhakar Lad
> 


-- 
Florian

Re: [Query]: DSA Understanding

[Lad, Prabhakar]

Hi Florian,

On Fri, Aug 10, 2018 at 6:36 PM Florian Fainelli <f.fainelli@gmail.com> wrote:
>
> On 08/10/2018 04:26 AM, Lad, Prabhakar wrote:
> > Hi Andrew,
> >
> > On Thu, Aug 9, 2018 at 6:23 PM Andrew Lunn <andrew@lunn.ch> wrote:
> >>
> >>> Its coming from the switch lan4 I have attached the png, where
> >>> C4:F3:12:08:FE:7F is
> >>> the mac of lan4, which is broadcast to ff:ff:ff:ff:ff:ff, which is
> >>> causing rx counter on
> >>> PC to go up.
> >>
> >> So, big packets are making it from the switch to the PC. But the small
> >> ARP packets are not.
> >>
> >> This is what Florian was suggesting.
> >>
> >> ARP packets are smaller than 64 bytes, which is the minimum packet
> >> size for Ethernet. Any packets smaller than 64 bytes are called runt
> >> packets. They have to be padded upto 64 bytes in order to make them
> >> valid. Otherwise the destination, or any switch along the path, might
> >> throw them away.
> >>
> >> What could be happening is that the CSPW driver or hardware is padding
> >> the packet to 64 bytes. But that packet has a DSA header in it. The
> >> switch removes the header, recalculate the checksum and sends the
> >> packet. It is now either 4 or 8 bytes smaller, depending on what DSA
> >> header was used. It then becomes a runt packet.
> >>
> > Thank you for the clarification, this really helped me out.
> >
> >> Florian had to fix this problem recently.
> >>
> >> http://patchwork.ozlabs.org/patch/836534/
> >>
> > But seems like this patch was never accepted, instead
> > brcm_tag_xmit_ll() does it if I am understanding it correctly.
> > similarly to this ksz_xmit() is taking care of padding.
>
> net/dsa/tag_brcm.c ended up doing the padding because that was a more
> generic and central location:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/tree/net/dsa/tag_brcm.c#n73
>
> >
> >> You probably need something similar for the cpsw.
> >>
> > looking at the tag_ksz.c in xmit function this is taken care of
>
> I agree, this should be padding packets correctly, can you still
> instrument cpsw to make sure that what comes to its ndo_start_xmit() is
> ETH_ZLEN + tag_len or more?
>
Yes I can confirm the skb->len is always >= 62 (ETH_ZLEN + 2)

Cheers,
--Prabhakar

Re: [Query]: DSA Understanding

[Andrew Lunn]

> > I agree, this should be padding packets correctly, can you still
> > instrument cpsw to make sure that what comes to its ndo_start_xmit() is
> > ETH_ZLEN + tag_len or more?
> >
> Yes I can confirm the skb->len is always >= 62 (ETH_ZLEN + 2)

Which switch are you using?

Marvell switches use either 4 or 8 bytes of tag. Broadcom has 4, KSZ
has 1 for packets going to the switch, lan9303 has 4, mtd uses 4, qca
has 2.

    Andrew

Re: [Query]: DSA Understanding

[Lad, Prabhakar]

Hi Andrew/Florain,

On Mon, Aug 13, 2018 at 2:38 PM Andrew Lunn <andrew@lunn.ch> wrote:
>
> > > I agree, this should be padding packets correctly, can you still
> > > instrument cpsw to make sure that what comes to its ndo_start_xmit() is
> > > ETH_ZLEN + tag_len or more?
> > >
> > Yes I can confirm the skb->len is always >= 62 (ETH_ZLEN + 2)
>
> Which switch are you using?
>
> Marvell switches use either 4 or 8 bytes of tag. Broadcom has 4, KSZ
> has 1 for packets going to the switch, lan9303 has 4, mtd uses 4, qca
> has 2.
>
I am using the KSZ switch. for Ingress it has 1 byte and for Egress it
has 2 bytes.
I came across patch [1] and padded 2 more bytes in ksz_xmit() and I was
successfully able to ping from lan4 to PC. Thank you very much for
your guidance/support.

Now I have stumbled into a different issue:

Case 1 Works:
=================
lan0 = 192.168.0.1
PC1 = 192.168.0.10
For the above ping works from both directions.

CASE 2 Doesn’t Work:
=========================
lan0 = 192.168.0.1
PC1 = 192.168.0.10
lan4 = 192.168.0.4
PC2 = 192.168.0.11

Ping from lan0 to PC1 and PC1 to lan0 works
But ping from PC2 to lan4 and lan4 to PC2 fails.

CASE 3 Works:
=========================
lan0 = 192.168.0.1
PC1 = 192.168.0.10
lan4 = 192.168.4.4
PC2 = 192.168.4.11

With the above setup ping works.

[Query] Why does ping fail in case 2. Any thoughts what I am missing here ?
or is it the expected behaviour ?

[1] https://lore.kernel.org/patchwork/patch/851457/

Cheers,
--Prabhakar Lad

Re: [Query]: DSA Understanding

[Florian Fainelli]

On 08/13/2018 08:58 AM, Lad, Prabhakar wrote:
> Hi Andrew/Florain,
> 
> On Mon, Aug 13, 2018 at 2:38 PM Andrew Lunn <andrew@lunn.ch> wrote:
>>
>>>> I agree, this should be padding packets correctly, can you still
>>>> instrument cpsw to make sure that what comes to its ndo_start_xmit() is
>>>> ETH_ZLEN + tag_len or more?
>>>>
>>> Yes I can confirm the skb->len is always >= 62 (ETH_ZLEN + 2)
>>
>> Which switch are you using?
>>
>> Marvell switches use either 4 or 8 bytes of tag. Broadcom has 4, KSZ
>> has 1 for packets going to the switch, lan9303 has 4, mtd uses 4, qca
>> has 2.
>>
> I am using the KSZ switch. for Ingress it has 1 byte and for Egress it
> has 2 bytes.
> I came across patch [1] and padded 2 more bytes in ksz_xmit() and I was
> successfully able to ping from lan4 to PC. Thank you very much for
> your guidance/support.
> 
> Now I have stumbled into a different issue:
> 
> Case 1 Works:
> =================
> lan0 = 192.168.0.1
> PC1 = 192.168.0.10
> For the above ping works from both directions.
> 
> CASE 2 Doesn’t Work:
> =========================
> lan0 = 192.168.0.1
> PC1 = 192.168.0.10
> lan4 = 192.168.0.4
> PC2 = 192.168.0.11
> 
> Ping from lan0 to PC1 and PC1 to lan0 works
> But ping from PC2 to lan4 and lan4 to PC2 fails.
> 
> CASE 3 Works:
> =========================
> lan0 = 192.168.0.1
> PC1 = 192.168.0.10
> lan4 = 192.168.4.4
> PC2 = 192.168.4.11
> 
> With the above setup ping works.
> 
> [Query] Why does ping fail in case 2. Any thoughts what I am missing here ?
> or is it the expected behaviour ?

For case 2, what I suspect is happening is that the machine that has
lan1/lan4, because you have put lan1/lan4 in the same subnet, does not
know how to respond to PC2 because it is unable to select an appropriate
output interface. In such cases, you might have to add an explicit /32
route that forces telling the kernel that PC2 is accessible via lan2.

Andrew, do you see an other explanation for that?

> 
> [1] https://lore.kernel.org/patchwork/patch/851457/

This patch works, but I think it is still working by "accident" in that
if you have both VLAN tags + KSZ tag, you would likely still be too
short by a few bytes.
-- 
Florian

Re: [Query]: DSA Understanding

[Lad, Prabhakar]

Hi Florian,

On Mon, Aug 13, 2018 at 7:57 PM Florian Fainelli <f.fainelli@gmail.com> wrote:
>
> On 08/13/2018 08:58 AM, Lad, Prabhakar wrote:
> > Hi Andrew/Florain,
> >
> > On Mon, Aug 13, 2018 at 2:38 PM Andrew Lunn <andrew@lunn.ch> wrote:
> >>
> >>>> I agree, this should be padding packets correctly, can you still
> >>>> instrument cpsw to make sure that what comes to its ndo_start_xmit() is
> >>>> ETH_ZLEN + tag_len or more?
> >>>>
> >>> Yes I can confirm the skb->len is always >= 62 (ETH_ZLEN + 2)
> >>
> >> Which switch are you using?
> >>
> >> Marvell switches use either 4 or 8 bytes of tag. Broadcom has 4, KSZ
> >> has 1 for packets going to the switch, lan9303 has 4, mtd uses 4, qca
> >> has 2.
> >>
> > I am using the KSZ switch. for Ingress it has 1 byte and for Egress it
> > has 2 bytes.
> > I came across patch [1] and padded 2 more bytes in ksz_xmit() and I was
> > successfully able to ping from lan4 to PC. Thank you very much for
> > your guidance/support.
> >
> > Now I have stumbled into a different issue:
> >
> > Case 1 Works:
> > =================
> > lan0 = 192.168.0.1
> > PC1 = 192.168.0.10
> > For the above ping works from both directions.
> >
> > CASE 2 Doesn’t Work:
> > =========================
> > lan0 = 192.168.0.1
> > PC1 = 192.168.0.10
> > lan4 = 192.168.0.4
> > PC2 = 192.168.0.11
> >
> > Ping from lan0 to PC1 and PC1 to lan0 works
> > But ping from PC2 to lan4 and lan4 to PC2 fails.
> >
> > CASE 3 Works:
> > =========================
> > lan0 = 192.168.0.1
> > PC1 = 192.168.0.10
> > lan4 = 192.168.4.4
> > PC2 = 192.168.4.11
> >
> > With the above setup ping works.
> >
> > [Query] Why does ping fail in case 2. Any thoughts what I am missing here ?
> > or is it the expected behaviour ?
>
> For case 2, what I suspect is happening is that the machine that has
> lan1/lan4, because you have put lan1/lan4 in the same subnet, does not
> know how to respond to PC2 because it is unable to select an appropriate
> output interface. In such cases, you might have to add an explicit /32
> route that forces telling the kernel that PC2 is accessible via lan2.
>
That did the trick thank you, following is my setup
lan0 = 192.168.0.1
PC1 = 192.168.0.10
lan4 = 192.168.0.4
PC2 = 192.168.0.11

route add 192.168.0.11  gw 192.168.0.4
route add 192.168.0.10  gw 192.168.0.1

And now ping works either ways. Is this setup for adding the route a valid way
to do it ? Or is there some standard way I need to follow which I am missing.

> Andrew, do you see an other explanation for that?
>
> >
> > [1] https://lore.kernel.org/patchwork/patch/851457/
>
> This patch works, but I think it is still working by "accident" in that
> if you have both VLAN tags + KSZ tag, you would likely still be too
> short by a few bytes.
>
too take care of this now I making sure the skb->len is padded to VLAN_ETH_ZLEN
in tag_ksz.c ksz_xmit() function.

Cheers,
--Prabhakar Lad