DSA

DSA

[Dave Richards]

Hello,

I am building a prototype for a new product based on a Lanner, Inc. embedded PC.  It is an Intel Celeron-based system with two host I210 GbE chips connected to 2 MV88E6172 chips (one NIC to one switch).  Everything appears to show up hardware-wise.  My question is, what is the next step?  How does DSA know which NICs are intended to be masters?  Is this supposed to be auto-detected or is this knowledge supposed to be communicated explicitly.  Reading through the DSA driver code I see that there is a check of the OF property list for the device for a "label"/"cpu" property/value pair that needs to be present.  Who sets this and when?

I'm sorry for this basic question, but Google has not enlightened me.

	Thanks!

	Dave


Dave Richards
VP Software Engineering
Impinj, Inc
400 Fairview Ave N. #1200
Seattle, WA 
O: (206) 812-9863

Re: DSA

[Florian Fainelli]

Hello,

On 04/27/2018 11:10 AM, Dave Richards wrote:
> Hello,
> 
> I am building a prototype for a new product based on a Lanner, Inc. embedded PC.  It is an Intel Celeron-based system with two host I210 GbE chips connected to 2 MV88E6172 chips (one NIC to one switch).  Everything appears to show up hardware-wise.  My question is, what is the next step?  How does DSA know which NICs are intended to be masters?  Is this supposed to be auto-detected or is this knowledge supposed to be communicated explicitly.  Reading through the DSA driver code I see that there is a check of the OF property list for the device for a "label"/"cpu" property/value pair that needs to be present.  Who sets this and when?

On system where Device Tree can be used, we expect you to declare all
relevant peripherals in Device Tree and those would include the Ethernet
controller (i210) and the Ethernet switches. An example of this can be
found here:

On x86, there is not an universal use of Device Tree, so we can use
something along these lines to register an Ethernet switch through DSA:

https://github.com/lunn/linux/commit/34055b931848545b6ba11ee50b88e89aeb02c9a5

There might be a way for you to use a conjuction of DMI Match entries to
match your specific board design and based on that run the DSA switch
registration code, indicating the port mapping and the Ethernet
controller to be used as a "master device.
-- 
Florian

Re: DSA

[Andrew Lunn]

On Fri, Apr 27, 2018 at 06:10:55PM +0000, Dave Richards wrote:
> Hello,
> 
> I am building a prototype for a new product based on a Lanner, Inc. embedded PC.  It is an Intel Celeron-based system with two host I210 GbE chips connected to 2 MV88E6172 chips (one NIC to one switch).  Everything appears to show up hardware-wise.  My question is, what is the next step?  How does DSA know which NICs are intended to be masters?  Is this supposed to be auto-detected or is this knowledge supposed to be communicated explicitly.  Reading through the DSA driver code I see that there is a check of the OF property list for the device for a "label"/"cpu" property/value pair that needs to be present.  Who sets this and when?

Hi Dave

Since you are on Intel, you don't have simple access to Device
tree. So you need to use platform data instead. Or possibly start
hacking on ACPI support for DSA. For the moment, i would suggest
platform data.

I'm also working on a similar setup, intel CPU connected to an
MV88E6532. I have some work in progress code i can share with you,
which i want to submit for inclusion to mainline in the next few
weeks.  This adds platform data support to the mv88e6xxx driver, and
will give you an idea how you link the MAC to the switch.

What MDIO bus do you connect the switches to? The i210 MDIO bus? If
so, this is going to cause you a problem. The igb driver ignores the
Linux MDIO and PHY code, and does it all itself. DSA assumes the
switch can be accessed using Linux standard MDIO interfaces. So you
have going to have to hack on the igb driver to make it use standard
MDIO.

	Andrew

Re: DSA

[Wyse, Chris]

--

Chris Wyse

Embedded Software Development
(203) 888-7914 ext 203

Canoga Perkins
100 Bank St,
Seymour, CT 06483

On Mon, 2018-04-30 at 14:50 +0200, Andrew Lunn wrote:
> On Fri, Apr 27, 2018 at 06:10:55PM +0000, Dave Richards wrote:
>
> Hello,
>
> I am building a prototype for a new product based on a Lanner, Inc.
> embedded PC.  It is an Intel Celeron-based system with two host I210
> GbE chips connected to 2 MV88E6172 chips (one NIC to one
> switch).  Everything appears to show up hardware-wise.  My question
> is, what is the next step?  How does DSA know which NICs are intended
> to be masters?  Is this supposed to be auto-detected or is this
> knowledge supposed to be communicated explicitly.  Reading through
> the DSA driver code I see that there is a check of the OF property
> list for the device for a "label"/"cpu" property/value pair that
> needs to be present.  Who sets this and when?
>
> Hi Dave
>
> Since you are on Intel, you don't have simple access to Device
> tree. So you need to use platform data instead. Or possibly start
> hacking on ACPI support for DSA. For the moment, i would suggest
> platform data.
>
> I'm also working on a similar setup, intel CPU connected to an
> MV88E6532. I have some work in progress code i can share with you,
> which i want to submit for inclusion to mainline in the next few
> weeks.  This adds platform data support to the mv88e6xxx driver, and
> will give you an idea how you link the MAC to the switch.
>
> What MDIO bus do you connect the switches to? The i210 MDIO bus? If
> so, this is going to cause you a problem. The igb driver ignores the
> Linux MDIO and PHY code, and does it all itself. DSA assumes the
> switch can be accessed using Linux standard MDIO interfaces. So you
> have going to have to hack on the igb driver to make it use standard
> MDIO.
>
> Andrew
>

I have a board that uses the Intel i210, and I'd like it be the DSA
master.  I'm looking for suggestions on how to proceed.

My configuration is an Intel E3950 CPU running Linux 4.19.62, using
UEFI/ACPI.  The board has a Xilinx FPGA that supports SFP & QSFP
devices.  The SFP ports use the standard SFP driver & phylink.  The
QSFP ports use a modified version of the SFP driver.  It also includes
an interface to an Intel i210 ethernet.

We use device tree overlays to load the information for the devices
supported by the FPGA, then load an MFD FPGA driver that instantiates
platform drivers for each of those devices.  One of the drivers that
gets loaded is a DSA driver that has the SFP & QSFP devices as its
slaves.  The intent is to use the Intel i210 on the master port of the
DSA driver.

At first glance, I believe I need to complete these tasks:
  1.  Create a device tree node for the i210, providing information on
the already loaded driver, that can be used by the DSA driver.
  2.  Obtain or update a i210 driver that will work with DSA

I'm open to any suggestions on how to proceed.  We're relatively time
limited - so I'm hoping to find a quick solution, even if we only use
it for the short-term.

Regarding the i210 driver, I looked through the latest version of the
driver.  I didn't see any commits from you for the igb driver
changes.  Did they ever get mainlined?  If not, would you please give
me access to your modified version of the driver source?

Thanks.

Chris Wyse

________________________________

Canoga Perkins
20600 Prairie Street
Chatsworth, CA 91311
(818) 718-6300

This e-mail and any attached document(s) is confidential and is intended only for the review of the party to whom it is addressed. If you have received this transmission in error, please notify the sender immediately and discard the original message and any attachment(s).

Re: DSA

[Andrew Lunn]

> I have a board that uses the Intel i210, and I'd like it be the DSA
> master.  I'm looking for suggestions on how to proceed.
> 
> My configuration is an Intel E3950 CPU running Linux 4.19.62,

Hi Chris

That is old. Can you update to 5.10.22? If you plan submitting any
kernel patches, you will need to be using net-next.

> using
> UEFI/ACPI.  The board has a Xilinx FPGA that supports SFP & QSFP
> devices.  The SFP ports use the standard SFP driver & phylink.  The
> QSFP ports use a modified version of the SFP driver.

Rusell King has some currently out of tree patches for QSFP. Since he
wrote the SFP code, you should talk to him, see what is needed to get
his changes merged.

> It also includes an interface to an Intel i210 ethernet.
> 
> We use device tree overlays to load the information for the devices
> supported by the FPGA, then load an MFD FPGA driver that instantiates
> platform drivers for each of those devices.  One of the drivers that
> gets loaded is a DSA driver that has the SFP & QSFP devices as its
> slaves.  The intent is to use the Intel i210 on the master port of the
> DSA driver.

So the switch is inside the FPGA? What is the control path for this
switch? The biggest problem with i210 is its MDIO bus. MDIO is often
used to control an Ethernet switch. But if your switch is inside the
FPGA, i guess you are not using MDIO?

> At first glance, I believe I need to complete these tasks:
>   1.  Create a device tree node for the i210, providing information on
> the already loaded driver, that can be used by the DSA driver.
>   2.  Obtain or update a i210 driver that will work with DSA

Take a look at arch/arm/boot/dts/imx6qdl-zii-rdu2.dtsi

&pcie {
        pinctrl-names = "default";
        pinctrl-0 = <&pinctrl_pcie>;
        reset-gpio = <&gpio7 12 GPIO_ACTIVE_LOW>;
        status = "okay";

        host@0 {
                reg = <0 0 0 0 0>;

                #address-cells = <3>;
                #size-cells = <2>;

                i210: i210@0 {
                        reg = <0 0 0 0 0>;
                };
        };
};

So this is an i210 on the PCIe bus of an IMX6.

It never made it into mailine, but we did have a setup where instead
of using the FEC Ethernet controller inside the IMX6, we used the i210
as the master. The usual phandle to this i210 just worked. The switch
was managed via MDIO from the FEC, or bit banging.

So assuming you switch control is not via the i210 MDIO bus, with the
correct DT, it should 'just work'.

      Andrew

Re: DSA

[Wyse, Chris]

On Tue, 2021-03-09 at 17:46 +0100, Andrew Lunn wrote:
>
> > I have a board that uses the Intel i210, and I'd like it be the DSA
> > master.  I'm looking for suggestions on how to proceed.
> >
> > My configuration is an Intel E3950 CPU running Linux 4.19.62,
>
> Hi Chris
>
> That is old. Can you update to 5.10.22? If you plan submitting any
> kernel patches, you will need to be using net-next.

First -- thanks for the quick response.  I'm not as quick as you...

Anyway, yes - the kernel is old.  I can't update it right now, but
we're planning a new version that would use a current kernel.

>
> > using
> > UEFI/ACPI.  The board has a Xilinx FPGA that supports SFP & QSFP
> > devices.  The SFP ports use the standard SFP driver & phylink.  The
> > QSFP ports use a modified version of the SFP driver.
>
> Rusell King has some currently out of tree patches for QSFP. Since he
> wrote the SFP code, you should talk to him, see what is needed to get
> his changes merged.

That's great information.  We searched all over for QSFP support, but
ended up writing our own.  I'll get in touch with Rusell.

>
> > It also includes an interface to an Intel i210 ethernet.
> >
> > We use device tree overlays to load the information for the devices
> > supported by the FPGA, then load an MFD FPGA driver that
> instantiates
> > platform drivers for each of those devices.  One of the drivers
> that
> > gets loaded is a DSA driver that has the SFP & QSFP devices as its
> > slaves.  The intent is to use the Intel i210 on the master port of
> the
> > DSA driver.
>
> So the switch is inside the FPGA? What is the control path for this
> switch? The biggest problem with i210 is its MDIO bus. MDIO is often
> used to control an Ethernet switch. But if your switch is inside the
> FPGA, i guess you are not using MDIO?
>
Yes, the switch is inside the FPGA.  The i210 is connected to the CPU
via a PCIe bus.  It's connected to a PHY which is connected to a
modified EMACLite IP module in the FPGA which only supports MDIO, and
to a custom MAC implemented in the FPGA.  The FPGA logic directs
packets to the appropriate ports of the switch, adding and removing
tags as appropriate.  The switch contains multiple MACs, which are
connected to SFPs and QSFPs (not shown).

Assuming the drawing below doesn't get corrupted with line breaks, it
is a pretty good representation of our system.

+-----------------------------+
|                             |
|  | |     +-----------+      |  +--------+
|  |P|     |           | MDI  |  |BroadCom|
|  |C+-----+   i210    +------+->|  PHY   |
|  |I|     |           |      |  |        |
|  |e|     +-----------+      |  +-+----+-+
|  | |                        |    |    |
|  |B|     +----------+-----+ |    |    |
|  |u|     |DSA Driver|     | |    |    |  MDIO Only
|  |s+-----+   TAG\x7f\x7f\x7f\x7f    | \x7fCPU | |    |    +-------------+
|  | |     | Protocol |     | |    |                  |
|  | |     +----------+-+-+-+ |    |                  |
|                       |P|   |    |                  |
+-----------------------+C+---+    |                  |
                        |I|        |                  |
                        |e|        |                  |
                        | |        | SGMII            |
                        |B|        +-----+            |
                        |u|              |            |
                        |s|              |            |
                        | |              v            v
+-----------------+-----+-+------+--+----------+-+------------+----+
|                 | PCIe Control |  |Custom MAC| | EMAC Lite  |    |
|  +---------+    | Block        |  |(No MDIO) | | IP Module  |    |
|  |Switch   |    +-+------------+  +-+--------+ +------------+    |
|  |Registers+------+                 |   ^                        |
|  +---------+                        |   |                        |
|              \x7f       +---------------+   |                        |
|                      |                  |                        |
|                      |          +-------+       +----------+-----+
|                      |          |               |          |     |
|                      |    +-----+-----+         |          | MAC |
|                      |    | Map & Add |         |          |     |
|   F P G A            |    | Port Tags |<--------+          +-----+
|                      |    +-----------+         |          |     |
|                      |                          |   Switch | MAC |
|                      |    +-----------+         |          |     |
|                      +--->| Del & Map +-------->|          +-----+
|                           | Port Tags |         |          |     |
|                           +-----------+         |          | MAC |
|                                                 |          |     |
|                                                 +----------+-----+
|                                                                  |
+------------------------------------------------------------------+
> > At first glance, I believe I need to complete these tasks:
> >   1.  Create a device tree node for the i210, providing information
> on
> > the already loaded driver, that can be used by the DSA driver.
> >   2.  Obtain or update a i210 driver that will work with DSA
>
> Take a look at arch/arm/boot/dts/imx6qdl-zii-rdu2.dtsi
>
> &pcie {
>         pinctrl-names = "default";
>         pinctrl-0 = <&pinctrl_pcie>;
>         reset-gpio = <&gpio7 12 GPIO_ACTIVE_LOW>;
>         status = "okay";
>
>         host@0 {
>                 reg = <0 0 0 0 0>;
>
>                 #address-cells = <3>;
>                 #size-cells = <2>;
>
>                 i210: i210@0 {
>                         reg = <0 0 0 0 0>;
>                 };
>         };
> };
>
I'll look at this, but one thing I see initially is that there are
references to other nodes that are not present in our device tree
overlay.  The overlay solely supports the IP modules in the FPGA.  Both
of our PCIe buses are handled via the ACPI table.  I'm not sure how to
handle something that already has an ACPI node.

Definitely a bit lost on how to proceed with the combination of ACPI &
device tree.
> So this is an i210 on the PCIe bus of an IMX6.
>
> It never made it into mailine, but we did have a setup where instead
> of using the FEC Ethernet controller inside the IMX6, we used the
> i210
> as the master. The usual phandle to this i210 just worked. The switch
> was managed via MDIO from the FEC, or bit banging.
>
> So assuming you switch control is not via the i210 MDIO bus, with the
> correct DT, it should 'just work'.
>
>       Andrew
--

Chris Wyse


Embedded Software Development
(203) 888-7914 ext 203


Canoga Perkins
100 Bank St,
Seymour, CT 06483


________________________________

Canoga Perkins
20600 Prairie Street
Chatsworth, CA 91311
(818) 718-6300

This e-mail and any attached document(s) is confidential and is intended only for the review of the party to whom it is addressed. If you have received this transmission in error, please notify the sender immediately and discard the original message and any attachment(s).

Re: DSA

[Andrew Lunn]

> > Take a look at arch/arm/boot/dts/imx6qdl-zii-rdu2.dtsi
> >
> > &pcie {
> >         pinctrl-names = "default";
> >         pinctrl-0 = <&pinctrl_pcie>;
> >         reset-gpio = <&gpio7 12 GPIO_ACTIVE_LOW>;
> >         status = "okay";
> >
> >         host@0 {
> >                 reg = <0 0 0 0 0>;
> >
> >                 #address-cells = <3>;
> >                 #size-cells = <2>;
> >
> >                 i210: i210@0 {
> >                         reg = <0 0 0 0 0>;
> >                 };
> >         };
> > };
> >
> I'll look at this, but one thing I see initially is that there are
> references to other nodes that are not present in our device tree
> overlay.  The overlay solely supports the IP modules in the FPGA.  Both
> of our PCIe buses are handled via the ACPI table.  I'm not sure how to
> handle something that already has an ACPI node.

Overlay is also possibly too late. Maybe. I guess you need the DT
available at the time the PCIe controller probes the bus. The core
PCIe code then pokes around in the DT and finds the node which
corresponds to the device on the bus. You might be able to work around
this with pci hotplugging? Load the overlay, and then trigger a hot
unplug/plug of the i210 via files in /sys? The relevant bit of code is
pci_set_of_node() which appears to get called independent of ACPI or
DT.

Otherwise you need to go the platform driver route. What works for mv88e6xxx is

static struct dsa_mv88e6xxx_pdata dsa_mv88e6xxx_pdata = {
        .cd = {
                .port_names[0] = NULL,
                .port_names[1] = "cpu",
                .port_names[2] = "red",
                .port_names[3] = "blue",
                .port_names[4] = "green",
                .port_names[5] = NULL,
                .port_names[6] = NULL,
                .port_names[7] = NULL,
                .port_names[8] = "waic0",
        },
        .compatible = "marvell,mv88e6190",
        .enabled_ports = BIT(1) | BIT(2) | BIT(3) | BIT(4) | BIT(8),
        .eeprom_len = 65536,
};

static const struct mdio_board_info bdinfo = {
        .bus_id = "gpio-0",
        .modalias = "mv88e6085",
        .mdio_addr = 0,
        .platform_data = &dsa_mv88e6xxx_pdata,
};

        dsa_mv88e6xxx_pdata.netdev = dev_get_by_name(&init_net, "eth0");
        if (!dsa_mv88e6xxx_pdata.netdev) {
                dev_err(dev, "Error finding Ethernet device\n");
                return -ENODEV;
        }

        err = mdiobus_register_board_info(&bdinfo, 1);
        if (err) {
                dev_err(dev, "Error setting up MDIO board info\n");
                goto out;
        }

On this device, there is a bit-banging MDIO driver. The MDIO core has
the needed code to associate the mdio_board_info to the bus, such that
after the bus probes, it adds a platform device on that bus, the
switch. The mv88e6xxx gets the dsa_mv88e6xxx_pdata, containing the
name of the Ethernet interface. You can probably do something similar
in your MFD for the FPGA.

This a bit fragile. systemd can come in and rename your interface from
eth0 to enp1s0, and then dev_get_by_name(). The advantage of DT is
that the name does not matter, you point directly at the device.

The other problem with this is you don't have a DT representation of
the switch, making it hard to use phylink for the SFPs, etc. So
getting overlays working would be best.

     Andrew

Re: DSA

[Wyse, Chris]

Subject: Re: DSA

> On Mon, 30 Apr 2018 14:50:30 +0200, Andrew Lunn wrote:
>
> > On Fri, Apr 27, 2018 at 06:10:55PM +0000, Dave Richards wrote:
> > > Hello,
> > >
> > > I am building a prototype for a new product based on a Lanner,
Inc. embedded PC.
> > > It is an Intel Celeron-based system with two host I210 GbE chips
connected to 2
> > > MV88E6172 chips (one NIC to one switch).  Everything appears to
show up
> > > hardware-wise.  My question is, what is the next step?  How does
DSA know which
> > > NICs are intended to be masters?  Is this supposed to be auto-
detected or is this
> > > knowledge supposed to be communicated explicitly.  Reading
through the DSA driver
> > > code I see that there is a check of the OF property list for the
device for a
> > > "label"/"cpu" property/value pair that needs to be present.  Who
sets this and when?
> >
> > Hi Dave
> >
> > Since you are on Intel, you don't have simple access to Device
> > tree. So you need to use platform data instead. Or possibly start
> > hacking on ACPI support for DSA. For the moment, i would suggest
> > platform data.
> >
> > I'm also working on a similar setup, intel CPU connected to an
> > MV88E6532. I have some work in progress code i can share with you,
> > which i want to submit for inclusion to mainline in the next few
> > weeks.  This adds platform data support to the mv88e6xxx driver,
and
> > will give you an idea how you link the MAC to the switch.
> >
> > What MDIO bus do you connect the switches to? The i210 MDIO bus? If
> > so, this is going to cause you a problem. The igb driver ignores
the
> > Linux MDIO and PHY code, and does it all itself. DSA assumes the
> > switch can be accessed using Linux standard MDIO interfaces. So you
> > have going to have to hack on the igb driver to make it use
standard
> > MDIO.
> >
> > Andrew
>
> > > Take a look at arch/arm/boot/dts/imx6qdl-zii-rdu2.dtsi
> > >
> > > &pcie {
> > >         pinctrl-names = "default";
> > >         pinctrl-0 = <&pinctrl_pcie>;
> > >         reset-gpio = <&gpio7 12 GPIO_ACTIVE_LOW>;
> > >         status = "okay";
> > >
> > >         host@0 {
> > >                 reg = <0 0 0 0 0>;
> > >
> > >                 #address-cells = <3>;
> > >                 #size-cells = <2>;
> > >
> > >                 i210: i210@0 {
> > >                         reg = <0 0 0 0 0>;
> > >                 };
> > >         };
> > > };
> > >
> > I'll look at this, but one thing I see initially is that there are
> > references to other nodes that are not present in our device tree
> > overlay.  The overlay solely supports the IP modules in the
FPGA.  Both
> > of our PCIe buses are handled via the ACPI table.  I'm not sure how
to
> > handle something that already has an ACPI node.
>
> Overlay is also possibly too late.

The current boot sequence is below:
1. i210 driver loads
2. Device Tree overlay installed (in addition to ACPI)
   - Includes DSA switch and ports, but master port was incorrectly
specified as the EMACLite IP module, which had a DT node
   - No DT node for the i210
3. MFD driver reads DT overlay and instantiates supporting devices and
the DSA driver

My thought was to create a DT entry for the i210 in the overlay, even
though the driver is already loaded via ACPI.  The DT overlay node for
the i210 would provide any needed information to the DSA driver.  It
would be essentially a reference to the already loaded driver.

If I could do that, the sequence above would remain the same.  The only
difference would be the additional DT overlay node for the i210 and
referenced devices.  I'm not an ACPI or DT expert, so I'm not sure what
types of issues I'd run into with this approach.

> Maybe. I guess you need the DT
> available at the time the PCIe controller probes the bus. The core
> PCIe code then pokes around in the DT and finds the node which
> corresponds to the device on the bus. You might be able to work
around
> this with pci hotplugging?

I think hot-plugging would work, but I'm not sure I see the need, since
the DSA driver would not be instantiated until the MFD driver is
loaded.

> Load the overlay, and then trigger a hot
> unplug/plug of the i210 via files in /sys? The relevant bit of code
is
> pci_set_of_node() which appears to get called independent of ACPI or
> DT.

So I think you're saying that the DT node for the i210 should work fine
if I emulate a PCI hotplug event.  Won't there be an issue since there
are ACPI nodes for the i210 and referenced devices?  I could unload the
drivers, then issue the hotplug event.

>
> Otherwise you need to go the platform driver route. What works for
mv88e6xxx is
>
> static struct dsa_mv88e6xxx_pdata dsa_mv88e6xxx_pdata = {
>         .cd = {
>                 .port_names[0] = NULL,
>                 .port_names[1] = "cpu",
>                 .port_names[2] = "red",
>                 .port_names[3] = "blue",
>                 .port_names[4] = "green",
>                 .port_names[5] = NULL,
>                 .port_names[6] = NULL,
>                 .port_names[7] = NULL,
>                 .port_names[8] = "waic0",
>         },
>         .compatible = "marvell,mv88e6190",
>         .enabled_ports = BIT(1) | BIT(2) | BIT(3) | BIT(4) | BIT(8),
>         .eeprom_len = 65536,
> };
>
> static const struct mdio_board_info bdinfo = {
>         .bus_id = "gpio-0",
>         .modalias = "mv88e6085",
>         .mdio_addr = 0,
>         .platform_data = &dsa_mv88e6xxx_pdata,
> };
>
>         dsa_mv88e6xxx_pdata.netdev = dev_get_by_name(&init_net,
"eth0");
>         if (!dsa_mv88e6xxx_pdata.netdev) {
>                 dev_err(dev, "Error finding Ethernet device\n");
>                 return -ENODEV;
>         }
>
>         err = mdiobus_register_board_info(&bdinfo, 1);
>         if (err) {
>                 dev_err(dev, "Error setting up MDIO board info\n");
>                 goto out;
>         }
>
> On this device, there is a bit-banging MDIO driver. The MDIO core has
> the needed code to associate the mdio_board_info to the bus, such
that
> after the bus probes, it adds a platform device on that bus, the
> switch. The mv88e6xxx gets the dsa_mv88e6xxx_pdata, containing the
> name of the Ethernet interface. You can probably do something similar
> in your MFD for the FPGA.

I think I need to see what you did for that.  That might be the right
approach for our purposes.  I could add a DT overlay node with the name
of the interface.  We're in a controlled environment - the name won't
change.  The DSA driver (modified) could read the i210 DT node,
retrieve the device name, then call dev_get_by_name().

>
> This a bit fragile. systemd can come in and rename your interface
from
> eth0 to enp1s0, and then dev_get_by_name(). The advantage of DT is
> that the name does not matter, you point directly at the device.
>
> The other problem with this is you don't have a DT representation of
> the switch, making it hard to use phylink for the SFPs, etc. So
> getting overlays working would be best.

Hmm...  Overlays are working except for the missing master port node.
I think I would use the same DT representation I have now, except to
reference the new i210 node as the master.



________________________________

Canoga Perkins
20600 Prairie Street
Chatsworth, CA 91311
(818) 718-6300

This e-mail and any attached document(s) is confidential and is intended only for the review of the party to whom it is addressed. If you have received this transmission in error, please notify the sender immediately and discard the original message and any attachment(s).

Re: DSA

[Andrew Lunn]

> The current boot sequence is below:
> 1. i210 driver loads
> 2. Device Tree overlay installed (in addition to ACPI)
>    - Includes DSA switch and ports, but master port was incorrectly
> specified as the EMACLite IP module, which had a DT node
>    - No DT node for the i210
> 3. MFD driver reads DT overlay and instantiates supporting devices and
> the DSA driver
> 
> My thought was to create a DT entry for the i210 in the overlay, even
> though the driver is already loaded via ACPI.  The DT overlay node for
> the i210 would provide any needed information to the DSA driver.  It
> would be essentially a reference to the already loaded driver.

I don't think what works. The normal sequence is that the PCI bus is
probed and devices found. At that point in time, the PCI core looks in
DT and finds the DT node associated with the device, and assigns
dev->of_node to point to the DT node. The device is then registered
with the device core. It will go off and try to find a driver of the
device, probe it, etc.

Sometime later, the DSA driver will probe. The phandle in device tree
is turned into a pointer. And then all devices registered for the
netdev class are walked to see if any have dev->of_node set to the
node.

So for this sequence to work, the overload needs to be loaded at the
point the PCI bus is scanned for devices. Now, there could be some
magic going on when an overlay is loaded, scanning the DT for devices
which have already loaded, and assigning there dev->of_node? I've no
idea, i've not used overlays. You probably want to add some printk()
into pci_set_of_node().

The idea with hotplug is that i guess it should rescan the PCI bus. So
that should cause pci_set_of_node() to be called, and now there is a
DT node for the device.

So i suggest you scatter some printk() in the PCI code, and
of_find_net_device_by_node() and the functions it calls to see what is
really going on, do i have the sequencing correct.

       Andrew

Re: DSA

[Wyse, Chris]

> > The current boot sequence is below:
> > 1. i210 driver loads
> > 2. Device Tree overlay installed (in addition to ACPI)
> >    - Includes DSA switch and ports, but master port was incorrectly
> > specified as the EMACLite IP module, which had a DT node
> >    - No DT node for the i210
> > 3. MFD driver reads DT overlay and instantiates supporting devices
and
> > the DSA driver
> >
> > My thought was to create a DT entry for the i210 in the overlay,
even
> > though the driver is already loaded via ACPI.  The DT overlay node
for
> > the i210 would provide any needed information to the DSA
driver.  It
> > would be essentially a reference to the already loaded driver.
>
> I don't think what works. The normal sequence is that the PCI bus is
> probed and devices found. At that point in time, the PCI core looks
in
> DT and finds the DT node associated with the device, and assigns
> dev->of_node to point to the DT node. The device is then registered
> with the device core. It will go off and try to find a driver of the
> device, probe it, etc.
>
> Sometime later, the DSA driver will probe. The phandle in device tree
> is turned into a pointer. And then all devices registered for the
> netdev class are walked to see if any have dev->of_node set to the
> node.
>
> So for this sequence to work, the overload needs to be loaded at the
> point the PCI bus is scanned for devices. Now, there could be some
> magic going on when an overlay is loaded, scanning the DT for devices
> which have already loaded, and assigning there dev->of_node? I've no
> idea, i've not used overlays. You probably want to add some printk()
> into pci_set_of_node().
>
> The idea with hotplug is that i guess it should rescan the PCI bus.
So
> that should cause pci_set_of_node() to be called, and now there is a
> DT node for the device.
>
> So i suggest you scatter some printk() in the PCI code, and
> of_find_net_device_by_node() and the functions it calls to see what
is
> really going on, do i have the sequencing correct.
>
Great information and ideas.  Thank you.

Currently, the PCI bus is probed at boot, and the FPGA the i210 are
detected.  The i210 driver gets loaded at boot, the DT overlays are
applied, and then we load the MFD driver for the FPGA.  The MFD driver
loads all supporting drivers, then loads the DSA driver.

I'm wondering if I could build the igb driver as a module (not quite
the same as hotplug, but seems like it would work), then load it after
the DT overlay has been added.  The PCI core would need to look for a
device tree node prior to an ACPI node (or I'd need to modify it to
work that way).  Otherwise, I believe that the DT code would be
skipped.

I _think_ that I could use something similar to your host@0 node for
the i210 DT definition and addressing.  I wouldn't need a node for the
PCIe as long as I provided the correct addressing for the i210 node.

---- Another thought ---

The loading of the device tree overlays was based on the device tree
unit test code.  From that, I created a simple device driver that takes
a dtbo file and loads it.  Instead of creating it as a module, I could
instantiate it prior to the the PCI driver.  If the PCI code favors the
DT over the ACPI, the maybe it would load the driver for me via DT
without any other changes.

----

Looking into the code now to see how these options would play out, with
emphasis on the areas you mentioned.  Please let me know if I've
misinterpreted any of your comments or if it's apparent that I've
missed a critical piece.  I'm not an expert in any of these area, and
struggling a bit to provide coherent and reasonable responses, so your
feedback is welcome.

Chris

________________________________

Canoga Perkins
20600 Prairie Street
Chatsworth, CA 91311
(818) 718-6300

This e-mail and any attached document(s) is confidential and is intended only for the review of the party to whom it is addressed. If you have received this transmission in error, please notify the sender immediately and discard the original message and any attachment(s).

Re: DSA

[George McCollister]

[snip]

I'm using an i210 w/ mdio to connect to cascaded mv88e6390 switches on
an Intel platform. I have patches based on patches sent in 2014 that
allow use of mdio and configuration with device-tree. I'd like to get
them upstreamed but there are some unresolved problems. For one I have
no way of testing external mdio connected phys since my board only
connects to a switch. Second it's not clear what the "correct" mode
configured in the EEPROM should be SGMII, 1000BASE_KX or SERDES. Based
on 2014 mailing list discussion I'm using SGMII in the EEPROM
configuration but it ends up having to change the link mode to
E1000_CTRL_EXT_LINK_MODE_PCIE_SERDES when a phy is not detected or it
won't link with the mv88e6390. I think there may be push back from
upstream in changing the behavior of various modes even if external
mdio is enabled in the EEPROM due to fear that vendors may have
enabled it for no reason (needs more investigation). I've also had to
add a hack to the mv88e6xxx driver to stop calling
phylink_helper_basex_speed, otherwise it always tries to use
2500base-x which doesn't work with the i210. I posted to the ML about
this but no one bothered to reply.

My Intel Atom platform happens to be using u-boot rather than UEFI so
I'm able to apply DT overlays based on detected PCI subsystem VID/ID
before the linux kernel is started.

I'm on #linuxswitch on Freenode if there is anything you want to
discuss off the list. If any of you have interest in getting these
patches upstream it would be great to work with someone on it.

You can find what I have here:
https://github.com/gmccollister/linux/tree/net-i210-mv88e6390

Regards,
George McCollister