[PATCH 0/4] MIPS: ralink: pci: driver for Pcie controller in MT7621 SoCs

[PATCH 0/4] MIPS: ralink: pci: driver for Pcie controller in MT7621 SoCs

[Sergio Paracuellos]

MediaTek MT7621 PCIe subsys supports single Root complex (RC)
with 3 Root Ports. Each Root Ports supports a Gen1 1-lane Link.
Topology is as follows:

                          MT7621 PCIe HOST Topology

                                   .-------.
                                   |       |
                                   |  CPU  |
                                   |       |
                                   '-------'
                                       |
                                       |
                                       |
                                       v
                              .------------------.
                  .-----------|  HOST/PCI Bridge |------------.
                  |           '------------------'            |     Type1 
         BUS0     |                     |                     |    Access 
                  v                     v                     v    On Bus0
          .-------------.        .-------------.       .-------------.
          | VIRTUAL P2P |        | VIRTUAL P2P |       | VIRTUAL P2P |
          |    BUS0     |        |    BUS0     |       |    BUS0     |
          |    DEV0     |        |    DEV1     |       |    DEV2     |
          '-------------'        '-------------'       '-------------'
    Type0        |          Type0       |         Type0       |
   Access   BUS1 |         Access   BUS2|        Access   BUS3|
   On Bus1       v         On Bus2      v        On Bus3      v
           .----------.           .----------.          .----------.
           | Device 0 |           | Device 0 |          | Device 0 |
           |  Func 0  |           |  Func 0  |          |  Func 0  |
           '----------'           '----------'          '----------'

This driver has been very long time in staging and I have been cleaning
it from its first versions where there was code kaos and PCI_LEGACY support.
Original code came probably from openWRT based on mediatek's SDK code. There
is no documentation at all about the mt7621 PCI subsystem.
I have been cleaning it targeting mt7621 SoC which is the one I use in
my GNUBee PC1 board and HiLink HLK-MT7621A evaluation board.

Now I think is clean enough to be moved into 'arch/mips/pci'.

This driver also uses already mainlined pci phy driver located in
'drivers/phy/ralink/phy-mt7621-pci.c'. There are two instances of
the phy being the first one dual ported for pci0 and pci1, and the 
second one not dual ported dedicated to pci2. Because of writing twice 
some phy registers of the dual-ported one sometimes become in not
confident boot cycles we have to take care of this when device link
is checked here in controller driver. We power on the dual ported-phy
if there is something connected in pcie0 or pcie1. In the same manner
we have to properly disable it only if nothing is connected in of both
pcie0 and pci1 slots.

Another thing that must be mentioned is that this driver uses IO
in physical address 0x001e160000. IO_SPACE_LIMIT for MIPS is 0xffff
so some generic PCI functions (like of_pci_range_to_resource) won't
work and the resource ranges part for IO is set manually.

I had already sent binding documentation to be reviewed but I am
include also here with the driver itself and this cover letter
to make easy review process.

Best regards,
    Sergio Paracuellos

Sergio Paracuellos (4):
  dt-bindings: mt7621-pci: PCIe binding documentation for MT7621 SoCs
  MIPS: pci: Add driver for MT7621 PCIe controller
  staging: mt7621-pci: remove driver from staging
  MAINTAINERS: add myself as maintainer of the MT7621 PCI controller
    driver

 .../bindings/pci/mediatek,mt7621-pci.yaml     | 149 ++++++++++++++++++
 MAINTAINERS                                   |   6 +
 arch/mips/pci/Makefile                        |   1 +
 .../mt7621-pci => arch/mips/pci}/pci-mt7621.c |   0
 arch/mips/ralink/Kconfig                      |   9 +-
 drivers/staging/Kconfig                       |   2 -
 drivers/staging/Makefile                      |   1 -
 drivers/staging/mt7621-pci/Kconfig            |   8 -
 drivers/staging/mt7621-pci/Makefile           |   2 -
 drivers/staging/mt7621-pci/TODO               |   4 -
 .../mt7621-pci/mediatek,mt7621-pci.txt        | 104 ------------
 11 files changed, 164 insertions(+), 122 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml
 rename {drivers/staging/mt7621-pci => arch/mips/pci}/pci-mt7621.c (100%)
 delete mode 100644 drivers/staging/mt7621-pci/Kconfig
 delete mode 100644 drivers/staging/mt7621-pci/Makefile
 delete mode 100644 drivers/staging/mt7621-pci/TODO
 delete mode 100644 drivers/staging/mt7621-pci/mediatek,mt7621-pci.txt

-- 
2.25.1

[PATCH 1/4] dt-bindings: mt7621-pci: PCIe binding documentation for MT7621 SoCs

[Sergio Paracuellos]

Add device tree binding documentation for PCIe in MT7621 SoCs.

Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
---
 .../bindings/pci/mediatek,mt7621-pci.yaml     | 149 ++++++++++++++++++
 1 file changed, 149 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml

diff --git a/Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml b/Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml
new file mode 100644
index 000000000000..7f5f9d583032
--- /dev/null
+++ b/Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml
@@ -0,0 +1,149 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/pci/mediatek,mt7621-pci.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: MediaTek MT7621 PCIe controller
+
+maintainers:
+  - Sergio Paracuellos <sergio.paracuellos@gmail.com>
+
+description: |+
+  MediaTek MT7621 PCIe subsys supports single Root complex (RC)
+  with 3 Root Ports. Each Root Ports supports a Gen1 1-lane Link
+
+allOf:
+  - $ref: /schemas/pci/pci-bus.yaml#
+
+properties:
+  compatible:
+    const: mediatek,mt7621-pci
+
+  reg:
+    items:
+      - description: host-pci bridge registers
+      - description: pcie port 0 RC control registers
+      - description: pcie port 1 RC control registers
+      - description: pcie port 2 RC control registers
+
+  ranges:
+    maxItems: 2
+
+  resets:
+    items:
+      - description: pcie port 0 reset.
+      - description: pcie port 1 reset.
+      - description: pcie port 2 reset.
+
+  reset-names:
+    items:
+      - const: pcie0
+      - const: pcie1
+      - const: pcie2
+
+  clocks:
+    items:
+      - description: pcie port 0 clock.
+      - description: pcie port 1 clock.
+      - description: pcie port 2 clock.
+
+  clock-names:
+    items:
+      - const: pcie0
+      - const: pcie1
+      - const: pcie2
+
+  phys:
+    items:
+      - description: Dual-ported phy for pcie port 0 and 1.
+      - description: Phy for pcie port 2.
+
+  phy-names:
+    items:
+      - const: pcie-phy0
+      - const: pcie-phy2
+
+required:
+  - compatible
+  - reg
+  - ranges
+  - "#interrupt-cells"
+  - interrupt-map-mask
+  - interrupt-map
+  - resets
+  - reset-names
+  - clocks
+  - clock-names
+  - phys
+  - phy-names
+  - reset-gpios
+
+unevaluatedProperties: false
+
+examples:
+  - |
+    #include <dt-bindings/gpio/gpio.h>
+    #include <dt-bindings/interrupt-controller/mips-gic.h>
+
+    pcie: pcie@1e140000 {
+        compatible = "mediatek,mt7621-pci";
+        reg = <0x1e140000 0x100>,
+              <0x1e142000 0x100>,
+              <0x1e143000 0x100>,
+              <0x1e144000 0x100>;
+
+        #address-cells = <3>;
+        #size-cells = <2>;
+        pinctrl-names = "default";
+        pinctrl-0 = <&pcie_pins>;
+        device_type = "pci";
+        ranges = <0x02000000 0 0x00000000 0x60000000 0 0x10000000>,  /* pci memory */
+                 <0x01000000 0 0x00000000 0x1e160000 0 0x00010000>;  /* io space */
+        #interrupt-cells = <1>;
+        interrupt-map-mask = <0xF800 0 0 0>;
+        interrupt-map = <0x0000 0 0 0 &gic GIC_SHARED 4 IRQ_TYPE_LEVEL_HIGH>,
+                        <0x0800 0 0 0 &gic GIC_SHARED 24 IRQ_TYPE_LEVEL_HIGH>,
+                        <0x1000 0 0 0 &gic GIC_SHARED 25 IRQ_TYPE_LEVEL_HIGH>;
+        resets = <&rstctrl 24>, <&rstctrl 25>, <&rstctrl 26>;
+        reset-names = "pcie0", "pcie1", "pcie2";
+        clocks = <&clkctrl 24>, <&clkctrl 25>, <&clkctrl 26>;
+        clock-names = "pcie0", "pcie1", "pcie2";
+        phys = <&pcie0_phy 1>, <&pcie2_phy 0>;
+        phy-names = "pcie-phy0", "pcie-phy2";
+        reset-gpios = <&gpio 19 GPIO_ACTIVE_LOW>;
+
+        pcie@0,0 {
+            reg = <0x0000 0 0 0 0>;
+            #address-cells = <3>;
+            #size-cells = <2>;
+            device_type = "pci";
+            #interrupt-cells = <1>;
+            interrupt-map-mask = <0 0 0 0>;
+            interrupt-map = <0 0 0 0 &gic GIC_SHARED 4 IRQ_TYPE_LEVEL_HIGH>;
+            ranges;
+        };
+
+        pcie@1,0 {
+            reg = <0x0800 0 0 0 0>;
+            #address-cells = <3>;
+            #size-cells = <2>;
+            device_type = "pci";
+            #interrupt-cells = <1>;
+            interrupt-map-mask = <0 0 0 0>;
+            interrupt-map = <0 0 0 0 &gic GIC_SHARED 24 IRQ_TYPE_LEVEL_HIGH>;
+            ranges;
+        };
+
+        pcie@2,0 {
+            reg = <0x1000 0 0 0 0>;
+            #address-cells = <3>;
+            #size-cells = <2>;
+            device_type = "pci";
+            #interrupt-cells = <1>;
+            interrupt-map-mask = <0 0 0 0>;
+            interrupt-map = <0 0 0 0 &gic GIC_SHARED 25 IRQ_TYPE_LEVEL_HIGH>;
+            ranges;
+        };
+    };
+...
-- 
2.25.1

Re: [PATCH 1/4] dt-bindings: mt7621-pci: PCIe binding documentation for MT7621 SoCs

[Sergio Paracuellos]

Hi Rob,

On Sat, May 15, 2021 at 2:40 PM Sergio Paracuellos
<sergio.paracuellos@gmail.com> wrote:
>
> Add device tree binding documentation for PCIe in MT7621 SoCs.
>
> Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> ---
>  .../bindings/pci/mediatek,mt7621-pci.yaml     | 149 ++++++++++++++++++
>  1 file changed, 149 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml
>
> diff --git a/Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml b/Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml
> new file mode 100644
> index 000000000000..7f5f9d583032
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml
> @@ -0,0 +1,149 @@
> +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> +%YAML 1.2
> +---
> +$id: http://devicetree.org/schemas/pci/mediatek,mt7621-pci.yaml#
> +$schema: http://devicetree.org/meta-schemas/core.yaml#
> +
> +title: MediaTek MT7621 PCIe controller
> +
> +maintainers:
> +  - Sergio Paracuellos <sergio.paracuellos@gmail.com>
> +
> +description: |+
> +  MediaTek MT7621 PCIe subsys supports single Root complex (RC)
> +  with 3 Root Ports. Each Root Ports supports a Gen1 1-lane Link
> +
> +allOf:
> +  - $ref: /schemas/pci/pci-bus.yaml#
> +
> +properties:
> +  compatible:
> +    const: mediatek,mt7621-pci
> +
> +  reg:
> +    items:
> +      - description: host-pci bridge registers
> +      - description: pcie port 0 RC control registers
> +      - description: pcie port 1 RC control registers
> +      - description: pcie port 2 RC control registers
> +
> +  ranges:
> +    maxItems: 2
> +
> +  resets:
> +    items:
> +      - description: pcie port 0 reset.
> +      - description: pcie port 1 reset.
> +      - description: pcie port 2 reset.
> +
> +  reset-names:
> +    items:
> +      - const: pcie0
> +      - const: pcie1
> +      - const: pcie2
> +
> +  clocks:
> +    items:
> +      - description: pcie port 0 clock.
> +      - description: pcie port 1 clock.
> +      - description: pcie port 2 clock.
> +
> +  clock-names:
> +    items:
> +      - const: pcie0
> +      - const: pcie1
> +      - const: pcie2
> +
> +  phys:
> +    items:
> +      - description: Dual-ported phy for pcie port 0 and 1.
> +      - description: Phy for pcie port 2.
> +
> +  phy-names:
> +    items:
> +      - const: pcie-phy0
> +      - const: pcie-phy2
> +
> +required:
> +  - compatible
> +  - reg
> +  - ranges
> +  - "#interrupt-cells"
> +  - interrupt-map-mask
> +  - interrupt-map
> +  - resets
> +  - reset-names
> +  - clocks
> +  - clock-names
> +  - phys
> +  - phy-names
> +  - reset-gpios
> +
> +unevaluatedProperties: false
> +
> +examples:
> +  - |
> +    #include <dt-bindings/gpio/gpio.h>
> +    #include <dt-bindings/interrupt-controller/mips-gic.h>
> +
> +    pcie: pcie@1e140000 {
> +        compatible = "mediatek,mt7621-pci";
> +        reg = <0x1e140000 0x100>,
> +              <0x1e142000 0x100>,
> +              <0x1e143000 0x100>,
> +              <0x1e144000 0x100>;
> +
> +        #address-cells = <3>;
> +        #size-cells = <2>;
> +        pinctrl-names = "default";
> +        pinctrl-0 = <&pcie_pins>;
> +        device_type = "pci";
> +        ranges = <0x02000000 0 0x00000000 0x60000000 0 0x10000000>,  /* pci memory */
> +                 <0x01000000 0 0x00000000 0x1e160000 0 0x00010000>;  /* io space */
> +        #interrupt-cells = <1>;
> +        interrupt-map-mask = <0xF800 0 0 0>;
> +        interrupt-map = <0x0000 0 0 0 &gic GIC_SHARED 4 IRQ_TYPE_LEVEL_HIGH>,
> +                        <0x0800 0 0 0 &gic GIC_SHARED 24 IRQ_TYPE_LEVEL_HIGH>,
> +                        <0x1000 0 0 0 &gic GIC_SHARED 25 IRQ_TYPE_LEVEL_HIGH>;
> +        resets = <&rstctrl 24>, <&rstctrl 25>, <&rstctrl 26>;
> +        reset-names = "pcie0", "pcie1", "pcie2";
> +        clocks = <&clkctrl 24>, <&clkctrl 25>, <&clkctrl 26>;
> +        clock-names = "pcie0", "pcie1", "pcie2";
> +        phys = <&pcie0_phy 1>, <&pcie2_phy 0>;
> +        phy-names = "pcie-phy0", "pcie-phy2";
> +        reset-gpios = <&gpio 19 GPIO_ACTIVE_LOW>;
> +
> +        pcie@0,0 {
> +            reg = <0x0000 0 0 0 0>;
> +            #address-cells = <3>;
> +            #size-cells = <2>;
> +            device_type = "pci";
> +            #interrupt-cells = <1>;
> +            interrupt-map-mask = <0 0 0 0>;
> +            interrupt-map = <0 0 0 0 &gic GIC_SHARED 4 IRQ_TYPE_LEVEL_HIGH>;
> +            ranges;
> +        };
> +
> +        pcie@1,0 {
> +            reg = <0x0800 0 0 0 0>;
> +            #address-cells = <3>;
> +            #size-cells = <2>;
> +            device_type = "pci";
> +            #interrupt-cells = <1>;
> +            interrupt-map-mask = <0 0 0 0>;
> +            interrupt-map = <0 0 0 0 &gic GIC_SHARED 24 IRQ_TYPE_LEVEL_HIGH>;
> +            ranges;
> +        };
> +
> +        pcie@2,0 {
> +            reg = <0x1000 0 0 0 0>;
> +            #address-cells = <3>;
> +            #size-cells = <2>;
> +            device_type = "pci";
> +            #interrupt-cells = <1>;
> +            interrupt-map-mask = <0 0 0 0>;
> +            interrupt-map = <0 0 0 0 &gic GIC_SHARED 25 IRQ_TYPE_LEVEL_HIGH>;
> +            ranges;
> +        };
> +    };
> +...
> --
> 2.25.1
>

Any comments on this binding? It's been a while in devicetree
patchwork review list without comments.

Thanks in advance for your time.

Best regards,
     Sergio Paracuellos

Re: [PATCH 1/4] dt-bindings: mt7621-pci: PCIe binding documentation for MT7621 SoCs

[Rob Herring]

On Sat, May 15, 2021 at 02:40:52PM +0200, Sergio Paracuellos wrote:
> Add device tree binding documentation for PCIe in MT7621 SoCs.
> 
> Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> ---
>  .../bindings/pci/mediatek,mt7621-pci.yaml     | 149 ++++++++++++++++++
>  1 file changed, 149 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml
> 
> diff --git a/Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml b/Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml
> new file mode 100644
> index 000000000000..7f5f9d583032
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml
> @@ -0,0 +1,149 @@
> +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> +%YAML 1.2
> +---
> +$id: http://devicetree.org/schemas/pci/mediatek,mt7621-pci.yaml#
> +$schema: http://devicetree.org/meta-schemas/core.yaml#
> +
> +title: MediaTek MT7621 PCIe controller
> +
> +maintainers:
> +  - Sergio Paracuellos <sergio.paracuellos@gmail.com>
> +
> +description: |+
> +  MediaTek MT7621 PCIe subsys supports single Root complex (RC)
> +  with 3 Root Ports. Each Root Ports supports a Gen1 1-lane Link
> +
> +allOf:
> +  - $ref: /schemas/pci/pci-bus.yaml#
> +
> +properties:
> +  compatible:
> +    const: mediatek,mt7621-pci
> +
> +  reg:
> +    items:
> +      - description: host-pci bridge registers
> +      - description: pcie port 0 RC control registers
> +      - description: pcie port 1 RC control registers
> +      - description: pcie port 2 RC control registers
> +
> +  ranges:
> +    maxItems: 2
> +
> +  resets:
> +    items:
> +      - description: pcie port 0 reset.
> +      - description: pcie port 1 reset.
> +      - description: pcie port 2 reset.
> +
> +  reset-names:
> +    items:
> +      - const: pcie0
> +      - const: pcie1
> +      - const: pcie2
> +
> +  clocks:
> +    items:
> +      - description: pcie port 0 clock.
> +      - description: pcie port 1 clock.
> +      - description: pcie port 2 clock.
> +
> +  clock-names:
> +    items:
> +      - const: pcie0
> +      - const: pcie1
> +      - const: pcie2
> +
> +  phys:
> +    items:
> +      - description: Dual-ported phy for pcie port 0 and 1.
> +      - description: Phy for pcie port 2.
> +
> +  phy-names:
> +    items:
> +      - const: pcie-phy0
> +      - const: pcie-phy2

If you're going to keep the ports (and I think that's right because 
there's only a single PCI address space AFAICT), then I think you should 
move resets, clocks, and phys into each port node.

So you'll need to define 'pcie@[0-2],0' node with those properties under 
it.

> +
> +required:
> +  - compatible
> +  - reg
> +  - ranges
> +  - "#interrupt-cells"
> +  - interrupt-map-mask
> +  - interrupt-map
> +  - resets
> +  - reset-names
> +  - clocks
> +  - clock-names
> +  - phys
> +  - phy-names
> +  - reset-gpios
> +
> +unevaluatedProperties: false
> +
> +examples:
> +  - |
> +    #include <dt-bindings/gpio/gpio.h>
> +    #include <dt-bindings/interrupt-controller/mips-gic.h>
> +
> +    pcie: pcie@1e140000 {
> +        compatible = "mediatek,mt7621-pci";
> +        reg = <0x1e140000 0x100>,
> +              <0x1e142000 0x100>,
> +              <0x1e143000 0x100>,
> +              <0x1e144000 0x100>;
> +
> +        #address-cells = <3>;
> +        #size-cells = <2>;
> +        pinctrl-names = "default";
> +        pinctrl-0 = <&pcie_pins>;
> +        device_type = "pci";
> +        ranges = <0x02000000 0 0x00000000 0x60000000 0 0x10000000>,  /* pci memory */
> +                 <0x01000000 0 0x00000000 0x1e160000 0 0x00010000>;  /* io space */
> +        #interrupt-cells = <1>;
> +        interrupt-map-mask = <0xF800 0 0 0>;
> +        interrupt-map = <0x0000 0 0 0 &gic GIC_SHARED 4 IRQ_TYPE_LEVEL_HIGH>,
> +                        <0x0800 0 0 0 &gic GIC_SHARED 24 IRQ_TYPE_LEVEL_HIGH>,
> +                        <0x1000 0 0 0 &gic GIC_SHARED 25 IRQ_TYPE_LEVEL_HIGH>;
> +        resets = <&rstctrl 24>, <&rstctrl 25>, <&rstctrl 26>;
> +        reset-names = "pcie0", "pcie1", "pcie2";
> +        clocks = <&clkctrl 24>, <&clkctrl 25>, <&clkctrl 26>;
> +        clock-names = "pcie0", "pcie1", "pcie2";
> +        phys = <&pcie0_phy 1>, <&pcie2_phy 0>;
> +        phy-names = "pcie-phy0", "pcie-phy2";
> +        reset-gpios = <&gpio 19 GPIO_ACTIVE_LOW>;
> +
> +        pcie@0,0 {
> +            reg = <0x0000 0 0 0 0>;
> +            #address-cells = <3>;
> +            #size-cells = <2>;
> +            device_type = "pci";
> +            #interrupt-cells = <1>;
> +            interrupt-map-mask = <0 0 0 0>;
> +            interrupt-map = <0 0 0 0 &gic GIC_SHARED 4 IRQ_TYPE_LEVEL_HIGH>;
> +            ranges;
> +        };
> +
> +        pcie@1,0 {
> +            reg = <0x0800 0 0 0 0>;
> +            #address-cells = <3>;
> +            #size-cells = <2>;
> +            device_type = "pci";
> +            #interrupt-cells = <1>;
> +            interrupt-map-mask = <0 0 0 0>;
> +            interrupt-map = <0 0 0 0 &gic GIC_SHARED 24 IRQ_TYPE_LEVEL_HIGH>;
> +            ranges;
> +        };
> +
> +        pcie@2,0 {
> +            reg = <0x1000 0 0 0 0>;
> +            #address-cells = <3>;
> +            #size-cells = <2>;
> +            device_type = "pci";
> +            #interrupt-cells = <1>;
> +            interrupt-map-mask = <0 0 0 0>;
> +            interrupt-map = <0 0 0 0 &gic GIC_SHARED 25 IRQ_TYPE_LEVEL_HIGH>;
> +            ranges;
> +        };
> +    };
> +...
> -- 
> 2.25.1

Re: [PATCH 1/4] dt-bindings: mt7621-pci: PCIe binding documentation for MT7621 SoCs

[Sergio Paracuellos]

Hi Rob,

Thanks for the review.

On Fri, Jun 4, 2021 at 9:35 PM Rob Herring <robh@kernel.org> wrote:
>
> On Sat, May 15, 2021 at 02:40:52PM +0200, Sergio Paracuellos wrote:
> > Add device tree binding documentation for PCIe in MT7621 SoCs.
> >
> > Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > ---
> >  .../bindings/pci/mediatek,mt7621-pci.yaml     | 149 ++++++++++++++++++
> >  1 file changed, 149 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml
> >
> > diff --git a/Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml b/Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml
> > new file mode 100644
> > index 000000000000..7f5f9d583032
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml
> > @@ -0,0 +1,149 @@
> > +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> > +%YAML 1.2
> > +---
> > +$id: http://devicetree.org/schemas/pci/mediatek,mt7621-pci.yaml#
> > +$schema: http://devicetree.org/meta-schemas/core.yaml#
> > +
> > +title: MediaTek MT7621 PCIe controller
> > +
> > +maintainers:
> > +  - Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > +
> > +description: |+
> > +  MediaTek MT7621 PCIe subsys supports single Root complex (RC)
> > +  with 3 Root Ports. Each Root Ports supports a Gen1 1-lane Link
> > +
> > +allOf:
> > +  - $ref: /schemas/pci/pci-bus.yaml#
> > +
> > +properties:
> > +  compatible:
> > +    const: mediatek,mt7621-pci
> > +
> > +  reg:
> > +    items:
> > +      - description: host-pci bridge registers
> > +      - description: pcie port 0 RC control registers
> > +      - description: pcie port 1 RC control registers
> > +      - description: pcie port 2 RC control registers
> > +
> > +  ranges:
> > +    maxItems: 2
> > +
> > +  resets:
> > +    items:
> > +      - description: pcie port 0 reset.
> > +      - description: pcie port 1 reset.
> > +      - description: pcie port 2 reset.
> > +
> > +  reset-names:
> > +    items:
> > +      - const: pcie0
> > +      - const: pcie1
> > +      - const: pcie2
> > +
> > +  clocks:
> > +    items:
> > +      - description: pcie port 0 clock.
> > +      - description: pcie port 1 clock.
> > +      - description: pcie port 2 clock.
> > +
> > +  clock-names:
> > +    items:
> > +      - const: pcie0
> > +      - const: pcie1
> > +      - const: pcie2
> > +
> > +  phys:
> > +    items:
> > +      - description: Dual-ported phy for pcie port 0 and 1.
> > +      - description: Phy for pcie port 2.
> > +
> > +  phy-names:
> > +    items:
> > +      - const: pcie-phy0
> > +      - const: pcie-phy2
>
> If you're going to keep the ports (and I think that's right because
> there's only a single PCI address space AFAICT), then I think you should
> move resets, clocks, and phys into each port node.
>
> So you'll need to define 'pcie@[0-2],0' node with those properties under
> it.

Ok I will move these stuff to each port node. So each port node will
be similar to:

pcie@0,0 {
    reg = <0x0000 0 0 0 0>;
    #address-cells = <3>;
    #size-cells = <2>;
    device_type = "pci";
    #interrupt-cells = <1>;
    clocks = <&clkctrl 24>;
    resets = <&rstctrl 24>;
    phys = <&pcie0_phy 1>;
    interrupt-map-mask = <0 0 0 0>;
    interrupt-map = <0 0 0 0 &gic GIC_SHARED 4 IRQ_TYPE_LEVEL_HIGH>;
    ranges;
};

How can I be sure by schema that the clocks, reset and phy properties
are in each port node if I move them from the parent? By now each port
node is just validating because of ' $ref:
/schemas/pci/pci-bus.yaml#'.

Thanks in advance for your time.

Best regards,
    Sergio Paracuellos

>
> > +
> > +required:
> > +  - compatible
> > +  - reg
> > +  - ranges
> > +  - "#interrupt-cells"
> > +  - interrupt-map-mask
> > +  - interrupt-map
> > +  - resets
> > +  - reset-names
> > +  - clocks
> > +  - clock-names
> > +  - phys
> > +  - phy-names
> > +  - reset-gpios
> > +
> > +unevaluatedProperties: false
> > +
> > +examples:
> > +  - |
> > +    #include <dt-bindings/gpio/gpio.h>
> > +    #include <dt-bindings/interrupt-controller/mips-gic.h>
> > +
> > +    pcie: pcie@1e140000 {
> > +        compatible = "mediatek,mt7621-pci";
> > +        reg = <0x1e140000 0x100>,
> > +              <0x1e142000 0x100>,
> > +              <0x1e143000 0x100>,
> > +              <0x1e144000 0x100>;
> > +
> > +        #address-cells = <3>;
> > +        #size-cells = <2>;
> > +        pinctrl-names = "default";
> > +        pinctrl-0 = <&pcie_pins>;
> > +        device_type = "pci";
> > +        ranges = <0x02000000 0 0x00000000 0x60000000 0 0x10000000>,  /* pci memory */
> > +                 <0x01000000 0 0x00000000 0x1e160000 0 0x00010000>;  /* io space */
> > +        #interrupt-cells = <1>;
> > +        interrupt-map-mask = <0xF800 0 0 0>;
> > +        interrupt-map = <0x0000 0 0 0 &gic GIC_SHARED 4 IRQ_TYPE_LEVEL_HIGH>,
> > +                        <0x0800 0 0 0 &gic GIC_SHARED 24 IRQ_TYPE_LEVEL_HIGH>,
> > +                        <0x1000 0 0 0 &gic GIC_SHARED 25 IRQ_TYPE_LEVEL_HIGH>;
> > +        resets = <&rstctrl 24>, <&rstctrl 25>, <&rstctrl 26>;
> > +        reset-names = "pcie0", "pcie1", "pcie2";
> > +        clocks = <&clkctrl 24>, <&clkctrl 25>, <&clkctrl 26>;
> > +        clock-names = "pcie0", "pcie1", "pcie2";
> > +        phys = <&pcie0_phy 1>, <&pcie2_phy 0>;
> > +        phy-names = "pcie-phy0", "pcie-phy2";
> > +        reset-gpios = <&gpio 19 GPIO_ACTIVE_LOW>;
> > +
> > +        pcie@0,0 {
> > +            reg = <0x0000 0 0 0 0>;
> > +            #address-cells = <3>;
> > +            #size-cells = <2>;
> > +            device_type = "pci";
> > +            #interrupt-cells = <1>;
> > +            interrupt-map-mask = <0 0 0 0>;
> > +            interrupt-map = <0 0 0 0 &gic GIC_SHARED 4 IRQ_TYPE_LEVEL_HIGH>;
> > +            ranges;
> > +        };
> > +
> > +        pcie@1,0 {
> > +            reg = <0x0800 0 0 0 0>;
> > +            #address-cells = <3>;
> > +            #size-cells = <2>;
> > +            device_type = "pci";
> > +            #interrupt-cells = <1>;
> > +            interrupt-map-mask = <0 0 0 0>;
> > +            interrupt-map = <0 0 0 0 &gic GIC_SHARED 24 IRQ_TYPE_LEVEL_HIGH>;
> > +            ranges;
> > +        };
> > +
> > +        pcie@2,0 {
> > +            reg = <0x1000 0 0 0 0>;
> > +            #address-cells = <3>;
> > +            #size-cells = <2>;
> > +            device_type = "pci";
> > +            #interrupt-cells = <1>;
> > +            interrupt-map-mask = <0 0 0 0>;
> > +            interrupt-map = <0 0 0 0 &gic GIC_SHARED 25 IRQ_TYPE_LEVEL_HIGH>;
> > +            ranges;
> > +        };
> > +    };
> > +...
> > --
> > 2.25.1

Re: [PATCH 1/4] dt-bindings: mt7621-pci: PCIe binding documentation for MT7621 SoCs

[Sergio Paracuellos]

Hi Rob,

On Fri, Jun 4, 2021 at 11:32 PM Sergio Paracuellos
<sergio.paracuellos@gmail.com> wrote:
>
> Hi Rob,
>
> Thanks for the review.
>
> On Fri, Jun 4, 2021 at 9:35 PM Rob Herring <robh@kernel.org> wrote:
> >
> > On Sat, May 15, 2021 at 02:40:52PM +0200, Sergio Paracuellos wrote:
> > > Add device tree binding documentation for PCIe in MT7621 SoCs.
> > >
> > > Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > > ---
> > >  .../bindings/pci/mediatek,mt7621-pci.yaml     | 149 ++++++++++++++++++
> > >  1 file changed, 149 insertions(+)
> > >  create mode 100644 Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml
> > >
> > > diff --git a/Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml b/Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml
> > > new file mode 100644
> > > index 000000000000..7f5f9d583032
> > > --- /dev/null
> > > +++ b/Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml
> > > @@ -0,0 +1,149 @@
> > > +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> > > +%YAML 1.2
> > > +---
> > > +$id: http://devicetree.org/schemas/pci/mediatek,mt7621-pci.yaml#
> > > +$schema: http://devicetree.org/meta-schemas/core.yaml#
> > > +
> > > +title: MediaTek MT7621 PCIe controller
> > > +
> > > +maintainers:
> > > +  - Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > > +
> > > +description: |+
> > > +  MediaTek MT7621 PCIe subsys supports single Root complex (RC)
> > > +  with 3 Root Ports. Each Root Ports supports a Gen1 1-lane Link
> > > +
> > > +allOf:
> > > +  - $ref: /schemas/pci/pci-bus.yaml#
> > > +
> > > +properties:
> > > +  compatible:
> > > +    const: mediatek,mt7621-pci
> > > +
> > > +  reg:
> > > +    items:
> > > +      - description: host-pci bridge registers
> > > +      - description: pcie port 0 RC control registers
> > > +      - description: pcie port 1 RC control registers
> > > +      - description: pcie port 2 RC control registers
> > > +
> > > +  ranges:
> > > +    maxItems: 2
> > > +
> > > +  resets:
> > > +    items:
> > > +      - description: pcie port 0 reset.
> > > +      - description: pcie port 1 reset.
> > > +      - description: pcie port 2 reset.
> > > +
> > > +  reset-names:
> > > +    items:
> > > +      - const: pcie0
> > > +      - const: pcie1
> > > +      - const: pcie2
> > > +
> > > +  clocks:
> > > +    items:
> > > +      - description: pcie port 0 clock.
> > > +      - description: pcie port 1 clock.
> > > +      - description: pcie port 2 clock.
> > > +
> > > +  clock-names:
> > > +    items:
> > > +      - const: pcie0
> > > +      - const: pcie1
> > > +      - const: pcie2
> > > +
> > > +  phys:
> > > +    items:
> > > +      - description: Dual-ported phy for pcie port 0 and 1.
> > > +      - description: Phy for pcie port 2.
> > > +
> > > +  phy-names:
> > > +    items:
> > > +      - const: pcie-phy0
> > > +      - const: pcie-phy2
> >
> > If you're going to keep the ports (and I think that's right because
> > there's only a single PCI address space AFAICT), then I think you should
> > move resets, clocks, and phys into each port node.
> >
> > So you'll need to define 'pcie@[0-2],0' node with those properties under
> > it.
>
> Ok I will move these stuff to each port node. So each port node will
> be similar to:
>
> pcie@0,0 {
>     reg = <0x0000 0 0 0 0>;
>     #address-cells = <3>;
>     #size-cells = <2>;
>     device_type = "pci";
>     #interrupt-cells = <1>;
>     clocks = <&clkctrl 24>;
>     resets = <&rstctrl 24>;
>     phys = <&pcie0_phy 1>;
>     interrupt-map-mask = <0 0 0 0>;
>     interrupt-map = <0 0 0 0 &gic GIC_SHARED 4 IRQ_TYPE_LEVEL_HIGH>;
>     ranges;
> };
>
> How can I be sure by schema that the clocks, reset and phy properties
> are in each port node if I move them from the parent? By now each port
> node is just validating because of ' $ref:
> /schemas/pci/pci-bus.yaml#'.
>
> Thanks in advance for your time.
>

So I think, this should be enough:

%YAML 1.2
---
$id: http://devicetree.org/schemas/pci/mediatek,mt7621-pci.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#

title: MediaTek MT7621 PCIe controller

maintainers:
  - Sergio Paracuellos <sergio.paracuellos@gmail.com>

description: |+
  MediaTek MT7621 PCIe subsys supports single Root complex (RC)
  with 3 Root Ports. Each Root Ports supports a Gen1 1-lane Link

allOf:
  - $ref: /schemas/pci/pci-bus.yaml#

properties:
  compatible:
    const: mediatek,mt7621-pci

  reg:
    items:
      - description: host-pci bridge registers
      - description: pcie port 0 RC control registers
      - description: pcie port 1 RC control registers
      - description: pcie port 2 RC control registers

  ranges:
    maxItems: 2

patternProperties:
  'pcie@[0-2],0':
    type: object
    $ref: /schemas/pci/pci-bus.yaml#

    properties:
      resets:
        maxItems: 1

      clocks:
        maxItems: 1

      phys:
        maxItems: 1

    required:
      - "#interrupt-cells"
      - interrupt-map-mask
      - interrupt-map
      - resets
      - clocks
      - phys
      - phy-names
      - ranges

    unevaluatedProperties: false

required:
  - compatible
  - reg
  - ranges
  - "#interrupt-cells"
  - interrupt-map-mask
  - interrupt-map
  - reset-gpios

unevaluatedProperties: false

examples:
  - |
    #include <dt-bindings/gpio/gpio.h>
    #include <dt-bindings/interrupt-controller/mips-gic.h>

    pcie: pcie@1e140000 {
        compatible = "mediatek,mt7621-pci";
        reg = <0x1e140000 0x100>,
              <0x1e142000 0x100>,
              <0x1e143000 0x100>,
              <0x1e144000 0x100>;

        #address-cells = <3>;
        #size-cells = <2>;
        pinctrl-names = "default";
        pinctrl-0 = <&pcie_pins>;
        device_type = "pci";
        ranges = <0x02000000 0 0x00000000 0x60000000 0 0x10000000>,
/* pci memory */
                 <0x01000000 0 0x00000000 0x1e160000 0 0x00010000>;
/* io space */
        #interrupt-cells = <1>;
        interrupt-map-mask = <0xF800 0 0 0>;
        interrupt-map = <0x0000 0 0 0 &gic GIC_SHARED 4 IRQ_TYPE_LEVEL_HIGH>,
                        <0x0800 0 0 0 &gic GIC_SHARED 24 IRQ_TYPE_LEVEL_HIGH>,
                        <0x1000 0 0 0 &gic GIC_SHARED 25 IRQ_TYPE_LEVEL_HIGH>;
        reset-gpios = <&gpio 19 GPIO_ACTIVE_LOW>;

        pcie@0,0 {
            reg = <0x0000 0 0 0 0>;
            #address-cells = <3>;
            #size-cells = <2>;
            device_type = "pci";
            #interrupt-cells = <1>;
            interrupt-map-mask = <0 0 0 0>;
            interrupt-map = <0 0 0 0 &gic GIC_SHARED 4 IRQ_TYPE_LEVEL_HIGH>;
            resets = <&rstctrl 24>;
            clocks = <&clkctrl 24>;
            phys = <&pcie0_phy 1>;
            phy-names = "pcie-phy0";
            ranges;
        };

        pcie@1,0 {
            reg = <0x0800 0 0 0 0>;
            #address-cells = <3>;
            #size-cells = <2>;
            device_type = "pci";
            #interrupt-cells = <1>;
            interrupt-map-mask = <0 0 0 0>;
            interrupt-map = <0 0 0 0 &gic GIC_SHARED 24 IRQ_TYPE_LEVEL_HIGH>;
            resets = <&rstctrl 25>;
            clocks = <&clkctrl 25>;
            phys = <&pcie0_phy 1>;
            phy-names = "pcie-phy1";
            ranges;
        };

        pcie@2,0 {
            reg = <0x1000 0 0 0 0>;
            #address-cells = <3>;
            #size-cells = <2>;
            device_type = "pci";
            #interrupt-cells = <1>;
            interrupt-map-mask = <0 0 0 0>;
            interrupt-map = <0 0 0 0 &gic GIC_SHARED 25 IRQ_TYPE_LEVEL_HIGH>;
            resets = <&rstctrl 26>;
            clocks = <&clkctrl 26>;
            phys = <&pcie2_phy 0>;
            phy-names = "pcie-phy2";
            ranges;
        };
    };
...

I will send these bindings for v2 if you are ok with them.

Thanks,
     Sergio Paracuellos

> Best regards,
>     Sergio Paracuellos
>
> >
> > > +
> > > +required:
> > > +  - compatible
> > > +  - reg
> > > +  - ranges
> > > +  - "#interrupt-cells"
> > > +  - interrupt-map-mask
> > > +  - interrupt-map
> > > +  - resets
> > > +  - reset-names
> > > +  - clocks
> > > +  - clock-names
> > > +  - phys
> > > +  - phy-names
> > > +  - reset-gpios
> > > +
> > > +unevaluatedProperties: false
> > > +
> > > +examples:
> > > +  - |
> > > +    #include <dt-bindings/gpio/gpio.h>
> > > +    #include <dt-bindings/interrupt-controller/mips-gic.h>
> > > +
> > > +    pcie: pcie@1e140000 {
> > > +        compatible = "mediatek,mt7621-pci";
> > > +        reg = <0x1e140000 0x100>,
> > > +              <0x1e142000 0x100>,
> > > +              <0x1e143000 0x100>,
> > > +              <0x1e144000 0x100>;
> > > +
> > > +        #address-cells = <3>;
> > > +        #size-cells = <2>;
> > > +        pinctrl-names = "default";
> > > +        pinctrl-0 = <&pcie_pins>;
> > > +        device_type = "pci";
> > > +        ranges = <0x02000000 0 0x00000000 0x60000000 0 0x10000000>,  /* pci memory */
> > > +                 <0x01000000 0 0x00000000 0x1e160000 0 0x00010000>;  /* io space */
> > > +        #interrupt-cells = <1>;
> > > +        interrupt-map-mask = <0xF800 0 0 0>;
> > > +        interrupt-map = <0x0000 0 0 0 &gic GIC_SHARED 4 IRQ_TYPE_LEVEL_HIGH>,
> > > +                        <0x0800 0 0 0 &gic GIC_SHARED 24 IRQ_TYPE_LEVEL_HIGH>,
> > > +                        <0x1000 0 0 0 &gic GIC_SHARED 25 IRQ_TYPE_LEVEL_HIGH>;
> > > +        resets = <&rstctrl 24>, <&rstctrl 25>, <&rstctrl 26>;
> > > +        reset-names = "pcie0", "pcie1", "pcie2";
> > > +        clocks = <&clkctrl 24>, <&clkctrl 25>, <&clkctrl 26>;
> > > +        clock-names = "pcie0", "pcie1", "pcie2";
> > > +        phys = <&pcie0_phy 1>, <&pcie2_phy 0>;
> > > +        phy-names = "pcie-phy0", "pcie-phy2";
> > > +        reset-gpios = <&gpio 19 GPIO_ACTIVE_LOW>;
> > > +
> > > +        pcie@0,0 {
> > > +            reg = <0x0000 0 0 0 0>;
> > > +            #address-cells = <3>;
> > > +            #size-cells = <2>;
> > > +            device_type = "pci";
> > > +            #interrupt-cells = <1>;
> > > +            interrupt-map-mask = <0 0 0 0>;
> > > +            interrupt-map = <0 0 0 0 &gic GIC_SHARED 4 IRQ_TYPE_LEVEL_HIGH>;
> > > +            ranges;
> > > +        };
> > > +
> > > +        pcie@1,0 {
> > > +            reg = <0x0800 0 0 0 0>;
> > > +            #address-cells = <3>;
> > > +            #size-cells = <2>;
> > > +            device_type = "pci";
> > > +            #interrupt-cells = <1>;
> > > +            interrupt-map-mask = <0 0 0 0>;
> > > +            interrupt-map = <0 0 0 0 &gic GIC_SHARED 24 IRQ_TYPE_LEVEL_HIGH>;
> > > +            ranges;
> > > +        };
> > > +
> > > +        pcie@2,0 {
> > > +            reg = <0x1000 0 0 0 0>;
> > > +            #address-cells = <3>;
> > > +            #size-cells = <2>;
> > > +            device_type = "pci";
> > > +            #interrupt-cells = <1>;
> > > +            interrupt-map-mask = <0 0 0 0>;
> > > +            interrupt-map = <0 0 0 0 &gic GIC_SHARED 25 IRQ_TYPE_LEVEL_HIGH>;
> > > +            ranges;
> > > +        };
> > > +    };
> > > +...
> > > --
> > > 2.25.1

[PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Sergio Paracuellos]

This patch adds a driver for the PCIe controller of MT7621 SoC.

Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
---
 arch/mips/pci/Makefile     |   1 +
 arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
 arch/mips/ralink/Kconfig   |   9 +-
 3 files changed, 633 insertions(+), 1 deletion(-)
 create mode 100644 arch/mips/pci/pci-mt7621.c

diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
index f3eecc065e5c..178c550739c4 100644
--- a/arch/mips/pci/Makefile
+++ b/arch/mips/pci/Makefile
@@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)	+= pci-ar2315.o
 obj-$(CONFIG_SOC_AR71XX)	+= pci-ar71xx.o
 obj-$(CONFIG_PCI_AR724X)	+= pci-ar724x.o
 obj-$(CONFIG_PCI_XTALK_BRIDGE)	+= pci-xtalk-bridge.o
+obj-$(CONFIG_PCI_MT7621)	+= pci-mt7621.o
 #
 # These are still pretty much in the old state, watch, go blind.
 #
diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
new file mode 100644
index 000000000000..fe1945819d25
--- /dev/null
+++ b/arch/mips/pci/pci-mt7621.c
@@ -0,0 +1,624 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * BRIEF MODULE DESCRIPTION
+ *     PCI init for Ralink RT2880 solution
+ *
+ * Copyright 2007 Ralink Inc. (bruce_chang@ralinktech.com.tw)
+ *
+ * May 2007 Bruce Chang
+ * Initial Release
+ *
+ * May 2009 Bruce Chang
+ * support RT2880/RT3883 PCIe
+ *
+ * May 2011 Bruce Chang
+ * support RT6855/MT7620 PCIe
+ */
+
+#include <linux/bitops.h>
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/gpio/consumer.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_address.h>
+#include <linux/of_pci.h>
+#include <linux/of_platform.h>
+#include <linux/pci.h>
+#include <linux/phy/phy.h>
+#include <linux/platform_device.h>
+#include <linux/reset.h>
+#include <linux/sys_soc.h>
+
+/* MediaTek specific configuration registers */
+#define PCIE_FTS_NUM			0x70c
+#define PCIE_FTS_NUM_MASK		GENMASK(15, 8)
+#define PCIE_FTS_NUM_L0(x)		(((x) & 0xff) << 8)
+
+/* Host-PCI bridge registers */
+#define RALINK_PCI_PCICFG_ADDR		0x0000
+#define RALINK_PCI_PCIMSK_ADDR		0x000C
+#define RALINK_PCI_CONFIG_ADDR		0x0020
+#define RALINK_PCI_CONFIG_DATA		0x0024
+#define RALINK_PCI_MEMBASE		0x0028
+#define RALINK_PCI_IOBASE		0x002C
+
+/* PCIe RC control registers */
+#define MT7621_PCIE_OFFSET		0x2000
+#define MT7621_NEXT_PORT		0x1000
+
+#define RALINK_PCI_BAR0SETUP_ADDR	0x0010
+#define RALINK_PCI_ID			0x0030
+#define RALINK_PCI_CLASS		0x0034
+#define RALINK_PCI_SUBID		0x0038
+#define RALINK_PCI_STATUS		0x0050
+
+/* Some definition values */
+#define PCIE_REVISION_ID		BIT(0)
+#define PCIE_CLASS_CODE			(0x60400 << 8)
+#define PCIE_BAR_MAP_MAX		GENMASK(30, 16)
+#define PCIE_BAR_ENABLE			BIT(0)
+#define PCIE_PORT_INT_EN(x)		BIT(20 + (x))
+#define PCIE_PORT_LINKUP		BIT(0)
+
+#define PERST_DELAY_MS			100
+
+/**
+ * struct mt7621_pcie_port - PCIe port information
+ * @base: I/O mapped register base
+ * @list: port list
+ * @pcie: pointer to PCIe host info
+ * @clk: pointer to the port clock gate
+ * @phy: pointer to PHY control block
+ * @pcie_rst: pointer to port reset control
+ * @gpio_rst: gpio reset
+ * @slot: port slot
+ * @enabled: indicates if port is enabled
+ */
+struct mt7621_pcie_port {
+	void __iomem *base;
+	struct list_head list;
+	struct mt7621_pcie *pcie;
+	struct clk *clk;
+	struct phy *phy;
+	struct reset_control *pcie_rst;
+	struct gpio_desc *gpio_rst;
+	u32 slot;
+	bool enabled;
+};
+
+/**
+ * struct mt7621_pcie - PCIe host information
+ * @base: IO Mapped Register Base
+ * @io: IO resource
+ * @mem: pointer to non-prefetchable memory resource
+ * @dev: Pointer to PCIe device
+ * @io_map_base: virtual memory base address for io
+ * @ports: pointer to PCIe port information
+ * @resets_inverted: depends on chip revision
+ * reset lines are inverted.
+ */
+struct mt7621_pcie {
+	void __iomem *base;
+	struct device *dev;
+	struct resource io;
+	struct resource *mem;
+	unsigned long io_map_base;
+	struct list_head ports;
+	bool resets_inverted;
+};
+
+static inline u32 pcie_read(struct mt7621_pcie *pcie, u32 reg)
+{
+	return readl(pcie->base + reg);
+}
+
+static inline void pcie_write(struct mt7621_pcie *pcie, u32 val, u32 reg)
+{
+	writel(val, pcie->base + reg);
+}
+
+static inline void pcie_rmw(struct mt7621_pcie *pcie, u32 reg, u32 clr, u32 set)
+{
+	u32 val = readl(pcie->base + reg);
+
+	val &= ~clr;
+	val |= set;
+	writel(val, pcie->base + reg);
+}
+
+static inline u32 pcie_port_read(struct mt7621_pcie_port *port, u32 reg)
+{
+	return readl(port->base + reg);
+}
+
+static inline void pcie_port_write(struct mt7621_pcie_port *port,
+				   u32 val, u32 reg)
+{
+	writel(val, port->base + reg);
+}
+
+static inline u32 mt7621_pci_get_cfgaddr(unsigned int bus, unsigned int slot,
+					 unsigned int func, unsigned int where)
+{
+	return (((where & 0xF00) >> 8) << 24) | (bus << 16) | (slot << 11) |
+		(func << 8) | (where & 0xfc) | 0x80000000;
+}
+
+static void __iomem *mt7621_pcie_map_bus(struct pci_bus *bus,
+					 unsigned int devfn, int where)
+{
+	struct mt7621_pcie *pcie = bus->sysdata;
+	u32 address = mt7621_pci_get_cfgaddr(bus->number, PCI_SLOT(devfn),
+					     PCI_FUNC(devfn), where);
+
+	writel(address, pcie->base + RALINK_PCI_CONFIG_ADDR);
+
+	return pcie->base + RALINK_PCI_CONFIG_DATA + (where & 3);
+}
+
+struct pci_ops mt7621_pci_ops = {
+	.map_bus	= mt7621_pcie_map_bus,
+	.read		= pci_generic_config_read,
+	.write		= pci_generic_config_write,
+};
+
+static u32 read_config(struct mt7621_pcie *pcie, unsigned int dev, u32 reg)
+{
+	u32 address = mt7621_pci_get_cfgaddr(0, dev, 0, reg);
+
+	pcie_write(pcie, address, RALINK_PCI_CONFIG_ADDR);
+	return pcie_read(pcie, RALINK_PCI_CONFIG_DATA);
+}
+
+static void write_config(struct mt7621_pcie *pcie, unsigned int dev,
+			 u32 reg, u32 val)
+{
+	u32 address = mt7621_pci_get_cfgaddr(0, dev, 0, reg);
+
+	pcie_write(pcie, address, RALINK_PCI_CONFIG_ADDR);
+	pcie_write(pcie, val, RALINK_PCI_CONFIG_DATA);
+}
+
+static inline void mt7621_rst_gpio_pcie_assert(struct mt7621_pcie_port *port)
+{
+	if (port->gpio_rst)
+		gpiod_set_value(port->gpio_rst, 1);
+}
+
+static inline void mt7621_rst_gpio_pcie_deassert(struct mt7621_pcie_port *port)
+{
+	if (port->gpio_rst)
+		gpiod_set_value(port->gpio_rst, 0);
+}
+
+static inline bool mt7621_pcie_port_is_linkup(struct mt7621_pcie_port *port)
+{
+	return (pcie_port_read(port, RALINK_PCI_STATUS) & PCIE_PORT_LINKUP) != 0;
+}
+
+static inline void mt7621_control_assert(struct mt7621_pcie_port *port)
+{
+	struct mt7621_pcie *pcie = port->pcie;
+
+	if (pcie->resets_inverted)
+		reset_control_assert(port->pcie_rst);
+	else
+		reset_control_deassert(port->pcie_rst);
+}
+
+static inline void mt7621_control_deassert(struct mt7621_pcie_port *port)
+{
+	struct mt7621_pcie *pcie = port->pcie;
+
+	if (pcie->resets_inverted)
+		reset_control_deassert(port->pcie_rst);
+	else
+		reset_control_assert(port->pcie_rst);
+}
+
+static void setup_cm_memory_region(struct mt7621_pcie *pcie)
+{
+	struct resource *mem_resource = pcie->mem;
+	struct device *dev = pcie->dev;
+	resource_size_t mask;
+
+	if (mips_cps_numiocu(0)) {
+		/*
+		 * FIXME: hardware doesn't accept mask values with 1s after
+		 * 0s (e.g. 0xffef), so it would be great to warn if that's
+		 * about to happen
+		 */
+		mask = ~(mem_resource->end - mem_resource->start);
+
+		write_gcr_reg1_base(mem_resource->start);
+		write_gcr_reg1_mask(mask | CM_GCR_REGn_MASK_CMTGT_IOCU0);
+		dev_info(dev, "PCI coherence region base: 0x%08llx, mask/settings: 0x%08llx\n",
+			 (unsigned long long)read_gcr_reg1_base(),
+			 (unsigned long long)read_gcr_reg1_mask());
+	}
+}
+
+static int mt7621_pci_parse_request_of_pci_ranges(struct pci_host_bridge *host)
+{
+	struct mt7621_pcie *pcie = pci_host_bridge_priv(host);
+	struct device *dev = pcie->dev;
+	struct device_node *node = dev->of_node;
+	struct of_pci_range_parser parser;
+	struct resource_entry *entry;
+	struct of_pci_range range;
+	LIST_HEAD(res);
+
+	if (of_pci_range_parser_init(&parser, node)) {
+		dev_err(dev, "missing \"ranges\" property\n");
+		return -EINVAL;
+	}
+
+	/*
+	 * IO_SPACE_LIMIT for MIPS is 0xffff but this platform uses IO at
+	 * upper address 0x001e160000. of_pci_range_to_resource does not work
+	 * well for MIPS platforms that don't define PCI_IOBASE, so set the IO
+	 * resource manually instead.
+	 */
+	for_each_of_pci_range(&parser, &range) {
+		switch (range.flags & IORESOURCE_TYPE_BITS) {
+		case IORESOURCE_IO:
+			pcie->io_map_base =
+				(unsigned long)ioremap(range.cpu_addr,
+						       range.size);
+			pcie->io.name = node->full_name;
+			pcie->io.flags = range.flags;
+			pcie->io.start = range.cpu_addr;
+			pcie->io.end = range.cpu_addr + range.size - 1;
+			pcie->io.parent = pcie->io.child = pcie->io.sibling = NULL;
+			set_io_port_base(pcie->io_map_base);
+			break;
+		}
+	}
+
+	entry = resource_list_first_type(&host->windows, IORESOURCE_MEM);
+	if (!entry) {
+		dev_err(dev, "Cannot get memory resource");
+		return -EINVAL;
+	}
+
+	pcie->mem = entry->res;
+	pci_add_resource(&res, &pcie->io);
+	pci_add_resource(&res, entry->res);
+	list_splice_init(&res, &host->windows);
+
+	return 0;
+}
+
+static int mt7621_pcie_parse_port(struct mt7621_pcie *pcie,
+				  int slot)
+{
+	struct mt7621_pcie_port *port;
+	struct device *dev = pcie->dev;
+	struct platform_device *pdev = to_platform_device(dev);
+	char name[10];
+
+	port = devm_kzalloc(dev, sizeof(*port), GFP_KERNEL);
+	if (!port)
+		return -ENOMEM;
+
+	port->base = devm_platform_ioremap_resource(pdev, slot + 1);
+	if (IS_ERR(port->base))
+		return PTR_ERR(port->base);
+
+	snprintf(name, sizeof(name), "pcie%d", slot);
+	port->clk = devm_clk_get(dev, name);
+	if (IS_ERR(port->clk)) {
+		dev_err(dev, "failed to get pcie%d clock\n", slot);
+		return PTR_ERR(port->clk);
+	}
+
+	snprintf(name, sizeof(name), "pcie%d", slot);
+	port->pcie_rst = devm_reset_control_get_exclusive(dev, name);
+	if (PTR_ERR(port->pcie_rst) == -EPROBE_DEFER) {
+		dev_err(dev, "failed to get pcie%d reset control\n", slot);
+		return PTR_ERR(port->pcie_rst);
+	}
+
+	snprintf(name, sizeof(name), "pcie-phy%d", slot);
+	port->phy = devm_phy_get(dev, name);
+	if (IS_ERR(port->phy) && slot != 1)
+		return PTR_ERR(port->phy);
+
+	port->gpio_rst = devm_gpiod_get_index_optional(dev, "reset", slot,
+						       GPIOD_OUT_LOW);
+	if (IS_ERR(port->gpio_rst)) {
+		dev_err(dev, "Failed to get GPIO for PCIe%d\n", slot);
+		return PTR_ERR(port->gpio_rst);
+	}
+
+	port->slot = slot;
+	port->pcie = pcie;
+
+	INIT_LIST_HEAD(&port->list);
+	list_add_tail(&port->list, &pcie->ports);
+
+	return 0;
+}
+
+static int mt7621_pcie_parse_dt(struct mt7621_pcie *pcie)
+{
+	struct device *dev = pcie->dev;
+	struct platform_device *pdev = to_platform_device(dev);
+	struct device_node *node = dev->of_node, *child;
+	int err;
+
+	pcie->base = devm_platform_ioremap_resource(pdev, 0);
+	if (IS_ERR(pcie->base))
+		return PTR_ERR(pcie->base);
+
+	for_each_available_child_of_node(node, child) {
+		int slot;
+
+		err = of_pci_get_devfn(child);
+		if (err < 0) {
+			of_node_put(child);
+			dev_err(dev, "failed to parse devfn: %d\n", err);
+			return err;
+		}
+
+		slot = PCI_SLOT(err);
+
+		err = mt7621_pcie_parse_port(pcie, slot);
+		if (err) {
+			of_node_put(child);
+			return err;
+		}
+	}
+
+	return 0;
+}
+
+static int mt7621_pcie_init_port(struct mt7621_pcie_port *port)
+{
+	struct mt7621_pcie *pcie = port->pcie;
+	struct device *dev = pcie->dev;
+	u32 slot = port->slot;
+	int err;
+
+	err = phy_init(port->phy);
+	if (err) {
+		dev_err(dev, "failed to initialize port%d phy\n", slot);
+		return err;
+	}
+
+	err = phy_power_on(port->phy);
+	if (err) {
+		dev_err(dev, "failed to power on port%d phy\n", slot);
+		phy_exit(port->phy);
+		return err;
+	}
+
+	port->enabled = true;
+
+	return 0;
+}
+
+static void mt7621_pcie_reset_assert(struct mt7621_pcie *pcie)
+{
+	struct mt7621_pcie_port *port;
+
+	list_for_each_entry(port, &pcie->ports, list) {
+		/* PCIe RC reset assert */
+		mt7621_control_assert(port);
+
+		/* PCIe EP reset assert */
+		mt7621_rst_gpio_pcie_assert(port);
+	}
+
+	msleep(PERST_DELAY_MS);
+}
+
+static void mt7621_pcie_reset_rc_deassert(struct mt7621_pcie *pcie)
+{
+	struct mt7621_pcie_port *port;
+
+	list_for_each_entry(port, &pcie->ports, list)
+		mt7621_control_deassert(port);
+}
+
+static void mt7621_pcie_reset_ep_deassert(struct mt7621_pcie *pcie)
+{
+	struct mt7621_pcie_port *port;
+
+	list_for_each_entry(port, &pcie->ports, list)
+		mt7621_rst_gpio_pcie_deassert(port);
+
+	msleep(PERST_DELAY_MS);
+}
+
+static void mt7621_pcie_init_ports(struct mt7621_pcie *pcie)
+{
+	struct device *dev = pcie->dev;
+	struct mt7621_pcie_port *port, *tmp;
+	int err;
+
+	mt7621_pcie_reset_assert(pcie);
+	mt7621_pcie_reset_rc_deassert(pcie);
+
+	list_for_each_entry_safe(port, tmp, &pcie->ports, list) {
+		u32 slot = port->slot;
+
+		if (slot == 1) {
+			port->enabled = true;
+			continue;
+		}
+
+		err = mt7621_pcie_init_port(port);
+		if (err) {
+			dev_err(dev, "Initiating port %d failed\n", slot);
+			list_del(&port->list);
+		}
+	}
+
+	mt7621_pcie_reset_ep_deassert(pcie);
+
+	tmp = NULL;
+	list_for_each_entry(port, &pcie->ports, list) {
+		u32 slot = port->slot;
+
+		if (!mt7621_pcie_port_is_linkup(port)) {
+			dev_err(dev, "pcie%d no card, disable it (RST & CLK)\n",
+				slot);
+			mt7621_control_assert(port);
+			clk_disable_unprepare(port->clk);
+			port->enabled = false;
+
+			if (slot == 0) {
+				tmp = port;
+				continue;
+			}
+
+			if (slot == 1 && tmp && !tmp->enabled)
+				phy_power_off(tmp->phy);
+		}
+	}
+}
+
+static void mt7621_pcie_enable_port(struct mt7621_pcie_port *port)
+{
+	struct mt7621_pcie *pcie = port->pcie;
+	u32 slot = port->slot;
+	u32 offset = MT7621_PCIE_OFFSET + (slot * MT7621_NEXT_PORT);
+	u32 val;
+
+	/* enable pcie interrupt */
+	val = pcie_read(pcie, RALINK_PCI_PCIMSK_ADDR);
+	val |= PCIE_PORT_INT_EN(slot);
+	pcie_write(pcie, val, RALINK_PCI_PCIMSK_ADDR);
+
+	/* map 2G DDR region */
+	pcie_write(pcie, PCIE_BAR_MAP_MAX | PCIE_BAR_ENABLE,
+		   offset + RALINK_PCI_BAR0SETUP_ADDR);
+
+	/* configure class code and revision ID */
+	pcie_write(pcie, PCIE_CLASS_CODE | PCIE_REVISION_ID,
+		   offset + RALINK_PCI_CLASS);
+}
+
+static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
+{
+	struct device *dev = pcie->dev;
+	struct mt7621_pcie_port *port;
+	u8 num_slots_enabled = 0;
+	u32 slot;
+	u32 val;
+	int err;
+
+	/* Setup MEMWIN and IOWIN */
+	pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
+	pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
+
+	list_for_each_entry(port, &pcie->ports, list) {
+		if (port->enabled) {
+			err = clk_prepare_enable(port->clk);
+			if (err) {
+				dev_err(dev, "enabling clk pcie%d\n", slot);
+				return err;
+			}
+
+			mt7621_pcie_enable_port(port);
+			dev_info(dev, "PCIE%d enabled\n", port->slot);
+			num_slots_enabled++;
+		}
+	}
+
+	for (slot = 0; slot < num_slots_enabled; slot++) {
+		val = read_config(pcie, slot, PCI_COMMAND);
+		val |= PCI_COMMAND_MASTER;
+		write_config(pcie, slot, PCI_COMMAND, val);
+		/* configure RC FTS number to 250 when it leaves L0s */
+		val = read_config(pcie, slot, PCIE_FTS_NUM);
+		val &= ~PCIE_FTS_NUM_MASK;
+		val |= PCIE_FTS_NUM_L0(0x50);
+		write_config(pcie, slot, PCIE_FTS_NUM, val);
+	}
+
+	return 0;
+}
+
+static int mt7621_pcie_register_host(struct pci_host_bridge *host)
+{
+	struct mt7621_pcie *pcie = pci_host_bridge_priv(host);
+
+	host->ops = &mt7621_pci_ops;
+	host->sysdata = pcie;
+	return pci_host_probe(host);
+}
+
+static const struct soc_device_attribute mt7621_pci_quirks_match[] = {
+	{ .soc_id = "mt7621", .revision = "E2" }
+};
+
+static int mt7621_pci_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	const struct soc_device_attribute *attr;
+	struct mt7621_pcie *pcie;
+	struct pci_host_bridge *bridge;
+	int err;
+
+	if (!dev->of_node)
+		return -ENODEV;
+
+	bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
+	if (!bridge)
+		return -ENOMEM;
+
+	pcie = pci_host_bridge_priv(bridge);
+	pcie->dev = dev;
+	platform_set_drvdata(pdev, pcie);
+	INIT_LIST_HEAD(&pcie->ports);
+
+	attr = soc_device_match(mt7621_pci_quirks_match);
+	if (attr)
+		pcie->resets_inverted = true;
+
+	err = mt7621_pcie_parse_dt(pcie);
+	if (err) {
+		dev_err(dev, "Parsing DT failed\n");
+		return err;
+	}
+
+	err = mt7621_pci_parse_request_of_pci_ranges(bridge);
+	if (err) {
+		dev_err(dev, "Error requesting pci resources from ranges");
+		return err;
+	}
+
+	/* set resources limits */
+	ioport_resource.start = pcie->io.start;
+	ioport_resource.end = pcie->io.end;
+
+	mt7621_pcie_init_ports(pcie);
+
+	err = mt7621_pcie_enable_ports(pcie);
+	if (err) {
+		dev_err(dev, "Error enabling pcie ports\n");
+		return err;
+	}
+
+	setup_cm_memory_region(pcie);
+
+	return mt7621_pcie_register_host(bridge);
+}
+
+static const struct of_device_id mt7621_pci_ids[] = {
+	{ .compatible = "mediatek,mt7621-pci" },
+	{},
+};
+MODULE_DEVICE_TABLE(of, mt7621_pci_ids);
+
+static struct platform_driver mt7621_pci_driver = {
+	.probe = mt7621_pci_probe,
+	.driver = {
+		.name = "mt7621-pci",
+		.of_match_table = of_match_ptr(mt7621_pci_ids),
+	},
+};
+builtin_platform_driver(mt7621_pci_driver);
diff --git a/arch/mips/ralink/Kconfig b/arch/mips/ralink/Kconfig
index ec4daa63c5e3..50e5a54f7d9e 100644
--- a/arch/mips/ralink/Kconfig
+++ b/arch/mips/ralink/Kconfig
@@ -56,7 +56,7 @@ choice
 		select MIPS_GIC
 		select COMMON_CLK
 		select CLKSRC_MIPS_GIC
-		select HAVE_PCI if PCI_MT7621
+		select HAVE_PCI
 		select SOC_BUS
 endchoice
 
@@ -101,4 +101,11 @@ choice
 
 endchoice
 
+config PCI_MT7621
+	bool "MediaTek MT7621 PCI Controller"
+	depends on RALINK && SOC_MT7621
+	select PCI_DRIVERS_GENERIC
+	help
+	  This selects a driver for the MediaTek MT7621 PCI Controller.
+
 endif
-- 
2.25.1

Re: [PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Pali Rohár]

On Saturday 15 May 2021 14:40:53 Sergio Paracuellos wrote:
> This patch adds a driver for the PCIe controller of MT7621 SoC.
> 
> Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> ---
>  arch/mips/pci/Makefile     |   1 +
>  arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
>  arch/mips/ralink/Kconfig   |   9 +-
>  3 files changed, 633 insertions(+), 1 deletion(-)
>  create mode 100644 arch/mips/pci/pci-mt7621.c
> 
> diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
> index f3eecc065e5c..178c550739c4 100644
> --- a/arch/mips/pci/Makefile
> +++ b/arch/mips/pci/Makefile
> @@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)	+= pci-ar2315.o
>  obj-$(CONFIG_SOC_AR71XX)	+= pci-ar71xx.o
>  obj-$(CONFIG_PCI_AR724X)	+= pci-ar724x.o
>  obj-$(CONFIG_PCI_XTALK_BRIDGE)	+= pci-xtalk-bridge.o
> +obj-$(CONFIG_PCI_MT7621)	+= pci-mt7621.o
>  #
>  # These are still pretty much in the old state, watch, go blind.
>  #
> diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
> new file mode 100644
> index 000000000000..fe1945819d25
> --- /dev/null
> +++ b/arch/mips/pci/pci-mt7621.c
...
> +static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> +{
> +	struct device *dev = pcie->dev;
> +	struct mt7621_pcie_port *port;
> +	u8 num_slots_enabled = 0;
> +	u32 slot;
> +	u32 val;
> +	int err;
> +
> +	/* Setup MEMWIN and IOWIN */
> +	pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> +	pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> +
> +	list_for_each_entry(port, &pcie->ports, list) {
> +		if (port->enabled) {
> +			err = clk_prepare_enable(port->clk);
> +			if (err) {
> +				dev_err(dev, "enabling clk pcie%d\n", slot);
> +				return err;
> +			}
> +
> +			mt7621_pcie_enable_port(port);
> +			dev_info(dev, "PCIE%d enabled\n", port->slot);
> +			num_slots_enabled++;
> +		}
> +	}
> +
> +	for (slot = 0; slot < num_slots_enabled; slot++) {
> +		val = read_config(pcie, slot, PCI_COMMAND);
> +		val |= PCI_COMMAND_MASTER;
> +		write_config(pcie, slot, PCI_COMMAND, val);

Hello! Is this part of code correct? Because it looks strange if PCIe
controller driver automatically enables PCI bus mastering, prior device
driver initialize itself.

Moreover kernel has already function pci_set_master() for this purpose
which is used by device drivers.

So I think this code can confuse some device drivers...

> +		/* configure RC FTS number to 250 when it leaves L0s */
> +		val = read_config(pcie, slot, PCIE_FTS_NUM);
> +		val &= ~PCIE_FTS_NUM_MASK;
> +		val |= PCIE_FTS_NUM_L0(0x50);
> +		write_config(pcie, slot, PCIE_FTS_NUM, val);
> +	}
> +
> +	return 0;
> +}

Re: [PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Sergio Paracuellos]

Hi Pali,

Thanks for your comments.

On Mon, May 31, 2021 at 3:14 PM Pali Rohár <pali@kernel.org> wrote:
>
> On Saturday 15 May 2021 14:40:53 Sergio Paracuellos wrote:
> > This patch adds a driver for the PCIe controller of MT7621 SoC.
> >
> > Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > ---
> >  arch/mips/pci/Makefile     |   1 +
> >  arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
> >  arch/mips/ralink/Kconfig   |   9 +-
> >  3 files changed, 633 insertions(+), 1 deletion(-)
> >  create mode 100644 arch/mips/pci/pci-mt7621.c
> >
> > diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
> > index f3eecc065e5c..178c550739c4 100644
> > --- a/arch/mips/pci/Makefile
> > +++ b/arch/mips/pci/Makefile
> > @@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)    += pci-ar2315.o
> >  obj-$(CONFIG_SOC_AR71XX)     += pci-ar71xx.o
> >  obj-$(CONFIG_PCI_AR724X)     += pci-ar724x.o
> >  obj-$(CONFIG_PCI_XTALK_BRIDGE)       += pci-xtalk-bridge.o
> > +obj-$(CONFIG_PCI_MT7621)     += pci-mt7621.o
> >  #
> >  # These are still pretty much in the old state, watch, go blind.
> >  #
> > diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
> > new file mode 100644
> > index 000000000000..fe1945819d25
> > --- /dev/null
> > +++ b/arch/mips/pci/pci-mt7621.c
> ...
> > +static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> > +{
> > +     struct device *dev = pcie->dev;
> > +     struct mt7621_pcie_port *port;
> > +     u8 num_slots_enabled = 0;
> > +     u32 slot;
> > +     u32 val;
> > +     int err;
> > +
> > +     /* Setup MEMWIN and IOWIN */
> > +     pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> > +     pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> > +
> > +     list_for_each_entry(port, &pcie->ports, list) {
> > +             if (port->enabled) {
> > +                     err = clk_prepare_enable(port->clk);
> > +                     if (err) {
> > +                             dev_err(dev, "enabling clk pcie%d\n", slot);
> > +                             return err;
> > +                     }
> > +
> > +                     mt7621_pcie_enable_port(port);
> > +                     dev_info(dev, "PCIE%d enabled\n", port->slot);
> > +                     num_slots_enabled++;
> > +             }
> > +     }
> > +
> > +     for (slot = 0; slot < num_slots_enabled; slot++) {
> > +             val = read_config(pcie, slot, PCI_COMMAND);
> > +             val |= PCI_COMMAND_MASTER;
> > +             write_config(pcie, slot, PCI_COMMAND, val);
>
> Hello! Is this part of code correct? Because it looks strange if PCIe
> controller driver automatically enables PCI bus mastering, prior device
> driver initialize itself.
>
> Moreover kernel has already function pci_set_master() for this purpose
> which is used by device drivers.
>
> So I think this code can confuse some device drivers...

I agree that we have pci_set_master() to be used in pci device driver
code. Original controller driver set this bit for enabled slots. Since
there is no documentation at all for the PCI in this SoC I have
maintained the setting in the driver in a cleaner way. See original
driver code and the setting here [0]. There is no other reason than
that. I am ok with removing this from here and testing with my two
devices that everything is still ok if having this setting in the pci
controller driver is a real problem.

[0]: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/tree/drivers/staging/mt7621-pci/pci-mt7621.c?h=v4.18#n676

Best regards,
    Sergio Paracuellos
>
> > +             /* configure RC FTS number to 250 when it leaves L0s */
> > +             val = read_config(pcie, slot, PCIE_FTS_NUM);
> > +             val &= ~PCIE_FTS_NUM_MASK;
> > +             val |= PCIE_FTS_NUM_L0(0x50);
> > +             write_config(pcie, slot, PCIE_FTS_NUM, val);
> > +     }
> > +
> > +     return 0;
> > +}

Re: [PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Pali Rohár]

On Monday 31 May 2021 15:39:55 Sergio Paracuellos wrote:
> Hi Pali,
> 
> Thanks for your comments.
> 
> On Mon, May 31, 2021 at 3:14 PM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Saturday 15 May 2021 14:40:53 Sergio Paracuellos wrote:
> > > This patch adds a driver for the PCIe controller of MT7621 SoC.
> > >
> > > Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > > ---
> > >  arch/mips/pci/Makefile     |   1 +
> > >  arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
> > >  arch/mips/ralink/Kconfig   |   9 +-
> > >  3 files changed, 633 insertions(+), 1 deletion(-)
> > >  create mode 100644 arch/mips/pci/pci-mt7621.c
> > >
> > > diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
> > > index f3eecc065e5c..178c550739c4 100644
> > > --- a/arch/mips/pci/Makefile
> > > +++ b/arch/mips/pci/Makefile
> > > @@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)    += pci-ar2315.o
> > >  obj-$(CONFIG_SOC_AR71XX)     += pci-ar71xx.o
> > >  obj-$(CONFIG_PCI_AR724X)     += pci-ar724x.o
> > >  obj-$(CONFIG_PCI_XTALK_BRIDGE)       += pci-xtalk-bridge.o
> > > +obj-$(CONFIG_PCI_MT7621)     += pci-mt7621.o
> > >  #
> > >  # These are still pretty much in the old state, watch, go blind.
> > >  #
> > > diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
> > > new file mode 100644
> > > index 000000000000..fe1945819d25
> > > --- /dev/null
> > > +++ b/arch/mips/pci/pci-mt7621.c
> > ...
> > > +static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> > > +{
> > > +     struct device *dev = pcie->dev;
> > > +     struct mt7621_pcie_port *port;
> > > +     u8 num_slots_enabled = 0;
> > > +     u32 slot;
> > > +     u32 val;
> > > +     int err;
> > > +
> > > +     /* Setup MEMWIN and IOWIN */
> > > +     pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> > > +     pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> > > +
> > > +     list_for_each_entry(port, &pcie->ports, list) {
> > > +             if (port->enabled) {
> > > +                     err = clk_prepare_enable(port->clk);
> > > +                     if (err) {
> > > +                             dev_err(dev, "enabling clk pcie%d\n", slot);
> > > +                             return err;
> > > +                     }
> > > +
> > > +                     mt7621_pcie_enable_port(port);
> > > +                     dev_info(dev, "PCIE%d enabled\n", port->slot);
> > > +                     num_slots_enabled++;
> > > +             }
> > > +     }
> > > +
> > > +     for (slot = 0; slot < num_slots_enabled; slot++) {
> > > +             val = read_config(pcie, slot, PCI_COMMAND);
> > > +             val |= PCI_COMMAND_MASTER;
> > > +             write_config(pcie, slot, PCI_COMMAND, val);
> >
> > Hello! Is this part of code correct? Because it looks strange if PCIe
> > controller driver automatically enables PCI bus mastering, prior device
> > driver initialize itself.
> >
> > Moreover kernel has already function pci_set_master() for this purpose
> > which is used by device drivers.
> >
> > So I think this code can confuse some device drivers...
> 
> I agree that we have pci_set_master() to be used in pci device driver
> code. Original controller driver set this bit for enabled slots. Since
> there is no documentation at all for the PCI in this SoC

I see... this is really a big problem to do any driver development...

> I have
> maintained the setting in the driver in a cleaner way. See original
> driver code and the setting here [0]. There is no other reason than
> that. I am ok with removing this from here and testing with my two
> devices that everything is still ok if having this setting in the pci
> controller driver is a real problem.

You can run lspci -nnvv with and without PCI_COMMAND_MASTER code and
then compare outputs.

Device drivers for sure enable PCI_COMMAND_MASTER at the time when it is
needed, so it is possible that there would be no difference in lspci
output.

> [0]: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/tree/drivers/staging/mt7621-pci/pci-mt7621.c?h=v4.18#n676
> 
> Best regards,
>     Sergio Paracuellos
> >
> > > +             /* configure RC FTS number to 250 when it leaves L0s */
> > > +             val = read_config(pcie, slot, PCIE_FTS_NUM);
> > > +             val &= ~PCIE_FTS_NUM_MASK;
> > > +             val |= PCIE_FTS_NUM_L0(0x50);
> > > +             write_config(pcie, slot, PCIE_FTS_NUM, val);
> > > +     }
> > > +
> > > +     return 0;
> > > +}

Re: [PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Sergio Paracuellos]

On Mon, May 31, 2021 at 3:50 PM Pali Rohár <pali@kernel.org> wrote:
>
> On Monday 31 May 2021 15:39:55 Sergio Paracuellos wrote:
> > Hi Pali,
> >
> > Thanks for your comments.
> >
> > On Mon, May 31, 2021 at 3:14 PM Pali Rohár <pali@kernel.org> wrote:
> > >
> > > On Saturday 15 May 2021 14:40:53 Sergio Paracuellos wrote:
> > > > This patch adds a driver for the PCIe controller of MT7621 SoC.
> > > >
> > > > Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > > > ---
> > > >  arch/mips/pci/Makefile     |   1 +
> > > >  arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
> > > >  arch/mips/ralink/Kconfig   |   9 +-
> > > >  3 files changed, 633 insertions(+), 1 deletion(-)
> > > >  create mode 100644 arch/mips/pci/pci-mt7621.c
> > > >
> > > > diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
> > > > index f3eecc065e5c..178c550739c4 100644
> > > > --- a/arch/mips/pci/Makefile
> > > > +++ b/arch/mips/pci/Makefile
> > > > @@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)    += pci-ar2315.o
> > > >  obj-$(CONFIG_SOC_AR71XX)     += pci-ar71xx.o
> > > >  obj-$(CONFIG_PCI_AR724X)     += pci-ar724x.o
> > > >  obj-$(CONFIG_PCI_XTALK_BRIDGE)       += pci-xtalk-bridge.o
> > > > +obj-$(CONFIG_PCI_MT7621)     += pci-mt7621.o
> > > >  #
> > > >  # These are still pretty much in the old state, watch, go blind.
> > > >  #
> > > > diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
> > > > new file mode 100644
> > > > index 000000000000..fe1945819d25
> > > > --- /dev/null
> > > > +++ b/arch/mips/pci/pci-mt7621.c
> > > ...
> > > > +static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> > > > +{
> > > > +     struct device *dev = pcie->dev;
> > > > +     struct mt7621_pcie_port *port;
> > > > +     u8 num_slots_enabled = 0;
> > > > +     u32 slot;
> > > > +     u32 val;
> > > > +     int err;
> > > > +
> > > > +     /* Setup MEMWIN and IOWIN */
> > > > +     pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> > > > +     pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> > > > +
> > > > +     list_for_each_entry(port, &pcie->ports, list) {
> > > > +             if (port->enabled) {
> > > > +                     err = clk_prepare_enable(port->clk);
> > > > +                     if (err) {
> > > > +                             dev_err(dev, "enabling clk pcie%d\n", slot);
> > > > +                             return err;
> > > > +                     }
> > > > +
> > > > +                     mt7621_pcie_enable_port(port);
> > > > +                     dev_info(dev, "PCIE%d enabled\n", port->slot);
> > > > +                     num_slots_enabled++;
> > > > +             }
> > > > +     }
> > > > +
> > > > +     for (slot = 0; slot < num_slots_enabled; slot++) {
> > > > +             val = read_config(pcie, slot, PCI_COMMAND);
> > > > +             val |= PCI_COMMAND_MASTER;
> > > > +             write_config(pcie, slot, PCI_COMMAND, val);
> > >
> > > Hello! Is this part of code correct? Because it looks strange if PCIe
> > > controller driver automatically enables PCI bus mastering, prior device
> > > driver initialize itself.
> > >
> > > Moreover kernel has already function pci_set_master() for this purpose
> > > which is used by device drivers.
> > >
> > > So I think this code can confuse some device drivers...
> >
> > I agree that we have pci_set_master() to be used in pci device driver
> > code. Original controller driver set this bit for enabled slots. Since
> > there is no documentation at all for the PCI in this SoC
>
> I see... this is really a big problem to do any driver development...

For sure it is :(.

>
> > I have
> > maintained the setting in the driver in a cleaner way. See original
> > driver code and the setting here [0]. There is no other reason than
> > that. I am ok with removing this from here and testing with my two
> > devices that everything is still ok if having this setting in the pci
> > controller driver is a real problem.
>
> You can run lspci -nnvv with and without PCI_COMMAND_MASTER code and
> then compare outputs.
>
> Device drivers for sure enable PCI_COMMAND_MASTER at the time when it is
> needed, so it is possible that there would be no difference in lspci
> output.

Thanks. I will take this into account when v2 is submitted after more
review comments come :).

>
> > [0]: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/tree/drivers/staging/mt7621-pci/pci-mt7621.c?h=v4.18#n676
> >
> > Best regards,
> >     Sergio Paracuellos
> > >
> > > > +             /* configure RC FTS number to 250 when it leaves L0s */
> > > > +             val = read_config(pcie, slot, PCIE_FTS_NUM);
> > > > +             val &= ~PCIE_FTS_NUM_MASK;
> > > > +             val |= PCIE_FTS_NUM_L0(0x50);
> > > > +             write_config(pcie, slot, PCIE_FTS_NUM, val);
> > > > +     }
> > > > +
> > > > +     return 0;
> > > > +}

Re: [PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Sergio Paracuellos]

Hi Pali,

On Mon, May 31, 2021 at 4:19 PM Sergio Paracuellos
<sergio.paracuellos@gmail.com> wrote:
>
> On Mon, May 31, 2021 at 3:50 PM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Monday 31 May 2021 15:39:55 Sergio Paracuellos wrote:
> > > Hi Pali,
> > >
> > > Thanks for your comments.
> > >
> > > On Mon, May 31, 2021 at 3:14 PM Pali Rohár <pali@kernel.org> wrote:
> > > >
> > > > On Saturday 15 May 2021 14:40:53 Sergio Paracuellos wrote:
> > > > > This patch adds a driver for the PCIe controller of MT7621 SoC.
> > > > >
> > > > > Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > > > > ---
> > > > >  arch/mips/pci/Makefile     |   1 +
> > > > >  arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
> > > > >  arch/mips/ralink/Kconfig   |   9 +-
> > > > >  3 files changed, 633 insertions(+), 1 deletion(-)
> > > > >  create mode 100644 arch/mips/pci/pci-mt7621.c
> > > > >
> > > > > diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
> > > > > index f3eecc065e5c..178c550739c4 100644
> > > > > --- a/arch/mips/pci/Makefile
> > > > > +++ b/arch/mips/pci/Makefile
> > > > > @@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)    += pci-ar2315.o
> > > > >  obj-$(CONFIG_SOC_AR71XX)     += pci-ar71xx.o
> > > > >  obj-$(CONFIG_PCI_AR724X)     += pci-ar724x.o
> > > > >  obj-$(CONFIG_PCI_XTALK_BRIDGE)       += pci-xtalk-bridge.o
> > > > > +obj-$(CONFIG_PCI_MT7621)     += pci-mt7621.o
> > > > >  #
> > > > >  # These are still pretty much in the old state, watch, go blind.
> > > > >  #
> > > > > diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
> > > > > new file mode 100644
> > > > > index 000000000000..fe1945819d25
> > > > > --- /dev/null
> > > > > +++ b/arch/mips/pci/pci-mt7621.c
> > > > ...
> > > > > +static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> > > > > +{
> > > > > +     struct device *dev = pcie->dev;
> > > > > +     struct mt7621_pcie_port *port;
> > > > > +     u8 num_slots_enabled = 0;
> > > > > +     u32 slot;
> > > > > +     u32 val;
> > > > > +     int err;
> > > > > +
> > > > > +     /* Setup MEMWIN and IOWIN */
> > > > > +     pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> > > > > +     pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> > > > > +
> > > > > +     list_for_each_entry(port, &pcie->ports, list) {
> > > > > +             if (port->enabled) {
> > > > > +                     err = clk_prepare_enable(port->clk);
> > > > > +                     if (err) {
> > > > > +                             dev_err(dev, "enabling clk pcie%d\n", slot);
> > > > > +                             return err;
> > > > > +                     }
> > > > > +
> > > > > +                     mt7621_pcie_enable_port(port);
> > > > > +                     dev_info(dev, "PCIE%d enabled\n", port->slot);
> > > > > +                     num_slots_enabled++;
> > > > > +             }
> > > > > +     }
> > > > > +
> > > > > +     for (slot = 0; slot < num_slots_enabled; slot++) {
> > > > > +             val = read_config(pcie, slot, PCI_COMMAND);
> > > > > +             val |= PCI_COMMAND_MASTER;
> > > > > +             write_config(pcie, slot, PCI_COMMAND, val);
> > > >
> > > > Hello! Is this part of code correct? Because it looks strange if PCIe
> > > > controller driver automatically enables PCI bus mastering, prior device
> > > > driver initialize itself.
> > > >
> > > > Moreover kernel has already function pci_set_master() for this purpose
> > > > which is used by device drivers.
> > > >
> > > > So I think this code can confuse some device drivers...
> > >
> > > I agree that we have pci_set_master() to be used in pci device driver
> > > code. Original controller driver set this bit for enabled slots. Since
> > > there is no documentation at all for the PCI in this SoC
> >
> > I see... this is really a big problem to do any driver development...
>
> For sure it is :(.
>
> >
> > > I have
> > > maintained the setting in the driver in a cleaner way. See original
> > > driver code and the setting here [0]. There is no other reason than
> > > that. I am ok with removing this from here and testing with my two
> > > devices that everything is still ok if having this setting in the pci
> > > controller driver is a real problem.
> >
> > You can run lspci -nnvv with and without PCI_COMMAND_MASTER code and
> > then compare outputs.
> >
> > Device drivers for sure enable PCI_COMMAND_MASTER at the time when it is
> > needed, so it is possible that there would be no difference in lspci
> > output.
>
> Thanks. I will take this into account when v2 is submitted after more
> review comments come :).

I have tested to remove this and check lspci -nnvv output with and
without PCI_COMMAND_MASTER code and, as you pointed out, there is no
difference between them. Also, both boards are working without
regressions at all. So I will remove this code for next version.

Thanks,
    Sergio Paracuellos
>
> >
> > > [0]: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/tree/drivers/staging/mt7621-pci/pci-mt7621.c?h=v4.18#n676
> > >
> > > Best regards,
> > >     Sergio Paracuellos
> > > >
> > > > > +             /* configure RC FTS number to 250 when it leaves L0s */
> > > > > +             val = read_config(pcie, slot, PCIE_FTS_NUM);
> > > > > +             val &= ~PCIE_FTS_NUM_MASK;
> > > > > +             val |= PCIE_FTS_NUM_L0(0x50);
> > > > > +             write_config(pcie, slot, PCIE_FTS_NUM, val);
> > > > > +     }
> > > > > +
> > > > > +     return 0;
> > > > > +}

Re: [PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Pali Rohár]

On Wednesday 02 June 2021 14:16:26 Sergio Paracuellos wrote:
> Hi Pali,
> 
> On Mon, May 31, 2021 at 4:19 PM Sergio Paracuellos
> <sergio.paracuellos@gmail.com> wrote:
> >
> > On Mon, May 31, 2021 at 3:50 PM Pali Rohár <pali@kernel.org> wrote:
> > >
> > > On Monday 31 May 2021 15:39:55 Sergio Paracuellos wrote:
> > > > Hi Pali,
> > > >
> > > > Thanks for your comments.
> > > >
> > > > On Mon, May 31, 2021 at 3:14 PM Pali Rohár <pali@kernel.org> wrote:
> > > > >
> > > > > On Saturday 15 May 2021 14:40:53 Sergio Paracuellos wrote:
> > > > > > This patch adds a driver for the PCIe controller of MT7621 SoC.
> > > > > >
> > > > > > Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > > > > > ---
> > > > > >  arch/mips/pci/Makefile     |   1 +
> > > > > >  arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
> > > > > >  arch/mips/ralink/Kconfig   |   9 +-
> > > > > >  3 files changed, 633 insertions(+), 1 deletion(-)
> > > > > >  create mode 100644 arch/mips/pci/pci-mt7621.c
> > > > > >
> > > > > > diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
> > > > > > index f3eecc065e5c..178c550739c4 100644
> > > > > > --- a/arch/mips/pci/Makefile
> > > > > > +++ b/arch/mips/pci/Makefile
> > > > > > @@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)    += pci-ar2315.o
> > > > > >  obj-$(CONFIG_SOC_AR71XX)     += pci-ar71xx.o
> > > > > >  obj-$(CONFIG_PCI_AR724X)     += pci-ar724x.o
> > > > > >  obj-$(CONFIG_PCI_XTALK_BRIDGE)       += pci-xtalk-bridge.o
> > > > > > +obj-$(CONFIG_PCI_MT7621)     += pci-mt7621.o
> > > > > >  #
> > > > > >  # These are still pretty much in the old state, watch, go blind.
> > > > > >  #
> > > > > > diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
> > > > > > new file mode 100644
> > > > > > index 000000000000..fe1945819d25
> > > > > > --- /dev/null
> > > > > > +++ b/arch/mips/pci/pci-mt7621.c
> > > > > ...
> > > > > > +static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> > > > > > +{
> > > > > > +     struct device *dev = pcie->dev;
> > > > > > +     struct mt7621_pcie_port *port;
> > > > > > +     u8 num_slots_enabled = 0;
> > > > > > +     u32 slot;
> > > > > > +     u32 val;
> > > > > > +     int err;
> > > > > > +
> > > > > > +     /* Setup MEMWIN and IOWIN */
> > > > > > +     pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> > > > > > +     pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> > > > > > +
> > > > > > +     list_for_each_entry(port, &pcie->ports, list) {
> > > > > > +             if (port->enabled) {
> > > > > > +                     err = clk_prepare_enable(port->clk);
> > > > > > +                     if (err) {
> > > > > > +                             dev_err(dev, "enabling clk pcie%d\n", slot);
> > > > > > +                             return err;
> > > > > > +                     }
> > > > > > +
> > > > > > +                     mt7621_pcie_enable_port(port);
> > > > > > +                     dev_info(dev, "PCIE%d enabled\n", port->slot);
> > > > > > +                     num_slots_enabled++;
> > > > > > +             }
> > > > > > +     }
> > > > > > +
> > > > > > +     for (slot = 0; slot < num_slots_enabled; slot++) {
> > > > > > +             val = read_config(pcie, slot, PCI_COMMAND);
> > > > > > +             val |= PCI_COMMAND_MASTER;
> > > > > > +             write_config(pcie, slot, PCI_COMMAND, val);
> > > > >
> > > > > Hello! Is this part of code correct? Because it looks strange if PCIe
> > > > > controller driver automatically enables PCI bus mastering, prior device
> > > > > driver initialize itself.
> > > > >
> > > > > Moreover kernel has already function pci_set_master() for this purpose
> > > > > which is used by device drivers.
> > > > >
> > > > > So I think this code can confuse some device drivers...
> > > >
> > > > I agree that we have pci_set_master() to be used in pci device driver
> > > > code. Original controller driver set this bit for enabled slots. Since
> > > > there is no documentation at all for the PCI in this SoC
> > >
> > > I see... this is really a big problem to do any driver development...
> >
> > For sure it is :(.
> >
> > >
> > > > I have
> > > > maintained the setting in the driver in a cleaner way. See original
> > > > driver code and the setting here [0]. There is no other reason than
> > > > that. I am ok with removing this from here and testing with my two
> > > > devices that everything is still ok if having this setting in the pci
> > > > controller driver is a real problem.
> > >
> > > You can run lspci -nnvv with and without PCI_COMMAND_MASTER code and
> > > then compare outputs.
> > >
> > > Device drivers for sure enable PCI_COMMAND_MASTER at the time when it is
> > > needed, so it is possible that there would be no difference in lspci
> > > output.
> >
> > Thanks. I will take this into account when v2 is submitted after more
> > review comments come :).
> 
> I have tested to remove this and check lspci -nnvv output with and
> without PCI_COMMAND_MASTER code and, as you pointed out, there is no
> difference between them. Also, both boards are working without
> regressions at all. So I will remove this code for next version.

Perfect!

> Thanks,
>     Sergio Paracuellos
> >
> > >
> > > > [0]: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/tree/drivers/staging/mt7621-pci/pci-mt7621.c?h=v4.18#n676
> > > >
> > > > Best regards,
> > > >     Sergio Paracuellos
> > > > >
> > > > > > +             /* configure RC FTS number to 250 when it leaves L0s */
> > > > > > +             val = read_config(pcie, slot, PCIE_FTS_NUM);
> > > > > > +             val &= ~PCIE_FTS_NUM_MASK;
> > > > > > +             val |= PCIE_FTS_NUM_L0(0x50);
> > > > > > +             write_config(pcie, slot, PCIE_FTS_NUM, val);

Could you look also what is doing this code (PCIE_FTS_NUM)? It is marked
as MT specific register. But from this code for me it looks like that it
just access config space of some device and therefore it could be some
standard PCIe register. Just with hardcoded calculated offset.

Could you provide output from lspci -nnvv? So other people could look at
it and maybe we decode what is this code doing and if it is needed.

> > > > > > +     }
> > > > > > +
> > > > > > +     return 0;
> > > > > > +}

Re: [PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Sergio Paracuellos]

Hi Pali,

On Wed, Jun 2, 2021 at 2:23 PM Pali Rohár <pali@kernel.org> wrote:
>
> On Wednesday 02 June 2021 14:16:26 Sergio Paracuellos wrote:
> > Hi Pali,
> >
> > On Mon, May 31, 2021 at 4:19 PM Sergio Paracuellos
> > <sergio.paracuellos@gmail.com> wrote:
> > >
> > > On Mon, May 31, 2021 at 3:50 PM Pali Rohár <pali@kernel.org> wrote:
> > > >
> > > > On Monday 31 May 2021 15:39:55 Sergio Paracuellos wrote:
> > > > > Hi Pali,
> > > > >
> > > > > Thanks for your comments.
> > > > >
> > > > > On Mon, May 31, 2021 at 3:14 PM Pali Rohár <pali@kernel.org> wrote:
> > > > > >
> > > > > > On Saturday 15 May 2021 14:40:53 Sergio Paracuellos wrote:
> > > > > > > This patch adds a driver for the PCIe controller of MT7621 SoC.
> > > > > > >
> > > > > > > Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > > > > > > ---
> > > > > > >  arch/mips/pci/Makefile     |   1 +
> > > > > > >  arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
> > > > > > >  arch/mips/ralink/Kconfig   |   9 +-
> > > > > > >  3 files changed, 633 insertions(+), 1 deletion(-)
> > > > > > >  create mode 100644 arch/mips/pci/pci-mt7621.c
> > > > > > >
> > > > > > > diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
> > > > > > > index f3eecc065e5c..178c550739c4 100644
> > > > > > > --- a/arch/mips/pci/Makefile
> > > > > > > +++ b/arch/mips/pci/Makefile
> > > > > > > @@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)    += pci-ar2315.o
> > > > > > >  obj-$(CONFIG_SOC_AR71XX)     += pci-ar71xx.o
> > > > > > >  obj-$(CONFIG_PCI_AR724X)     += pci-ar724x.o
> > > > > > >  obj-$(CONFIG_PCI_XTALK_BRIDGE)       += pci-xtalk-bridge.o
> > > > > > > +obj-$(CONFIG_PCI_MT7621)     += pci-mt7621.o
> > > > > > >  #
> > > > > > >  # These are still pretty much in the old state, watch, go blind.
> > > > > > >  #
> > > > > > > diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
> > > > > > > new file mode 100644
> > > > > > > index 000000000000..fe1945819d25
> > > > > > > --- /dev/null
> > > > > > > +++ b/arch/mips/pci/pci-mt7621.c
> > > > > > ...
> > > > > > > +static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> > > > > > > +{
> > > > > > > +     struct device *dev = pcie->dev;
> > > > > > > +     struct mt7621_pcie_port *port;
> > > > > > > +     u8 num_slots_enabled = 0;
> > > > > > > +     u32 slot;
> > > > > > > +     u32 val;
> > > > > > > +     int err;
> > > > > > > +
> > > > > > > +     /* Setup MEMWIN and IOWIN */
> > > > > > > +     pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> > > > > > > +     pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> > > > > > > +
> > > > > > > +     list_for_each_entry(port, &pcie->ports, list) {
> > > > > > > +             if (port->enabled) {
> > > > > > > +                     err = clk_prepare_enable(port->clk);
> > > > > > > +                     if (err) {
> > > > > > > +                             dev_err(dev, "enabling clk pcie%d\n", slot);
> > > > > > > +                             return err;
> > > > > > > +                     }
> > > > > > > +
> > > > > > > +                     mt7621_pcie_enable_port(port);
> > > > > > > +                     dev_info(dev, "PCIE%d enabled\n", port->slot);
> > > > > > > +                     num_slots_enabled++;
> > > > > > > +             }
> > > > > > > +     }
> > > > > > > +
> > > > > > > +     for (slot = 0; slot < num_slots_enabled; slot++) {
> > > > > > > +             val = read_config(pcie, slot, PCI_COMMAND);
> > > > > > > +             val |= PCI_COMMAND_MASTER;
> > > > > > > +             write_config(pcie, slot, PCI_COMMAND, val);
> > > > > >
> > > > > > Hello! Is this part of code correct? Because it looks strange if PCIe
> > > > > > controller driver automatically enables PCI bus mastering, prior device
> > > > > > driver initialize itself.
> > > > > >
> > > > > > Moreover kernel has already function pci_set_master() for this purpose
> > > > > > which is used by device drivers.
> > > > > >
> > > > > > So I think this code can confuse some device drivers...
> > > > >
> > > > > I agree that we have pci_set_master() to be used in pci device driver
> > > > > code. Original controller driver set this bit for enabled slots. Since
> > > > > there is no documentation at all for the PCI in this SoC
> > > >
> > > > I see... this is really a big problem to do any driver development...
> > >
> > > For sure it is :(.
> > >
> > > >
> > > > > I have
> > > > > maintained the setting in the driver in a cleaner way. See original
> > > > > driver code and the setting here [0]. There is no other reason than
> > > > > that. I am ok with removing this from here and testing with my two
> > > > > devices that everything is still ok if having this setting in the pci
> > > > > controller driver is a real problem.
> > > >
> > > > You can run lspci -nnvv with and without PCI_COMMAND_MASTER code and
> > > > then compare outputs.
> > > >
> > > > Device drivers for sure enable PCI_COMMAND_MASTER at the time when it is
> > > > needed, so it is possible that there would be no difference in lspci
> > > > output.
> > >
> > > Thanks. I will take this into account when v2 is submitted after more
> > > review comments come :).
> >
> > I have tested to remove this and check lspci -nnvv output with and
> > without PCI_COMMAND_MASTER code and, as you pointed out, there is no
> > difference between them. Also, both boards are working without
> > regressions at all. So I will remove this code for next version.
>
> Perfect!
>
> > Thanks,
> >     Sergio Paracuellos
> > >
> > > >
> > > > > [0]: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/tree/drivers/staging/mt7621-pci/pci-mt7621.c?h=v4.18#n676
> > > > >
> > > > > Best regards,
> > > > >     Sergio Paracuellos
> > > > > >
> > > > > > > +             /* configure RC FTS number to 250 when it leaves L0s */
> > > > > > > +             val = read_config(pcie, slot, PCIE_FTS_NUM);
> > > > > > > +             val &= ~PCIE_FTS_NUM_MASK;
> > > > > > > +             val |= PCIE_FTS_NUM_L0(0x50);
> > > > > > > +             write_config(pcie, slot, PCIE_FTS_NUM, val);
>
> Could you look also what is doing this code (PCIE_FTS_NUM)? It is marked
> as MT specific register. But from this code for me it looks like that it
> just access config space of some device and therefore it could be some
> standard PCIe register. Just with hardcoded calculated offset.
>
> Could you provide output from lspci -nnvv? So other people could look at
> it and maybe we decode what is this code doing and if it is needed.

# lspci -nnvv
00:02.0 PCI bridge [0604]: Device [0e8d:0801] (rev 01) (prog-if 00
[Normal decode])
        Device tree node: /sys/firmware/devicetree/base/pcie@1e140000/pcie@2,0
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 255
        Region 1: Memory at 60200000 (32-bit, non-prefetchable) [size=64K]
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00000000-00000fff [size=4K]
        Memory behind bridge: 60000000-600fffff [size=1M]
        Prefetchable memory behind bridge: 60100000-601fffff [size=1M]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA
PME(D0+,D1+,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq-
AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1,
Exit Latency L0s <512ns, L1 <64us
                        ClockPM- Surprise- LLActRep+ BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 128 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
                        TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
                RootCap: CRSVisible-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
PMEIntEna- CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Not Supported,
TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not
Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported,
EmergencyPowerReductionInit-
                         FRS- LN System CLS Not Supported, TPHComp-
ExtTPHComp- ARIFwd-
                         AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
LTR- OBFF Disabled, ARIFwd-
                         AtomicOpsCtl: ReqEn- EgressBlck-
                LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink-
Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3-
LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr+ BadTLP- BadDLLP- Rollover- Timeout-
AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+
ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
                RootCmd: CERptEn- NFERptEn- FERptEn-
                RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
                         FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
                ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
        Capabilities: [140 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending- InProgress-
lspci: Unable to load libkmod resources: error -12

01:00.0 Network controller [0280]: MEDIATEK Corp. Device [14c3:7612]
        Subsystem: MEDIATEK Corp. Device [14c3:7612]
        Device tree node:
/sys/firmware/devicetree/base/pcie@1e140000/pcie@2,0/wifi@0,0
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 20
        Region 0: Memory at 60000000 (64-bit, non-prefetchable) [size=1M]
        Expansion ROM at 60100000 [virtual] [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+
FLReset- SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq-
AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1,
Exit Latency L0s <2us, L1 unlimited
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not
Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported,
EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink-
Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB,
EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3-
LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+
ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [158 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Capabilities: [160 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+
ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=50us PortTPowerOnTime=10us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=0us LTR1.2_Threshold=0ns
                L1SubCtl2: T_PwrOn=10us
        Kernel driver in use: mt76x2e

Best regards,
    Sergio Paracuellos

>
> > > > > > > +     }
> > > > > > > +
> > > > > > > +     return 0;
> > > > > > > +}

Re: [PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Pali Rohár]

On Wednesday 02 June 2021 14:43:53 Sergio Paracuellos wrote:
> Hi Pali,
> 
> On Wed, Jun 2, 2021 at 2:23 PM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Wednesday 02 June 2021 14:16:26 Sergio Paracuellos wrote:
> > > Hi Pali,
> > >
> > > On Mon, May 31, 2021 at 4:19 PM Sergio Paracuellos
> > > <sergio.paracuellos@gmail.com> wrote:
> > > >
> > > > On Mon, May 31, 2021 at 3:50 PM Pali Rohár <pali@kernel.org> wrote:
> > > > >
> > > > > On Monday 31 May 2021 15:39:55 Sergio Paracuellos wrote:
> > > > > > Hi Pali,
> > > > > >
> > > > > > Thanks for your comments.
> > > > > >
> > > > > > On Mon, May 31, 2021 at 3:14 PM Pali Rohár <pali@kernel.org> wrote:
> > > > > > >
> > > > > > > On Saturday 15 May 2021 14:40:53 Sergio Paracuellos wrote:
> > > > > > > > This patch adds a driver for the PCIe controller of MT7621 SoC.
> > > > > > > >
> > > > > > > > Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > > > > > > > ---
> > > > > > > >  arch/mips/pci/Makefile     |   1 +
> > > > > > > >  arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
> > > > > > > >  arch/mips/ralink/Kconfig   |   9 +-
> > > > > > > >  3 files changed, 633 insertions(+), 1 deletion(-)
> > > > > > > >  create mode 100644 arch/mips/pci/pci-mt7621.c
> > > > > > > >
> > > > > > > > diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
> > > > > > > > index f3eecc065e5c..178c550739c4 100644
> > > > > > > > --- a/arch/mips/pci/Makefile
> > > > > > > > +++ b/arch/mips/pci/Makefile
> > > > > > > > @@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)    += pci-ar2315.o
> > > > > > > >  obj-$(CONFIG_SOC_AR71XX)     += pci-ar71xx.o
> > > > > > > >  obj-$(CONFIG_PCI_AR724X)     += pci-ar724x.o
> > > > > > > >  obj-$(CONFIG_PCI_XTALK_BRIDGE)       += pci-xtalk-bridge.o
> > > > > > > > +obj-$(CONFIG_PCI_MT7621)     += pci-mt7621.o
> > > > > > > >  #
> > > > > > > >  # These are still pretty much in the old state, watch, go blind.
> > > > > > > >  #
> > > > > > > > diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
> > > > > > > > new file mode 100644
> > > > > > > > index 000000000000..fe1945819d25
> > > > > > > > --- /dev/null
> > > > > > > > +++ b/arch/mips/pci/pci-mt7621.c
> > > > > > > ...
> > > > > > > > +static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> > > > > > > > +{
> > > > > > > > +     struct device *dev = pcie->dev;
> > > > > > > > +     struct mt7621_pcie_port *port;
> > > > > > > > +     u8 num_slots_enabled = 0;
> > > > > > > > +     u32 slot;
> > > > > > > > +     u32 val;
> > > > > > > > +     int err;
> > > > > > > > +
> > > > > > > > +     /* Setup MEMWIN and IOWIN */
> > > > > > > > +     pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> > > > > > > > +     pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> > > > > > > > +
> > > > > > > > +     list_for_each_entry(port, &pcie->ports, list) {
> > > > > > > > +             if (port->enabled) {
> > > > > > > > +                     err = clk_prepare_enable(port->clk);
> > > > > > > > +                     if (err) {
> > > > > > > > +                             dev_err(dev, "enabling clk pcie%d\n", slot);
> > > > > > > > +                             return err;
> > > > > > > > +                     }
> > > > > > > > +
> > > > > > > > +                     mt7621_pcie_enable_port(port);
> > > > > > > > +                     dev_info(dev, "PCIE%d enabled\n", port->slot);
> > > > > > > > +                     num_slots_enabled++;
> > > > > > > > +             }
> > > > > > > > +     }
> > > > > > > > +
> > > > > > > > +     for (slot = 0; slot < num_slots_enabled; slot++) {
> > > > > > > > +             val = read_config(pcie, slot, PCI_COMMAND);
> > > > > > > > +             val |= PCI_COMMAND_MASTER;
> > > > > > > > +             write_config(pcie, slot, PCI_COMMAND, val);
> > > > > > >
> > > > > > > Hello! Is this part of code correct? Because it looks strange if PCIe
> > > > > > > controller driver automatically enables PCI bus mastering, prior device
> > > > > > > driver initialize itself.
> > > > > > >
> > > > > > > Moreover kernel has already function pci_set_master() for this purpose
> > > > > > > which is used by device drivers.
> > > > > > >
> > > > > > > So I think this code can confuse some device drivers...
> > > > > >
> > > > > > I agree that we have pci_set_master() to be used in pci device driver
> > > > > > code. Original controller driver set this bit for enabled slots. Since
> > > > > > there is no documentation at all for the PCI in this SoC
> > > > >
> > > > > I see... this is really a big problem to do any driver development...
> > > >
> > > > For sure it is :(.
> > > >
> > > > >
> > > > > > I have
> > > > > > maintained the setting in the driver in a cleaner way. See original
> > > > > > driver code and the setting here [0]. There is no other reason than
> > > > > > that. I am ok with removing this from here and testing with my two
> > > > > > devices that everything is still ok if having this setting in the pci
> > > > > > controller driver is a real problem.
> > > > >
> > > > > You can run lspci -nnvv with and without PCI_COMMAND_MASTER code and
> > > > > then compare outputs.
> > > > >
> > > > > Device drivers for sure enable PCI_COMMAND_MASTER at the time when it is
> > > > > needed, so it is possible that there would be no difference in lspci
> > > > > output.
> > > >
> > > > Thanks. I will take this into account when v2 is submitted after more
> > > > review comments come :).
> > >
> > > I have tested to remove this and check lspci -nnvv output with and
> > > without PCI_COMMAND_MASTER code and, as you pointed out, there is no
> > > difference between them. Also, both boards are working without
> > > regressions at all. So I will remove this code for next version.
> >
> > Perfect!
> >
> > > Thanks,
> > >     Sergio Paracuellos
> > > >
> > > > >
> > > > > > [0]: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/tree/drivers/staging/mt7621-pci/pci-mt7621.c?h=v4.18#n676
> > > > > >
> > > > > > Best regards,
> > > > > >     Sergio Paracuellos
> > > > > > >
> > > > > > > > +             /* configure RC FTS number to 250 when it leaves L0s */
> > > > > > > > +             val = read_config(pcie, slot, PCIE_FTS_NUM);
> > > > > > > > +             val &= ~PCIE_FTS_NUM_MASK;
> > > > > > > > +             val |= PCIE_FTS_NUM_L0(0x50);
> > > > > > > > +             write_config(pcie, slot, PCIE_FTS_NUM, val);
> >
> > Could you look also what is doing this code (PCIE_FTS_NUM)? It is marked
> > as MT specific register. But from this code for me it looks like that it
> > just access config space of some device and therefore it could be some
> > standard PCIe register. Just with hardcoded calculated offset.

So based on your lspci output, there is no PCIe capability register at
address PCIE_FTS_NUM (0x70c), right? It seems strange to trying access
capability register outside of capability list.

> > Could you provide output from lspci -nnvv? So other people could look at
> > it and maybe we decode what is this code doing and if it is needed.
> 
> # lspci -nnvv
> 00:02.0 PCI bridge [0604]: Device [0e8d:0801] (rev 01) (prog-if 00
> [Normal decode])

Hm... Device address is 02. But in your code is:

    u8 num_slots_enabled = 0;
    ...
    list_for_each_entry(port, &pcie->ports, list) {
        if (port->enabled) {
            ...
            num_slots_enabled++;
            ...
        }
    }
    ...
    for (slot = 0; slot < num_slots_enabled; slot++) {
        val = read_config(pcie, slot, ...);
        ...
        write_config(pcie, slot, ...);
    }

Which means that this code writes to config space of wrong device 0
(instead of 2)! In function write_config() can be seen that second
parameter specify device of BDF address for bus=0 and function=0.

>         Device tree node: /sys/firmware/devicetree/base/pcie@1e140000/pcie@2,0
>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0
>         Interrupt: pin A routed to IRQ 255
>         Region 1: Memory at 60200000 (32-bit, non-prefetchable) [size=64K]
>         Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>         I/O behind bridge: 00000000-00000fff [size=4K]
>         Memory behind bridge: 60000000-600fffff [size=1M]
>         Prefetchable memory behind bridge: 60100000-601fffff [size=1M]
>         Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>         BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
>                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>         Capabilities: [40] Power Management version 3
>                 Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA
> PME(D0+,D1+,D2-,D3hot+,D3cold-)
>                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
>         Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
>                 Address: 0000000000000000  Data: 0000
>         Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
>                 DevCap: MaxPayload 128 bytes, PhantFunc 0
>                         ExtTag- RBE+
>                 DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
>                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                         MaxPayload 128 bytes, MaxReadReq 128 bytes
>                 DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq-
> AuxPwr- TransPend-
>                 LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1,
> Exit Latency L0s <512ns, L1 <64us
>                         ClockPM- Surprise- LLActRep+ BwNot- ASPMOptComp-
>                 LnkCtl: ASPM Disabled; RCB 128 bytes, Disabled- CommClk-
>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>                 LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
>                         TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
>                 RootCap: CRSVisible-
>                 RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
> PMEIntEna- CRSVisible-
>                 RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>                 DevCap2: Completion Timeout: Not Supported,
> TimeoutDis+ NROPrPrP- LTR-
>                          10BitTagComp- 10BitTagReq- OBFF Not
> Supported, ExtFmt- EETLPPrefix-
>                          EmergencyPowerReduction Not Supported,
> EmergencyPowerReductionInit-
>                          FRS- LN System CLS Not Supported, TPHComp-
> ExtTPHComp- ARIFwd-
>                          AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
>                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
> LTR- OBFF Disabled, ARIFwd-
>                          AtomicOpsCtl: ReqEn- EgressBlck-
>                 LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink-
> Retimer- 2Retimers- DRS-
>                 LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
>                          Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>                          Compliance De-emphasis: -6dB
>                 LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete- EqualizationPhase1-
>                          EqualizationPhase2- EqualizationPhase3-
> LinkEqualizationRequest-
>                          Retimer- 2Retimers- CrosslinkRes: unsupported
>         Capabilities: [100 v1] Advanced Error Reporting
>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
> UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>                 CESta:  RxErr+ BadTLP- BadDLLP- Rollover- Timeout-
> AdvNonFatalErr-
>                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> AdvNonFatalErr+
>                 AERCap: First Error Pointer: 00, ECRCGenCap+
> ECRCGenEn- ECRCChkCap+ ECRCChkEn-
>                         MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
>                 HeaderLog: 00000000 00000000 00000000 00000000
>                 RootCmd: CERptEn- NFERptEn- FERptEn-
>                 RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
>                          FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
>                 ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
>         Capabilities: [140 v1] Virtual Channel
>                 Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
>                 Arb:    Fixed- WRR32- WRR64- WRR128-
>                 Ctrl:   ArbSelect=Fixed
>                 Status: InProgress-
>                 VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>                         Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
>                         Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>                         Status: NegoPending- InProgress-
> lspci: Unable to load libkmod resources: error -12
> 
> 01:00.0 Network controller [0280]: MEDIATEK Corp. Device [14c3:7612]
>         Subsystem: MEDIATEK Corp. Device [14c3:7612]
>         Device tree node:
> /sys/firmware/devicetree/base/pcie@1e140000/pcie@2,0/wifi@0,0
>         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0
>         Interrupt: pin A routed to IRQ 20
>         Region 0: Memory at 60000000 (64-bit, non-prefetchable) [size=1M]
>         Expansion ROM at 60100000 [virtual] [disabled] [size=64K]
>         Capabilities: [40] Power Management version 3
>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA
> PME(D0+,D1-,D2-,D3hot+,D3cold+)
>                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
>         Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
>                 Address: 0000000000000000  Data: 0000
>         Capabilities: [70] Express (v2) Endpoint, MSI 00
>                 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> unlimited, L1 unlimited
>                         ExtTag- AttnBtn- AttnInd- PwrInd- RBE+
> FLReset- SlotPowerLimit 0.000W
>                 DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
>                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                         MaxPayload 128 bytes, MaxReadReq 128 bytes
>                 DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq-
> AuxPwr+ TransPend-
>                 LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1,
> Exit Latency L0s <2us, L1 unlimited
>                         ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>                 LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>                 LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
>                         TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
>                 DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
> NROPrPrP- LTR-
>                          10BitTagComp- 10BitTagReq- OBFF Not
> Supported, ExtFmt- EETLPPrefix-
>                          EmergencyPowerReduction Not Supported,
> EmergencyPowerReductionInit-
>                          FRS- TPHComp- ExtTPHComp-
>                          AtomicOpsCap: 32bit- 64bit- 128bitCAS-
>                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
> LTR- OBFF Disabled,
>                          AtomicOpsCtl: ReqEn-
>                 LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink-
> Retimer- 2Retimers- DRS-
>                 LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
>                          Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>                          Compliance De-emphasis: -6dB
>                 LnkSta2: Current De-emphasis Level: -3.5dB,
> EqualizationComplete- EqualizationPhase1-
>                          EqualizationPhase2- EqualizationPhase3-
> LinkEqualizationRequest-
>                          Retimer- 2Retimers- CrosslinkRes: unsupported
>         Capabilities: [100 v2] Advanced Error Reporting
>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
> UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> AdvNonFatalErr-
>                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> AdvNonFatalErr+
>                 AERCap: First Error Pointer: 00, ECRCGenCap+
> ECRCGenEn- ECRCChkCap+ ECRCChkEn-
>                         MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
>                 HeaderLog: 00000000 00000000 00000000 00000000
>         Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
>         Capabilities: [158 v1] Latency Tolerance Reporting
>                 Max snoop latency: 0ns
>                 Max no snoop latency: 0ns
>         Capabilities: [160 v1] L1 PM Substates
>                 L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+
> ASPM_L1.1+ L1_PM_Substates+
>                           PortCommonModeRestoreTime=50us PortTPowerOnTime=10us
>                 L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
>                            T_CommonMode=0us LTR1.2_Threshold=0ns
>                 L1SubCtl2: T_PwrOn=10us
>         Kernel driver in use: mt76x2e
> 
> Best regards,
>     Sergio Paracuellos
> 
> >
> > > > > > > > +     }
> > > > > > > > +
> > > > > > > > +     return 0;
> > > > > > > > +}

Re: [PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Sergio Paracuellos]

Hi Pali,

Thanks for your comments.

On Fri, Jun 4, 2021 at 6:55 PM Pali Rohár <pali@kernel.org> wrote:
>
> On Wednesday 02 June 2021 14:43:53 Sergio Paracuellos wrote:
> > Hi Pali,
> >
> > On Wed, Jun 2, 2021 at 2:23 PM Pali Rohár <pali@kernel.org> wrote:
> > >
> > > On Wednesday 02 June 2021 14:16:26 Sergio Paracuellos wrote:
> > > > Hi Pali,
> > > >
> > > > On Mon, May 31, 2021 at 4:19 PM Sergio Paracuellos
> > > > <sergio.paracuellos@gmail.com> wrote:
> > > > >
> > > > > On Mon, May 31, 2021 at 3:50 PM Pali Rohár <pali@kernel.org> wrote:
> > > > > >
> > > > > > On Monday 31 May 2021 15:39:55 Sergio Paracuellos wrote:
> > > > > > > Hi Pali,
> > > > > > >
> > > > > > > Thanks for your comments.
> > > > > > >
> > > > > > > On Mon, May 31, 2021 at 3:14 PM Pali Rohár <pali@kernel.org> wrote:
> > > > > > > >
> > > > > > > > On Saturday 15 May 2021 14:40:53 Sergio Paracuellos wrote:
> > > > > > > > > This patch adds a driver for the PCIe controller of MT7621 SoC.
> > > > > > > > >
> > > > > > > > > Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > > > > > > > > ---
> > > > > > > > >  arch/mips/pci/Makefile     |   1 +
> > > > > > > > >  arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
> > > > > > > > >  arch/mips/ralink/Kconfig   |   9 +-
> > > > > > > > >  3 files changed, 633 insertions(+), 1 deletion(-)
> > > > > > > > >  create mode 100644 arch/mips/pci/pci-mt7621.c
> > > > > > > > >
> > > > > > > > > diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
> > > > > > > > > index f3eecc065e5c..178c550739c4 100644
> > > > > > > > > --- a/arch/mips/pci/Makefile
> > > > > > > > > +++ b/arch/mips/pci/Makefile
> > > > > > > > > @@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)    += pci-ar2315.o
> > > > > > > > >  obj-$(CONFIG_SOC_AR71XX)     += pci-ar71xx.o
> > > > > > > > >  obj-$(CONFIG_PCI_AR724X)     += pci-ar724x.o
> > > > > > > > >  obj-$(CONFIG_PCI_XTALK_BRIDGE)       += pci-xtalk-bridge.o
> > > > > > > > > +obj-$(CONFIG_PCI_MT7621)     += pci-mt7621.o
> > > > > > > > >  #
> > > > > > > > >  # These are still pretty much in the old state, watch, go blind.
> > > > > > > > >  #
> > > > > > > > > diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
> > > > > > > > > new file mode 100644
> > > > > > > > > index 000000000000..fe1945819d25
> > > > > > > > > --- /dev/null
> > > > > > > > > +++ b/arch/mips/pci/pci-mt7621.c
> > > > > > > > ...
> > > > > > > > > +static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> > > > > > > > > +{
> > > > > > > > > +     struct device *dev = pcie->dev;
> > > > > > > > > +     struct mt7621_pcie_port *port;
> > > > > > > > > +     u8 num_slots_enabled = 0;
> > > > > > > > > +     u32 slot;
> > > > > > > > > +     u32 val;
> > > > > > > > > +     int err;
> > > > > > > > > +
> > > > > > > > > +     /* Setup MEMWIN and IOWIN */
> > > > > > > > > +     pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> > > > > > > > > +     pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> > > > > > > > > +
> > > > > > > > > +     list_for_each_entry(port, &pcie->ports, list) {
> > > > > > > > > +             if (port->enabled) {
> > > > > > > > > +                     err = clk_prepare_enable(port->clk);
> > > > > > > > > +                     if (err) {
> > > > > > > > > +                             dev_err(dev, "enabling clk pcie%d\n", slot);
> > > > > > > > > +                             return err;
> > > > > > > > > +                     }
> > > > > > > > > +
> > > > > > > > > +                     mt7621_pcie_enable_port(port);
> > > > > > > > > +                     dev_info(dev, "PCIE%d enabled\n", port->slot);
> > > > > > > > > +                     num_slots_enabled++;
> > > > > > > > > +             }
> > > > > > > > > +     }
> > > > > > > > > +
> > > > > > > > > +     for (slot = 0; slot < num_slots_enabled; slot++) {
> > > > > > > > > +             val = read_config(pcie, slot, PCI_COMMAND);
> > > > > > > > > +             val |= PCI_COMMAND_MASTER;
> > > > > > > > > +             write_config(pcie, slot, PCI_COMMAND, val);
> > > > > > > >
> > > > > > > > Hello! Is this part of code correct? Because it looks strange if PCIe
> > > > > > > > controller driver automatically enables PCI bus mastering, prior device
> > > > > > > > driver initialize itself.
> > > > > > > >
> > > > > > > > Moreover kernel has already function pci_set_master() for this purpose
> > > > > > > > which is used by device drivers.
> > > > > > > >
> > > > > > > > So I think this code can confuse some device drivers...
> > > > > > >
> > > > > > > I agree that we have pci_set_master() to be used in pci device driver
> > > > > > > code. Original controller driver set this bit for enabled slots. Since
> > > > > > > there is no documentation at all for the PCI in this SoC
> > > > > >
> > > > > > I see... this is really a big problem to do any driver development...
> > > > >
> > > > > For sure it is :(.
> > > > >
> > > > > >
> > > > > > > I have
> > > > > > > maintained the setting in the driver in a cleaner way. See original
> > > > > > > driver code and the setting here [0]. There is no other reason than
> > > > > > > that. I am ok with removing this from here and testing with my two
> > > > > > > devices that everything is still ok if having this setting in the pci
> > > > > > > controller driver is a real problem.
> > > > > >
> > > > > > You can run lspci -nnvv with and without PCI_COMMAND_MASTER code and
> > > > > > then compare outputs.
> > > > > >
> > > > > > Device drivers for sure enable PCI_COMMAND_MASTER at the time when it is
> > > > > > needed, so it is possible that there would be no difference in lspci
> > > > > > output.
> > > > >
> > > > > Thanks. I will take this into account when v2 is submitted after more
> > > > > review comments come :).
> > > >
> > > > I have tested to remove this and check lspci -nnvv output with and
> > > > without PCI_COMMAND_MASTER code and, as you pointed out, there is no
> > > > difference between them. Also, both boards are working without
> > > > regressions at all. So I will remove this code for next version.
> > >
> > > Perfect!
> > >
> > > > Thanks,
> > > >     Sergio Paracuellos
> > > > >
> > > > > >
> > > > > > > [0]: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/tree/drivers/staging/mt7621-pci/pci-mt7621.c?h=v4.18#n676
> > > > > > >
> > > > > > > Best regards,
> > > > > > >     Sergio Paracuellos
> > > > > > > >
> > > > > > > > > +             /* configure RC FTS number to 250 when it leaves L0s */
> > > > > > > > > +             val = read_config(pcie, slot, PCIE_FTS_NUM);
> > > > > > > > > +             val &= ~PCIE_FTS_NUM_MASK;
> > > > > > > > > +             val |= PCIE_FTS_NUM_L0(0x50);
> > > > > > > > > +             write_config(pcie, slot, PCIE_FTS_NUM, val);
> > >
> > > Could you look also what is doing this code (PCIE_FTS_NUM)? It is marked
> > > as MT specific register. But from this code for me it looks like that it
> > > just access config space of some device and therefore it could be some
> > > standard PCIe register. Just with hardcoded calculated offset.
>
> So based on your lspci output, there is no PCIe capability register at
> address PCIE_FTS_NUM (0x70c), right? It seems strange to trying access
> capability register outside of capability list.

This setting is also in the original driver code from mediatek SDK. See [0].
Anyway, I have tried to remove this code and test what happens with my
two boards and both of them seem to
work properly just by deleting this code.

>
> > > Could you provide output from lspci -nnvv? So other people could look at
> > > it and maybe we decode what is this code doing and if it is needed.
> >
> > # lspci -nnvv
> > 00:02.0 PCI bridge [0604]: Device [0e8d:0801] (rev 01) (prog-if 00
> > [Normal decode])
>
> Hm... Device address is 02. But in your code is:
>
>     u8 num_slots_enabled = 0;
>     ...
>     list_for_each_entry(port, &pcie->ports, list) {
>         if (port->enabled) {
>             ...
>             num_slots_enabled++;
>             ...
>         }
>     }
>     ...
>     for (slot = 0; slot < num_slots_enabled; slot++) {
>         val = read_config(pcie, slot, ...);
>         ...
>         write_config(pcie, slot, ...);
>     }
>
> Which means that this code writes to config space of wrong device 0
> (instead of 2)! In function write_config() can be seen that second
> parameter specify device of BDF address for bus=0 and function=0.

Bridge enumeration depends on a virtual bridge register configuration.
But at the end devices connected to the bridge
are enumerated as 01:00.0, 02:00.0 and 03:00.0. So in this case the
phy used is the one for "pcie2" (00:02.0) and the device connected to
it
is 01:00.0. For example a board using all the virtual bridges will get
an output similar to:

[   16.487166] mt7621-pci 1e140000.pcie: host bridge /pcie@1e140000 ranges:
[   16.500627] mt7621-pci 1e140000.pcie:   No bus range found for
/pcie@1e140000, using [bus 00-ff]
[   16.518212] mt7621-pci 1e140000.pcie:      MEM
0x0060000000..0x006fffffff -> 0x0000000000
[   16.534531] mt7621-pci 1e140000.pcie:       IO
0x001e160000..0x001e16ffff -> 0x0000000000
[   16.786498] mt7621-pci 1e140000.pcie: PCIE0 enabled
[   16.796220] mt7621-pci 1e140000.pcie: PCIE1 enabled
[   16.805943] mt7621-pci 1e140000.pcie: PCIE2 enabled
[   16.815664] mt7621-pci 1e140000.pcie: PCI coherence region base:
0x60000000, mask/settings: 0xf0000002
[   16.834398] mt7621-pci 1e140000.pcie: PCI host bridge to bus 0000:00
[   16.847098] pci_bus 0000:00: root bus resource [io  0x1e160000-0x1e16ffff]
[   16.860806] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fffffff]
[   16.874504] pci_bus 0000:00: root bus resource [bus 00-ff]
[   16.885441] pci_bus 0000:00: root bus resource [mem
0x60000000-0x6fffffff] (bus address [0x00000000-0x0fffffff])
[   16.905773] pci 0000:00:00.0: [0e8d:0801] type 01 class 0x060400
[   16.917772] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x7fffffff]
[   16.930260] pci 0000:00:00.0: reg 0x14: initial BAR value 0x00000000 invalid
[   16.944304] pci 0000:00:00.0: reg 0x14: [mem size 0x00010000]
[   16.955841] pci 0000:00:00.0: supports D1
[   16.963837] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[   16.975747] pci 0000:00:01.0: [0e8d:0801] type 01 class 0x060400
[   16.987772] pci 0000:00:01.0: reg 0x10: [mem 0x00000000-0x7fffffff]
[   17.000273] pci 0000:00:01.0: reg 0x14: initial BAR value 0x00000000 invalid
[   17.014311] pci 0000:00:01.0: reg 0x14: [mem size 0x00010000]
[   17.025838] pci 0000:00:01.0: supports D1
[   17.033828] pci 0000:00:01.0: PME# supported from D0 D1 D3hot
[   17.045699] pci 0000:00:02.0: [0e8d:0801] type 01 class 0x060400
[   17.057726] pci 0000:00:02.0: reg 0x10: [mem 0x00000000-0x7fffffff]
[   17.070218] pci 0000:00:02.0: reg 0x14: initial BAR value 0x00000000 invalid
[   17.084260] pci 0000:00:02.0: reg 0x14: [mem size 0x00010000]
[   17.095788] pci 0000:00:02.0: supports D1
[   17.103785] pci 0000:00:02.0: PME# supported from D0 D1 D3hot
[   17.116598] pci 0000:00:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[   17.132566] pci 0000:00:01.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[   17.148514] pci 0000:00:02.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[   17.164739] pci 0000:01:00.0: [1b21:0611] type 00 class 0x010185
[   17.176775] pci 0000:01:00.0: reg 0x10: [io  0x0000-0x0007]
[   17.187892] pci 0000:01:00.0: reg 0x14: [io  0x0000-0x0003]
[   17.199009] pci 0000:01:00.0: reg 0x18: [io  0x0000-0x0007]
[   17.210130] pci 0000:01:00.0: reg 0x1c: [io  0x0000-0x0003]
[   17.221246] pci 0000:01:00.0: reg 0x20: [io  0x0000-0x000f]
[   17.232357] pci 0000:01:00.0: reg 0x24: initial BAR value 0x00000000 invalid
[   17.246411] pci 0000:01:00.0: reg 0x24: [mem size 0x00000200]
[   17.258031] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth,
limited by 2.5 GT/s PCIe x1 link at 0000:00:00.0 (capable of 4.000
Gb/s with 5.0 GT/s PCIe x1 link)
[   17.317563] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[   17.328033] pci 0000:00:00.0:   bridge window [io  0x0000-0x0fff]
[   17.340179] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[   17.353703] pci 0000:00:00.0:   bridge window [mem
0x60000000-0x600fffff pref]
[   17.368097] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[   17.381550] pci 0000:02:00.0: [1b21:0611] type 00 class 0x010185
[   17.393573] pci 0000:02:00.0: reg 0x10: [io  0x0000-0x0007]
[   17.404694] pci 0000:02:00.0: reg 0x14: [io  0x0000-0x0003]
[   17.415804] pci 0000:02:00.0: reg 0x18: [io  0x0000-0x0007]
[   17.426916] pci 0000:02:00.0: reg 0x1c: [io  0x0000-0x0003]
[   17.438040] pci 0000:02:00.0: reg 0x20: [io  0x0000-0x000f]
[   17.449148] pci 0000:02:00.0: reg 0x24: initial BAR value 0x00000000 invalid
[   17.463193] pci 0000:02:00.0: reg 0x24: [mem size 0x00000200]
[   17.474803] pci 0000:02:00.0: 2.000 Gb/s available PCIe bandwidth,
limited by 2.5 GT/s PCIe x1 link at 0000:00:01.0 (capable of 4.000
Gb/s with 5.0 GT/s PCIe x1 link)
[   17.527554] pci 0000:00:01.0: PCI bridge to [bus 02-ff]
[   17.538014] pci 0000:00:01.0:   bridge window [io  0x0000-0x0fff]
[   17.550159] pci 0000:00:01.0:   bridge window [mem 0x60000000-0x600fffff]
[   17.563682] pci 0000:00:01.0:   bridge window [mem
0x60000000-0x600fffff pref]
[   17.578078] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
[   17.591529] pci 0000:03:00.0: [1b21:0611] type 00 class 0x010185
[   17.603545] pci 0000:03:00.0: reg 0x10: [io  0x0000-0x0007]
[   17.614665] pci 0000:03:00.0: reg 0x14: [io  0x0000-0x0003]
[   17.625775] pci 0000:03:00.0: reg 0x18: [io  0x0000-0x0007]
[   17.636887] pci 0000:03:00.0: reg 0x1c: [io  0x0000-0x0003]
[   17.648009] pci 0000:03:00.0: reg 0x20: [io  0x0000-0x000f]
[   17.659119] pci 0000:03:00.0: reg 0x24: initial BAR value 0x00000000 invalid
[   17.673162] pci 0000:03:00.0: reg 0x24: [mem size 0x00000200]
[   17.684777] pci 0000:03:00.0: 2.000 Gb/s available PCIe bandwidth,
limited by 2.5 GT/s PCIe x1 link at 0000:00:02.0 (capable of 4.000
Gb/s with 5.0 GT/s PCIe x1 link)
[   17.737561] pci 0000:00:02.0: PCI bridge to [bus 03-ff]
[   17.748022] pci 0000:00:02.0:   bridge window [io  0x0000-0x0fff]
[   17.760167] pci 0000:00:02.0:   bridge window [mem 0x60000000-0x600fffff]
[   17.773690] pci 0000:00:02.0:   bridge window [mem
0x60000000-0x600fffff pref]
[   17.788085] pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03
[   17.801341] pci 0000:00:00.0: BAR 0: no space for [mem size 0x80000000]
[   17.814518] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x80000000]
[   17.828392] pci 0000:00:01.0: BAR 0: no space for [mem size 0x80000000]
[   17.841571] pci 0000:00:01.0: BAR 0: failed to assign [mem size 0x80000000]
[   17.855443] pci 0000:00:02.0: BAR 0: no space for [mem size 0x80000000]
[   17.868626] pci 0000:00:02.0: BAR 0: failed to assign [mem size 0x80000000]
[   17.882502] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[   17.896028] pci 0000:00:00.0: BAR 9: assigned [mem
0x60100000-0x601fffff pref]
[   17.910419] pci 0000:00:01.0: BAR 8: assigned [mem 0x60200000-0x602fffff]
[   17.923951] pci 0000:00:01.0: BAR 9: assigned [mem
0x60300000-0x603fffff pref]
[   17.938353] pci 0000:00:02.0: BAR 8: assigned [mem 0x60400000-0x604fffff]
[   17.951879] pci 0000:00:02.0: BAR 9: assigned [mem
0x60500000-0x605fffff pref]
[   17.966273] pci 0000:00:00.0: BAR 1: assigned [mem 0x60600000-0x6060ffff]
[   17.979809] pci 0000:00:01.0: BAR 1: assigned [mem 0x60610000-0x6061ffff]
[   17.993341] pci 0000:00:02.0: BAR 1: assigned [mem 0x60620000-0x6062ffff]
[   18.006877] pci 0000:00:00.0: BAR 7: assigned [io  0x1e160000-0x1e160fff]
[   18.020408] pci 0000:00:01.0: BAR 7: assigned [io  0x1e161000-0x1e161fff]
[   18.033932] pci 0000:00:02.0: BAR 7: assigned [io  0x1e162000-0x1e162fff]
[   18.047472] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
[   18.061005] pci 0000:01:00.0: BAR 4: assigned [io  0x1e160000-0x1e16000f]
[   18.074540] pci 0000:01:00.0: BAR 0: assigned [io  0x1e160010-0x1e160017]
[   18.088070] pci 0000:01:00.0: BAR 2: assigned [io  0x1e160018-0x1e16001f]
[   18.101606] pci 0000:01:00.0: BAR 1: assigned [io  0x1e160020-0x1e160023]
[   18.115139] pci 0000:01:00.0: BAR 3: assigned [io  0x1e160024-0x1e160027]
[   18.128676] pci 0000:00:00.0: PCI bridge to [bus 01]
[   18.138577] pci 0000:00:00.0:   bridge window [io  0x1e160000-0x1e160fff]
[   18.152100] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[   18.165627] pci 0000:00:00.0:   bridge window [mem
0x60100000-0x601fffff pref]
[   18.180030] pci 0000:02:00.0: BAR 5: assigned [mem 0x60200000-0x602001ff]
[   18.193566] pci 0000:02:00.0: BAR 4: assigned [io  0x1e161000-0x1e16100f]
[   18.207110] pci 0000:02:00.0: BAR 0: assigned [io  0x1e161010-0x1e161017]
[   18.220648] pci 0000:02:00.0: BAR 2: assigned [io  0x1e161018-0x1e16101f]
[   18.234183] pci 0000:02:00.0: BAR 1: assigned [io  0x1e161020-0x1e161023]
[   18.247723] pci 0000:02:00.0: BAR 3: assigned [io  0x1e161024-0x1e161027]
[   18.261257] pci 0000:00:01.0: PCI bridge to [bus 02]
[   18.271165] pci 0000:00:01.0:   bridge window [io  0x1e161000-0x1e161fff]
[   18.284695] pci 0000:00:01.0:   bridge window [mem 0x60200000-0x602fffff]
[   18.298225] pci 0000:00:01.0:   bridge window [mem
0x60300000-0x603fffff pref]
[   18.312630] pci 0000:03:00.0: BAR 5: assigned [mem 0x60400000-0x604001ff]
[   18.326166] pci 0000:03:00.0: BAR 4: assigned [io  0x1e162000-0x1e16200f]
[   18.339702] pci 0000:03:00.0: BAR 0: assigned [io  0x1e162010-0x1e162017]
[   18.353237] pci 0000:03:00.0: BAR 2: assigned [io  0x1e162018-0x1e16201f]
[   18.366775] pci 0000:03:00.0: BAR 1: assigned [io  0x1e162020-0x1e162023]
[   18.380311] pci 0000:03:00.0: BAR 3: assigned [io  0x1e162024-0x1e162027]
[   18.393841] pci 0000:00:02.0: PCI bridge to [bus 03]
[   18.403740] pci 0000:00:02.0:   bridge window [io  0x1e162000-0x1e162fff]
[   18.417270] pci 0000:00:02.0:   bridge window [mem 0x60400000-0x604fffff]
[   18.430801] pci 0000:00:02.0:   bridge window [mem
0x60500000-0x605fffff pref]
[   18.445529] ahci 0000:01:00.0: version 3.0
[   18.445559] pci 0000:00:00.0: enabling device (0000 -> 0003)
[   18.456853] ahci 0000:01:00.0: enabling device (0000 -> 0003)
[   18.468455] ahci 0000:01:00.0: SSS flag set, parallel bus scan disabled
[   18.481700] ahci 0000:01:00.0: AHCI 0001.0200 32 slots 2 ports 6
Gbps 0x3 impl IDE mode
[   18.497662] ahci 0000:01:00.0: flags: 64bit ncq sntf stag led clo
pmp pio slum part ccc sxs
[   18.516777] scsi host0: ahci
[   18.523629] scsi host1: ahci
[   18.529829] ata1: SATA max UDMA/133 abar m512@0x60000000 port
0x60000100 irq 22
[   18.544447] ata2: SATA max UDMA/133 abar m512@0x60000000 port
0x60000180 irq 22
[   18.559465] pci 0000:00:01.0: enabling device (0000 -> 0003)
[   18.570786] ahci 0000:02:00.0: enabling device (0000 -> 0003)
[   18.582414] ahci 0000:02:00.0: SSS flag set, parallel bus scan disabled
[   18.595665] ahci 0000:02:00.0: AHCI 0001.0200 32 slots 2 ports 6
Gbps 0x3 impl IDE mode
[   18.611614] ahci 0000:02:00.0: flags: 64bit ncq sntf stag led clo
pmp pio slum part ccc sxs
[   18.631053] scsi host2: ahci
[   18.637983] scsi host3: ahci
[   18.644138] ata3: SATA max UDMA/133 abar m512@0x60200000 port
0x60200100 irq 23
[   18.658792] ata4: SATA max UDMA/133 abar m512@0x60200000 port
0x60200180 irq 23
[   18.673827] pci 0000:00:02.0: enabling device (0000 -> 0003)
[   18.685151] ahci 0000:03:00.0: enabling device (0000 -> 0003)
[   18.696782] ahci 0000:03:00.0: SSS flag set, parallel bus scan disabled
[   18.710025] ahci 0000:03:00.0: AHCI 0001.0200 32 slots 2 ports 6
Gbps 0x3 impl IDE mode
[   18.725972] ahci 0000:03:00.0: flags: 64bit ncq sntf stag led clo
pmp pio slum part ccc sxs

And you are totally right, the setting is writing in the wrong place.
I changed the device tree and the way interrupts are mapped
to avoid using a custom 'map_irq' function [1]. Before that commit the
pci virtual bridge register was reordering the
buses enumeration depending on link status, so there I should also
properly rewrite the code in question.

I can rewrite the code to read and write config properly using the
slot moving the code into 'mt7621_pcie_enable_port' as follows:

static void mt7621_pcie_enable_port(struct mt7621_pcie_port *port)
{
    struct mt7621_pcie *pcie = port->pcie;
    u32 slot = port->slot;
    u32 offset = MT7621_PCIE_OFFSET + (slot * MT7621_NEXT_PORT);
    u32 val;

    /* enable pcie interrupt */
    val = pcie_read(pcie, RALINK_PCI_PCIMSK_ADDR);
    val |= PCIE_PORT_INT_EN(slot);

    /* map 2G DDR region */
    pcie_write(pcie, PCIE_BAR_MAP_MAX | PCIE_BAR_ENABLE,
           offset + RALINK_PCI_BAR0SETUP_ADDR);

    /* configure class code and revision ID */
    pcie_write(pcie, PCIE_CLASS_CODE | PCIE_REVISION_ID,
           offset + RALINK_PCI_CLASS);

    /* configure RC FTS number to 250 when it leaves L0s */
    val = read_config(pcie, slot, PCIE_FTS_NUM);
    val &= ~PCIE_FTS_NUM_MASK;
    val |= PCIE_FTS_NUM_L0(0x50);
    write_config(pcie, slot, PCIE_FTS_NUM, val);
}

static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
{
    struct device *dev = pcie->dev;
    struct mt7621_pcie_port *port;
    int err;

    /* Setup MEMWIN and IOWIN */
    pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
    pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);

    list_for_each_entry(port, &pcie->ports, list) {
        if (port->enabled) {
            err = clk_prepare_enable(port->clk);
            if (err) {
                dev_err(dev, "enabling clk pcie%d\n",
                    port->slot);
                return err;
            }

            mt7621_pcie_enable_port(port);
            dev_info(dev, "PCIE%d enabled\n", port->slot);
        }
    }

    return 0;
}

Or just delete the setting and the read and write config functions
since they are not being used in any other place. My two boards work
without this setting but I don't know about other boards.

What do you think?

Best regards,
    Sergio Paracuellos

[0]: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/tree/drivers/staging/mt7621-pci/pci-mt7621.c?h=v4.18#n663
[1]: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/commit/drivers/staging/mt7621-pci?h=staging-testing&id=b99cc3a2b6b62cf994acac5cced03298d9908c9b

>
> >         Device tree node: /sys/firmware/devicetree/base/pcie@1e140000/pcie@2,0
> >         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> > ParErr- Stepping- SERR- FastB2B- DisINTx-
> >         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> > <TAbort- <MAbort- >SERR- <PERR- INTx-
> >         Latency: 0
> >         Interrupt: pin A routed to IRQ 255
> >         Region 1: Memory at 60200000 (32-bit, non-prefetchable) [size=64K]
> >         Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
> >         I/O behind bridge: 00000000-00000fff [size=4K]
> >         Memory behind bridge: 60000000-600fffff [size=1M]
> >         Prefetchable memory behind bridge: 60100000-601fffff [size=1M]
> >         Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> > <TAbort- <MAbort- <SERR- <PERR-
> >         BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
> >                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> >         Capabilities: [40] Power Management version 3
> >                 Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA
> > PME(D0+,D1+,D2-,D3hot+,D3cold-)
> >                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
> >         Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
> >                 Address: 0000000000000000  Data: 0000
> >         Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
> >                 DevCap: MaxPayload 128 bytes, PhantFunc 0
> >                         ExtTag- RBE+
> >                 DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
> >                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
> >                         MaxPayload 128 bytes, MaxReadReq 128 bytes
> >                 DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq-
> > AuxPwr- TransPend-
> >                 LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1,
> > Exit Latency L0s <512ns, L1 <64us
> >                         ClockPM- Surprise- LLActRep+ BwNot- ASPMOptComp-
> >                 LnkCtl: ASPM Disabled; RCB 128 bytes, Disabled- CommClk-
> >                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> >                 LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
> >                         TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
> >                 RootCap: CRSVisible-
> >                 RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
> > PMEIntEna- CRSVisible-
> >                 RootSta: PME ReqID 0000, PMEStatus- PMEPending-
> >                 DevCap2: Completion Timeout: Not Supported,
> > TimeoutDis+ NROPrPrP- LTR-
> >                          10BitTagComp- 10BitTagReq- OBFF Not
> > Supported, ExtFmt- EETLPPrefix-
> >                          EmergencyPowerReduction Not Supported,
> > EmergencyPowerReductionInit-
> >                          FRS- LN System CLS Not Supported, TPHComp-
> > ExtTPHComp- ARIFwd-
> >                          AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
> >                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
> > LTR- OBFF Disabled, ARIFwd-
> >                          AtomicOpsCtl: ReqEn- EgressBlck-
> >                 LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink-
> > Retimer- 2Retimers- DRS-
> >                 LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
> >                          Transmit Margin: Normal Operating Range,
> > EnterModifiedCompliance- ComplianceSOS-
> >                          Compliance De-emphasis: -6dB
> >                 LnkSta2: Current De-emphasis Level: -6dB,
> > EqualizationComplete- EqualizationPhase1-
> >                          EqualizationPhase2- EqualizationPhase3-
> > LinkEqualizationRequest-
> >                          Retimer- 2Retimers- CrosslinkRes: unsupported
> >         Capabilities: [100 v1] Advanced Error Reporting
> >                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> >                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> >                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
> > UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> >                 CESta:  RxErr+ BadTLP- BadDLLP- Rollover- Timeout-
> > AdvNonFatalErr-
> >                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> > AdvNonFatalErr+
> >                 AERCap: First Error Pointer: 00, ECRCGenCap+
> > ECRCGenEn- ECRCChkCap+ ECRCChkEn-
> >                         MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
> >                 HeaderLog: 00000000 00000000 00000000 00000000
> >                 RootCmd: CERptEn- NFERptEn- FERptEn-
> >                 RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
> >                          FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
> >                 ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
> >         Capabilities: [140 v1] Virtual Channel
> >                 Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
> >                 Arb:    Fixed- WRR32- WRR64- WRR128-
> >                 Ctrl:   ArbSelect=Fixed
> >                 Status: InProgress-
> >                 VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
> >                         Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
> >                         Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
> >                         Status: NegoPending- InProgress-
> > lspci: Unable to load libkmod resources: error -12
> >
> > 01:00.0 Network controller [0280]: MEDIATEK Corp. Device [14c3:7612]
> >         Subsystem: MEDIATEK Corp. Device [14c3:7612]
> >         Device tree node:
> > /sys/firmware/devicetree/base/pcie@1e140000/pcie@2,0/wifi@0,0
> >         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> > ParErr- Stepping- SERR- FastB2B- DisINTx-
> >         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> > <TAbort- <MAbort- >SERR- <PERR- INTx-
> >         Latency: 0
> >         Interrupt: pin A routed to IRQ 20
> >         Region 0: Memory at 60000000 (64-bit, non-prefetchable) [size=1M]
> >         Expansion ROM at 60100000 [virtual] [disabled] [size=64K]
> >         Capabilities: [40] Power Management version 3
> >                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA
> > PME(D0+,D1-,D2-,D3hot+,D3cold+)
> >                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
> >         Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
> >                 Address: 0000000000000000  Data: 0000
> >         Capabilities: [70] Express (v2) Endpoint, MSI 00
> >                 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
> > unlimited, L1 unlimited
> >                         ExtTag- AttnBtn- AttnInd- PwrInd- RBE+
> > FLReset- SlotPowerLimit 0.000W
> >                 DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
> >                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
> >                         MaxPayload 128 bytes, MaxReadReq 128 bytes
> >                 DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq-
> > AuxPwr+ TransPend-
> >                 LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1,
> > Exit Latency L0s <2us, L1 unlimited
> >                         ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
> >                 LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
> >                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> >                 LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
> >                         TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> >                 DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
> > NROPrPrP- LTR-
> >                          10BitTagComp- 10BitTagReq- OBFF Not
> > Supported, ExtFmt- EETLPPrefix-
> >                          EmergencyPowerReduction Not Supported,
> > EmergencyPowerReductionInit-
> >                          FRS- TPHComp- ExtTPHComp-
> >                          AtomicOpsCap: 32bit- 64bit- 128bitCAS-
> >                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
> > LTR- OBFF Disabled,
> >                          AtomicOpsCtl: ReqEn-
> >                 LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink-
> > Retimer- 2Retimers- DRS-
> >                 LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
> >                          Transmit Margin: Normal Operating Range,
> > EnterModifiedCompliance- ComplianceSOS-
> >                          Compliance De-emphasis: -6dB
> >                 LnkSta2: Current De-emphasis Level: -3.5dB,
> > EqualizationComplete- EqualizationPhase1-
> >                          EqualizationPhase2- EqualizationPhase3-
> > LinkEqualizationRequest-
> >                          Retimer- 2Retimers- CrosslinkRes: unsupported
> >         Capabilities: [100 v2] Advanced Error Reporting
> >                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> >                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
> > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> >                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
> > UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> >                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> > AdvNonFatalErr-
> >                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
> > AdvNonFatalErr+
> >                 AERCap: First Error Pointer: 00, ECRCGenCap+
> > ECRCGenEn- ECRCChkCap+ ECRCChkEn-
> >                         MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
> >                 HeaderLog: 00000000 00000000 00000000 00000000
> >         Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
> >         Capabilities: [158 v1] Latency Tolerance Reporting
> >                 Max snoop latency: 0ns
> >                 Max no snoop latency: 0ns
> >         Capabilities: [160 v1] L1 PM Substates
> >                 L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+
> > ASPM_L1.1+ L1_PM_Substates+
> >                           PortCommonModeRestoreTime=50us PortTPowerOnTime=10us
> >                 L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
> >                            T_CommonMode=0us LTR1.2_Threshold=0ns
> >                 L1SubCtl2: T_PwrOn=10us
> >         Kernel driver in use: mt76x2e
> >
> > Best regards,
> >     Sergio Paracuellos
> >
> > >
> > > > > > > > > +     }
> > > > > > > > > +
> > > > > > > > > +     return 0;
> > > > > > > > > +}

Re: [PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Pali Rohár]

On Friday 04 June 2021 20:44:42 Sergio Paracuellos wrote:
> Hi Pali,
> 
> Thanks for your comments.
> 
> On Fri, Jun 4, 2021 at 6:55 PM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Wednesday 02 June 2021 14:43:53 Sergio Paracuellos wrote:
> > > Hi Pali,
> > >
> > > On Wed, Jun 2, 2021 at 2:23 PM Pali Rohár <pali@kernel.org> wrote:
> > > >
> > > > On Wednesday 02 June 2021 14:16:26 Sergio Paracuellos wrote:
> > > > > Hi Pali,
> > > > >
> > > > > On Mon, May 31, 2021 at 4:19 PM Sergio Paracuellos
> > > > > <sergio.paracuellos@gmail.com> wrote:
> > > > > >
> > > > > > On Mon, May 31, 2021 at 3:50 PM Pali Rohár <pali@kernel.org> wrote:
> > > > > > >
> > > > > > > On Monday 31 May 2021 15:39:55 Sergio Paracuellos wrote:
> > > > > > > > Hi Pali,
> > > > > > > >
> > > > > > > > Thanks for your comments.
> > > > > > > >
> > > > > > > > On Mon, May 31, 2021 at 3:14 PM Pali Rohár <pali@kernel.org> wrote:
> > > > > > > > >
> > > > > > > > > On Saturday 15 May 2021 14:40:53 Sergio Paracuellos wrote:
> > > > > > > > > > This patch adds a driver for the PCIe controller of MT7621 SoC.
> > > > > > > > > >
> > > > > > > > > > Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > > > > > > > > > ---
> > > > > > > > > >  arch/mips/pci/Makefile     |   1 +
> > > > > > > > > >  arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
> > > > > > > > > >  arch/mips/ralink/Kconfig   |   9 +-
> > > > > > > > > >  3 files changed, 633 insertions(+), 1 deletion(-)
> > > > > > > > > >  create mode 100644 arch/mips/pci/pci-mt7621.c
> > > > > > > > > >
> > > > > > > > > > diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
> > > > > > > > > > index f3eecc065e5c..178c550739c4 100644
> > > > > > > > > > --- a/arch/mips/pci/Makefile
> > > > > > > > > > +++ b/arch/mips/pci/Makefile
> > > > > > > > > > @@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)    += pci-ar2315.o
> > > > > > > > > >  obj-$(CONFIG_SOC_AR71XX)     += pci-ar71xx.o
> > > > > > > > > >  obj-$(CONFIG_PCI_AR724X)     += pci-ar724x.o
> > > > > > > > > >  obj-$(CONFIG_PCI_XTALK_BRIDGE)       += pci-xtalk-bridge.o
> > > > > > > > > > +obj-$(CONFIG_PCI_MT7621)     += pci-mt7621.o
> > > > > > > > > >  #
> > > > > > > > > >  # These are still pretty much in the old state, watch, go blind.
> > > > > > > > > >  #
> > > > > > > > > > diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
> > > > > > > > > > new file mode 100644
> > > > > > > > > > index 000000000000..fe1945819d25
> > > > > > > > > > --- /dev/null
> > > > > > > > > > +++ b/arch/mips/pci/pci-mt7621.c
> > > > > > > > > ...
> > > > > > > > > > +static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> > > > > > > > > > +{
> > > > > > > > > > +     struct device *dev = pcie->dev;
> > > > > > > > > > +     struct mt7621_pcie_port *port;
> > > > > > > > > > +     u8 num_slots_enabled = 0;
> > > > > > > > > > +     u32 slot;
> > > > > > > > > > +     u32 val;
> > > > > > > > > > +     int err;
> > > > > > > > > > +
> > > > > > > > > > +     /* Setup MEMWIN and IOWIN */
> > > > > > > > > > +     pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> > > > > > > > > > +     pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> > > > > > > > > > +
> > > > > > > > > > +     list_for_each_entry(port, &pcie->ports, list) {
> > > > > > > > > > +             if (port->enabled) {
> > > > > > > > > > +                     err = clk_prepare_enable(port->clk);
> > > > > > > > > > +                     if (err) {
> > > > > > > > > > +                             dev_err(dev, "enabling clk pcie%d\n", slot);
> > > > > > > > > > +                             return err;
> > > > > > > > > > +                     }
> > > > > > > > > > +
> > > > > > > > > > +                     mt7621_pcie_enable_port(port);
> > > > > > > > > > +                     dev_info(dev, "PCIE%d enabled\n", port->slot);
> > > > > > > > > > +                     num_slots_enabled++;
> > > > > > > > > > +             }
> > > > > > > > > > +     }
> > > > > > > > > > +
> > > > > > > > > > +     for (slot = 0; slot < num_slots_enabled; slot++) {
> > > > > > > > > > +             val = read_config(pcie, slot, PCI_COMMAND);
> > > > > > > > > > +             val |= PCI_COMMAND_MASTER;
> > > > > > > > > > +             write_config(pcie, slot, PCI_COMMAND, val);
> > > > > > > > >
> > > > > > > > > Hello! Is this part of code correct? Because it looks strange if PCIe
> > > > > > > > > controller driver automatically enables PCI bus mastering, prior device
> > > > > > > > > driver initialize itself.
> > > > > > > > >
> > > > > > > > > Moreover kernel has already function pci_set_master() for this purpose
> > > > > > > > > which is used by device drivers.
> > > > > > > > >
> > > > > > > > > So I think this code can confuse some device drivers...
> > > > > > > >
> > > > > > > > I agree that we have pci_set_master() to be used in pci device driver
> > > > > > > > code. Original controller driver set this bit for enabled slots. Since
> > > > > > > > there is no documentation at all for the PCI in this SoC
> > > > > > >
> > > > > > > I see... this is really a big problem to do any driver development...
> > > > > >
> > > > > > For sure it is :(.
> > > > > >
> > > > > > >
> > > > > > > > I have
> > > > > > > > maintained the setting in the driver in a cleaner way. See original
> > > > > > > > driver code and the setting here [0]. There is no other reason than
> > > > > > > > that. I am ok with removing this from here and testing with my two
> > > > > > > > devices that everything is still ok if having this setting in the pci
> > > > > > > > controller driver is a real problem.
> > > > > > >
> > > > > > > You can run lspci -nnvv with and without PCI_COMMAND_MASTER code and
> > > > > > > then compare outputs.
> > > > > > >
> > > > > > > Device drivers for sure enable PCI_COMMAND_MASTER at the time when it is
> > > > > > > needed, so it is possible that there would be no difference in lspci
> > > > > > > output.
> > > > > >
> > > > > > Thanks. I will take this into account when v2 is submitted after more
> > > > > > review comments come :).
> > > > >
> > > > > I have tested to remove this and check lspci -nnvv output with and
> > > > > without PCI_COMMAND_MASTER code and, as you pointed out, there is no
> > > > > difference between them. Also, both boards are working without
> > > > > regressions at all. So I will remove this code for next version.
> > > >
> > > > Perfect!
> > > >
> > > > > Thanks,
> > > > >     Sergio Paracuellos
> > > > > >
> > > > > > >
> > > > > > > > [0]: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/tree/drivers/staging/mt7621-pci/pci-mt7621.c?h=v4.18#n676
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > >     Sergio Paracuellos
> > > > > > > > >
> > > > > > > > > > +             /* configure RC FTS number to 250 when it leaves L0s */
> > > > > > > > > > +             val = read_config(pcie, slot, PCIE_FTS_NUM);
> > > > > > > > > > +             val &= ~PCIE_FTS_NUM_MASK;
> > > > > > > > > > +             val |= PCIE_FTS_NUM_L0(0x50);
> > > > > > > > > > +             write_config(pcie, slot, PCIE_FTS_NUM, val);
> > > >
> > > > Could you look also what is doing this code (PCIE_FTS_NUM)? It is marked
> > > > as MT specific register. But from this code for me it looks like that it
> > > > just access config space of some device and therefore it could be some
> > > > standard PCIe register. Just with hardcoded calculated offset.
> >
> > So based on your lspci output, there is no PCIe capability register at
> > address PCIE_FTS_NUM (0x70c), right? It seems strange to trying access
> > capability register outside of capability list.
> 
> This setting is also in the original driver code from mediatek SDK. See [0].
> Anyway, I have tried to remove this code and test what happens with my
> two boards and both of them seem to
> work properly just by deleting this code.

Looks like that this code is some designware stuff which seems to be
needed. So it would be better to let it here as is.

> > > > Could you provide output from lspci -nnvv? So other people could look at
> > > > it and maybe we decode what is this code doing and if it is needed.
> > >
> > > # lspci -nnvv
> > > 00:02.0 PCI bridge [0604]: Device [0e8d:0801] (rev 01) (prog-if 00
> > > [Normal decode])
> >
> > Hm... Device address is 02. But in your code is:
> >
> >     u8 num_slots_enabled = 0;
> >     ...
> >     list_for_each_entry(port, &pcie->ports, list) {
> >         if (port->enabled) {
> >             ...
> >             num_slots_enabled++;
> >             ...
> >         }
> >     }
> >     ...
> >     for (slot = 0; slot < num_slots_enabled; slot++) {
> >         val = read_config(pcie, slot, ...);
> >         ...
> >         write_config(pcie, slot, ...);
> >     }
> >
> > Which means that this code writes to config space of wrong device 0
> > (instead of 2)! In function write_config() can be seen that second
> > parameter specify device of BDF address for bus=0 and function=0.
> 
> Bridge enumeration depends on a virtual bridge register configuration.
> But at the end devices connected to the bridge
> are enumerated as 01:00.0, 02:00.0 and 03:00.0. So in this case the
> phy used is the one for "pcie2" (00:02.0) and the device connected to
> it
> is 01:00.0. For example a board using all the virtual bridges will get
> an output similar to:
> 
> [   16.487166] mt7621-pci 1e140000.pcie: host bridge /pcie@1e140000 ranges:
> [   16.500627] mt7621-pci 1e140000.pcie:   No bus range found for
> /pcie@1e140000, using [bus 00-ff]
> [   16.518212] mt7621-pci 1e140000.pcie:      MEM
> 0x0060000000..0x006fffffff -> 0x0000000000
> [   16.534531] mt7621-pci 1e140000.pcie:       IO
> 0x001e160000..0x001e16ffff -> 0x0000000000
> [   16.786498] mt7621-pci 1e140000.pcie: PCIE0 enabled
> [   16.796220] mt7621-pci 1e140000.pcie: PCIE1 enabled
> [   16.805943] mt7621-pci 1e140000.pcie: PCIE2 enabled
> [   16.815664] mt7621-pci 1e140000.pcie: PCI coherence region base:
> 0x60000000, mask/settings: 0xf0000002
> [   16.834398] mt7621-pci 1e140000.pcie: PCI host bridge to bus 0000:00
> [   16.847098] pci_bus 0000:00: root bus resource [io  0x1e160000-0x1e16ffff]
> [   16.860806] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fffffff]
> [   16.874504] pci_bus 0000:00: root bus resource [bus 00-ff]
> [   16.885441] pci_bus 0000:00: root bus resource [mem
> 0x60000000-0x6fffffff] (bus address [0x00000000-0x0fffffff])
> [   16.905773] pci 0000:00:00.0: [0e8d:0801] type 01 class 0x060400
> [   16.917772] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x7fffffff]
> [   16.930260] pci 0000:00:00.0: reg 0x14: initial BAR value 0x00000000 invalid
> [   16.944304] pci 0000:00:00.0: reg 0x14: [mem size 0x00010000]
> [   16.955841] pci 0000:00:00.0: supports D1
> [   16.963837] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
> [   16.975747] pci 0000:00:01.0: [0e8d:0801] type 01 class 0x060400
> [   16.987772] pci 0000:00:01.0: reg 0x10: [mem 0x00000000-0x7fffffff]
> [   17.000273] pci 0000:00:01.0: reg 0x14: initial BAR value 0x00000000 invalid
> [   17.014311] pci 0000:00:01.0: reg 0x14: [mem size 0x00010000]
> [   17.025838] pci 0000:00:01.0: supports D1
> [   17.033828] pci 0000:00:01.0: PME# supported from D0 D1 D3hot
> [   17.045699] pci 0000:00:02.0: [0e8d:0801] type 01 class 0x060400
> [   17.057726] pci 0000:00:02.0: reg 0x10: [mem 0x00000000-0x7fffffff]
> [   17.070218] pci 0000:00:02.0: reg 0x14: initial BAR value 0x00000000 invalid
> [   17.084260] pci 0000:00:02.0: reg 0x14: [mem size 0x00010000]
> [   17.095788] pci 0000:00:02.0: supports D1
> [   17.103785] pci 0000:00:02.0: PME# supported from D0 D1 D3hot
> [   17.116598] pci 0000:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
> [   17.132566] pci 0000:00:01.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
> [   17.148514] pci 0000:00:02.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
> [   17.164739] pci 0000:01:00.0: [1b21:0611] type 00 class 0x010185
> [   17.176775] pci 0000:01:00.0: reg 0x10: [io  0x0000-0x0007]
> [   17.187892] pci 0000:01:00.0: reg 0x14: [io  0x0000-0x0003]
> [   17.199009] pci 0000:01:00.0: reg 0x18: [io  0x0000-0x0007]
> [   17.210130] pci 0000:01:00.0: reg 0x1c: [io  0x0000-0x0003]
> [   17.221246] pci 0000:01:00.0: reg 0x20: [io  0x0000-0x000f]
> [   17.232357] pci 0000:01:00.0: reg 0x24: initial BAR value 0x00000000 invalid
> [   17.246411] pci 0000:01:00.0: reg 0x24: [mem size 0x00000200]
> [   17.258031] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth,
> limited by 2.5 GT/s PCIe x1 link at 0000:00:00.0 (capable of 4.000
> Gb/s with 5.0 GT/s PCIe x1 link)
> [   17.317563] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
> [   17.328033] pci 0000:00:00.0:   bridge window [io  0x0000-0x0fff]
> [   17.340179] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
> [   17.353703] pci 0000:00:00.0:   bridge window [mem
> 0x60000000-0x600fffff pref]
> [   17.368097] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
> [   17.381550] pci 0000:02:00.0: [1b21:0611] type 00 class 0x010185
> [   17.393573] pci 0000:02:00.0: reg 0x10: [io  0x0000-0x0007]
> [   17.404694] pci 0000:02:00.0: reg 0x14: [io  0x0000-0x0003]
> [   17.415804] pci 0000:02:00.0: reg 0x18: [io  0x0000-0x0007]
> [   17.426916] pci 0000:02:00.0: reg 0x1c: [io  0x0000-0x0003]
> [   17.438040] pci 0000:02:00.0: reg 0x20: [io  0x0000-0x000f]
> [   17.449148] pci 0000:02:00.0: reg 0x24: initial BAR value 0x00000000 invalid
> [   17.463193] pci 0000:02:00.0: reg 0x24: [mem size 0x00000200]
> [   17.474803] pci 0000:02:00.0: 2.000 Gb/s available PCIe bandwidth,
> limited by 2.5 GT/s PCIe x1 link at 0000:00:01.0 (capable of 4.000
> Gb/s with 5.0 GT/s PCIe x1 link)
> [   17.527554] pci 0000:00:01.0: PCI bridge to [bus 02-ff]
> [   17.538014] pci 0000:00:01.0:   bridge window [io  0x0000-0x0fff]
> [   17.550159] pci 0000:00:01.0:   bridge window [mem 0x60000000-0x600fffff]
> [   17.563682] pci 0000:00:01.0:   bridge window [mem
> 0x60000000-0x600fffff pref]
> [   17.578078] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
> [   17.591529] pci 0000:03:00.0: [1b21:0611] type 00 class 0x010185
> [   17.603545] pci 0000:03:00.0: reg 0x10: [io  0x0000-0x0007]
> [   17.614665] pci 0000:03:00.0: reg 0x14: [io  0x0000-0x0003]
> [   17.625775] pci 0000:03:00.0: reg 0x18: [io  0x0000-0x0007]
> [   17.636887] pci 0000:03:00.0: reg 0x1c: [io  0x0000-0x0003]
> [   17.648009] pci 0000:03:00.0: reg 0x20: [io  0x0000-0x000f]
> [   17.659119] pci 0000:03:00.0: reg 0x24: initial BAR value 0x00000000 invalid
> [   17.673162] pci 0000:03:00.0: reg 0x24: [mem size 0x00000200]
> [   17.684777] pci 0000:03:00.0: 2.000 Gb/s available PCIe bandwidth,
> limited by 2.5 GT/s PCIe x1 link at 0000:00:02.0 (capable of 4.000
> Gb/s with 5.0 GT/s PCIe x1 link)
> [   17.737561] pci 0000:00:02.0: PCI bridge to [bus 03-ff]
> [   17.748022] pci 0000:00:02.0:   bridge window [io  0x0000-0x0fff]
> [   17.760167] pci 0000:00:02.0:   bridge window [mem 0x60000000-0x600fffff]
> [   17.773690] pci 0000:00:02.0:   bridge window [mem
> 0x60000000-0x600fffff pref]
> [   17.788085] pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03
> [   17.801341] pci 0000:00:00.0: BAR 0: no space for [mem size 0x80000000]
> [   17.814518] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x80000000]
> [   17.828392] pci 0000:00:01.0: BAR 0: no space for [mem size 0x80000000]
> [   17.841571] pci 0000:00:01.0: BAR 0: failed to assign [mem size 0x80000000]
> [   17.855443] pci 0000:00:02.0: BAR 0: no space for [mem size 0x80000000]
> [   17.868626] pci 0000:00:02.0: BAR 0: failed to assign [mem size 0x80000000]
> [   17.882502] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
> [   17.896028] pci 0000:00:00.0: BAR 9: assigned [mem
> 0x60100000-0x601fffff pref]
> [   17.910419] pci 0000:00:01.0: BAR 8: assigned [mem 0x60200000-0x602fffff]
> [   17.923951] pci 0000:00:01.0: BAR 9: assigned [mem
> 0x60300000-0x603fffff pref]
> [   17.938353] pci 0000:00:02.0: BAR 8: assigned [mem 0x60400000-0x604fffff]
> [   17.951879] pci 0000:00:02.0: BAR 9: assigned [mem
> 0x60500000-0x605fffff pref]
> [   17.966273] pci 0000:00:00.0: BAR 1: assigned [mem 0x60600000-0x6060ffff]
> [   17.979809] pci 0000:00:01.0: BAR 1: assigned [mem 0x60610000-0x6061ffff]
> [   17.993341] pci 0000:00:02.0: BAR 1: assigned [mem 0x60620000-0x6062ffff]
> [   18.006877] pci 0000:00:00.0: BAR 7: assigned [io  0x1e160000-0x1e160fff]
> [   18.020408] pci 0000:00:01.0: BAR 7: assigned [io  0x1e161000-0x1e161fff]
> [   18.033932] pci 0000:00:02.0: BAR 7: assigned [io  0x1e162000-0x1e162fff]
> [   18.047472] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
> [   18.061005] pci 0000:01:00.0: BAR 4: assigned [io  0x1e160000-0x1e16000f]
> [   18.074540] pci 0000:01:00.0: BAR 0: assigned [io  0x1e160010-0x1e160017]
> [   18.088070] pci 0000:01:00.0: BAR 2: assigned [io  0x1e160018-0x1e16001f]
> [   18.101606] pci 0000:01:00.0: BAR 1: assigned [io  0x1e160020-0x1e160023]
> [   18.115139] pci 0000:01:00.0: BAR 3: assigned [io  0x1e160024-0x1e160027]
> [   18.128676] pci 0000:00:00.0: PCI bridge to [bus 01]
> [   18.138577] pci 0000:00:00.0:   bridge window [io  0x1e160000-0x1e160fff]
> [   18.152100] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
> [   18.165627] pci 0000:00:00.0:   bridge window [mem
> 0x60100000-0x601fffff pref]
> [   18.180030] pci 0000:02:00.0: BAR 5: assigned [mem 0x60200000-0x602001ff]
> [   18.193566] pci 0000:02:00.0: BAR 4: assigned [io  0x1e161000-0x1e16100f]
> [   18.207110] pci 0000:02:00.0: BAR 0: assigned [io  0x1e161010-0x1e161017]
> [   18.220648] pci 0000:02:00.0: BAR 2: assigned [io  0x1e161018-0x1e16101f]
> [   18.234183] pci 0000:02:00.0: BAR 1: assigned [io  0x1e161020-0x1e161023]
> [   18.247723] pci 0000:02:00.0: BAR 3: assigned [io  0x1e161024-0x1e161027]
> [   18.261257] pci 0000:00:01.0: PCI bridge to [bus 02]
> [   18.271165] pci 0000:00:01.0:   bridge window [io  0x1e161000-0x1e161fff]
> [   18.284695] pci 0000:00:01.0:   bridge window [mem 0x60200000-0x602fffff]
> [   18.298225] pci 0000:00:01.0:   bridge window [mem
> 0x60300000-0x603fffff pref]
> [   18.312630] pci 0000:03:00.0: BAR 5: assigned [mem 0x60400000-0x604001ff]
> [   18.326166] pci 0000:03:00.0: BAR 4: assigned [io  0x1e162000-0x1e16200f]
> [   18.339702] pci 0000:03:00.0: BAR 0: assigned [io  0x1e162010-0x1e162017]
> [   18.353237] pci 0000:03:00.0: BAR 2: assigned [io  0x1e162018-0x1e16201f]
> [   18.366775] pci 0000:03:00.0: BAR 1: assigned [io  0x1e162020-0x1e162023]
> [   18.380311] pci 0000:03:00.0: BAR 3: assigned [io  0x1e162024-0x1e162027]
> [   18.393841] pci 0000:00:02.0: PCI bridge to [bus 03]
> [   18.403740] pci 0000:00:02.0:   bridge window [io  0x1e162000-0x1e162fff]
> [   18.417270] pci 0000:00:02.0:   bridge window [mem 0x60400000-0x604fffff]
> [   18.430801] pci 0000:00:02.0:   bridge window [mem
> 0x60500000-0x605fffff pref]
> [   18.445529] ahci 0000:01:00.0: version 3.0
> [   18.445559] pci 0000:00:00.0: enabling device (0000 -> 0003)
> [   18.456853] ahci 0000:01:00.0: enabling device (0000 -> 0003)
> [   18.468455] ahci 0000:01:00.0: SSS flag set, parallel bus scan disabled
> [   18.481700] ahci 0000:01:00.0: AHCI 0001.0200 32 slots 2 ports 6
> Gbps 0x3 impl IDE mode
> [   18.497662] ahci 0000:01:00.0: flags: 64bit ncq sntf stag led clo
> pmp pio slum part ccc sxs
> [   18.516777] scsi host0: ahci
> [   18.523629] scsi host1: ahci
> [   18.529829] ata1: SATA max UDMA/133 abar m512@0x60000000 port
> 0x60000100 irq 22
> [   18.544447] ata2: SATA max UDMA/133 abar m512@0x60000000 port
> 0x60000180 irq 22
> [   18.559465] pci 0000:00:01.0: enabling device (0000 -> 0003)
> [   18.570786] ahci 0000:02:00.0: enabling device (0000 -> 0003)
> [   18.582414] ahci 0000:02:00.0: SSS flag set, parallel bus scan disabled
> [   18.595665] ahci 0000:02:00.0: AHCI 0001.0200 32 slots 2 ports 6
> Gbps 0x3 impl IDE mode
> [   18.611614] ahci 0000:02:00.0: flags: 64bit ncq sntf stag led clo
> pmp pio slum part ccc sxs
> [   18.631053] scsi host2: ahci
> [   18.637983] scsi host3: ahci
> [   18.644138] ata3: SATA max UDMA/133 abar m512@0x60200000 port
> 0x60200100 irq 23
> [   18.658792] ata4: SATA max UDMA/133 abar m512@0x60200000 port
> 0x60200180 irq 23
> [   18.673827] pci 0000:00:02.0: enabling device (0000 -> 0003)
> [   18.685151] ahci 0000:03:00.0: enabling device (0000 -> 0003)
> [   18.696782] ahci 0000:03:00.0: SSS flag set, parallel bus scan disabled
> [   18.710025] ahci 0000:03:00.0: AHCI 0001.0200 32 slots 2 ports 6
> Gbps 0x3 impl IDE mode
> [   18.725972] ahci 0000:03:00.0: flags: 64bit ncq sntf stag led clo
> pmp pio slum part ccc sxs
> 
> And you are totally right, the setting is writing in the wrong place.
> I changed the device tree and the way interrupts are mapped
> to avoid using a custom 'map_irq' function [1]. Before that commit the
> pci virtual bridge register was reordering the
> buses enumeration depending on link status, so there I should also
> properly rewrite the code in question.
> 
> I can rewrite the code to read and write config properly using the
> slot moving the code into 'mt7621_pcie_enable_port' as follows:
> 
> static void mt7621_pcie_enable_port(struct mt7621_pcie_port *port)
> {
>     struct mt7621_pcie *pcie = port->pcie;
>     u32 slot = port->slot;
>     u32 offset = MT7621_PCIE_OFFSET + (slot * MT7621_NEXT_PORT);
>     u32 val;
> 
>     /* enable pcie interrupt */
>     val = pcie_read(pcie, RALINK_PCI_PCIMSK_ADDR);
>     val |= PCIE_PORT_INT_EN(slot);
> 
>     /* map 2G DDR region */
>     pcie_write(pcie, PCIE_BAR_MAP_MAX | PCIE_BAR_ENABLE,
>            offset + RALINK_PCI_BAR0SETUP_ADDR);
> 
>     /* configure class code and revision ID */
>     pcie_write(pcie, PCIE_CLASS_CODE | PCIE_REVISION_ID,
>            offset + RALINK_PCI_CLASS);
> 
>     /* configure RC FTS number to 250 when it leaves L0s */
>     val = read_config(pcie, slot, PCIE_FTS_NUM);
>     val &= ~PCIE_FTS_NUM_MASK;
>     val |= PCIE_FTS_NUM_L0(0x50);
>     write_config(pcie, slot, PCIE_FTS_NUM, val);
> }
> 
> static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> {
>     struct device *dev = pcie->dev;
>     struct mt7621_pcie_port *port;
>     int err;
> 
>     /* Setup MEMWIN and IOWIN */
>     pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
>     pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> 
>     list_for_each_entry(port, &pcie->ports, list) {
>         if (port->enabled) {
>             err = clk_prepare_enable(port->clk);
>             if (err) {
>                 dev_err(dev, "enabling clk pcie%d\n",
>                     port->slot);
>                 return err;
>             }
> 
>             mt7621_pcie_enable_port(port);
>             dev_info(dev, "PCIE%d enabled\n", port->slot);
>         }
>     }
> 
>     return 0;
> }
> 
> Or just delete the setting and the read and write config functions
> since they are not being used in any other place. My two boards work
> without this setting but I don't know about other boards.
> 
> What do you think?

I think that your above modification is fine and make code more
straightforward. Thanks for looking at it!

> Best regards,
>     Sergio Paracuellos
> 
> [0]: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/tree/drivers/staging/mt7621-pci/pci-mt7621.c?h=v4.18#n663
> [1]: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/commit/drivers/staging/mt7621-pci?h=staging-testing&id=b99cc3a2b6b62cf994acac5cced03298d9908c9b

Re: [PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Sergio Paracuellos]

Hi Pali,

On Sat, Jun 5, 2021 at 1:07 AM Pali Rohár <pali@kernel.org> wrote:
>
> On Friday 04 June 2021 20:44:42 Sergio Paracuellos wrote:
> > Hi Pali,
> >
> > Thanks for your comments.
> >
> > On Fri, Jun 4, 2021 at 6:55 PM Pali Rohár <pali@kernel.org> wrote:
> > >
> > > On Wednesday 02 June 2021 14:43:53 Sergio Paracuellos wrote:
> > > > Hi Pali,
> > > >
> > > > On Wed, Jun 2, 2021 at 2:23 PM Pali Rohár <pali@kernel.org> wrote:
> > > > >
> > > > > On Wednesday 02 June 2021 14:16:26 Sergio Paracuellos wrote:
> > > > > > Hi Pali,
> > > > > >
> > > > > > On Mon, May 31, 2021 at 4:19 PM Sergio Paracuellos
> > > > > > <sergio.paracuellos@gmail.com> wrote:
> > > > > > >
> > > > > > > On Mon, May 31, 2021 at 3:50 PM Pali Rohár <pali@kernel.org> wrote:
> > > > > > > >
> > > > > > > > On Monday 31 May 2021 15:39:55 Sergio Paracuellos wrote:
> > > > > > > > > Hi Pali,
> > > > > > > > >
> > > > > > > > > Thanks for your comments.
> > > > > > > > >
> > > > > > > > > On Mon, May 31, 2021 at 3:14 PM Pali Rohár <pali@kernel.org> wrote:
> > > > > > > > > >
> > > > > > > > > > On Saturday 15 May 2021 14:40:53 Sergio Paracuellos wrote:
> > > > > > > > > > > This patch adds a driver for the PCIe controller of MT7621 SoC.
> > > > > > > > > > >
> > > > > > > > > > > Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > > > > > > > > > > ---
> > > > > > > > > > >  arch/mips/pci/Makefile     |   1 +
> > > > > > > > > > >  arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
> > > > > > > > > > >  arch/mips/ralink/Kconfig   |   9 +-
> > > > > > > > > > >  3 files changed, 633 insertions(+), 1 deletion(-)
> > > > > > > > > > >  create mode 100644 arch/mips/pci/pci-mt7621.c
> > > > > > > > > > >
> > > > > > > > > > > diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
> > > > > > > > > > > index f3eecc065e5c..178c550739c4 100644
> > > > > > > > > > > --- a/arch/mips/pci/Makefile
> > > > > > > > > > > +++ b/arch/mips/pci/Makefile
> > > > > > > > > > > @@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)    += pci-ar2315.o
> > > > > > > > > > >  obj-$(CONFIG_SOC_AR71XX)     += pci-ar71xx.o
> > > > > > > > > > >  obj-$(CONFIG_PCI_AR724X)     += pci-ar724x.o
> > > > > > > > > > >  obj-$(CONFIG_PCI_XTALK_BRIDGE)       += pci-xtalk-bridge.o
> > > > > > > > > > > +obj-$(CONFIG_PCI_MT7621)     += pci-mt7621.o
> > > > > > > > > > >  #
> > > > > > > > > > >  # These are still pretty much in the old state, watch, go blind.
> > > > > > > > > > >  #
> > > > > > > > > > > diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
> > > > > > > > > > > new file mode 100644
> > > > > > > > > > > index 000000000000..fe1945819d25
> > > > > > > > > > > --- /dev/null
> > > > > > > > > > > +++ b/arch/mips/pci/pci-mt7621.c
> > > > > > > > > > ...
> > > > > > > > > > > +static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> > > > > > > > > > > +{
> > > > > > > > > > > +     struct device *dev = pcie->dev;
> > > > > > > > > > > +     struct mt7621_pcie_port *port;
> > > > > > > > > > > +     u8 num_slots_enabled = 0;
> > > > > > > > > > > +     u32 slot;
> > > > > > > > > > > +     u32 val;
> > > > > > > > > > > +     int err;
> > > > > > > > > > > +
> > > > > > > > > > > +     /* Setup MEMWIN and IOWIN */
> > > > > > > > > > > +     pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> > > > > > > > > > > +     pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> > > > > > > > > > > +
> > > > > > > > > > > +     list_for_each_entry(port, &pcie->ports, list) {
> > > > > > > > > > > +             if (port->enabled) {
> > > > > > > > > > > +                     err = clk_prepare_enable(port->clk);
> > > > > > > > > > > +                     if (err) {
> > > > > > > > > > > +                             dev_err(dev, "enabling clk pcie%d\n", slot);
> > > > > > > > > > > +                             return err;
> > > > > > > > > > > +                     }
> > > > > > > > > > > +
> > > > > > > > > > > +                     mt7621_pcie_enable_port(port);
> > > > > > > > > > > +                     dev_info(dev, "PCIE%d enabled\n", port->slot);
> > > > > > > > > > > +                     num_slots_enabled++;
> > > > > > > > > > > +             }
> > > > > > > > > > > +     }
> > > > > > > > > > > +
> > > > > > > > > > > +     for (slot = 0; slot < num_slots_enabled; slot++) {
> > > > > > > > > > > +             val = read_config(pcie, slot, PCI_COMMAND);
> > > > > > > > > > > +             val |= PCI_COMMAND_MASTER;
> > > > > > > > > > > +             write_config(pcie, slot, PCI_COMMAND, val);
> > > > > > > > > >
> > > > > > > > > > Hello! Is this part of code correct? Because it looks strange if PCIe
> > > > > > > > > > controller driver automatically enables PCI bus mastering, prior device
> > > > > > > > > > driver initialize itself.
> > > > > > > > > >
> > > > > > > > > > Moreover kernel has already function pci_set_master() for this purpose
> > > > > > > > > > which is used by device drivers.
> > > > > > > > > >
> > > > > > > > > > So I think this code can confuse some device drivers...
> > > > > > > > >
> > > > > > > > > I agree that we have pci_set_master() to be used in pci device driver
> > > > > > > > > code. Original controller driver set this bit for enabled slots. Since
> > > > > > > > > there is no documentation at all for the PCI in this SoC
> > > > > > > >
> > > > > > > > I see... this is really a big problem to do any driver development...
> > > > > > >
> > > > > > > For sure it is :(.
> > > > > > >
> > > > > > > >
> > > > > > > > > I have
> > > > > > > > > maintained the setting in the driver in a cleaner way. See original
> > > > > > > > > driver code and the setting here [0]. There is no other reason than
> > > > > > > > > that. I am ok with removing this from here and testing with my two
> > > > > > > > > devices that everything is still ok if having this setting in the pci
> > > > > > > > > controller driver is a real problem.
> > > > > > > >
> > > > > > > > You can run lspci -nnvv with and without PCI_COMMAND_MASTER code and
> > > > > > > > then compare outputs.
> > > > > > > >
> > > > > > > > Device drivers for sure enable PCI_COMMAND_MASTER at the time when it is
> > > > > > > > needed, so it is possible that there would be no difference in lspci
> > > > > > > > output.
> > > > > > >
> > > > > > > Thanks. I will take this into account when v2 is submitted after more
> > > > > > > review comments come :).
> > > > > >
> > > > > > I have tested to remove this and check lspci -nnvv output with and
> > > > > > without PCI_COMMAND_MASTER code and, as you pointed out, there is no
> > > > > > difference between them. Also, both boards are working without
> > > > > > regressions at all. So I will remove this code for next version.
> > > > >
> > > > > Perfect!
> > > > >
> > > > > > Thanks,
> > > > > >     Sergio Paracuellos
> > > > > > >
> > > > > > > >
> > > > > > > > > [0]: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/tree/drivers/staging/mt7621-pci/pci-mt7621.c?h=v4.18#n676
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > >     Sergio Paracuellos
> > > > > > > > > >
> > > > > > > > > > > +             /* configure RC FTS number to 250 when it leaves L0s */
> > > > > > > > > > > +             val = read_config(pcie, slot, PCIE_FTS_NUM);
> > > > > > > > > > > +             val &= ~PCIE_FTS_NUM_MASK;
> > > > > > > > > > > +             val |= PCIE_FTS_NUM_L0(0x50);
> > > > > > > > > > > +             write_config(pcie, slot, PCIE_FTS_NUM, val);
> > > > >
> > > > > Could you look also what is doing this code (PCIE_FTS_NUM)? It is marked
> > > > > as MT specific register. But from this code for me it looks like that it
> > > > > just access config space of some device and therefore it could be some
> > > > > standard PCIe register. Just with hardcoded calculated offset.
> > >
> > > So based on your lspci output, there is no PCIe capability register at
> > > address PCIE_FTS_NUM (0x70c), right? It seems strange to trying access
> > > capability register outside of capability list.
> >
> > This setting is also in the original driver code from mediatek SDK. See [0].
> > Anyway, I have tried to remove this code and test what happens with my
> > two boards and both of them seem to
> > work properly just by deleting this code.
>
> Looks like that this code is some designware stuff which seems to be
> needed. So it would be better to let it here as is.

Ok, will do.

>
> > > > > Could you provide output from lspci -nnvv? So other people could look at
> > > > > it and maybe we decode what is this code doing and if it is needed.
> > > >
> > > > # lspci -nnvv
> > > > 00:02.0 PCI bridge [0604]: Device [0e8d:0801] (rev 01) (prog-if 00
> > > > [Normal decode])
> > >
> > > Hm... Device address is 02. But in your code is:
> > >
> > >     u8 num_slots_enabled = 0;
> > >     ...
> > >     list_for_each_entry(port, &pcie->ports, list) {
> > >         if (port->enabled) {
> > >             ...
> > >             num_slots_enabled++;
> > >             ...
> > >         }
> > >     }
> > >     ...
> > >     for (slot = 0; slot < num_slots_enabled; slot++) {
> > >         val = read_config(pcie, slot, ...);
> > >         ...
> > >         write_config(pcie, slot, ...);
> > >     }
> > >
> > > Which means that this code writes to config space of wrong device 0
> > > (instead of 2)! In function write_config() can be seen that second
> > > parameter specify device of BDF address for bus=0 and function=0.
> >
> > Bridge enumeration depends on a virtual bridge register configuration.
> > But at the end devices connected to the bridge
> > are enumerated as 01:00.0, 02:00.0 and 03:00.0. So in this case the
> > phy used is the one for "pcie2" (00:02.0) and the device connected to
> > it
> > is 01:00.0. For example a board using all the virtual bridges will get
> > an output similar to:
> >
> > [   16.487166] mt7621-pci 1e140000.pcie: host bridge /pcie@1e140000 ranges:
> > [   16.500627] mt7621-pci 1e140000.pcie:   No bus range found for
> > /pcie@1e140000, using [bus 00-ff]
> > [   16.518212] mt7621-pci 1e140000.pcie:      MEM
> > 0x0060000000..0x006fffffff -> 0x0000000000
> > [   16.534531] mt7621-pci 1e140000.pcie:       IO
> > 0x001e160000..0x001e16ffff -> 0x0000000000
> > [   16.786498] mt7621-pci 1e140000.pcie: PCIE0 enabled
> > [   16.796220] mt7621-pci 1e140000.pcie: PCIE1 enabled
> > [   16.805943] mt7621-pci 1e140000.pcie: PCIE2 enabled
> > [   16.815664] mt7621-pci 1e140000.pcie: PCI coherence region base:
> > 0x60000000, mask/settings: 0xf0000002
> > [   16.834398] mt7621-pci 1e140000.pcie: PCI host bridge to bus 0000:00
> > [   16.847098] pci_bus 0000:00: root bus resource [io  0x1e160000-0x1e16ffff]
> > [   16.860806] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fffffff]
> > [   16.874504] pci_bus 0000:00: root bus resource [bus 00-ff]
> > [   16.885441] pci_bus 0000:00: root bus resource [mem
> > 0x60000000-0x6fffffff] (bus address [0x00000000-0x0fffffff])
> > [   16.905773] pci 0000:00:00.0: [0e8d:0801] type 01 class 0x060400
> > [   16.917772] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x7fffffff]
> > [   16.930260] pci 0000:00:00.0: reg 0x14: initial BAR value 0x00000000 invalid
> > [   16.944304] pci 0000:00:00.0: reg 0x14: [mem size 0x00010000]
> > [   16.955841] pci 0000:00:00.0: supports D1
> > [   16.963837] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
> > [   16.975747] pci 0000:00:01.0: [0e8d:0801] type 01 class 0x060400
> > [   16.987772] pci 0000:00:01.0: reg 0x10: [mem 0x00000000-0x7fffffff]
> > [   17.000273] pci 0000:00:01.0: reg 0x14: initial BAR value 0x00000000 invalid
> > [   17.014311] pci 0000:00:01.0: reg 0x14: [mem size 0x00010000]
> > [   17.025838] pci 0000:00:01.0: supports D1
> > [   17.033828] pci 0000:00:01.0: PME# supported from D0 D1 D3hot
> > [   17.045699] pci 0000:00:02.0: [0e8d:0801] type 01 class 0x060400
> > [   17.057726] pci 0000:00:02.0: reg 0x10: [mem 0x00000000-0x7fffffff]
> > [   17.070218] pci 0000:00:02.0: reg 0x14: initial BAR value 0x00000000 invalid
> > [   17.084260] pci 0000:00:02.0: reg 0x14: [mem size 0x00010000]
> > [   17.095788] pci 0000:00:02.0: supports D1
> > [   17.103785] pci 0000:00:02.0: PME# supported from D0 D1 D3hot
> > [   17.116598] pci 0000:00:00.0: bridge configuration invalid ([bus
> > 00-00]), reconfiguring
> > [   17.132566] pci 0000:00:01.0: bridge configuration invalid ([bus
> > 00-00]), reconfiguring
> > [   17.148514] pci 0000:00:02.0: bridge configuration invalid ([bus
> > 00-00]), reconfiguring
> > [   17.164739] pci 0000:01:00.0: [1b21:0611] type 00 class 0x010185
> > [   17.176775] pci 0000:01:00.0: reg 0x10: [io  0x0000-0x0007]
> > [   17.187892] pci 0000:01:00.0: reg 0x14: [io  0x0000-0x0003]
> > [   17.199009] pci 0000:01:00.0: reg 0x18: [io  0x0000-0x0007]
> > [   17.210130] pci 0000:01:00.0: reg 0x1c: [io  0x0000-0x0003]
> > [   17.221246] pci 0000:01:00.0: reg 0x20: [io  0x0000-0x000f]
> > [   17.232357] pci 0000:01:00.0: reg 0x24: initial BAR value 0x00000000 invalid
> > [   17.246411] pci 0000:01:00.0: reg 0x24: [mem size 0x00000200]
> > [   17.258031] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth,
> > limited by 2.5 GT/s PCIe x1 link at 0000:00:00.0 (capable of 4.000
> > Gb/s with 5.0 GT/s PCIe x1 link)
> > [   17.317563] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
> > [   17.328033] pci 0000:00:00.0:   bridge window [io  0x0000-0x0fff]
> > [   17.340179] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
> > [   17.353703] pci 0000:00:00.0:   bridge window [mem
> > 0x60000000-0x600fffff pref]
> > [   17.368097] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
> > [   17.381550] pci 0000:02:00.0: [1b21:0611] type 00 class 0x010185
> > [   17.393573] pci 0000:02:00.0: reg 0x10: [io  0x0000-0x0007]
> > [   17.404694] pci 0000:02:00.0: reg 0x14: [io  0x0000-0x0003]
> > [   17.415804] pci 0000:02:00.0: reg 0x18: [io  0x0000-0x0007]
> > [   17.426916] pci 0000:02:00.0: reg 0x1c: [io  0x0000-0x0003]
> > [   17.438040] pci 0000:02:00.0: reg 0x20: [io  0x0000-0x000f]
> > [   17.449148] pci 0000:02:00.0: reg 0x24: initial BAR value 0x00000000 invalid
> > [   17.463193] pci 0000:02:00.0: reg 0x24: [mem size 0x00000200]
> > [   17.474803] pci 0000:02:00.0: 2.000 Gb/s available PCIe bandwidth,
> > limited by 2.5 GT/s PCIe x1 link at 0000:00:01.0 (capable of 4.000
> > Gb/s with 5.0 GT/s PCIe x1 link)
> > [   17.527554] pci 0000:00:01.0: PCI bridge to [bus 02-ff]
> > [   17.538014] pci 0000:00:01.0:   bridge window [io  0x0000-0x0fff]
> > [   17.550159] pci 0000:00:01.0:   bridge window [mem 0x60000000-0x600fffff]
> > [   17.563682] pci 0000:00:01.0:   bridge window [mem
> > 0x60000000-0x600fffff pref]
> > [   17.578078] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
> > [   17.591529] pci 0000:03:00.0: [1b21:0611] type 00 class 0x010185
> > [   17.603545] pci 0000:03:00.0: reg 0x10: [io  0x0000-0x0007]
> > [   17.614665] pci 0000:03:00.0: reg 0x14: [io  0x0000-0x0003]
> > [   17.625775] pci 0000:03:00.0: reg 0x18: [io  0x0000-0x0007]
> > [   17.636887] pci 0000:03:00.0: reg 0x1c: [io  0x0000-0x0003]
> > [   17.648009] pci 0000:03:00.0: reg 0x20: [io  0x0000-0x000f]
> > [   17.659119] pci 0000:03:00.0: reg 0x24: initial BAR value 0x00000000 invalid
> > [   17.673162] pci 0000:03:00.0: reg 0x24: [mem size 0x00000200]
> > [   17.684777] pci 0000:03:00.0: 2.000 Gb/s available PCIe bandwidth,
> > limited by 2.5 GT/s PCIe x1 link at 0000:00:02.0 (capable of 4.000
> > Gb/s with 5.0 GT/s PCIe x1 link)
> > [   17.737561] pci 0000:00:02.0: PCI bridge to [bus 03-ff]
> > [   17.748022] pci 0000:00:02.0:   bridge window [io  0x0000-0x0fff]
> > [   17.760167] pci 0000:00:02.0:   bridge window [mem 0x60000000-0x600fffff]
> > [   17.773690] pci 0000:00:02.0:   bridge window [mem
> > 0x60000000-0x600fffff pref]
> > [   17.788085] pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03
> > [   17.801341] pci 0000:00:00.0: BAR 0: no space for [mem size 0x80000000]
> > [   17.814518] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x80000000]
> > [   17.828392] pci 0000:00:01.0: BAR 0: no space for [mem size 0x80000000]
> > [   17.841571] pci 0000:00:01.0: BAR 0: failed to assign [mem size 0x80000000]
> > [   17.855443] pci 0000:00:02.0: BAR 0: no space for [mem size 0x80000000]
> > [   17.868626] pci 0000:00:02.0: BAR 0: failed to assign [mem size 0x80000000]
> > [   17.882502] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
> > [   17.896028] pci 0000:00:00.0: BAR 9: assigned [mem
> > 0x60100000-0x601fffff pref]
> > [   17.910419] pci 0000:00:01.0: BAR 8: assigned [mem 0x60200000-0x602fffff]
> > [   17.923951] pci 0000:00:01.0: BAR 9: assigned [mem
> > 0x60300000-0x603fffff pref]
> > [   17.938353] pci 0000:00:02.0: BAR 8: assigned [mem 0x60400000-0x604fffff]
> > [   17.951879] pci 0000:00:02.0: BAR 9: assigned [mem
> > 0x60500000-0x605fffff pref]
> > [   17.966273] pci 0000:00:00.0: BAR 1: assigned [mem 0x60600000-0x6060ffff]
> > [   17.979809] pci 0000:00:01.0: BAR 1: assigned [mem 0x60610000-0x6061ffff]
> > [   17.993341] pci 0000:00:02.0: BAR 1: assigned [mem 0x60620000-0x6062ffff]
> > [   18.006877] pci 0000:00:00.0: BAR 7: assigned [io  0x1e160000-0x1e160fff]
> > [   18.020408] pci 0000:00:01.0: BAR 7: assigned [io  0x1e161000-0x1e161fff]
> > [   18.033932] pci 0000:00:02.0: BAR 7: assigned [io  0x1e162000-0x1e162fff]
> > [   18.047472] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
> > [   18.061005] pci 0000:01:00.0: BAR 4: assigned [io  0x1e160000-0x1e16000f]
> > [   18.074540] pci 0000:01:00.0: BAR 0: assigned [io  0x1e160010-0x1e160017]
> > [   18.088070] pci 0000:01:00.0: BAR 2: assigned [io  0x1e160018-0x1e16001f]
> > [   18.101606] pci 0000:01:00.0: BAR 1: assigned [io  0x1e160020-0x1e160023]
> > [   18.115139] pci 0000:01:00.0: BAR 3: assigned [io  0x1e160024-0x1e160027]
> > [   18.128676] pci 0000:00:00.0: PCI bridge to [bus 01]
> > [   18.138577] pci 0000:00:00.0:   bridge window [io  0x1e160000-0x1e160fff]
> > [   18.152100] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
> > [   18.165627] pci 0000:00:00.0:   bridge window [mem
> > 0x60100000-0x601fffff pref]
> > [   18.180030] pci 0000:02:00.0: BAR 5: assigned [mem 0x60200000-0x602001ff]
> > [   18.193566] pci 0000:02:00.0: BAR 4: assigned [io  0x1e161000-0x1e16100f]
> > [   18.207110] pci 0000:02:00.0: BAR 0: assigned [io  0x1e161010-0x1e161017]
> > [   18.220648] pci 0000:02:00.0: BAR 2: assigned [io  0x1e161018-0x1e16101f]
> > [   18.234183] pci 0000:02:00.0: BAR 1: assigned [io  0x1e161020-0x1e161023]
> > [   18.247723] pci 0000:02:00.0: BAR 3: assigned [io  0x1e161024-0x1e161027]
> > [   18.261257] pci 0000:00:01.0: PCI bridge to [bus 02]
> > [   18.271165] pci 0000:00:01.0:   bridge window [io  0x1e161000-0x1e161fff]
> > [   18.284695] pci 0000:00:01.0:   bridge window [mem 0x60200000-0x602fffff]
> > [   18.298225] pci 0000:00:01.0:   bridge window [mem
> > 0x60300000-0x603fffff pref]
> > [   18.312630] pci 0000:03:00.0: BAR 5: assigned [mem 0x60400000-0x604001ff]
> > [   18.326166] pci 0000:03:00.0: BAR 4: assigned [io  0x1e162000-0x1e16200f]
> > [   18.339702] pci 0000:03:00.0: BAR 0: assigned [io  0x1e162010-0x1e162017]
> > [   18.353237] pci 0000:03:00.0: BAR 2: assigned [io  0x1e162018-0x1e16201f]
> > [   18.366775] pci 0000:03:00.0: BAR 1: assigned [io  0x1e162020-0x1e162023]
> > [   18.380311] pci 0000:03:00.0: BAR 3: assigned [io  0x1e162024-0x1e162027]
> > [   18.393841] pci 0000:00:02.0: PCI bridge to [bus 03]
> > [   18.403740] pci 0000:00:02.0:   bridge window [io  0x1e162000-0x1e162fff]
> > [   18.417270] pci 0000:00:02.0:   bridge window [mem 0x60400000-0x604fffff]
> > [   18.430801] pci 0000:00:02.0:   bridge window [mem
> > 0x60500000-0x605fffff pref]
> > [   18.445529] ahci 0000:01:00.0: version 3.0
> > [   18.445559] pci 0000:00:00.0: enabling device (0000 -> 0003)
> > [   18.456853] ahci 0000:01:00.0: enabling device (0000 -> 0003)
> > [   18.468455] ahci 0000:01:00.0: SSS flag set, parallel bus scan disabled
> > [   18.481700] ahci 0000:01:00.0: AHCI 0001.0200 32 slots 2 ports 6
> > Gbps 0x3 impl IDE mode
> > [   18.497662] ahci 0000:01:00.0: flags: 64bit ncq sntf stag led clo
> > pmp pio slum part ccc sxs
> > [   18.516777] scsi host0: ahci
> > [   18.523629] scsi host1: ahci
> > [   18.529829] ata1: SATA max UDMA/133 abar m512@0x60000000 port
> > 0x60000100 irq 22
> > [   18.544447] ata2: SATA max UDMA/133 abar m512@0x60000000 port
> > 0x60000180 irq 22
> > [   18.559465] pci 0000:00:01.0: enabling device (0000 -> 0003)
> > [   18.570786] ahci 0000:02:00.0: enabling device (0000 -> 0003)
> > [   18.582414] ahci 0000:02:00.0: SSS flag set, parallel bus scan disabled
> > [   18.595665] ahci 0000:02:00.0: AHCI 0001.0200 32 slots 2 ports 6
> > Gbps 0x3 impl IDE mode
> > [   18.611614] ahci 0000:02:00.0: flags: 64bit ncq sntf stag led clo
> > pmp pio slum part ccc sxs
> > [   18.631053] scsi host2: ahci
> > [   18.637983] scsi host3: ahci
> > [   18.644138] ata3: SATA max UDMA/133 abar m512@0x60200000 port
> > 0x60200100 irq 23
> > [   18.658792] ata4: SATA max UDMA/133 abar m512@0x60200000 port
> > 0x60200180 irq 23
> > [   18.673827] pci 0000:00:02.0: enabling device (0000 -> 0003)
> > [   18.685151] ahci 0000:03:00.0: enabling device (0000 -> 0003)
> > [   18.696782] ahci 0000:03:00.0: SSS flag set, parallel bus scan disabled
> > [   18.710025] ahci 0000:03:00.0: AHCI 0001.0200 32 slots 2 ports 6
> > Gbps 0x3 impl IDE mode
> > [   18.725972] ahci 0000:03:00.0: flags: 64bit ncq sntf stag led clo
> > pmp pio slum part ccc sxs
> >
> > And you are totally right, the setting is writing in the wrong place.
> > I changed the device tree and the way interrupts are mapped
> > to avoid using a custom 'map_irq' function [1]. Before that commit the
> > pci virtual bridge register was reordering the
> > buses enumeration depending on link status, so there I should also
> > properly rewrite the code in question.
> >
> > I can rewrite the code to read and write config properly using the
> > slot moving the code into 'mt7621_pcie_enable_port' as follows:
> >
> > static void mt7621_pcie_enable_port(struct mt7621_pcie_port *port)
> > {
> >     struct mt7621_pcie *pcie = port->pcie;
> >     u32 slot = port->slot;
> >     u32 offset = MT7621_PCIE_OFFSET + (slot * MT7621_NEXT_PORT);
> >     u32 val;
> >
> >     /* enable pcie interrupt */
> >     val = pcie_read(pcie, RALINK_PCI_PCIMSK_ADDR);
> >     val |= PCIE_PORT_INT_EN(slot);
> >
> >     /* map 2G DDR region */
> >     pcie_write(pcie, PCIE_BAR_MAP_MAX | PCIE_BAR_ENABLE,
> >            offset + RALINK_PCI_BAR0SETUP_ADDR);
> >
> >     /* configure class code and revision ID */
> >     pcie_write(pcie, PCIE_CLASS_CODE | PCIE_REVISION_ID,
> >            offset + RALINK_PCI_CLASS);
> >
> >     /* configure RC FTS number to 250 when it leaves L0s */
> >     val = read_config(pcie, slot, PCIE_FTS_NUM);
> >     val &= ~PCIE_FTS_NUM_MASK;
> >     val |= PCIE_FTS_NUM_L0(0x50);
> >     write_config(pcie, slot, PCIE_FTS_NUM, val);
> > }
> >
> > static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> > {
> >     struct device *dev = pcie->dev;
> >     struct mt7621_pcie_port *port;
> >     int err;
> >
> >     /* Setup MEMWIN and IOWIN */
> >     pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> >     pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> >
> >     list_for_each_entry(port, &pcie->ports, list) {
> >         if (port->enabled) {
> >             err = clk_prepare_enable(port->clk);
> >             if (err) {
> >                 dev_err(dev, "enabling clk pcie%d\n",
> >                     port->slot);
> >                 return err;
> >             }
> >
> >             mt7621_pcie_enable_port(port);
> >             dev_info(dev, "PCIE%d enabled\n", port->slot);
> >         }
> >     }
> >
> >     return 0;
> > }
> >
> > Or just delete the setting and the read and write config functions
> > since they are not being used in any other place. My two boards work
> > without this setting but I don't know about other boards.
> >
> > What do you think?
>
> I think that your above modification is fine and make code more
> straightforward. Thanks for looking at it!

Ok I will change this in this way, then.

Thanks for your comments and help.

Best regards,
    Sergio Paracuellos

>
> > Best regards,
> >     Sergio Paracuellos
> >
> > [0]: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/tree/drivers/staging/mt7621-pci/pci-mt7621.c?h=v4.18#n663
> > [1]: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/commit/drivers/staging/mt7621-pci?h=staging-testing&id=b99cc3a2b6b62cf994acac5cced03298d9908c9b

Re: [PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Rob Herring]

On Fri, Jun 04, 2021 at 06:55:25PM +0200, Pali Rohár wrote:
> On Wednesday 02 June 2021 14:43:53 Sergio Paracuellos wrote:
> > Hi Pali,
> > 
> > On Wed, Jun 2, 2021 at 2:23 PM Pali Rohár <pali@kernel.org> wrote:
> > >
> > > On Wednesday 02 June 2021 14:16:26 Sergio Paracuellos wrote:
> > > > Hi Pali,
> > > >
> > > > On Mon, May 31, 2021 at 4:19 PM Sergio Paracuellos
> > > > <sergio.paracuellos@gmail.com> wrote:
> > > > >
> > > > > On Mon, May 31, 2021 at 3:50 PM Pali Rohár <pali@kernel.org> wrote:
> > > > > >
> > > > > > On Monday 31 May 2021 15:39:55 Sergio Paracuellos wrote:
> > > > > > > Hi Pali,
> > > > > > >
> > > > > > > Thanks for your comments.
> > > > > > >
> > > > > > > On Mon, May 31, 2021 at 3:14 PM Pali Rohár <pali@kernel.org> wrote:
> > > > > > > >
> > > > > > > > On Saturday 15 May 2021 14:40:53 Sergio Paracuellos wrote:
> > > > > > > > > This patch adds a driver for the PCIe controller of MT7621 SoC.
> > > > > > > > >
> > > > > > > > > Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > > > > > > > > ---
> > > > > > > > >  arch/mips/pci/Makefile     |   1 +
> > > > > > > > >  arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
> > > > > > > > >  arch/mips/ralink/Kconfig   |   9 +-
> > > > > > > > >  3 files changed, 633 insertions(+), 1 deletion(-)
> > > > > > > > >  create mode 100644 arch/mips/pci/pci-mt7621.c
> > > > > > > > >
> > > > > > > > > diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
> > > > > > > > > index f3eecc065e5c..178c550739c4 100644
> > > > > > > > > --- a/arch/mips/pci/Makefile
> > > > > > > > > +++ b/arch/mips/pci/Makefile
> > > > > > > > > @@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)    += pci-ar2315.o
> > > > > > > > >  obj-$(CONFIG_SOC_AR71XX)     += pci-ar71xx.o
> > > > > > > > >  obj-$(CONFIG_PCI_AR724X)     += pci-ar724x.o
> > > > > > > > >  obj-$(CONFIG_PCI_XTALK_BRIDGE)       += pci-xtalk-bridge.o
> > > > > > > > > +obj-$(CONFIG_PCI_MT7621)     += pci-mt7621.o
> > > > > > > > >  #
> > > > > > > > >  # These are still pretty much in the old state, watch, go blind.
> > > > > > > > >  #
> > > > > > > > > diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
> > > > > > > > > new file mode 100644
> > > > > > > > > index 000000000000..fe1945819d25
> > > > > > > > > --- /dev/null
> > > > > > > > > +++ b/arch/mips/pci/pci-mt7621.c
> > > > > > > > ...
> > > > > > > > > +static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> > > > > > > > > +{
> > > > > > > > > +     struct device *dev = pcie->dev;
> > > > > > > > > +     struct mt7621_pcie_port *port;
> > > > > > > > > +     u8 num_slots_enabled = 0;
> > > > > > > > > +     u32 slot;
> > > > > > > > > +     u32 val;
> > > > > > > > > +     int err;
> > > > > > > > > +
> > > > > > > > > +     /* Setup MEMWIN and IOWIN */
> > > > > > > > > +     pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> > > > > > > > > +     pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> > > > > > > > > +
> > > > > > > > > +     list_for_each_entry(port, &pcie->ports, list) {
> > > > > > > > > +             if (port->enabled) {
> > > > > > > > > +                     err = clk_prepare_enable(port->clk);
> > > > > > > > > +                     if (err) {
> > > > > > > > > +                             dev_err(dev, "enabling clk pcie%d\n", slot);
> > > > > > > > > +                             return err;
> > > > > > > > > +                     }
> > > > > > > > > +
> > > > > > > > > +                     mt7621_pcie_enable_port(port);
> > > > > > > > > +                     dev_info(dev, "PCIE%d enabled\n", port->slot);
> > > > > > > > > +                     num_slots_enabled++;
> > > > > > > > > +             }
> > > > > > > > > +     }
> > > > > > > > > +
> > > > > > > > > +     for (slot = 0; slot < num_slots_enabled; slot++) {
> > > > > > > > > +             val = read_config(pcie, slot, PCI_COMMAND);
> > > > > > > > > +             val |= PCI_COMMAND_MASTER;
> > > > > > > > > +             write_config(pcie, slot, PCI_COMMAND, val);
> > > > > > > >
> > > > > > > > Hello! Is this part of code correct? Because it looks strange if PCIe
> > > > > > > > controller driver automatically enables PCI bus mastering, prior device
> > > > > > > > driver initialize itself.
> > > > > > > >
> > > > > > > > Moreover kernel has already function pci_set_master() for this purpose
> > > > > > > > which is used by device drivers.
> > > > > > > >
> > > > > > > > So I think this code can confuse some device drivers...
> > > > > > >
> > > > > > > I agree that we have pci_set_master() to be used in pci device driver
> > > > > > > code. Original controller driver set this bit for enabled slots. Since
> > > > > > > there is no documentation at all for the PCI in this SoC
> > > > > >
> > > > > > I see... this is really a big problem to do any driver development...
> > > > >
> > > > > For sure it is :(.
> > > > >
> > > > > >
> > > > > > > I have
> > > > > > > maintained the setting in the driver in a cleaner way. See original
> > > > > > > driver code and the setting here [0]. There is no other reason than
> > > > > > > that. I am ok with removing this from here and testing with my two
> > > > > > > devices that everything is still ok if having this setting in the pci
> > > > > > > controller driver is a real problem.
> > > > > >
> > > > > > You can run lspci -nnvv with and without PCI_COMMAND_MASTER code and
> > > > > > then compare outputs.
> > > > > >
> > > > > > Device drivers for sure enable PCI_COMMAND_MASTER at the time when it is
> > > > > > needed, so it is possible that there would be no difference in lspci
> > > > > > output.
> > > > >
> > > > > Thanks. I will take this into account when v2 is submitted after more
> > > > > review comments come :).
> > > >
> > > > I have tested to remove this and check lspci -nnvv output with and
> > > > without PCI_COMMAND_MASTER code and, as you pointed out, there is no
> > > > difference between them. Also, both boards are working without
> > > > regressions at all. So I will remove this code for next version.
> > >
> > > Perfect!
> > >
> > > > Thanks,
> > > >     Sergio Paracuellos
> > > > >
> > > > > >
> > > > > > > [0]: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/tree/drivers/staging/mt7621-pci/pci-mt7621.c?h=v4.18#n676
> > > > > > >
> > > > > > > Best regards,
> > > > > > >     Sergio Paracuellos
> > > > > > > >
> > > > > > > > > +             /* configure RC FTS number to 250 when it leaves L0s */
> > > > > > > > > +             val = read_config(pcie, slot, PCIE_FTS_NUM);
> > > > > > > > > +             val &= ~PCIE_FTS_NUM_MASK;
> > > > > > > > > +             val |= PCIE_FTS_NUM_L0(0x50);
> > > > > > > > > +             write_config(pcie, slot, PCIE_FTS_NUM, val);
> > >
> > > Could you look also what is doing this code (PCIE_FTS_NUM)? It is marked
> > > as MT specific register. But from this code for me it looks like that it
> > > just access config space of some device and therefore it could be some
> > > standard PCIe register. Just with hardcoded calculated offset.
> 
> So based on your lspci output, there is no PCIe capability register at
> address PCIE_FTS_NUM (0x70c), right? It seems strange to trying access
> capability register outside of capability list.

Looks like a DW PCIe port logic register:

drivers/pci/controller/dwc/pcie-designware.h-/* Synopsys-specific PCIe configuration registers */
drivers/pci/controller/dwc/pcie-designware.h:#define PCIE_PORT_AFR                      0x70C
drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_N_FTS_MASK                GENMASK(15, 8)
drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_N_FTS(n)          FIELD_PREP(PORT_AFR_N_FTS_MASK, n)
drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_CC_N_FTS_MASK             GENMASK(23, 16)
drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_CC_N_FTS(n)               FIELD_PREP(PORT_AFR_CC_N_FTS_MASK, n)
drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_ENTER_ASPM                BIT(30)
drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_L0S_ENTRANCE_LAT_SHIFT    24
drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_L0S_ENTRANCE_LAT_MASK     GENMASK(26, 24)
drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_L1_ENTRANCE_LAT_SHIFT     27
drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_L1_ENTRANCE_LAT_MASK      GENMASK(29, 27)

Rob

Re: [PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Pali Rohár]

On Friday 04 June 2021 13:49:39 Rob Herring wrote:
> On Fri, Jun 04, 2021 at 06:55:25PM +0200, Pali Rohár wrote:
> > On Wednesday 02 June 2021 14:43:53 Sergio Paracuellos wrote:
> > > Hi Pali,
> > > 
> > > On Wed, Jun 2, 2021 at 2:23 PM Pali Rohár <pali@kernel.org> wrote:
> > > >
> > > > On Wednesday 02 June 2021 14:16:26 Sergio Paracuellos wrote:
> > > > > Hi Pali,
> > > > >
> > > > > On Mon, May 31, 2021 at 4:19 PM Sergio Paracuellos
> > > > > <sergio.paracuellos@gmail.com> wrote:
> > > > > >
> > > > > > On Mon, May 31, 2021 at 3:50 PM Pali Rohár <pali@kernel.org> wrote:
> > > > > > >
> > > > > > > On Monday 31 May 2021 15:39:55 Sergio Paracuellos wrote:
> > > > > > > > Hi Pali,
> > > > > > > >
> > > > > > > > Thanks for your comments.
> > > > > > > >
> > > > > > > > On Mon, May 31, 2021 at 3:14 PM Pali Rohár <pali@kernel.org> wrote:
> > > > > > > > >
> > > > > > > > > On Saturday 15 May 2021 14:40:53 Sergio Paracuellos wrote:
> > > > > > > > > > This patch adds a driver for the PCIe controller of MT7621 SoC.
> > > > > > > > > >
> > > > > > > > > > Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > > > > > > > > > ---
> > > > > > > > > >  arch/mips/pci/Makefile     |   1 +
> > > > > > > > > >  arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
> > > > > > > > > >  arch/mips/ralink/Kconfig   |   9 +-
> > > > > > > > > >  3 files changed, 633 insertions(+), 1 deletion(-)
> > > > > > > > > >  create mode 100644 arch/mips/pci/pci-mt7621.c
> > > > > > > > > >
> > > > > > > > > > diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
> > > > > > > > > > index f3eecc065e5c..178c550739c4 100644
> > > > > > > > > > --- a/arch/mips/pci/Makefile
> > > > > > > > > > +++ b/arch/mips/pci/Makefile
> > > > > > > > > > @@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)    += pci-ar2315.o
> > > > > > > > > >  obj-$(CONFIG_SOC_AR71XX)     += pci-ar71xx.o
> > > > > > > > > >  obj-$(CONFIG_PCI_AR724X)     += pci-ar724x.o
> > > > > > > > > >  obj-$(CONFIG_PCI_XTALK_BRIDGE)       += pci-xtalk-bridge.o
> > > > > > > > > > +obj-$(CONFIG_PCI_MT7621)     += pci-mt7621.o
> > > > > > > > > >  #
> > > > > > > > > >  # These are still pretty much in the old state, watch, go blind.
> > > > > > > > > >  #
> > > > > > > > > > diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
> > > > > > > > > > new file mode 100644
> > > > > > > > > > index 000000000000..fe1945819d25
> > > > > > > > > > --- /dev/null
> > > > > > > > > > +++ b/arch/mips/pci/pci-mt7621.c
> > > > > > > > > ...
> > > > > > > > > > +static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> > > > > > > > > > +{
> > > > > > > > > > +     struct device *dev = pcie->dev;
> > > > > > > > > > +     struct mt7621_pcie_port *port;
> > > > > > > > > > +     u8 num_slots_enabled = 0;
> > > > > > > > > > +     u32 slot;
> > > > > > > > > > +     u32 val;
> > > > > > > > > > +     int err;
> > > > > > > > > > +
> > > > > > > > > > +     /* Setup MEMWIN and IOWIN */
> > > > > > > > > > +     pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> > > > > > > > > > +     pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> > > > > > > > > > +
> > > > > > > > > > +     list_for_each_entry(port, &pcie->ports, list) {
> > > > > > > > > > +             if (port->enabled) {
> > > > > > > > > > +                     err = clk_prepare_enable(port->clk);
> > > > > > > > > > +                     if (err) {
> > > > > > > > > > +                             dev_err(dev, "enabling clk pcie%d\n", slot);
> > > > > > > > > > +                             return err;
> > > > > > > > > > +                     }
> > > > > > > > > > +
> > > > > > > > > > +                     mt7621_pcie_enable_port(port);
> > > > > > > > > > +                     dev_info(dev, "PCIE%d enabled\n", port->slot);
> > > > > > > > > > +                     num_slots_enabled++;
> > > > > > > > > > +             }
> > > > > > > > > > +     }
> > > > > > > > > > +
> > > > > > > > > > +     for (slot = 0; slot < num_slots_enabled; slot++) {
> > > > > > > > > > +             val = read_config(pcie, slot, PCI_COMMAND);
> > > > > > > > > > +             val |= PCI_COMMAND_MASTER;
> > > > > > > > > > +             write_config(pcie, slot, PCI_COMMAND, val);
> > > > > > > > >
> > > > > > > > > Hello! Is this part of code correct? Because it looks strange if PCIe
> > > > > > > > > controller driver automatically enables PCI bus mastering, prior device
> > > > > > > > > driver initialize itself.
> > > > > > > > >
> > > > > > > > > Moreover kernel has already function pci_set_master() for this purpose
> > > > > > > > > which is used by device drivers.
> > > > > > > > >
> > > > > > > > > So I think this code can confuse some device drivers...
> > > > > > > >
> > > > > > > > I agree that we have pci_set_master() to be used in pci device driver
> > > > > > > > code. Original controller driver set this bit for enabled slots. Since
> > > > > > > > there is no documentation at all for the PCI in this SoC
> > > > > > >
> > > > > > > I see... this is really a big problem to do any driver development...
> > > > > >
> > > > > > For sure it is :(.
> > > > > >
> > > > > > >
> > > > > > > > I have
> > > > > > > > maintained the setting in the driver in a cleaner way. See original
> > > > > > > > driver code and the setting here [0]. There is no other reason than
> > > > > > > > that. I am ok with removing this from here and testing with my two
> > > > > > > > devices that everything is still ok if having this setting in the pci
> > > > > > > > controller driver is a real problem.
> > > > > > >
> > > > > > > You can run lspci -nnvv with and without PCI_COMMAND_MASTER code and
> > > > > > > then compare outputs.
> > > > > > >
> > > > > > > Device drivers for sure enable PCI_COMMAND_MASTER at the time when it is
> > > > > > > needed, so it is possible that there would be no difference in lspci
> > > > > > > output.
> > > > > >
> > > > > > Thanks. I will take this into account when v2 is submitted after more
> > > > > > review comments come :).
> > > > >
> > > > > I have tested to remove this and check lspci -nnvv output with and
> > > > > without PCI_COMMAND_MASTER code and, as you pointed out, there is no
> > > > > difference between them. Also, both boards are working without
> > > > > regressions at all. So I will remove this code for next version.
> > > >
> > > > Perfect!
> > > >
> > > > > Thanks,
> > > > >     Sergio Paracuellos
> > > > > >
> > > > > > >
> > > > > > > > [0]: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/tree/drivers/staging/mt7621-pci/pci-mt7621.c?h=v4.18#n676
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > >     Sergio Paracuellos
> > > > > > > > >
> > > > > > > > > > +             /* configure RC FTS number to 250 when it leaves L0s */
> > > > > > > > > > +             val = read_config(pcie, slot, PCIE_FTS_NUM);
> > > > > > > > > > +             val &= ~PCIE_FTS_NUM_MASK;
> > > > > > > > > > +             val |= PCIE_FTS_NUM_L0(0x50);
> > > > > > > > > > +             write_config(pcie, slot, PCIE_FTS_NUM, val);
> > > >
> > > > Could you look also what is doing this code (PCIE_FTS_NUM)? It is marked
> > > > as MT specific register. But from this code for me it looks like that it
> > > > just access config space of some device and therefore it could be some
> > > > standard PCIe register. Just with hardcoded calculated offset.
> > 
> > So based on your lspci output, there is no PCIe capability register at
> > address PCIE_FTS_NUM (0x70c), right? It seems strange to trying access
> > capability register outside of capability list.
> 
> Looks like a DW PCIe port logic register:
> 
> drivers/pci/controller/dwc/pcie-designware.h-/* Synopsys-specific PCIe configuration registers */
> drivers/pci/controller/dwc/pcie-designware.h:#define PCIE_PORT_AFR                      0x70C
> drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_N_FTS_MASK                GENMASK(15, 8)
> drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_N_FTS(n)          FIELD_PREP(PORT_AFR_N_FTS_MASK, n)
> drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_CC_N_FTS_MASK             GENMASK(23, 16)
> drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_CC_N_FTS(n)               FIELD_PREP(PORT_AFR_CC_N_FTS_MASK, n)
> drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_ENTER_ASPM                BIT(30)
> drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_L0S_ENTRANCE_LAT_SHIFT    24
> drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_L0S_ENTRANCE_LAT_MASK     GENMASK(26, 24)
> drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_L1_ENTRANCE_LAT_SHIFT     27
> drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_L1_ENTRANCE_LAT_MASK      GENMASK(29, 27)
> 
> Rob

Rob: nice catch!
Does it mean that this MT7621 SoC has dwc controller and driver can be
theoretically in future rewritten to use common dwc code?

Sergio: I have tried to find some information about it and seems that
MT7620, MT7621 and MT7628 SoC are really using some designware dwc IP.
Some details are available in section "Embedded/kernel developer
friendliness" in following blog post:
https://www.abclinuxu.cz/blog/GardenOfEdenConfiguration/2019/10/opus-magnum

And seems that "programming guide" documentation for MT7620 is available
on internet with description of PCIe registers. I do not know how MT7620
and MT7621 are different but maybe it could help to develop or understand
driver.

Re: [PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Sergio Paracuellos]

Hi Pali,

On Sat, Jun 5, 2021 at 12:58 AM Pali Rohár <pali@kernel.org> wrote:
>
> On Friday 04 June 2021 13:49:39 Rob Herring wrote:
> > On Fri, Jun 04, 2021 at 06:55:25PM +0200, Pali Rohár wrote:
> > > On Wednesday 02 June 2021 14:43:53 Sergio Paracuellos wrote:
> > > > Hi Pali,
> > > >
> > > > On Wed, Jun 2, 2021 at 2:23 PM Pali Rohár <pali@kernel.org> wrote:
> > > > >
> > > > > On Wednesday 02 June 2021 14:16:26 Sergio Paracuellos wrote:
> > > > > > Hi Pali,
> > > > > >
> > > > > > On Mon, May 31, 2021 at 4:19 PM Sergio Paracuellos
> > > > > > <sergio.paracuellos@gmail.com> wrote:
> > > > > > >
> > > > > > > On Mon, May 31, 2021 at 3:50 PM Pali Rohár <pali@kernel.org> wrote:
> > > > > > > >
> > > > > > > > On Monday 31 May 2021 15:39:55 Sergio Paracuellos wrote:
> > > > > > > > > Hi Pali,
> > > > > > > > >
> > > > > > > > > Thanks for your comments.
> > > > > > > > >
> > > > > > > > > On Mon, May 31, 2021 at 3:14 PM Pali Rohár <pali@kernel.org> wrote:
> > > > > > > > > >
> > > > > > > > > > On Saturday 15 May 2021 14:40:53 Sergio Paracuellos wrote:
> > > > > > > > > > > This patch adds a driver for the PCIe controller of MT7621 SoC.
> > > > > > > > > > >
> > > > > > > > > > > Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > > > > > > > > > > ---
> > > > > > > > > > >  arch/mips/pci/Makefile     |   1 +
> > > > > > > > > > >  arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
> > > > > > > > > > >  arch/mips/ralink/Kconfig   |   9 +-
> > > > > > > > > > >  3 files changed, 633 insertions(+), 1 deletion(-)
> > > > > > > > > > >  create mode 100644 arch/mips/pci/pci-mt7621.c
> > > > > > > > > > >
> > > > > > > > > > > diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
> > > > > > > > > > > index f3eecc065e5c..178c550739c4 100644
> > > > > > > > > > > --- a/arch/mips/pci/Makefile
> > > > > > > > > > > +++ b/arch/mips/pci/Makefile
> > > > > > > > > > > @@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)    += pci-ar2315.o
> > > > > > > > > > >  obj-$(CONFIG_SOC_AR71XX)     += pci-ar71xx.o
> > > > > > > > > > >  obj-$(CONFIG_PCI_AR724X)     += pci-ar724x.o
> > > > > > > > > > >  obj-$(CONFIG_PCI_XTALK_BRIDGE)       += pci-xtalk-bridge.o
> > > > > > > > > > > +obj-$(CONFIG_PCI_MT7621)     += pci-mt7621.o
> > > > > > > > > > >  #
> > > > > > > > > > >  # These are still pretty much in the old state, watch, go blind.
> > > > > > > > > > >  #
> > > > > > > > > > > diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
> > > > > > > > > > > new file mode 100644
> > > > > > > > > > > index 000000000000..fe1945819d25
> > > > > > > > > > > --- /dev/null
> > > > > > > > > > > +++ b/arch/mips/pci/pci-mt7621.c
> > > > > > > > > > ...
> > > > > > > > > > > +static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> > > > > > > > > > > +{
> > > > > > > > > > > +     struct device *dev = pcie->dev;
> > > > > > > > > > > +     struct mt7621_pcie_port *port;
> > > > > > > > > > > +     u8 num_slots_enabled = 0;
> > > > > > > > > > > +     u32 slot;
> > > > > > > > > > > +     u32 val;
> > > > > > > > > > > +     int err;
> > > > > > > > > > > +
> > > > > > > > > > > +     /* Setup MEMWIN and IOWIN */
> > > > > > > > > > > +     pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> > > > > > > > > > > +     pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> > > > > > > > > > > +
> > > > > > > > > > > +     list_for_each_entry(port, &pcie->ports, list) {
> > > > > > > > > > > +             if (port->enabled) {
> > > > > > > > > > > +                     err = clk_prepare_enable(port->clk);
> > > > > > > > > > > +                     if (err) {
> > > > > > > > > > > +                             dev_err(dev, "enabling clk pcie%d\n", slot);
> > > > > > > > > > > +                             return err;
> > > > > > > > > > > +                     }
> > > > > > > > > > > +
> > > > > > > > > > > +                     mt7621_pcie_enable_port(port);
> > > > > > > > > > > +                     dev_info(dev, "PCIE%d enabled\n", port->slot);
> > > > > > > > > > > +                     num_slots_enabled++;
> > > > > > > > > > > +             }
> > > > > > > > > > > +     }
> > > > > > > > > > > +
> > > > > > > > > > > +     for (slot = 0; slot < num_slots_enabled; slot++) {
> > > > > > > > > > > +             val = read_config(pcie, slot, PCI_COMMAND);
> > > > > > > > > > > +             val |= PCI_COMMAND_MASTER;
> > > > > > > > > > > +             write_config(pcie, slot, PCI_COMMAND, val);
> > > > > > > > > >
> > > > > > > > > > Hello! Is this part of code correct? Because it looks strange if PCIe
> > > > > > > > > > controller driver automatically enables PCI bus mastering, prior device
> > > > > > > > > > driver initialize itself.
> > > > > > > > > >
> > > > > > > > > > Moreover kernel has already function pci_set_master() for this purpose
> > > > > > > > > > which is used by device drivers.
> > > > > > > > > >
> > > > > > > > > > So I think this code can confuse some device drivers...
> > > > > > > > >
> > > > > > > > > I agree that we have pci_set_master() to be used in pci device driver
> > > > > > > > > code. Original controller driver set this bit for enabled slots. Since
> > > > > > > > > there is no documentation at all for the PCI in this SoC
> > > > > > > >
> > > > > > > > I see... this is really a big problem to do any driver development...
> > > > > > >
> > > > > > > For sure it is :(.
> > > > > > >
> > > > > > > >
> > > > > > > > > I have
> > > > > > > > > maintained the setting in the driver in a cleaner way. See original
> > > > > > > > > driver code and the setting here [0]. There is no other reason than
> > > > > > > > > that. I am ok with removing this from here and testing with my two
> > > > > > > > > devices that everything is still ok if having this setting in the pci
> > > > > > > > > controller driver is a real problem.
> > > > > > > >
> > > > > > > > You can run lspci -nnvv with and without PCI_COMMAND_MASTER code and
> > > > > > > > then compare outputs.
> > > > > > > >
> > > > > > > > Device drivers for sure enable PCI_COMMAND_MASTER at the time when it is
> > > > > > > > needed, so it is possible that there would be no difference in lspci
> > > > > > > > output.
> > > > > > >
> > > > > > > Thanks. I will take this into account when v2 is submitted after more
> > > > > > > review comments come :).
> > > > > >
> > > > > > I have tested to remove this and check lspci -nnvv output with and
> > > > > > without PCI_COMMAND_MASTER code and, as you pointed out, there is no
> > > > > > difference between them. Also, both boards are working without
> > > > > > regressions at all. So I will remove this code for next version.
> > > > >
> > > > > Perfect!
> > > > >
> > > > > > Thanks,
> > > > > >     Sergio Paracuellos
> > > > > > >
> > > > > > > >
> > > > > > > > > [0]: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/tree/drivers/staging/mt7621-pci/pci-mt7621.c?h=v4.18#n676
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > >     Sergio Paracuellos
> > > > > > > > > >
> > > > > > > > > > > +             /* configure RC FTS number to 250 when it leaves L0s */
> > > > > > > > > > > +             val = read_config(pcie, slot, PCIE_FTS_NUM);
> > > > > > > > > > > +             val &= ~PCIE_FTS_NUM_MASK;
> > > > > > > > > > > +             val |= PCIE_FTS_NUM_L0(0x50);
> > > > > > > > > > > +             write_config(pcie, slot, PCIE_FTS_NUM, val);
> > > > >
> > > > > Could you look also what is doing this code (PCIE_FTS_NUM)? It is marked
> > > > > as MT specific register. But from this code for me it looks like that it
> > > > > just access config space of some device and therefore it could be some
> > > > > standard PCIe register. Just with hardcoded calculated offset.
> > >
> > > So based on your lspci output, there is no PCIe capability register at
> > > address PCIE_FTS_NUM (0x70c), right? It seems strange to trying access
> > > capability register outside of capability list.
> >
> > Looks like a DW PCIe port logic register:
> >
> > drivers/pci/controller/dwc/pcie-designware.h-/* Synopsys-specific PCIe configuration registers */
> > drivers/pci/controller/dwc/pcie-designware.h:#define PCIE_PORT_AFR                      0x70C
> > drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_N_FTS_MASK                GENMASK(15, 8)
> > drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_N_FTS(n)          FIELD_PREP(PORT_AFR_N_FTS_MASK, n)
> > drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_CC_N_FTS_MASK             GENMASK(23, 16)
> > drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_CC_N_FTS(n)               FIELD_PREP(PORT_AFR_CC_N_FTS_MASK, n)
> > drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_ENTER_ASPM                BIT(30)
> > drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_L0S_ENTRANCE_LAT_SHIFT    24
> > drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_L0S_ENTRANCE_LAT_MASK     GENMASK(26, 24)
> > drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_L1_ENTRANCE_LAT_SHIFT     27
> > drivers/pci/controller/dwc/pcie-designware.h-#define PORT_AFR_L1_ENTRANCE_LAT_MASK      GENMASK(29, 27)
> >
> > Rob
>
> Rob: nice catch!
> Does it mean that this MT7621 SoC has dwc controller and driver can be
> theoretically in future rewritten to use common dwc code?
>
> Sergio: I have tried to find some information about it and seems that
> MT7620, MT7621 and MT7628 SoC are really using some designware dwc IP.
> Some details are available in section "Embedded/kernel developer
> friendliness" in following blog post:
> https://www.abclinuxu.cz/blog/GardenOfEdenConfiguration/2019/10/opus-magnum

Thanks. For what see in the datasheet linked there some of the
registers looks pretty similar. It also defines registers taking in
account three root ports but in mt7628 and mt7620 there is only one
available. It seems tons of things are not listed there but it is
better than nothing.

>
> And seems that "programming guide" documentation for MT7620 is available
> on internet with description of PCIe registers. I do not know how MT7620
> and MT7621 are different but maybe it could help to develop or understand
> driver.

Yes, it is available but none of the dwc stuff is inside. Only the
main register as the mt7628 datasheet has.

Best regards,
    Sergio Paracuellos

Re: [PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Rob Herring]

On Sat, May 15, 2021 at 02:40:53PM +0200, Sergio Paracuellos wrote:
> This patch adds a driver for the PCIe controller of MT7621 SoC.
> 
> Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> ---
>  arch/mips/pci/Makefile     |   1 +
>  arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
>  arch/mips/ralink/Kconfig   |   9 +-
>  3 files changed, 633 insertions(+), 1 deletion(-)
>  create mode 100644 arch/mips/pci/pci-mt7621.c
> 
> diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
> index f3eecc065e5c..178c550739c4 100644
> --- a/arch/mips/pci/Makefile
> +++ b/arch/mips/pci/Makefile
> @@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)	+= pci-ar2315.o
>  obj-$(CONFIG_SOC_AR71XX)	+= pci-ar71xx.o
>  obj-$(CONFIG_PCI_AR724X)	+= pci-ar724x.o
>  obj-$(CONFIG_PCI_XTALK_BRIDGE)	+= pci-xtalk-bridge.o
> +obj-$(CONFIG_PCI_MT7621)	+= pci-mt7621.o
>  #
>  # These are still pretty much in the old state, watch, go blind.
>  #
> diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
> new file mode 100644
> index 000000000000..fe1945819d25
> --- /dev/null
> +++ b/arch/mips/pci/pci-mt7621.c
> @@ -0,0 +1,624 @@
> +// SPDX-License-Identifier: GPL-2.0+
> +/*
> + * BRIEF MODULE DESCRIPTION
> + *     PCI init for Ralink RT2880 solution
> + *
> + * Copyright 2007 Ralink Inc. (bruce_chang@ralinktech.com.tw)
> + *
> + * May 2007 Bruce Chang
> + * Initial Release
> + *
> + * May 2009 Bruce Chang
> + * support RT2880/RT3883 PCIe
> + *
> + * May 2011 Bruce Chang
> + * support RT6855/MT7620 PCIe
> + */
> +
> +#include <linux/bitops.h>
> +#include <linux/clk.h>
> +#include <linux/delay.h>
> +#include <linux/gpio/consumer.h>
> +#include <linux/module.h>
> +#include <linux/of.h>
> +#include <linux/of_address.h>
> +#include <linux/of_pci.h>
> +#include <linux/of_platform.h>
> +#include <linux/pci.h>
> +#include <linux/phy/phy.h>
> +#include <linux/platform_device.h>
> +#include <linux/reset.h>
> +#include <linux/sys_soc.h>
> +
> +/* MediaTek specific configuration registers */
> +#define PCIE_FTS_NUM			0x70c
> +#define PCIE_FTS_NUM_MASK		GENMASK(15, 8)
> +#define PCIE_FTS_NUM_L0(x)		(((x) & 0xff) << 8)
> +
> +/* Host-PCI bridge registers */
> +#define RALINK_PCI_PCICFG_ADDR		0x0000
> +#define RALINK_PCI_PCIMSK_ADDR		0x000C
> +#define RALINK_PCI_CONFIG_ADDR		0x0020
> +#define RALINK_PCI_CONFIG_DATA		0x0024
> +#define RALINK_PCI_MEMBASE		0x0028
> +#define RALINK_PCI_IOBASE		0x002C
> +
> +/* PCIe RC control registers */
> +#define MT7621_PCIE_OFFSET		0x2000
> +#define MT7621_NEXT_PORT		0x1000
> +
> +#define RALINK_PCI_BAR0SETUP_ADDR	0x0010

Standard BAR0 register?

> +#define RALINK_PCI_ID			0x0030
> +#define RALINK_PCI_CLASS		0x0034
> +#define RALINK_PCI_SUBID		0x0038
> +#define RALINK_PCI_STATUS		0x0050
> +
> +/* Some definition values */
> +#define PCIE_REVISION_ID		BIT(0)
> +#define PCIE_CLASS_CODE			(0x60400 << 8)
> +#define PCIE_BAR_MAP_MAX		GENMASK(30, 16)
> +#define PCIE_BAR_ENABLE			BIT(0)
> +#define PCIE_PORT_INT_EN(x)		BIT(20 + (x))
> +#define PCIE_PORT_LINKUP		BIT(0)
> +
> +#define PERST_DELAY_MS			100
> +
> +/**
> + * struct mt7621_pcie_port - PCIe port information
> + * @base: I/O mapped register base
> + * @list: port list
> + * @pcie: pointer to PCIe host info
> + * @clk: pointer to the port clock gate
> + * @phy: pointer to PHY control block
> + * @pcie_rst: pointer to port reset control
> + * @gpio_rst: gpio reset
> + * @slot: port slot
> + * @enabled: indicates if port is enabled
> + */
> +struct mt7621_pcie_port {
> +	void __iomem *base;
> +	struct list_head list;
> +	struct mt7621_pcie *pcie;
> +	struct clk *clk;
> +	struct phy *phy;
> +	struct reset_control *pcie_rst;
> +	struct gpio_desc *gpio_rst;
> +	u32 slot;
> +	bool enabled;
> +};
> +
> +/**
> + * struct mt7621_pcie - PCIe host information
> + * @base: IO Mapped Register Base
> + * @io: IO resource
> + * @mem: pointer to non-prefetchable memory resource
> + * @dev: Pointer to PCIe device
> + * @io_map_base: virtual memory base address for io
> + * @ports: pointer to PCIe port information
> + * @resets_inverted: depends on chip revision
> + * reset lines are inverted.
> + */
> +struct mt7621_pcie {
> +	void __iomem *base;
> +	struct device *dev;

> +	struct resource io;
> +	struct resource *mem;
> +	unsigned long io_map_base;

These are all stored in the host bridge struct, no need for you to store 
them.

> +	struct list_head ports;

A list is kind of an overkill for 3 entries and you know how many ports. 
Just embed an array of struct mt7621_pcie_port. Then you only need 1 
alloc.

> +	bool resets_inverted;
> +};
> +
> +static inline u32 pcie_read(struct mt7621_pcie *pcie, u32 reg)
> +{
> +	return readl(pcie->base + reg);

Can use _relaxed variants here and through out.

> +}
> +
> +static inline void pcie_write(struct mt7621_pcie *pcie, u32 val, u32 reg)
> +{
> +	writel(val, pcie->base + reg);
> +}
> +
> +static inline void pcie_rmw(struct mt7621_pcie *pcie, u32 reg, u32 clr, u32 set)
> +{
> +	u32 val = readl(pcie->base + reg);
> +
> +	val &= ~clr;
> +	val |= set;
> +	writel(val, pcie->base + reg);
> +}
> +
> +static inline u32 pcie_port_read(struct mt7621_pcie_port *port, u32 reg)
> +{
> +	return readl(port->base + reg);
> +}
> +
> +static inline void pcie_port_write(struct mt7621_pcie_port *port,
> +				   u32 val, u32 reg)
> +{
> +	writel(val, port->base + reg);
> +}
> +
> +static inline u32 mt7621_pci_get_cfgaddr(unsigned int bus, unsigned int slot,
> +					 unsigned int func, unsigned int where)
> +{
> +	return (((where & 0xF00) >> 8) << 24) | (bus << 16) | (slot << 11) |
> +		(func << 8) | (where & 0xfc) | 0x80000000;
> +}
> +
> +static void __iomem *mt7621_pcie_map_bus(struct pci_bus *bus,
> +					 unsigned int devfn, int where)
> +{
> +	struct mt7621_pcie *pcie = bus->sysdata;
> +	u32 address = mt7621_pci_get_cfgaddr(bus->number, PCI_SLOT(devfn),
> +					     PCI_FUNC(devfn), where);
> +
> +	writel(address, pcie->base + RALINK_PCI_CONFIG_ADDR);
> +
> +	return pcie->base + RALINK_PCI_CONFIG_DATA + (where & 3);
> +}
> +
> +struct pci_ops mt7621_pci_ops = {
> +	.map_bus	= mt7621_pcie_map_bus,
> +	.read		= pci_generic_config_read,
> +	.write		= pci_generic_config_write,
> +};
> +
> +static u32 read_config(struct mt7621_pcie *pcie, unsigned int dev, u32 reg)
> +{
> +	u32 address = mt7621_pci_get_cfgaddr(0, dev, 0, reg);
> +
> +	pcie_write(pcie, address, RALINK_PCI_CONFIG_ADDR);
> +	return pcie_read(pcie, RALINK_PCI_CONFIG_DATA);
> +}
> +
> +static void write_config(struct mt7621_pcie *pcie, unsigned int dev,
> +			 u32 reg, u32 val)
> +{
> +	u32 address = mt7621_pci_get_cfgaddr(0, dev, 0, reg);
> +
> +	pcie_write(pcie, address, RALINK_PCI_CONFIG_ADDR);
> +	pcie_write(pcie, val, RALINK_PCI_CONFIG_DATA);
> +}
> +
> +static inline void mt7621_rst_gpio_pcie_assert(struct mt7621_pcie_port *port)
> +{
> +	if (port->gpio_rst)

Don't need the if. gpiod_set_value should work with NULL.

> +		gpiod_set_value(port->gpio_rst, 1);
> +}
> +
> +static inline void mt7621_rst_gpio_pcie_deassert(struct mt7621_pcie_port *port)
> +{
> +	if (port->gpio_rst)
> +		gpiod_set_value(port->gpio_rst, 0);
> +}
> +
> +static inline bool mt7621_pcie_port_is_linkup(struct mt7621_pcie_port *port)
> +{
> +	return (pcie_port_read(port, RALINK_PCI_STATUS) & PCIE_PORT_LINKUP) != 0;
> +}
> +
> +static inline void mt7621_control_assert(struct mt7621_pcie_port *port)
> +{
> +	struct mt7621_pcie *pcie = port->pcie;
> +
> +	if (pcie->resets_inverted)
> +		reset_control_assert(port->pcie_rst);
> +	else
> +		reset_control_deassert(port->pcie_rst);
> +}
> +
> +static inline void mt7621_control_deassert(struct mt7621_pcie_port *port)
> +{
> +	struct mt7621_pcie *pcie = port->pcie;
> +
> +	if (pcie->resets_inverted)
> +		reset_control_deassert(port->pcie_rst);
> +	else
> +		reset_control_assert(port->pcie_rst);
> +}
> +
> +static void setup_cm_memory_region(struct mt7621_pcie *pcie)
> +{
> +	struct resource *mem_resource = pcie->mem;
> +	struct device *dev = pcie->dev;
> +	resource_size_t mask;
> +
> +	if (mips_cps_numiocu(0)) {
> +		/*
> +		 * FIXME: hardware doesn't accept mask values with 1s after
> +		 * 0s (e.g. 0xffef), so it would be great to warn if that's
> +		 * about to happen
> +		 */
> +		mask = ~(mem_resource->end - mem_resource->start);
> +
> +		write_gcr_reg1_base(mem_resource->start);
> +		write_gcr_reg1_mask(mask | CM_GCR_REGn_MASK_CMTGT_IOCU0);
> +		dev_info(dev, "PCI coherence region base: 0x%08llx, mask/settings: 0x%08llx\n",
> +			 (unsigned long long)read_gcr_reg1_base(),
> +			 (unsigned long long)read_gcr_reg1_mask());
> +	}
> +}
> +
> +static int mt7621_pci_parse_request_of_pci_ranges(struct pci_host_bridge *host)
> +{
> +	struct mt7621_pcie *pcie = pci_host_bridge_priv(host);
> +	struct device *dev = pcie->dev;
> +	struct device_node *node = dev->of_node;
> +	struct of_pci_range_parser parser;
> +	struct resource_entry *entry;
> +	struct of_pci_range range;
> +	LIST_HEAD(res);
> +
> +	if (of_pci_range_parser_init(&parser, node)) {
> +		dev_err(dev, "missing \"ranges\" property\n");
> +		return -EINVAL;
> +	}
> +
> +	/*
> +	 * IO_SPACE_LIMIT for MIPS is 0xffff but this platform uses IO at
> +	 * upper address 0x001e160000. of_pci_range_to_resource does not work

I think that's normal...

> +	 * well for MIPS platforms that don't define PCI_IOBASE, so set the IO
> +	 * resource manually instead.

Can't this be fixed?

> +	 */
> +	for_each_of_pci_range(&parser, &range) {
> +		switch (range.flags & IORESOURCE_TYPE_BITS) {

The core code already parses ranges for you. Try not to do it again. 
(Use the resource instead)

> +		case IORESOURCE_IO:
> +			pcie->io_map_base =
> +				(unsigned long)ioremap(range.cpu_addr,
> +						       range.size);
> +			pcie->io.name = node->full_name;
> +			pcie->io.flags = range.flags;
> +			pcie->io.start = range.cpu_addr;
> +			pcie->io.end = range.cpu_addr + range.size - 1;
> +			pcie->io.parent = pcie->io.child = pcie->io.sibling = NULL;
> +			set_io_port_base(pcie->io_map_base);
> +			break;
> +		}
> +	}
> +
> +	entry = resource_list_first_type(&host->windows, IORESOURCE_MEM);
> +	if (!entry) {
> +		dev_err(dev, "Cannot get memory resource");
> +		return -EINVAL;
> +	}
> +
> +	pcie->mem = entry->res;
> +	pci_add_resource(&res, &pcie->io);
> +	pci_add_resource(&res, entry->res);
> +	list_splice_init(&res, &host->windows);

This should already be done for you.

> +
> +	return 0;
> +}
> +
> +static int mt7621_pcie_parse_port(struct mt7621_pcie *pcie,
> +				  int slot)
> +{
> +	struct mt7621_pcie_port *port;
> +	struct device *dev = pcie->dev;
> +	struct platform_device *pdev = to_platform_device(dev);
> +	char name[10];
> +
> +	port = devm_kzalloc(dev, sizeof(*port), GFP_KERNEL);
> +	if (!port)
> +		return -ENOMEM;
> +
> +	port->base = devm_platform_ioremap_resource(pdev, slot + 1);
> +	if (IS_ERR(port->base))
> +		return PTR_ERR(port->base);
> +
> +	snprintf(name, sizeof(name), "pcie%d", slot);
> +	port->clk = devm_clk_get(dev, name);
> +	if (IS_ERR(port->clk)) {
> +		dev_err(dev, "failed to get pcie%d clock\n", slot);
> +		return PTR_ERR(port->clk);
> +	}
> +
> +	snprintf(name, sizeof(name), "pcie%d", slot);
> +	port->pcie_rst = devm_reset_control_get_exclusive(dev, name);
> +	if (PTR_ERR(port->pcie_rst) == -EPROBE_DEFER) {
> +		dev_err(dev, "failed to get pcie%d reset control\n", slot);
> +		return PTR_ERR(port->pcie_rst);
> +	}
> +
> +	snprintf(name, sizeof(name), "pcie-phy%d", slot);
> +	port->phy = devm_phy_get(dev, name);
> +	if (IS_ERR(port->phy) && slot != 1)
> +		return PTR_ERR(port->phy);
> +
> +	port->gpio_rst = devm_gpiod_get_index_optional(dev, "reset", slot,
> +						       GPIOD_OUT_LOW);
> +	if (IS_ERR(port->gpio_rst)) {
> +		dev_err(dev, "Failed to get GPIO for PCIe%d\n", slot);
> +		return PTR_ERR(port->gpio_rst);
> +	}
> +
> +	port->slot = slot;
> +	port->pcie = pcie;
> +
> +	INIT_LIST_HEAD(&port->list);
> +	list_add_tail(&port->list, &pcie->ports);
> +
> +	return 0;
> +}
> +
> +static int mt7621_pcie_parse_dt(struct mt7621_pcie *pcie)
> +{
> +	struct device *dev = pcie->dev;
> +	struct platform_device *pdev = to_platform_device(dev);
> +	struct device_node *node = dev->of_node, *child;
> +	int err;
> +
> +	pcie->base = devm_platform_ioremap_resource(pdev, 0);
> +	if (IS_ERR(pcie->base))
> +		return PTR_ERR(pcie->base);
> +
> +	for_each_available_child_of_node(node, child) {
> +		int slot;
> +
> +		err = of_pci_get_devfn(child);
> +		if (err < 0) {
> +			of_node_put(child);
> +			dev_err(dev, "failed to parse devfn: %d\n", err);
> +			return err;
> +		}
> +
> +		slot = PCI_SLOT(err);
> +
> +		err = mt7621_pcie_parse_port(pcie, slot);
> +		if (err) {
> +			of_node_put(child);
> +			return err;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static int mt7621_pcie_init_port(struct mt7621_pcie_port *port)
> +{
> +	struct mt7621_pcie *pcie = port->pcie;
> +	struct device *dev = pcie->dev;
> +	u32 slot = port->slot;
> +	int err;
> +
> +	err = phy_init(port->phy);
> +	if (err) {
> +		dev_err(dev, "failed to initialize port%d phy\n", slot);
> +		return err;
> +	}
> +
> +	err = phy_power_on(port->phy);
> +	if (err) {
> +		dev_err(dev, "failed to power on port%d phy\n", slot);
> +		phy_exit(port->phy);
> +		return err;
> +	}
> +
> +	port->enabled = true;
> +
> +	return 0;
> +}
> +
> +static void mt7621_pcie_reset_assert(struct mt7621_pcie *pcie)
> +{
> +	struct mt7621_pcie_port *port;
> +
> +	list_for_each_entry(port, &pcie->ports, list) {
> +		/* PCIe RC reset assert */
> +		mt7621_control_assert(port);
> +
> +		/* PCIe EP reset assert */
> +		mt7621_rst_gpio_pcie_assert(port);
> +	}
> +
> +	msleep(PERST_DELAY_MS);
> +}
> +
> +static void mt7621_pcie_reset_rc_deassert(struct mt7621_pcie *pcie)
> +{
> +	struct mt7621_pcie_port *port;
> +
> +	list_for_each_entry(port, &pcie->ports, list)
> +		mt7621_control_deassert(port);
> +}
> +
> +static void mt7621_pcie_reset_ep_deassert(struct mt7621_pcie *pcie)
> +{
> +	struct mt7621_pcie_port *port;
> +
> +	list_for_each_entry(port, &pcie->ports, list)
> +		mt7621_rst_gpio_pcie_deassert(port);
> +
> +	msleep(PERST_DELAY_MS);
> +}
> +
> +static void mt7621_pcie_init_ports(struct mt7621_pcie *pcie)
> +{
> +	struct device *dev = pcie->dev;
> +	struct mt7621_pcie_port *port, *tmp;
> +	int err;
> +
> +	mt7621_pcie_reset_assert(pcie);
> +	mt7621_pcie_reset_rc_deassert(pcie);
> +
> +	list_for_each_entry_safe(port, tmp, &pcie->ports, list) {
> +		u32 slot = port->slot;
> +
> +		if (slot == 1) {
> +			port->enabled = true;
> +			continue;
> +		}
> +
> +		err = mt7621_pcie_init_port(port);
> +		if (err) {
> +			dev_err(dev, "Initiating port %d failed\n", slot);
> +			list_del(&port->list);
> +		}
> +	}
> +
> +	mt7621_pcie_reset_ep_deassert(pcie);
> +
> +	tmp = NULL;
> +	list_for_each_entry(port, &pcie->ports, list) {
> +		u32 slot = port->slot;
> +
> +		if (!mt7621_pcie_port_is_linkup(port)) {
> +			dev_err(dev, "pcie%d no card, disable it (RST & CLK)\n",
> +				slot);
> +			mt7621_control_assert(port);
> +			clk_disable_unprepare(port->clk);
> +			port->enabled = false;
> +
> +			if (slot == 0) {
> +				tmp = port;
> +				continue;
> +			}
> +
> +			if (slot == 1 && tmp && !tmp->enabled)
> +				phy_power_off(tmp->phy);
> +		}
> +	}
> +}
> +
> +static void mt7621_pcie_enable_port(struct mt7621_pcie_port *port)
> +{
> +	struct mt7621_pcie *pcie = port->pcie;
> +	u32 slot = port->slot;
> +	u32 offset = MT7621_PCIE_OFFSET + (slot * MT7621_NEXT_PORT);

I don't see how this works unless the ports happen to be at the same VA 
offset. The writes below are to the 'common' registers which are 0x100 
bytes long based on the DT example, but the offset lines up with the 
port offsets.

> +	u32 val;
> +
> +	/* enable pcie interrupt */
> +	val = pcie_read(pcie, RALINK_PCI_PCIMSK_ADDR);
> +	val |= PCIE_PORT_INT_EN(slot);
> +	pcie_write(pcie, val, RALINK_PCI_PCIMSK_ADDR);
> +
> +	/* map 2G DDR region */
> +	pcie_write(pcie, PCIE_BAR_MAP_MAX | PCIE_BAR_ENABLE,
> +		   offset + RALINK_PCI_BAR0SETUP_ADDR);
> +
> +	/* configure class code and revision ID */
> +	pcie_write(pcie, PCIE_CLASS_CODE | PCIE_REVISION_ID,
> +		   offset + RALINK_PCI_CLASS);
> +}
> +
> +static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> +{
> +	struct device *dev = pcie->dev;
> +	struct mt7621_pcie_port *port;
> +	u8 num_slots_enabled = 0;
> +	u32 slot;
> +	u32 val;
> +	int err;
> +
> +	/* Setup MEMWIN and IOWIN */
> +	pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> +	pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> +
> +	list_for_each_entry(port, &pcie->ports, list) {
> +		if (port->enabled) {
> +			err = clk_prepare_enable(port->clk);
> +			if (err) {
> +				dev_err(dev, "enabling clk pcie%d\n", slot);
> +				return err;
> +			}
> +
> +			mt7621_pcie_enable_port(port);
> +			dev_info(dev, "PCIE%d enabled\n", port->slot);
> +			num_slots_enabled++;
> +		}
> +	}
> +
> +	for (slot = 0; slot < num_slots_enabled; slot++) {
> +		val = read_config(pcie, slot, PCI_COMMAND);
> +		val |= PCI_COMMAND_MASTER;
> +		write_config(pcie, slot, PCI_COMMAND, val);
> +		/* configure RC FTS number to 250 when it leaves L0s */
> +		val = read_config(pcie, slot, PCIE_FTS_NUM);
> +		val &= ~PCIE_FTS_NUM_MASK;
> +		val |= PCIE_FTS_NUM_L0(0x50);
> +		write_config(pcie, slot, PCIE_FTS_NUM, val);
> +	}
> +
> +	return 0;
> +}
> +
> +static int mt7621_pcie_register_host(struct pci_host_bridge *host)
> +{
> +	struct mt7621_pcie *pcie = pci_host_bridge_priv(host);
> +
> +	host->ops = &mt7621_pci_ops;
> +	host->sysdata = pcie;
> +	return pci_host_probe(host);
> +}
> +
> +static const struct soc_device_attribute mt7621_pci_quirks_match[] = {
> +	{ .soc_id = "mt7621", .revision = "E2" }
> +};
> +
> +static int mt7621_pci_probe(struct platform_device *pdev)
> +{
> +	struct device *dev = &pdev->dev;
> +	const struct soc_device_attribute *attr;
> +	struct mt7621_pcie *pcie;
> +	struct pci_host_bridge *bridge;
> +	int err;
> +
> +	if (!dev->of_node)
> +		return -ENODEV;
> +
> +	bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
> +	if (!bridge)
> +		return -ENOMEM;
> +
> +	pcie = pci_host_bridge_priv(bridge);
> +	pcie->dev = dev;
> +	platform_set_drvdata(pdev, pcie);
> +	INIT_LIST_HEAD(&pcie->ports);
> +
> +	attr = soc_device_match(mt7621_pci_quirks_match);
> +	if (attr)
> +		pcie->resets_inverted = true;
> +
> +	err = mt7621_pcie_parse_dt(pcie);
> +	if (err) {
> +		dev_err(dev, "Parsing DT failed\n");
> +		return err;
> +	}
> +
> +	err = mt7621_pci_parse_request_of_pci_ranges(bridge);
> +	if (err) {
> +		dev_err(dev, "Error requesting pci resources from ranges");
> +		return err;
> +	}
> +
> +	/* set resources limits */
> +	ioport_resource.start = pcie->io.start;
> +	ioport_resource.end = pcie->io.end;
> +
> +	mt7621_pcie_init_ports(pcie);
> +
> +	err = mt7621_pcie_enable_ports(pcie);

Really need 2 functions here?

> +	if (err) {
> +		dev_err(dev, "Error enabling pcie ports\n");
> +		return err;
> +	}
> +
> +	setup_cm_memory_region(pcie);
> +
> +	return mt7621_pcie_register_host(bridge);
> +}
> +
> +static const struct of_device_id mt7621_pci_ids[] = {
> +	{ .compatible = "mediatek,mt7621-pci" },
> +	{},
> +};
> +MODULE_DEVICE_TABLE(of, mt7621_pci_ids);
> +
> +static struct platform_driver mt7621_pci_driver = {
> +	.probe = mt7621_pci_probe,
> +	.driver = {
> +		.name = "mt7621-pci",
> +		.of_match_table = of_match_ptr(mt7621_pci_ids),
> +	},
> +};
> +builtin_platform_driver(mt7621_pci_driver);
> diff --git a/arch/mips/ralink/Kconfig b/arch/mips/ralink/Kconfig
> index ec4daa63c5e3..50e5a54f7d9e 100644
> --- a/arch/mips/ralink/Kconfig
> +++ b/arch/mips/ralink/Kconfig
> @@ -56,7 +56,7 @@ choice
>  		select MIPS_GIC
>  		select COMMON_CLK
>  		select CLKSRC_MIPS_GIC
> -		select HAVE_PCI if PCI_MT7621
> +		select HAVE_PCI
>  		select SOC_BUS
>  endchoice
>  
> @@ -101,4 +101,11 @@ choice
>  
>  endchoice
>  
> +config PCI_MT7621
> +	bool "MediaTek MT7621 PCI Controller"
> +	depends on RALINK && SOC_MT7621

Ideally this should also have (|| COMPILE_TEST), but looks like there 
are a few MIPS dependencies in here. So maybe (|| (MIPS && 
COMPILE_TEST)? If it's not built by allmodconfig or defconfig, it's hard 
for people to compile test it.


> +	select PCI_DRIVERS_GENERIC
> +	help
> +	  This selects a driver for the MediaTek MT7621 PCI Controller.
> +
>  endif
> -- 
> 2.25.1

Re: [PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Sergio Paracuellos]

Hi Rob,

Thanks for the review.

On Fri, Jun 4, 2021 at 9:30 PM Rob Herring <robh@kernel.org> wrote:
>
> On Sat, May 15, 2021 at 02:40:53PM +0200, Sergio Paracuellos wrote:
> > This patch adds a driver for the PCIe controller of MT7621 SoC.
> >
> > Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > ---
> >  arch/mips/pci/Makefile     |   1 +
> >  arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
> >  arch/mips/ralink/Kconfig   |   9 +-
> >  3 files changed, 633 insertions(+), 1 deletion(-)
> >  create mode 100644 arch/mips/pci/pci-mt7621.c
> >
> > diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
> > index f3eecc065e5c..178c550739c4 100644
> > --- a/arch/mips/pci/Makefile
> > +++ b/arch/mips/pci/Makefile
> > @@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)    += pci-ar2315.o
> >  obj-$(CONFIG_SOC_AR71XX)     += pci-ar71xx.o
> >  obj-$(CONFIG_PCI_AR724X)     += pci-ar724x.o
> >  obj-$(CONFIG_PCI_XTALK_BRIDGE)       += pci-xtalk-bridge.o
> > +obj-$(CONFIG_PCI_MT7621)     += pci-mt7621.o
> >  #
> >  # These are still pretty much in the old state, watch, go blind.
> >  #
> > diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
> > new file mode 100644
> > index 000000000000..fe1945819d25
> > --- /dev/null
> > +++ b/arch/mips/pci/pci-mt7621.c
> > @@ -0,0 +1,624 @@
> > +// SPDX-License-Identifier: GPL-2.0+
> > +/*
> > + * BRIEF MODULE DESCRIPTION
> > + *     PCI init for Ralink RT2880 solution
> > + *
> > + * Copyright 2007 Ralink Inc. (bruce_chang@ralinktech.com.tw)
> > + *
> > + * May 2007 Bruce Chang
> > + * Initial Release
> > + *
> > + * May 2009 Bruce Chang
> > + * support RT2880/RT3883 PCIe
> > + *
> > + * May 2011 Bruce Chang
> > + * support RT6855/MT7620 PCIe
> > + */
> > +
> > +#include <linux/bitops.h>
> > +#include <linux/clk.h>
> > +#include <linux/delay.h>
> > +#include <linux/gpio/consumer.h>
> > +#include <linux/module.h>
> > +#include <linux/of.h>
> > +#include <linux/of_address.h>
> > +#include <linux/of_pci.h>
> > +#include <linux/of_platform.h>
> > +#include <linux/pci.h>
> > +#include <linux/phy/phy.h>
> > +#include <linux/platform_device.h>
> > +#include <linux/reset.h>
> > +#include <linux/sys_soc.h>
> > +
> > +/* MediaTek specific configuration registers */
> > +#define PCIE_FTS_NUM                 0x70c
> > +#define PCIE_FTS_NUM_MASK            GENMASK(15, 8)
> > +#define PCIE_FTS_NUM_L0(x)           (((x) & 0xff) << 8)
> > +
> > +/* Host-PCI bridge registers */
> > +#define RALINK_PCI_PCICFG_ADDR               0x0000
> > +#define RALINK_PCI_PCIMSK_ADDR               0x000C
> > +#define RALINK_PCI_CONFIG_ADDR               0x0020
> > +#define RALINK_PCI_CONFIG_DATA               0x0024
> > +#define RALINK_PCI_MEMBASE           0x0028
> > +#define RALINK_PCI_IOBASE            0x002C
> > +
> > +/* PCIe RC control registers */
> > +#define MT7621_PCIE_OFFSET           0x2000
> > +#define MT7621_NEXT_PORT             0x1000
> > +
> > +#define RALINK_PCI_BAR0SETUP_ADDR    0x0010
>
> Standard BAR0 register?

Ok, I will remove this and use 'PCI_BASE_ADDRESS_0' instead.

>
> > +#define RALINK_PCI_ID                        0x0030
> > +#define RALINK_PCI_CLASS             0x0034
> > +#define RALINK_PCI_SUBID             0x0038
> > +#define RALINK_PCI_STATUS            0x0050
> > +
> > +/* Some definition values */
> > +#define PCIE_REVISION_ID             BIT(0)
> > +#define PCIE_CLASS_CODE                      (0x60400 << 8)
> > +#define PCIE_BAR_MAP_MAX             GENMASK(30, 16)
> > +#define PCIE_BAR_ENABLE                      BIT(0)
> > +#define PCIE_PORT_INT_EN(x)          BIT(20 + (x))
> > +#define PCIE_PORT_LINKUP             BIT(0)
> > +
> > +#define PERST_DELAY_MS                       100
> > +
> > +/**
> > + * struct mt7621_pcie_port - PCIe port information
> > + * @base: I/O mapped register base
> > + * @list: port list
> > + * @pcie: pointer to PCIe host info
> > + * @clk: pointer to the port clock gate
> > + * @phy: pointer to PHY control block
> > + * @pcie_rst: pointer to port reset control
> > + * @gpio_rst: gpio reset
> > + * @slot: port slot
> > + * @enabled: indicates if port is enabled
> > + */
> > +struct mt7621_pcie_port {
> > +     void __iomem *base;
> > +     struct list_head list;
> > +     struct mt7621_pcie *pcie;
> > +     struct clk *clk;
> > +     struct phy *phy;
> > +     struct reset_control *pcie_rst;
> > +     struct gpio_desc *gpio_rst;
> > +     u32 slot;
> > +     bool enabled;
> > +};
> > +
> > +/**
> > + * struct mt7621_pcie - PCIe host information
> > + * @base: IO Mapped Register Base
> > + * @io: IO resource
> > + * @mem: pointer to non-prefetchable memory resource
> > + * @dev: Pointer to PCIe device
> > + * @io_map_base: virtual memory base address for io
> > + * @ports: pointer to PCIe port information
> > + * @resets_inverted: depends on chip revision
> > + * reset lines are inverted.
> > + */
> > +struct mt7621_pcie {
> > +     void __iomem *base;
> > +     struct device *dev;
>
> > +     struct resource io;
> > +     struct resource *mem;
> > +     unsigned long io_map_base;
>
> These are all stored in the host bridge struct, no need for you to store
> them.

IO resources must be requested and mapped manually (see my explanation
below) and I use mem resource to also setup mips iocu regions, that is
why I am storing it also here.

>
> > +     struct list_head ports;
>
> A list is kind of an overkill for 3 entries and you know how many ports.
> Just embed an array of struct mt7621_pcie_port. Then you only need 1
> alloc.

Since ports are at most three but can be one or two also depending on
the board I ended up using a list instead of a fixed array of three
ports. This list is dynamically updated depending on link status. If
some of the ports are not present in the board nodes initially stored
after device tree parsing are deleted for the list. If it is not a big
problem I prefer to maintain this list as it is.

>
> > +     bool resets_inverted;
> > +};
> > +
> > +static inline u32 pcie_read(struct mt7621_pcie *pcie, u32 reg)
> > +{
> > +     return readl(pcie->base + reg);
>
> Can use _relaxed variants here and through out.

Ok will change into _relaxed variants if is preferred.

>
> > +}
> > +
> > +static inline void pcie_write(struct mt7621_pcie *pcie, u32 val, u32 reg)
> > +{
> > +     writel(val, pcie->base + reg);
> > +}
> > +
> > +static inline void pcie_rmw(struct mt7621_pcie *pcie, u32 reg, u32 clr, u32 set)
> > +{
> > +     u32 val = readl(pcie->base + reg);
> > +
> > +     val &= ~clr;
> > +     val |= set;
> > +     writel(val, pcie->base + reg);
> > +}
> > +
> > +static inline u32 pcie_port_read(struct mt7621_pcie_port *port, u32 reg)
> > +{
> > +     return readl(port->base + reg);
> > +}
> > +
> > +static inline void pcie_port_write(struct mt7621_pcie_port *port,
> > +                                u32 val, u32 reg)
> > +{
> > +     writel(val, port->base + reg);
> > +}
> > +
> > +static inline u32 mt7621_pci_get_cfgaddr(unsigned int bus, unsigned int slot,
> > +                                      unsigned int func, unsigned int where)
> > +{
> > +     return (((where & 0xF00) >> 8) << 24) | (bus << 16) | (slot << 11) |
> > +             (func << 8) | (where & 0xfc) | 0x80000000;
> > +}
> > +
> > +static void __iomem *mt7621_pcie_map_bus(struct pci_bus *bus,
> > +                                      unsigned int devfn, int where)
> > +{
> > +     struct mt7621_pcie *pcie = bus->sysdata;
> > +     u32 address = mt7621_pci_get_cfgaddr(bus->number, PCI_SLOT(devfn),
> > +                                          PCI_FUNC(devfn), where);
> > +
> > +     writel(address, pcie->base + RALINK_PCI_CONFIG_ADDR);
> > +
> > +     return pcie->base + RALINK_PCI_CONFIG_DATA + (where & 3);
> > +}
> > +
> > +struct pci_ops mt7621_pci_ops = {
> > +     .map_bus        = mt7621_pcie_map_bus,
> > +     .read           = pci_generic_config_read,
> > +     .write          = pci_generic_config_write,
> > +};
> > +
> > +static u32 read_config(struct mt7621_pcie *pcie, unsigned int dev, u32 reg)
> > +{
> > +     u32 address = mt7621_pci_get_cfgaddr(0, dev, 0, reg);
> > +
> > +     pcie_write(pcie, address, RALINK_PCI_CONFIG_ADDR);
> > +     return pcie_read(pcie, RALINK_PCI_CONFIG_DATA);
> > +}
> > +
> > +static void write_config(struct mt7621_pcie *pcie, unsigned int dev,
> > +                      u32 reg, u32 val)
> > +{
> > +     u32 address = mt7621_pci_get_cfgaddr(0, dev, 0, reg);
> > +
> > +     pcie_write(pcie, address, RALINK_PCI_CONFIG_ADDR);
> > +     pcie_write(pcie, val, RALINK_PCI_CONFIG_DATA);
> > +}
> > +
> > +static inline void mt7621_rst_gpio_pcie_assert(struct mt7621_pcie_port *port)
> > +{
> > +     if (port->gpio_rst)
>
> Don't need the if. gpiod_set_value should work with NULL.

Gpio resets can be optional. Some boards use one gpio reset pin for
each port but others only one for all of them. So if no reset is not
requested for a port I don't want to do anything. Hence, the check.

>
> > +             gpiod_set_value(port->gpio_rst, 1);
> > +}
> > +
> > +static inline void mt7621_rst_gpio_pcie_deassert(struct mt7621_pcie_port *port)
> > +{
> > +     if (port->gpio_rst)
> > +             gpiod_set_value(port->gpio_rst, 0);
> > +}
> > +
> > +static inline bool mt7621_pcie_port_is_linkup(struct mt7621_pcie_port *port)
> > +{
> > +     return (pcie_port_read(port, RALINK_PCI_STATUS) & PCIE_PORT_LINKUP) != 0;
> > +}
> > +
> > +static inline void mt7621_control_assert(struct mt7621_pcie_port *port)
> > +{
> > +     struct mt7621_pcie *pcie = port->pcie;
> > +
> > +     if (pcie->resets_inverted)
> > +             reset_control_assert(port->pcie_rst);
> > +     else
> > +             reset_control_deassert(port->pcie_rst);
> > +}
> > +
> > +static inline void mt7621_control_deassert(struct mt7621_pcie_port *port)
> > +{
> > +     struct mt7621_pcie *pcie = port->pcie;
> > +
> > +     if (pcie->resets_inverted)
> > +             reset_control_deassert(port->pcie_rst);
> > +     else
> > +             reset_control_assert(port->pcie_rst);
> > +}
> > +
> > +static void setup_cm_memory_region(struct mt7621_pcie *pcie)
> > +{
> > +     struct resource *mem_resource = pcie->mem;
> > +     struct device *dev = pcie->dev;
> > +     resource_size_t mask;
> > +
> > +     if (mips_cps_numiocu(0)) {
> > +             /*
> > +              * FIXME: hardware doesn't accept mask values with 1s after
> > +              * 0s (e.g. 0xffef), so it would be great to warn if that's
> > +              * about to happen
> > +              */
> > +             mask = ~(mem_resource->end - mem_resource->start);
> > +
> > +             write_gcr_reg1_base(mem_resource->start);
> > +             write_gcr_reg1_mask(mask | CM_GCR_REGn_MASK_CMTGT_IOCU0);
> > +             dev_info(dev, "PCI coherence region base: 0x%08llx, mask/settings: 0x%08llx\n",
> > +                      (unsigned long long)read_gcr_reg1_base(),
> > +                      (unsigned long long)read_gcr_reg1_mask());
> > +     }
> > +}
> > +
> > +static int mt7621_pci_parse_request_of_pci_ranges(struct pci_host_bridge *host)
> > +{
> > +     struct mt7621_pcie *pcie = pci_host_bridge_priv(host);
> > +     struct device *dev = pcie->dev;
> > +     struct device_node *node = dev->of_node;
> > +     struct of_pci_range_parser parser;
> > +     struct resource_entry *entry;
> > +     struct of_pci_range range;
> > +     LIST_HEAD(res);
> > +
> > +     if (of_pci_range_parser_init(&parser, node)) {
> > +             dev_err(dev, "missing \"ranges\" property\n");
> > +             return -EINVAL;
> > +     }
> > +
> > +     /*
> > +      * IO_SPACE_LIMIT for MIPS is 0xffff but this platform uses IO at
> > +      * upper address 0x001e160000. of_pci_range_to_resource does not work
>
> I think that's normal...

No, pci_address_to_pio will fail and io resources are not properly
assigned at all.

>
> > +      * well for MIPS platforms that don't define PCI_IOBASE, so set the IO
> > +      * resource manually instead.
>
> Can't this be fixed?

This is the way the current PCI architecture is in mips... Maybe
Thomas has a strong opinion on this.

>
> > +      */
> > +     for_each_of_pci_range(&parser, &range) {
> > +             switch (range.flags & IORESOURCE_TYPE_BITS) {
>
> The core code already parses ranges for you. Try not to do it again.
> (Use the resource instead)

See below...

>
> > +             case IORESOURCE_IO:
> > +                     pcie->io_map_base =
> > +                             (unsigned long)ioremap(range.cpu_addr,
> > +                                                    range.size);
> > +                     pcie->io.name = node->full_name;
> > +                     pcie->io.flags = range.flags;
> > +                     pcie->io.start = range.cpu_addr;
> > +                     pcie->io.end = range.cpu_addr + range.size - 1;
> > +                     pcie->io.parent = pcie->io.child = pcie->io.sibling = NULL;
> > +                     set_io_port_base(pcie->io_map_base);
> > +                     break;
> > +             }
> > +     }
> > +
> > +     entry = resource_list_first_type(&host->windows, IORESOURCE_MEM);
> > +     if (!entry) {
> > +             dev_err(dev, "Cannot get memory resource");
> > +             return -EINVAL;
> > +     }
> > +
> > +     pcie->mem = entry->res;
> > +     pci_add_resource(&res, &pcie->io);
> > +     pci_add_resource(&res, entry->res);
> > +     list_splice_init(&res, &host->windows);
>
> This should already be done for you.

Most MIPS platforms do not define PCI_IOBASE, nor implement
pci_address_to_pio(). Moreover, IO_SPACE_LIMIT is 0xffff for most MIPS
platforms. of_pci_range_to_resource passes the _start address_ of the
IO range into pci_address_to_pio, which then checks it against
IO_SPACE_LIMIT and fails. So I have to do this manually or nothing
will work properly...

>
> > +
> > +     return 0;
> > +}
> > +
> > +static int mt7621_pcie_parse_port(struct mt7621_pcie *pcie,
> > +                               int slot)
> > +{
> > +     struct mt7621_pcie_port *port;
> > +     struct device *dev = pcie->dev;
> > +     struct platform_device *pdev = to_platform_device(dev);
> > +     char name[10];
> > +
> > +     port = devm_kzalloc(dev, sizeof(*port), GFP_KERNEL);
> > +     if (!port)
> > +             return -ENOMEM;
> > +
> > +     port->base = devm_platform_ioremap_resource(pdev, slot + 1);
> > +     if (IS_ERR(port->base))
> > +             return PTR_ERR(port->base);
> > +
> > +     snprintf(name, sizeof(name), "pcie%d", slot);
> > +     port->clk = devm_clk_get(dev, name);
> > +     if (IS_ERR(port->clk)) {
> > +             dev_err(dev, "failed to get pcie%d clock\n", slot);
> > +             return PTR_ERR(port->clk);
> > +     }
> > +
> > +     snprintf(name, sizeof(name), "pcie%d", slot);
> > +     port->pcie_rst = devm_reset_control_get_exclusive(dev, name);
> > +     if (PTR_ERR(port->pcie_rst) == -EPROBE_DEFER) {
> > +             dev_err(dev, "failed to get pcie%d reset control\n", slot);
> > +             return PTR_ERR(port->pcie_rst);
> > +     }
> > +
> > +     snprintf(name, sizeof(name), "pcie-phy%d", slot);
> > +     port->phy = devm_phy_get(dev, name);
> > +     if (IS_ERR(port->phy) && slot != 1)
> > +             return PTR_ERR(port->phy);
> > +
> > +     port->gpio_rst = devm_gpiod_get_index_optional(dev, "reset", slot,
> > +                                                    GPIOD_OUT_LOW);
> > +     if (IS_ERR(port->gpio_rst)) {
> > +             dev_err(dev, "Failed to get GPIO for PCIe%d\n", slot);
> > +             return PTR_ERR(port->gpio_rst);
> > +     }
> > +
> > +     port->slot = slot;
> > +     port->pcie = pcie;
> > +
> > +     INIT_LIST_HEAD(&port->list);
> > +     list_add_tail(&port->list, &pcie->ports);
> > +
> > +     return 0;
> > +}
> > +
> > +static int mt7621_pcie_parse_dt(struct mt7621_pcie *pcie)
> > +{
> > +     struct device *dev = pcie->dev;
> > +     struct platform_device *pdev = to_platform_device(dev);
> > +     struct device_node *node = dev->of_node, *child;
> > +     int err;
> > +
> > +     pcie->base = devm_platform_ioremap_resource(pdev, 0);
> > +     if (IS_ERR(pcie->base))
> > +             return PTR_ERR(pcie->base);
> > +
> > +     for_each_available_child_of_node(node, child) {
> > +             int slot;
> > +
> > +             err = of_pci_get_devfn(child);
> > +             if (err < 0) {
> > +                     of_node_put(child);
> > +                     dev_err(dev, "failed to parse devfn: %d\n", err);
> > +                     return err;
> > +             }
> > +
> > +             slot = PCI_SLOT(err);
> > +
> > +             err = mt7621_pcie_parse_port(pcie, slot);
> > +             if (err) {
> > +                     of_node_put(child);
> > +                     return err;
> > +             }
> > +     }
> > +
> > +     return 0;
> > +}
> > +
> > +static int mt7621_pcie_init_port(struct mt7621_pcie_port *port)
> > +{
> > +     struct mt7621_pcie *pcie = port->pcie;
> > +     struct device *dev = pcie->dev;
> > +     u32 slot = port->slot;
> > +     int err;
> > +
> > +     err = phy_init(port->phy);
> > +     if (err) {
> > +             dev_err(dev, "failed to initialize port%d phy\n", slot);
> > +             return err;
> > +     }
> > +
> > +     err = phy_power_on(port->phy);
> > +     if (err) {
> > +             dev_err(dev, "failed to power on port%d phy\n", slot);
> > +             phy_exit(port->phy);
> > +             return err;
> > +     }
> > +
> > +     port->enabled = true;
> > +
> > +     return 0;
> > +}
> > +
> > +static void mt7621_pcie_reset_assert(struct mt7621_pcie *pcie)
> > +{
> > +     struct mt7621_pcie_port *port;
> > +
> > +     list_for_each_entry(port, &pcie->ports, list) {
> > +             /* PCIe RC reset assert */
> > +             mt7621_control_assert(port);
> > +
> > +             /* PCIe EP reset assert */
> > +             mt7621_rst_gpio_pcie_assert(port);
> > +     }
> > +
> > +     msleep(PERST_DELAY_MS);
> > +}
> > +
> > +static void mt7621_pcie_reset_rc_deassert(struct mt7621_pcie *pcie)
> > +{
> > +     struct mt7621_pcie_port *port;
> > +
> > +     list_for_each_entry(port, &pcie->ports, list)
> > +             mt7621_control_deassert(port);
> > +}
> > +
> > +static void mt7621_pcie_reset_ep_deassert(struct mt7621_pcie *pcie)
> > +{
> > +     struct mt7621_pcie_port *port;
> > +
> > +     list_for_each_entry(port, &pcie->ports, list)
> > +             mt7621_rst_gpio_pcie_deassert(port);
> > +
> > +     msleep(PERST_DELAY_MS);
> > +}
> > +
> > +static void mt7621_pcie_init_ports(struct mt7621_pcie *pcie)
> > +{
> > +     struct device *dev = pcie->dev;
> > +     struct mt7621_pcie_port *port, *tmp;
> > +     int err;
> > +
> > +     mt7621_pcie_reset_assert(pcie);
> > +     mt7621_pcie_reset_rc_deassert(pcie);
> > +
> > +     list_for_each_entry_safe(port, tmp, &pcie->ports, list) {
> > +             u32 slot = port->slot;
> > +
> > +             if (slot == 1) {
> > +                     port->enabled = true;
> > +                     continue;
> > +             }
> > +
> > +             err = mt7621_pcie_init_port(port);
> > +             if (err) {
> > +                     dev_err(dev, "Initiating port %d failed\n", slot);
> > +                     list_del(&port->list);
> > +             }
> > +     }
> > +
> > +     mt7621_pcie_reset_ep_deassert(pcie);
> > +
> > +     tmp = NULL;
> > +     list_for_each_entry(port, &pcie->ports, list) {
> > +             u32 slot = port->slot;
> > +
> > +             if (!mt7621_pcie_port_is_linkup(port)) {
> > +                     dev_err(dev, "pcie%d no card, disable it (RST & CLK)\n",
> > +                             slot);
> > +                     mt7621_control_assert(port);
> > +                     clk_disable_unprepare(port->clk);
> > +                     port->enabled = false;
> > +
> > +                     if (slot == 0) {
> > +                             tmp = port;
> > +                             continue;
> > +                     }
> > +
> > +                     if (slot == 1 && tmp && !tmp->enabled)
> > +                             phy_power_off(tmp->phy)
> > +             }
> > +     }
> > +}
> > +
> > +static void mt7621_pcie_enable_port(struct mt7621_pcie_port *port)
> > +{
> > +     struct mt7621_pcie *pcie = port->pcie;
> > +     u32 slot = port->slot;
> > +     u32 offset = MT7621_PCIE_OFFSET + (slot * MT7621_NEXT_PORT);
>
> I don't see how this works unless the ports happen to be at the same VA
> offset. The writes below are to the 'common' registers which are 0x100
> bytes long based on the DT example, but the offset lines up with the
> port offsets.

RC registers for port 0 start at physical address 0x1e142000, for port
1 0x1e143000 and for port 2 0x1e144000. We are using
'pcie_write' which internally uses pcie->base which is 0x1e140000, but
all the previous ones have been already requested and mapped for each
port so we calculate the offset from this 0x1e140000 and write there.

>
> > +     u32 val;
> > +
> > +     /* enable pcie interrupt */
> > +     val = pcie_read(pcie, RALINK_PCI_PCIMSK_ADDR);
> > +     val |= PCIE_PORT_INT_EN(slot);
> > +     pcie_write(pcie, val, RALINK_PCI_PCIMSK_ADDR);
> > +
> > +     /* map 2G DDR region */
> > +     pcie_write(pcie, PCIE_BAR_MAP_MAX | PCIE_BAR_ENABLE,
> > +                offset + RALINK_PCI_BAR0SETUP_ADDR);
> > +
> > +     /* configure class code and revision ID */
> > +     pcie_write(pcie, PCIE_CLASS_CODE | PCIE_REVISION_ID,
> > +                offset + RALINK_PCI_CLASS);
> > +}
> > +
> > +static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> > +{
> > +     struct device *dev = pcie->dev;
> > +     struct mt7621_pcie_port *port;
> > +     u8 num_slots_enabled = 0;
> > +     u32 slot;
> > +     u32 val;
> > +     int err;
> > +
> > +     /* Setup MEMWIN and IOWIN */
> > +     pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> > +     pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> > +
> > +     list_for_each_entry(port, &pcie->ports, list) {
> > +             if (port->enabled) {
> > +                     err = clk_prepare_enable(port->clk);
> > +                     if (err) {
> > +                             dev_err(dev, "enabling clk pcie%d\n", slot);
> > +                             return err;
> > +                     }
> > +
> > +                     mt7621_pcie_enable_port(port);
> > +                     dev_info(dev, "PCIE%d enabled\n", port->slot);
> > +                     num_slots_enabled++;
> > +             }
> > +     }
> > +
> > +     for (slot = 0; slot < num_slots_enabled; slot++) {
> > +             val = read_config(pcie, slot, PCI_COMMAND);
> > +             val |= PCI_COMMAND_MASTER;
> > +             write_config(pcie, slot, PCI_COMMAND, val);
> > +             /* configure RC FTS number to 250 when it leaves L0s */
> > +             val = read_config(pcie, slot, PCIE_FTS_NUM);
> > +             val &= ~PCIE_FTS_NUM_MASK;
> > +             val |= PCIE_FTS_NUM_L0(0x50);
> > +             write_config(pcie, slot, PCIE_FTS_NUM, val);
> > +     }
> > +
> > +     return 0;
> > +}
> > +
> > +static int mt7621_pcie_register_host(struct pci_host_bridge *host)
> > +{
> > +     struct mt7621_pcie *pcie = pci_host_bridge_priv(host);
> > +
> > +     host->ops = &mt7621_pci_ops;
> > +     host->sysdata = pcie;
> > +     return pci_host_probe(host);
> > +}
> > +
> > +static const struct soc_device_attribute mt7621_pci_quirks_match[] = {
> > +     { .soc_id = "mt7621", .revision = "E2" }
> > +};
> > +
> > +static int mt7621_pci_probe(struct platform_device *pdev)
> > +{
> > +     struct device *dev = &pdev->dev;
> > +     const struct soc_device_attribute *attr;
> > +     struct mt7621_pcie *pcie;
> > +     struct pci_host_bridge *bridge;
> > +     int err;
> > +
> > +     if (!dev->of_node)
> > +             return -ENODEV;
> > +
> > +     bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
> > +     if (!bridge)
> > +             return -ENOMEM;
> > +
> > +     pcie = pci_host_bridge_priv(bridge);
> > +     pcie->dev = dev;
> > +     platform_set_drvdata(pdev, pcie);
> > +     INIT_LIST_HEAD(&pcie->ports);
> > +
> > +     attr = soc_device_match(mt7621_pci_quirks_match);
> > +     if (attr)
> > +             pcie->resets_inverted = true;
> > +
> > +     err = mt7621_pcie_parse_dt(pcie);
> > +     if (err) {
> > +             dev_err(dev, "Parsing DT failed\n");
> > +             return err;
> > +     }
> > +
> > +     err = mt7621_pci_parse_request_of_pci_ranges(bridge);
> > +     if (err) {
> > +             dev_err(dev, "Error requesting pci resources from ranges");
> > +             return err;
> > +     }
> > +
> > +     /* set resources limits */
> > +     ioport_resource.start = pcie->io.start;
> > +     ioport_resource.end = pcie->io.end;
> > +
> > +     mt7621_pcie_init_ports(pcie);
> > +
> > +     err = mt7621_pcie_enable_ports(pcie);
>
> Really need 2 functions here?

mt7621_pcie_init_ports is in charge of power on the phy and checks for
link status to see which ports are in use and mt7621_pcie_enable_ports
just enable all the stuff to be ready to be used.

>
> > +     if (err) {
> > +             dev_err(dev, "Error enabling pcie ports\n");
> > +             return err;
> > +     }
> > +
> > +     setup_cm_memory_region(pcie);
> > +
> > +     return mt7621_pcie_register_host(bridge);
> > +}
> > +
> > +static const struct of_device_id mt7621_pci_ids[] = {
> > +     { .compatible = "mediatek,mt7621-pci" },
> > +     {},
> > +};
> > +MODULE_DEVICE_TABLE(of, mt7621_pci_ids);
> > +
> > +static struct platform_driver mt7621_pci_driver = {
> > +     .probe = mt7621_pci_probe,
> > +     .driver = {
> > +             .name = "mt7621-pci",
> > +             .of_match_table = of_match_ptr(mt7621_pci_ids),
> > +     },
> > +};
> > +builtin_platform_driver(mt7621_pci_driver);
> > diff --git a/arch/mips/ralink/Kconfig b/arch/mips/ralink/Kconfig
> > index ec4daa63c5e3..50e5a54f7d9e 100644
> > --- a/arch/mips/ralink/Kconfig
> > +++ b/arch/mips/ralink/Kconfig
> > @@ -56,7 +56,7 @@ choice
> >               select MIPS_GIC
> >               select COMMON_CLK
> >               select CLKSRC_MIPS_GIC
> > -             select HAVE_PCI if PCI_MT7621
> > +             select HAVE_PCI
> >               select SOC_BUS
> >  endchoice
> >
> > @@ -101,4 +101,11 @@ choice
> >
> >  endchoice
> >
> > +config PCI_MT7621
> > +     bool "MediaTek MT7621 PCI Controller"
> > +     depends on RALINK && SOC_MT7621
>
> Ideally this should also have (|| COMPILE_TEST), but looks like there
> are a few MIPS dependencies in here. So maybe (|| (MIPS &&
> COMPILE_TEST)? If it's not built by allmodconfig or defconfig, it's hard
> for people to compile test it.

Ok, I will add (|| (MIPS && COMPILE_TEST) here.

Best regards,
    Sergio Paracuellos

>
>
> > +     select PCI_DRIVERS_GENERIC
> > +     help
> > +       This selects a driver for the MediaTek MT7621 PCI Controller.
> > +
> >  endif
> > --
> > 2.25.1

Re: [PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Sergio Paracuellos]

Hi Rob,

On Sat, Jun 5, 2021 at 12:25 AM Sergio Paracuellos
<sergio.paracuellos@gmail.com> wrote:
>
> Hi Rob,
>
> Thanks for the review.
>
> On Fri, Jun 4, 2021 at 9:30 PM Rob Herring <robh@kernel.org> wrote:
> >
> > On Sat, May 15, 2021 at 02:40:53PM +0200, Sergio Paracuellos wrote:
> > > This patch adds a driver for the PCIe controller of MT7621 SoC.
> > >
> > > Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > > ---
> > >  arch/mips/pci/Makefile     |   1 +
> > >  arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
> > >  arch/mips/ralink/Kconfig   |   9 +-
> > >  3 files changed, 633 insertions(+), 1 deletion(-)
> > >  create mode 100644 arch/mips/pci/pci-mt7621.c
> > >
> > > diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
> > > index f3eecc065e5c..178c550739c4 100644
> > > --- a/arch/mips/pci/Makefile
> > > +++ b/arch/mips/pci/Makefile
> > > @@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)    += pci-ar2315.o
> > >  obj-$(CONFIG_SOC_AR71XX)     += pci-ar71xx.o
> > >  obj-$(CONFIG_PCI_AR724X)     += pci-ar724x.o
> > >  obj-$(CONFIG_PCI_XTALK_BRIDGE)       += pci-xtalk-bridge.o
> > > +obj-$(CONFIG_PCI_MT7621)     += pci-mt7621.o
> > >  #
> > >  # These are still pretty much in the old state, watch, go blind.
> > >  #
> > > diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
> > > new file mode 100644
> > > index 000000000000..fe1945819d25
> > > --- /dev/null
> > > +++ b/arch/mips/pci/pci-mt7621.c
> > > @@ -0,0 +1,624 @@
> > > +// SPDX-License-Identifier: GPL-2.0+
> > > +/*
> > > + * BRIEF MODULE DESCRIPTION
> > > + *     PCI init for Ralink RT2880 solution
> > > + *
> > > + * Copyright 2007 Ralink Inc. (bruce_chang@ralinktech.com.tw)
> > > + *
> > > + * May 2007 Bruce Chang
> > > + * Initial Release
> > > + *
> > > + * May 2009 Bruce Chang
> > > + * support RT2880/RT3883 PCIe
> > > + *
> > > + * May 2011 Bruce Chang
> > > + * support RT6855/MT7620 PCIe
> > > + */
> > > +
> > > +#include <linux/bitops.h>
> > > +#include <linux/clk.h>
> > > +#include <linux/delay.h>
> > > +#include <linux/gpio/consumer.h>
> > > +#include <linux/module.h>
> > > +#include <linux/of.h>
> > > +#include <linux/of_address.h>
> > > +#include <linux/of_pci.h>
> > > +#include <linux/of_platform.h>
> > > +#include <linux/pci.h>
> > > +#include <linux/phy/phy.h>
> > > +#include <linux/platform_device.h>
> > > +#include <linux/reset.h>
> > > +#include <linux/sys_soc.h>
> > > +
> > > +/* MediaTek specific configuration registers */
> > > +#define PCIE_FTS_NUM                 0x70c
> > > +#define PCIE_FTS_NUM_MASK            GENMASK(15, 8)
> > > +#define PCIE_FTS_NUM_L0(x)           (((x) & 0xff) << 8)
> > > +
> > > +/* Host-PCI bridge registers */
> > > +#define RALINK_PCI_PCICFG_ADDR               0x0000
> > > +#define RALINK_PCI_PCIMSK_ADDR               0x000C
> > > +#define RALINK_PCI_CONFIG_ADDR               0x0020
> > > +#define RALINK_PCI_CONFIG_DATA               0x0024
> > > +#define RALINK_PCI_MEMBASE           0x0028
> > > +#define RALINK_PCI_IOBASE            0x002C
> > > +
> > > +/* PCIe RC control registers */
> > > +#define MT7621_PCIE_OFFSET           0x2000
> > > +#define MT7621_NEXT_PORT             0x1000
> > > +
> > > +#define RALINK_PCI_BAR0SETUP_ADDR    0x0010
> >
> > Standard BAR0 register?
>
> Ok, I will remove this and use 'PCI_BASE_ADDRESS_0' instead.
>
> >
> > > +#define RALINK_PCI_ID                        0x0030
> > > +#define RALINK_PCI_CLASS             0x0034
> > > +#define RALINK_PCI_SUBID             0x0038
> > > +#define RALINK_PCI_STATUS            0x0050
> > > +
> > > +/* Some definition values */
> > > +#define PCIE_REVISION_ID             BIT(0)
> > > +#define PCIE_CLASS_CODE                      (0x60400 << 8)
> > > +#define PCIE_BAR_MAP_MAX             GENMASK(30, 16)
> > > +#define PCIE_BAR_ENABLE                      BIT(0)
> > > +#define PCIE_PORT_INT_EN(x)          BIT(20 + (x))
> > > +#define PCIE_PORT_LINKUP             BIT(0)
> > > +
> > > +#define PERST_DELAY_MS                       100
> > > +
> > > +/**
> > > + * struct mt7621_pcie_port - PCIe port information
> > > + * @base: I/O mapped register base
> > > + * @list: port list
> > > + * @pcie: pointer to PCIe host info
> > > + * @clk: pointer to the port clock gate
> > > + * @phy: pointer to PHY control block
> > > + * @pcie_rst: pointer to port reset control
> > > + * @gpio_rst: gpio reset
> > > + * @slot: port slot
> > > + * @enabled: indicates if port is enabled
> > > + */
> > > +struct mt7621_pcie_port {
> > > +     void __iomem *base;
> > > +     struct list_head list;
> > > +     struct mt7621_pcie *pcie;
> > > +     struct clk *clk;
> > > +     struct phy *phy;
> > > +     struct reset_control *pcie_rst;
> > > +     struct gpio_desc *gpio_rst;
> > > +     u32 slot;
> > > +     bool enabled;
> > > +};
> > > +
> > > +/**
> > > + * struct mt7621_pcie - PCIe host information
> > > + * @base: IO Mapped Register Base
> > > + * @io: IO resource
> > > + * @mem: pointer to non-prefetchable memory resource
> > > + * @dev: Pointer to PCIe device
> > > + * @io_map_base: virtual memory base address for io
> > > + * @ports: pointer to PCIe port information
> > > + * @resets_inverted: depends on chip revision
> > > + * reset lines are inverted.
> > > + */
> > > +struct mt7621_pcie {
> > > +     void __iomem *base;
> > > +     struct device *dev;
> >
> > > +     struct resource io;
> > > +     struct resource *mem;
> > > +     unsigned long io_map_base;
> >
> > These are all stored in the host bridge struct, no need for you to store
> > them.
>
> IO resources must be requested and mapped manually (see my explanation
> below) and I use mem resource to also setup mips iocu regions, that is
> why I am storing it also here.
>
> >
> > > +     struct list_head ports;
> >
> > A list is kind of an overkill for 3 entries and you know how many ports.
> > Just embed an array of struct mt7621_pcie_port. Then you only need 1
> > alloc.
>
> Since ports are at most three but can be one or two also depending on
> the board I ended up using a list instead of a fixed array of three
> ports. This list is dynamically updated depending on link status. If
> some of the ports are not present in the board nodes initially stored
> after device tree parsing are deleted for the list. If it is not a big
> problem I prefer to maintain this list as it is.
>
> >
> > > +     bool resets_inverted;
> > > +};
> > > +
> > > +static inline u32 pcie_read(struct mt7621_pcie *pcie, u32 reg)
> > > +{
> > > +     return readl(pcie->base + reg);
> >
> > Can use _relaxed variants here and through out.
>
> Ok will change into _relaxed variants if is preferred.
>
> >
> > > +}
> > > +
> > > +static inline void pcie_write(struct mt7621_pcie *pcie, u32 val, u32 reg)
> > > +{
> > > +     writel(val, pcie->base + reg);
> > > +}
> > > +
> > > +static inline void pcie_rmw(struct mt7621_pcie *pcie, u32 reg, u32 clr, u32 set)
> > > +{
> > > +     u32 val = readl(pcie->base + reg);
> > > +
> > > +     val &= ~clr;
> > > +     val |= set;
> > > +     writel(val, pcie->base + reg);
> > > +}
> > > +
> > > +static inline u32 pcie_port_read(struct mt7621_pcie_port *port, u32 reg)
> > > +{
> > > +     return readl(port->base + reg);
> > > +}
> > > +
> > > +static inline void pcie_port_write(struct mt7621_pcie_port *port,
> > > +                                u32 val, u32 reg)
> > > +{
> > > +     writel(val, port->base + reg);
> > > +}
> > > +
> > > +static inline u32 mt7621_pci_get_cfgaddr(unsigned int bus, unsigned int slot,
> > > +                                      unsigned int func, unsigned int where)
> > > +{
> > > +     return (((where & 0xF00) >> 8) << 24) | (bus << 16) | (slot << 11) |
> > > +             (func << 8) | (where & 0xfc) | 0x80000000;
> > > +}
> > > +
> > > +static void __iomem *mt7621_pcie_map_bus(struct pci_bus *bus,
> > > +                                      unsigned int devfn, int where)
> > > +{
> > > +     struct mt7621_pcie *pcie = bus->sysdata;
> > > +     u32 address = mt7621_pci_get_cfgaddr(bus->number, PCI_SLOT(devfn),
> > > +                                          PCI_FUNC(devfn), where);
> > > +
> > > +     writel(address, pcie->base + RALINK_PCI_CONFIG_ADDR);
> > > +
> > > +     return pcie->base + RALINK_PCI_CONFIG_DATA + (where & 3);
> > > +}
> > > +
> > > +struct pci_ops mt7621_pci_ops = {
> > > +     .map_bus        = mt7621_pcie_map_bus,
> > > +     .read           = pci_generic_config_read,
> > > +     .write          = pci_generic_config_write,
> > > +};
> > > +
> > > +static u32 read_config(struct mt7621_pcie *pcie, unsigned int dev, u32 reg)
> > > +{
> > > +     u32 address = mt7621_pci_get_cfgaddr(0, dev, 0, reg);
> > > +
> > > +     pcie_write(pcie, address, RALINK_PCI_CONFIG_ADDR);
> > > +     return pcie_read(pcie, RALINK_PCI_CONFIG_DATA);
> > > +}
> > > +
> > > +static void write_config(struct mt7621_pcie *pcie, unsigned int dev,
> > > +                      u32 reg, u32 val)
> > > +{
> > > +     u32 address = mt7621_pci_get_cfgaddr(0, dev, 0, reg);
> > > +
> > > +     pcie_write(pcie, address, RALINK_PCI_CONFIG_ADDR);
> > > +     pcie_write(pcie, val, RALINK_PCI_CONFIG_DATA);
> > > +}
> > > +
> > > +static inline void mt7621_rst_gpio_pcie_assert(struct mt7621_pcie_port *port)
> > > +{
> > > +     if (port->gpio_rst)
> >
> > Don't need the if. gpiod_set_value should work with NULL.
>
> Gpio resets can be optional. Some boards use one gpio reset pin for
> each port but others only one for all of them. So if no reset is not
> requested for a port I don't want to do anything. Hence, the check.
>
> >
> > > +             gpiod_set_value(port->gpio_rst, 1);
> > > +}
> > > +
> > > +static inline void mt7621_rst_gpio_pcie_deassert(struct mt7621_pcie_port *port)
> > > +{
> > > +     if (port->gpio_rst)
> > > +             gpiod_set_value(port->gpio_rst, 0);
> > > +}
> > > +
> > > +static inline bool mt7621_pcie_port_is_linkup(struct mt7621_pcie_port *port)
> > > +{
> > > +     return (pcie_port_read(port, RALINK_PCI_STATUS) & PCIE_PORT_LINKUP) != 0;
> > > +}
> > > +
> > > +static inline void mt7621_control_assert(struct mt7621_pcie_port *port)
> > > +{
> > > +     struct mt7621_pcie *pcie = port->pcie;
> > > +
> > > +     if (pcie->resets_inverted)
> > > +             reset_control_assert(port->pcie_rst);
> > > +     else
> > > +             reset_control_deassert(port->pcie_rst);
> > > +}
> > > +
> > > +static inline void mt7621_control_deassert(struct mt7621_pcie_port *port)
> > > +{
> > > +     struct mt7621_pcie *pcie = port->pcie;
> > > +
> > > +     if (pcie->resets_inverted)
> > > +             reset_control_deassert(port->pcie_rst);
> > > +     else
> > > +             reset_control_assert(port->pcie_rst);
> > > +}
> > > +
> > > +static void setup_cm_memory_region(struct mt7621_pcie *pcie)
> > > +{
> > > +     struct resource *mem_resource = pcie->mem;
> > > +     struct device *dev = pcie->dev;
> > > +     resource_size_t mask;
> > > +
> > > +     if (mips_cps_numiocu(0)) {
> > > +             /*
> > > +              * FIXME: hardware doesn't accept mask values with 1s after
> > > +              * 0s (e.g. 0xffef), so it would be great to warn if that's
> > > +              * about to happen
> > > +              */
> > > +             mask = ~(mem_resource->end - mem_resource->start);
> > > +
> > > +             write_gcr_reg1_base(mem_resource->start);
> > > +             write_gcr_reg1_mask(mask | CM_GCR_REGn_MASK_CMTGT_IOCU0);
> > > +             dev_info(dev, "PCI coherence region base: 0x%08llx, mask/settings: 0x%08llx\n",
> > > +                      (unsigned long long)read_gcr_reg1_base(),
> > > +                      (unsigned long long)read_gcr_reg1_mask());
> > > +     }
> > > +}
> > > +
> > > +static int mt7621_pci_parse_request_of_pci_ranges(struct pci_host_bridge *host)
> > > +{
> > > +     struct mt7621_pcie *pcie = pci_host_bridge_priv(host);
> > > +     struct device *dev = pcie->dev;
> > > +     struct device_node *node = dev->of_node;
> > > +     struct of_pci_range_parser parser;
> > > +     struct resource_entry *entry;
> > > +     struct of_pci_range range;
> > > +     LIST_HEAD(res);
> > > +
> > > +     if (of_pci_range_parser_init(&parser, node)) {
> > > +             dev_err(dev, "missing \"ranges\" property\n");
> > > +             return -EINVAL;
> > > +     }
> > > +
> > > +     /*
> > > +      * IO_SPACE_LIMIT for MIPS is 0xffff but this platform uses IO at
> > > +      * upper address 0x001e160000. of_pci_range_to_resource does not work
> >
> > I think that's normal...
>
> No, pci_address_to_pio will fail and io resources are not properly
> assigned at all.
>
> >
> > > +      * well for MIPS platforms that don't define PCI_IOBASE, so set the IO
> > > +      * resource manually instead.
> >
> > Can't this be fixed?
>
> This is the way the current PCI architecture is in mips... Maybe
> Thomas has a strong opinion on this.
>
> >
> > > +      */
> > > +     for_each_of_pci_range(&parser, &range) {
> > > +             switch (range.flags & IORESOURCE_TYPE_BITS) {
> >
> > The core code already parses ranges for you. Try not to do it again.
> > (Use the resource instead)
>
> See below...
>
> >
> > > +             case IORESOURCE_IO:
> > > +                     pcie->io_map_base =
> > > +                             (unsigned long)ioremap(range.cpu_addr,
> > > +                                                    range.size);
> > > +                     pcie->io.name = node->full_name;
> > > +                     pcie->io.flags = range.flags;
> > > +                     pcie->io.start = range.cpu_addr;
> > > +                     pcie->io.end = range.cpu_addr + range.size - 1;
> > > +                     pcie->io.parent = pcie->io.child = pcie->io.sibling = NULL;
> > > +                     set_io_port_base(pcie->io_map_base);
> > > +                     break;
> > > +             }
> > > +     }
> > > +
> > > +     entry = resource_list_first_type(&host->windows, IORESOURCE_MEM);
> > > +     if (!entry) {
> > > +             dev_err(dev, "Cannot get memory resource");
> > > +             return -EINVAL;
> > > +     }
> > > +
> > > +     pcie->mem = entry->res;
> > > +     pci_add_resource(&res, &pcie->io);
> > > +     pci_add_resource(&res, entry->res);
> > > +     list_splice_init(&res, &host->windows);
> >
> > This should already be done for you.
>
> Most MIPS platforms do not define PCI_IOBASE, nor implement
> pci_address_to_pio(). Moreover, IO_SPACE_LIMIT is 0xffff for most MIPS
> platforms. of_pci_range_to_resource passes the _start address_ of the
> IO range into pci_address_to_pio, which then checks it against
> IO_SPACE_LIMIT and fails. So I have to do this manually or nothing
> will work properly...
>
> >
> > > +
> > > +     return 0;
> > > +}
> > > +
> > > +static int mt7621_pcie_parse_port(struct mt7621_pcie *pcie,
> > > +                               int slot)
> > > +{
> > > +     struct mt7621_pcie_port *port;
> > > +     struct device *dev = pcie->dev;
> > > +     struct platform_device *pdev = to_platform_device(dev);
> > > +     char name[10];
> > > +
> > > +     port = devm_kzalloc(dev, sizeof(*port), GFP_KERNEL);
> > > +     if (!port)
> > > +             return -ENOMEM;
> > > +
> > > +     port->base = devm_platform_ioremap_resource(pdev, slot + 1);
> > > +     if (IS_ERR(port->base))
> > > +             return PTR_ERR(port->base);
> > > +
> > > +     snprintf(name, sizeof(name), "pcie%d", slot);
> > > +     port->clk = devm_clk_get(dev, name);
> > > +     if (IS_ERR(port->clk)) {
> > > +             dev_err(dev, "failed to get pcie%d clock\n", slot);
> > > +             return PTR_ERR(port->clk);
> > > +     }
> > > +
> > > +     snprintf(name, sizeof(name), "pcie%d", slot);
> > > +     port->pcie_rst = devm_reset_control_get_exclusive(dev, name);
> > > +     if (PTR_ERR(port->pcie_rst) == -EPROBE_DEFER) {
> > > +             dev_err(dev, "failed to get pcie%d reset control\n", slot);
> > > +             return PTR_ERR(port->pcie_rst);
> > > +     }
> > > +
> > > +     snprintf(name, sizeof(name), "pcie-phy%d", slot);
> > > +     port->phy = devm_phy_get(dev, name);
> > > +     if (IS_ERR(port->phy) && slot != 1)
> > > +             return PTR_ERR(port->phy);
> > > +
> > > +     port->gpio_rst = devm_gpiod_get_index_optional(dev, "reset", slot,
> > > +                                                    GPIOD_OUT_LOW);
> > > +     if (IS_ERR(port->gpio_rst)) {
> > > +             dev_err(dev, "Failed to get GPIO for PCIe%d\n", slot);
> > > +             return PTR_ERR(port->gpio_rst);
> > > +     }
> > > +
> > > +     port->slot = slot;
> > > +     port->pcie = pcie;
> > > +
> > > +     INIT_LIST_HEAD(&port->list);
> > > +     list_add_tail(&port->list, &pcie->ports);
> > > +
> > > +     return 0;
> > > +}
> > > +
> > > +static int mt7621_pcie_parse_dt(struct mt7621_pcie *pcie)
> > > +{
> > > +     struct device *dev = pcie->dev;
> > > +     struct platform_device *pdev = to_platform_device(dev);
> > > +     struct device_node *node = dev->of_node, *child;
> > > +     int err;
> > > +
> > > +     pcie->base = devm_platform_ioremap_resource(pdev, 0);
> > > +     if (IS_ERR(pcie->base))
> > > +             return PTR_ERR(pcie->base);
> > > +
> > > +     for_each_available_child_of_node(node, child) {
> > > +             int slot;
> > > +
> > > +             err = of_pci_get_devfn(child);
> > > +             if (err < 0) {
> > > +                     of_node_put(child);
> > > +                     dev_err(dev, "failed to parse devfn: %d\n", err);
> > > +                     return err;
> > > +             }
> > > +
> > > +             slot = PCI_SLOT(err);
> > > +
> > > +             err = mt7621_pcie_parse_port(pcie, slot);
> > > +             if (err) {
> > > +                     of_node_put(child);
> > > +                     return err;
> > > +             }
> > > +     }
> > > +
> > > +     return 0;
> > > +}
> > > +
> > > +static int mt7621_pcie_init_port(struct mt7621_pcie_port *port)
> > > +{
> > > +     struct mt7621_pcie *pcie = port->pcie;
> > > +     struct device *dev = pcie->dev;
> > > +     u32 slot = port->slot;
> > > +     int err;
> > > +
> > > +     err = phy_init(port->phy);
> > > +     if (err) {
> > > +             dev_err(dev, "failed to initialize port%d phy\n", slot);
> > > +             return err;
> > > +     }
> > > +
> > > +     err = phy_power_on(port->phy);
> > > +     if (err) {
> > > +             dev_err(dev, "failed to power on port%d phy\n", slot);
> > > +             phy_exit(port->phy);
> > > +             return err;
> > > +     }
> > > +
> > > +     port->enabled = true;
> > > +
> > > +     return 0;
> > > +}
> > > +
> > > +static void mt7621_pcie_reset_assert(struct mt7621_pcie *pcie)
> > > +{
> > > +     struct mt7621_pcie_port *port;
> > > +
> > > +     list_for_each_entry(port, &pcie->ports, list) {
> > > +             /* PCIe RC reset assert */
> > > +             mt7621_control_assert(port);
> > > +
> > > +             /* PCIe EP reset assert */
> > > +             mt7621_rst_gpio_pcie_assert(port);
> > > +     }
> > > +
> > > +     msleep(PERST_DELAY_MS);
> > > +}
> > > +
> > > +static void mt7621_pcie_reset_rc_deassert(struct mt7621_pcie *pcie)
> > > +{
> > > +     struct mt7621_pcie_port *port;
> > > +
> > > +     list_for_each_entry(port, &pcie->ports, list)
> > > +             mt7621_control_deassert(port);
> > > +}
> > > +
> > > +static void mt7621_pcie_reset_ep_deassert(struct mt7621_pcie *pcie)
> > > +{
> > > +     struct mt7621_pcie_port *port;
> > > +
> > > +     list_for_each_entry(port, &pcie->ports, list)
> > > +             mt7621_rst_gpio_pcie_deassert(port);
> > > +
> > > +     msleep(PERST_DELAY_MS);
> > > +}
> > > +
> > > +static void mt7621_pcie_init_ports(struct mt7621_pcie *pcie)
> > > +{
> > > +     struct device *dev = pcie->dev;
> > > +     struct mt7621_pcie_port *port, *tmp;
> > > +     int err;
> > > +
> > > +     mt7621_pcie_reset_assert(pcie);
> > > +     mt7621_pcie_reset_rc_deassert(pcie);
> > > +
> > > +     list_for_each_entry_safe(port, tmp, &pcie->ports, list) {
> > > +             u32 slot = port->slot;
> > > +
> > > +             if (slot == 1) {
> > > +                     port->enabled = true;
> > > +                     continue;
> > > +             }
> > > +
> > > +             err = mt7621_pcie_init_port(port);
> > > +             if (err) {
> > > +                     dev_err(dev, "Initiating port %d failed\n", slot);
> > > +                     list_del(&port->list);
> > > +             }
> > > +     }
> > > +
> > > +     mt7621_pcie_reset_ep_deassert(pcie);
> > > +
> > > +     tmp = NULL;
> > > +     list_for_each_entry(port, &pcie->ports, list) {
> > > +             u32 slot = port->slot;
> > > +
> > > +             if (!mt7621_pcie_port_is_linkup(port)) {
> > > +                     dev_err(dev, "pcie%d no card, disable it (RST & CLK)\n",
> > > +                             slot);
> > > +                     mt7621_control_assert(port);
> > > +                     clk_disable_unprepare(port->clk);
> > > +                     port->enabled = false;
> > > +
> > > +                     if (slot == 0) {
> > > +                             tmp = port;
> > > +                             continue;
> > > +                     }
> > > +
> > > +                     if (slot == 1 && tmp && !tmp->enabled)
> > > +                             phy_power_off(tmp->phy)
> > > +             }
> > > +     }
> > > +}
> > > +
> > > +static void mt7621_pcie_enable_port(struct mt7621_pcie_port *port)
> > > +{
> > > +     struct mt7621_pcie *pcie = port->pcie;
> > > +     u32 slot = port->slot;
> > > +     u32 offset = MT7621_PCIE_OFFSET + (slot * MT7621_NEXT_PORT);
> >
> > I don't see how this works unless the ports happen to be at the same VA
> > offset. The writes below are to the 'common' registers which are 0x100
> > bytes long based on the DT example, but the offset lines up with the
> > port offsets.
>
> RC registers for port 0 start at physical address 0x1e142000, for port
> 1 0x1e143000 and for port 2 0x1e144000. We are using
> 'pcie_write' which internally uses pcie->base which is 0x1e140000, but
> all the previous ones have been already requested and mapped for each
> port so we calculate the offset from this 0x1e140000 and write there.
>
> >
> > > +     u32 val;
> > > +
> > > +     /* enable pcie interrupt */
> > > +     val = pcie_read(pcie, RALINK_PCI_PCIMSK_ADDR);
> > > +     val |= PCIE_PORT_INT_EN(slot);
> > > +     pcie_write(pcie, val, RALINK_PCI_PCIMSK_ADDR);
> > > +
> > > +     /* map 2G DDR region */
> > > +     pcie_write(pcie, PCIE_BAR_MAP_MAX | PCIE_BAR_ENABLE,
> > > +                offset + RALINK_PCI_BAR0SETUP_ADDR);
> > > +
> > > +     /* configure class code and revision ID */
> > > +     pcie_write(pcie, PCIE_CLASS_CODE | PCIE_REVISION_ID,
> > > +                offset + RALINK_PCI_CLASS);
> > > +}
> > > +
> > > +static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> > > +{
> > > +     struct device *dev = pcie->dev;
> > > +     struct mt7621_pcie_port *port;
> > > +     u8 num_slots_enabled = 0;
> > > +     u32 slot;
> > > +     u32 val;
> > > +     int err;
> > > +
> > > +     /* Setup MEMWIN and IOWIN */
> > > +     pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> > > +     pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> > > +
> > > +     list_for_each_entry(port, &pcie->ports, list) {
> > > +             if (port->enabled) {
> > > +                     err = clk_prepare_enable(port->clk);
> > > +                     if (err) {
> > > +                             dev_err(dev, "enabling clk pcie%d\n", slot);
> > > +                             return err;
> > > +                     }
> > > +
> > > +                     mt7621_pcie_enable_port(port);
> > > +                     dev_info(dev, "PCIE%d enabled\n", port->slot);
> > > +                     num_slots_enabled++;
> > > +             }
> > > +     }
> > > +
> > > +     for (slot = 0; slot < num_slots_enabled; slot++) {
> > > +             val = read_config(pcie, slot, PCI_COMMAND);
> > > +             val |= PCI_COMMAND_MASTER;
> > > +             write_config(pcie, slot, PCI_COMMAND, val);
> > > +             /* configure RC FTS number to 250 when it leaves L0s */
> > > +             val = read_config(pcie, slot, PCIE_FTS_NUM);
> > > +             val &= ~PCIE_FTS_NUM_MASK;
> > > +             val |= PCIE_FTS_NUM_L0(0x50);
> > > +             write_config(pcie, slot, PCIE_FTS_NUM, val);
> > > +     }
> > > +
> > > +     return 0;
> > > +}
> > > +
> > > +static int mt7621_pcie_register_host(struct pci_host_bridge *host)
> > > +{
> > > +     struct mt7621_pcie *pcie = pci_host_bridge_priv(host);
> > > +
> > > +     host->ops = &mt7621_pci_ops;
> > > +     host->sysdata = pcie;
> > > +     return pci_host_probe(host);
> > > +}
> > > +
> > > +static const struct soc_device_attribute mt7621_pci_quirks_match[] = {
> > > +     { .soc_id = "mt7621", .revision = "E2" }
> > > +};
> > > +
> > > +static int mt7621_pci_probe(struct platform_device *pdev)
> > > +{
> > > +     struct device *dev = &pdev->dev;
> > > +     const struct soc_device_attribute *attr;
> > > +     struct mt7621_pcie *pcie;
> > > +     struct pci_host_bridge *bridge;
> > > +     int err;
> > > +
> > > +     if (!dev->of_node)
> > > +             return -ENODEV;
> > > +
> > > +     bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
> > > +     if (!bridge)
> > > +             return -ENOMEM;
> > > +
> > > +     pcie = pci_host_bridge_priv(bridge);
> > > +     pcie->dev = dev;
> > > +     platform_set_drvdata(pdev, pcie);
> > > +     INIT_LIST_HEAD(&pcie->ports);
> > > +
> > > +     attr = soc_device_match(mt7621_pci_quirks_match);
> > > +     if (attr)
> > > +             pcie->resets_inverted = true;
> > > +
> > > +     err = mt7621_pcie_parse_dt(pcie);
> > > +     if (err) {
> > > +             dev_err(dev, "Parsing DT failed\n");
> > > +             return err;
> > > +     }
> > > +
> > > +     err = mt7621_pci_parse_request_of_pci_ranges(bridge);
> > > +     if (err) {
> > > +             dev_err(dev, "Error requesting pci resources from ranges");
> > > +             return err;
> > > +     }
> > > +
> > > +     /* set resources limits */
> > > +     ioport_resource.start = pcie->io.start;
> > > +     ioport_resource.end = pcie->io.end;
> > > +
> > > +     mt7621_pcie_init_ports(pcie);
> > > +
> > > +     err = mt7621_pcie_enable_ports(pcie);
> >
> > Really need 2 functions here?
>
> mt7621_pcie_init_ports is in charge of power on the phy and checks for
> link status to see which ports are in use and mt7621_pcie_enable_ports
> just enable all the stuff to be ready to be used.
>
> >
> > > +     if (err) {
> > > +             dev_err(dev, "Error enabling pcie ports\n");
> > > +             return err;
> > > +     }
> > > +
> > > +     setup_cm_memory_region(pcie);
> > > +
> > > +     return mt7621_pcie_register_host(bridge);
> > > +}
> > > +
> > > +static const struct of_device_id mt7621_pci_ids[] = {
> > > +     { .compatible = "mediatek,mt7621-pci" },
> > > +     {},
> > > +};
> > > +MODULE_DEVICE_TABLE(of, mt7621_pci_ids);
> > > +
> > > +static struct platform_driver mt7621_pci_driver = {
> > > +     .probe = mt7621_pci_probe,
> > > +     .driver = {
> > > +             .name = "mt7621-pci",
> > > +             .of_match_table = of_match_ptr(mt7621_pci_ids),
> > > +     },
> > > +};
> > > +builtin_platform_driver(mt7621_pci_driver);
> > > diff --git a/arch/mips/ralink/Kconfig b/arch/mips/ralink/Kconfig
> > > index ec4daa63c5e3..50e5a54f7d9e 100644
> > > --- a/arch/mips/ralink/Kconfig
> > > +++ b/arch/mips/ralink/Kconfig
> > > @@ -56,7 +56,7 @@ choice
> > >               select MIPS_GIC
> > >               select COMMON_CLK
> > >               select CLKSRC_MIPS_GIC
> > > -             select HAVE_PCI if PCI_MT7621
> > > +             select HAVE_PCI
> > >               select SOC_BUS
> > >  endchoice
> > >
> > > @@ -101,4 +101,11 @@ choice
> > >
> > >  endchoice
> > >
> > > +config PCI_MT7621
> > > +     bool "MediaTek MT7621 PCI Controller"
> > > +     depends on RALINK && SOC_MT7621
> >
> > Ideally this should also have (|| COMPILE_TEST), but looks like there
> > are a few MIPS dependencies in here. So maybe (|| (MIPS &&
> > COMPILE_TEST)? If it's not built by allmodconfig or defconfig, it's hard
> > for people to compile test it.
>
> Ok, I will add (|| (MIPS && COMPILE_TEST) here.
>
> Best regards,
>     Sergio Paracuellos
>
> >
> >
> > > +     select PCI_DRIVERS_GENERIC
> > > +     help
> > > +       This selects a driver for the MediaTek MT7621 PCI Controller.
> > > +
> > >  endif
> > > --
> > > 2.25.1

I have already sent to staging this patchset[0] regarding your
comments before moving this driver to 'drivers/pci/controller' and
sending v2:

[0]: https://lore.kernel.org/linux-staging/20210606161358.7114-1-sergio.paracuellos@gmail.com/T/#t

Thanks,
    Sergio Paracuellos

Re: [PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Bjorn Helgaas]

On Sat, May 15, 2021 at 02:40:53PM +0200, Sergio Paracuellos wrote:
> This patch adds a driver for the PCIe controller of MT7621 SoC.

If you put this in drivers/pci/, update the subject/commit log like
this:

  PCI: mt7621: Add MediaTek MT7621 PCIe host controller driver

  Add driver for the PCIe controller of the MT7621 SoC.

> Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> ---
>  arch/mips/pci/Makefile     |   1 +
>  arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
>  arch/mips/ralink/Kconfig   |   9 +-
>  3 files changed, 633 insertions(+), 1 deletion(-)
>  create mode 100644 arch/mips/pci/pci-mt7621.c

I'd suggest a single commit that basically does a "git mv" from
drivers/staging/mt7621-pci to drivers/pci/controller/, with zero
changes to the driver itself.

> diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
> index f3eecc065e5c..178c550739c4 100644
> --- a/arch/mips/pci/Makefile
> +++ b/arch/mips/pci/Makefile
> @@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)	+= pci-ar2315.o
>  obj-$(CONFIG_SOC_AR71XX)	+= pci-ar71xx.o
>  obj-$(CONFIG_PCI_AR724X)	+= pci-ar724x.o
>  obj-$(CONFIG_PCI_XTALK_BRIDGE)	+= pci-xtalk-bridge.o
> +obj-$(CONFIG_PCI_MT7621)	+= pci-mt7621.o
>  #
>  # These are still pretty much in the old state, watch, go blind.
>  #
> diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
> new file mode 100644
> index 000000000000..fe1945819d25
> --- /dev/null
> +++ b/arch/mips/pci/pci-mt7621.c
> @@ -0,0 +1,624 @@
> +// SPDX-License-Identifier: GPL-2.0+
> +/*
> + * BRIEF MODULE DESCRIPTION
> + *     PCI init for Ralink RT2880 solution
> + *
> + * Copyright 2007 Ralink Inc. (bruce_chang@ralinktech.com.tw)
> + *
> + * May 2007 Bruce Chang
> + * Initial Release
> + *
> + * May 2009 Bruce Chang
> + * support RT2880/RT3883 PCIe
> + *
> + * May 2011 Bruce Chang
> + * support RT6855/MT7620 PCIe
> + */
> +
> +#include <linux/bitops.h>
> +#include <linux/clk.h>
> +#include <linux/delay.h>
> +#include <linux/gpio/consumer.h>
> +#include <linux/module.h>
> +#include <linux/of.h>
> +#include <linux/of_address.h>
> +#include <linux/of_pci.h>
> +#include <linux/of_platform.h>
> +#include <linux/pci.h>
> +#include <linux/phy/phy.h>
> +#include <linux/platform_device.h>
> +#include <linux/reset.h>
> +#include <linux/sys_soc.h>
> +
> +/* MediaTek specific configuration registers */
> +#define PCIE_FTS_NUM			0x70c
> +#define PCIE_FTS_NUM_MASK		GENMASK(15, 8)
> +#define PCIE_FTS_NUM_L0(x)		(((x) & 0xff) << 8)
> +
> +/* Host-PCI bridge registers */
> +#define RALINK_PCI_PCICFG_ADDR		0x0000
> +#define RALINK_PCI_PCIMSK_ADDR		0x000C
> +#define RALINK_PCI_CONFIG_ADDR		0x0020
> +#define RALINK_PCI_CONFIG_DATA		0x0024
> +#define RALINK_PCI_MEMBASE		0x0028
> +#define RALINK_PCI_IOBASE		0x002C
> +
> +/* PCIe RC control registers */
> +#define MT7621_PCIE_OFFSET		0x2000
> +#define MT7621_NEXT_PORT		0x1000
> +
> +#define RALINK_PCI_BAR0SETUP_ADDR	0x0010
> +#define RALINK_PCI_ID			0x0030
> +#define RALINK_PCI_CLASS		0x0034
> +#define RALINK_PCI_SUBID		0x0038
> +#define RALINK_PCI_STATUS		0x0050
> +
> +/* Some definition values */
> +#define PCIE_REVISION_ID		BIT(0)
> +#define PCIE_CLASS_CODE			(0x60400 << 8)
> +#define PCIE_BAR_MAP_MAX		GENMASK(30, 16)
> +#define PCIE_BAR_ENABLE			BIT(0)
> +#define PCIE_PORT_INT_EN(x)		BIT(20 + (x))
> +#define PCIE_PORT_LINKUP		BIT(0)
> +
> +#define PERST_DELAY_MS			100
> +
> +/**
> + * struct mt7621_pcie_port - PCIe port information
> + * @base: I/O mapped register base
> + * @list: port list
> + * @pcie: pointer to PCIe host info
> + * @clk: pointer to the port clock gate
> + * @phy: pointer to PHY control block
> + * @pcie_rst: pointer to port reset control
> + * @gpio_rst: gpio reset
> + * @slot: port slot
> + * @enabled: indicates if port is enabled
> + */
> +struct mt7621_pcie_port {
> +	void __iomem *base;
> +	struct list_head list;
> +	struct mt7621_pcie *pcie;
> +	struct clk *clk;
> +	struct phy *phy;
> +	struct reset_control *pcie_rst;
> +	struct gpio_desc *gpio_rst;
> +	u32 slot;
> +	bool enabled;
> +};
> +
> +/**
> + * struct mt7621_pcie - PCIe host information
> + * @base: IO Mapped Register Base
> + * @io: IO resource
> + * @mem: pointer to non-prefetchable memory resource
> + * @dev: Pointer to PCIe device
> + * @io_map_base: virtual memory base address for io
> + * @ports: pointer to PCIe port information
> + * @resets_inverted: depends on chip revision
> + * reset lines are inverted.
> + */
> +struct mt7621_pcie {
> +	void __iomem *base;
> +	struct device *dev;
> +	struct resource io;
> +	struct resource *mem;
> +	unsigned long io_map_base;
> +	struct list_head ports;
> +	bool resets_inverted;
> +};
> +
> +static inline u32 pcie_read(struct mt7621_pcie *pcie, u32 reg)
> +{
> +	return readl(pcie->base + reg);
> +}
> +
> +static inline void pcie_write(struct mt7621_pcie *pcie, u32 val, u32 reg)
> +{
> +	writel(val, pcie->base + reg);
> +}
> +
> +static inline void pcie_rmw(struct mt7621_pcie *pcie, u32 reg, u32 clr, u32 set)
> +{
> +	u32 val = readl(pcie->base + reg);
> +
> +	val &= ~clr;
> +	val |= set;
> +	writel(val, pcie->base + reg);
> +}
> +
> +static inline u32 pcie_port_read(struct mt7621_pcie_port *port, u32 reg)
> +{
> +	return readl(port->base + reg);
> +}
> +
> +static inline void pcie_port_write(struct mt7621_pcie_port *port,
> +				   u32 val, u32 reg)
> +{
> +	writel(val, port->base + reg);
> +}
> +
> +static inline u32 mt7621_pci_get_cfgaddr(unsigned int bus, unsigned int slot,
> +					 unsigned int func, unsigned int where)
> +{
> +	return (((where & 0xF00) >> 8) << 24) | (bus << 16) | (slot << 11) |
> +		(func << 8) | (where & 0xfc) | 0x80000000;
> +}
> +
> +static void __iomem *mt7621_pcie_map_bus(struct pci_bus *bus,
> +					 unsigned int devfn, int where)
> +{
> +	struct mt7621_pcie *pcie = bus->sysdata;
> +	u32 address = mt7621_pci_get_cfgaddr(bus->number, PCI_SLOT(devfn),
> +					     PCI_FUNC(devfn), where);
> +
> +	writel(address, pcie->base + RALINK_PCI_CONFIG_ADDR);
> +
> +	return pcie->base + RALINK_PCI_CONFIG_DATA + (where & 3);
> +}
> +
> +struct pci_ops mt7621_pci_ops = {
> +	.map_bus	= mt7621_pcie_map_bus,
> +	.read		= pci_generic_config_read,
> +	.write		= pci_generic_config_write,
> +};
> +
> +static u32 read_config(struct mt7621_pcie *pcie, unsigned int dev, u32 reg)
> +{
> +	u32 address = mt7621_pci_get_cfgaddr(0, dev, 0, reg);
> +
> +	pcie_write(pcie, address, RALINK_PCI_CONFIG_ADDR);
> +	return pcie_read(pcie, RALINK_PCI_CONFIG_DATA);
> +}
> +
> +static void write_config(struct mt7621_pcie *pcie, unsigned int dev,
> +			 u32 reg, u32 val)
> +{
> +	u32 address = mt7621_pci_get_cfgaddr(0, dev, 0, reg);
> +
> +	pcie_write(pcie, address, RALINK_PCI_CONFIG_ADDR);
> +	pcie_write(pcie, val, RALINK_PCI_CONFIG_DATA);
> +}
> +
> +static inline void mt7621_rst_gpio_pcie_assert(struct mt7621_pcie_port *port)
> +{
> +	if (port->gpio_rst)
> +		gpiod_set_value(port->gpio_rst, 1);
> +}
> +
> +static inline void mt7621_rst_gpio_pcie_deassert(struct mt7621_pcie_port *port)
> +{
> +	if (port->gpio_rst)
> +		gpiod_set_value(port->gpio_rst, 0);
> +}
> +
> +static inline bool mt7621_pcie_port_is_linkup(struct mt7621_pcie_port *port)
> +{
> +	return (pcie_port_read(port, RALINK_PCI_STATUS) & PCIE_PORT_LINKUP) != 0;
> +}
> +
> +static inline void mt7621_control_assert(struct mt7621_pcie_port *port)
> +{
> +	struct mt7621_pcie *pcie = port->pcie;
> +
> +	if (pcie->resets_inverted)
> +		reset_control_assert(port->pcie_rst);
> +	else
> +		reset_control_deassert(port->pcie_rst);
> +}
> +
> +static inline void mt7621_control_deassert(struct mt7621_pcie_port *port)
> +{
> +	struct mt7621_pcie *pcie = port->pcie;
> +
> +	if (pcie->resets_inverted)
> +		reset_control_deassert(port->pcie_rst);
> +	else
> +		reset_control_assert(port->pcie_rst);
> +}
> +
> +static void setup_cm_memory_region(struct mt7621_pcie *pcie)
> +{
> +	struct resource *mem_resource = pcie->mem;
> +	struct device *dev = pcie->dev;
> +	resource_size_t mask;
> +
> +	if (mips_cps_numiocu(0)) {
> +		/*
> +		 * FIXME: hardware doesn't accept mask values with 1s after
> +		 * 0s (e.g. 0xffef), so it would be great to warn if that's
> +		 * about to happen
> +		 */
> +		mask = ~(mem_resource->end - mem_resource->start);
> +
> +		write_gcr_reg1_base(mem_resource->start);
> +		write_gcr_reg1_mask(mask | CM_GCR_REGn_MASK_CMTGT_IOCU0);
> +		dev_info(dev, "PCI coherence region base: 0x%08llx, mask/settings: 0x%08llx\n",
> +			 (unsigned long long)read_gcr_reg1_base(),
> +			 (unsigned long long)read_gcr_reg1_mask());
> +	}
> +}
> +
> +static int mt7621_pci_parse_request_of_pci_ranges(struct pci_host_bridge *host)
> +{
> +	struct mt7621_pcie *pcie = pci_host_bridge_priv(host);
> +	struct device *dev = pcie->dev;
> +	struct device_node *node = dev->of_node;
> +	struct of_pci_range_parser parser;
> +	struct resource_entry *entry;
> +	struct of_pci_range range;
> +	LIST_HEAD(res);
> +
> +	if (of_pci_range_parser_init(&parser, node)) {
> +		dev_err(dev, "missing \"ranges\" property\n");
> +		return -EINVAL;
> +	}
> +
> +	/*
> +	 * IO_SPACE_LIMIT for MIPS is 0xffff but this platform uses IO at
> +	 * upper address 0x001e160000. of_pci_range_to_resource does not work
> +	 * well for MIPS platforms that don't define PCI_IOBASE, so set the IO
> +	 * resource manually instead.
> +	 */
> +	for_each_of_pci_range(&parser, &range) {
> +		switch (range.flags & IORESOURCE_TYPE_BITS) {
> +		case IORESOURCE_IO:
> +			pcie->io_map_base =
> +				(unsigned long)ioremap(range.cpu_addr,
> +						       range.size);
> +			pcie->io.name = node->full_name;
> +			pcie->io.flags = range.flags;
> +			pcie->io.start = range.cpu_addr;
> +			pcie->io.end = range.cpu_addr + range.size - 1;
> +			pcie->io.parent = pcie->io.child = pcie->io.sibling = NULL;
> +			set_io_port_base(pcie->io_map_base);
> +			break;
> +		}
> +	}
> +
> +	entry = resource_list_first_type(&host->windows, IORESOURCE_MEM);
> +	if (!entry) {
> +		dev_err(dev, "Cannot get memory resource");
> +		return -EINVAL;
> +	}
> +
> +	pcie->mem = entry->res;
> +	pci_add_resource(&res, &pcie->io);
> +	pci_add_resource(&res, entry->res);
> +	list_splice_init(&res, &host->windows);
> +
> +	return 0;
> +}
> +
> +static int mt7621_pcie_parse_port(struct mt7621_pcie *pcie,
> +				  int slot)
> +{
> +	struct mt7621_pcie_port *port;
> +	struct device *dev = pcie->dev;
> +	struct platform_device *pdev = to_platform_device(dev);
> +	char name[10];
> +
> +	port = devm_kzalloc(dev, sizeof(*port), GFP_KERNEL);
> +	if (!port)
> +		return -ENOMEM;
> +
> +	port->base = devm_platform_ioremap_resource(pdev, slot + 1);
> +	if (IS_ERR(port->base))
> +		return PTR_ERR(port->base);
> +
> +	snprintf(name, sizeof(name), "pcie%d", slot);
> +	port->clk = devm_clk_get(dev, name);
> +	if (IS_ERR(port->clk)) {
> +		dev_err(dev, "failed to get pcie%d clock\n", slot);
> +		return PTR_ERR(port->clk);
> +	}
> +
> +	snprintf(name, sizeof(name), "pcie%d", slot);
> +	port->pcie_rst = devm_reset_control_get_exclusive(dev, name);
> +	if (PTR_ERR(port->pcie_rst) == -EPROBE_DEFER) {
> +		dev_err(dev, "failed to get pcie%d reset control\n", slot);
> +		return PTR_ERR(port->pcie_rst);
> +	}
> +
> +	snprintf(name, sizeof(name), "pcie-phy%d", slot);
> +	port->phy = devm_phy_get(dev, name);
> +	if (IS_ERR(port->phy) && slot != 1)
> +		return PTR_ERR(port->phy);
> +
> +	port->gpio_rst = devm_gpiod_get_index_optional(dev, "reset", slot,
> +						       GPIOD_OUT_LOW);
> +	if (IS_ERR(port->gpio_rst)) {
> +		dev_err(dev, "Failed to get GPIO for PCIe%d\n", slot);
> +		return PTR_ERR(port->gpio_rst);
> +	}
> +
> +	port->slot = slot;
> +	port->pcie = pcie;
> +
> +	INIT_LIST_HEAD(&port->list);
> +	list_add_tail(&port->list, &pcie->ports);
> +
> +	return 0;
> +}
> +
> +static int mt7621_pcie_parse_dt(struct mt7621_pcie *pcie)
> +{
> +	struct device *dev = pcie->dev;
> +	struct platform_device *pdev = to_platform_device(dev);
> +	struct device_node *node = dev->of_node, *child;
> +	int err;
> +
> +	pcie->base = devm_platform_ioremap_resource(pdev, 0);
> +	if (IS_ERR(pcie->base))
> +		return PTR_ERR(pcie->base);
> +
> +	for_each_available_child_of_node(node, child) {
> +		int slot;
> +
> +		err = of_pci_get_devfn(child);
> +		if (err < 0) {
> +			of_node_put(child);
> +			dev_err(dev, "failed to parse devfn: %d\n", err);
> +			return err;
> +		}
> +
> +		slot = PCI_SLOT(err);
> +
> +		err = mt7621_pcie_parse_port(pcie, slot);
> +		if (err) {
> +			of_node_put(child);
> +			return err;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static int mt7621_pcie_init_port(struct mt7621_pcie_port *port)
> +{
> +	struct mt7621_pcie *pcie = port->pcie;
> +	struct device *dev = pcie->dev;
> +	u32 slot = port->slot;
> +	int err;
> +
> +	err = phy_init(port->phy);
> +	if (err) {
> +		dev_err(dev, "failed to initialize port%d phy\n", slot);
> +		return err;
> +	}
> +
> +	err = phy_power_on(port->phy);
> +	if (err) {
> +		dev_err(dev, "failed to power on port%d phy\n", slot);
> +		phy_exit(port->phy);
> +		return err;
> +	}
> +
> +	port->enabled = true;
> +
> +	return 0;
> +}
> +
> +static void mt7621_pcie_reset_assert(struct mt7621_pcie *pcie)
> +{
> +	struct mt7621_pcie_port *port;
> +
> +	list_for_each_entry(port, &pcie->ports, list) {
> +		/* PCIe RC reset assert */
> +		mt7621_control_assert(port);
> +
> +		/* PCIe EP reset assert */
> +		mt7621_rst_gpio_pcie_assert(port);
> +	}
> +
> +	msleep(PERST_DELAY_MS);
> +}
> +
> +static void mt7621_pcie_reset_rc_deassert(struct mt7621_pcie *pcie)
> +{
> +	struct mt7621_pcie_port *port;
> +
> +	list_for_each_entry(port, &pcie->ports, list)
> +		mt7621_control_deassert(port);
> +}
> +
> +static void mt7621_pcie_reset_ep_deassert(struct mt7621_pcie *pcie)
> +{
> +	struct mt7621_pcie_port *port;
> +
> +	list_for_each_entry(port, &pcie->ports, list)
> +		mt7621_rst_gpio_pcie_deassert(port);
> +
> +	msleep(PERST_DELAY_MS);
> +}
> +
> +static void mt7621_pcie_init_ports(struct mt7621_pcie *pcie)
> +{
> +	struct device *dev = pcie->dev;
> +	struct mt7621_pcie_port *port, *tmp;
> +	int err;
> +
> +	mt7621_pcie_reset_assert(pcie);
> +	mt7621_pcie_reset_rc_deassert(pcie);
> +
> +	list_for_each_entry_safe(port, tmp, &pcie->ports, list) {
> +		u32 slot = port->slot;
> +
> +		if (slot == 1) {
> +			port->enabled = true;
> +			continue;
> +		}
> +
> +		err = mt7621_pcie_init_port(port);
> +		if (err) {
> +			dev_err(dev, "Initiating port %d failed\n", slot);
> +			list_del(&port->list);
> +		}
> +	}
> +
> +	mt7621_pcie_reset_ep_deassert(pcie);
> +
> +	tmp = NULL;
> +	list_for_each_entry(port, &pcie->ports, list) {
> +		u32 slot = port->slot;
> +
> +		if (!mt7621_pcie_port_is_linkup(port)) {
> +			dev_err(dev, "pcie%d no card, disable it (RST & CLK)\n",
> +				slot);
> +			mt7621_control_assert(port);
> +			clk_disable_unprepare(port->clk);
> +			port->enabled = false;
> +
> +			if (slot == 0) {
> +				tmp = port;
> +				continue;
> +			}
> +
> +			if (slot == 1 && tmp && !tmp->enabled)
> +				phy_power_off(tmp->phy);
> +		}
> +	}
> +}
> +
> +static void mt7621_pcie_enable_port(struct mt7621_pcie_port *port)
> +{
> +	struct mt7621_pcie *pcie = port->pcie;
> +	u32 slot = port->slot;
> +	u32 offset = MT7621_PCIE_OFFSET + (slot * MT7621_NEXT_PORT);
> +	u32 val;
> +
> +	/* enable pcie interrupt */
> +	val = pcie_read(pcie, RALINK_PCI_PCIMSK_ADDR);
> +	val |= PCIE_PORT_INT_EN(slot);
> +	pcie_write(pcie, val, RALINK_PCI_PCIMSK_ADDR);
> +
> +	/* map 2G DDR region */
> +	pcie_write(pcie, PCIE_BAR_MAP_MAX | PCIE_BAR_ENABLE,
> +		   offset + RALINK_PCI_BAR0SETUP_ADDR);
> +
> +	/* configure class code and revision ID */
> +	pcie_write(pcie, PCIE_CLASS_CODE | PCIE_REVISION_ID,
> +		   offset + RALINK_PCI_CLASS);
> +}
> +
> +static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> +{
> +	struct device *dev = pcie->dev;
> +	struct mt7621_pcie_port *port;
> +	u8 num_slots_enabled = 0;
> +	u32 slot;
> +	u32 val;
> +	int err;
> +
> +	/* Setup MEMWIN and IOWIN */
> +	pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> +	pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> +
> +	list_for_each_entry(port, &pcie->ports, list) {
> +		if (port->enabled) {
> +			err = clk_prepare_enable(port->clk);
> +			if (err) {
> +				dev_err(dev, "enabling clk pcie%d\n", slot);
> +				return err;
> +			}
> +
> +			mt7621_pcie_enable_port(port);
> +			dev_info(dev, "PCIE%d enabled\n", port->slot);
> +			num_slots_enabled++;
> +		}
> +	}
> +
> +	for (slot = 0; slot < num_slots_enabled; slot++) {
> +		val = read_config(pcie, slot, PCI_COMMAND);
> +		val |= PCI_COMMAND_MASTER;
> +		write_config(pcie, slot, PCI_COMMAND, val);
> +		/* configure RC FTS number to 250 when it leaves L0s */
> +		val = read_config(pcie, slot, PCIE_FTS_NUM);
> +		val &= ~PCIE_FTS_NUM_MASK;
> +		val |= PCIE_FTS_NUM_L0(0x50);
> +		write_config(pcie, slot, PCIE_FTS_NUM, val);
> +	}
> +
> +	return 0;
> +}
> +
> +static int mt7621_pcie_register_host(struct pci_host_bridge *host)
> +{
> +	struct mt7621_pcie *pcie = pci_host_bridge_priv(host);
> +
> +	host->ops = &mt7621_pci_ops;
> +	host->sysdata = pcie;
> +	return pci_host_probe(host);
> +}
> +
> +static const struct soc_device_attribute mt7621_pci_quirks_match[] = {
> +	{ .soc_id = "mt7621", .revision = "E2" }
> +};
> +
> +static int mt7621_pci_probe(struct platform_device *pdev)
> +{
> +	struct device *dev = &pdev->dev;
> +	const struct soc_device_attribute *attr;
> +	struct mt7621_pcie *pcie;
> +	struct pci_host_bridge *bridge;
> +	int err;
> +
> +	if (!dev->of_node)
> +		return -ENODEV;
> +
> +	bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
> +	if (!bridge)
> +		return -ENOMEM;
> +
> +	pcie = pci_host_bridge_priv(bridge);
> +	pcie->dev = dev;
> +	platform_set_drvdata(pdev, pcie);
> +	INIT_LIST_HEAD(&pcie->ports);
> +
> +	attr = soc_device_match(mt7621_pci_quirks_match);
> +	if (attr)
> +		pcie->resets_inverted = true;
> +
> +	err = mt7621_pcie_parse_dt(pcie);
> +	if (err) {
> +		dev_err(dev, "Parsing DT failed\n");
> +		return err;
> +	}
> +
> +	err = mt7621_pci_parse_request_of_pci_ranges(bridge);
> +	if (err) {
> +		dev_err(dev, "Error requesting pci resources from ranges");
> +		return err;
> +	}
> +
> +	/* set resources limits */
> +	ioport_resource.start = pcie->io.start;
> +	ioport_resource.end = pcie->io.end;
> +
> +	mt7621_pcie_init_ports(pcie);
> +
> +	err = mt7621_pcie_enable_ports(pcie);
> +	if (err) {
> +		dev_err(dev, "Error enabling pcie ports\n");
> +		return err;
> +	}
> +
> +	setup_cm_memory_region(pcie);
> +
> +	return mt7621_pcie_register_host(bridge);
> +}
> +
> +static const struct of_device_id mt7621_pci_ids[] = {
> +	{ .compatible = "mediatek,mt7621-pci" },
> +	{},
> +};
> +MODULE_DEVICE_TABLE(of, mt7621_pci_ids);
> +
> +static struct platform_driver mt7621_pci_driver = {
> +	.probe = mt7621_pci_probe,
> +	.driver = {
> +		.name = "mt7621-pci",
> +		.of_match_table = of_match_ptr(mt7621_pci_ids),
> +	},
> +};
> +builtin_platform_driver(mt7621_pci_driver);
> diff --git a/arch/mips/ralink/Kconfig b/arch/mips/ralink/Kconfig
> index ec4daa63c5e3..50e5a54f7d9e 100644
> --- a/arch/mips/ralink/Kconfig
> +++ b/arch/mips/ralink/Kconfig
> @@ -56,7 +56,7 @@ choice
>  		select MIPS_GIC
>  		select COMMON_CLK
>  		select CLKSRC_MIPS_GIC
> -		select HAVE_PCI if PCI_MT7621
> +		select HAVE_PCI
>  		select SOC_BUS
>  endchoice
>  
> @@ -101,4 +101,11 @@ choice
>  
>  endchoice
>  
> +config PCI_MT7621
> +	bool "MediaTek MT7621 PCI Controller"
> +	depends on RALINK && SOC_MT7621
> +	select PCI_DRIVERS_GENERIC
> +	help
> +	  This selects a driver for the MediaTek MT7621 PCI Controller.
> +
>  endif
> -- 
> 2.25.1
> 

Re: [PATCH 2/4] MIPS: pci: Add driver for MT7621 PCIe controller

[Sergio Paracuellos]

Hi Bjorn,

On Fri, Jun 4, 2021 at 9:49 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Sat, May 15, 2021 at 02:40:53PM +0200, Sergio Paracuellos wrote:
> > This patch adds a driver for the PCIe controller of MT7621 SoC.
>
> If you put this in drivers/pci/, update the subject/commit log like
> this:
>
>   PCI: mt7621: Add MediaTek MT7621 PCIe host controller driver
>
>   Add driver for the PCIe controller of the MT7621 SoC.

Will do, thanks.

>
> > Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> > ---
> >  arch/mips/pci/Makefile     |   1 +
> >  arch/mips/pci/pci-mt7621.c | 624 +++++++++++++++++++++++++++++++++++++
> >  arch/mips/ralink/Kconfig   |   9 +-
> >  3 files changed, 633 insertions(+), 1 deletion(-)
> >  create mode 100644 arch/mips/pci/pci-mt7621.c
>
> I'd suggest a single commit that basically does a "git mv" from
> drivers/staging/mt7621-pci to drivers/pci/controller/, with zero
> changes to the driver itself.

Ok, I will address all the suggested changes in staging and then move
the driver using 'git mv' instead of doing an add and a later remove.

Best regards,
    Sergio Paracuellos

>
> > diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
> > index f3eecc065e5c..178c550739c4 100644
> > --- a/arch/mips/pci/Makefile
> > +++ b/arch/mips/pci/Makefile
> > @@ -24,6 +24,7 @@ obj-$(CONFIG_PCI_AR2315)    += pci-ar2315.o
> >  obj-$(CONFIG_SOC_AR71XX)     += pci-ar71xx.o
> >  obj-$(CONFIG_PCI_AR724X)     += pci-ar724x.o
> >  obj-$(CONFIG_PCI_XTALK_BRIDGE)       += pci-xtalk-bridge.o
> > +obj-$(CONFIG_PCI_MT7621)     += pci-mt7621.o
> >  #
> >  # These are still pretty much in the old state, watch, go blind.
> >  #
> > diff --git a/arch/mips/pci/pci-mt7621.c b/arch/mips/pci/pci-mt7621.c
> > new file mode 100644
> > index 000000000000..fe1945819d25
> > --- /dev/null
> > +++ b/arch/mips/pci/pci-mt7621.c
> > @@ -0,0 +1,624 @@
> > +// SPDX-License-Identifier: GPL-2.0+
> > +/*
> > + * BRIEF MODULE DESCRIPTION
> > + *     PCI init for Ralink RT2880 solution
> > + *
> > + * Copyright 2007 Ralink Inc. (bruce_chang@ralinktech.com.tw)
> > + *
> > + * May 2007 Bruce Chang
> > + * Initial Release
> > + *
> > + * May 2009 Bruce Chang
> > + * support RT2880/RT3883 PCIe
> > + *
> > + * May 2011 Bruce Chang
> > + * support RT6855/MT7620 PCIe
> > + */
> > +
> > +#include <linux/bitops.h>
> > +#include <linux/clk.h>
> > +#include <linux/delay.h>
> > +#include <linux/gpio/consumer.h>
> > +#include <linux/module.h>
> > +#include <linux/of.h>
> > +#include <linux/of_address.h>
> > +#include <linux/of_pci.h>
> > +#include <linux/of_platform.h>
> > +#include <linux/pci.h>
> > +#include <linux/phy/phy.h>
> > +#include <linux/platform_device.h>
> > +#include <linux/reset.h>
> > +#include <linux/sys_soc.h>
> > +
> > +/* MediaTek specific configuration registers */
> > +#define PCIE_FTS_NUM                 0x70c
> > +#define PCIE_FTS_NUM_MASK            GENMASK(15, 8)
> > +#define PCIE_FTS_NUM_L0(x)           (((x) & 0xff) << 8)
> > +
> > +/* Host-PCI bridge registers */
> > +#define RALINK_PCI_PCICFG_ADDR               0x0000
> > +#define RALINK_PCI_PCIMSK_ADDR               0x000C
> > +#define RALINK_PCI_CONFIG_ADDR               0x0020
> > +#define RALINK_PCI_CONFIG_DATA               0x0024
> > +#define RALINK_PCI_MEMBASE           0x0028
> > +#define RALINK_PCI_IOBASE            0x002C
> > +
> > +/* PCIe RC control registers */
> > +#define MT7621_PCIE_OFFSET           0x2000
> > +#define MT7621_NEXT_PORT             0x1000
> > +
> > +#define RALINK_PCI_BAR0SETUP_ADDR    0x0010
> > +#define RALINK_PCI_ID                        0x0030
> > +#define RALINK_PCI_CLASS             0x0034
> > +#define RALINK_PCI_SUBID             0x0038
> > +#define RALINK_PCI_STATUS            0x0050
> > +
> > +/* Some definition values */
> > +#define PCIE_REVISION_ID             BIT(0)
> > +#define PCIE_CLASS_CODE                      (0x60400 << 8)
> > +#define PCIE_BAR_MAP_MAX             GENMASK(30, 16)
> > +#define PCIE_BAR_ENABLE                      BIT(0)
> > +#define PCIE_PORT_INT_EN(x)          BIT(20 + (x))
> > +#define PCIE_PORT_LINKUP             BIT(0)
> > +
> > +#define PERST_DELAY_MS                       100
> > +
> > +/**
> > + * struct mt7621_pcie_port - PCIe port information
> > + * @base: I/O mapped register base
> > + * @list: port list
> > + * @pcie: pointer to PCIe host info
> > + * @clk: pointer to the port clock gate
> > + * @phy: pointer to PHY control block
> > + * @pcie_rst: pointer to port reset control
> > + * @gpio_rst: gpio reset
> > + * @slot: port slot
> > + * @enabled: indicates if port is enabled
> > + */
> > +struct mt7621_pcie_port {
> > +     void __iomem *base;
> > +     struct list_head list;
> > +     struct mt7621_pcie *pcie;
> > +     struct clk *clk;
> > +     struct phy *phy;
> > +     struct reset_control *pcie_rst;
> > +     struct gpio_desc *gpio_rst;
> > +     u32 slot;
> > +     bool enabled;
> > +};
> > +
> > +/**
> > + * struct mt7621_pcie - PCIe host information
> > + * @base: IO Mapped Register Base
> > + * @io: IO resource
> > + * @mem: pointer to non-prefetchable memory resource
> > + * @dev: Pointer to PCIe device
> > + * @io_map_base: virtual memory base address for io
> > + * @ports: pointer to PCIe port information
> > + * @resets_inverted: depends on chip revision
> > + * reset lines are inverted.
> > + */
> > +struct mt7621_pcie {
> > +     void __iomem *base;
> > +     struct device *dev;
> > +     struct resource io;
> > +     struct resource *mem;
> > +     unsigned long io_map_base;
> > +     struct list_head ports;
> > +     bool resets_inverted;
> > +};
> > +
> > +static inline u32 pcie_read(struct mt7621_pcie *pcie, u32 reg)
> > +{
> > +     return readl(pcie->base + reg);
> > +}
> > +
> > +static inline void pcie_write(struct mt7621_pcie *pcie, u32 val, u32 reg)
> > +{
> > +     writel(val, pcie->base + reg);
> > +}
> > +
> > +static inline void pcie_rmw(struct mt7621_pcie *pcie, u32 reg, u32 clr, u32 set)
> > +{
> > +     u32 val = readl(pcie->base + reg);
> > +
> > +     val &= ~clr;
> > +     val |= set;
> > +     writel(val, pcie->base + reg);
> > +}
> > +
> > +static inline u32 pcie_port_read(struct mt7621_pcie_port *port, u32 reg)
> > +{
> > +     return readl(port->base + reg);
> > +}
> > +
> > +static inline void pcie_port_write(struct mt7621_pcie_port *port,
> > +                                u32 val, u32 reg)
> > +{
> > +     writel(val, port->base + reg);
> > +}
> > +
> > +static inline u32 mt7621_pci_get_cfgaddr(unsigned int bus, unsigned int slot,
> > +                                      unsigned int func, unsigned int where)
> > +{
> > +     return (((where & 0xF00) >> 8) << 24) | (bus << 16) | (slot << 11) |
> > +             (func << 8) | (where & 0xfc) | 0x80000000;
> > +}
> > +
> > +static void __iomem *mt7621_pcie_map_bus(struct pci_bus *bus,
> > +                                      unsigned int devfn, int where)
> > +{
> > +     struct mt7621_pcie *pcie = bus->sysdata;
> > +     u32 address = mt7621_pci_get_cfgaddr(bus->number, PCI_SLOT(devfn),
> > +                                          PCI_FUNC(devfn), where);
> > +
> > +     writel(address, pcie->base + RALINK_PCI_CONFIG_ADDR);
> > +
> > +     return pcie->base + RALINK_PCI_CONFIG_DATA + (where & 3);
> > +}
> > +
> > +struct pci_ops mt7621_pci_ops = {
> > +     .map_bus        = mt7621_pcie_map_bus,
> > +     .read           = pci_generic_config_read,
> > +     .write          = pci_generic_config_write,
> > +};
> > +
> > +static u32 read_config(struct mt7621_pcie *pcie, unsigned int dev, u32 reg)
> > +{
> > +     u32 address = mt7621_pci_get_cfgaddr(0, dev, 0, reg);
> > +
> > +     pcie_write(pcie, address, RALINK_PCI_CONFIG_ADDR);
> > +     return pcie_read(pcie, RALINK_PCI_CONFIG_DATA);
> > +}
> > +
> > +static void write_config(struct mt7621_pcie *pcie, unsigned int dev,
> > +                      u32 reg, u32 val)
> > +{
> > +     u32 address = mt7621_pci_get_cfgaddr(0, dev, 0, reg);
> > +
> > +     pcie_write(pcie, address, RALINK_PCI_CONFIG_ADDR);
> > +     pcie_write(pcie, val, RALINK_PCI_CONFIG_DATA);
> > +}
> > +
> > +static inline void mt7621_rst_gpio_pcie_assert(struct mt7621_pcie_port *port)
> > +{
> > +     if (port->gpio_rst)
> > +             gpiod_set_value(port->gpio_rst, 1);
> > +}
> > +
> > +static inline void mt7621_rst_gpio_pcie_deassert(struct mt7621_pcie_port *port)
> > +{
> > +     if (port->gpio_rst)
> > +             gpiod_set_value(port->gpio_rst, 0);
> > +}
> > +
> > +static inline bool mt7621_pcie_port_is_linkup(struct mt7621_pcie_port *port)
> > +{
> > +     return (pcie_port_read(port, RALINK_PCI_STATUS) & PCIE_PORT_LINKUP) != 0;
> > +}
> > +
> > +static inline void mt7621_control_assert(struct mt7621_pcie_port *port)
> > +{
> > +     struct mt7621_pcie *pcie = port->pcie;
> > +
> > +     if (pcie->resets_inverted)
> > +             reset_control_assert(port->pcie_rst);
> > +     else
> > +             reset_control_deassert(port->pcie_rst);
> > +}
> > +
> > +static inline void mt7621_control_deassert(struct mt7621_pcie_port *port)
> > +{
> > +     struct mt7621_pcie *pcie = port->pcie;
> > +
> > +     if (pcie->resets_inverted)
> > +             reset_control_deassert(port->pcie_rst);
> > +     else
> > +             reset_control_assert(port->pcie_rst);
> > +}
> > +
> > +static void setup_cm_memory_region(struct mt7621_pcie *pcie)
> > +{
> > +     struct resource *mem_resource = pcie->mem;
> > +     struct device *dev = pcie->dev;
> > +     resource_size_t mask;
> > +
> > +     if (mips_cps_numiocu(0)) {
> > +             /*
> > +              * FIXME: hardware doesn't accept mask values with 1s after
> > +              * 0s (e.g. 0xffef), so it would be great to warn if that's
> > +              * about to happen
> > +              */
> > +             mask = ~(mem_resource->end - mem_resource->start);
> > +
> > +             write_gcr_reg1_base(mem_resource->start);
> > +             write_gcr_reg1_mask(mask | CM_GCR_REGn_MASK_CMTGT_IOCU0);
> > +             dev_info(dev, "PCI coherence region base: 0x%08llx, mask/settings: 0x%08llx\n",
> > +                      (unsigned long long)read_gcr_reg1_base(),
> > +                      (unsigned long long)read_gcr_reg1_mask());
> > +     }
> > +}
> > +
> > +static int mt7621_pci_parse_request_of_pci_ranges(struct pci_host_bridge *host)
> > +{
> > +     struct mt7621_pcie *pcie = pci_host_bridge_priv(host);
> > +     struct device *dev = pcie->dev;
> > +     struct device_node *node = dev->of_node;
> > +     struct of_pci_range_parser parser;
> > +     struct resource_entry *entry;
> > +     struct of_pci_range range;
> > +     LIST_HEAD(res);
> > +
> > +     if (of_pci_range_parser_init(&parser, node)) {
> > +             dev_err(dev, "missing \"ranges\" property\n");
> > +             return -EINVAL;
> > +     }
> > +
> > +     /*
> > +      * IO_SPACE_LIMIT for MIPS is 0xffff but this platform uses IO at
> > +      * upper address 0x001e160000. of_pci_range_to_resource does not work
> > +      * well for MIPS platforms that don't define PCI_IOBASE, so set the IO
> > +      * resource manually instead.
> > +      */
> > +     for_each_of_pci_range(&parser, &range) {
> > +             switch (range.flags & IORESOURCE_TYPE_BITS) {
> > +             case IORESOURCE_IO:
> > +                     pcie->io_map_base =
> > +                             (unsigned long)ioremap(range.cpu_addr,
> > +                                                    range.size);
> > +                     pcie->io.name = node->full_name;
> > +                     pcie->io.flags = range.flags;
> > +                     pcie->io.start = range.cpu_addr;
> > +                     pcie->io.end = range.cpu_addr + range.size - 1;
> > +                     pcie->io.parent = pcie->io.child = pcie->io.sibling = NULL;
> > +                     set_io_port_base(pcie->io_map_base);
> > +                     break;
> > +             }
> > +     }
> > +
> > +     entry = resource_list_first_type(&host->windows, IORESOURCE_MEM);
> > +     if (!entry) {
> > +             dev_err(dev, "Cannot get memory resource");
> > +             return -EINVAL;
> > +     }
> > +
> > +     pcie->mem = entry->res;
> > +     pci_add_resource(&res, &pcie->io);
> > +     pci_add_resource(&res, entry->res);
> > +     list_splice_init(&res, &host->windows);
> > +
> > +     return 0;
> > +}
> > +
> > +static int mt7621_pcie_parse_port(struct mt7621_pcie *pcie,
> > +                               int slot)
> > +{
> > +     struct mt7621_pcie_port *port;
> > +     struct device *dev = pcie->dev;
> > +     struct platform_device *pdev = to_platform_device(dev);
> > +     char name[10];
> > +
> > +     port = devm_kzalloc(dev, sizeof(*port), GFP_KERNEL);
> > +     if (!port)
> > +             return -ENOMEM;
> > +
> > +     port->base = devm_platform_ioremap_resource(pdev, slot + 1);
> > +     if (IS_ERR(port->base))
> > +             return PTR_ERR(port->base);
> > +
> > +     snprintf(name, sizeof(name), "pcie%d", slot);
> > +     port->clk = devm_clk_get(dev, name);
> > +     if (IS_ERR(port->clk)) {
> > +             dev_err(dev, "failed to get pcie%d clock\n", slot);
> > +             return PTR_ERR(port->clk);
> > +     }
> > +
> > +     snprintf(name, sizeof(name), "pcie%d", slot);
> > +     port->pcie_rst = devm_reset_control_get_exclusive(dev, name);
> > +     if (PTR_ERR(port->pcie_rst) == -EPROBE_DEFER) {
> > +             dev_err(dev, "failed to get pcie%d reset control\n", slot);
> > +             return PTR_ERR(port->pcie_rst);
> > +     }
> > +
> > +     snprintf(name, sizeof(name), "pcie-phy%d", slot);
> > +     port->phy = devm_phy_get(dev, name);
> > +     if (IS_ERR(port->phy) && slot != 1)
> > +             return PTR_ERR(port->phy);
> > +
> > +     port->gpio_rst = devm_gpiod_get_index_optional(dev, "reset", slot,
> > +                                                    GPIOD_OUT_LOW);
> > +     if (IS_ERR(port->gpio_rst)) {
> > +             dev_err(dev, "Failed to get GPIO for PCIe%d\n", slot);
> > +             return PTR_ERR(port->gpio_rst);
> > +     }
> > +
> > +     port->slot = slot;
> > +     port->pcie = pcie;
> > +
> > +     INIT_LIST_HEAD(&port->list);
> > +     list_add_tail(&port->list, &pcie->ports);
> > +
> > +     return 0;
> > +}
> > +
> > +static int mt7621_pcie_parse_dt(struct mt7621_pcie *pcie)
> > +{
> > +     struct device *dev = pcie->dev;
> > +     struct platform_device *pdev = to_platform_device(dev);
> > +     struct device_node *node = dev->of_node, *child;
> > +     int err;
> > +
> > +     pcie->base = devm_platform_ioremap_resource(pdev, 0);
> > +     if (IS_ERR(pcie->base))
> > +             return PTR_ERR(pcie->base);
> > +
> > +     for_each_available_child_of_node(node, child) {
> > +             int slot;
> > +
> > +             err = of_pci_get_devfn(child);
> > +             if (err < 0) {
> > +                     of_node_put(child);
> > +                     dev_err(dev, "failed to parse devfn: %d\n", err);
> > +                     return err;
> > +             }
> > +
> > +             slot = PCI_SLOT(err);
> > +
> > +             err = mt7621_pcie_parse_port(pcie, slot);
> > +             if (err) {
> > +                     of_node_put(child);
> > +                     return err;
> > +             }
> > +     }
> > +
> > +     return 0;
> > +}
> > +
> > +static int mt7621_pcie_init_port(struct mt7621_pcie_port *port)
> > +{
> > +     struct mt7621_pcie *pcie = port->pcie;
> > +     struct device *dev = pcie->dev;
> > +     u32 slot = port->slot;
> > +     int err;
> > +
> > +     err = phy_init(port->phy);
> > +     if (err) {
> > +             dev_err(dev, "failed to initialize port%d phy\n", slot);
> > +             return err;
> > +     }
> > +
> > +     err = phy_power_on(port->phy);
> > +     if (err) {
> > +             dev_err(dev, "failed to power on port%d phy\n", slot);
> > +             phy_exit(port->phy);
> > +             return err;
> > +     }
> > +
> > +     port->enabled = true;
> > +
> > +     return 0;
> > +}
> > +
> > +static void mt7621_pcie_reset_assert(struct mt7621_pcie *pcie)
> > +{
> > +     struct mt7621_pcie_port *port;
> > +
> > +     list_for_each_entry(port, &pcie->ports, list) {
> > +             /* PCIe RC reset assert */
> > +             mt7621_control_assert(port);
> > +
> > +             /* PCIe EP reset assert */
> > +             mt7621_rst_gpio_pcie_assert(port);
> > +     }
> > +
> > +     msleep(PERST_DELAY_MS);
> > +}
> > +
> > +static void mt7621_pcie_reset_rc_deassert(struct mt7621_pcie *pcie)
> > +{
> > +     struct mt7621_pcie_port *port;
> > +
> > +     list_for_each_entry(port, &pcie->ports, list)
> > +             mt7621_control_deassert(port);
> > +}
> > +
> > +static void mt7621_pcie_reset_ep_deassert(struct mt7621_pcie *pcie)
> > +{
> > +     struct mt7621_pcie_port *port;
> > +
> > +     list_for_each_entry(port, &pcie->ports, list)
> > +             mt7621_rst_gpio_pcie_deassert(port);
> > +
> > +     msleep(PERST_DELAY_MS);
> > +}
> > +
> > +static void mt7621_pcie_init_ports(struct mt7621_pcie *pcie)
> > +{
> > +     struct device *dev = pcie->dev;
> > +     struct mt7621_pcie_port *port, *tmp;
> > +     int err;
> > +
> > +     mt7621_pcie_reset_assert(pcie);
> > +     mt7621_pcie_reset_rc_deassert(pcie);
> > +
> > +     list_for_each_entry_safe(port, tmp, &pcie->ports, list) {
> > +             u32 slot = port->slot;
> > +
> > +             if (slot == 1) {
> > +                     port->enabled = true;
> > +                     continue;
> > +             }
> > +
> > +             err = mt7621_pcie_init_port(port);
> > +             if (err) {
> > +                     dev_err(dev, "Initiating port %d failed\n", slot);
> > +                     list_del(&port->list);
> > +             }
> > +     }
> > +
> > +     mt7621_pcie_reset_ep_deassert(pcie);
> > +
> > +     tmp = NULL;
> > +     list_for_each_entry(port, &pcie->ports, list) {
> > +             u32 slot = port->slot;
> > +
> > +             if (!mt7621_pcie_port_is_linkup(port)) {
> > +                     dev_err(dev, "pcie%d no card, disable it (RST & CLK)\n",
> > +                             slot);
> > +                     mt7621_control_assert(port);
> > +                     clk_disable_unprepare(port->clk);
> > +                     port->enabled = false;
> > +
> > +                     if (slot == 0) {
> > +                             tmp = port;
> > +                             continue;
> > +                     }
> > +
> > +                     if (slot == 1 && tmp && !tmp->enabled)
> > +                             phy_power_off(tmp->phy);
> > +             }
> > +     }
> > +}
> > +
> > +static void mt7621_pcie_enable_port(struct mt7621_pcie_port *port)
> > +{
> > +     struct mt7621_pcie *pcie = port->pcie;
> > +     u32 slot = port->slot;
> > +     u32 offset = MT7621_PCIE_OFFSET + (slot * MT7621_NEXT_PORT);
> > +     u32 val;
> > +
> > +     /* enable pcie interrupt */
> > +     val = pcie_read(pcie, RALINK_PCI_PCIMSK_ADDR);
> > +     val |= PCIE_PORT_INT_EN(slot);
> > +     pcie_write(pcie, val, RALINK_PCI_PCIMSK_ADDR);
> > +
> > +     /* map 2G DDR region */
> > +     pcie_write(pcie, PCIE_BAR_MAP_MAX | PCIE_BAR_ENABLE,
> > +                offset + RALINK_PCI_BAR0SETUP_ADDR);
> > +
> > +     /* configure class code and revision ID */
> > +     pcie_write(pcie, PCIE_CLASS_CODE | PCIE_REVISION_ID,
> > +                offset + RALINK_PCI_CLASS);
> > +}
> > +
> > +static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
> > +{
> > +     struct device *dev = pcie->dev;
> > +     struct mt7621_pcie_port *port;
> > +     u8 num_slots_enabled = 0;
> > +     u32 slot;
> > +     u32 val;
> > +     int err;
> > +
> > +     /* Setup MEMWIN and IOWIN */
> > +     pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
> > +     pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
> > +
> > +     list_for_each_entry(port, &pcie->ports, list) {
> > +             if (port->enabled) {
> > +                     err = clk_prepare_enable(port->clk);
> > +                     if (err) {
> > +                             dev_err(dev, "enabling clk pcie%d\n", slot);
> > +                             return err;
> > +                     }
> > +
> > +                     mt7621_pcie_enable_port(port);
> > +                     dev_info(dev, "PCIE%d enabled\n", port->slot);
> > +                     num_slots_enabled++;
> > +             }
> > +     }
> > +
> > +     for (slot = 0; slot < num_slots_enabled; slot++) {
> > +             val = read_config(pcie, slot, PCI_COMMAND);
> > +             val |= PCI_COMMAND_MASTER;
> > +             write_config(pcie, slot, PCI_COMMAND, val);
> > +             /* configure RC FTS number to 250 when it leaves L0s */
> > +             val = read_config(pcie, slot, PCIE_FTS_NUM);
> > +             val &= ~PCIE_FTS_NUM_MASK;
> > +             val |= PCIE_FTS_NUM_L0(0x50);
> > +             write_config(pcie, slot, PCIE_FTS_NUM, val);
> > +     }
> > +
> > +     return 0;
> > +}
> > +
> > +static int mt7621_pcie_register_host(struct pci_host_bridge *host)
> > +{
> > +     struct mt7621_pcie *pcie = pci_host_bridge_priv(host);
> > +
> > +     host->ops = &mt7621_pci_ops;
> > +     host->sysdata = pcie;
> > +     return pci_host_probe(host);
> > +}
> > +
> > +static const struct soc_device_attribute mt7621_pci_quirks_match[] = {
> > +     { .soc_id = "mt7621", .revision = "E2" }
> > +};
> > +
> > +static int mt7621_pci_probe(struct platform_device *pdev)
> > +{
> > +     struct device *dev = &pdev->dev;
> > +     const struct soc_device_attribute *attr;
> > +     struct mt7621_pcie *pcie;
> > +     struct pci_host_bridge *bridge;
> > +     int err;
> > +
> > +     if (!dev->of_node)
> > +             return -ENODEV;
> > +
> > +     bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
> > +     if (!bridge)
> > +             return -ENOMEM;
> > +
> > +     pcie = pci_host_bridge_priv(bridge);
> > +     pcie->dev = dev;
> > +     platform_set_drvdata(pdev, pcie);
> > +     INIT_LIST_HEAD(&pcie->ports);
> > +
> > +     attr = soc_device_match(mt7621_pci_quirks_match);
> > +     if (attr)
> > +             pcie->resets_inverted = true;
> > +
> > +     err = mt7621_pcie_parse_dt(pcie);
> > +     if (err) {
> > +             dev_err(dev, "Parsing DT failed\n");
> > +             return err;
> > +     }
> > +
> > +     err = mt7621_pci_parse_request_of_pci_ranges(bridge);
> > +     if (err) {
> > +             dev_err(dev, "Error requesting pci resources from ranges");
> > +             return err;
> > +     }
> > +
> > +     /* set resources limits */
> > +     ioport_resource.start = pcie->io.start;
> > +     ioport_resource.end = pcie->io.end;
> > +
> > +     mt7621_pcie_init_ports(pcie);
> > +
> > +     err = mt7621_pcie_enable_ports(pcie);
> > +     if (err) {
> > +             dev_err(dev, "Error enabling pcie ports\n");
> > +             return err;
> > +     }
> > +
> > +     setup_cm_memory_region(pcie);
> > +
> > +     return mt7621_pcie_register_host(bridge);
> > +}
> > +
> > +static const struct of_device_id mt7621_pci_ids[] = {
> > +     { .compatible = "mediatek,mt7621-pci" },
> > +     {},
> > +};
> > +MODULE_DEVICE_TABLE(of, mt7621_pci_ids);
> > +
> > +static struct platform_driver mt7621_pci_driver = {
> > +     .probe = mt7621_pci_probe,
> > +     .driver = {
> > +             .name = "mt7621-pci",
> > +             .of_match_table = of_match_ptr(mt7621_pci_ids),
> > +     },
> > +};
> > +builtin_platform_driver(mt7621_pci_driver);
> > diff --git a/arch/mips/ralink/Kconfig b/arch/mips/ralink/Kconfig
> > index ec4daa63c5e3..50e5a54f7d9e 100644
> > --- a/arch/mips/ralink/Kconfig
> > +++ b/arch/mips/ralink/Kconfig
> > @@ -56,7 +56,7 @@ choice
> >               select MIPS_GIC
> >               select COMMON_CLK
> >               select CLKSRC_MIPS_GIC
> > -             select HAVE_PCI if PCI_MT7621
> > +             select HAVE_PCI
> >               select SOC_BUS
> >  endchoice
> >
> > @@ -101,4 +101,11 @@ choice
> >
> >  endchoice
> >
> > +config PCI_MT7621
> > +     bool "MediaTek MT7621 PCI Controller"
> > +     depends on RALINK && SOC_MT7621
> > +     select PCI_DRIVERS_GENERIC
> > +     help
> > +       This selects a driver for the MediaTek MT7621 PCI Controller.
> > +
> >  endif
> > --
> > 2.25.1
> >

[PATCH 3/4] staging: mt7621-pci: remove driver from staging

[Sergio Paracuellos]

Driver has been moved into its proper place in the
kernel 'arch/mips/pci'. Hence, remove it from staging.

Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
---
 drivers/staging/Kconfig                       |   2 -
 drivers/staging/Makefile                      |   1 -
 drivers/staging/mt7621-pci/Kconfig            |   8 -
 drivers/staging/mt7621-pci/Makefile           |   2 -
 drivers/staging/mt7621-pci/TODO               |   4 -
 .../mt7621-pci/mediatek,mt7621-pci.txt        | 104 ---
 drivers/staging/mt7621-pci/pci-mt7621.c       | 624 ------------------
 7 files changed, 745 deletions(-)
 delete mode 100644 drivers/staging/mt7621-pci/Kconfig
 delete mode 100644 drivers/staging/mt7621-pci/Makefile
 delete mode 100644 drivers/staging/mt7621-pci/TODO
 delete mode 100644 drivers/staging/mt7621-pci/mediatek,mt7621-pci.txt
 delete mode 100644 drivers/staging/mt7621-pci/pci-mt7621.c

diff --git a/drivers/staging/Kconfig b/drivers/staging/Kconfig
index b7ae5bdc4eb5..9a21d730ab2b 100644
--- a/drivers/staging/Kconfig
+++ b/drivers/staging/Kconfig
@@ -86,8 +86,6 @@ source "drivers/staging/vc04_services/Kconfig"
 
 source "drivers/staging/pi433/Kconfig"
 
-source "drivers/staging/mt7621-pci/Kconfig"
-
 source "drivers/staging/mt7621-dma/Kconfig"
 
 source "drivers/staging/ralink-gdma/Kconfig"
diff --git a/drivers/staging/Makefile b/drivers/staging/Makefile
index 075c979bfe7c..b7b4916761d4 100644
--- a/drivers/staging/Makefile
+++ b/drivers/staging/Makefile
@@ -33,7 +33,6 @@ obj-$(CONFIG_KS7010)		+= ks7010/
 obj-$(CONFIG_GREYBUS)		+= greybus/
 obj-$(CONFIG_BCM2835_VCHIQ)	+= vc04_services/
 obj-$(CONFIG_PI433)		+= pi433/
-obj-$(CONFIG_PCI_MT7621)	+= mt7621-pci/
 obj-$(CONFIG_SOC_MT7621)	+= mt7621-dma/
 obj-$(CONFIG_DMA_RALINK)	+= ralink-gdma/
 obj-$(CONFIG_SOC_MT7621)	+= mt7621-dts/
diff --git a/drivers/staging/mt7621-pci/Kconfig b/drivers/staging/mt7621-pci/Kconfig
deleted file mode 100644
index ce58042f2f21..000000000000
--- a/drivers/staging/mt7621-pci/Kconfig
+++ /dev/null
@@ -1,8 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0
-config PCI_MT7621
-	tristate "MediaTek MT7621 PCI Controller"
-	depends on RALINK
-	select PCI_DRIVERS_GENERIC
-	help
-	  This selects a driver for the MediaTek MT7621 PCI Controller.
-
diff --git a/drivers/staging/mt7621-pci/Makefile b/drivers/staging/mt7621-pci/Makefile
deleted file mode 100644
index f4e651cf7ce3..000000000000
--- a/drivers/staging/mt7621-pci/Makefile
+++ /dev/null
@@ -1,2 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0
-obj-$(CONFIG_PCI_MT7621)       += pci-mt7621.o
diff --git a/drivers/staging/mt7621-pci/TODO b/drivers/staging/mt7621-pci/TODO
deleted file mode 100644
index d674a9ac85c1..000000000000
--- a/drivers/staging/mt7621-pci/TODO
+++ /dev/null
@@ -1,4 +0,0 @@
-
-- general code review and cleanup
-
-Cc: NeilBrown <neil@brown.name>
diff --git a/drivers/staging/mt7621-pci/mediatek,mt7621-pci.txt b/drivers/staging/mt7621-pci/mediatek,mt7621-pci.txt
deleted file mode 100644
index 327a68267309..000000000000
--- a/drivers/staging/mt7621-pci/mediatek,mt7621-pci.txt
+++ /dev/null
@@ -1,104 +0,0 @@
-MediaTek MT7621 PCIe controller
-
-Required properties:
-- compatible: "mediatek,mt7621-pci"
-- device_type: Must be "pci"
-- reg: Base addresses and lengths of the PCIe subsys and root ports.
-- bus-range: Range of bus numbers associated with this controller.
-- #address-cells: Address representation for root ports (must be 3)
-- pinctrl-names : The pin control state names.
-- pinctrl-0: The "default" pinctrl state.
-- #size-cells: Size representation for root ports (must be 2)
-- ranges: Ranges for the PCI memory and I/O regions.
-- #interrupt-cells: Must be 1
-- interrupt-map-mask and interrupt-map: Standard PCI IRQ mapping properties.
-  Please refer to the standard PCI bus binding document for a more detailed
-  explanation.
-- status: either "disabled" or "okay".
-- resets: Must contain an entry for each entry in reset-names.
-  See ../reset/reset.txt for details.
-- reset-names: Must be "pcie0", "pcie1", "pcieN"... based on the number of
-  root ports.
-- clocks: Must contain an entry for each entry in clock-names.
-  See ../clocks/clock-bindings.txt for details.
-- clock-names: Must be "pcie0", "pcie1", "pcieN"... based on the number of
-  root ports.
-- reset-gpios: GPIO specs for the reset pins.
-
-In addition, the device tree node must have sub-nodes describing each PCIe port
-interface, having the following mandatory properties:
-
-Required properties:
-- reg: Only the first four bytes are used to refer to the correct bus number
-      and device number.
-- #address-cells: Must be 3
-- #size-cells: Must be 2
-- ranges: Sub-ranges distributed from the PCIe controller node. An empty
-  property is sufficient.
-- bus-range: Range of bus numbers associated with this port.
-
-Example for MT7621:
-
-	pcie: pcie@1e140000 {
-		compatible = "mediatek,mt7621-pci";
-        reg = <0x1e140000 0x100    /* host-pci bridge registers */
-               0x1e142000 0x100    /* pcie port 0 RC control registers */
-               0x1e143000 0x100    /* pcie port 1 RC control registers */
-               0x1e144000 0x100>;  /* pcie port 2 RC control registers */
-
-		#address-cells = <3>;
-		#size-cells = <2>;
-
-		pinctrl-names = "default";
-		pinctrl-0 = <&pcie_pins>;
-
-		device_type = "pci";
-
-		bus-range = <0 255>;
-		ranges = <
-			0x02000000 0 0x00000000 0x60000000 0 0x10000000 /* pci memory */
-			0x01000000 0 0x00000000 0x1e160000 0 0x00010000 /* io space */
-		>;
-
-		#interrupt-cells = <1>;
-		interrupt-map-mask = <0xF0000 0 0 1>;
-		interrupt-map = <0x10000 0 0 1 &gic GIC_SHARED 4 IRQ_TYPE_LEVEL_HIGH>,
-				<0x20000 0 0 1 &gic GIC_SHARED 24 IRQ_TYPE_LEVEL_HIGH>,
-				<0x30000 0 0 1 &gic GIC_SHARED 25 IRQ_TYPE_LEVEL_HIGH>;
-
-		status = "disabled";
-
-		resets = <&rstctrl 24 &rstctrl 25 &rstctrl 26>;
-		reset-names = "pcie0", "pcie1", "pcie2";
-		clocks = <&clkctrl 24 &clkctrl 25 &clkctrl 26>;
-		clock-names = "pcie0", "pcie1", "pcie2";
-
-		reset-gpios = <&gpio 19 GPIO_ACTIVE_LOW>,
-				<&gpio 8 GPIO_ACTIVE_LOW>,
-				<&gpio 7 GPIO_ACTIVE_LOW>;
-
-		pcie@0,0 {
-			reg = <0x0000 0 0 0 0>;
-			#address-cells = <3>;
-			#size-cells = <2>;
-			ranges;
-			bus-range = <0x00 0xff>;
-		};
-
-		pcie@1,0 {
-			reg = <0x0800 0 0 0 0>;
-			#address-cells = <3>;
-			#size-cells = <2>;
-			ranges;
-			bus-range = <0x00 0xff>;
-		};
-
-		pcie@2,0 {
-			reg = <0x1000 0 0 0 0>;
-			#address-cells = <3>;
-			#size-cells = <2>;
-			ranges;
-			bus-range = <0x00 0xff>;
-		};
-	};
-
diff --git a/drivers/staging/mt7621-pci/pci-mt7621.c b/drivers/staging/mt7621-pci/pci-mt7621.c
deleted file mode 100644
index fe1945819d25..000000000000
--- a/drivers/staging/mt7621-pci/pci-mt7621.c
+++ /dev/null
@@ -1,624 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0+
-/*
- * BRIEF MODULE DESCRIPTION
- *     PCI init for Ralink RT2880 solution
- *
- * Copyright 2007 Ralink Inc. (bruce_chang@ralinktech.com.tw)
- *
- * May 2007 Bruce Chang
- * Initial Release
- *
- * May 2009 Bruce Chang
- * support RT2880/RT3883 PCIe
- *
- * May 2011 Bruce Chang
- * support RT6855/MT7620 PCIe
- */
-
-#include <linux/bitops.h>
-#include <linux/clk.h>
-#include <linux/delay.h>
-#include <linux/gpio/consumer.h>
-#include <linux/module.h>
-#include <linux/of.h>
-#include <linux/of_address.h>
-#include <linux/of_pci.h>
-#include <linux/of_platform.h>
-#include <linux/pci.h>
-#include <linux/phy/phy.h>
-#include <linux/platform_device.h>
-#include <linux/reset.h>
-#include <linux/sys_soc.h>
-
-/* MediaTek specific configuration registers */
-#define PCIE_FTS_NUM			0x70c
-#define PCIE_FTS_NUM_MASK		GENMASK(15, 8)
-#define PCIE_FTS_NUM_L0(x)		(((x) & 0xff) << 8)
-
-/* Host-PCI bridge registers */
-#define RALINK_PCI_PCICFG_ADDR		0x0000
-#define RALINK_PCI_PCIMSK_ADDR		0x000C
-#define RALINK_PCI_CONFIG_ADDR		0x0020
-#define RALINK_PCI_CONFIG_DATA		0x0024
-#define RALINK_PCI_MEMBASE		0x0028
-#define RALINK_PCI_IOBASE		0x002C
-
-/* PCIe RC control registers */
-#define MT7621_PCIE_OFFSET		0x2000
-#define MT7621_NEXT_PORT		0x1000
-
-#define RALINK_PCI_BAR0SETUP_ADDR	0x0010
-#define RALINK_PCI_ID			0x0030
-#define RALINK_PCI_CLASS		0x0034
-#define RALINK_PCI_SUBID		0x0038
-#define RALINK_PCI_STATUS		0x0050
-
-/* Some definition values */
-#define PCIE_REVISION_ID		BIT(0)
-#define PCIE_CLASS_CODE			(0x60400 << 8)
-#define PCIE_BAR_MAP_MAX		GENMASK(30, 16)
-#define PCIE_BAR_ENABLE			BIT(0)
-#define PCIE_PORT_INT_EN(x)		BIT(20 + (x))
-#define PCIE_PORT_LINKUP		BIT(0)
-
-#define PERST_DELAY_MS			100
-
-/**
- * struct mt7621_pcie_port - PCIe port information
- * @base: I/O mapped register base
- * @list: port list
- * @pcie: pointer to PCIe host info
- * @clk: pointer to the port clock gate
- * @phy: pointer to PHY control block
- * @pcie_rst: pointer to port reset control
- * @gpio_rst: gpio reset
- * @slot: port slot
- * @enabled: indicates if port is enabled
- */
-struct mt7621_pcie_port {
-	void __iomem *base;
-	struct list_head list;
-	struct mt7621_pcie *pcie;
-	struct clk *clk;
-	struct phy *phy;
-	struct reset_control *pcie_rst;
-	struct gpio_desc *gpio_rst;
-	u32 slot;
-	bool enabled;
-};
-
-/**
- * struct mt7621_pcie - PCIe host information
- * @base: IO Mapped Register Base
- * @io: IO resource
- * @mem: pointer to non-prefetchable memory resource
- * @dev: Pointer to PCIe device
- * @io_map_base: virtual memory base address for io
- * @ports: pointer to PCIe port information
- * @resets_inverted: depends on chip revision
- * reset lines are inverted.
- */
-struct mt7621_pcie {
-	void __iomem *base;
-	struct device *dev;
-	struct resource io;
-	struct resource *mem;
-	unsigned long io_map_base;
-	struct list_head ports;
-	bool resets_inverted;
-};
-
-static inline u32 pcie_read(struct mt7621_pcie *pcie, u32 reg)
-{
-	return readl(pcie->base + reg);
-}
-
-static inline void pcie_write(struct mt7621_pcie *pcie, u32 val, u32 reg)
-{
-	writel(val, pcie->base + reg);
-}
-
-static inline void pcie_rmw(struct mt7621_pcie *pcie, u32 reg, u32 clr, u32 set)
-{
-	u32 val = readl(pcie->base + reg);
-
-	val &= ~clr;
-	val |= set;
-	writel(val, pcie->base + reg);
-}
-
-static inline u32 pcie_port_read(struct mt7621_pcie_port *port, u32 reg)
-{
-	return readl(port->base + reg);
-}
-
-static inline void pcie_port_write(struct mt7621_pcie_port *port,
-				   u32 val, u32 reg)
-{
-	writel(val, port->base + reg);
-}
-
-static inline u32 mt7621_pci_get_cfgaddr(unsigned int bus, unsigned int slot,
-					 unsigned int func, unsigned int where)
-{
-	return (((where & 0xF00) >> 8) << 24) | (bus << 16) | (slot << 11) |
-		(func << 8) | (where & 0xfc) | 0x80000000;
-}
-
-static void __iomem *mt7621_pcie_map_bus(struct pci_bus *bus,
-					 unsigned int devfn, int where)
-{
-	struct mt7621_pcie *pcie = bus->sysdata;
-	u32 address = mt7621_pci_get_cfgaddr(bus->number, PCI_SLOT(devfn),
-					     PCI_FUNC(devfn), where);
-
-	writel(address, pcie->base + RALINK_PCI_CONFIG_ADDR);
-
-	return pcie->base + RALINK_PCI_CONFIG_DATA + (where & 3);
-}
-
-struct pci_ops mt7621_pci_ops = {
-	.map_bus	= mt7621_pcie_map_bus,
-	.read		= pci_generic_config_read,
-	.write		= pci_generic_config_write,
-};
-
-static u32 read_config(struct mt7621_pcie *pcie, unsigned int dev, u32 reg)
-{
-	u32 address = mt7621_pci_get_cfgaddr(0, dev, 0, reg);
-
-	pcie_write(pcie, address, RALINK_PCI_CONFIG_ADDR);
-	return pcie_read(pcie, RALINK_PCI_CONFIG_DATA);
-}
-
-static void write_config(struct mt7621_pcie *pcie, unsigned int dev,
-			 u32 reg, u32 val)
-{
-	u32 address = mt7621_pci_get_cfgaddr(0, dev, 0, reg);
-
-	pcie_write(pcie, address, RALINK_PCI_CONFIG_ADDR);
-	pcie_write(pcie, val, RALINK_PCI_CONFIG_DATA);
-}
-
-static inline void mt7621_rst_gpio_pcie_assert(struct mt7621_pcie_port *port)
-{
-	if (port->gpio_rst)
-		gpiod_set_value(port->gpio_rst, 1);
-}
-
-static inline void mt7621_rst_gpio_pcie_deassert(struct mt7621_pcie_port *port)
-{
-	if (port->gpio_rst)
-		gpiod_set_value(port->gpio_rst, 0);
-}
-
-static inline bool mt7621_pcie_port_is_linkup(struct mt7621_pcie_port *port)
-{
-	return (pcie_port_read(port, RALINK_PCI_STATUS) & PCIE_PORT_LINKUP) != 0;
-}
-
-static inline void mt7621_control_assert(struct mt7621_pcie_port *port)
-{
-	struct mt7621_pcie *pcie = port->pcie;
-
-	if (pcie->resets_inverted)
-		reset_control_assert(port->pcie_rst);
-	else
-		reset_control_deassert(port->pcie_rst);
-}
-
-static inline void mt7621_control_deassert(struct mt7621_pcie_port *port)
-{
-	struct mt7621_pcie *pcie = port->pcie;
-
-	if (pcie->resets_inverted)
-		reset_control_deassert(port->pcie_rst);
-	else
-		reset_control_assert(port->pcie_rst);
-}
-
-static void setup_cm_memory_region(struct mt7621_pcie *pcie)
-{
-	struct resource *mem_resource = pcie->mem;
-	struct device *dev = pcie->dev;
-	resource_size_t mask;
-
-	if (mips_cps_numiocu(0)) {
-		/*
-		 * FIXME: hardware doesn't accept mask values with 1s after
-		 * 0s (e.g. 0xffef), so it would be great to warn if that's
-		 * about to happen
-		 */
-		mask = ~(mem_resource->end - mem_resource->start);
-
-		write_gcr_reg1_base(mem_resource->start);
-		write_gcr_reg1_mask(mask | CM_GCR_REGn_MASK_CMTGT_IOCU0);
-		dev_info(dev, "PCI coherence region base: 0x%08llx, mask/settings: 0x%08llx\n",
-			 (unsigned long long)read_gcr_reg1_base(),
-			 (unsigned long long)read_gcr_reg1_mask());
-	}
-}
-
-static int mt7621_pci_parse_request_of_pci_ranges(struct pci_host_bridge *host)
-{
-	struct mt7621_pcie *pcie = pci_host_bridge_priv(host);
-	struct device *dev = pcie->dev;
-	struct device_node *node = dev->of_node;
-	struct of_pci_range_parser parser;
-	struct resource_entry *entry;
-	struct of_pci_range range;
-	LIST_HEAD(res);
-
-	if (of_pci_range_parser_init(&parser, node)) {
-		dev_err(dev, "missing \"ranges\" property\n");
-		return -EINVAL;
-	}
-
-	/*
-	 * IO_SPACE_LIMIT for MIPS is 0xffff but this platform uses IO at
-	 * upper address 0x001e160000. of_pci_range_to_resource does not work
-	 * well for MIPS platforms that don't define PCI_IOBASE, so set the IO
-	 * resource manually instead.
-	 */
-	for_each_of_pci_range(&parser, &range) {
-		switch (range.flags & IORESOURCE_TYPE_BITS) {
-		case IORESOURCE_IO:
-			pcie->io_map_base =
-				(unsigned long)ioremap(range.cpu_addr,
-						       range.size);
-			pcie->io.name = node->full_name;
-			pcie->io.flags = range.flags;
-			pcie->io.start = range.cpu_addr;
-			pcie->io.end = range.cpu_addr + range.size - 1;
-			pcie->io.parent = pcie->io.child = pcie->io.sibling = NULL;
-			set_io_port_base(pcie->io_map_base);
-			break;
-		}
-	}
-
-	entry = resource_list_first_type(&host->windows, IORESOURCE_MEM);
-	if (!entry) {
-		dev_err(dev, "Cannot get memory resource");
-		return -EINVAL;
-	}
-
-	pcie->mem = entry->res;
-	pci_add_resource(&res, &pcie->io);
-	pci_add_resource(&res, entry->res);
-	list_splice_init(&res, &host->windows);
-
-	return 0;
-}
-
-static int mt7621_pcie_parse_port(struct mt7621_pcie *pcie,
-				  int slot)
-{
-	struct mt7621_pcie_port *port;
-	struct device *dev = pcie->dev;
-	struct platform_device *pdev = to_platform_device(dev);
-	char name[10];
-
-	port = devm_kzalloc(dev, sizeof(*port), GFP_KERNEL);
-	if (!port)
-		return -ENOMEM;
-
-	port->base = devm_platform_ioremap_resource(pdev, slot + 1);
-	if (IS_ERR(port->base))
-		return PTR_ERR(port->base);
-
-	snprintf(name, sizeof(name), "pcie%d", slot);
-	port->clk = devm_clk_get(dev, name);
-	if (IS_ERR(port->clk)) {
-		dev_err(dev, "failed to get pcie%d clock\n", slot);
-		return PTR_ERR(port->clk);
-	}
-
-	snprintf(name, sizeof(name), "pcie%d", slot);
-	port->pcie_rst = devm_reset_control_get_exclusive(dev, name);
-	if (PTR_ERR(port->pcie_rst) == -EPROBE_DEFER) {
-		dev_err(dev, "failed to get pcie%d reset control\n", slot);
-		return PTR_ERR(port->pcie_rst);
-	}
-
-	snprintf(name, sizeof(name), "pcie-phy%d", slot);
-	port->phy = devm_phy_get(dev, name);
-	if (IS_ERR(port->phy) && slot != 1)
-		return PTR_ERR(port->phy);
-
-	port->gpio_rst = devm_gpiod_get_index_optional(dev, "reset", slot,
-						       GPIOD_OUT_LOW);
-	if (IS_ERR(port->gpio_rst)) {
-		dev_err(dev, "Failed to get GPIO for PCIe%d\n", slot);
-		return PTR_ERR(port->gpio_rst);
-	}
-
-	port->slot = slot;
-	port->pcie = pcie;
-
-	INIT_LIST_HEAD(&port->list);
-	list_add_tail(&port->list, &pcie->ports);
-
-	return 0;
-}
-
-static int mt7621_pcie_parse_dt(struct mt7621_pcie *pcie)
-{
-	struct device *dev = pcie->dev;
-	struct platform_device *pdev = to_platform_device(dev);
-	struct device_node *node = dev->of_node, *child;
-	int err;
-
-	pcie->base = devm_platform_ioremap_resource(pdev, 0);
-	if (IS_ERR(pcie->base))
-		return PTR_ERR(pcie->base);
-
-	for_each_available_child_of_node(node, child) {
-		int slot;
-
-		err = of_pci_get_devfn(child);
-		if (err < 0) {
-			of_node_put(child);
-			dev_err(dev, "failed to parse devfn: %d\n", err);
-			return err;
-		}
-
-		slot = PCI_SLOT(err);
-
-		err = mt7621_pcie_parse_port(pcie, slot);
-		if (err) {
-			of_node_put(child);
-			return err;
-		}
-	}
-
-	return 0;
-}
-
-static int mt7621_pcie_init_port(struct mt7621_pcie_port *port)
-{
-	struct mt7621_pcie *pcie = port->pcie;
-	struct device *dev = pcie->dev;
-	u32 slot = port->slot;
-	int err;
-
-	err = phy_init(port->phy);
-	if (err) {
-		dev_err(dev, "failed to initialize port%d phy\n", slot);
-		return err;
-	}
-
-	err = phy_power_on(port->phy);
-	if (err) {
-		dev_err(dev, "failed to power on port%d phy\n", slot);
-		phy_exit(port->phy);
-		return err;
-	}
-
-	port->enabled = true;
-
-	return 0;
-}
-
-static void mt7621_pcie_reset_assert(struct mt7621_pcie *pcie)
-{
-	struct mt7621_pcie_port *port;
-
-	list_for_each_entry(port, &pcie->ports, list) {
-		/* PCIe RC reset assert */
-		mt7621_control_assert(port);
-
-		/* PCIe EP reset assert */
-		mt7621_rst_gpio_pcie_assert(port);
-	}
-
-	msleep(PERST_DELAY_MS);
-}
-
-static void mt7621_pcie_reset_rc_deassert(struct mt7621_pcie *pcie)
-{
-	struct mt7621_pcie_port *port;
-
-	list_for_each_entry(port, &pcie->ports, list)
-		mt7621_control_deassert(port);
-}
-
-static void mt7621_pcie_reset_ep_deassert(struct mt7621_pcie *pcie)
-{
-	struct mt7621_pcie_port *port;
-
-	list_for_each_entry(port, &pcie->ports, list)
-		mt7621_rst_gpio_pcie_deassert(port);
-
-	msleep(PERST_DELAY_MS);
-}
-
-static void mt7621_pcie_init_ports(struct mt7621_pcie *pcie)
-{
-	struct device *dev = pcie->dev;
-	struct mt7621_pcie_port *port, *tmp;
-	int err;
-
-	mt7621_pcie_reset_assert(pcie);
-	mt7621_pcie_reset_rc_deassert(pcie);
-
-	list_for_each_entry_safe(port, tmp, &pcie->ports, list) {
-		u32 slot = port->slot;
-
-		if (slot == 1) {
-			port->enabled = true;
-			continue;
-		}
-
-		err = mt7621_pcie_init_port(port);
-		if (err) {
-			dev_err(dev, "Initiating port %d failed\n", slot);
-			list_del(&port->list);
-		}
-	}
-
-	mt7621_pcie_reset_ep_deassert(pcie);
-
-	tmp = NULL;
-	list_for_each_entry(port, &pcie->ports, list) {
-		u32 slot = port->slot;
-
-		if (!mt7621_pcie_port_is_linkup(port)) {
-			dev_err(dev, "pcie%d no card, disable it (RST & CLK)\n",
-				slot);
-			mt7621_control_assert(port);
-			clk_disable_unprepare(port->clk);
-			port->enabled = false;
-
-			if (slot == 0) {
-				tmp = port;
-				continue;
-			}
-
-			if (slot == 1 && tmp && !tmp->enabled)
-				phy_power_off(tmp->phy);
-		}
-	}
-}
-
-static void mt7621_pcie_enable_port(struct mt7621_pcie_port *port)
-{
-	struct mt7621_pcie *pcie = port->pcie;
-	u32 slot = port->slot;
-	u32 offset = MT7621_PCIE_OFFSET + (slot * MT7621_NEXT_PORT);
-	u32 val;
-
-	/* enable pcie interrupt */
-	val = pcie_read(pcie, RALINK_PCI_PCIMSK_ADDR);
-	val |= PCIE_PORT_INT_EN(slot);
-	pcie_write(pcie, val, RALINK_PCI_PCIMSK_ADDR);
-
-	/* map 2G DDR region */
-	pcie_write(pcie, PCIE_BAR_MAP_MAX | PCIE_BAR_ENABLE,
-		   offset + RALINK_PCI_BAR0SETUP_ADDR);
-
-	/* configure class code and revision ID */
-	pcie_write(pcie, PCIE_CLASS_CODE | PCIE_REVISION_ID,
-		   offset + RALINK_PCI_CLASS);
-}
-
-static int mt7621_pcie_enable_ports(struct mt7621_pcie *pcie)
-{
-	struct device *dev = pcie->dev;
-	struct mt7621_pcie_port *port;
-	u8 num_slots_enabled = 0;
-	u32 slot;
-	u32 val;
-	int err;
-
-	/* Setup MEMWIN and IOWIN */
-	pcie_write(pcie, 0xffffffff, RALINK_PCI_MEMBASE);
-	pcie_write(pcie, pcie->io.start, RALINK_PCI_IOBASE);
-
-	list_for_each_entry(port, &pcie->ports, list) {
-		if (port->enabled) {
-			err = clk_prepare_enable(port->clk);
-			if (err) {
-				dev_err(dev, "enabling clk pcie%d\n", slot);
-				return err;
-			}
-
-			mt7621_pcie_enable_port(port);
-			dev_info(dev, "PCIE%d enabled\n", port->slot);
-			num_slots_enabled++;
-		}
-	}
-
-	for (slot = 0; slot < num_slots_enabled; slot++) {
-		val = read_config(pcie, slot, PCI_COMMAND);
-		val |= PCI_COMMAND_MASTER;
-		write_config(pcie, slot, PCI_COMMAND, val);
-		/* configure RC FTS number to 250 when it leaves L0s */
-		val = read_config(pcie, slot, PCIE_FTS_NUM);
-		val &= ~PCIE_FTS_NUM_MASK;
-		val |= PCIE_FTS_NUM_L0(0x50);
-		write_config(pcie, slot, PCIE_FTS_NUM, val);
-	}
-
-	return 0;
-}
-
-static int mt7621_pcie_register_host(struct pci_host_bridge *host)
-{
-	struct mt7621_pcie *pcie = pci_host_bridge_priv(host);
-
-	host->ops = &mt7621_pci_ops;
-	host->sysdata = pcie;
-	return pci_host_probe(host);
-}
-
-static const struct soc_device_attribute mt7621_pci_quirks_match[] = {
-	{ .soc_id = "mt7621", .revision = "E2" }
-};
-
-static int mt7621_pci_probe(struct platform_device *pdev)
-{
-	struct device *dev = &pdev->dev;
-	const struct soc_device_attribute *attr;
-	struct mt7621_pcie *pcie;
-	struct pci_host_bridge *bridge;
-	int err;
-
-	if (!dev->of_node)
-		return -ENODEV;
-
-	bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
-	if (!bridge)
-		return -ENOMEM;
-
-	pcie = pci_host_bridge_priv(bridge);
-	pcie->dev = dev;
-	platform_set_drvdata(pdev, pcie);
-	INIT_LIST_HEAD(&pcie->ports);
-
-	attr = soc_device_match(mt7621_pci_quirks_match);
-	if (attr)
-		pcie->resets_inverted = true;
-
-	err = mt7621_pcie_parse_dt(pcie);
-	if (err) {
-		dev_err(dev, "Parsing DT failed\n");
-		return err;
-	}
-
-	err = mt7621_pci_parse_request_of_pci_ranges(bridge);
-	if (err) {
-		dev_err(dev, "Error requesting pci resources from ranges");
-		return err;
-	}
-
-	/* set resources limits */
-	ioport_resource.start = pcie->io.start;
-	ioport_resource.end = pcie->io.end;
-
-	mt7621_pcie_init_ports(pcie);
-
-	err = mt7621_pcie_enable_ports(pcie);
-	if (err) {
-		dev_err(dev, "Error enabling pcie ports\n");
-		return err;
-	}
-
-	setup_cm_memory_region(pcie);
-
-	return mt7621_pcie_register_host(bridge);
-}
-
-static const struct of_device_id mt7621_pci_ids[] = {
-	{ .compatible = "mediatek,mt7621-pci" },
-	{},
-};
-MODULE_DEVICE_TABLE(of, mt7621_pci_ids);
-
-static struct platform_driver mt7621_pci_driver = {
-	.probe = mt7621_pci_probe,
-	.driver = {
-		.name = "mt7621-pci",
-		.of_match_table = of_match_ptr(mt7621_pci_ids),
-	},
-};
-builtin_platform_driver(mt7621_pci_driver);
-- 
2.25.1

Re: [PATCH 3/4] staging: mt7621-pci: remove driver from staging

[Greg KH]

On Sat, May 15, 2021 at 02:40:54PM +0200, Sergio Paracuellos wrote:
> Driver has been moved into its proper place in the
> kernel 'arch/mips/pci'. Hence, remove it from staging.
> 
> Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
> ---

Acked-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

[PATCH 4/4] MAINTAINERS: add myself as maintainer of the MT7621 PCI controller driver

[Sergio Paracuellos]

Add myself as maintainer of the PCie Controlller driver for
MT7621 SoCs.

Signed-off-by: Sergio Paracuellos <sergio.paracuellos@gmail.com>
---
 MAINTAINERS | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index bd7aff0c120f..312ea2cad79b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -11562,6 +11562,12 @@ S:	Maintained
 F:	Documentation/devicetree/bindings/i2c/i2c-mt7621.txt
 F:	drivers/i2c/busses/i2c-mt7621.c
 
+MEDIATEK MT7621 PCI CONTROLLER DRIVER
+M:	Sergio Paracuellos <sergio.paracuellos@gmail.com>
+S:	Maintained
+F:	Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml
+F:	arch/mips/pci/pci-mt7621.c
+
 MEDIATEK MT7621 PHY PCI DRIVER
 M:	Sergio Paracuellos <sergio.paracuellos@gmail.com>
 S:	Maintained
-- 
2.25.1

Re: [PATCH 0/4] MIPS: ralink: pci: driver for Pcie controller in MT7621 SoCs

[Bjorn Helgaas]

On Sat, May 15, 2021 at 02:40:51PM +0200, Sergio Paracuellos wrote:
> MediaTek MT7621 PCIe subsys supports single Root complex (RC)
> with 3 Root Ports. Each Root Ports supports a Gen1 1-lane Link.
> Topology is as follows:
> 
>                           MT7621 PCIe HOST Topology
> 
>                                    .-------.
>                                    |       |
>                                    |  CPU  |
>                                    |       |
>                                    '-------'
>                                        |
>                                        |
>                                        |
>                                        v
>                               .------------------.
>                   .-----------|  HOST/PCI Bridge |------------.
>                   |           '------------------'            |     Type1 
>          BUS0     |                     |                     |    Access 
>                   v                     v                     v    On Bus0
>           .-------------.        .-------------.       .-------------.
>           | VIRTUAL P2P |        | VIRTUAL P2P |       | VIRTUAL P2P |
>           |    BUS0     |        |    BUS0     |       |    BUS0     |
>           |    DEV0     |        |    DEV1     |       |    DEV2     |
>           '-------------'        '-------------'       '-------------'
>     Type0        |          Type0       |         Type0       |
>    Access   BUS1 |         Access   BUS2|        Access   BUS3|
>    On Bus1       v         On Bus2      v        On Bus3      v
>            .----------.           .----------.          .----------.
>            | Device 0 |           | Device 0 |          | Device 0 |
>            |  Func 0  |           |  Func 0  |          |  Func 0  |
>            '----------'           '----------'          '----------'
> 
> This driver has been very long time in staging and I have been cleaning
> it from its first versions where there was code kaos and PCI_LEGACY support.
> Original code came probably from openWRT based on mediatek's SDK code. There
> is no documentation at all about the mt7621 PCI subsystem.
> I have been cleaning it targeting mt7621 SoC which is the one I use in
> my GNUBee PC1 board and HiLink HLK-MT7621A evaluation board.
> 
> Now I think is clean enough to be moved into 'arch/mips/pci'.
> 
> This driver also uses already mainlined pci phy driver located in
> 'drivers/phy/ralink/phy-mt7621-pci.c'. There are two instances of
> the phy being the first one dual ported for pci0 and pci1, and the 
> second one not dual ported dedicated to pci2. Because of writing twice 
> some phy registers of the dual-ported one sometimes become in not
> confident boot cycles we have to take care of this when device link
> is checked here in controller driver. We power on the dual ported-phy
> if there is something connected in pcie0 or pcie1. In the same manner
> we have to properly disable it only if nothing is connected in of both
> pcie0 and pci1 slots.
> 
> Another thing that must be mentioned is that this driver uses IO
> in physical address 0x001e160000. IO_SPACE_LIMIT for MIPS is 0xffff
> so some generic PCI functions (like of_pci_range_to_resource) won't
> work and the resource ranges part for IO is set manually.
> 
> I had already sent binding documentation to be reviewed but I am
> include also here with the driver itself and this cover letter
> to make easy review process.
> 
> Best regards,
>     Sergio Paracuellos
> 
> Sergio Paracuellos (4):
>   dt-bindings: mt7621-pci: PCIe binding documentation for MT7621 SoCs
>   MIPS: pci: Add driver for MT7621 PCIe controller
>   staging: mt7621-pci: remove driver from staging

Generally it's better if the move can be done in one commit instead of
an add followed by a remove.

I see there are a bunch of MIPS PCI controller drivers in
arch/mips/pci/, so I see the argument for putting this one there as
well.

But most of the similar drivers are in drivers/pci/controller/, where
I think it's easier to keep them up to date with changes in the PCI
core.  Have you considered putting this one there?

>   MAINTAINERS: add myself as maintainer of the MT7621 PCI controller
>     driver
> 
>  .../bindings/pci/mediatek,mt7621-pci.yaml     | 149 ++++++++++++++++++
>  MAINTAINERS                                   |   6 +
>  arch/mips/pci/Makefile                        |   1 +
>  .../mt7621-pci => arch/mips/pci}/pci-mt7621.c |   0
>  arch/mips/ralink/Kconfig                      |   9 +-
>  drivers/staging/Kconfig                       |   2 -
>  drivers/staging/Makefile                      |   1 -
>  drivers/staging/mt7621-pci/Kconfig            |   8 -
>  drivers/staging/mt7621-pci/Makefile           |   2 -
>  drivers/staging/mt7621-pci/TODO               |   4 -
>  .../mt7621-pci/mediatek,mt7621-pci.txt        | 104 ------------
>  11 files changed, 164 insertions(+), 122 deletions(-)
>  create mode 100644 Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml
>  rename {drivers/staging/mt7621-pci => arch/mips/pci}/pci-mt7621.c (100%)
>  delete mode 100644 drivers/staging/mt7621-pci/Kconfig
>  delete mode 100644 drivers/staging/mt7621-pci/Makefile
>  delete mode 100644 drivers/staging/mt7621-pci/TODO
>  delete mode 100644 drivers/staging/mt7621-pci/mediatek,mt7621-pci.txt
> 
> -- 
> 2.25.1

Re: [PATCH 0/4] MIPS: ralink: pci: driver for Pcie controller in MT7621 SoCs

[Sergio Paracuellos]

Hi Bjorn,

On Wed, May 19, 2021 at 10:36 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Sat, May 15, 2021 at 02:40:51PM +0200, Sergio Paracuellos wrote:
> > MediaTek MT7621 PCIe subsys supports single Root complex (RC)
> > with 3 Root Ports. Each Root Ports supports a Gen1 1-lane Link.
> > Topology is as follows:
> >
> >                           MT7621 PCIe HOST Topology
> >
> >                                    .-------.
> >                                    |       |
> >                                    |  CPU  |
> >                                    |       |
> >                                    '-------'
> >                                        |
> >                                        |
> >                                        |
> >                                        v
> >                               .------------------.
> >                   .-----------|  HOST/PCI Bridge |------------.
> >                   |           '------------------'            |     Type1
> >          BUS0     |                     |                     |    Access
> >                   v                     v                     v    On Bus0
> >           .-------------.        .-------------.       .-------------.
> >           | VIRTUAL P2P |        | VIRTUAL P2P |       | VIRTUAL P2P |
> >           |    BUS0     |        |    BUS0     |       |    BUS0     |
> >           |    DEV0     |        |    DEV1     |       |    DEV2     |
> >           '-------------'        '-------------'       '-------------'
> >     Type0        |          Type0       |         Type0       |
> >    Access   BUS1 |         Access   BUS2|        Access   BUS3|
> >    On Bus1       v         On Bus2      v        On Bus3      v
> >            .----------.           .----------.          .----------.
> >            | Device 0 |           | Device 0 |          | Device 0 |
> >            |  Func 0  |           |  Func 0  |          |  Func 0  |
> >            '----------'           '----------'          '----------'
> >
> > This driver has been very long time in staging and I have been cleaning
> > it from its first versions where there was code kaos and PCI_LEGACY support.
> > Original code came probably from openWRT based on mediatek's SDK code. There
> > is no documentation at all about the mt7621 PCI subsystem.
> > I have been cleaning it targeting mt7621 SoC which is the one I use in
> > my GNUBee PC1 board and HiLink HLK-MT7621A evaluation board.
> >
> > Now I think is clean enough to be moved into 'arch/mips/pci'.
> >
> > This driver also uses already mainlined pci phy driver located in
> > 'drivers/phy/ralink/phy-mt7621-pci.c'. There are two instances of
> > the phy being the first one dual ported for pci0 and pci1, and the
> > second one not dual ported dedicated to pci2. Because of writing twice
> > some phy registers of the dual-ported one sometimes become in not
> > confident boot cycles we have to take care of this when device link
> > is checked here in controller driver. We power on the dual ported-phy
> > if there is something connected in pcie0 or pcie1. In the same manner
> > we have to properly disable it only if nothing is connected in of both
> > pcie0 and pci1 slots.
> >
> > Another thing that must be mentioned is that this driver uses IO
> > in physical address 0x001e160000. IO_SPACE_LIMIT for MIPS is 0xffff
> > so some generic PCI functions (like of_pci_range_to_resource) won't
> > work and the resource ranges part for IO is set manually.
> >
> > I had already sent binding documentation to be reviewed but I am
> > include also here with the driver itself and this cover letter
> > to make easy review process.
> >
> > Best regards,
> >     Sergio Paracuellos
> >
> > Sergio Paracuellos (4):
> >   dt-bindings: mt7621-pci: PCIe binding documentation for MT7621 SoCs
> >   MIPS: pci: Add driver for MT7621 PCIe controller
> >   staging: mt7621-pci: remove driver from staging
>
> Generally it's better if the move can be done in one commit instead of
> an add followed by a remove.

I have no problem at all to just move the driver instead of add and
remove. But it is easier to review initially in this way. No other
reason but this one. So, if after review is ok, we can just move this
in the way you are pointing out here.

>
> I see there are a bunch of MIPS PCI controller drivers in
> arch/mips/pci/, so I see the argument for putting this one there as
> well.
>
> But most of the similar drivers are in drivers/pci/controller/, where
> I think it's easier to keep them up to date with changes in the PCI
> core.  Have you considered putting this one there?

Most pci drivers in 'arch/mips/pci' are still using PCI_LEGACY stuff.
In contrast mt7621-pci is using current pci generic apis but even most
of the code is generic enough, there is one remaining thing which
depends on mips architecture which is the iocu region configuration
which must be done in the driver itself. This is the only reason to
move this driver into 'arch/mips/pci' instead of
'drivers/pci/controller/'. So... I am all ears to listen to
suggestions for the proper place for this driver. Thomas, do you have
any thoughts on this?

Thanks,
    Sergio Paracuellos
>
> >   MAINTAINERS: add myself as maintainer of the MT7621 PCI controller
> >     driver
> >
> >  .../bindings/pci/mediatek,mt7621-pci.yaml     | 149 ++++++++++++++++++
> >  MAINTAINERS                                   |   6 +
> >  arch/mips/pci/Makefile                        |   1 +
> >  .../mt7621-pci => arch/mips/pci}/pci-mt7621.c |   0
> >  arch/mips/ralink/Kconfig                      |   9 +-
> >  drivers/staging/Kconfig                       |   2 -
> >  drivers/staging/Makefile                      |   1 -
> >  drivers/staging/mt7621-pci/Kconfig            |   8 -
> >  drivers/staging/mt7621-pci/Makefile           |   2 -
> >  drivers/staging/mt7621-pci/TODO               |   4 -
> >  .../mt7621-pci/mediatek,mt7621-pci.txt        | 104 ------------
> >  11 files changed, 164 insertions(+), 122 deletions(-)
> >  create mode 100644 Documentation/devicetree/bindings/pci/mediatek,mt7621-pci.yaml
> >  rename {drivers/staging/mt7621-pci => arch/mips/pci}/pci-mt7621.c (100%)
> >  delete mode 100644 drivers/staging/mt7621-pci/Kconfig
> >  delete mode 100644 drivers/staging/mt7621-pci/Makefile
> >  delete mode 100644 drivers/staging/mt7621-pci/TODO
> >  delete mode 100644 drivers/staging/mt7621-pci/mediatek,mt7621-pci.txt
> >
> > --
> > 2.25.1

Re: [PATCH 0/4] MIPS: ralink: pci: driver for Pcie controller in MT7621 SoCs

[Thomas Bogendoerfer]

On Wed, May 19, 2021 at 11:18:36PM +0200, Sergio Paracuellos wrote:
> On Wed, May 19, 2021 at 10:36 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > But most of the similar drivers are in drivers/pci/controller/, where
> > I think it's easier to keep them up to date with changes in the PCI
> > core.  Have you considered putting this one there?
> 
> Most pci drivers in 'arch/mips/pci' are still using PCI_LEGACY stuff.
> In contrast mt7621-pci is using current pci generic apis but even most
> of the code is generic enough, there is one remaining thing which
> depends on mips architecture which is the iocu region configuration
> which must be done in the driver itself. This is the only reason to
> move this driver into 'arch/mips/pci' instead of
> 'drivers/pci/controller/'. So... I am all ears to listen to
> suggestions for the proper place for this driver. Thomas, do you have
> any thoughts on this?

I tried to put a pci-xtalk driver into drivers/pci/controller, but
Lorenzo didn't want it there for being MIPS and not DT based. So this
one is DT based, but still MIPS. I'm perfectly fine putting this
driver into drivers/pci/controller/

Lorenzo, what's your opinion ?

Thomas.

-- 
Crap can work. Given enough thrust pigs will fly, but it's not necessarily a
good idea.                                                [ RFC1925, 2.3 ]

Re: [PATCH 0/4] MIPS: ralink: pci: driver for Pcie controller in MT7621 SoCs

[Pali Rohár]

On Friday 21 May 2021 12:23:38 Thomas Bogendoerfer wrote:
> On Wed, May 19, 2021 at 11:18:36PM +0200, Sergio Paracuellos wrote:
> > On Wed, May 19, 2021 at 10:36 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > But most of the similar drivers are in drivers/pci/controller/, where
> > > I think it's easier to keep them up to date with changes in the PCI
> > > core.  Have you considered putting this one there?
> > 
> > Most pci drivers in 'arch/mips/pci' are still using PCI_LEGACY stuff.
> > In contrast mt7621-pci is using current pci generic apis but even most
> > of the code is generic enough, there is one remaining thing which
> > depends on mips architecture which is the iocu region configuration
> > which must be done in the driver itself. This is the only reason to
> > move this driver into 'arch/mips/pci' instead of
> > 'drivers/pci/controller/'. So... I am all ears to listen to
> > suggestions for the proper place for this driver. Thomas, do you have
> > any thoughts on this?
> 
> I tried to put a pci-xtalk driver into drivers/pci/controller, but
> Lorenzo didn't want it there for being MIPS and not DT based. So this
> one is DT based, but still MIPS. I'm perfectly fine putting this
> driver into drivers/pci/controller/

In my personal opinion this driver could go into drivers/pci/controller/

Re: [PATCH 0/4] MIPS: ralink: pci: driver for Pcie controller in MT7621 SoCs

[Bjorn Helgaas]

On Mon, May 31, 2021 at 03:18:45PM +0200, Pali Rohár wrote:
> On Friday 21 May 2021 12:23:38 Thomas Bogendoerfer wrote:
> > On Wed, May 19, 2021 at 11:18:36PM +0200, Sergio Paracuellos wrote:
> > > On Wed, May 19, 2021 at 10:36 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > But most of the similar drivers are in drivers/pci/controller/, where
> > > > I think it's easier to keep them up to date with changes in the PCI
> > > > core.  Have you considered putting this one there?
> > > 
> > > Most pci drivers in 'arch/mips/pci' are still using PCI_LEGACY stuff.
> > > In contrast mt7621-pci is using current pci generic apis but even most
> > > of the code is generic enough, there is one remaining thing which
> > > depends on mips architecture which is the iocu region configuration
> > > which must be done in the driver itself. This is the only reason to
> > > move this driver into 'arch/mips/pci' instead of
> > > 'drivers/pci/controller/'. So... I am all ears to listen to
> > > suggestions for the proper place for this driver. Thomas, do you have
> > > any thoughts on this?
> > 
> > I tried to put a pci-xtalk driver into drivers/pci/controller, but
> > Lorenzo didn't want it there for being MIPS and not DT based. So this
> > one is DT based, but still MIPS. I'm perfectly fine putting this
> > driver into drivers/pci/controller/
> 
> In my personal opinion this driver could go into drivers/pci/controller/

I'm not sure exactly what "PCI_LEGACY" above refers to.

I don't see any direct #includes of arch/mips in the driver.  I do see
that it uses mips_cps_numiocu(), which is certainly MIPS-specific.

But we do have some things in drivers/pci/controller/ that only build
on certain arches, enforced mostly by Kconfig rules, so I think you
could do that.  We try to make so things can at least be *compiled* on
any arch, but I know that's not always possible.

So I think it would be useful to put this in drivers/pci/controller/
somewhere because it will make it easier to see common patterns and
refactoring opportunities.

Bjorn

Re: [PATCH 0/4] MIPS: ralink: pci: driver for Pcie controller in MT7621 SoCs

[Sergio Paracuellos]

Hi Bjorn,

On Fri, Jun 4, 2021 at 9:43 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Mon, May 31, 2021 at 03:18:45PM +0200, Pali Rohár wrote:
> > On Friday 21 May 2021 12:23:38 Thomas Bogendoerfer wrote:
> > > On Wed, May 19, 2021 at 11:18:36PM +0200, Sergio Paracuellos wrote:
> > > > On Wed, May 19, 2021 at 10:36 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > > But most of the similar drivers are in drivers/pci/controller/, where
> > > > > I think it's easier to keep them up to date with changes in the PCI
> > > > > core.  Have you considered putting this one there?
> > > >
> > > > Most pci drivers in 'arch/mips/pci' are still using PCI_LEGACY stuff.
> > > > In contrast mt7621-pci is using current pci generic apis but even most
> > > > of the code is generic enough, there is one remaining thing which
> > > > depends on mips architecture which is the iocu region configuration
> > > > which must be done in the driver itself. This is the only reason to
> > > > move this driver into 'arch/mips/pci' instead of
> > > > 'drivers/pci/controller/'. So... I am all ears to listen to
> > > > suggestions for the proper place for this driver. Thomas, do you have
> > > > any thoughts on this?
> > >
> > > I tried to put a pci-xtalk driver into drivers/pci/controller, but
> > > Lorenzo didn't want it there for being MIPS and not DT based. So this
> > > one is DT based, but still MIPS. I'm perfectly fine putting this
> > > driver into drivers/pci/controller/
> >
> > In my personal opinion this driver could go into drivers/pci/controller/
>
> I'm not sure exactly what "PCI_LEGACY" above refers to.

I meant most of the drivers there are not using current generic pci
apis but still using pci legacy ones.

>
> I don't see any direct #includes of arch/mips in the driver.  I do see
> that it uses mips_cps_numiocu(), which is certainly MIPS-specific.

Yes, mips_cps_numiocu is the only stuff needed and arch specific used
by this driver.

>
> But we do have some things in drivers/pci/controller/ that only build
> on certain arches, enforced mostly by Kconfig rules, so I think you
> could do that.  We try to make so things can at least be *compiled* on
> any arch, but I know that's not always possible.
>
> So I think it would be useful to put this in drivers/pci/controller/
> somewhere because it will make it easier to see common patterns and
> refactoring opportunities.

Ok, so I will move the driver into 'drivers/pci/controller/' in v2.

Thanks,
    Sergio Paracuellos
>
> Bjorn