Enabling PCIe support on Hikey 970

[Mauro Carvalho Chehab <mchehab+huawei@kernel.org>]

Hi Bjorn/Rob,

I've been trying to make a Hikey 970 board to work properly upstream.

This specific hardware is similar to a previous model (Hikey 960),
and it uses the same PCIe driver, with a few additions
(drivers/pci/controller/dwc/pcie-kirin.c).

The major difference between those two models is that, on Hikey 960,
the PCIe is mapped as [1]:

	+---------+      +--------+
	|Kirin 960|----> |PCIe M.2|
	|  SoC    |	 |        |
	+---------+      +--------+

While, on Hikey 970, the connection is more complex[2]:

	+---------+	 +--------+
	|         |	 |        |     +--------+
	|         |	 |        |---->|PCIe M.2|-->M.2 slot
	|         |	 |        |     +--------+
	|         |	 |        |
	|         |	 |        |     +--------+
	|Kirin 970|----> |Switch  |---->|Mini 1x |-->mini PCIe slot
	|         |	 |PEX 8606|     +--------+
	|  SoC    |	 |        |
	|         |	 |        |     +-------+
	|         |	 |        |---->|RTL8169|---> Ethernet
	|         |	 |        |     +-------+
	+---------+	 +--------+



[1] see https://www.96boards.org/documentation/consumer/hikey/hikey960/hardware-docs/hardware-user-manual.md.html
[2] see https://www.96boards.org/documentation/consumer/hikey/hikey970/hardware-docs/files/hikey970-schematics.pdf

When the driver is properly loaded, this is what can be seen there:

	$ lspci
	00:00.0 PCI bridge: Huawei Technologies Co., Ltd. Device 3670 (rev 01)
	01:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
	02:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
	02:04.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
	02:05.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
	02:07.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
	02:09.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
	06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 07)

(without anything connected to M.2 or mini 1x slots)

All devices after the SoC require a regulator line to be
enabled (LDO33). Starting the PCIe bus before turning them on
causes PCIe probe to fail.

There are also separate PERST lines for the switch (Broadcom PEX 8606), 
PCIe M.2, Mini 1x and for the Ethernet hardware (RTL 8169).

To make it a little more fun, the M.2, the Mini 1x and the
RTL 8169 also requires a clockreq in order to work.

Both the 4 PERST reset lines and the 3 CLOCKREQ lines are initialized
during the PCIe power on logic, at probing time.

I'm currently thinking about the best way to report this via
device tree.

It sounds to me that the best would be to add those 4 data at the DTS file:

	reset-gpios = <&gpio7 0 0 >, <&gpio25 2 0 >,
		      <&gpio3 1 0 >, <&gpio27 4 0 >;
	reset-names = "pcie_switch_reset", "pcie_eth_reset",
		      "pcie_m_2_reset",    "pcie_mini1_reset";
	clkreq-gpios = <&gpio20 6 0 >, <&gpio27 3 0 >,
		       <&gpio17 0 0 >;
	clkreq-names = "pcie_eth_clkreq", "pcie_m_2_clkreq",
		       "pcie_mini1_clkreq";

E. g. the complete representation for the PCIe controller will
be:

	soc {
		compatible = "simple-bus";
		#address-cells = <2>;
		#size-cells = <2>;
		ranges;
...
		pcie@f4000000 {
			compatible = "hisilicon,kirin970-pcie";
			reg = <0x0 0xf4000000 0x0 0x1000000>,
			      <0x0 0xfc180000 0x0 0x1000>,
			      <0x0 0xfc000000 0x0 0x80000>,
			      <0x0 0xf5000000 0x0 0x2000>;
			pci-supply = <&ldo33>;
			reg-names = "dbi", "apb", "phy", "config";
			bus-range = <0x0  0x1>;
			msi-parent = <&its_pcie>;
			#address-cells = <3>;
			#size-cells = <2>;
			device_type = "pci";
			ranges = <0x02000000 0x0 0x00000000
				  0x0 0xf6000000
				  0x0 0x02000000>;
			num-lanes = <1>;
			#interrupt-cells = <1>;
			interrupts = <0 283 4>;
			interrupt-names = "msi";
			interrupt-map-mask = <0 0 0 7>;
			interrupt-map = <0x0 0 0 1
					 &gic GIC_SPI 282 IRQ_TYPE_LEVEL_HIGH>,
					<0x0 0 0 2
					 &gic GIC_SPI 283 IRQ_TYPE_LEVEL_HIGH>,
					<0x0 0 0 3
					 &gic GIC_SPI 284 IRQ_TYPE_LEVEL_HIGH>,
					<0x0 0 0 4
					 &gic GIC_SPI 285 IRQ_TYPE_LEVEL_HIGH>;
			clocks = <&crg_ctrl HI3670_CLK_GATE_PCIEPHY_REF>,
				 <&crg_ctrl HI3670_CLK_GATE_PCIEAUX>,
				 <&crg_ctrl HI3670_PCLK_GATE_PCIE_PHY>,
				 <&crg_ctrl HI3670_PCLK_GATE_PCIE_SYS>,
				 <&crg_ctrl HI3670_ACLK_GATE_PCIE>;

			clock-names = "pcie_phy_ref", "pcie_aux",
				      "pcie_apb_phy", "pcie_apb_sys",
				      "pcie_aclk";
			reset-gpios = <&gpio7 0 0 >, <&gpio25 2 0 >,
				      <&gpio3 1 0 >, <&gpio27 4 0 >;

			reset-names = "pcie_switch_reset", "pcie_eth_reset",
				      "pcie_m_2_reset",    "pcie_mini1_reset";

			clkreq-gpios = <&gpio20 6 0 >, <&gpio27 3 0 >,
				       <&gpio17 0 0 >;

			clkreq-names = "pcie_eth_clkreq", "pcie_m_2_clkreq",
				       "pcie_mini1_clkreq";

			/* vboost iboost pre post main */
			eye_param = <0xFFFFFFFF 0xFFFFFFFF 0xFFFFFFFF 0xFFFFFFFF 0xFFFFFFFF>;

			pinctrl-names = "default";
			pinctrl-0 = <&pcie_clkreq_pmx_func &pcie_clkreq_cfg_func>;
		};
	};

Would this be ok?

---

FYI, the current driver is on this branch:

	https://github.com/mchehab/linux/tree/staging/hikey970-pcie-v2

The DTS file is at:

	https://github.com/mchehab/linux/blob/staging/hikey970-pcie-v2/arch/arm64/boot/dts/hisilicon/hi3670.dtsi

The DT bindings at:
	https://github.com/mchehab/linux/blob/staging/hikey970-pcie-v2/Documentation/devicetree/bindings/pci/hisilicon%2Ckirin-pcie.yaml

And the driver, modified to support Hikey 970, at:
	https://github.com/mchehab/linux/blob/staging/hikey970-pcie-v2/drivers/pci/controller/dwc/pcie-kirin.c

Thanks,
Mauro