[RFC PATCH 0/3] PCI: Define slot-power-limit DT property

[RFC PATCH 0/3] PCI: Define slot-power-limit DT property

[Pali Rohár]

This is RFC patch series which define a new PCIe slot-power-limit DT
property (first patch) and show example how to use it in pci-aardvark.c
driver (second and third patch). Second and third patches depends on other
pci-aardvark patches which are under review. I included them here just to
show how this new slot-power-limit DT property can be used.

Please let me know what do you think about this approach of defining Slot
Power Limit in DTS file.

Pali Rohár (3):
  dt-bindings: Add 'slot-power-limit' PCIe port property
  PCI: aardvark: Add support for sending Set_Slot_Power_Limit message
  arm64: dts: armada-3720-turris-mox: Define slot-power-limit for PCIe

 .../devicetree/bindings/pci/aardvark-pci.txt  |  2 +
 Documentation/devicetree/bindings/pci/pci.txt |  6 ++
 .../dts/marvell/armada-3720-turris-mox.dts    |  1 +
 drivers/pci/controller/pci-aardvark.c         | 66 ++++++++++++++++++-
 4 files changed, 74 insertions(+), 1 deletion(-)

-- 
2.20.1

[RFC PATCH 1/3] dt-bindings: Add 'slot-power-limit' PCIe port property

[Pali Rohár]

This property specifies slot power limit in mW unit. It is form-factor and
board specific value and must be initialized by hardware.

Some PCIe controllers delegates this work to software to allow hardware
flexibility and therefore this property basically specifies what should
host bridge programs into PCIe Slot Capabilities registers.

Property needs to be specified in mW unit, and not in special format
defined by Slot Capabilities (which encodes scaling factor or different
unit). Host drivers should convert value from mW unit to their format.

Signed-off-by: Pali Rohár <pali@kernel.org>
---
 Documentation/devicetree/bindings/pci/pci.txt | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/Documentation/devicetree/bindings/pci/pci.txt b/Documentation/devicetree/bindings/pci/pci.txt
index 6a8f2874a24d..e67d5db21514 100644
--- a/Documentation/devicetree/bindings/pci/pci.txt
+++ b/Documentation/devicetree/bindings/pci/pci.txt
@@ -32,6 +32,12 @@ driver implementation may support the following properties:
    root port to downstream device and host bridge drivers can do programming
    which depends on CLKREQ signal existence. For example, programming root port
    not to advertise ASPM L1 Sub-States support if there is no CLKREQ signal.
+- slot-power-limit:
+   If present this property specifies slot power limit in mW unit. Host drivers
+   can parse this slot power limit and use it for programming Root Port or host
+   bridge, or for composing and sending PCIe Set_Slot_Power_Limit message
+   through the Root Port or host bridge when transitioning PCIe link from a
+   non-DL_Up Status to a DL_Up Status.
 
 PCI-PCI Bridge properties
 -------------------------
-- 
2.20.1

[RFC PATCH 2/3] PCI: aardvark: Add support for sending Set_Slot_Power_Limit message

[Pali Rohár]

According to PCIe Base specification 3.0, when transitioning from a
non-DL_Up Status to a DL_Up Status, the Port must initiate the
transmission of a Set_Slot_Power_Limit Message to the other component
on the Link to convey the value programmed in the Slot Power Limit
Scale and Value fields of the Slot Capabilities register. This
Transmission is optional if the Slot Capabilities register has not
yet been initialized.

As PCIe Root Bridge is emulated by kernel emulate readback of Slot Power
Limit Scale and Value bits in Slot Capabilities register.

Also send that Set_Slot_Power_Limit message via Message Generation Control
Register in Link Up handler when link changes from down to up state and
slot power limit value was defined.

Slot power limit value is read from DT property 'slot-power-limit' which
value is in mW unit. When this DT property is not specified then it is
treated as "Slot Capabilities register has not yet been initialized".

Signed-off-by: Pali Rohár <pali@kernel.org>
---
 .../devicetree/bindings/pci/aardvark-pci.txt  |  2 +
 drivers/pci/controller/pci-aardvark.c         | 66 ++++++++++++++++++-
 2 files changed, 67 insertions(+), 1 deletion(-)

diff --git a/Documentation/devicetree/bindings/pci/aardvark-pci.txt b/Documentation/devicetree/bindings/pci/aardvark-pci.txt
index 2b8ca920a7fa..bb658f261db0 100644
--- a/Documentation/devicetree/bindings/pci/aardvark-pci.txt
+++ b/Documentation/devicetree/bindings/pci/aardvark-pci.txt
@@ -20,6 +20,7 @@ contain the following properties:
    define the mapping of the PCIe interface to interrupt numbers.
  - bus-range: PCI bus numbers covered
  - phys: the PCIe PHY handle
+ - slot-power-limit: see pci.txt
  - max-link-speed: see pci.txt
  - reset-gpios: see pci.txt
 
@@ -52,6 +53,7 @@ Example:
 				<0 0 0 3 &pcie_intc 2>,
 				<0 0 0 4 &pcie_intc 3>;
 		phys = <&comphy1 0>;
+		slot-power-limit: <10000>;
 		pcie_intc: interrupt-controller {
 			interrupt-controller;
 			#interrupt-cells = <1>;
diff --git a/drivers/pci/controller/pci-aardvark.c b/drivers/pci/controller/pci-aardvark.c
index f94898f6072f..cf704c199c15 100644
--- a/drivers/pci/controller/pci-aardvark.c
+++ b/drivers/pci/controller/pci-aardvark.c
@@ -166,6 +166,11 @@
 #define VENDOR_ID_REG				(LMI_BASE_ADDR + 0x44)
 #define DEBUG_MUX_CTRL_REG			(LMI_BASE_ADDR + 0x208)
 #define     DIS_ORD_CHK				BIT(30)
+#define PME_MSG_GEN_CTRL			(LMI_BASE_ADDR + 0x220)
+#define     SEND_SET_SLOT_POWER_LIMIT		BIT(13)
+#define     SEND_PME_TURN_OFF			BIT(14)
+#define     SLOT_POWER_LIMIT_DATA_SHIFT		16
+#define     SLOT_POWER_LIMIT_DATA_MASK		GENMASK(25, 16)
 
 /* PCIe core controller registers */
 #define CTRL_CORE_BASE_ADDR			0x18000
@@ -267,6 +272,7 @@ static bool advk_pcie_link_up(struct advk_pcie *pcie)
 {
 	u32 val, ltssm_state;
 	u16 slotsta, slotctl;
+	u32 slotpwr;
 	bool link_up;
 
 	val = advk_readl(pcie, CFG_REG);
@@ -277,7 +283,25 @@ static bool advk_pcie_link_up(struct advk_pcie *pcie)
 		pcie->link_up = true;
 		slotsta = le16_to_cpu(pcie->bridge.pcie_conf.slotsta);
 		slotctl = le16_to_cpu(pcie->bridge.pcie_conf.slotctl);
+		slotpwr = (le32_to_cpu(pcie->bridge.pcie_conf.slotcap) &
+			   (PCI_EXP_SLTCAP_SPLV | PCI_EXP_SLTCAP_SPLS)) >> 7;
 		pcie->bridge.pcie_conf.slotsta = cpu_to_le16(slotsta | PCI_EXP_SLTSTA_DLLSC);
+		if (!(slotctl & PCI_EXP_SLTCTL_ASPL_DISABLE) && slotpwr) {
+			/*
+			 * According to PCIe Base specification 3.0, when transitioning from a
+			 * non-DL_Up Status to a DL_Up Status, the Port must initiate the
+			 * transmission of a Set_Slot_Power_Limit Message to the other component
+			 * on the Link to convey the value programmed in the Slot Power Limit
+			 * Scale and Value fields of the Slot Capabilities register. This
+			 * Transmission is optional if the Slot Capabilities register has not
+			 * yet been initialized.
+			 */
+			val = advk_readl(pcie, PME_MSG_GEN_CTRL);
+			val &= ~SLOT_POWER_LIMIT_DATA_MASK;
+			val |= slotpwr << SLOT_POWER_LIMIT_DATA_SHIFT;
+			val |= SEND_SET_SLOT_POWER_LIMIT;
+			advk_writel(pcie, val, PME_MSG_GEN_CTRL);
+		}
 		if ((slotctl & PCI_EXP_SLTCTL_DLLSCE) && (slotctl & PCI_EXP_SLTCTL_HPIE))
 			mod_timer(&pcie->link_up_irq_timer, jiffies + 1);
 	}
@@ -956,6 +980,9 @@ static struct pci_bridge_emul_ops advk_pci_bridge_emul_ops = {
 static int advk_sw_pci_bridge_init(struct advk_pcie *pcie)
 {
 	struct pci_bridge_emul *bridge = &pcie->bridge;
+	struct device *dev = &pcie->pdev->dev;
+	u8 slot_power_limit_scale, slot_power_limit_value;
+	u32 slot_power_limit;
 	int ret;
 
 	bridge->conf.vendor =
@@ -988,6 +1015,40 @@ static int advk_sw_pci_bridge_init(struct advk_pcie *pcie)
 	/* Indicates supports for Completion Retry Status */
 	bridge->pcie_conf.rootcap = cpu_to_le16(PCI_EXP_RTCAP_CRSVIS);
 
+	if (of_property_read_u32(dev->of_node, "slot-power-limit", &slot_power_limit))
+		slot_power_limit = 0;
+
+	if (slot_power_limit)
+		dev_info(dev, "Slot power limit %u.%uW\n", slot_power_limit / 1000,
+			 (slot_power_limit / 100) % 10);
+
+	/* Calculate Slot Power Limit Value and Slot Power Limit Scale */
+	if (slot_power_limit == 0) {
+		slot_power_limit_scale = 0;
+		slot_power_limit_value = 0x00;
+	} else if (slot_power_limit <= 255) {
+		slot_power_limit_scale = 3;
+		slot_power_limit_value = slot_power_limit;
+	} else if (slot_power_limit <= 255*10) {
+		slot_power_limit_scale = 2;
+		slot_power_limit_value = slot_power_limit / 10;
+	} else if (slot_power_limit <= 255*100) {
+		slot_power_limit_scale = 1;
+		slot_power_limit_value = slot_power_limit / 100;
+	} else if (slot_power_limit <= 239*1000) {
+		slot_power_limit_scale = 0;
+		slot_power_limit_value = slot_power_limit / 1000;
+	} else if (slot_power_limit <= 250*1000) {
+		slot_power_limit_scale = 0;
+		slot_power_limit_value = 0xF0;
+	} else if (slot_power_limit <= 275*1000) {
+		slot_power_limit_scale = 0;
+		slot_power_limit_value = 0xF1;
+	} else {
+		slot_power_limit_scale = 0;
+		slot_power_limit_value = 0xF2;
+	}
+
 	/*
 	 * Mark bridge as Hot-Plug Capable to allow delivering Data Link Layer
 	 * State Changed interrupt. No Command Completed Support is set because
@@ -996,7 +1057,10 @@ static int advk_sw_pci_bridge_init(struct advk_pcie *pcie)
 	 * bit permanently as there is no support for unplugging PCIe card from
 	 * the slot. Assume that PCIe card is always connected in slot.
 	 */
-	bridge->pcie_conf.slotcap = cpu_to_le32(PCI_EXP_SLTCAP_NCCS | PCI_EXP_SLTCAP_HPC);
+	bridge->pcie_conf.slotcap = cpu_to_le32(PCI_EXP_SLTCAP_NCCS |
+						PCI_EXP_SLTCAP_HPC |
+						slot_power_limit_value << 7 |
+						slot_power_limit_scale << 15);
 	bridge->pcie_conf.slotsta = cpu_to_le16(PCI_EXP_SLTSTA_PDS);
 
 	return 0;
-- 
2.20.1

[RFC PATCH 3/3] arm64: dts: armada-3720-turris-mox: Define slot-power-limit for PCIe

[Pali Rohár]

PCIe Slot Power Limit on Turris Mox is 10W.

Signed-off-by: Pali Rohár <pali@kernel.org>
---
 arch/arm64/boot/dts/marvell/armada-3720-turris-mox.dts | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm64/boot/dts/marvell/armada-3720-turris-mox.dts b/arch/arm64/boot/dts/marvell/armada-3720-turris-mox.dts
index 86b3025f174b..1db928dff9ec 100644
--- a/arch/arm64/boot/dts/marvell/armada-3720-turris-mox.dts
+++ b/arch/arm64/boot/dts/marvell/armada-3720-turris-mox.dts
@@ -134,6 +134,7 @@
 	pinctrl-0 = <&pcie_reset_pins &pcie_clkreq_pins>;
 	status = "okay";
 	reset-gpios = <&gpiosb 3 GPIO_ACTIVE_LOW>;
+	slot-power-limit = <10000>;
 	/*
 	 * U-Boot port for Turris Mox has a bug which always expects that "ranges" DT property
 	 * contains exactly 2 ranges with 3 (child) address cells, 2 (parent) address cells and
-- 
2.20.1

Re: [RFC PATCH 1/3] dt-bindings: Add 'slot-power-limit' PCIe port property

[Marek Behún]

On Fri, 20 Aug 2021 18:00:21 +0200
Pali Rohár <pali@kernel.org> wrote:

> This property specifies slot power limit in mW unit. It is form-factor and
> board specific value and must be initialized by hardware.
> 
> Some PCIe controllers delegates this work to software to allow hardware
> flexibility and therefore this property basically specifies what should
> host bridge programs into PCIe Slot Capabilities registers.
> 
> Property needs to be specified in mW unit, and not in special format
> defined by Slot Capabilities (which encodes scaling factor or different
> unit). Host drivers should convert value from mW unit to their format.
> 
> Signed-off-by: Pali Rohár <pali@kernel.org>
> ---
>  Documentation/devicetree/bindings/pci/pci.txt | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/Documentation/devicetree/bindings/pci/pci.txt b/Documentation/devicetree/bindings/pci/pci.txt
> index 6a8f2874a24d..e67d5db21514 100644
> --- a/Documentation/devicetree/bindings/pci/pci.txt
> +++ b/Documentation/devicetree/bindings/pci/pci.txt
> @@ -32,6 +32,12 @@ driver implementation may support the following properties:
>     root port to downstream device and host bridge drivers can do programming
>     which depends on CLKREQ signal existence. For example, programming root port
>     not to advertise ASPM L1 Sub-States support if there is no CLKREQ signal.
> +- slot-power-limit:
> +   If present this property specifies slot power limit in mW unit. Host drivers
> +   can parse this slot power limit and use it for programming Root Port or host
> +   bridge, or for composing and sending PCIe Set_Slot_Power_Limit message
> +   through the Root Port or host bridge when transitioning PCIe link from a
> +   non-DL_Up Status to a DL_Up Status.
>  
>  PCI-PCI Bridge properties
>  -------------------------

SFP uses something similar for maximum power, but since the units are
in milliwatt, the name of the property is maximum-power-milliwatt, to
avoid problems. I think it should be done here as well, i.e.
  slot-maximum-power-milliwatt
or maybe even
  maximum-power-milliwatt.

Marek

Re: [RFC PATCH 1/3] dt-bindings: Add 'slot-power-limit' PCIe port property

[Rob Herring]

On Fri, Aug 20, 2021 at 06:00:21PM +0200, Pali Rohár wrote:
> This property specifies slot power limit in mW unit. It is form-factor and
> board specific value and must be initialized by hardware.
> 
> Some PCIe controllers delegates this work to software to allow hardware
> flexibility and therefore this property basically specifies what should
> host bridge programs into PCIe Slot Capabilities registers.
> 
> Property needs to be specified in mW unit, and not in special format
> defined by Slot Capabilities (which encodes scaling factor or different
> unit). Host drivers should convert value from mW unit to their format.
> 
> Signed-off-by: Pali Rohár <pali@kernel.org>
> ---
>  Documentation/devicetree/bindings/pci/pci.txt | 6 ++++++
>  1 file changed, 6 insertions(+)

This needs to be in dtschema schemas/pci/pci-bus.yaml instead.

(pci.txt is still here because it needs to be relicensed to move all the 
descriptions to pci-bus.yaml.)

> 
> diff --git a/Documentation/devicetree/bindings/pci/pci.txt b/Documentation/devicetree/bindings/pci/pci.txt
> index 6a8f2874a24d..e67d5db21514 100644
> --- a/Documentation/devicetree/bindings/pci/pci.txt
> +++ b/Documentation/devicetree/bindings/pci/pci.txt
> @@ -32,6 +32,12 @@ driver implementation may support the following properties:
>     root port to downstream device and host bridge drivers can do programming
>     which depends on CLKREQ signal existence. For example, programming root port
>     not to advertise ASPM L1 Sub-States support if there is no CLKREQ signal.
> +- slot-power-limit:
> +   If present this property specifies slot power limit in mW unit. Host drivers

As mentioned, this should have a unit suffix. I'm not sure it is 
beneficial to share with SFP in this case though.

> +   can parse this slot power limit and use it for programming Root Port or host
> +   bridge, or for composing and sending PCIe Set_Slot_Power_Limit message
> +   through the Root Port or host bridge when transitioning PCIe link from a
> +   non-DL_Up Status to a DL_Up Status.

I no nothing about how this mechanism works, but I think this belongs in 
the next section as for PCIe, a slot is always below a PCI-PCI bridge. 
If we have N slots, then there's N bridges and needs to be N 
slot-power-limit properties, right?

(The same is probably true for all the properties here except 
linux,pci-domain.) There's no distinction between host and PCI bridges  
in pci-bus.yaml though.

Rob

Re: [RFC PATCH 2/3] PCI: aardvark: Add support for sending Set_Slot_Power_Limit message

[Rob Herring]

On Fri, Aug 20, 2021 at 06:00:22PM +0200, Pali Rohár wrote:
> According to PCIe Base specification 3.0, when transitioning from a
> non-DL_Up Status to a DL_Up Status, the Port must initiate the
> transmission of a Set_Slot_Power_Limit Message to the other component
> on the Link to convey the value programmed in the Slot Power Limit
> Scale and Value fields of the Slot Capabilities register. This
> Transmission is optional if the Slot Capabilities register has not
> yet been initialized.
> 
> As PCIe Root Bridge is emulated by kernel emulate readback of Slot Power
> Limit Scale and Value bits in Slot Capabilities register.
> 
> Also send that Set_Slot_Power_Limit message via Message Generation Control
> Register in Link Up handler when link changes from down to up state and
> slot power limit value was defined.
> 
> Slot power limit value is read from DT property 'slot-power-limit' which
> value is in mW unit. When this DT property is not specified then it is
> treated as "Slot Capabilities register has not yet been initialized".
> 
> Signed-off-by: Pali Rohár <pali@kernel.org>
> ---
>  .../devicetree/bindings/pci/aardvark-pci.txt  |  2 +
>  drivers/pci/controller/pci-aardvark.c         | 66 ++++++++++++++++++-
>  2 files changed, 67 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/devicetree/bindings/pci/aardvark-pci.txt b/Documentation/devicetree/bindings/pci/aardvark-pci.txt
> index 2b8ca920a7fa..bb658f261db0 100644
> --- a/Documentation/devicetree/bindings/pci/aardvark-pci.txt
> +++ b/Documentation/devicetree/bindings/pci/aardvark-pci.txt
> @@ -20,6 +20,7 @@ contain the following properties:
>     define the mapping of the PCIe interface to interrupt numbers.
>   - bus-range: PCI bus numbers covered
>   - phys: the PCIe PHY handle
> + - slot-power-limit: see pci.txt
>   - max-link-speed: see pci.txt
>   - reset-gpios: see pci.txt
>  
> @@ -52,6 +53,7 @@ Example:
>  				<0 0 0 3 &pcie_intc 2>,
>  				<0 0 0 4 &pcie_intc 3>;
>  		phys = <&comphy1 0>;
> +		slot-power-limit: <10000>;
>  		pcie_intc: interrupt-controller {
>  			interrupt-controller;
>  			#interrupt-cells = <1>;
> diff --git a/drivers/pci/controller/pci-aardvark.c b/drivers/pci/controller/pci-aardvark.c
> index f94898f6072f..cf704c199c15 100644
> --- a/drivers/pci/controller/pci-aardvark.c
> +++ b/drivers/pci/controller/pci-aardvark.c
> @@ -166,6 +166,11 @@
>  #define VENDOR_ID_REG				(LMI_BASE_ADDR + 0x44)
>  #define DEBUG_MUX_CTRL_REG			(LMI_BASE_ADDR + 0x208)
>  #define     DIS_ORD_CHK				BIT(30)
> +#define PME_MSG_GEN_CTRL			(LMI_BASE_ADDR + 0x220)
> +#define     SEND_SET_SLOT_POWER_LIMIT		BIT(13)
> +#define     SEND_PME_TURN_OFF			BIT(14)
> +#define     SLOT_POWER_LIMIT_DATA_SHIFT		16
> +#define     SLOT_POWER_LIMIT_DATA_MASK		GENMASK(25, 16)
>  
>  /* PCIe core controller registers */
>  #define CTRL_CORE_BASE_ADDR			0x18000
> @@ -267,6 +272,7 @@ static bool advk_pcie_link_up(struct advk_pcie *pcie)
>  {
>  	u32 val, ltssm_state;
>  	u16 slotsta, slotctl;
> +	u32 slotpwr;
>  	bool link_up;
>  
>  	val = advk_readl(pcie, CFG_REG);
> @@ -277,7 +283,25 @@ static bool advk_pcie_link_up(struct advk_pcie *pcie)
>  		pcie->link_up = true;
>  		slotsta = le16_to_cpu(pcie->bridge.pcie_conf.slotsta);
>  		slotctl = le16_to_cpu(pcie->bridge.pcie_conf.slotctl);
> +		slotpwr = (le32_to_cpu(pcie->bridge.pcie_conf.slotcap) &
> +			   (PCI_EXP_SLTCAP_SPLV | PCI_EXP_SLTCAP_SPLS)) >> 7;
>  		pcie->bridge.pcie_conf.slotsta = cpu_to_le16(slotsta | PCI_EXP_SLTSTA_DLLSC);
> +		if (!(slotctl & PCI_EXP_SLTCTL_ASPL_DISABLE) && slotpwr) {
> +			/*
> +			 * According to PCIe Base specification 3.0, when transitioning from a
> +			 * non-DL_Up Status to a DL_Up Status, the Port must initiate the
> +			 * transmission of a Set_Slot_Power_Limit Message to the other component
> +			 * on the Link to convey the value programmed in the Slot Power Limit
> +			 * Scale and Value fields of the Slot Capabilities register. This
> +			 * Transmission is optional if the Slot Capabilities register has not
> +			 * yet been initialized.
> +			 */
> +			val = advk_readl(pcie, PME_MSG_GEN_CTRL);
> +			val &= ~SLOT_POWER_LIMIT_DATA_MASK;
> +			val |= slotpwr << SLOT_POWER_LIMIT_DATA_SHIFT;
> +			val |= SEND_SET_SLOT_POWER_LIMIT;
> +			advk_writel(pcie, val, PME_MSG_GEN_CTRL);
> +		}
>  		if ((slotctl & PCI_EXP_SLTCTL_DLLSCE) && (slotctl & PCI_EXP_SLTCTL_HPIE))
>  			mod_timer(&pcie->link_up_irq_timer, jiffies + 1);
>  	}
> @@ -956,6 +980,9 @@ static struct pci_bridge_emul_ops advk_pci_bridge_emul_ops = {
>  static int advk_sw_pci_bridge_init(struct advk_pcie *pcie)
>  {
>  	struct pci_bridge_emul *bridge = &pcie->bridge;
> +	struct device *dev = &pcie->pdev->dev;
> +	u8 slot_power_limit_scale, slot_power_limit_value;
> +	u32 slot_power_limit;
>  	int ret;
>  
>  	bridge->conf.vendor =
> @@ -988,6 +1015,40 @@ static int advk_sw_pci_bridge_init(struct advk_pcie *pcie)
>  	/* Indicates supports for Completion Retry Status */
>  	bridge->pcie_conf.rootcap = cpu_to_le16(PCI_EXP_RTCAP_CRSVIS);
>  

> +	if (of_property_read_u32(dev->of_node, "slot-power-limit", &slot_power_limit))
> +		slot_power_limit = 0;
> +
> +	if (slot_power_limit)
> +		dev_info(dev, "Slot power limit %u.%uW\n", slot_power_limit / 1000,
> +			 (slot_power_limit / 100) % 10);

Add a wrapper function for this.

> +
> +	/* Calculate Slot Power Limit Value and Slot Power Limit Scale */
> +	if (slot_power_limit == 0) {
> +		slot_power_limit_scale = 0;
> +		slot_power_limit_value = 0x00;
> +	} else if (slot_power_limit <= 255) {
> +		slot_power_limit_scale = 3;
> +		slot_power_limit_value = slot_power_limit;
> +	} else if (slot_power_limit <= 255*10) {
> +		slot_power_limit_scale = 2;
> +		slot_power_limit_value = slot_power_limit / 10;
> +	} else if (slot_power_limit <= 255*100) {
> +		slot_power_limit_scale = 1;
> +		slot_power_limit_value = slot_power_limit / 100;
> +	} else if (slot_power_limit <= 239*1000) {
> +		slot_power_limit_scale = 0;
> +		slot_power_limit_value = slot_power_limit / 1000;
> +	} else if (slot_power_limit <= 250*1000) {
> +		slot_power_limit_scale = 0;
> +		slot_power_limit_value = 0xF0;
> +	} else if (slot_power_limit <= 275*1000) {
> +		slot_power_limit_scale = 0;
> +		slot_power_limit_value = 0xF1;
> +	} else {
> +		slot_power_limit_scale = 0;
> +		slot_power_limit_value = 0xF2;
> +	}

This all looks like it should be common code.

> +
>  	/*
>  	 * Mark bridge as Hot-Plug Capable to allow delivering Data Link Layer
>  	 * State Changed interrupt. No Command Completed Support is set because
> @@ -996,7 +1057,10 @@ static int advk_sw_pci_bridge_init(struct advk_pcie *pcie)
>  	 * bit permanently as there is no support for unplugging PCIe card from
>  	 * the slot. Assume that PCIe card is always connected in slot.
>  	 */
> -	bridge->pcie_conf.slotcap = cpu_to_le32(PCI_EXP_SLTCAP_NCCS | PCI_EXP_SLTCAP_HPC);
> +	bridge->pcie_conf.slotcap = cpu_to_le32(PCI_EXP_SLTCAP_NCCS |
> +						PCI_EXP_SLTCAP_HPC |

While not new, how does the driver know the board is hotplug capable. 
IIRC, we have a property for that.

> +						slot_power_limit_value << 7 |
> +						slot_power_limit_scale << 15);
>  	bridge->pcie_conf.slotsta = cpu_to_le16(PCI_EXP_SLTSTA_PDS);
>  
>  	return 0;
> -- 
> 2.20.1
> 
> 

Re: [RFC PATCH 1/3] dt-bindings: Add 'slot-power-limit' PCIe port property

[Pali Rohár]

On Tuesday 24 August 2021 10:35:34 Rob Herring wrote:
> On Fri, Aug 20, 2021 at 06:00:21PM +0200, Pali Rohár wrote:
> > This property specifies slot power limit in mW unit. It is form-factor and
> > board specific value and must be initialized by hardware.
> > 
> > Some PCIe controllers delegates this work to software to allow hardware
> > flexibility and therefore this property basically specifies what should
> > host bridge programs into PCIe Slot Capabilities registers.
> > 
> > Property needs to be specified in mW unit, and not in special format
> > defined by Slot Capabilities (which encodes scaling factor or different
> > unit). Host drivers should convert value from mW unit to their format.
> > 
> > Signed-off-by: Pali Rohár <pali@kernel.org>
> > ---
> >  Documentation/devicetree/bindings/pci/pci.txt | 6 ++++++
> >  1 file changed, 6 insertions(+)
> 
> This needs to be in dtschema schemas/pci/pci-bus.yaml instead.
> 
> (pci.txt is still here because it needs to be relicensed to move all the 
> descriptions to pci-bus.yaml.)

Ok, this is just a proposal for a new DTS property. So documentation
issues will be fixed in real patch.

> > 
> > diff --git a/Documentation/devicetree/bindings/pci/pci.txt b/Documentation/devicetree/bindings/pci/pci.txt
> > index 6a8f2874a24d..e67d5db21514 100644
> > --- a/Documentation/devicetree/bindings/pci/pci.txt
> > +++ b/Documentation/devicetree/bindings/pci/pci.txt
> > @@ -32,6 +32,12 @@ driver implementation may support the following properties:
> >     root port to downstream device and host bridge drivers can do programming
> >     which depends on CLKREQ signal existence. For example, programming root port
> >     not to advertise ASPM L1 Sub-States support if there is no CLKREQ signal.
> > +- slot-power-limit:
> > +   If present this property specifies slot power limit in mW unit. Host drivers
> 
> As mentioned, this should have a unit suffix. I'm not sure it is 
> beneficial to share with SFP in this case though.
> 
> > +   can parse this slot power limit and use it for programming Root Port or host
> > +   bridge, or for composing and sending PCIe Set_Slot_Power_Limit message
> > +   through the Root Port or host bridge when transitioning PCIe link from a
> > +   non-DL_Up Status to a DL_Up Status.
> 
> I no nothing about how this mechanism works, but I think this belongs in 
> the next section as for PCIe, a slot is always below a PCI-PCI bridge. 
> If we have N slots, then there's N bridges and needs to be N 
> slot-power-limit properties, right?
> 
> (The same is probably true for all the properties here except 
> linux,pci-domain.) There's no distinction between host and PCI bridges  
> in pci-bus.yaml though.
> 
> Rob

This slot-power-limit property belongs to same place where are also
other slot properties (link speed, reset gpios, clkreq). So I put it in
place where others are.

But I'm not sure where it should be as it affects link/slot. Because
link has two sides. I guess that link speed and slot power limit could
belong to the root/downstream port and reset gpio to the endpoint card
or upstream port...

Re: [RFC PATCH 2/3] PCI: aardvark: Add support for sending Set_Slot_Power_Limit message

[Pali Rohár]

On Tuesday 24 August 2021 10:58:31 Rob Herring wrote:
> On Fri, Aug 20, 2021 at 06:00:22PM +0200, Pali Rohár wrote:
> > According to PCIe Base specification 3.0, when transitioning from a
> > non-DL_Up Status to a DL_Up Status, the Port must initiate the
> > transmission of a Set_Slot_Power_Limit Message to the other component
> > on the Link to convey the value programmed in the Slot Power Limit
> > Scale and Value fields of the Slot Capabilities register. This
> > Transmission is optional if the Slot Capabilities register has not
> > yet been initialized.
> > 
> > As PCIe Root Bridge is emulated by kernel emulate readback of Slot Power
> > Limit Scale and Value bits in Slot Capabilities register.
> > 
> > Also send that Set_Slot_Power_Limit message via Message Generation Control
> > Register in Link Up handler when link changes from down to up state and
> > slot power limit value was defined.
> > 
> > Slot power limit value is read from DT property 'slot-power-limit' which
> > value is in mW unit. When this DT property is not specified then it is
> > treated as "Slot Capabilities register has not yet been initialized".
> > 
> > Signed-off-by: Pali Rohár <pali@kernel.org>
> > ---
> >  .../devicetree/bindings/pci/aardvark-pci.txt  |  2 +
> >  drivers/pci/controller/pci-aardvark.c         | 66 ++++++++++++++++++-
> >  2 files changed, 67 insertions(+), 1 deletion(-)
> > 
> > diff --git a/Documentation/devicetree/bindings/pci/aardvark-pci.txt b/Documentation/devicetree/bindings/pci/aardvark-pci.txt
> > index 2b8ca920a7fa..bb658f261db0 100644
> > --- a/Documentation/devicetree/bindings/pci/aardvark-pci.txt
> > +++ b/Documentation/devicetree/bindings/pci/aardvark-pci.txt
> > @@ -20,6 +20,7 @@ contain the following properties:
> >     define the mapping of the PCIe interface to interrupt numbers.
> >   - bus-range: PCI bus numbers covered
> >   - phys: the PCIe PHY handle
> > + - slot-power-limit: see pci.txt
> >   - max-link-speed: see pci.txt
> >   - reset-gpios: see pci.txt
> >  
> > @@ -52,6 +53,7 @@ Example:
> >  				<0 0 0 3 &pcie_intc 2>,
> >  				<0 0 0 4 &pcie_intc 3>;
> >  		phys = <&comphy1 0>;
> > +		slot-power-limit: <10000>;
> >  		pcie_intc: interrupt-controller {
> >  			interrupt-controller;
> >  			#interrupt-cells = <1>;
> > diff --git a/drivers/pci/controller/pci-aardvark.c b/drivers/pci/controller/pci-aardvark.c
> > index f94898f6072f..cf704c199c15 100644
> > --- a/drivers/pci/controller/pci-aardvark.c
> > +++ b/drivers/pci/controller/pci-aardvark.c
> > @@ -166,6 +166,11 @@
> >  #define VENDOR_ID_REG				(LMI_BASE_ADDR + 0x44)
> >  #define DEBUG_MUX_CTRL_REG			(LMI_BASE_ADDR + 0x208)
> >  #define     DIS_ORD_CHK				BIT(30)
> > +#define PME_MSG_GEN_CTRL			(LMI_BASE_ADDR + 0x220)
> > +#define     SEND_SET_SLOT_POWER_LIMIT		BIT(13)
> > +#define     SEND_PME_TURN_OFF			BIT(14)
> > +#define     SLOT_POWER_LIMIT_DATA_SHIFT		16
> > +#define     SLOT_POWER_LIMIT_DATA_MASK		GENMASK(25, 16)
> >  
> >  /* PCIe core controller registers */
> >  #define CTRL_CORE_BASE_ADDR			0x18000
> > @@ -267,6 +272,7 @@ static bool advk_pcie_link_up(struct advk_pcie *pcie)
> >  {
> >  	u32 val, ltssm_state;
> >  	u16 slotsta, slotctl;
> > +	u32 slotpwr;
> >  	bool link_up;
> >  
> >  	val = advk_readl(pcie, CFG_REG);
> > @@ -277,7 +283,25 @@ static bool advk_pcie_link_up(struct advk_pcie *pcie)
> >  		pcie->link_up = true;
> >  		slotsta = le16_to_cpu(pcie->bridge.pcie_conf.slotsta);
> >  		slotctl = le16_to_cpu(pcie->bridge.pcie_conf.slotctl);
> > +		slotpwr = (le32_to_cpu(pcie->bridge.pcie_conf.slotcap) &
> > +			   (PCI_EXP_SLTCAP_SPLV | PCI_EXP_SLTCAP_SPLS)) >> 7;
> >  		pcie->bridge.pcie_conf.slotsta = cpu_to_le16(slotsta | PCI_EXP_SLTSTA_DLLSC);
> > +		if (!(slotctl & PCI_EXP_SLTCTL_ASPL_DISABLE) && slotpwr) {
> > +			/*
> > +			 * According to PCIe Base specification 3.0, when transitioning from a
> > +			 * non-DL_Up Status to a DL_Up Status, the Port must initiate the
> > +			 * transmission of a Set_Slot_Power_Limit Message to the other component
> > +			 * on the Link to convey the value programmed in the Slot Power Limit
> > +			 * Scale and Value fields of the Slot Capabilities register. This
> > +			 * Transmission is optional if the Slot Capabilities register has not
> > +			 * yet been initialized.
> > +			 */
> > +			val = advk_readl(pcie, PME_MSG_GEN_CTRL);
> > +			val &= ~SLOT_POWER_LIMIT_DATA_MASK;
> > +			val |= slotpwr << SLOT_POWER_LIMIT_DATA_SHIFT;
> > +			val |= SEND_SET_SLOT_POWER_LIMIT;
> > +			advk_writel(pcie, val, PME_MSG_GEN_CTRL);
> > +		}
> >  		if ((slotctl & PCI_EXP_SLTCTL_DLLSCE) && (slotctl & PCI_EXP_SLTCTL_HPIE))
> >  			mod_timer(&pcie->link_up_irq_timer, jiffies + 1);
> >  	}
> > @@ -956,6 +980,9 @@ static struct pci_bridge_emul_ops advk_pci_bridge_emul_ops = {
> >  static int advk_sw_pci_bridge_init(struct advk_pcie *pcie)
> >  {
> >  	struct pci_bridge_emul *bridge = &pcie->bridge;
> > +	struct device *dev = &pcie->pdev->dev;
> > +	u8 slot_power_limit_scale, slot_power_limit_value;
> > +	u32 slot_power_limit;
> >  	int ret;
> >  
> >  	bridge->conf.vendor =
> > @@ -988,6 +1015,40 @@ static int advk_sw_pci_bridge_init(struct advk_pcie *pcie)
> >  	/* Indicates supports for Completion Retry Status */
> >  	bridge->pcie_conf.rootcap = cpu_to_le16(PCI_EXP_RTCAP_CRSVIS);
> >  
> 
> > +	if (of_property_read_u32(dev->of_node, "slot-power-limit", &slot_power_limit))
> > +		slot_power_limit = 0;
> > +
> > +	if (slot_power_limit)
> > +		dev_info(dev, "Slot power limit %u.%uW\n", slot_power_limit / 1000,
> > +			 (slot_power_limit / 100) % 10);
> 
> Add a wrapper function for this.
> 
> > +
> > +	/* Calculate Slot Power Limit Value and Slot Power Limit Scale */
> > +	if (slot_power_limit == 0) {
> > +		slot_power_limit_scale = 0;
> > +		slot_power_limit_value = 0x00;
> > +	} else if (slot_power_limit <= 255) {
> > +		slot_power_limit_scale = 3;
> > +		slot_power_limit_value = slot_power_limit;
> > +	} else if (slot_power_limit <= 255*10) {
> > +		slot_power_limit_scale = 2;
> > +		slot_power_limit_value = slot_power_limit / 10;
> > +	} else if (slot_power_limit <= 255*100) {
> > +		slot_power_limit_scale = 1;
> > +		slot_power_limit_value = slot_power_limit / 100;
> > +	} else if (slot_power_limit <= 239*1000) {
> > +		slot_power_limit_scale = 0;
> > +		slot_power_limit_value = slot_power_limit / 1000;
> > +	} else if (slot_power_limit <= 250*1000) {
> > +		slot_power_limit_scale = 0;
> > +		slot_power_limit_value = 0xF0;
> > +	} else if (slot_power_limit <= 275*1000) {
> > +		slot_power_limit_scale = 0;
> > +		slot_power_limit_value = 0xF1;
> > +	} else {
> > +		slot_power_limit_scale = 0;
> > +		slot_power_limit_value = 0xF2;
> > +	}
> 
> This all looks like it should be common code.

Yes, this is common code. I will put it into some common function.

> > +
> >  	/*
> >  	 * Mark bridge as Hot-Plug Capable to allow delivering Data Link Layer
> >  	 * State Changed interrupt. No Command Completed Support is set because
> > @@ -996,7 +1057,10 @@ static int advk_sw_pci_bridge_init(struct advk_pcie *pcie)
> >  	 * bit permanently as there is no support for unplugging PCIe card from
> >  	 * the slot. Assume that PCIe card is always connected in slot.
> >  	 */
> > -	bridge->pcie_conf.slotcap = cpu_to_le32(PCI_EXP_SLTCAP_NCCS | PCI_EXP_SLTCAP_HPC);
> > +	bridge->pcie_conf.slotcap = cpu_to_le32(PCI_EXP_SLTCAP_NCCS |
> > +						PCI_EXP_SLTCAP_HPC |
> 
> While not new, how does the driver know the board is hotplug capable. 
> IIRC, we have a property for that.

Just ignore it for now. It is part of unfinished patches. I sent it to
show how slot_power_limit would be configured via emulated root bridge.

> > +						slot_power_limit_value << 7 |
> > +						slot_power_limit_scale << 15);
> >  	bridge->pcie_conf.slotsta = cpu_to_le16(PCI_EXP_SLTSTA_PDS);
> >  
> >  	return 0;
> > -- 
> > 2.20.1
> > 
> > 

Re: [RFC PATCH 1/3] dt-bindings: Add 'slot-power-limit' PCIe port property

[Rob Herring]

On Tue, Aug 24, 2021 at 11:14 AM Pali Rohár <pali@kernel.org> wrote:
>
> On Tuesday 24 August 2021 10:35:34 Rob Herring wrote:
> > On Fri, Aug 20, 2021 at 06:00:21PM +0200, Pali Rohár wrote:
> > > This property specifies slot power limit in mW unit. It is form-factor and
> > > board specific value and must be initialized by hardware.
> > >
> > > Some PCIe controllers delegates this work to software to allow hardware
> > > flexibility and therefore this property basically specifies what should
> > > host bridge programs into PCIe Slot Capabilities registers.
> > >
> > > Property needs to be specified in mW unit, and not in special format
> > > defined by Slot Capabilities (which encodes scaling factor or different
> > > unit). Host drivers should convert value from mW unit to their format.
> > >
> > > Signed-off-by: Pali Rohár <pali@kernel.org>
> > > ---
> > >  Documentation/devicetree/bindings/pci/pci.txt | 6 ++++++
> > >  1 file changed, 6 insertions(+)
> >
> > This needs to be in dtschema schemas/pci/pci-bus.yaml instead.
> >
> > (pci.txt is still here because it needs to be relicensed to move all the
> > descriptions to pci-bus.yaml.)
>
> Ok, this is just a proposal for a new DTS property. So documentation
> issues will be fixed in real patch.
>
> > >
> > > diff --git a/Documentation/devicetree/bindings/pci/pci.txt b/Documentation/devicetree/bindings/pci/pci.txt
> > > index 6a8f2874a24d..e67d5db21514 100644
> > > --- a/Documentation/devicetree/bindings/pci/pci.txt
> > > +++ b/Documentation/devicetree/bindings/pci/pci.txt
> > > @@ -32,6 +32,12 @@ driver implementation may support the following properties:
> > >     root port to downstream device and host bridge drivers can do programming
> > >     which depends on CLKREQ signal existence. For example, programming root port
> > >     not to advertise ASPM L1 Sub-States support if there is no CLKREQ signal.
> > > +- slot-power-limit:
> > > +   If present this property specifies slot power limit in mW unit. Host drivers
> >
> > As mentioned, this should have a unit suffix. I'm not sure it is
> > beneficial to share with SFP in this case though.
> >
> > > +   can parse this slot power limit and use it for programming Root Port or host
> > > +   bridge, or for composing and sending PCIe Set_Slot_Power_Limit message
> > > +   through the Root Port or host bridge when transitioning PCIe link from a
> > > +   non-DL_Up Status to a DL_Up Status.
> >
> > I no nothing about how this mechanism works, but I think this belongs in
> > the next section as for PCIe, a slot is always below a PCI-PCI bridge.
> > If we have N slots, then there's N bridges and needs to be N
> > slot-power-limit properties, right?
> >
> > (The same is probably true for all the properties here except
> > linux,pci-domain.) There's no distinction between host and PCI bridges
> > in pci-bus.yaml though.
> >
> > Rob
>
> This slot-power-limit property belongs to same place where are also
> other slot properties (link speed, reset gpios, clkreq). So I put it in
> place where others are.
>
> But I'm not sure where it should be as it affects link/slot. Because
> link has two sides. I guess that link speed and slot power limit could
> belong to the root/downstream port and reset gpio to the endpoint card
> or upstream port...

I wasn't debating whether it goes upstream or downstream, but just
that it can apply to more than just the host bridge or root port(s).
We have that now already with reset-gpios. Look at the hikey970 case
that's queued for 5.15. It's got RP -> switch -> slots/devices with
reset gpio on each slot.

As for upstream vs. downstream side, this is one of those cases where
we didn't represent the downstream side in DT, so everything gets
stuffed in the upstream side. As PCIe is point to point, it doesn't
matter so much. It would be a bigger issue on old PCI.

Rob

Re: [RFC PATCH 1/3] dt-bindings: Add 'slot-power-limit' PCIe port property

[Pali Rohár]

On Wednesday 25 August 2021 09:57:52 Rob Herring wrote:
> On Tue, Aug 24, 2021 at 11:14 AM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Tuesday 24 August 2021 10:35:34 Rob Herring wrote:
> > > On Fri, Aug 20, 2021 at 06:00:21PM +0200, Pali Rohár wrote:
> > > > This property specifies slot power limit in mW unit. It is form-factor and
> > > > board specific value and must be initialized by hardware.
> > > >
> > > > Some PCIe controllers delegates this work to software to allow hardware
> > > > flexibility and therefore this property basically specifies what should
> > > > host bridge programs into PCIe Slot Capabilities registers.
> > > >
> > > > Property needs to be specified in mW unit, and not in special format
> > > > defined by Slot Capabilities (which encodes scaling factor or different
> > > > unit). Host drivers should convert value from mW unit to their format.
> > > >
> > > > Signed-off-by: Pali Rohár <pali@kernel.org>
> > > > ---
> > > >  Documentation/devicetree/bindings/pci/pci.txt | 6 ++++++
> > > >  1 file changed, 6 insertions(+)
> > >
> > > This needs to be in dtschema schemas/pci/pci-bus.yaml instead.
> > >
> > > (pci.txt is still here because it needs to be relicensed to move all the
> > > descriptions to pci-bus.yaml.)
> >
> > Ok, this is just a proposal for a new DTS property. So documentation
> > issues will be fixed in real patch.
> >
> > > >
> > > > diff --git a/Documentation/devicetree/bindings/pci/pci.txt b/Documentation/devicetree/bindings/pci/pci.txt
> > > > index 6a8f2874a24d..e67d5db21514 100644
> > > > --- a/Documentation/devicetree/bindings/pci/pci.txt
> > > > +++ b/Documentation/devicetree/bindings/pci/pci.txt
> > > > @@ -32,6 +32,12 @@ driver implementation may support the following properties:
> > > >     root port to downstream device and host bridge drivers can do programming
> > > >     which depends on CLKREQ signal existence. For example, programming root port
> > > >     not to advertise ASPM L1 Sub-States support if there is no CLKREQ signal.
> > > > +- slot-power-limit:
> > > > +   If present this property specifies slot power limit in mW unit. Host drivers
> > >
> > > As mentioned, this should have a unit suffix. I'm not sure it is
> > > beneficial to share with SFP in this case though.
> > >
> > > > +   can parse this slot power limit and use it for programming Root Port or host
> > > > +   bridge, or for composing and sending PCIe Set_Slot_Power_Limit message
> > > > +   through the Root Port or host bridge when transitioning PCIe link from a
> > > > +   non-DL_Up Status to a DL_Up Status.
> > >
> > > I no nothing about how this mechanism works, but I think this belongs in
> > > the next section as for PCIe, a slot is always below a PCI-PCI bridge.
> > > If we have N slots, then there's N bridges and needs to be N
> > > slot-power-limit properties, right?
> > >
> > > (The same is probably true for all the properties here except
> > > linux,pci-domain.) There's no distinction between host and PCI bridges
> > > in pci-bus.yaml though.
> > >
> > > Rob
> >
> > This slot-power-limit property belongs to same place where are also
> > other slot properties (link speed, reset gpios, clkreq). So I put it in
> > place where others are.
> >
> > But I'm not sure where it should be as it affects link/slot. Because
> > link has two sides. I guess that link speed and slot power limit could
> > belong to the root/downstream port and reset gpio to the endpoint card
> > or upstream port...
> 
> I wasn't debating whether it goes upstream or downstream, but just
> that it can apply to more than just the host bridge or root port(s).
> We have that now already with reset-gpios. Look at the hikey970 case
> that's queued for 5.15. It's got RP -> switch -> slots/devices with
> reset gpio on each slot.
> 
> As for upstream vs. downstream side, this is one of those cases where
> we didn't represent the downstream side in DT, so everything gets
> stuffed in the upstream side. As PCIe is point to point, it doesn't
> matter so much. It would be a bigger issue on old PCI.

Upstream vs downstream matters for hotplugging PCIe cases. Upstream
part (endpoint card) can be unplugged from hotplugging PCIe bridge and
so does not have any node in sysfs (or lspci output). But downstream
part (or root port) of PCIe bridge is always present.