linux-pci.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v8 0/8] PCI: brcmstb: have portdrv turn on sub-device power
@ 2021-11-10 22:14 Jim Quinlan
  2021-11-10 22:14 ` [PATCH v8 1/8] PCI: brcmstb: Change brcm_phy_stop() to return void Jim Quinlan
                   ` (7 more replies)
  0 siblings, 8 replies; 35+ messages in thread
From: Jim Quinlan @ 2021-11-10 22:14 UTC (permalink / raw)
  To: linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring,
	Mark Brown, bcm-kernel-feedback-list, jim2101024, james.quinlan
  Cc: open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS,
	Jonathan Cameron, Keith Busch, Krzysztof Wilczyński,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	open list,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	Qiuxu Zhuo, Saenz Julienne, Sean V Kelley

v8  -- Only the two binding commits and the "Change brcm_phy_stop()" commit
       are unchanged.
    -- The code has been moved to portdrv_pci.c and bus.c.  The regulators   
       are placed in bus->dev.driver_data (bus->sysdata is already occupied
       by the Broadcom PCIe).  Two functions, pci_subdev_regulators_{add,remove}_bus()
       are created to turn the regulators on/off.  The pcie_portdriver also sets
       its pci_driver methods suspend and resume when the condirtions are
       right for this feature.  (Robh for suggestions, although I probably
       erred in following them).
    -- Have the root complex return 0xffffffff on accesses even when
       the link is down and the HW doesn't support such accesses (PaliR).
    -- Just call devm_bulk_regulator_get() on standard supplies; don't
       bother pre-scanning the DT for them (MarkB).

v7  -- RobH suggested putting the "vpcixxx-supply" property under the
       bridge-node rather than the endpoint device node.  Also, he said to
       use the pci-ops add_bus/remove methods.  Doing so simplifies the
       code greatly and three commits were dropped.  Thanks!

    -- Rob also suggested (I think) having this patchset be a general
       feature which is activated by an OF property under the bridge node.
       I tried to do that but realized that our root complex driver
       controls the regulators with its dev_pm_ops and there is no way to
       transfer this control when using general mechanism.  Note that
       although the regulator core deals with suspend, our RC driver wants
       the right to sometimes to preclude this for WOL scenarios.

    -- One commit was added to change the response to the return value of
       the pci_ops add_bus() method.  Currently, an error causes a WARNING,
       a dev_err(...), and continues to return the child bus.  The
       modification was, for returning -ENOLINK only, to skip WARNING &
       dev_err() and return NULL.  This is necessary for our RC HW, as if
       the code continues on it will do a pci_read_config_dword() for the
       vendor/id, and our HW flags a CPU abort (instead of returning
       0xffffffff) when the is no pcie-link established.

       [NOTE: MarkB, I did not add one of your two "Reviewed-by"s
              because the commit had a decent amount of change.]

v6   -- Dropped the idea of a placeholder regulator
        property (brcm-ep-a-supply). (MarkB)
     -- device_initialize() now called once.  Two
        commits were added for this.  (GKH)
     -- In two cases, separated a single function 
        into two or more functions (MarkB)
     -- "(void)foo();" => "foo()".  Note that although
        foo() returns an int, in this instance it is being
	invoked within a function returning void, and foo()
	already executes a dev_err() on error. (MarkB)
     -- Added a commit to correct PCIe interrupts in YAML.
     -- Removed "device_type = "pci";" for the EP node
        in the YAML example.
     -- Updated the URL related to the voltage regulator
        names on GitHub.  Note that I added vpciev3v3aux.

v5 [NOTE: It has been a while since v4.  Sorry]
     -- See "PCI: allow for callback to prepare nascent subdev"
        commit message for the cornerstone of this patchset
        and the reasons behind it.  This is a new commit.
     -- The RC driver now looks into its DT children and
        turns on regulators for a sub-device, and this occurs
	prior to PCIe link as it must.
     -- Dropped commits not related to the focus of this patchset.

v4 [NOTE: I'm not sure this fixes RobH and MarkB constraints but I'd
          like to use this pullreq as a basis for future discussion.]
   [Commit: Add bindings for ...]
     -- Fix syntax error in YAML bindings example (RobH)
     -- {vpcie12v,vpcie3v3}-supply props are back in root complex DT node
        (I believe RobH said this was okay)
   [Commit: Add control of ..]
     -- Do not do global search for regulator; now we look specifically
        for the property {vpcie12v,vpcie3v3}-supply in the root complex
	DT node and then call devm_regulator_bulk_get() (MarkB)
     -- Use devm_regulator_bulk_get() (Bjorn)
     -- s/EP/slot0 device/ (Bjorn)
     -- Spelling, capitalization (Bjorn)
     -- Have brcm_phy_stop() return a void (Bjorn)
   [Commit: Do not turn off ...]
     -- Capitalization (Bjorn)
   [Commit: Check return value ...]
     -- Commit message content (Bjorn)
     -- Move 6/6 hunk to 2/6 where it belongs (Bjorn)
     -- Move the rest of 6/6 before all other commits (Bjorn)

v3 -- Driver now searches for EP DT subnode for any regulators to turn on.
      If present, these regulators have the property names
      "vpcie12v-supply" and "vpcie3v3-supply".  The existence of these
      regulators in the EP subnode are currently pending as a pullreq
      in pci-bus.yaml at
      https://github.com/devicetree-org/dt-schema/pull/54
      (MarkB, RobH).
   -- Check return of brcm_set_regulators() (Florian)
   -- Specify one regulator string per line for easier update (Florian)
   -- Author/Committer/Signoff email changed from that of V2 from
      'james.quinlan@broadcom.com' to 'jim2101024@gmail.com'.

v2 -- Use regulator bulk API rather than multiple calls (MarkB).

v1 -- Bindings are added for fixed regulators that may power the EP device.
   -- The brcmstb RC driver is modified to control these regulators
      during probe, suspend, and resume.
   -- 7216 type SOCs have additional error reporting HW and a
      panic handler is added to dump its info.
   -- A missing return value check is added.

Jim Quinlan (8):
  PCI: brcmstb: Change brcm_phy_stop() to return void
  dt-bindings: PCI: Correct brcmstb interrupts, interrupt-map.
  dt-bindings: PCI: Add bindings for Brcmstb EP voltage regulators
  PCI/portdrv: Create pcie_is_port_dev() func from existing code
  PCI/portdrv: add mechanism to turn on subdev regulators
  PCI/portdrv: Do not turn off subdev regulators if EP can wake up
  PCI: brcmstb: Split brcm_pcie_setup() into two funcs
  PCI: brcmstb: Add control of subdevice voltage regulators

 .../bindings/pci/brcm,stb-pcie.yaml           |  31 ++++-
 drivers/pci/bus.c                             |  72 ++++++++++
 drivers/pci/controller/pcie-brcmstb.c         | 131 +++++++++++++-----
 drivers/pci/pci.h                             |  11 ++
 drivers/pci/pcie/portdrv_pci.c                |  86 +++++++++++-
 5 files changed, 292 insertions(+), 39 deletions(-)


base-commit: 8e37395c3a5dceff62a5010ebbbc107f4145935c
-- 
2.17.1


^ permalink raw reply	[flat|nested] 35+ messages in thread

* [PATCH v8 1/8] PCI: brcmstb: Change brcm_phy_stop() to return void
  2021-11-10 22:14 [PATCH v8 0/8] PCI: brcmstb: have portdrv turn on sub-device power Jim Quinlan
@ 2021-11-10 22:14 ` Jim Quinlan
  2021-11-11 21:57   ` Bjorn Helgaas
  2021-11-10 22:14 ` [PATCH v8 2/8] dt-bindings: PCI: Correct brcmstb interrupts, interrupt-map Jim Quinlan
                   ` (6 subsequent siblings)
  7 siblings, 1 reply; 35+ messages in thread
From: Jim Quinlan @ 2021-11-10 22:14 UTC (permalink / raw)
  To: linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring,
	Mark Brown, bcm-kernel-feedback-list, jim2101024, james.quinlan
  Cc: Florian Fainelli, Lorenzo Pieralisi, Krzysztof Wilczyński,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	open list

We do not use the result of this function so make it void.

Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
---
 drivers/pci/controller/pcie-brcmstb.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/drivers/pci/controller/pcie-brcmstb.c b/drivers/pci/controller/pcie-brcmstb.c
index cc30215f5a43..ff7d0d291531 100644
--- a/drivers/pci/controller/pcie-brcmstb.c
+++ b/drivers/pci/controller/pcie-brcmstb.c
@@ -1111,9 +1111,10 @@ static inline int brcm_phy_start(struct brcm_pcie *pcie)
 	return pcie->rescal ? brcm_phy_cntl(pcie, 1) : 0;
 }
 
-static inline int brcm_phy_stop(struct brcm_pcie *pcie)
+static inline void brcm_phy_stop(struct brcm_pcie *pcie)
 {
-	return pcie->rescal ? brcm_phy_cntl(pcie, 0) : 0;
+	if (pcie->rescal)
+		brcm_phy_cntl(pcie, 0);
 }
 
 static void brcm_pcie_turn_off(struct brcm_pcie *pcie)
@@ -1143,14 +1144,13 @@ static void brcm_pcie_turn_off(struct brcm_pcie *pcie)
 static int brcm_pcie_suspend(struct device *dev)
 {
 	struct brcm_pcie *pcie = dev_get_drvdata(dev);
-	int ret;
 
 	brcm_pcie_turn_off(pcie);
-	ret = brcm_phy_stop(pcie);
+	brcm_phy_stop(pcie);
 	reset_control_rearm(pcie->rescal);
 	clk_disable_unprepare(pcie->clk);
 
-	return ret;
+	return 0;
 }
 
 static int brcm_pcie_resume(struct device *dev)
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [PATCH v8 2/8] dt-bindings: PCI: Correct brcmstb interrupts, interrupt-map.
  2021-11-10 22:14 [PATCH v8 0/8] PCI: brcmstb: have portdrv turn on sub-device power Jim Quinlan
  2021-11-10 22:14 ` [PATCH v8 1/8] PCI: brcmstb: Change brcm_phy_stop() to return void Jim Quinlan
@ 2021-11-10 22:14 ` Jim Quinlan
  2021-11-10 22:14 ` [PATCH v8 3/8] dt-bindings: PCI: Add bindings for Brcmstb EP voltage regulators Jim Quinlan
                   ` (5 subsequent siblings)
  7 siblings, 0 replies; 35+ messages in thread
From: Jim Quinlan @ 2021-11-10 22:14 UTC (permalink / raw)
  To: linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring,
	Mark Brown, bcm-kernel-feedback-list, jim2101024, james.quinlan
  Cc: Florian Fainelli, Rob Herring, Saenz Julienne,
	moderated list:BROADCOM BCM7XXX ARM ARCHITECTURE,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS,
	open list

The "pcie" and "msi" interrupts were given the same interrupt when they are
actually different.  Interrupt-map only had the INTA entry; the INTB, INTC,
and INTD entries are added.

Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
Acked-by: Florian Fainelli <f.fainelli@gmail.com>
Acked-by: Rob Herring <robh@kernel.org>
---
 Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml b/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
index b9589a0daa5c..508e5dce1282 100644
--- a/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
+++ b/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
@@ -142,11 +142,15 @@ examples:
                     #address-cells = <3>;
                     #size-cells = <2>;
                     #interrupt-cells = <1>;
-                    interrupts = <GIC_SPI 148 IRQ_TYPE_LEVEL_HIGH>,
+                    interrupts = <GIC_SPI 147 IRQ_TYPE_LEVEL_HIGH>,
                                  <GIC_SPI 148 IRQ_TYPE_LEVEL_HIGH>;
                     interrupt-names = "pcie", "msi";
                     interrupt-map-mask = <0x0 0x0 0x0 0x7>;
-                    interrupt-map = <0 0 0 1 &gicv2 GIC_SPI 143 IRQ_TYPE_LEVEL_HIGH>;
+                    interrupt-map = <0 0 0 1 &gicv2 GIC_SPI 143 IRQ_TYPE_LEVEL_HIGH
+                                     0 0 0 2 &gicv2 GIC_SPI 144 IRQ_TYPE_LEVEL_HIGH
+                                     0 0 0 3 &gicv2 GIC_SPI 145 IRQ_TYPE_LEVEL_HIGH
+                                     0 0 0 4 &gicv2 GIC_SPI 146 IRQ_TYPE_LEVEL_HIGH>;
+
                     msi-parent = <&pcie0>;
                     msi-controller;
                     ranges = <0x02000000 0x0 0xf8000000 0x6 0x00000000 0x0 0x04000000>;
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [PATCH v8 3/8] dt-bindings: PCI: Add bindings for Brcmstb EP voltage regulators
  2021-11-10 22:14 [PATCH v8 0/8] PCI: brcmstb: have portdrv turn on sub-device power Jim Quinlan
  2021-11-10 22:14 ` [PATCH v8 1/8] PCI: brcmstb: Change brcm_phy_stop() to return void Jim Quinlan
  2021-11-10 22:14 ` [PATCH v8 2/8] dt-bindings: PCI: Correct brcmstb interrupts, interrupt-map Jim Quinlan
@ 2021-11-10 22:14 ` Jim Quinlan
  2021-11-11 22:17   ` Bjorn Helgaas
  2021-11-13 11:38   ` Pali Rohár
  2021-11-10 22:14 ` [PATCH v8 4/8] PCI/portdrv: Create pcie_is_port_dev() func from existing code Jim Quinlan
                   ` (4 subsequent siblings)
  7 siblings, 2 replies; 35+ messages in thread
From: Jim Quinlan @ 2021-11-10 22:14 UTC (permalink / raw)
  To: linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring,
	Mark Brown, bcm-kernel-feedback-list, jim2101024, james.quinlan
  Cc: Florian Fainelli, Rob Herring, Saenz Julienne,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS,
	open list

Similar to the regulator bindings found in "rockchip-pcie-host.txt", this
allows optional regulators to be attached and controlled by the PCIe RC
driver.  That being said, this driver searches in the DT subnode (the EP
node, eg pci-ep@0,0) for the regulator property.

The use of a regulator property in the pcie EP subnode such as
"vpcie12v-supply" depends on a pending pullreq to the pci-bus.yaml
file at

https://github.com/devicetree-org/dt-schema/pull/63

Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
---
 .../bindings/pci/brcm,stb-pcie.yaml           | 23 +++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml b/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
index 508e5dce1282..ef2427320b7d 100644
--- a/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
+++ b/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
@@ -158,5 +158,28 @@ examples:
                                  <0x42000000 0x1 0x80000000 0x3 0x00000000 0x0 0x80000000>;
                     brcm,enable-ssc;
                     brcm,scb-sizes =  <0x0000000080000000 0x0000000080000000>;
+
+                    /* PCIe bridge */
+                    pci@0,0 {
+                            #address-cells = <3>;
+                            #size-cells = <2>;
+                            reg = <0x0 0x0 0x0 0x0 0x0>;
+                            compatible = "pciclass,0604";
+                            device_type = "pci";
+                            vpcie3v3-supply = <&vreg7>;
+                            ranges;
+
+                            /* PCIe endpoint */
+                            pci-ep@0,0 {
+                                    assigned-addresses =
+                                        <0x82010000 0x0 0xf8000000 0x6 0x00000000 0x0 0x2000>;
+                                    reg = <0x0 0x0 0x0 0x0 0x0>;
+                                    compatible = "pci14e4,1688";
+                                    #address-cells = <3>;
+                                    #size-cells = <2>;
+
+                                    ranges;
+                            };
+                    };
             };
     };
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [PATCH v8 4/8] PCI/portdrv: Create pcie_is_port_dev() func from existing code
  2021-11-10 22:14 [PATCH v8 0/8] PCI: brcmstb: have portdrv turn on sub-device power Jim Quinlan
                   ` (2 preceding siblings ...)
  2021-11-10 22:14 ` [PATCH v8 3/8] dt-bindings: PCI: Add bindings for Brcmstb EP voltage regulators Jim Quinlan
@ 2021-11-10 22:14 ` Jim Quinlan
  2021-11-11 21:51   ` Florian Fainelli
  2021-11-11 23:50   ` Krzysztof Wilczyński
  2021-11-10 22:14 ` [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators Jim Quinlan
                   ` (3 subsequent siblings)
  7 siblings, 2 replies; 35+ messages in thread
From: Jim Quinlan @ 2021-11-10 22:14 UTC (permalink / raw)
  To: linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring,
	Mark Brown, bcm-kernel-feedback-list, jim2101024, james.quinlan
  Cc: Sean V Kelley, Jonathan Cameron, Qiuxu Zhuo, Keith Busch, open list

The function will be needed elsewhere in a few commits.

Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
---
 drivers/pci/pci.h              |  2 ++
 drivers/pci/pcie/portdrv_pci.c | 23 ++++++++++++++++++-----
 2 files changed, 20 insertions(+), 5 deletions(-)

diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 1cce56c2aea0..c2bd1995d3a9 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -744,4 +744,6 @@ extern const struct attribute_group aspm_ctrl_attr_group;
 
 extern const struct attribute_group pci_dev_reset_method_attr_group;
 
+bool pcie_is_port_dev(struct pci_dev *dev);
+
 #endif /* DRIVERS_PCI_H */
diff --git a/drivers/pci/pcie/portdrv_pci.c b/drivers/pci/pcie/portdrv_pci.c
index c7ff1eea225a..63f2a87e9db8 100644
--- a/drivers/pci/pcie/portdrv_pci.c
+++ b/drivers/pci/pcie/portdrv_pci.c
@@ -90,6 +90,23 @@ static const struct dev_pm_ops pcie_portdrv_pm_ops = {
 #define PCIE_PORTDRV_PM_OPS	NULL
 #endif /* !PM */
 
+bool pcie_is_port_dev(struct pci_dev *dev)
+{
+	int type;
+
+	if (!dev)
+		return false;
+
+	type = pci_pcie_type(dev);
+
+	return pci_is_pcie(dev) &&
+		((type == PCI_EXP_TYPE_ROOT_PORT) ||
+		 (type == PCI_EXP_TYPE_UPSTREAM) ||
+		 (type == PCI_EXP_TYPE_DOWNSTREAM) ||
+		 (type == PCI_EXP_TYPE_RC_EC));
+}
+EXPORT_SYMBOL_GPL(pcie_is_port_dev);
+
 /*
  * pcie_portdrv_probe - Probe PCI-Express port devices
  * @dev: PCI-Express port device being probed
@@ -104,11 +121,7 @@ static int pcie_portdrv_probe(struct pci_dev *dev,
 	int type = pci_pcie_type(dev);
 	int status;
 
-	if (!pci_is_pcie(dev) ||
-	    ((type != PCI_EXP_TYPE_ROOT_PORT) &&
-	     (type != PCI_EXP_TYPE_UPSTREAM) &&
-	     (type != PCI_EXP_TYPE_DOWNSTREAM) &&
-	     (type != PCI_EXP_TYPE_RC_EC)))
+	if (!pcie_is_port_dev(dev))
 		return -ENODEV;
 
 	if (type == PCI_EXP_TYPE_RC_EC)
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators
  2021-11-10 22:14 [PATCH v8 0/8] PCI: brcmstb: have portdrv turn on sub-device power Jim Quinlan
                   ` (3 preceding siblings ...)
  2021-11-10 22:14 ` [PATCH v8 4/8] PCI/portdrv: Create pcie_is_port_dev() func from existing code Jim Quinlan
@ 2021-11-10 22:14 ` Jim Quinlan
  2021-11-11  9:44   ` kernel test robot
                     ` (3 more replies)
  2021-11-10 22:14 ` [PATCH v8 6/8] PCI/portdrv: Do not turn off subdev regulators if EP can wake up Jim Quinlan
                   ` (2 subsequent siblings)
  7 siblings, 4 replies; 35+ messages in thread
From: Jim Quinlan @ 2021-11-10 22:14 UTC (permalink / raw)
  To: linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring,
	Mark Brown, bcm-kernel-feedback-list, jim2101024, james.quinlan
  Cc: Sean V Kelley, Jonathan Cameron, Qiuxu Zhuo, Keith Busch, open list

Adds a mechanism inside the root port device to identify standard PCIe
regulators in the DT, allocate them, and turn them on before the rest of
the bus is scanned during pci_host_probe().  A root complex driver can
leverage this mechanism by setting the pci_ops methods add_bus and
remove_bus to pci_subdev_regulators_{add,remove}_bus.

The allocated structure that contains the regulators is stored in
dev.driver_data.

The unabridged reason for doing this is as follows.  We would like the
Broadcom STB PCIe root complex driver (and others) to be able to turn
off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
the drivers of these endpoint devices are stock Linux drivers that are not
aware that these regulator(s) exist and must be turned on for the driver to
be probed.  The simple solution of course is to turn these regulators on at
boot and keep them on.  However, this solution does not satisfy at least
three of our usage modes:

1. For example, one customer uses multiple PCIe controllers, but wants the
ability to, by script invoking and unbind, turn any or all of them by and
their subdevices off to save power, e.g. when in battery mode.

2. Another example is when a watchdog script discovers that an endpoint
device is in an unresponsive state and would like to unbind, power toggle,
and re-bind just the PCIe endpoint and controller.

3. Of course we also want power turned off during suspend mode.  However,
some endpoint devices may be able to "wake" during suspend and we need to
recognise this case and veto the nominal act of turning off its regulator.
Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
end-point device needs to be kept powered on in order to receive network
packets and wake-up the system.

In all of these cases it is advantageous for the PCIe controller to govern
the turning off/on the regulators needed by the endpoint device.  The first
two cases can be done by simply unbinding and binding the PCIe controller,
if the controller has control of these regulators.

[1] These regulators typically govern the actual power supply to the
    endpoint chip.  Sometimes they may be a the official PCIe socket
    power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
    the regulator(s) that supply power to the EP chip.

[2] The 99% configuration of our boards is a single endpoint device
    attached to the PCIe controller.  I use the term endpoint but it could
    possible mean a switch as well.

Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
---
 drivers/pci/bus.c              | 72 ++++++++++++++++++++++++++++++++++
 drivers/pci/pci.h              |  8 ++++
 drivers/pci/pcie/portdrv_pci.c | 32 +++++++++++++++
 3 files changed, 112 insertions(+)

diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index 3cef835b375f..c39fdf36b0ad 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -419,3 +419,75 @@ void pci_bus_put(struct pci_bus *bus)
 	if (bus)
 		put_device(&bus->dev);
 }
+
+static void *alloc_subdev_regulators(struct device *dev)
+{
+	static const char * const supplies[] = {
+		"vpcie3v3",
+		"vpcie3v3aux",
+		"vpcie12v",
+	};
+	const size_t size = sizeof(struct subdev_regulators)
+		+ sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
+	struct subdev_regulators *sr;
+	int i;
+
+	sr = devm_kzalloc(dev, size, GFP_KERNEL);
+
+	if (sr) {
+		sr->num_supplies = ARRAY_SIZE(supplies);
+		for (i = 0; i < ARRAY_SIZE(supplies); i++)
+			sr->supplies[i].supply = supplies[i];
+	}
+
+	return sr;
+}
+
+
+int pci_subdev_regulators_add_bus(struct pci_bus *bus)
+{
+	struct device *dev = &bus->dev;
+	struct subdev_regulators *sr;
+	int ret;
+
+	if (!pcie_is_port_dev(bus->self))
+		return 0;
+
+	if (WARN_ON(bus->dev.driver_data))
+		dev_err(dev, "multiple clients using dev.driver_data\n");
+
+	sr = alloc_subdev_regulators(&bus->dev);
+	if (!sr)
+		return -ENOMEM;
+
+	bus->dev.driver_data = sr;
+	ret = regulator_bulk_get(dev, sr->num_supplies, sr->supplies);
+	if (ret)
+		return ret;
+
+	ret = regulator_bulk_enable(sr->num_supplies, sr->supplies);
+	if (ret) {
+		dev_err(dev, "failed to enable regulators for downstream device\n");
+		return ret;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(pci_subdev_regulators_add_bus);
+
+void pci_subdev_regulators_remove_bus(struct pci_bus *bus)
+{
+	struct device *dev = &bus->dev;
+	struct subdev_regulators *sr;
+
+	if (!pcie_is_port_dev(bus->self))
+		return;
+
+	sr = bus->dev.driver_data;
+	if (!sr)
+		return;
+
+	if (regulator_bulk_disable(sr->num_supplies, sr->supplies))
+		dev_err(dev, "failed to disable regulators for downstream device\n");
+}
+EXPORT_SYMBOL_GPL(pci_subdev_regulators_remove_bus);
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index c2bd1995d3a9..3f6cf75b91cc 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -3,6 +3,7 @@
 #define DRIVERS_PCI_H
 
 #include <linux/pci.h>
+#include <linux/regulator/consumer.h>
 
 /* Number of possible devfns: 0.0 to 1f.7 inclusive */
 #define MAX_NR_DEVFNS 256
@@ -744,6 +745,13 @@ extern const struct attribute_group aspm_ctrl_attr_group;
 
 extern const struct attribute_group pci_dev_reset_method_attr_group;
 
+struct subdev_regulators {
+	unsigned int num_supplies;
+	struct regulator_bulk_data supplies[];
+};
+
 bool pcie_is_port_dev(struct pci_dev *dev);
+int pci_subdev_regulators_add_bus(struct pci_bus *bus);
+void pci_subdev_regulators_remove_bus(struct pci_bus *bus);
 
 #endif /* DRIVERS_PCI_H */
diff --git a/drivers/pci/pcie/portdrv_pci.c b/drivers/pci/pcie/portdrv_pci.c
index 63f2a87e9db8..9330cfbebdc1 100644
--- a/drivers/pci/pcie/portdrv_pci.c
+++ b/drivers/pci/pcie/portdrv_pci.c
@@ -15,6 +15,7 @@
 #include <linux/init.h>
 #include <linux/aer.h>
 #include <linux/dmi.h>
+#include <linux/regulator/consumer.h>
 
 #include "../pci.h"
 #include "portdrv.h"
@@ -35,6 +36,9 @@ bool pcie_ports_native;
  */
 bool pcie_ports_dpc_native;
 
+/* forward declaration */
+static struct pci_driver pcie_portdriver;
+
 static int __init pcie_port_setup(char *str)
 {
 	if (!strncmp(str, "compat", 6))
@@ -107,6 +111,26 @@ bool pcie_is_port_dev(struct pci_dev *dev)
 }
 EXPORT_SYMBOL_GPL(pcie_is_port_dev);
 
+static int subdev_regulator_resume(struct pci_dev *dev)
+{
+	struct subdev_regulators *sr = dev->dev.driver_data;
+
+	if (sr)
+		return regulator_bulk_enable(sr->num_supplies, sr->supplies);
+
+	return 0;
+}
+
+static int subdev_regulator_suspend(struct pci_dev *dev, pm_message_t state)
+{
+	struct subdev_regulators *sr = dev->dev.driver_data;
+
+	if (sr)
+		return regulator_bulk_disable(sr->num_supplies, sr->supplies);
+
+	return 0;
+}
+
 /*
  * pcie_portdrv_probe - Probe PCI-Express port devices
  * @dev: PCI-Express port device being probed
@@ -131,6 +155,13 @@ static int pcie_portdrv_probe(struct pci_dev *dev,
 	if (status)
 		return status;
 
+	if (dev->bus->ops &&
+	    dev->bus->ops->add_bus &&
+	    dev->bus->dev.driver_data) {
+		pcie_portdriver.resume = subdev_regulator_resume;
+		pcie_portdriver.suspend = subdev_regulator_suspend;
+	}
+
 	pci_save_state(dev);
 
 	dev_pm_set_driver_flags(&dev->dev, DPM_FLAG_NO_DIRECT_COMPLETE |
@@ -237,6 +268,7 @@ static struct pci_driver pcie_portdriver = {
 	.err_handler	= &pcie_portdrv_err_handler,
 
 	.driver.pm	= PCIE_PORTDRV_PM_OPS,
+	/* Note: suspend and resume may be set during probe */
 };
 
 static int __init dmi_pcie_pme_disable_msi(const struct dmi_system_id *d)
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [PATCH v8 6/8] PCI/portdrv: Do not turn off subdev regulators if EP can wake up
  2021-11-10 22:14 [PATCH v8 0/8] PCI: brcmstb: have portdrv turn on sub-device power Jim Quinlan
                   ` (4 preceding siblings ...)
  2021-11-10 22:14 ` [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators Jim Quinlan
@ 2021-11-10 22:14 ` Jim Quinlan
  2021-11-10 22:14 ` [PATCH v8 7/8] PCI: brcmstb: Split brcm_pcie_setup() into two funcs Jim Quinlan
  2021-11-10 22:14 ` [PATCH v8 8/8] PCI: brcmstb: Add control of subdevice voltage regulators Jim Quinlan
  7 siblings, 0 replies; 35+ messages in thread
From: Jim Quinlan @ 2021-11-10 22:14 UTC (permalink / raw)
  To: linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring,
	Mark Brown, bcm-kernel-feedback-list, jim2101024, james.quinlan
  Cc: Sean V Kelley, Jonathan Cameron, Qiuxu Zhuo, Keith Busch, open list

If any downstream device may wake up during S2/S3 suspend, we do not want
to turn off its power when suspending.

Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
---
 drivers/pci/pci.h              |  1 +
 drivers/pci/pcie/portdrv_pci.c | 43 +++++++++++++++++++++++++++++-----
 2 files changed, 38 insertions(+), 6 deletions(-)

diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 3f6cf75b91cc..27abb977d0ec 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -746,6 +746,7 @@ extern const struct attribute_group aspm_ctrl_attr_group;
 extern const struct attribute_group pci_dev_reset_method_attr_group;
 
 struct subdev_regulators {
+	bool ep_wakeup_capable;
 	unsigned int num_supplies;
 	struct regulator_bulk_data supplies[];
 };
diff --git a/drivers/pci/pcie/portdrv_pci.c b/drivers/pci/pcie/portdrv_pci.c
index 9330cfbebdc1..717e816d0fc8 100644
--- a/drivers/pci/pcie/portdrv_pci.c
+++ b/drivers/pci/pcie/portdrv_pci.c
@@ -111,24 +111,55 @@ bool pcie_is_port_dev(struct pci_dev *dev)
 }
 EXPORT_SYMBOL_GPL(pcie_is_port_dev);
 
+static int pci_dev_may_wakeup(struct pci_dev *dev, void *data)
+{
+	bool *ret = data;
+
+	if (device_may_wakeup(&dev->dev)) {
+		*ret = true;
+		dev_dbg(&dev->dev, "disable cancelled for wake-up device\n");
+	}
+	return (int) *ret;
+}
+
 static int subdev_regulator_resume(struct pci_dev *dev)
 {
 	struct subdev_regulators *sr = dev->dev.driver_data;
 
-	if (sr)
-		return regulator_bulk_enable(sr->num_supplies, sr->supplies);
+	if (!sr)
+		return 0;
 
-	return 0;
+	if (sr->ep_wakeup_capable) {
+		/*
+		 * We are resuming from a suspend.  In the previous suspend
+		 * we did not disable the power supplies, so there is no
+		 * need to enable them (and falsely increase their usage
+		 * count).
+		 */
+		sr->ep_wakeup_capable = false;
+		return 0;
+	}
+
+	return regulator_bulk_enable(sr->num_supplies, sr->supplies);
 }
 
 static int subdev_regulator_suspend(struct pci_dev *dev, pm_message_t state)
 {
 	struct subdev_regulators *sr = dev->dev.driver_data;
 
-	if (sr)
-		return regulator_bulk_disable(sr->num_supplies, sr->supplies);
+	if (!sr)
+		return 0;
+	/*
+	 * If at least one device on this bus is enabled as a
+	 * wake-up source, do not turn off regulators.
+	 */
+	sr->ep_wakeup_capable = false;
+	pci_walk_bus(dev->bus, pci_dev_may_wakeup,
+		     &sr->ep_wakeup_capable);
+	if (sr->ep_wakeup_capable)
+		return 0;
 
-	return 0;
+	return regulator_bulk_disable(sr->num_supplies, sr->supplies);
 }
 
 /*
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [PATCH v8 7/8] PCI: brcmstb: Split brcm_pcie_setup() into two funcs
  2021-11-10 22:14 [PATCH v8 0/8] PCI: brcmstb: have portdrv turn on sub-device power Jim Quinlan
                   ` (5 preceding siblings ...)
  2021-11-10 22:14 ` [PATCH v8 6/8] PCI/portdrv: Do not turn off subdev regulators if EP can wake up Jim Quinlan
@ 2021-11-10 22:14 ` Jim Quinlan
  2021-11-10 22:14 ` [PATCH v8 8/8] PCI: brcmstb: Add control of subdevice voltage regulators Jim Quinlan
  7 siblings, 0 replies; 35+ messages in thread
From: Jim Quinlan @ 2021-11-10 22:14 UTC (permalink / raw)
  To: linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring,
	Mark Brown, bcm-kernel-feedback-list, jim2101024, james.quinlan
  Cc: Florian Fainelli, Lorenzo Pieralisi, Krzysztof Wilczyński,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	open list

We need to take some code in brcm_pcie_setup() and put it in a new function
brcm_pcie_linkup().  In future commits the brcm_pcie_linkup() function will
be called indirectly by pci_host_probe() as opposed to the host driver
invoking it directly.

Some code that was executed after the PCIe linkup is now placed so that it
executes prior to linkup, since this code has to run prior to the
invocation of pci_host_probe().

Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
---
 drivers/pci/controller/pcie-brcmstb.c | 65 ++++++++++++++++-----------
 1 file changed, 39 insertions(+), 26 deletions(-)

diff --git a/drivers/pci/controller/pcie-brcmstb.c b/drivers/pci/controller/pcie-brcmstb.c
index ff7d0d291531..1a841c240abb 100644
--- a/drivers/pci/controller/pcie-brcmstb.c
+++ b/drivers/pci/controller/pcie-brcmstb.c
@@ -863,16 +863,9 @@ static inline int brcm_pcie_get_rc_bar2_size_and_offset(struct brcm_pcie *pcie,
 
 static int brcm_pcie_setup(struct brcm_pcie *pcie)
 {
-	struct pci_host_bridge *bridge = pci_host_bridge_from_priv(pcie);
 	u64 rc_bar2_offset, rc_bar2_size;
 	void __iomem *base = pcie->base;
-	struct device *dev = pcie->dev;
-	struct resource_entry *entry;
-	bool ssc_good = false;
-	struct resource *res;
-	int num_out_wins = 0;
-	u16 nlw, cls, lnksta;
-	int i, ret, memc;
+	int ret, memc;
 	u32 tmp, burst, aspm_support;
 
 	/* Reset the bridge */
@@ -957,6 +950,40 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
 	if (pcie->gen)
 		brcm_pcie_set_gen(pcie, pcie->gen);
 
+	/* Don't advertise L0s capability if 'aspm-no-l0s' */
+	aspm_support = PCIE_LINK_STATE_L1;
+	if (!of_property_read_bool(pcie->np, "aspm-no-l0s"))
+		aspm_support |= PCIE_LINK_STATE_L0S;
+	tmp = readl(base + PCIE_RC_CFG_PRIV1_LINK_CAPABILITY);
+	u32p_replace_bits(&tmp, aspm_support,
+		PCIE_RC_CFG_PRIV1_LINK_CAPABILITY_ASPM_SUPPORT_MASK);
+	writel(tmp, base + PCIE_RC_CFG_PRIV1_LINK_CAPABILITY);
+
+	/*
+	 * For config space accesses on the RC, show the right class for
+	 * a PCIe-PCIe bridge (the default setting is to be EP mode).
+	 */
+	tmp = readl(base + PCIE_RC_CFG_PRIV1_ID_VAL3);
+	u32p_replace_bits(&tmp, 0x060400,
+			  PCIE_RC_CFG_PRIV1_ID_VAL3_CLASS_CODE_MASK);
+	writel(tmp, base + PCIE_RC_CFG_PRIV1_ID_VAL3);
+
+	return 0;
+}
+
+static int brcm_pcie_linkup(struct brcm_pcie *pcie)
+{
+	struct pci_host_bridge *bridge = pci_host_bridge_from_priv(pcie);
+	struct device *dev = pcie->dev;
+	void __iomem *base = pcie->base;
+	struct resource_entry *entry;
+	struct resource *res;
+	int num_out_wins = 0;
+	u16 nlw, cls, lnksta;
+	bool ssc_good = false;
+	u32 tmp;
+	int ret, i;
+
 	/* Unassert the fundamental reset */
 	pcie->perst_set(pcie, 0);
 
@@ -994,24 +1021,6 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
 		num_out_wins++;
 	}
 
-	/* Don't advertise L0s capability if 'aspm-no-l0s' */
-	aspm_support = PCIE_LINK_STATE_L1;
-	if (!of_property_read_bool(pcie->np, "aspm-no-l0s"))
-		aspm_support |= PCIE_LINK_STATE_L0S;
-	tmp = readl(base + PCIE_RC_CFG_PRIV1_LINK_CAPABILITY);
-	u32p_replace_bits(&tmp, aspm_support,
-		PCIE_RC_CFG_PRIV1_LINK_CAPABILITY_ASPM_SUPPORT_MASK);
-	writel(tmp, base + PCIE_RC_CFG_PRIV1_LINK_CAPABILITY);
-
-	/*
-	 * For config space accesses on the RC, show the right class for
-	 * a PCIe-PCIe bridge (the default setting is to be EP mode).
-	 */
-	tmp = readl(base + PCIE_RC_CFG_PRIV1_ID_VAL3);
-	u32p_replace_bits(&tmp, 0x060400,
-			  PCIE_RC_CFG_PRIV1_ID_VAL3_CLASS_CODE_MASK);
-	writel(tmp, base + PCIE_RC_CFG_PRIV1_ID_VAL3);
-
 	if (pcie->ssc) {
 		ret = brcm_pcie_set_ssc(pcie);
 		if (ret == 0)
@@ -1186,6 +1195,10 @@ static int brcm_pcie_resume(struct device *dev)
 	if (ret)
 		goto err_reset;
 
+	ret = brcm_pcie_linkup(pcie);
+	if (ret)
+		goto err_reset;
+
 	if (pcie->msi)
 		brcm_msi_set_regs(pcie->msi);
 
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [PATCH v8 8/8] PCI: brcmstb: Add control of subdevice voltage regulators
  2021-11-10 22:14 [PATCH v8 0/8] PCI: brcmstb: have portdrv turn on sub-device power Jim Quinlan
                   ` (6 preceding siblings ...)
  2021-11-10 22:14 ` [PATCH v8 7/8] PCI: brcmstb: Split brcm_pcie_setup() into two funcs Jim Quinlan
@ 2021-11-10 22:14 ` Jim Quinlan
  7 siblings, 0 replies; 35+ messages in thread
From: Jim Quinlan @ 2021-11-10 22:14 UTC (permalink / raw)
  To: linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring,
	Mark Brown, bcm-kernel-feedback-list, jim2101024, james.quinlan
  Cc: Florian Fainelli, Lorenzo Pieralisi, Krzysztof Wilczyński,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	open list

This Broadcom STB PCIe RC driver has one port and connects directly to one
device, be it a switch or an endpoint.  We want to be able to leverage
the recently added mechansim that allocates and turns on/off subdevice
regulators.

All that needs to be done is to put the regulator DT nodes in the bridge
below host and to set the pci_ops methods add_bus and remove_bus.

Note that the pci_subdev_regulators_add_bus() method is wrapped for two
reasons:

   1. To acheive linkup after the voltage regulators are turned on.

   2. If, in the case of an unsuccessful linkup, to redirect
      any PCIe accesses to subdevices, e.g. the scan for
      DEV/ID.  This redirection is needed because the Broadcom
      PCIe HW wil issue a CPU abort if such an access is made when
      there is no linkup.

Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
---
 drivers/pci/controller/pcie-brcmstb.c | 56 ++++++++++++++++++++++++++-
 1 file changed, 55 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/controller/pcie-brcmstb.c b/drivers/pci/controller/pcie-brcmstb.c
index 1a841c240abb..692df3dda77a 100644
--- a/drivers/pci/controller/pcie-brcmstb.c
+++ b/drivers/pci/controller/pcie-brcmstb.c
@@ -191,6 +191,7 @@ static inline void brcm_pcie_bridge_sw_init_set_generic(struct brcm_pcie *pcie,
 static inline void brcm_pcie_perst_set_4908(struct brcm_pcie *pcie, u32 val);
 static inline void brcm_pcie_perst_set_7278(struct brcm_pcie *pcie, u32 val);
 static inline void brcm_pcie_perst_set_generic(struct brcm_pcie *pcie, u32 val);
+static int brcm_pcie_add_bus(struct pci_bus *bus);
 
 enum {
 	RGR1_SW_INIT_1,
@@ -295,6 +296,7 @@ struct brcm_pcie {
 	u32			hw_rev;
 	void			(*perst_set)(struct brcm_pcie *pcie, u32 val);
 	void			(*bridge_sw_init_set)(struct brcm_pcie *pcie, u32 val);
+	bool			refusal_mode;
 };
 
 /*
@@ -711,6 +713,18 @@ static void __iomem *brcm_pcie_map_conf(struct pci_bus *bus, unsigned int devfn,
 	/* Accesses to the RC go right to the RC registers if slot==0 */
 	if (pci_is_root_bus(bus))
 		return PCI_SLOT(devfn) ? NULL : base + where;
+	if (pcie->refusal_mode) {
+		/*
+		 * At this point we do not have link.  There will be a CPU
+		 * abort -- a quirk with this controller --if Linux tries
+		 * to read any config-space registers besides those
+		 * targeting the host bridge.  To prevent this we hijack
+		 * the address to point to a safe access that will return
+		 * 0xffffffff.
+		 */
+		writel(0xffffffff, base + PCIE_MISC_RC_BAR2_CONFIG_HI);
+		return base + PCIE_MISC_RC_BAR2_CONFIG_HI + (where & 0x3);
+	}
 
 	/* For devices, write to the config space index register */
 	idx = PCIE_ECAM_OFFSET(bus->number, devfn, 0);
@@ -722,6 +736,8 @@ static struct pci_ops brcm_pcie_ops = {
 	.map_bus = brcm_pcie_map_conf,
 	.read = pci_generic_config_read,
 	.write = pci_generic_config_write,
+	.add_bus = brcm_pcie_add_bus,
+	.remove_bus = pci_subdev_regulators_remove_bus,
 };
 
 static inline void brcm_pcie_bridge_sw_init_set_generic(struct brcm_pcie *pcie, u32 val)
@@ -1242,6 +1258,34 @@ static const struct of_device_id brcm_pcie_match[] = {
 	{},
 };
 
+static int brcm_pcie_add_bus(struct pci_bus *bus)
+{
+	struct pci_host_bridge *hb;
+	struct brcm_pcie *pcie;
+	int ret;
+
+	if (!pcie_is_port_dev(bus->self))
+		return 0;
+
+	hb = pci_find_host_bridge(bus);
+	pcie = (struct brcm_pcie *) hb->sysdata;
+
+	ret = pci_subdev_regulators_add_bus(bus);
+	if (ret)
+		return ret;
+
+	/*
+	 * If we have failed linkup there is no point to return an error as
+	 * currently it will cause a WARNING() from pci_alloc_child_bus().
+	 * We return 0 and turn on the "refusal_mode" so that any further
+	 * accesses to the pci_dev just get 0xffffffff
+	 */
+	if (brcm_pcie_linkup(pcie) != 0)
+		pcie->refusal_mode = true;
+
+	return 0;
+}
+
 static int brcm_pcie_probe(struct platform_device *pdev)
 {
 	struct device_node *np = pdev->dev.of_node, *msi_np;
@@ -1333,7 +1377,17 @@ static int brcm_pcie_probe(struct platform_device *pdev)
 
 	platform_set_drvdata(pdev, pcie);
 
-	return pci_host_probe(bridge);
+	ret = pci_host_probe(bridge);
+	if (!ret && !brcm_pcie_link_up(pcie))
+		ret = -ENODEV;
+
+	if (ret) {
+		brcm_pcie_remove(pdev);
+		return ret;
+	}
+
+	return 0;
+
 fail:
 	__brcm_pcie_remove(pcie);
 	return ret;
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators
  2021-11-10 22:14 ` [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators Jim Quinlan
@ 2021-11-11  9:44   ` kernel test robot
  2021-11-11 22:12   ` Bjorn Helgaas
                     ` (2 subsequent siblings)
  3 siblings, 0 replies; 35+ messages in thread
From: kernel test robot @ 2021-11-11  9:44 UTC (permalink / raw)
  To: Jim Quinlan, linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne,
	Rob Herring, Mark Brown, bcm-kernel-feedback-list, james.quinlan
  Cc: kbuild-all, Sean V Kelley, Jonathan Cameron

[-- Attachment #1: Type: text/plain, Size: 1638 bytes --]

Hi Jim,

I love your patch! Yet something to improve:

[auto build test ERROR on 8e37395c3a5dceff62a5010ebbbc107f4145935c]

url:    https://github.com/0day-ci/linux/commits/Jim-Quinlan/PCI-brcmstb-have-portdrv-turn-on-sub-device-power/20211111-061940
base:   8e37395c3a5dceff62a5010ebbbc107f4145935c
config: ia64-defconfig (attached as .config)
compiler: ia64-linux-gcc (GCC) 11.2.0
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # https://github.com/0day-ci/linux/commit/9751f2d806d6efe0a7e389c3d64ad2fbd84b6be0
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Jim-Quinlan/PCI-brcmstb-have-portdrv-turn-on-sub-device-power/20211111-061940
        git checkout 9751f2d806d6efe0a7e389c3d64ad2fbd84b6be0
        # save the attached .config to linux build tree
        mkdir build_dir
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross O=build_dir ARCH=ia64 SHELL=/bin/bash

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>

All errors (new ones prefixed by >>):

   ia64-linux-ld: drivers/pci/bus.o: in function `pci_subdev_regulators_remove_bus':
>> (.text+0x3f2): undefined reference to `pcie_is_port_dev'
   ia64-linux-ld: drivers/pci/bus.o: in function `pci_subdev_regulators_add_bus':
   (.text+0x442): undefined reference to `pcie_is_port_dev'

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 19955 bytes --]

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 4/8] PCI/portdrv: Create pcie_is_port_dev() func from existing code
  2021-11-10 22:14 ` [PATCH v8 4/8] PCI/portdrv: Create pcie_is_port_dev() func from existing code Jim Quinlan
@ 2021-11-11 21:51   ` Florian Fainelli
  2021-11-11 22:53     ` Rob Herring
  2021-11-11 23:50   ` Krzysztof Wilczyński
  1 sibling, 1 reply; 35+ messages in thread
From: Florian Fainelli @ 2021-11-11 21:51 UTC (permalink / raw)
  To: Jim Quinlan, linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne,
	Rob Herring, Mark Brown, bcm-kernel-feedback-list, james.quinlan
  Cc: Sean V Kelley, Jonathan Cameron, Qiuxu Zhuo, Keith Busch, open list

On 11/10/21 2:14 PM, Jim Quinlan wrote:
> The function will be needed elsewhere in a few commits.
> 
> Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> ---
>  drivers/pci/pci.h              |  2 ++
>  drivers/pci/pcie/portdrv_pci.c | 23 ++++++++++++++++++-----
>  2 files changed, 20 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> index 1cce56c2aea0..c2bd1995d3a9 100644
> --- a/drivers/pci/pci.h
> +++ b/drivers/pci/pci.h
> @@ -744,4 +744,6 @@ extern const struct attribute_group aspm_ctrl_attr_group;
>  
>  extern const struct attribute_group pci_dev_reset_method_attr_group;
>  
> +bool pcie_is_port_dev(struct pci_dev *dev);

Looks like you need an inline stub here when CONFIG_PCIEPORTBUS is
disabled to avoid the linking failure reported by the kbuild robot:

static inline bool pcie_is_port_dev(struct pci_dev *dev)
{
	return false;
}

Thanks!
--
Florian

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 1/8] PCI: brcmstb: Change brcm_phy_stop() to return void
  2021-11-10 22:14 ` [PATCH v8 1/8] PCI: brcmstb: Change brcm_phy_stop() to return void Jim Quinlan
@ 2021-11-11 21:57   ` Bjorn Helgaas
  2021-11-15 20:56     ` Jim Quinlan
  0 siblings, 1 reply; 35+ messages in thread
From: Bjorn Helgaas @ 2021-11-11 21:57 UTC (permalink / raw)
  To: Jim Quinlan
  Cc: linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring,
	Mark Brown, bcm-kernel-feedback-list, james.quinlan,
	Florian Fainelli, Lorenzo Pieralisi, Krzysztof Wilczyński,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	open list

On Wed, Nov 10, 2021 at 05:14:41PM -0500, Jim Quinlan wrote:
> We do not use the result of this function so make it void.

I don't get it.  Can you expand on this?

brcm_phy_cntl() can return -EIO, which means brcm_phy_stop() can
return -EIO, which means brcm_pcie_suspend can return -EIO.
brcm_pcie_suspend() is the dev_pm_ops.suspend() method.

So are you saying we never use the result of dev_pm_ops.suspend()?

> Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> ---
>  drivers/pci/controller/pcie-brcmstb.c | 10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/pci/controller/pcie-brcmstb.c b/drivers/pci/controller/pcie-brcmstb.c
> index cc30215f5a43..ff7d0d291531 100644
> --- a/drivers/pci/controller/pcie-brcmstb.c
> +++ b/drivers/pci/controller/pcie-brcmstb.c
> @@ -1111,9 +1111,10 @@ static inline int brcm_phy_start(struct brcm_pcie *pcie)
>  	return pcie->rescal ? brcm_phy_cntl(pcie, 1) : 0;
>  }
>  
> -static inline int brcm_phy_stop(struct brcm_pcie *pcie)
> +static inline void brcm_phy_stop(struct brcm_pcie *pcie)
>  {
> -	return pcie->rescal ? brcm_phy_cntl(pcie, 0) : 0;
> +	if (pcie->rescal)
> +		brcm_phy_cntl(pcie, 0);
>  }
>  
>  static void brcm_pcie_turn_off(struct brcm_pcie *pcie)
> @@ -1143,14 +1144,13 @@ static void brcm_pcie_turn_off(struct brcm_pcie *pcie)
>  static int brcm_pcie_suspend(struct device *dev)
>  {
>  	struct brcm_pcie *pcie = dev_get_drvdata(dev);
> -	int ret;
>  
>  	brcm_pcie_turn_off(pcie);
> -	ret = brcm_phy_stop(pcie);
> +	brcm_phy_stop(pcie);
>  	reset_control_rearm(pcie->rescal);
>  	clk_disable_unprepare(pcie->clk);
>  
> -	return ret;
> +	return 0;
>  }
>  
>  static int brcm_pcie_resume(struct device *dev)
> -- 
> 2.17.1
> 

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators
  2021-11-10 22:14 ` [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators Jim Quinlan
  2021-11-11  9:44   ` kernel test robot
@ 2021-11-11 22:12   ` Bjorn Helgaas
  2021-11-11 22:50     ` Rob Herring
  2021-11-11 22:56   ` Rob Herring
  2021-11-11 23:38   ` Krzysztof Wilczyński
  3 siblings, 1 reply; 35+ messages in thread
From: Bjorn Helgaas @ 2021-11-11 22:12 UTC (permalink / raw)
  To: Jim Quinlan
  Cc: linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring,
	Mark Brown, bcm-kernel-feedback-list, james.quinlan,
	Sean V Kelley, Jonathan Cameron, Qiuxu Zhuo, Keith Busch,
	open list

Capitalize "Add" in subject.

On Wed, Nov 10, 2021 at 05:14:45PM -0500, Jim Quinlan wrote:
> Adds a mechanism inside the root port device to identify standard PCIe
> regulators in the DT, allocate them, and turn them on before the rest of
> the bus is scanned during pci_host_probe().  A root complex driver can
> leverage this mechanism by setting the pci_ops methods add_bus and
> remove_bus to pci_subdev_regulators_{add,remove}_bus.

s/Adds a/Add a/

> The allocated structure that contains the regulators is stored in
> dev.driver_data.
> 
> The unabridged reason for doing this is as follows.  We would like the
> Broadcom STB PCIe root complex driver (and others) to be able to turn
> off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> the drivers of these endpoint devices are stock Linux drivers that are not
> aware that these regulator(s) exist and must be turned on for the driver to
> be probed.  The simple solution of course is to turn these regulators on at
> boot and keep them on.  However, this solution does not satisfy at least
> three of our usage modes:
> 
> 1. For example, one customer uses multiple PCIe controllers, but wants the
> ability to, by script invoking and unbind, turn any or all of them by and
> their subdevices off to save power, e.g. when in battery mode.
> 
> 2. Another example is when a watchdog script discovers that an endpoint
> device is in an unresponsive state and would like to unbind, power toggle,
> and re-bind just the PCIe endpoint and controller.
> 
> 3. Of course we also want power turned off during suspend mode.  However,
> some endpoint devices may be able to "wake" during suspend and we need to
> recognise this case and veto the nominal act of turning off its regulator.
> Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> end-point device needs to be kept powered on in order to receive network
> packets and wake-up the system.
> 
> In all of these cases it is advantageous for the PCIe controller to govern
> the turning off/on the regulators needed by the endpoint device.  The first
> two cases can be done by simply unbinding and binding the PCIe controller,
> if the controller has control of these regulators.
> 
> [1] These regulators typically govern the actual power supply to the
>     endpoint chip.  Sometimes they may be a the official PCIe socket
>     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
>     the regulator(s) that supply power to the EP chip.

s/may be a the/may be the/

> [2] The 99% configuration of our boards is a single endpoint device
>     attached to the PCIe controller.  I use the term endpoint but it could
>     possible mean a switch as well.

s/possible/possibly/

This adds generic code, so it needs some connection to the generic DT
binding for these things, e.g., a commit in this series that adds it
(I see commits that touch brcm,stb-pcie.yaml, but not a generic
place).

> +static void *alloc_subdev_regulators(struct device *dev)
> +{
> +	static const char * const supplies[] = {
> +		"vpcie3v3",
> +		"vpcie3v3aux",
> +		"vpcie12v",
> +	};
> +	const size_t size = sizeof(struct subdev_regulators)
> +		+ sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
> +	struct subdev_regulators *sr;
> +	int i;
> +
> +	sr = devm_kzalloc(dev, size, GFP_KERNEL);
> +
> +	if (sr) {
> +		sr->num_supplies = ARRAY_SIZE(supplies);
> +		for (i = 0; i < ARRAY_SIZE(supplies); i++)
> +			sr->supplies[i].supply = supplies[i];
> +	}
> +
> +	return sr;
> +}
> +
> +

Spurious blank line.

> +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> ...

> @@ -131,6 +155,13 @@ static int pcie_portdrv_probe(struct pci_dev *dev,
>  	if (status)
>  		return status;
>  
> +	if (dev->bus->ops &&
> +	    dev->bus->ops->add_bus &&
> +	    dev->bus->dev.driver_data) {
> +		pcie_portdriver.resume = subdev_regulator_resume;
> +		pcie_portdriver.suspend = subdev_regulator_suspend;

The pci_driver.resume() and pci_driver.suspend() methods are going
away, so we shouldn't add new uses like this.

Doesn't this trigger the warning in pci_has_legacy_pm_support() about
supporting both legacy PM and the new PM?

Bjorn

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 3/8] dt-bindings: PCI: Add bindings for Brcmstb EP voltage regulators
  2021-11-10 22:14 ` [PATCH v8 3/8] dt-bindings: PCI: Add bindings for Brcmstb EP voltage regulators Jim Quinlan
@ 2021-11-11 22:17   ` Bjorn Helgaas
  2021-11-12 18:25     ` Jim Quinlan
  2021-11-13 11:38   ` Pali Rohár
  1 sibling, 1 reply; 35+ messages in thread
From: Bjorn Helgaas @ 2021-11-11 22:17 UTC (permalink / raw)
  To: Jim Quinlan
  Cc: linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring,
	Mark Brown, bcm-kernel-feedback-list, james.quinlan,
	Florian Fainelli, Rob Herring, Saenz Julienne,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS,
	open list

On Wed, Nov 10, 2021 at 05:14:43PM -0500, Jim Quinlan wrote:
> Similar to the regulator bindings found in "rockchip-pcie-host.txt", this
> allows optional regulators to be attached and controlled by the PCIe RC
> driver.  That being said, this driver searches in the DT subnode (the EP
> node, eg pci-ep@0,0) for the regulator property.
> 
> The use of a regulator property in the pcie EP subnode such as
> "vpcie12v-supply" depends on a pending pullreq to the pci-bus.yaml
> file at
> 
> https://github.com/devicetree-org/dt-schema/pull/63

Can you use a lore URL here?  github.com is sort of outside the Linux
ecosystem and this link is more likely to remain useful if it's to
something in kernel.org.

The subject says what this patch does, but the commit log doesn't.
It's OK to repeat the subject in the commit log if that's what makes
the most sense.

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators
  2021-11-11 22:12   ` Bjorn Helgaas
@ 2021-11-11 22:50     ` Rob Herring
  0 siblings, 0 replies; 35+ messages in thread
From: Rob Herring @ 2021-11-11 22:50 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: Jim Quinlan, PCI, Bjorn Helgaas, Nicolas Saenz Julienne,
	Mark Brown, maintainer:BROADCOM BCM7XXX ARM ARCHITECTURE,
	Jim Quinlan, Sean V Kelley, Jonathan Cameron, Qiuxu Zhuo,
	Keith Busch, open list

On Thu, Nov 11, 2021 at 4:12 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> Capitalize "Add" in subject.
>
> On Wed, Nov 10, 2021 at 05:14:45PM -0500, Jim Quinlan wrote:
> > Adds a mechanism inside the root port device to identify standard PCIe
> > regulators in the DT, allocate them, and turn them on before the rest of
> > the bus is scanned during pci_host_probe().  A root complex driver can
> > leverage this mechanism by setting the pci_ops methods add_bus and
> > remove_bus to pci_subdev_regulators_{add,remove}_bus.
>
> s/Adds a/Add a/
>
> > The allocated structure that contains the regulators is stored in
> > dev.driver_data.
> >
> > The unabridged reason for doing this is as follows.  We would like the
> > Broadcom STB PCIe root complex driver (and others) to be able to turn
> > off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> > the drivers of these endpoint devices are stock Linux drivers that are not
> > aware that these regulator(s) exist and must be turned on for the driver to
> > be probed.  The simple solution of course is to turn these regulators on at
> > boot and keep them on.  However, this solution does not satisfy at least
> > three of our usage modes:
> >
> > 1. For example, one customer uses multiple PCIe controllers, but wants the
> > ability to, by script invoking and unbind, turn any or all of them by and
> > their subdevices off to save power, e.g. when in battery mode.
> >
> > 2. Another example is when a watchdog script discovers that an endpoint
> > device is in an unresponsive state and would like to unbind, power toggle,
> > and re-bind just the PCIe endpoint and controller.
> >
> > 3. Of course we also want power turned off during suspend mode.  However,
> > some endpoint devices may be able to "wake" during suspend and we need to
> > recognise this case and veto the nominal act of turning off its regulator.
> > Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> > end-point device needs to be kept powered on in order to receive network
> > packets and wake-up the system.
> >
> > In all of these cases it is advantageous for the PCIe controller to govern
> > the turning off/on the regulators needed by the endpoint device.  The first
> > two cases can be done by simply unbinding and binding the PCIe controller,
> > if the controller has control of these regulators.
> >
> > [1] These regulators typically govern the actual power supply to the
> >     endpoint chip.  Sometimes they may be a the official PCIe socket
> >     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
> >     the regulator(s) that supply power to the EP chip.
>
> s/may be a the/may be the/
>
> > [2] The 99% configuration of our boards is a single endpoint device
> >     attached to the PCIe controller.  I use the term endpoint but it could
> >     possible mean a switch as well.
>
> s/possible/possibly/
>
> This adds generic code, so it needs some connection to the generic DT
> binding for these things, e.g., a commit in this series that adds it
> (I see commits that touch brcm,stb-pcie.yaml, but not a generic
> place).

That's pending here: https://github.com/devicetree-org/dt-schema/pull/63

Rob

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 4/8] PCI/portdrv: Create pcie_is_port_dev() func from existing code
  2021-11-11 21:51   ` Florian Fainelli
@ 2021-11-11 22:53     ` Rob Herring
  0 siblings, 0 replies; 35+ messages in thread
From: Rob Herring @ 2021-11-11 22:53 UTC (permalink / raw)
  To: Florian Fainelli
  Cc: Jim Quinlan, PCI, Bjorn Helgaas, Nicolas Saenz Julienne,
	Mark Brown, maintainer:BROADCOM BCM7XXX ARM ARCHITECTURE,
	Jim Quinlan, Sean V Kelley, Jonathan Cameron, Qiuxu Zhuo,
	Keith Busch, open list

On Thu, Nov 11, 2021 at 3:51 PM Florian Fainelli <f.fainelli@gmail.com> wrote:
>
> On 11/10/21 2:14 PM, Jim Quinlan wrote:
> > The function will be needed elsewhere in a few commits.
> >
> > Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> > ---
> >  drivers/pci/pci.h              |  2 ++
> >  drivers/pci/pcie/portdrv_pci.c | 23 ++++++++++++++++++-----
> >  2 files changed, 20 insertions(+), 5 deletions(-)
> >
> > diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> > index 1cce56c2aea0..c2bd1995d3a9 100644
> > --- a/drivers/pci/pci.h
> > +++ b/drivers/pci/pci.h
> > @@ -744,4 +744,6 @@ extern const struct attribute_group aspm_ctrl_attr_group;
> >
> >  extern const struct attribute_group pci_dev_reset_method_attr_group;
> >
> > +bool pcie_is_port_dev(struct pci_dev *dev);
>
> Looks like you need an inline stub here when CONFIG_PCIEPORTBUS is
> disabled to avoid the linking failure reported by the kbuild robot:

Probably always an inline function. It has nothing to do with the
driver, so portdrv_pci.c is not the right place.

Rob

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators
  2021-11-10 22:14 ` [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators Jim Quinlan
  2021-11-11  9:44   ` kernel test robot
  2021-11-11 22:12   ` Bjorn Helgaas
@ 2021-11-11 22:56   ` Rob Herring
  2021-11-15 20:44     ` Jim Quinlan
  2021-11-11 23:38   ` Krzysztof Wilczyński
  3 siblings, 1 reply; 35+ messages in thread
From: Rob Herring @ 2021-11-11 22:56 UTC (permalink / raw)
  To: Jim Quinlan
  Cc: PCI, Bjorn Helgaas, Nicolas Saenz Julienne, Mark Brown,
	maintainer:BROADCOM BCM7XXX ARM ARCHITECTURE, Jim Quinlan,
	Sean V Kelley, Jonathan Cameron, Qiuxu Zhuo, Keith Busch,
	open list

On Wed, Nov 10, 2021 at 4:15 PM Jim Quinlan <jim2101024@gmail.com> wrote:
>
> Adds a mechanism inside the root port device to identify standard PCIe
> regulators in the DT, allocate them, and turn them on before the rest of
> the bus is scanned during pci_host_probe().  A root complex driver can
> leverage this mechanism by setting the pci_ops methods add_bus and
> remove_bus to pci_subdev_regulators_{add,remove}_bus.
>
> The allocated structure that contains the regulators is stored in
> dev.driver_data.
>
> The unabridged reason for doing this is as follows.  We would like the
> Broadcom STB PCIe root complex driver (and others) to be able to turn
> off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> the drivers of these endpoint devices are stock Linux drivers that are not
> aware that these regulator(s) exist and must be turned on for the driver to
> be probed.  The simple solution of course is to turn these regulators on at
> boot and keep them on.  However, this solution does not satisfy at least
> three of our usage modes:
>
> 1. For example, one customer uses multiple PCIe controllers, but wants the
> ability to, by script invoking and unbind, turn any or all of them by and
> their subdevices off to save power, e.g. when in battery mode.
>
> 2. Another example is when a watchdog script discovers that an endpoint
> device is in an unresponsive state and would like to unbind, power toggle,
> and re-bind just the PCIe endpoint and controller.
>
> 3. Of course we also want power turned off during suspend mode.  However,
> some endpoint devices may be able to "wake" during suspend and we need to
> recognise this case and veto the nominal act of turning off its regulator.
> Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> end-point device needs to be kept powered on in order to receive network
> packets and wake-up the system.
>
> In all of these cases it is advantageous for the PCIe controller to govern
> the turning off/on the regulators needed by the endpoint device.  The first
> two cases can be done by simply unbinding and binding the PCIe controller,
> if the controller has control of these regulators.
>
> [1] These regulators typically govern the actual power supply to the
>     endpoint chip.  Sometimes they may be a the official PCIe socket
>     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
>     the regulator(s) that supply power to the EP chip.
>
> [2] The 99% configuration of our boards is a single endpoint device
>     attached to the PCIe controller.  I use the term endpoint but it could
>     possible mean a switch as well.
>
> Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> ---
>  drivers/pci/bus.c              | 72 ++++++++++++++++++++++++++++++++++
>  drivers/pci/pci.h              |  8 ++++
>  drivers/pci/pcie/portdrv_pci.c | 32 +++++++++++++++
>  3 files changed, 112 insertions(+)
>
> diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> index 3cef835b375f..c39fdf36b0ad 100644
> --- a/drivers/pci/bus.c
> +++ b/drivers/pci/bus.c
> @@ -419,3 +419,75 @@ void pci_bus_put(struct pci_bus *bus)
>         if (bus)
>                 put_device(&bus->dev);
>  }
> +
> +static void *alloc_subdev_regulators(struct device *dev)
> +{
> +       static const char * const supplies[] = {
> +               "vpcie3v3",
> +               "vpcie3v3aux",
> +               "vpcie12v",
> +       };
> +       const size_t size = sizeof(struct subdev_regulators)
> +               + sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
> +       struct subdev_regulators *sr;
> +       int i;
> +
> +       sr = devm_kzalloc(dev, size, GFP_KERNEL);
> +
> +       if (sr) {
> +               sr->num_supplies = ARRAY_SIZE(supplies);
> +               for (i = 0; i < ARRAY_SIZE(supplies); i++)
> +                       sr->supplies[i].supply = supplies[i];
> +       }
> +
> +       return sr;
> +}
> +
> +
> +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> +{
> +       struct device *dev = &bus->dev;
> +       struct subdev_regulators *sr;
> +       int ret;
> +
> +       if (!pcie_is_port_dev(bus->self))
> +               return 0;
> +
> +       if (WARN_ON(bus->dev.driver_data))
> +               dev_err(dev, "multiple clients using dev.driver_data\n");
> +
> +       sr = alloc_subdev_regulators(&bus->dev);
> +       if (!sr)
> +               return -ENOMEM;
> +
> +       bus->dev.driver_data = sr;
> +       ret = regulator_bulk_get(dev, sr->num_supplies, sr->supplies);
> +       if (ret)
> +               return ret;
> +
> +       ret = regulator_bulk_enable(sr->num_supplies, sr->supplies);
> +       if (ret) {
> +               dev_err(dev, "failed to enable regulators for downstream device\n");
> +               return ret;
> +       }
> +
> +       return 0;
> +}
> +EXPORT_SYMBOL_GPL(pci_subdev_regulators_add_bus);

Can't these just go in the portdrv probe and remove functions now?

Rob

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators
  2021-11-10 22:14 ` [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators Jim Quinlan
                     ` (2 preceding siblings ...)
  2021-11-11 22:56   ` Rob Herring
@ 2021-11-11 23:38   ` Krzysztof Wilczyński
  2021-11-15 20:26     ` Jim Quinlan
  3 siblings, 1 reply; 35+ messages in thread
From: Krzysztof Wilczyński @ 2021-11-11 23:38 UTC (permalink / raw)
  To: Jim Quinlan
  Cc: linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring,
	Mark Brown, bcm-kernel-feedback-list, james.quinlan,
	Sean V Kelley, Jonathan Cameron, Qiuxu Zhuo, Keith Busch,
	open list

Hi Jim,

[...]
> [1] These regulators typically govern the actual power supply to the
>     endpoint chip.  Sometimes they may be a the official PCIe socket

In the above, did you mean to say "be at the"?

> +static void *alloc_subdev_regulators(struct device *dev)
> +{
> +	static const char * const supplies[] = {
> +		"vpcie3v3",
> +		"vpcie3v3aux",
> +		"vpcie12v",
> +	};
> +	const size_t size = sizeof(struct subdev_regulators)
> +		+ sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);

[...]
> +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> +{
> +	struct device *dev = &bus->dev;
> +	struct subdev_regulators *sr;
> +	int ret;
> +
> +	if (!pcie_is_port_dev(bus->self))
> +		return 0;
> +
> +	if (WARN_ON(bus->dev.driver_data))
> +		dev_err(dev, "multiple clients using dev.driver_data\n");

I have to ask - is the WARN_ON() above adding value given the nature of the
error?  Would dumping a stack be of interest to someone?

Having said that, why do we even need to assert this?  Can there be some
sort of a race condition with access happening here?

I am asking as pci_subdev_regulators_remove_bus() does not seem to be
concerned about this sort of thing yet it also accesses the same driver
data, and such.

[...]
> +/* forward declaration */
> +static struct pci_driver pcie_portdriver;

The comment above might not be needed as it's quite obvious what the code
at this line is for, I believe.

[...]
> @@ -131,6 +155,13 @@ static int pcie_portdrv_probe(struct pci_dev *dev,
>  	if (status)
>  		return status;
>  
> +	if (dev->bus->ops &&
> +	    dev->bus->ops->add_bus &&
> +	    dev->bus->dev.driver_data) {
> +		pcie_portdriver.resume = subdev_regulator_resume;
> +		pcie_portdriver.suspend = subdev_regulator_suspend;
> +	}
> +
>  	pci_save_state(dev);

[...]
> @@ -237,6 +268,7 @@ static struct pci_driver pcie_portdriver = {
>  	.err_handler	= &pcie_portdrv_err_handler,
>  
>  	.driver.pm	= PCIE_PORTDRV_PM_OPS,
> +	/* Note: suspend and resume may be set during probe */

This comment here is for the "driver.pm" line above, correct?  If so, then
I would move it above the statement.  It's a little bit confusing
otherwise.

	Krzysztof


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 4/8] PCI/portdrv: Create pcie_is_port_dev() func from existing code
  2021-11-10 22:14 ` [PATCH v8 4/8] PCI/portdrv: Create pcie_is_port_dev() func from existing code Jim Quinlan
  2021-11-11 21:51   ` Florian Fainelli
@ 2021-11-11 23:50   ` Krzysztof Wilczyński
  2021-11-12 18:14     ` Jim Quinlan
  1 sibling, 1 reply; 35+ messages in thread
From: Krzysztof Wilczyński @ 2021-11-11 23:50 UTC (permalink / raw)
  To: Jim Quinlan
  Cc: linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring,
	Mark Brown, bcm-kernel-feedback-list, james.quinlan,
	Sean V Kelley, Jonathan Cameron, Qiuxu Zhuo, Keith Busch,
	open list

Hi Jim,

[...]
> +bool pcie_is_port_dev(struct pci_dev *dev)
> +{
> +	int type;
> +
> +	if (!dev)
> +		return false;
> +
> +	type = pci_pcie_type(dev);
> +
> +	return pci_is_pcie(dev) &&
> +		((type == PCI_EXP_TYPE_ROOT_PORT) ||
> +		 (type == PCI_EXP_TYPE_UPSTREAM) ||
> +		 (type == PCI_EXP_TYPE_DOWNSTREAM) ||
> +		 (type == PCI_EXP_TYPE_RC_EC));
> +}
> +EXPORT_SYMBOL_GPL(pcie_is_port_dev);

It would be really nice to document what the above function does (not that
some of the logic has been extracted from other function).  You know, for
the future generations of kernel hackers.

	Krzysztof

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 4/8] PCI/portdrv: Create pcie_is_port_dev() func from existing code
  2021-11-11 23:50   ` Krzysztof Wilczyński
@ 2021-11-12 18:14     ` Jim Quinlan
  0 siblings, 0 replies; 35+ messages in thread
From: Jim Quinlan @ 2021-11-12 18:14 UTC (permalink / raw)
  To: Krzysztof Wilczyński
  Cc: linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring,
	Mark Brown, bcm-kernel-feedback-list, Jim Quinlan, Sean V Kelley,
	Jonathan Cameron, Qiuxu Zhuo, Keith Busch, open list,
	Jim Quinlan

On Thu, Nov 11, 2021 at 6:50 PM Krzysztof Wilczyński <kw@linux.com> wrote:
>
> Hi Jim,
>
> [...]
> > +bool pcie_is_port_dev(struct pci_dev *dev)
> > +{
> > +     int type;
> > +
> > +     if (!dev)
> > +             return false;
> > +
> > +     type = pci_pcie_type(dev);
> > +
> > +     return pci_is_pcie(dev) &&
> > +             ((type == PCI_EXP_TYPE_ROOT_PORT) ||
> > +              (type == PCI_EXP_TYPE_UPSTREAM) ||
> > +              (type == PCI_EXP_TYPE_DOWNSTREAM) ||
> > +              (type == PCI_EXP_TYPE_RC_EC));
> > +}
> > +EXPORT_SYMBOL_GPL(pcie_is_port_dev);
>
> It would be really nice to document what the above function does (not that
> some of the logic has been extracted from other function).  You know, for
> the future generations of kernel hackers.

Hi Krzysztof and others,

I gave this a second look and realized that the portdrv's
pci_device_id list for the probe is doing filtering that is not
included in the function.  So perhaps the code must be the following
in order to live up to its name:

static inline bool pci_is_port_dev(struct pci_dev *dev)
{
    int type, class;

    if (!dev || !pci_is_pcie(dev))
        return false;

    class = dev->class;

    /* This must be kept in sync with port_pci_ids[] of protdev_pci.c */
    if (!(class == ((PCI_CLASS_BRIDGE_PCI << 8) | 0x00) ||
          class == ((PCI_CLASS_BRIDGE_PCI << 8) | 0x01) ||
          class == ((PCI_CLASS_SYSTEM_RCEC << 8) | 0x00)))
        return false;

    type = pci_pcie_type(dev);

    return ((type == PCI_EXP_TYPE_ROOT_PORT) ||
        (type == PCI_EXP_TYPE_UPSTREAM) ||
        (type == PCI_EXP_TYPE_DOWNSTREAM) ||
        (type == PCI_EXP_TYPE_RC_EC));
}

Kind of large for an inline, plus the code must be kept in sync with
the device list.   Suggestions?

As for a description, my understanding is that the code identifies a
pci_dev that is directly under a host bridge device.  I'm not really
sure about the PCI_CLASS_SYSTEM_RCEC though.

Regards,
Jim Quinlan
Broadcom STB

>
>         Krzysztof

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 3/8] dt-bindings: PCI: Add bindings for Brcmstb EP voltage regulators
  2021-11-11 22:17   ` Bjorn Helgaas
@ 2021-11-12 18:25     ` Jim Quinlan
  2021-11-12 20:20       ` Bjorn Helgaas
  0 siblings, 1 reply; 35+ messages in thread
From: Jim Quinlan @ 2021-11-12 18:25 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring,
	Mark Brown, bcm-kernel-feedback-list, Jim Quinlan,
	Florian Fainelli, Rob Herring, Saenz Julienne,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS,
	open list

On Thu, Nov 11, 2021 at 5:17 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Wed, Nov 10, 2021 at 05:14:43PM -0500, Jim Quinlan wrote:
> > Similar to the regulator bindings found in "rockchip-pcie-host.txt", this
> > allows optional regulators to be attached and controlled by the PCIe RC
> > driver.  That being said, this driver searches in the DT subnode (the EP
> > node, eg pci-ep@0,0) for the regulator property.
> >
> > The use of a regulator property in the pcie EP subnode such as
> > "vpcie12v-supply" depends on a pending pullreq to the pci-bus.yaml
> > file at
> >
> > https://github.com/devicetree-org/dt-schema/pull/63
>
> Can you use a lore URL here?  github.com is sort of outside the Linux
> ecosystem and this link is more likely to remain useful if it's to
> something in kernel.org.
Hi Bjorn,
I'm afraid I don't know how or if  this github repo transfers
information to Linux.  RobH, what should I be doing here?
>
> The subject says what this patch does, but the commit log doesn't.
> It's OK to repeat the subject in the commit log if that's what makes
> the most sense.
Got it.
Thanks,
Jim Quinlan
Broadcom STB

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 3/8] dt-bindings: PCI: Add bindings for Brcmstb EP voltage regulators
  2021-11-12 18:25     ` Jim Quinlan
@ 2021-11-12 20:20       ` Bjorn Helgaas
  2021-11-12 21:46         ` Rob Herring
  0 siblings, 1 reply; 35+ messages in thread
From: Bjorn Helgaas @ 2021-11-12 20:20 UTC (permalink / raw)
  To: Jim Quinlan
  Cc: linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring,
	Mark Brown, bcm-kernel-feedback-list, Jim Quinlan,
	Florian Fainelli, Rob Herring, Saenz Julienne,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS,
	open list

On Fri, Nov 12, 2021 at 01:25:11PM -0500, Jim Quinlan wrote:
> On Thu, Nov 11, 2021 at 5:17 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > On Wed, Nov 10, 2021 at 05:14:43PM -0500, Jim Quinlan wrote:
> > > Similar to the regulator bindings found in "rockchip-pcie-host.txt", this
> > > allows optional regulators to be attached and controlled by the PCIe RC
> > > driver.  That being said, this driver searches in the DT subnode (the EP
> > > node, eg pci-ep@0,0) for the regulator property.
> > >
> > > The use of a regulator property in the pcie EP subnode such as
> > > "vpcie12v-supply" depends on a pending pullreq to the pci-bus.yaml
> > > file at
> > >
> > > https://github.com/devicetree-org/dt-schema/pull/63
> >
> > Can you use a lore URL here?  github.com is sort of outside the Linux
> > ecosystem and this link is more likely to remain useful if it's to
> > something in kernel.org.
> Hi Bjorn,
> I'm afraid I don't know how or if  this github repo transfers
> information to Linux.  RobH, what should I be doing here?

Does this change get posted to any mailing lists where people can
review it?  Or would people have to watch the github devicetree-org
repo if they wanted to do that?  I was assuming this pci-bus.yaml
change was something that would eventually end up in the Linux kernel
source tree, but dt-scheme doesn't seem to be based on Linus' tree, so
I don't know if there's a connection.

Bjorn

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 3/8] dt-bindings: PCI: Add bindings for Brcmstb EP voltage regulators
  2021-11-12 20:20       ` Bjorn Helgaas
@ 2021-11-12 21:46         ` Rob Herring
  0 siblings, 0 replies; 35+ messages in thread
From: Rob Herring @ 2021-11-12 21:46 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: Jim Quinlan, linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne,
	Mark Brown, bcm-kernel-feedback-list, Jim Quinlan,
	Florian Fainelli, Saenz Julienne,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS,
	open list

On Fri, Nov 12, 2021 at 2:20 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Fri, Nov 12, 2021 at 01:25:11PM -0500, Jim Quinlan wrote:
> > On Thu, Nov 11, 2021 at 5:17 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > On Wed, Nov 10, 2021 at 05:14:43PM -0500, Jim Quinlan wrote:
> > > > Similar to the regulator bindings found in "rockchip-pcie-host.txt", this
> > > > allows optional regulators to be attached and controlled by the PCIe RC
> > > > driver.  That being said, this driver searches in the DT subnode (the EP
> > > > node, eg pci-ep@0,0) for the regulator property.
> > > >
> > > > The use of a regulator property in the pcie EP subnode such as
> > > > "vpcie12v-supply" depends on a pending pullreq to the pci-bus.yaml
> > > > file at
> > > >
> > > > https://github.com/devicetree-org/dt-schema/pull/63
> > >
> > > Can you use a lore URL here?  github.com is sort of outside the Linux
> > > ecosystem and this link is more likely to remain useful if it's to
> > > something in kernel.org.
> > Hi Bjorn,
> > I'm afraid I don't know how or if  this github repo transfers
> > information to Linux.  RobH, what should I be doing here?
>
> Does this change get posted to any mailing lists where people can
> review it?

devicetree-spec is where I direct folks to. It's not in lore, but we
could add it I guess. But I take PRs too. There's so few other
contributions I'm looking to make it as painless as possible for
contributors. I'd be happy for more reviewers other than me, but I
don't think where changes are posted is the problem there. :( Someone
should review all the crap Python code I write too.

Generally the flow is I redirect things submitted to the kernel to
dtschema instead. So that review happens first at least.

> Or would people have to watch the github devicetree-org
> repo if they wanted to do that?  I was assuming this pci-bus.yaml
> change was something that would eventually end up in the Linux kernel
> source tree, but dt-scheme doesn't seem to be based on Linus' tree, so
> I don't know if there's a connection.

It's more the other way around. The 'rule' is common bindings go in
dtschema and device specific bindings in the kernel tree. Reality is
some common bindings are in the kernel tree primarily because I want
everything in dtschema dual licensed and relicensing is a pain. That's
why we have pci.txt and pci-bus.yaml still.

Rob

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 3/8] dt-bindings: PCI: Add bindings for Brcmstb EP voltage regulators
  2021-11-10 22:14 ` [PATCH v8 3/8] dt-bindings: PCI: Add bindings for Brcmstb EP voltage regulators Jim Quinlan
  2021-11-11 22:17   ` Bjorn Helgaas
@ 2021-11-13 11:38   ` Pali Rohár
  1 sibling, 0 replies; 35+ messages in thread
From: Pali Rohár @ 2021-11-13 11:38 UTC (permalink / raw)
  To: Jim Quinlan
  Cc: linux-pci, Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring,
	Mark Brown, bcm-kernel-feedback-list, james.quinlan,
	Florian Fainelli, Rob Herring, Saenz Julienne,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS,
	open list

On Wednesday 10 November 2021 17:14:43 Jim Quinlan wrote:
> Similar to the regulator bindings found in "rockchip-pcie-host.txt", this
> allows optional regulators to be attached and controlled by the PCIe RC
> driver.  That being said, this driver searches in the DT subnode (the EP
> node, eg pci-ep@0,0) for the regulator property.
> 
> The use of a regulator property in the pcie EP subnode such as
> "vpcie12v-supply" depends on a pending pullreq to the pci-bus.yaml
> file at
> 
> https://github.com/devicetree-org/dt-schema/pull/63
> 
> Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> ---
>  .../bindings/pci/brcm,stb-pcie.yaml           | 23 +++++++++++++++++++
>  1 file changed, 23 insertions(+)
> 
> diff --git a/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml b/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
> index 508e5dce1282..ef2427320b7d 100644
> --- a/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
> +++ b/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
> @@ -158,5 +158,28 @@ examples:
>                                   <0x42000000 0x1 0x80000000 0x3 0x00000000 0x0 0x80000000>;
>                      brcm,enable-ssc;
>                      brcm,scb-sizes =  <0x0000000080000000 0x0000000080000000>;
> +
> +                    /* PCIe bridge */
> +                    pci@0,0 {

Hello! I think that above comment should be "PCIe Root Port" (and not
PCIe bridge) as this node really describes Root Port device in PCIe Root
Complex (correct me if I deduced it wrongly). lspci (-vv) show type of
device on the line "Capabilities: [..] Express".

> +                            #address-cells = <3>;
> +                            #size-cells = <2>;
> +                            reg = <0x0 0x0 0x0 0x0 0x0>;
> +                            compatible = "pciclass,0604";
> +                            device_type = "pci";
> +                            vpcie3v3-supply = <&vreg7>;
> +                            ranges;
> +
> +                            /* PCIe endpoint */
> +                            pci-ep@0,0 {
> +                                    assigned-addresses =
> +                                        <0x82010000 0x0 0xf8000000 0x6 0x00000000 0x0 0x2000>;
> +                                    reg = <0x0 0x0 0x0 0x0 0x0>;
> +                                    compatible = "pci14e4,1688";
> +                                    #address-cells = <3>;
> +                                    #size-cells = <2>;
> +
> +                                    ranges;
> +                            };
> +                    };
>              };
>      };
> -- 
> 2.17.1
> 

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators
  2021-11-11 23:38   ` Krzysztof Wilczyński
@ 2021-11-15 20:26     ` Jim Quinlan
  0 siblings, 0 replies; 35+ messages in thread
From: Jim Quinlan @ 2021-11-15 20:26 UTC (permalink / raw)
  To: Krzysztof Wilczyński
  Cc: Jim Quinlan,
	open list:PCI NATIVE HOST BRIDGE AND ENDPOINT DRIVERS,
	Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring, Mark Brown,
	maintainer:BROADCOM BCM7XXX ARM ARCHITECTURE, Sean V Kelley,
	Jonathan Cameron, Qiuxu Zhuo, Keith Busch, open list

On Thu, Nov 11, 2021 at 6:38 PM Krzysztof Wilczyński <kw@linux.com> wrote:
>
> Hi Jim,
>
> [...]
> > [1] These regulators typically govern the actual power supply to the
> >     endpoint chip.  Sometimes they may be a the official PCIe socket
>
> In the above, did you mean to say "be at the"?
Yep.
>
> > +static void *alloc_subdev_regulators(struct device *dev)
> > +{
> > +     static const char * const supplies[] = {
> > +             "vpcie3v3",
> > +             "vpcie3v3aux",
> > +             "vpcie12v",
> > +     };
> > +     const size_t size = sizeof(struct subdev_regulators)
> > +             + sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
>
> [...]
> > +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> > +{
> > +     struct device *dev = &bus->dev;
> > +     struct subdev_regulators *sr;
> > +     int ret;
> > +
> > +     if (!pcie_is_port_dev(bus->self))
> > +             return 0;
> > +
> > +     if (WARN_ON(bus->dev.driver_data))
> > +             dev_err(dev, "multiple clients using dev.driver_data\n");
>
> I have to ask - is the WARN_ON() above adding value given the nature of the
> error?  Would dumping a stack be of interest to someone?
Hello Krzysztof,

It doesn't need to be a warning.  You are right, the backtrace will
not help anyone figure out how to fix the problem.

>
> Having said that, why do we even need to assert this?  Can there be some
> sort of a race condition with access happening here?
This commit-set is claiming the driver_data field of the PCIe port
device and I am concerned  that something else in the future would
unknowingly do the same.  It would not be a race, just two separate
pieces of code stomping on the same variable.  If I am over-worrying I
can use a dev_err or nothing at all.


>
> I am asking as pci_subdev_regulators_remove_bus() does not seem to be
> concerned about this sort of thing yet it also accesses the same driver
> data, and such.
Yes, but when pci_subdev_regulators_remove_bus() accesses the port
driver driver_data and it is non-NULL  it  does not  know whether it
is the expected pointer or something else.

>
> [...]
> > +/* forward declaration */
> > +static struct pci_driver pcie_portdriver;
>
> The comment above might not be needed as it's quite obvious what the code
> at this line is for, I believe.
Okay.
>
> [...]
> > @@ -131,6 +155,13 @@ static int pcie_portdrv_probe(struct pci_dev *dev,
> >       if (status)
> >               return status;
> >
> > +     if (dev->bus->ops &&
> > +         dev->bus->ops->add_bus &&
> > +         dev->bus->dev.driver_data) {
> > +             pcie_portdriver.resume = subdev_regulator_resume;
> > +             pcie_portdriver.suspend = subdev_regulator_suspend;
> > +     }
> > +
> >       pci_save_state(dev);
>
> [...]
> > @@ -237,6 +268,7 @@ static struct pci_driver pcie_portdriver = {
> >       .err_handler    = &pcie_portdrv_err_handler,
> >
> >       .driver.pm      = PCIE_PORTDRV_PM_OPS,
> > +     /* Note: suspend and resume may be set during probe */
>
> This comment here is for the "driver.pm" line above, correct?  If so, then
> I would move it above the statement.  It's a little bit confusing
> otherwise.
I'm planning to remove this comment and the code that sets
pcie_portdriver.{resume,suspend} and instead put this code into the
int pcie_port_device_{suspend,remove}() functions.

Regards,
Jim Quinlan
Broadcom STB

>
>         Krzysztof
>

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators
  2021-11-11 22:56   ` Rob Herring
@ 2021-11-15 20:44     ` Jim Quinlan
  2021-11-16 17:41       ` Rob Herring
  0 siblings, 1 reply; 35+ messages in thread
From: Jim Quinlan @ 2021-11-15 20:44 UTC (permalink / raw)
  To: Rob Herring
  Cc: Jim Quinlan, PCI, Bjorn Helgaas, Nicolas Saenz Julienne,
	Mark Brown, maintainer:BROADCOM BCM7XXX ARM ARCHITECTURE,
	Sean V Kelley, Jonathan Cameron, Qiuxu Zhuo, Keith Busch,
	open list

On Thu, Nov 11, 2021 at 5:57 PM Rob Herring <robh@kernel.org> wrote:
>
> On Wed, Nov 10, 2021 at 4:15 PM Jim Quinlan <jim2101024@gmail.com> wrote:
> >
> > Adds a mechanism inside the root port device to identify standard PCIe
> > regulators in the DT, allocate them, and turn them on before the rest of
> > the bus is scanned during pci_host_probe().  A root complex driver can
> > leverage this mechanism by setting the pci_ops methods add_bus and
> > remove_bus to pci_subdev_regulators_{add,remove}_bus.
> >
> > The allocated structure that contains the regulators is stored in
> > dev.driver_data.
> >
> > The unabridged reason for doing this is as follows.  We would like the
> > Broadcom STB PCIe root complex driver (and others) to be able to turn
> > off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> > the drivers of these endpoint devices are stock Linux drivers that are not
> > aware that these regulator(s) exist and must be turned on for the driver to
> > be probed.  The simple solution of course is to turn these regulators on at
> > boot and keep them on.  However, this solution does not satisfy at least
> > three of our usage modes:
> >
> > 1. For example, one customer uses multiple PCIe controllers, but wants the
> > ability to, by script invoking and unbind, turn any or all of them by and
> > their subdevices off to save power, e.g. when in battery mode.
> >
> > 2. Another example is when a watchdog script discovers that an endpoint
> > device is in an unresponsive state and would like to unbind, power toggle,
> > and re-bind just the PCIe endpoint and controller.
> >
> > 3. Of course we also want power turned off during suspend mode.  However,
> > some endpoint devices may be able to "wake" during suspend and we need to
> > recognise this case and veto the nominal act of turning off its regulator.
> > Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> > end-point device needs to be kept powered on in order to receive network
> > packets and wake-up the system.
> >
> > In all of these cases it is advantageous for the PCIe controller to govern
> > the turning off/on the regulators needed by the endpoint device.  The first
> > two cases can be done by simply unbinding and binding the PCIe controller,
> > if the controller has control of these regulators.
> >
> > [1] These regulators typically govern the actual power supply to the
> >     endpoint chip.  Sometimes they may be a the official PCIe socket
> >     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
> >     the regulator(s) that supply power to the EP chip.
> >
> > [2] The 99% configuration of our boards is a single endpoint device
> >     attached to the PCIe controller.  I use the term endpoint but it could
> >     possible mean a switch as well.
> >
> > Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> > ---
> >  drivers/pci/bus.c              | 72 ++++++++++++++++++++++++++++++++++
> >  drivers/pci/pci.h              |  8 ++++
> >  drivers/pci/pcie/portdrv_pci.c | 32 +++++++++++++++
> >  3 files changed, 112 insertions(+)
> >
> > diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> > index 3cef835b375f..c39fdf36b0ad 100644
> > --- a/drivers/pci/bus.c
> > +++ b/drivers/pci/bus.c
> > @@ -419,3 +419,75 @@ void pci_bus_put(struct pci_bus *bus)
> >         if (bus)
> >                 put_device(&bus->dev);
> >  }
> > +
> > +static void *alloc_subdev_regulators(struct device *dev)
> > +{
> > +       static const char * const supplies[] = {
> > +               "vpcie3v3",
> > +               "vpcie3v3aux",
> > +               "vpcie12v",
> > +       };
> > +       const size_t size = sizeof(struct subdev_regulators)
> > +               + sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
> > +       struct subdev_regulators *sr;
> > +       int i;
> > +
> > +       sr = devm_kzalloc(dev, size, GFP_KERNEL);
> > +
> > +       if (sr) {
> > +               sr->num_supplies = ARRAY_SIZE(supplies);
> > +               for (i = 0; i < ARRAY_SIZE(supplies); i++)
> > +                       sr->supplies[i].supply = supplies[i];
> > +       }
> > +
> > +       return sr;
> > +}
> > +
> > +
> > +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> > +{
> > +       struct device *dev = &bus->dev;
> > +       struct subdev_regulators *sr;
> > +       int ret;
> > +
> > +       if (!pcie_is_port_dev(bus->self))
> > +               return 0;
> > +
> > +       if (WARN_ON(bus->dev.driver_data))
> > +               dev_err(dev, "multiple clients using dev.driver_data\n");
> > +
> > +       sr = alloc_subdev_regulators(&bus->dev);
> > +       if (!sr)
> > +               return -ENOMEM;
> > +
> > +       bus->dev.driver_data = sr;
> > +       ret = regulator_bulk_get(dev, sr->num_supplies, sr->supplies);
> > +       if (ret)
> > +               return ret;
> > +
> > +       ret = regulator_bulk_enable(sr->num_supplies, sr->supplies);
> > +       if (ret) {
> > +               dev_err(dev, "failed to enable regulators for downstream device\n");
> > +               return ret;
> > +       }
> > +
> > +       return 0;
> > +}
> > +EXPORT_SYMBOL_GPL(pci_subdev_regulators_add_bus);
>
> Can't these just go in the portdrv probe and remove functions now?
>
> Rob

Not really.  The idea is that  only when a host controller driver does this

static struct pci_ops my_pcie_ops = {
    .add_bus = pci_subdev_regulators_add_bus , /* see  note below */
    .remove_bus = pci_subdev_regulators_remove_bus,
    ...
}

does it explicitly want this feature.  Without doing this, every PCI
port in the world will execute a devm_kzalloc() and
devm_regulator_bulk_get() to (likely) grab nothing, and then there
will be three superfluous lines in the boot log:

pci_bus 0001:01: 0001:01 supply vpcie12v not found, using dummy regulator
pci_bus 0001:01: 0001:01 supply vpcie3v3 not found, using dummy regulator
pci_bus 0001:01: 0001:01 supply vpcie3v3aux not found, using dummy regulator

Secondly, our  HW needs to know when the  alloc/get/enable of
regulators is done so that the PCIe link can then be attempted.   This
is pretty much the cornerstone of this patchset.   To do this the brcm
RC driver's call to pci_subdev_regulators_add_bus() is wrapped by
brcm_pcie_add_bus() so that we can do this:

static struct pci_ops my_pcie_ops = {
    .add_bus = brcm_pcie_add_bus ,   /* calls pci_subdev_regulators_add_bus() */
    .remove_bus = pci_subdev_regulators_remove_bus,
    ...
}

Regards,
Jim Quinlan
Broadcom STB

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 1/8] PCI: brcmstb: Change brcm_phy_stop() to return void
  2021-11-11 21:57   ` Bjorn Helgaas
@ 2021-11-15 20:56     ` Jim Quinlan
  2021-11-16 20:40       ` Bjorn Helgaas
  0 siblings, 1 reply; 35+ messages in thread
From: Jim Quinlan @ 2021-11-15 20:56 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: Jim Quinlan,
	open list:PCI NATIVE HOST BRIDGE AND ENDPOINT DRIVERS,
	Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring, Mark Brown,
	maintainer:BROADCOM BCM7XXX ARM ARCHITECTURE, Florian Fainelli,
	Lorenzo Pieralisi, Krzysztof Wilczyński,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	open list

On Thu, Nov 11, 2021 at 4:57 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Wed, Nov 10, 2021 at 05:14:41PM -0500, Jim Quinlan wrote:
> > We do not use the result of this function so make it void.
>
> I don't get it.  Can you expand on this?
>
> brcm_phy_cntl() can return -EIO, which means brcm_phy_stop() can
> return -EIO, which means brcm_pcie_suspend can return -EIO.
> brcm_pcie_suspend() is the dev_pm_ops.suspend() method.
>
> So are you saying we never use the result of dev_pm_ops.suspend()?
Hi Bjorn,

In this situation we are going into suspend.  In doing so, any
problems with the brcm phy may be erased/forgiven upon resume, since
clocks are turned off and most power removed/reduced.  An error from
brcm_phy_stop() that becomes the return value of  brcm_pcie_suspend()
will bring a halt to the entire suspend IIRC.   In fact, I forced a
-EIO in this code and it panic'd on suspend.  This is not really the
behavior I want for what is most likely a recoverable error.

Perhaps a dev_err(...) will suffice while still returning 0?

I noticed that reset_control_rearm() also returns a value, and if that
is in error it will not be erease/forgiven  by suspend sleep.  I will
fix this.

Regards,
Jim Quinlan
Broadcom STB


>
> > Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> > ---
> >  drivers/pci/controller/pcie-brcmstb.c | 10 +++++-----
> >  1 file changed, 5 insertions(+), 5 deletions(-)
> >
> > diff --git a/drivers/pci/controller/pcie-brcmstb.c b/drivers/pci/controller/pcie-brcmstb.c
> > index cc30215f5a43..ff7d0d291531 100644
> > --- a/drivers/pci/controller/pcie-brcmstb.c
> > +++ b/drivers/pci/controller/pcie-brcmstb.c
> > @@ -1111,9 +1111,10 @@ static inline int brcm_phy_start(struct brcm_pcie *pcie)
> >       return pcie->rescal ? brcm_phy_cntl(pcie, 1) : 0;
> >  }
> >
> > -static inline int brcm_phy_stop(struct brcm_pcie *pcie)
> > +static inline void brcm_phy_stop(struct brcm_pcie *pcie)
> >  {
> > -     return pcie->rescal ? brcm_phy_cntl(pcie, 0) : 0;
> > +     if (pcie->rescal)
> > +             brcm_phy_cntl(pcie, 0);
> >  }
> >
> >  static void brcm_pcie_turn_off(struct brcm_pcie *pcie)
> > @@ -1143,14 +1144,13 @@ static void brcm_pcie_turn_off(struct brcm_pcie *pcie)
> >  static int brcm_pcie_suspend(struct device *dev)
> >  {
> >       struct brcm_pcie *pcie = dev_get_drvdata(dev);
> > -     int ret;
> >
> >       brcm_pcie_turn_off(pcie);
> > -     ret = brcm_phy_stop(pcie);
> > +     brcm_phy_stop(pcie);
> >       reset_control_rearm(pcie->rescal);
> >       clk_disable_unprepare(pcie->clk);
> >
> > -     return ret;
> > +     return 0;
> >  }
> >
> >  static int brcm_pcie_resume(struct device *dev)
> > --
> > 2.17.1
> >

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators
  2021-11-15 20:44     ` Jim Quinlan
@ 2021-11-16 17:41       ` Rob Herring
  2021-11-16 20:53         ` Pali Rohár
  2021-11-17 14:46         ` Jim Quinlan
  0 siblings, 2 replies; 35+ messages in thread
From: Rob Herring @ 2021-11-16 17:41 UTC (permalink / raw)
  To: Jim Quinlan, Pali Rohár
  Cc: Jim Quinlan, PCI, Bjorn Helgaas, Nicolas Saenz Julienne,
	Mark Brown, maintainer:BROADCOM BCM7XXX ARM ARCHITECTURE,
	Sean V Kelley, Jonathan Cameron, Qiuxu Zhuo, Keith Busch,
	open list

+Pali

On Mon, Nov 15, 2021 at 2:44 PM Jim Quinlan <james.quinlan@broadcom.com> wrote:
>
> On Thu, Nov 11, 2021 at 5:57 PM Rob Herring <robh@kernel.org> wrote:
> >
> > On Wed, Nov 10, 2021 at 4:15 PM Jim Quinlan <jim2101024@gmail.com> wrote:
> > >
> > > Adds a mechanism inside the root port device to identify standard PCIe
> > > regulators in the DT, allocate them, and turn them on before the rest of
> > > the bus is scanned during pci_host_probe().  A root complex driver can
> > > leverage this mechanism by setting the pci_ops methods add_bus and
> > > remove_bus to pci_subdev_regulators_{add,remove}_bus.
> > >
> > > The allocated structure that contains the regulators is stored in
> > > dev.driver_data.
> > >
> > > The unabridged reason for doing this is as follows.  We would like the
> > > Broadcom STB PCIe root complex driver (and others) to be able to turn
> > > off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> > > the drivers of these endpoint devices are stock Linux drivers that are not
> > > aware that these regulator(s) exist and must be turned on for the driver to
> > > be probed.  The simple solution of course is to turn these regulators on at
> > > boot and keep them on.  However, this solution does not satisfy at least
> > > three of our usage modes:
> > >
> > > 1. For example, one customer uses multiple PCIe controllers, but wants the
> > > ability to, by script invoking and unbind, turn any or all of them by and
> > > their subdevices off to save power, e.g. when in battery mode.
> > >
> > > 2. Another example is when a watchdog script discovers that an endpoint
> > > device is in an unresponsive state and would like to unbind, power toggle,
> > > and re-bind just the PCIe endpoint and controller.
> > >
> > > 3. Of course we also want power turned off during suspend mode.  However,
> > > some endpoint devices may be able to "wake" during suspend and we need to
> > > recognise this case and veto the nominal act of turning off its regulator.
> > > Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> > > end-point device needs to be kept powered on in order to receive network
> > > packets and wake-up the system.
> > >
> > > In all of these cases it is advantageous for the PCIe controller to govern
> > > the turning off/on the regulators needed by the endpoint device.  The first
> > > two cases can be done by simply unbinding and binding the PCIe controller,
> > > if the controller has control of these regulators.
> > >
> > > [1] These regulators typically govern the actual power supply to the
> > >     endpoint chip.  Sometimes they may be a the official PCIe socket
> > >     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
> > >     the regulator(s) that supply power to the EP chip.
> > >
> > > [2] The 99% configuration of our boards is a single endpoint device
> > >     attached to the PCIe controller.  I use the term endpoint but it could
> > >     possible mean a switch as well.
> > >
> > > Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> > > ---
> > >  drivers/pci/bus.c              | 72 ++++++++++++++++++++++++++++++++++
> > >  drivers/pci/pci.h              |  8 ++++
> > >  drivers/pci/pcie/portdrv_pci.c | 32 +++++++++++++++
> > >  3 files changed, 112 insertions(+)
> > >
> > > diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> > > index 3cef835b375f..c39fdf36b0ad 100644
> > > --- a/drivers/pci/bus.c
> > > +++ b/drivers/pci/bus.c
> > > @@ -419,3 +419,75 @@ void pci_bus_put(struct pci_bus *bus)
> > >         if (bus)
> > >                 put_device(&bus->dev);
> > >  }
> > > +
> > > +static void *alloc_subdev_regulators(struct device *dev)
> > > +{
> > > +       static const char * const supplies[] = {
> > > +               "vpcie3v3",
> > > +               "vpcie3v3aux",
> > > +               "vpcie12v",
> > > +       };
> > > +       const size_t size = sizeof(struct subdev_regulators)
> > > +               + sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
> > > +       struct subdev_regulators *sr;
> > > +       int i;
> > > +
> > > +       sr = devm_kzalloc(dev, size, GFP_KERNEL);
> > > +
> > > +       if (sr) {
> > > +               sr->num_supplies = ARRAY_SIZE(supplies);
> > > +               for (i = 0; i < ARRAY_SIZE(supplies); i++)
> > > +                       sr->supplies[i].supply = supplies[i];
> > > +       }
> > > +
> > > +       return sr;
> > > +}
> > > +
> > > +
> > > +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> > > +{
> > > +       struct device *dev = &bus->dev;
> > > +       struct subdev_regulators *sr;
> > > +       int ret;
> > > +
> > > +       if (!pcie_is_port_dev(bus->self))
> > > +               return 0;
> > > +
> > > +       if (WARN_ON(bus->dev.driver_data))
> > > +               dev_err(dev, "multiple clients using dev.driver_data\n");
> > > +
> > > +       sr = alloc_subdev_regulators(&bus->dev);
> > > +       if (!sr)
> > > +               return -ENOMEM;
> > > +
> > > +       bus->dev.driver_data = sr;
> > > +       ret = regulator_bulk_get(dev, sr->num_supplies, sr->supplies);
> > > +       if (ret)
> > > +               return ret;
> > > +
> > > +       ret = regulator_bulk_enable(sr->num_supplies, sr->supplies);
> > > +       if (ret) {
> > > +               dev_err(dev, "failed to enable regulators for downstream device\n");
> > > +               return ret;
> > > +       }
> > > +
> > > +       return 0;
> > > +}
> > > +EXPORT_SYMBOL_GPL(pci_subdev_regulators_add_bus);
> >
> > Can't these just go in the portdrv probe and remove functions now?
> >
> > Rob
>
> Not really.  The idea is that  only when a host controller driver does this
>
> static struct pci_ops my_pcie_ops = {
>     .add_bus = pci_subdev_regulators_add_bus , /* see  note below */
>     .remove_bus = pci_subdev_regulators_remove_bus,
>     ...
> }
>
> does it explicitly want this feature.  Without doing this, every PCI
> port in the world will execute a devm_kzalloc() and
> devm_regulator_bulk_get() to (likely) grab nothing, and then there
> will be three superfluous lines in the boot log:

You can opt-in based on there being a DT node.

> pci_bus 0001:01: 0001:01 supply vpcie12v not found, using dummy regulator
> pci_bus 0001:01: 0001:01 supply vpcie3v3 not found, using dummy regulator
> pci_bus 0001:01: 0001:01 supply vpcie3v3aux not found, using dummy regulator

This would be annoying, but not really a reason for how to design this.

> Secondly, our  HW needs to know when the  alloc/get/enable of
> regulators is done so that the PCIe link can then be attempted.   This
> is pretty much the cornerstone of this patchset.   To do this the brcm
> RC driver's call to pci_subdev_regulators_add_bus() is wrapped by
> brcm_pcie_add_bus() so that we can do this:
>
> static struct pci_ops my_pcie_ops = {
>     .add_bus = brcm_pcie_add_bus ,   /* calls pci_subdev_regulators_add_bus() */
>     .remove_bus = pci_subdev_regulators_remove_bus,

Do add_bus/remove_bus get called during resume/suspend? If not, how do
you handle the link during resume?

Maybe there needs to be explicit hooks for link handling. Pali has
been looking into this some.

Rob

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 1/8] PCI: brcmstb: Change brcm_phy_stop() to return void
  2021-11-15 20:56     ` Jim Quinlan
@ 2021-11-16 20:40       ` Bjorn Helgaas
  0 siblings, 0 replies; 35+ messages in thread
From: Bjorn Helgaas @ 2021-11-16 20:40 UTC (permalink / raw)
  To: Jim Quinlan
  Cc: Jim Quinlan,
	open list:PCI NATIVE HOST BRIDGE AND ENDPOINT DRIVERS,
	Bjorn Helgaas, Nicolas Saenz Julienne, Rob Herring, Mark Brown,
	maintainer:BROADCOM BCM7XXX ARM ARCHITECTURE, Florian Fainelli,
	Lorenzo Pieralisi, Krzysztof Wilczyński,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	moderated list:BROADCOM BCM2711/BCM2835 ARM ARCHITECTURE,
	open list

On Mon, Nov 15, 2021 at 03:56:14PM -0500, Jim Quinlan wrote:
> On Thu, Nov 11, 2021 at 4:57 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > On Wed, Nov 10, 2021 at 05:14:41PM -0500, Jim Quinlan wrote:
> > > We do not use the result of this function so make it void.
> >
> > I don't get it.  Can you expand on this?
> >
> > brcm_phy_cntl() can return -EIO, which means brcm_phy_stop() can
> > return -EIO, which means brcm_pcie_suspend can return -EIO.
> > brcm_pcie_suspend() is the dev_pm_ops.suspend() method.
> >
> > So are you saying we never use the result of dev_pm_ops.suspend()?
> 
> In this situation we are going into suspend.  In doing so, any
> problems with the brcm phy may be erased/forgiven upon resume, since
> clocks are turned off and most power removed/reduced.  An error from
> brcm_phy_stop() that becomes the return value of  brcm_pcie_suspend()
> will bring a halt to the entire suspend IIRC.   In fact, I forced a
> -EIO in this code and it panic'd on suspend.  This is not really the
> behavior I want for what is most likely a recoverable error.
> 
> Perhaps a dev_err(...) will suffice while still returning 0?

Maybe we just need a note about why we want to intentionally ignore
any errors here.

> I noticed that reset_control_rearm() also returns a value, and if that
> is in error it will not be erease/forgiven  by suspend sleep.  I will
> fix this.
> 
> Regards,
> Jim Quinlan
> Broadcom STB
> 
> 
> >
> > > Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> > > ---
> > >  drivers/pci/controller/pcie-brcmstb.c | 10 +++++-----
> > >  1 file changed, 5 insertions(+), 5 deletions(-)
> > >
> > > diff --git a/drivers/pci/controller/pcie-brcmstb.c b/drivers/pci/controller/pcie-brcmstb.c
> > > index cc30215f5a43..ff7d0d291531 100644
> > > --- a/drivers/pci/controller/pcie-brcmstb.c
> > > +++ b/drivers/pci/controller/pcie-brcmstb.c
> > > @@ -1111,9 +1111,10 @@ static inline int brcm_phy_start(struct brcm_pcie *pcie)
> > >       return pcie->rescal ? brcm_phy_cntl(pcie, 1) : 0;
> > >  }
> > >
> > > -static inline int brcm_phy_stop(struct brcm_pcie *pcie)
> > > +static inline void brcm_phy_stop(struct brcm_pcie *pcie)
> > >  {
> > > -     return pcie->rescal ? brcm_phy_cntl(pcie, 0) : 0;
> > > +     if (pcie->rescal)
> > > +             brcm_phy_cntl(pcie, 0);
> > >  }
> > >
> > >  static void brcm_pcie_turn_off(struct brcm_pcie *pcie)
> > > @@ -1143,14 +1144,13 @@ static void brcm_pcie_turn_off(struct brcm_pcie *pcie)
> > >  static int brcm_pcie_suspend(struct device *dev)
> > >  {
> > >       struct brcm_pcie *pcie = dev_get_drvdata(dev);
> > > -     int ret;
> > >
> > >       brcm_pcie_turn_off(pcie);
> > > -     ret = brcm_phy_stop(pcie);
> > > +     brcm_phy_stop(pcie);
> > >       reset_control_rearm(pcie->rescal);
> > >       clk_disable_unprepare(pcie->clk);
> > >
> > > -     return ret;
> > > +     return 0;
> > >  }
> > >
> > >  static int brcm_pcie_resume(struct device *dev)
> > > --
> > > 2.17.1
> > >

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators
  2021-11-16 17:41       ` Rob Herring
@ 2021-11-16 20:53         ` Pali Rohár
  2021-11-17 15:14           ` Jim Quinlan
  2021-11-17 14:46         ` Jim Quinlan
  1 sibling, 1 reply; 35+ messages in thread
From: Pali Rohár @ 2021-11-16 20:53 UTC (permalink / raw)
  To: Rob Herring
  Cc: Jim Quinlan, Jim Quinlan, PCI, Bjorn Helgaas,
	Nicolas Saenz Julienne, Mark Brown,
	maintainer:BROADCOM BCM7XXX ARM ARCHITECTURE, Sean V Kelley,
	Jonathan Cameron, Qiuxu Zhuo, Keith Busch, open list

On Tuesday 16 November 2021 11:41:22 Rob Herring wrote:
> +Pali
> 
> On Mon, Nov 15, 2021 at 2:44 PM Jim Quinlan <james.quinlan@broadcom.com> wrote:
> >
> > On Thu, Nov 11, 2021 at 5:57 PM Rob Herring <robh@kernel.org> wrote:
> > >
> > > On Wed, Nov 10, 2021 at 4:15 PM Jim Quinlan <jim2101024@gmail.com> wrote:
> > > >
> > > > Adds a mechanism inside the root port device to identify standard PCIe
> > > > regulators in the DT, allocate them, and turn them on before the rest of
> > > > the bus is scanned during pci_host_probe().  A root complex driver can
> > > > leverage this mechanism by setting the pci_ops methods add_bus and
> > > > remove_bus to pci_subdev_regulators_{add,remove}_bus.
> > > >
> > > > The allocated structure that contains the regulators is stored in
> > > > dev.driver_data.
> > > >
> > > > The unabridged reason for doing this is as follows.  We would like the
> > > > Broadcom STB PCIe root complex driver (and others) to be able to turn
> > > > off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> > > > the drivers of these endpoint devices are stock Linux drivers that are not
> > > > aware that these regulator(s) exist and must be turned on for the driver to
> > > > be probed.  The simple solution of course is to turn these regulators on at
> > > > boot and keep them on.  However, this solution does not satisfy at least
> > > > three of our usage modes:
> > > >
> > > > 1. For example, one customer uses multiple PCIe controllers, but wants the
> > > > ability to, by script invoking and unbind, turn any or all of them by and
> > > > their subdevices off to save power, e.g. when in battery mode.
> > > >
> > > > 2. Another example is when a watchdog script discovers that an endpoint
> > > > device is in an unresponsive state and would like to unbind, power toggle,
> > > > and re-bind just the PCIe endpoint and controller.
> > > >
> > > > 3. Of course we also want power turned off during suspend mode.  However,
> > > > some endpoint devices may be able to "wake" during suspend and we need to
> > > > recognise this case and veto the nominal act of turning off its regulator.
> > > > Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> > > > end-point device needs to be kept powered on in order to receive network
> > > > packets and wake-up the system.
> > > >
> > > > In all of these cases it is advantageous for the PCIe controller to govern
> > > > the turning off/on the regulators needed by the endpoint device.  The first
> > > > two cases can be done by simply unbinding and binding the PCIe controller,
> > > > if the controller has control of these regulators.
> > > >
> > > > [1] These regulators typically govern the actual power supply to the
> > > >     endpoint chip.  Sometimes they may be a the official PCIe socket
> > > >     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
> > > >     the regulator(s) that supply power to the EP chip.
> > > >
> > > > [2] The 99% configuration of our boards is a single endpoint device
> > > >     attached to the PCIe controller.  I use the term endpoint but it could
> > > >     possible mean a switch as well.
> > > >
> > > > Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> > > > ---
> > > >  drivers/pci/bus.c              | 72 ++++++++++++++++++++++++++++++++++
> > > >  drivers/pci/pci.h              |  8 ++++
> > > >  drivers/pci/pcie/portdrv_pci.c | 32 +++++++++++++++
> > > >  3 files changed, 112 insertions(+)
> > > >
> > > > diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> > > > index 3cef835b375f..c39fdf36b0ad 100644
> > > > --- a/drivers/pci/bus.c
> > > > +++ b/drivers/pci/bus.c
> > > > @@ -419,3 +419,75 @@ void pci_bus_put(struct pci_bus *bus)
> > > >         if (bus)
> > > >                 put_device(&bus->dev);
> > > >  }
> > > > +
> > > > +static void *alloc_subdev_regulators(struct device *dev)
> > > > +{
> > > > +       static const char * const supplies[] = {
> > > > +               "vpcie3v3",
> > > > +               "vpcie3v3aux",
> > > > +               "vpcie12v",
> > > > +       };
> > > > +       const size_t size = sizeof(struct subdev_regulators)
> > > > +               + sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
> > > > +       struct subdev_regulators *sr;
> > > > +       int i;
> > > > +
> > > > +       sr = devm_kzalloc(dev, size, GFP_KERNEL);
> > > > +
> > > > +       if (sr) {
> > > > +               sr->num_supplies = ARRAY_SIZE(supplies);
> > > > +               for (i = 0; i < ARRAY_SIZE(supplies); i++)
> > > > +                       sr->supplies[i].supply = supplies[i];
> > > > +       }
> > > > +
> > > > +       return sr;
> > > > +}
> > > > +
> > > > +
> > > > +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> > > > +{
> > > > +       struct device *dev = &bus->dev;
> > > > +       struct subdev_regulators *sr;
> > > > +       int ret;
> > > > +
> > > > +       if (!pcie_is_port_dev(bus->self))
> > > > +               return 0;
> > > > +
> > > > +       if (WARN_ON(bus->dev.driver_data))
> > > > +               dev_err(dev, "multiple clients using dev.driver_data\n");
> > > > +
> > > > +       sr = alloc_subdev_regulators(&bus->dev);
> > > > +       if (!sr)
> > > > +               return -ENOMEM;
> > > > +
> > > > +       bus->dev.driver_data = sr;
> > > > +       ret = regulator_bulk_get(dev, sr->num_supplies, sr->supplies);
> > > > +       if (ret)
> > > > +               return ret;
> > > > +
> > > > +       ret = regulator_bulk_enable(sr->num_supplies, sr->supplies);
> > > > +       if (ret) {
> > > > +               dev_err(dev, "failed to enable regulators for downstream device\n");
> > > > +               return ret;
> > > > +       }
> > > > +
> > > > +       return 0;
> > > > +}
> > > > +EXPORT_SYMBOL_GPL(pci_subdev_regulators_add_bus);
> > >
> > > Can't these just go in the portdrv probe and remove functions now?
> > >
> > > Rob
> >
> > Not really.  The idea is that  only when a host controller driver does this
> >
> > static struct pci_ops my_pcie_ops = {
> >     .add_bus = pci_subdev_regulators_add_bus , /* see  note below */
> >     .remove_bus = pci_subdev_regulators_remove_bus,
> >     ...
> > }
> >
> > does it explicitly want this feature.  Without doing this, every PCI
> > port in the world will execute a devm_kzalloc() and
> > devm_regulator_bulk_get() to (likely) grab nothing, and then there
> > will be three superfluous lines in the boot log:
> 
> You can opt-in based on there being a DT node.
> 
> > pci_bus 0001:01: 0001:01 supply vpcie12v not found, using dummy regulator
> > pci_bus 0001:01: 0001:01 supply vpcie3v3 not found, using dummy regulator
> > pci_bus 0001:01: 0001:01 supply vpcie3v3aux not found, using dummy regulator
> 
> This would be annoying, but not really a reason for how to design this.
> 
> > Secondly, our  HW needs to know when the  alloc/get/enable of
> > regulators is done so that the PCIe link can then be attempted.   This
> > is pretty much the cornerstone of this patchset.   To do this the brcm
> > RC driver's call to pci_subdev_regulators_add_bus() is wrapped by
> > brcm_pcie_add_bus() so that we can do this:
> >
> > static struct pci_ops my_pcie_ops = {
> >     .add_bus = brcm_pcie_add_bus ,   /* calls pci_subdev_regulators_add_bus() */
> >     .remove_bus = pci_subdev_regulators_remove_bus,
> 
> Do add_bus/remove_bus get called during resume/suspend? If not, how do
> you handle the link during resume?
> 
> Maybe there needs to be explicit hooks for link handling. Pali has
> been looking into this some.
> 
> Rob

Yes, I was looking at it... main power (12V/3.3V) and AUX power (3.3V)
needs to be supplied at the "correct" time during establishing link
procedure. I wrote it in my RFC email:
https://lore.kernel.org/linux-pci/20211022183808.jdeo7vntnagqkg7g@pali/

I'm not sure if regulator API is the most suitable for this task in PCI
core code as there are planty ways how it can be controllers. My idea
presented in that email was that driver provides power callback and core
pci code would use it.

Because power needs to be enabled at the "correct" time during link up,
I think that add/remove bus callbacks are unsuitable for this task. This
would just cause adding another msleep() calls on different places to
make correct timing of link up...

I think it is needed to implement generic function for establishing link
in pci core code with all required steps.

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators
  2021-11-16 17:41       ` Rob Herring
  2021-11-16 20:53         ` Pali Rohár
@ 2021-11-17 14:46         ` Jim Quinlan
  1 sibling, 0 replies; 35+ messages in thread
From: Jim Quinlan @ 2021-11-17 14:46 UTC (permalink / raw)
  To: Rob Herring
  Cc: Pali Rohár, Jim Quinlan, PCI, Bjorn Helgaas,
	Nicolas Saenz Julienne, Mark Brown,
	maintainer:BROADCOM BCM7XXX ARM ARCHITECTURE, Sean V Kelley,
	Jonathan Cameron, Qiuxu Zhuo, Keith Busch, open list

On Tue, Nov 16, 2021 at 12:41 PM Rob Herring <robh@kernel.org> wrote:
>
> +Pali
>
> On Mon, Nov 15, 2021 at 2:44 PM Jim Quinlan <james.quinlan@broadcom.com> wrote:
> >
> > On Thu, Nov 11, 2021 at 5:57 PM Rob Herring <robh@kernel.org> wrote:
> > >
> > > On Wed, Nov 10, 2021 at 4:15 PM Jim Quinlan <jim2101024@gmail.com> wrote:
> > > >
> > > > Adds a mechanism inside the root port device to identify standard PCIe
> > > > regulators in the DT, allocate them, and turn them on before the rest of
> > > > the bus is scanned during pci_host_probe().  A root complex driver can
> > > > leverage this mechanism by setting the pci_ops methods add_bus and
> > > > remove_bus to pci_subdev_regulators_{add,remove}_bus.
> > > >
> > > > The allocated structure that contains the regulators is stored in
> > > > dev.driver_data.
> > > >
> > > > The unabridged reason for doing this is as follows.  We would like the
> > > > Broadcom STB PCIe root complex driver (and others) to be able to turn
> > > > off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> > > > the drivers of these endpoint devices are stock Linux drivers that are not
> > > > aware that these regulator(s) exist and must be turned on for the driver to
> > > > be probed.  The simple solution of course is to turn these regulators on at
> > > > boot and keep them on.  However, this solution does not satisfy at least
> > > > three of our usage modes:drivers/pci/pcie/aspm.cdrivers/pci/pcie/aspm.c
> > > >
> > > > 1. For example, one customer uses multiple PCIe controllers, but wants the
> > > > ability to, by script invoking and unbind, turn any or all of them by and
> > > > their subdevices off to save power, e.g. when in battery mode.
> > > >
> > > > 2. Another example is when a watchdog script discovers that an endpoint
> > > > device is in an unresponsive state and would like to unbind, power toggle,
> > > > and re-bind just the PCIe endpoint and controller.
> > > >
> > > > 3. Of course we also want power turned off during suspend mode.  However,
> > > > some endpoint devices may be able to "wake" during suspend and we need to
> > > > recognise this case and veto the nominal act of turning off its regulator.
> > > > Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> > > > end-point device needs to be kept powered on in order to receive network
> > > > packets and wake-up the system.
> > > >
> > > > In all of these cases it is advantageous for the PCIe controller to govern
> > > > the turning off/on the regulators needed by the endpoint device.  The first
> > > > two cases can be done by simply unbinding and binding the PCIe controller,
> > > > if the controller has control of these regulators.
> > > >
> > > > [1] These regulators typically govern the actual power supply to the
> > > >     endpoint chip.  Sometimes they may be a the official PCIe socket
> > > >     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
> > > >     the regulator(s) that supply power to the EP chip.
> > > >
> > > > [2] The 99% configuration of our boards is a single endpoint device
> > > >     attached to the PCIe controller.  I use the term endpoint but it could
> > > >     possible mean a switch as well.
> > > >
> > > > Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> > > > ---
> > > >  drivers/pci/bus.c              | 72 ++++++++++++++++++++++++++++++++++
> > > >  drivers/pci/pci.h              |  8 ++++
> > > >  drivers/pci/pcie/portdrv_pci.c | 32 +++++++++++++++
> > > >  3 files changed, 112 insertions(+)
> > > >
> > > > diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> > > > index 3cef835b375f..c39fdf36b0ad 100644
> > > > --- a/drivers/pci/bus.c
> > > > +++ b/drivers/pci/bus.c
> > > > @@ -419,3 +419,75 @@ void pci_bus_put(struct pci_bus *bus)
> > > >         if (bus)
> > > >                 put_device(&bus->dev);
> > > >  }
> > > > +
> > > > +static void *alloc_subdev_regulators(struct device *dev)
> > > > +{
> > > > +       static const char * const supplies[] = {
> > > > +               "vpcie3v3",
> > > > +               "vpcie3v3aux",
> > > > +               "vpcie12v",
> > > > +       };
> > > > +       const size_t size = sizeof(struct subdev_regulators)
> > > > +               + sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
> > > > +       struct subdev_regulators *sr;
> > > > +       int i;
> > > > +
> > > > +       sr = devm_kzalloc(dev, size, GFP_KERNEL);
> > > > +
> > > > +       if (sr) {
> > > > +               sr->num_supplies = ARRAY_SIZE(supplies);
> > > > +               for (i = 0; i < ARRAY_SIZE(supplies); i++)
> > > > +                       sr->supplies[i].supply = supplies[i];
> > > > +       }
> > > > +
> > > > +       return sr;
> > > > +}
> > > > +
> > > > +
> > > > +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> > > > +{
> > > > +       struct device *dev = &bus->dev;
> > > > +       struct subdev_regulators *sr;
> > > > +       int ret;
> > > > +
> > > > +       if (!pcie_is_port_dev(bus->self))
> > > > +               return 0;
> > > > +
> > > > +       if (WARN_ON(bus->dev.driver_data))
> > > > +               dev_err(dev, "multiple clients using dev.driver_data\n");
> > > > +
> > > > +       sr = alloc_subdev_regulators(&bus->dev);
> > > > +       if (!sr)
> > > > +               return -ENOMEM;
> > > > +
> > > > +       bus->dev.driver_data = sr;
> > > > +       ret = regulator_bulk_get(dev, sr->num_supplies, sr->supplies);
> > > > +       if (ret)
> > > > +               return ret;
> > > > +
> > > > +       ret = regulator_bulk_enable(sr->num_supplies, sr->supplies);
> > > > +       if (ret) {
> > > > +               dev_err(dev, "failed to enable regulators for downstream device\n");
> > > > +               return ret;
> > > > +       }
> > > > +
> > > > +       return 0;
> > > > +}
> > > > +EXPORT_SYMBOL_GPL(pci_subdev_regulators_add_bus);
> > >
> > > Can't these just go in the portdrv probe and remove functions now?
> > >
> > > Rob
> >
> > Not really.  The idea is that  only when a host controller driver does this
> >
> > static struct pci_ops my_pcie_ops = {
> >     .add_bus = pci_subdev_regulators_add_bus , /* see  note below */
> >     .remove_bus = pci_subdev_regulators_remove_bus,
> >     ...
> > }
> >
> > does it explicitly want this feature.  Without doing this, every PCI
> > port in the world will execute a devm_kzalloc() and
> > devm_regulator_bulk_get() to (likely) grab nothing, and then there
> > will be three superfluous lines in the boot log:
>
> You can opt-in based on there being a DT node.
Will do, thanks.
>
> > pci_bus 0001:01: 0001:01 supply vpcie12v not found, using dummy regulator
> > pci_bus 0001:01: 0001:01 supply vpcie3v3 not found, using dummy regulator
> > pci_bus 0001:01: 0001:01 supply vpcie3v3aux not found, using dummy regulator
>
> This would be annoying, but not really a reason for how to design this.
>
> > Secondly, our  HW needs to know when the  alloc/get/enable of
> > regulators is done so that the PCIe link can then be attempted.   This
> > is pretty much the cornerstone of this patchset.   To do this the brcm
> > RC driver's call to pci_subdev_regulators_add_bus() is wrapped by
> > brcm_pcie_add_bus() so that we can do this:
> >
> > static struct pci_ops my_pcie_ops = {
> >     .add_bus = brcm_pcie_add_bus ,   /* calls pci_subdev_regulators_add_bus() */
> >     .remove_bus = pci_subdev_regulators_remove_bus,
>
> Do add_bus/remove_bus get called during resume/suspend? If not, how do
> you handle the link during resume?
No to the first question, and incorrectly to the second.  As it turns
out, my suspend/resume test for this patchset was flawed and wasn't
testing what I thought it was.  The suspend/resume is broken in V8.

Using pcie_port_device_resume_noirq() is not early enough to turn on
the regulators during resume.  However,  if I have my RC driver handle
 only the suspend/resume of the regulators, all is fine and as an
added benefit the patchset has no modifications to
portdrv_{pci,core}.c.  This is what I will submit for V9, assuming
there is one.

>
> Maybe there needs to be explicit hooks for link handling. Pali has
> been looking into this some.
I will reply to Pali's recent email now.
Regards,
Jim Quinlan
Broadcom STB
>
> Rob

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators
  2021-11-16 20:53         ` Pali Rohár
@ 2021-11-17 15:14           ` Jim Quinlan
  2021-11-17 15:45             ` Pali Rohár
  0 siblings, 1 reply; 35+ messages in thread
From: Jim Quinlan @ 2021-11-17 15:14 UTC (permalink / raw)
  To: Pali Rohár
  Cc: Rob Herring, Jim Quinlan, PCI, Bjorn Helgaas,
	Nicolas Saenz Julienne, Mark Brown,
	maintainer:BROADCOM BCM7XXX ARM ARCHITECTURE, Sean V Kelley,
	Jonathan Cameron, Qiuxu Zhuo, Keith Busch, open list

On Tue, Nov 16, 2021 at 3:53 PM Pali Rohár <pali@kernel.org> wrote:
>
> On Tuesday 16 November 2021 11:41:22 Rob Herring wrote:
> > +Pali
> >
> > On Mon, Nov 15, 2021 at 2:44 PM Jim Quinlan <james.quinlan@broadcom.com> wrote:
> > >
> > > On Thu, Nov 11, 2021 at 5:57 PM Rob Herring <robh@kernel.org> wrote:
> > > >
> > > > On Wed, Nov 10, 2021 at 4:15 PM Jim Quinlan <jim2101024@gmail.com> wrote:
> > > > >
> > > > > Adds a mechanism inside the root port device to identify standard PCIe
> > > > > regulators in the DT, allocate them, and turn them on before the rest of
> > > > > the bus is scanned during pci_host_probe().  A root complex driver can
> > > > > leverage this mechanism by setting the pci_ops methods add_bus and
> > > > > remove_bus to pci_subdev_regulators_{add,remove}_bus.
> > > > >
> > > > > The allocated structure that contains the regulators is stored in
> > > > > dev.driver_data.
> > > > >
> > > > > The unabridged reason for doing this is as follows.  We would like the
> > > > > Broadcom STB PCIe root complex driver (and others) to be able to turn
> > > > > off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> > > > > the drivers of these endpoint devices are stock Linux drivers that are not
> > > > > aware that these regulator(s) exist and must be turned on for the driver to
> > > > > be probed.  The simple solution of course is to turn these regulators on at
> > > > > boot and keep them on.  However, this solution does not satisfy at least
> > > > > three of our usage modes:
> > > > >
> > > > > 1. For example, one customer uses multiple PCIe controllers, but wants the
> > > > > ability to, by script invoking and unbind, turn any or all of them by and
> > > > > their subdevices off to save power, e.g. when in battery mode.
> > > > >
> > > > > 2. Another example is when a watchdog script discovers that an endpoint
> > > > > device is in an unresponsive state and would like to unbind, power toggle,
> > > > > and re-bind just the PCIe endpoint and controller.
> > > > >
> > > > > 3. Of course we also want power turned off during suspend mode.  However,
> > > > > some endpoint devices may be able to "wake" during suspend and we need to
> > > > > recognise this case and veto the nominal act of turning off its regulator.
> > > > > Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> > > > > end-point device needs to be kept powered on in order to receive network
> > > > > packets and wake-up the system.
> > > > >
> > > > > In all of these cases it is advantageous for the PCIe controller to govern
> > > > > the turning off/on the regulators needed by the endpoint device.  The first
> > > > > two cases can be done by simply unbinding and binding the PCIe controller,
> > > > > if the controller has control of these regulators.
> > > > >
> > > > > [1] These regulators typically govern the actual power supply to the
> > > > >     endpoint chip.  Sometimes they may be a the official PCIe socket
> > > > >     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
> > > > >     the regulator(s) that supply power to the EP chip.
> > > > >
> > > > > [2] The 99% configuration of our boards is a single endpoint device
> > > > >     attached to the PCIe controller.  I use the term endpoint but it could
> > > > >     possible mean a switch as well.
> > > > >
> > > > > Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> > > > > ---
> > > > >  drivers/pci/bus.c              | 72 ++++++++++++++++++++++++++++++++++
> > > > >  drivers/pci/pci.h              |  8 ++++
> > > > >  drivers/pci/pcie/portdrv_pci.c | 32 +++++++++++++++
> > > > >  3 files changed, 112 insertions(+)
> > > > >
> > > > > diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> > > > > index 3cef835b375f..c39fdf36b0ad 100644
> > > > > --- a/drivers/pci/bus.c
> > > > > +++ b/drivers/pci/bus.c
> > > > > @@ -419,3 +419,75 @@ void pci_bus_put(struct pci_bus *bus)
> > > > >         if (bus)
> > > > >                 put_device(&bus->dev);
> > > > >  }
> > > > > +
> > > > > +static void *alloc_subdev_regulators(struct device *dev)
> > > > > +{
> > > > > +       static const char * const supplies[] = {
> > > > > +               "vpcie3v3",
> > > > > +               "vpcie3v3aux",
> > > > > +               "vpcie12v",
> > > > > +       };
> > > > > +       const size_t size = sizeof(struct subdev_regulators)
> > > > > +               + sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
> > > > > +       struct subdev_regulators *sr;
> > > > > +       int i;
> > > > > +
> > > > > +       sr = devm_kzalloc(dev, size, GFP_KERNEL);
> > > > > +
> > > > > +       if (sr) {
> > > > > +               sr->num_supplies = ARRAY_SIZE(supplies);
> > > > > +               for (i = 0; i < ARRAY_SIZE(supplies); i++)
> > > > > +                       sr->supplies[i].supply = supplies[i];
> > > > > +       }
> > > > > +
> > > > > +       return sr;
> > > > > +}
> > > > > +
> > > > > +
> > > > > +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> > > > > +{
> > > > > +       struct device *dev = &bus->dev;
> > > > > +       struct subdev_regulators *sr;
> > > > > +       int ret;
> > > > > +
> > > > > +       if (!pcie_is_port_dev(bus->self))
> > > > > +               return 0;
> > > > > +
> > > > > +       if (WARN_ON(bus->dev.driver_data))
> > > > > +               dev_err(dev, "multiple clients using dev.driver_data\n");
> > > > > +
> > > > > +       sr = alloc_subdev_regulators(&bus->dev);
> > > > > +       if (!sr)
> > > > > +               return -ENOMEM;
> > > > > +
> > > > > +       bus->dev.driver_data = sr;
> > > > > +       ret = regulator_bulk_get(dev, sr->num_supplies, sr->supplies);
> > > > > +       if (ret)
> > > > > +               return ret;
> > > > > +
> > > > > +       ret = regulator_bulk_enable(sr->num_supplies, sr->supplies);
> > > > > +       if (ret) {
> > > > > +               dev_err(dev, "failed to enable regulators for downstream device\n");
> > > > > +               return ret;
> > > > > +       }
> > > > > +
> > > > > +       return 0;
> > > > > +}
> > > > > +EXPORT_SYMBOL_GPL(pci_subdev_regulators_add_bus);
> > > >
> > > > Can't these just go in the portdrv probe and remove functions now?
> > > >
> > > > Rob
> > >
> > > Not really.  The idea is that  only when a host controller driver does this
> > >
> > > static struct pci_ops my_pcie_ops = {
> > >     .add_bus = pci_subdev_regulators_add_bus , /* see  note below */
> > >     .remove_bus = pci_subdev_regulators_remove_bus,
> > >     ...
> > > }
> > >
> > > does it explicitly want this feature.  Without doing this, every PCI
> > > port in the world will execute a devm_kzalloc() and
> > > devm_regulator_bulk_get() to (likely) grab nothing, and then there
> > > will be three superfluous lines in the boot log:
> >
> > You can opt-in based on there being a DT node.
> >
> > > pci_bus 0001:01: 0001:01 supply vpcie12v not found, using dummy regulator
> > > pci_bus 0001:01: 0001:01 supply vpcie3v3 not found, using dummy regulator
> > > pci_bus 0001:01: 0001:01 supply vpcie3v3aux not found, using dummy regulator
> >
> > This would be annoying, but not really a reason for how to design this.
> >
> > > Secondly, our  HW needs to know when the  alloc/get/enable of
> > > regulators is done so that the PCIe link can then be attempted.   This
> > > is pretty much the cornerstone of this patchset.   To do this the brcm
> > > RC driver's call to pci_subdev_regulators_add_bus() is wrapped by
> > > brcm_pcie_add_bus() so that we can do this:
> > >
> > > static struct pci_ops my_pcie_ops = {
> > >     .add_bus = brcm_pcie_add_bus ,   /* calls pci_subdev_regulators_add_bus() */
> > >     .remove_bus = pci_subdev_regulators_remove_bus,
> >
> > Do add_bus/remove_bus get called during resume/suspend? If not, how do
> > you handle the link during resume?
> >
> > Maybe there needs to be explicit hooks for link handling. Pali has
> > been looking into this some.
> >
> > Rob
>
> Yes, I was looking at it... main power (12V/3.3V) and AUX power (3.3V)
> needs to be supplied at the "correct" time during establishing link
> procedure. I wrote it in my RFC email:
> https://lore.kernel.org/linux-pci/20211022183808.jdeo7vntnagqkg7g@pali/
Hello Pali,

I really like your proposal although I would like to get my patchset
first :-) :-)

Suppose you came up with a patchset for your ideas-- would that include
changes to existing RC drivers to use the proposed framework?  If so,
I am wary that it would
break at least a few of them.  Or would you just present the framework
and allow the
RC drivers' authors to opt-in, one by one?

At any rate, if you want someone to test some of your ideas I can work
with you.

Regards,
Jim Quinlan
Broadcom STB


>
> I'm not sure if regulator API is the most suitable for this task in PCI
> core code as there are planty ways how it can be controllers. My idea
> presented in that email was that driver provides power callback and core
> pci code would use it.
>
> Because power needs to be enabled at the "correct" time during link up,
> I think that add/remove bus callbacks are unsuitable for this task. This
> would just cause adding another msleep() calls on different places to
> make correct timing of link up...
>
> I think it is needed to implement generic function for establishing link
> in pci core code with all required steps.

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators
  2021-11-17 15:14           ` Jim Quinlan
@ 2021-11-17 15:45             ` Pali Rohár
  2021-11-18 15:36               ` Jim Quinlan
  0 siblings, 1 reply; 35+ messages in thread
From: Pali Rohár @ 2021-11-17 15:45 UTC (permalink / raw)
  To: Jim Quinlan
  Cc: Rob Herring, Jim Quinlan, PCI, Bjorn Helgaas,
	Nicolas Saenz Julienne, Mark Brown,
	maintainer:BROADCOM BCM7XXX ARM ARCHITECTURE, Sean V Kelley,
	Jonathan Cameron, Qiuxu Zhuo, Keith Busch, open list

On Wednesday 17 November 2021 10:14:19 Jim Quinlan wrote:
> On Tue, Nov 16, 2021 at 3:53 PM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Tuesday 16 November 2021 11:41:22 Rob Herring wrote:
> > > +Pali
> > >
> > > On Mon, Nov 15, 2021 at 2:44 PM Jim Quinlan <james.quinlan@broadcom.com> wrote:
> > > >
> > > > On Thu, Nov 11, 2021 at 5:57 PM Rob Herring <robh@kernel.org> wrote:
> > > > >
> > > > > On Wed, Nov 10, 2021 at 4:15 PM Jim Quinlan <jim2101024@gmail.com> wrote:
> > > > > >
> > > > > > Adds a mechanism inside the root port device to identify standard PCIe
> > > > > > regulators in the DT, allocate them, and turn them on before the rest of
> > > > > > the bus is scanned during pci_host_probe().  A root complex driver can
> > > > > > leverage this mechanism by setting the pci_ops methods add_bus and
> > > > > > remove_bus to pci_subdev_regulators_{add,remove}_bus.
> > > > > >
> > > > > > The allocated structure that contains the regulators is stored in
> > > > > > dev.driver_data.
> > > > > >
> > > > > > The unabridged reason for doing this is as follows.  We would like the
> > > > > > Broadcom STB PCIe root complex driver (and others) to be able to turn
> > > > > > off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> > > > > > the drivers of these endpoint devices are stock Linux drivers that are not
> > > > > > aware that these regulator(s) exist and must be turned on for the driver to
> > > > > > be probed.  The simple solution of course is to turn these regulators on at
> > > > > > boot and keep them on.  However, this solution does not satisfy at least
> > > > > > three of our usage modes:
> > > > > >
> > > > > > 1. For example, one customer uses multiple PCIe controllers, but wants the
> > > > > > ability to, by script invoking and unbind, turn any or all of them by and
> > > > > > their subdevices off to save power, e.g. when in battery mode.
> > > > > >
> > > > > > 2. Another example is when a watchdog script discovers that an endpoint
> > > > > > device is in an unresponsive state and would like to unbind, power toggle,
> > > > > > and re-bind just the PCIe endpoint and controller.
> > > > > >
> > > > > > 3. Of course we also want power turned off during suspend mode.  However,
> > > > > > some endpoint devices may be able to "wake" during suspend and we need to
> > > > > > recognise this case and veto the nominal act of turning off its regulator.
> > > > > > Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> > > > > > end-point device needs to be kept powered on in order to receive network
> > > > > > packets and wake-up the system.
> > > > > >
> > > > > > In all of these cases it is advantageous for the PCIe controller to govern
> > > > > > the turning off/on the regulators needed by the endpoint device.  The first
> > > > > > two cases can be done by simply unbinding and binding the PCIe controller,
> > > > > > if the controller has control of these regulators.
> > > > > >
> > > > > > [1] These regulators typically govern the actual power supply to the
> > > > > >     endpoint chip.  Sometimes they may be a the official PCIe socket
> > > > > >     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
> > > > > >     the regulator(s) that supply power to the EP chip.
> > > > > >
> > > > > > [2] The 99% configuration of our boards is a single endpoint device
> > > > > >     attached to the PCIe controller.  I use the term endpoint but it could
> > > > > >     possible mean a switch as well.
> > > > > >
> > > > > > Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> > > > > > ---
> > > > > >  drivers/pci/bus.c              | 72 ++++++++++++++++++++++++++++++++++
> > > > > >  drivers/pci/pci.h              |  8 ++++
> > > > > >  drivers/pci/pcie/portdrv_pci.c | 32 +++++++++++++++
> > > > > >  3 files changed, 112 insertions(+)
> > > > > >
> > > > > > diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> > > > > > index 3cef835b375f..c39fdf36b0ad 100644
> > > > > > --- a/drivers/pci/bus.c
> > > > > > +++ b/drivers/pci/bus.c
> > > > > > @@ -419,3 +419,75 @@ void pci_bus_put(struct pci_bus *bus)
> > > > > >         if (bus)
> > > > > >                 put_device(&bus->dev);
> > > > > >  }
> > > > > > +
> > > > > > +static void *alloc_subdev_regulators(struct device *dev)
> > > > > > +{
> > > > > > +       static const char * const supplies[] = {
> > > > > > +               "vpcie3v3",
> > > > > > +               "vpcie3v3aux",
> > > > > > +               "vpcie12v",
> > > > > > +       };
> > > > > > +       const size_t size = sizeof(struct subdev_regulators)
> > > > > > +               + sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
> > > > > > +       struct subdev_regulators *sr;
> > > > > > +       int i;
> > > > > > +
> > > > > > +       sr = devm_kzalloc(dev, size, GFP_KERNEL);
> > > > > > +
> > > > > > +       if (sr) {
> > > > > > +               sr->num_supplies = ARRAY_SIZE(supplies);
> > > > > > +               for (i = 0; i < ARRAY_SIZE(supplies); i++)
> > > > > > +                       sr->supplies[i].supply = supplies[i];
> > > > > > +       }
> > > > > > +
> > > > > > +       return sr;
> > > > > > +}
> > > > > > +
> > > > > > +
> > > > > > +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> > > > > > +{
> > > > > > +       struct device *dev = &bus->dev;
> > > > > > +       struct subdev_regulators *sr;
> > > > > > +       int ret;
> > > > > > +
> > > > > > +       if (!pcie_is_port_dev(bus->self))
> > > > > > +               return 0;
> > > > > > +
> > > > > > +       if (WARN_ON(bus->dev.driver_data))
> > > > > > +               dev_err(dev, "multiple clients using dev.driver_data\n");
> > > > > > +
> > > > > > +       sr = alloc_subdev_regulators(&bus->dev);
> > > > > > +       if (!sr)
> > > > > > +               return -ENOMEM;
> > > > > > +
> > > > > > +       bus->dev.driver_data = sr;
> > > > > > +       ret = regulator_bulk_get(dev, sr->num_supplies, sr->supplies);
> > > > > > +       if (ret)
> > > > > > +               return ret;
> > > > > > +
> > > > > > +       ret = regulator_bulk_enable(sr->num_supplies, sr->supplies);
> > > > > > +       if (ret) {
> > > > > > +               dev_err(dev, "failed to enable regulators for downstream device\n");
> > > > > > +               return ret;
> > > > > > +       }
> > > > > > +
> > > > > > +       return 0;
> > > > > > +}
> > > > > > +EXPORT_SYMBOL_GPL(pci_subdev_regulators_add_bus);
> > > > >
> > > > > Can't these just go in the portdrv probe and remove functions now?
> > > > >
> > > > > Rob
> > > >
> > > > Not really.  The idea is that  only when a host controller driver does this
> > > >
> > > > static struct pci_ops my_pcie_ops = {
> > > >     .add_bus = pci_subdev_regulators_add_bus , /* see  note below */
> > > >     .remove_bus = pci_subdev_regulators_remove_bus,
> > > >     ...
> > > > }
> > > >
> > > > does it explicitly want this feature.  Without doing this, every PCI
> > > > port in the world will execute a devm_kzalloc() and
> > > > devm_regulator_bulk_get() to (likely) grab nothing, and then there
> > > > will be three superfluous lines in the boot log:
> > >
> > > You can opt-in based on there being a DT node.
> > >
> > > > pci_bus 0001:01: 0001:01 supply vpcie12v not found, using dummy regulator
> > > > pci_bus 0001:01: 0001:01 supply vpcie3v3 not found, using dummy regulator
> > > > pci_bus 0001:01: 0001:01 supply vpcie3v3aux not found, using dummy regulator
> > >
> > > This would be annoying, but not really a reason for how to design this.
> > >
> > > > Secondly, our  HW needs to know when the  alloc/get/enable of
> > > > regulators is done so that the PCIe link can then be attempted.   This
> > > > is pretty much the cornerstone of this patchset.   To do this the brcm
> > > > RC driver's call to pci_subdev_regulators_add_bus() is wrapped by
> > > > brcm_pcie_add_bus() so that we can do this:
> > > >
> > > > static struct pci_ops my_pcie_ops = {
> > > >     .add_bus = brcm_pcie_add_bus ,   /* calls pci_subdev_regulators_add_bus() */
> > > >     .remove_bus = pci_subdev_regulators_remove_bus,
> > >
> > > Do add_bus/remove_bus get called during resume/suspend? If not, how do
> > > you handle the link during resume?
> > >
> > > Maybe there needs to be explicit hooks for link handling. Pali has
> > > been looking into this some.
> > >
> > > Rob
> >
> > Yes, I was looking at it... main power (12V/3.3V) and AUX power (3.3V)
> > needs to be supplied at the "correct" time during establishing link
> > procedure. I wrote it in my RFC email:
> > https://lore.kernel.org/linux-pci/20211022183808.jdeo7vntnagqkg7g@pali/
> Hello Pali,
> 
> I really like your proposal although I would like to get my patchset
> first :-) :-)
> 
> Suppose you came up with a patchset for your ideas-- would that include
> changes to existing RC drivers to use the proposed framework?  If so,
> I am wary that it would
> break at least a few of them.  Or would you just present the framework
> and allow the
> RC drivers' authors to opt-in, one by one?

My idea is to add new "framework" to allow drivers implement new
callbacks for this "framework". There would be no change in drivers
which do not provide these callbacks to ensure that nothing is going to
be broken. I'm planning to implement these callbacks only for RC drivers
for which I have hardware and can properly test to not introduce any
regression. For other existing RC drivers it is up to other authors +
testers. But to decrease future maintenance cost of all RC drivers I
expect that new drivers would not implement any ad-hoc solution in their
"probe" function and instead implement these new callbacks. That is my
idea.

> At any rate, if you want someone to test some of your ideas I can work
> with you.

Perfect! If you have any concerns or you see any issues, please reply
that my RFC email. So I can collect feedback.

Also I sent draft for updating DTS schema for PCIe devices:
https://github.com/devicetree-org/dt-schema/pull/64

> Regards,
> Jim Quinlan
> Broadcom STB
> 
> 
> >
> > I'm not sure if regulator API is the most suitable for this task in PCI
> > core code as there are planty ways how it can be controllers. My idea
> > presented in that email was that driver provides power callback and core
> > pci code would use it.
> >
> > Because power needs to be enabled at the "correct" time during link up,
> > I think that add/remove bus callbacks are unsuitable for this task. This
> > would just cause adding another msleep() calls on different places to
> > make correct timing of link up...
> >
> > I think it is needed to implement generic function for establishing link
> > in pci core code with all required steps.

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators
  2021-11-17 15:45             ` Pali Rohár
@ 2021-11-18 15:36               ` Jim Quinlan
  2021-11-18 15:50                 ` Pali Rohár
  0 siblings, 1 reply; 35+ messages in thread
From: Jim Quinlan @ 2021-11-18 15:36 UTC (permalink / raw)
  To: Pali Rohár
  Cc: Rob Herring, Jim Quinlan, PCI, Bjorn Helgaas,
	Nicolas Saenz Julienne, Mark Brown,
	maintainer:BROADCOM BCM7XXX ARM ARCHITECTURE, Sean V Kelley,
	Jonathan Cameron, Qiuxu Zhuo, Keith Busch, open list

On Wed, Nov 17, 2021 at 10:45 AM Pali Rohár <pali@kernel.org> wrote:
>
> On Wednesday 17 November 2021 10:14:19 Jim Quinlan wrote:
> > On Tue, Nov 16, 2021 at 3:53 PM Pali Rohár <pali@kernel.org> wrote:
> > >
> > > On Tuesday 16 November 2021 11:41:22 Rob Herring wrote:
> > > > +Pali
> > > >
> > > > On Mon, Nov 15, 2021 at 2:44 PM Jim Quinlan <james.quinlan@broadcom.com> wrote:
> > > > >
> > > > > On Thu, Nov 11, 2021 at 5:57 PM Rob Herring <robh@kernel.org> wrote:
> > > > > >
> > > > > > On Wed, Nov 10, 2021 at 4:15 PM Jim Quinlan <jim2101024@gmail.com> wrote:
> > > > > > >
> > > > > > > Adds a mechanism inside the root port device to identify standard PCIe
> > > > > > > regulators in the DT, allocate them, and turn them on before the rest of
> > > > > > > the bus is scanned during pci_host_probe().  A root complex driver can
> > > > > > > leverage this mechanism by setting the pci_ops methods add_bus and
> > > > > > > remove_bus to pci_subdev_regulators_{add,remove}_bus.
> > > > > > >
> > > > > > > The allocated structure that contains the regulators is stored in
> > > > > > > dev.driver_data.
> > > > > > >
> > > > > > > The unabridge    /* PCIe endpoint */d reason for doing this is as follows.  We would like the
> > > > > > > Broadcom STB PCIe root complex driver (and others) to be able to turn
> > > > > > > off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> > > > > > > the drivers of these endpoint devices are stock Linux drivers that are not
> > > > > > > aware that these regulator(s) exist and must be turned on for the driver to
> > > > > > > be probed.  The simple solution of course is to turn these regulators on at
> > > > > > > boot and keep them on.  However, this solution does not satisfy at least
> > > > > > > three of our usage modes:
> > > > > > >
> > > > > > > 1. For example, one customer uses multiple PCIe controllers, but wants the
> > > > > > > ability to, by script invoking and unbind, turn any or all of them by and
> > > > > > > their subdevices off to save power, e.g. when in battery mode.
> > > > > > >
> > > > > > > 2. Another example is when a watchdog script discovers that an endpoint
> > > > > > > device is in an unresponsive state and would like to unbind, power toggle,
> > > > > > > and re-bind just the PCIe endpoint and controller.
> > > > > > >
> > > > > > > 3. Of course we also want power turned off during suspend mode.  However,
> > > > > > > some endpoint devices may be able to "wake" during suspend and we need to
> > > > > > > recognise this case and veto the nominal act of turning off its regulator.
> > > > > > > Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> > > > > > > end-point device needs to be kept powered on in order to receive network
> > > > > > > packets and wake-up the system.
> > > > > > >
> > > > > > > In all of these cases it is advantageous for the PCIe controller to govern
> > > > > > > the turning off/on the regulators needed by the endpoint device.  The first
> > > > > > > two cases can be done by simply unbinding and binding the PCIe controller,
> > > > > > > if the controller has control of these regulators.
> > > > > > >
> > > > > > > [1] These regulators typically govern the actual power supply to the
> > > > > > >     endpoint chip.  Sometimes they may be a the official PCIe socket
> > > > > > >     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
> > > > > > >     the regulator(s) that supply power to the EP chip.
> > > > > > >
> > > > > > > [2] The 99% configuration of our boards is a single endpoint device
> > > > > > >     attached to the PCIe controller.  I use the term endpoint but it could
> > > > > > >     possible mean a switch as well.
> > > > > > >
> > > > > > > Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> > > > > > > ---
> > > > > > >  drivers/pci/bus.c              | 72 ++++++++++++++++++++++++++++++++++
> > > > > > >  drivers/pci/pci.h              |  8 ++++
> > > > > > >  drivers/pci/pcie/portdrv_pci.c | 32 +++++++++++++++
> > > > > > >  3 files changed, 112 insertions(+)
> > > > > > >
> > > > > > > diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> > > > > > > index 3cef835b375f..c39fdf36b0ad 100644
> > > > > > > --- a/drivers/pci/bus.c
> > > > > > > +++ b/drivers/pci/bus.c
> > > > > > > @@ -419,3 +419,75 @@ void pci_bus_put(struct pci_bus *bus)
> > > > > > >         if (bus)
> > > > > > >                 put_device(&bus->dev);
> > > > > > >  }
> > > > > > > +
> > > > > > > +static void *alloc_subdev_regulators(struct device *dev)
> > > > > > > +{
> > > > > > > +       static const char * const supplies[] = {
> > > > > > > +               "vpcie3v3",
> > > > > > > +               "vpcie3v3aux",
> > > > > > > +               "vpcie12v",
> > > > > > > +       };
> > > > > > > +       const size_t size = sizeof(struct subdev_regulators)
> > > > > > > +               + sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
> > > > > > > +       struct subdev_regulators *sr;
> > > > > > > +       int i;
> > > > > > > +
> > > > > > > +       sr = devm_kzalloc(dev, size, GFP_KERNEL);
> > > > > > > +
> > > > > > > +       if (sr) {
> > > > > > > +               sr->num_supplies = ARRAY_SIZE(supplies);
> > > > > > > +               for (i = 0; i < ARRAY_SIZE(supplies); i++)
> > > > > > > +                       sr->supplies[i].supply = supplies[i];
> > > > > > > +       }
> > > > > > > +
> > > > > > > +       return sr;
> > > > > > > +}
> > > > > > > +
> > > > > > > +
> > > > > > > +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> > > > > > > +{
> > > > > > > +       struct device *dev = &bus->dev;
> > > > > > > +       struct subdev_regulators *sr;
> > > > > > > +       int ret;
> > > > > > > +
> > > > > > > +       if (!pcie_is_port_dev(bus->self))
> > > > > > > +               return 0;
> > > > > > > +
> > > > > > > +       if (WARN_ON(bus->dev.driver_data))
> > > > > > > +               dev_err(dev, "multiple clients using dev.driver_data\n");
> > > > > > > +
> > > > > > > +       sr = alloc_subdev_regulators(&bus->dev);
> > > > > > > +       if (!sr)
> > > > > > > +               return -ENOMEM;
> > > > > > > +
> > > > > > > +       bus->dev.driver_data = sr;
> > > > > > > +       ret = regulator_bulk_get(dev, sr->num_supplies, sr->supplies);
> > > > > > > +       if (ret)
> > > > > > > +               return ret;
> > > > > > > +
> > > > > > > +       ret = regulator_bulk_enable(sr->num_supplies, sr->supplies);
> > > > > > > +       if (ret) {
> > > > > > > +               dev_err(dev, "failed to enable regulators for downstream device\n");
> > > > > > > +               return ret;
> > > > > > > +       }
> > > > > > > +
> > > > > > > +       return 0;
> > > > > > > +}
> > > > > > > +EXPORT_SYMBOL_GPL(pci_subdev_regulators_add_bus);
> > > > > >
> > > > > > Can't these just go in the portdrv probe and remove functions now?
> > > > > >
> > > > > > Rob
> > > > >
> > > > > Not really.  The idea is that  only when a host controller driver does this
> > > > >
> > > > > static struct pci_ops my_pcie_ops = {
> > > > >     .add_bus = pci_subdev_regulators_add_bus , /* see  note below */
> > > > >     .remove_bus = pci_subdev_regulators_remove_bus,
> > > > >     ...
> > > > > }
> > > > >
> > > > > does it explicitly want this feature.  Without doing this, every PCI
> > > > > port in the world will execute a devm_kzalloc() and
> > > > > devm_regulator_bulk_get() to (likely) grab nothing, and then there
> > > > > will be three superfluous lines in the boot log:
> > > >
> > > > You can opt-in based on there being a DT node.
> > > >
> > > > > pci_bus 0001:01: 0001:01 supply vpcie12v not found, using dummy regulator
> > > > > pci_bus 0001:01: 0001:01 supply vpcie3v3 not found, using dummy regulator
> > > > > pci_bus 0001:01: 0001:01 supply vpcie3v3aux not found, using dummy regulator
> > > >
> > > > This would be annoying, but not really a reason for how to design this.
> > > >
> > > > > Secondly, our  HW needs to know when the  alloc/get/enable of
> > > > > regulators is done so that the PCIe link can then be attempted.   This
> > > > > is pretty much the cornerstone of this patchset.   To do this the brcm
> > > > > RC driver's call to pci_subdev_regulators_add_bus() is wrapped by
> > > > > brcm_pcie_add_bus() so that we can do this:
> > > > >
> > > > > static struct pci_ops my_pcie_ops = {
> > > > >     .add_bus = brcm_pcie_add_bus ,   /* calls pci_subdev_regulators_add_bus() */
> > > > >     .remove_bus = pci_subdev_regulators_remove_bus,
> > > >
> > > > Do add_bus/remove_bus get called during resume/suspend? If not, how do
> > > > you handle the link during resume?
> > > >
> > > > Maybe there needs to be explicit hooks for link handling. Pali has
> > > > been looking into this some.
> > > >
> > > > Rob
> > >
> > > Yes, I was looking at it... main power (12V/3.3V) and AUX power (3.3V)
> > > needs to be supplied at the "correct" time during establishing link
> > > procedure. I wrote it in my RFC email:
> > > https://lore.kernel.org/linux-pci/20211022183808.jdeo7vntnagqkg7g@pali/
> > Hello Pali,
> >
> > I really like your proposal although I would like to get my patchset
> > first :-) :-)
> >
> > Suppose you came up with a patchset for your ideas-- would that include
> > changes to existing RC drivers to use the proposed framework?  If so,
> > I am wary that it would
> > break at least a few of them.  Or would you just present the framework
> > and allow the
> > RC drivers' authors to opt-in, one by one?
>
> My idea is to add new "framework" to allow drivers implement new
> callbacks for this "framework". There would be no change in drivers
> which do not provide these callbacks to ensure that nothing is going to
> be broken. I'm planning to implement these callbacks only for RC drivers
> for which I have hardware and can properly test to not introduce any
> regression. For other existing RC drivers it is up to other authors +
> testers. But to decrease future maintenance cost of all RC drivers I
> expect that new drivers would not implement any ad-hoc solution in their
> "probe" function and instead implement these new callbacks. That is my
> idea.
>
> > At any rate, if you want someone to test some of your ideas I can work
> > with you.
>
> Perfect! If you have any concerns or you see any issues, please reply
> that my RFC email. So I can collect feedback.
>
> Also I sent draft for updating DTS schema for PCIe devices:
> https://github.com/devicetree-org/dt-schema/pull/64

Hi Pali,
I don't see any mention or placement of the regulator nodes for power;
do you agree with where
I proposed we place them, ie in the first bridge under the root-complex,  e.g.

            pcie0: pcie@7d500000 {                                /*
root complex */
                    compatible = "brcm,bcm2711-pcie";
                    reg = <0x0 0x7d500000 0x9310>;

                    /* PCIe bridge */
                    pci@0,0 {
                            #address-cells = <3>;
                            #size-cells = <2>;
                            reg = <0x0 0x0 0x0 0x0 0x0>;
                            compatible = "pciclass,0604";
                            device_type = "pci";
                            vpcie3v3-supply = <&vreg7>;     /*
<------------- HERE  */
                            ranges;

                            pci-ep@0,0 {        /* PCIe endpoint */
                                    assigned-addresses =
                                        <0x82010000 0x0 0xf8000000 0x6
0x00000000 0x0 0x2000>;
                                    reg = <0x0 0x0 0x0 0x0 0x0>;
                                    compatible = "pci14e4,1688";
                                    #address-cells = <3>;
                                    #size-cells = <2>;

                                    ranges;
                            };
                    };
            };


Regards,
Jim

>
> > Regards,
> > Jim Quinlan
> > Broadcom STB
> >
> >
> > >
> > > I'm not sure if regulator API is the most suitable for this task in PCI
> > > core code as there are planty ways how it can be controllers. My idea
> > > presented in that email was that driver provides power callback and core
> > > pci code would use it.
> > >
> > > Because power needs to be enabled at the "correct" time during link up,
> > > I think that add/remove bus callbacks are unsuitable for this task. This
> > > would just cause adding another msleep() calls on different places to
> > > make correct timing of link up...
> > >
> > > I think it is needed to implement generic function for establishing link
> > > in pci core code with all required steps.

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators
  2021-11-18 15:36               ` Jim Quinlan
@ 2021-11-18 15:50                 ` Pali Rohár
  0 siblings, 0 replies; 35+ messages in thread
From: Pali Rohár @ 2021-11-18 15:50 UTC (permalink / raw)
  To: Jim Quinlan
  Cc: Rob Herring, Jim Quinlan, PCI, Bjorn Helgaas,
	Nicolas Saenz Julienne, Mark Brown,
	maintainer:BROADCOM BCM7XXX ARM ARCHITECTURE, Sean V Kelley,
	Jonathan Cameron, Qiuxu Zhuo, Keith Busch, open list

On Thursday 18 November 2021 10:36:00 Jim Quinlan wrote:
> On Wed, Nov 17, 2021 at 10:45 AM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Wednesday 17 November 2021 10:14:19 Jim Quinlan wrote:
> > > On Tue, Nov 16, 2021 at 3:53 PM Pali Rohár <pali@kernel.org> wrote:
> > > >
> > > > Yes, I was looking at it... main power (12V/3.3V) and AUX power (3.3V)
> > > > needs to be supplied at the "correct" time during establishing link
> > > > procedure. I wrote it in my RFC email:
> > > > https://lore.kernel.org/linux-pci/20211022183808.jdeo7vntnagqkg7g@pali/
> > > Hello Pali,
> > >
> > > I really like your proposal although I would like to get my patchset
> > > first :-) :-)
> > >
> > > Suppose you came up with a patchset for your ideas-- would that include
> > > changes to existing RC drivers to use the proposed framework?  If so,
> > > I am wary that it would
> > > break at least a few of them.  Or would you just present the framework
> > > and allow the
> > > RC drivers' authors to opt-in, one by one?
> >
> > My idea is to add new "framework" to allow drivers implement new
> > callbacks for this "framework". There would be no change in drivers
> > which do not provide these callbacks to ensure that nothing is going to
> > be broken. I'm planning to implement these callbacks only for RC drivers
> > for which I have hardware and can properly test to not introduce any
> > regression. For other existing RC drivers it is up to other authors +
> > testers. But to decrease future maintenance cost of all RC drivers I
> > expect that new drivers would not implement any ad-hoc solution in their
> > "probe" function and instead implement these new callbacks. That is my
> > idea.
> >
> > > At any rate, if you want someone to test some of your ideas I can work
> > > with you.
> >
> > Perfect! If you have any concerns or you see any issues, please reply
> > that my RFC email. So I can collect feedback.
> >
> > Also I sent draft for updating DTS schema for PCIe devices:
> > https://github.com/devicetree-org/dt-schema/pull/64
> 
> Hi Pali,
> I don't see any mention or placement of the regulator nodes for power;

I put in above pull request draft only existing attributes (from
pci.txt), I have not introduce anything new yet.

> do you agree with where
> I proposed we place them, ie in the first bridge under the root-complex,  e.g.
> 
>             pcie0: pcie@7d500000 {                                /*
> root complex */
>                     compatible = "brcm,bcm2711-pcie";
>                     reg = <0x0 0x7d500000 0x9310>;
> 
>                     /* PCIe bridge */
>                     pci@0,0 {
>                             #address-cells = <3>;
>                             #size-cells = <2>;
>                             reg = <0x0 0x0 0x0 0x0 0x0>;
>                             compatible = "pciclass,0604";
>                             device_type = "pci";
>                             vpcie3v3-supply = <&vreg7>;     /*
> <------------- HERE  */

This node 'pci@0,0' describes PCIe Root Port. So yes, it is place where
power regulators belongs. I agree with you.

(Note: I would suggest to use /* PCIe Root Port */ comment instead of
/* PCIe bridge */. As PCIe bridge is ambiguous name which could mean
also other devices.)

>                             ranges;
> 
>                             pci-ep@0,0 {        /* PCIe endpoint */
>                                     assigned-addresses =
>                                         <0x82010000 0x0 0xf8000000 0x6
> 0x00000000 0x0 0x2000>;
>                                     reg = <0x0 0x0 0x0 0x0 0x0>;
>                                     compatible = "pci14e4,1688";
>                                     #address-cells = <3>;
>                                     #size-cells = <2>;
> 
>                                     ranges;
>                             };
>                     };
>             };
> 
> 
> Regards,
> Jim

^ permalink raw reply	[flat|nested] 35+ messages in thread

end of thread, other threads:[~2021-11-18 15:50 UTC | newest]

Thread overview: 35+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-10 22:14 [PATCH v8 0/8] PCI: brcmstb: have portdrv turn on sub-device power Jim Quinlan
2021-11-10 22:14 ` [PATCH v8 1/8] PCI: brcmstb: Change brcm_phy_stop() to return void Jim Quinlan
2021-11-11 21:57   ` Bjorn Helgaas
2021-11-15 20:56     ` Jim Quinlan
2021-11-16 20:40       ` Bjorn Helgaas
2021-11-10 22:14 ` [PATCH v8 2/8] dt-bindings: PCI: Correct brcmstb interrupts, interrupt-map Jim Quinlan
2021-11-10 22:14 ` [PATCH v8 3/8] dt-bindings: PCI: Add bindings for Brcmstb EP voltage regulators Jim Quinlan
2021-11-11 22:17   ` Bjorn Helgaas
2021-11-12 18:25     ` Jim Quinlan
2021-11-12 20:20       ` Bjorn Helgaas
2021-11-12 21:46         ` Rob Herring
2021-11-13 11:38   ` Pali Rohár
2021-11-10 22:14 ` [PATCH v8 4/8] PCI/portdrv: Create pcie_is_port_dev() func from existing code Jim Quinlan
2021-11-11 21:51   ` Florian Fainelli
2021-11-11 22:53     ` Rob Herring
2021-11-11 23:50   ` Krzysztof Wilczyński
2021-11-12 18:14     ` Jim Quinlan
2021-11-10 22:14 ` [PATCH v8 5/8] PCI/portdrv: add mechanism to turn on subdev regulators Jim Quinlan
2021-11-11  9:44   ` kernel test robot
2021-11-11 22:12   ` Bjorn Helgaas
2021-11-11 22:50     ` Rob Herring
2021-11-11 22:56   ` Rob Herring
2021-11-15 20:44     ` Jim Quinlan
2021-11-16 17:41       ` Rob Herring
2021-11-16 20:53         ` Pali Rohár
2021-11-17 15:14           ` Jim Quinlan
2021-11-17 15:45             ` Pali Rohár
2021-11-18 15:36               ` Jim Quinlan
2021-11-18 15:50                 ` Pali Rohár
2021-11-17 14:46         ` Jim Quinlan
2021-11-11 23:38   ` Krzysztof Wilczyński
2021-11-15 20:26     ` Jim Quinlan
2021-11-10 22:14 ` [PATCH v8 6/8] PCI/portdrv: Do not turn off subdev regulators if EP can wake up Jim Quinlan
2021-11-10 22:14 ` [PATCH v8 7/8] PCI: brcmstb: Split brcm_pcie_setup() into two funcs Jim Quinlan
2021-11-10 22:14 ` [PATCH v8 8/8] PCI: brcmstb: Add control of subdevice voltage regulators Jim Quinlan

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).