[PATCH v2 0/3] PCI: brcmstb: CLKREQ# accomodations of downstream device

[PATCH v2 0/3] PCI: brcmstb: CLKREQ# accomodations of downstream device

[Jim Quinlan]

v2 -- Changed binding property 'brcm,completion-timeout-msec' to
      'brcm,completion-timeout-us'.  (StefanW for standard suffix).
   -- Warn when clamping timeout value, and include clamped
      region in message. Also add min and max in YAML. (StefanW)
   -- Qualify description of "brcm,completion-timeout-us" so that
      it refers to PCIe transactions. (StefanW)
   -- Remvove mention of Linux specifics in binding description. (StefanW)
   -- s/clkreq#/CLKREQ#/g (Bjorn)
   -- Refactor completion-timeout-us code to compare max and min to
      value given by the property (as opposed to the computed value).

v1 -- The current driver assumes the downstream devices can
      provide CLKREQ# for ASPM.  These commits accomodate devices
      w/ or w/o clkreq# and also handle L1SS-capable devices.

   -- The Raspian Linux folks have already been using a PCIe RC
      property "brcm,enable-l1ss".  These commits use the same
      property, in a backward-compatible manner, and the implementaion
      adds more detail and also automatically identifies devices w/o
      a clkreq# signal, i.e. most devices plugged into an RPi CM4
      IO board.

Jim Quinlan (3):
  PCI: brcmstb: CLKREQ# accomodations of downstream device
  PCI: brcmstb: Set PCIe transaction completion timeout
  blah blah

 drivers/pci/controller/pcie-brcmstb.c | 99 ++++++++++++++++++++++++---
 init/do_mounts.c                      | 16 ++++-
 2 files changed, 102 insertions(+), 13 deletions(-)


base-commit: 76f598ba7d8e2bfb4855b5298caedd5af0c374a8
prerequisite-patch-id: f38de8681d8746126d60b3430eaf218d2dd169cd
prerequisite-patch-id: 23e13189f200358976abf5bf3600973a20cf386c
prerequisite-patch-id: edfbe6ea39ed6a4937e2cec3bb8ee0e60091546d
prerequisite-patch-id: c87dd155e8506a2a277726c47d85bf3fa83727d5
prerequisite-patch-id: 579841e1dc179517506a7a7c42e0e651b3bc3649
prerequisite-patch-id: b5b079998ea451821edffd7c52cd3d89d06046a1
prerequisite-patch-id: b51b3918e554e279b2ace1f68ed6b4176f8ccc24
prerequisite-patch-id: 333e5188fb27d0ed010f5359e83e539172a67690
prerequisite-patch-id: bb107ee7b4811a9719508ea667cad2466933dec0
prerequisite-patch-id: 1258db336e778eb57d5cbea88834cd25aa1346ba
-- 
2.17.1

[PATCH v2 1/3] dt-bindings: PCI: brcmstb: Add two optional props

[Jim Quinlan]

Regarding "brcm,enable-l1ss":

  The Broadcom STB/CM PCIe HW -- a core that is also used by RPi SOCs --
  requires the driver probe() to deliberately place the HW one of three
  CLKREQ# modes:

  (a) CLKREQ# driven by the RC unconditionally
  (b) CLKREQ# driven by the EP for ASPM L0s, L1
  (c) Bidirectional CLKREQ#, as used for L1 Substates (L1SS).

  The HW+driver can tell the difference between downstream devices that
  need (a) and (b), but does not know when to configure (c).  Further, the
  HW may cause a CPU abort on boot if guesses wrong regarding the need for
  (c).  So we introduce the boolean "brcm,enable-l1ss" property to indicate
  that (c) is desired.  Setting this property only makes sense when the
  downstream device is L1SS-capable and the OS is configured to activate
  this mode (e.g. policy==superpowersave).

  This property is already present in the Raspian version of Linux, but the
  upstream driver implementaion that will follow adds more details and
  discerns between (a) and (b).

Regarding "brcm,completion-timeout-us"

  Our HW will cause a CPU abort if the L1SS exit time is longer than the
  PCIe transaction completion abort timeout.  We've been asked to make this
  configurable, so we are introducing "brcm,completion-timeout-us".

Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
---
 .../devicetree/bindings/pci/brcm,stb-pcie.yaml   | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml b/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
index 7e15aae7d69e..f7fc2f6561bb 100644
--- a/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
+++ b/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
@@ -64,6 +64,22 @@ properties:
 
   aspm-no-l0s: true
 
+  brcm,enable-l1ss:
+    description: Indicates that PCIe L1SS power savings
+      are desired, the downstream device is L1SS-capable, and the
+      OS has been configured to enable this mode.  Note that when
+      in this mode, this particular HW may not meet the requirement
+      that requires CLKREQ# assertion to clock active to be
+      within 400ns.
+    type: boolean
+
+  brcm,completion-timeout-us:
+    description: Number of microseconds before PCI transaction
+      completion timeout abort is signalled.
+    minimum: 16
+    default: 1000000
+    maximum: 19884107
+
   brcm,scb-sizes:
     description: u64 giving the 64bit PCIe memory
       viewport size of a memory controller.  There may be up to
-- 
2.17.1

[PATCH v2 2/3] PCI: brcmstb: CLKREQ# accomodations of downstream device

[Jim Quinlan]

The Broadcom STB/CM PCIe HW core, which is also used in RPi SOCs, must be
deliberately set by the probe() into one of three mutually exclusive modes:

  (a) No CLKREQ# expected or required, refclk is always available.
  (b) CLKREQ# is expected to be driven by downstream device when needed.
  (c) Bidirectional CLKREQ# for L1SS capable devices.

Previously, only (b) was supported by the driver, as almost all STB/CM
boards operate in this mode.  But now there is interest in activating L1SS
power savings from STB/CM customers, and also interest in accomodating mode
(a) for designs such as the RPi CM4 with IO board.

The HW+driver is able to tell us when mode (a) mode is needed.  But there
is no easy way to tell if L1SS mode should be configured.  In certain
situations, getting this wrong may cause a panic during boot time.  So we
rely on the DT prop "brcm,enable-l1ss" to tell us when mode (c) is desired.
Using this mode only makes sense when the downstream device is L1SS-capable
and the OS has been configured to activate L1SS
(e.g. policy==powersupersave).

This property has already been in use by Raspian Linux, but this
immplementation adds more details and discerns between (a) and (b)
automatically.

Link: https://bugzilla.kernel.org/show_bug.cgi?id=217276
Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
---
 drivers/pci/controller/pcie-brcmstb.c | 69 +++++++++++++++++++++++----
 1 file changed, 59 insertions(+), 10 deletions(-)

diff --git a/drivers/pci/controller/pcie-brcmstb.c b/drivers/pci/controller/pcie-brcmstb.c
index edf283e2b5dd..56b96aa02221 100644
--- a/drivers/pci/controller/pcie-brcmstb.c
+++ b/drivers/pci/controller/pcie-brcmstb.c
@@ -48,10 +48,17 @@
 #define PCIE_RC_CFG_PRIV1_LINK_CAPABILITY			0x04dc
 #define  PCIE_RC_CFG_PRIV1_LINK_CAPABILITY_ASPM_SUPPORT_MASK	0xc00
 
+#define PCIE_RC_CFG_PRIV1_ROOT_CAP			0x4f8
+#define  PCIE_RC_CFG_PRIV1_ROOT_CAP_L1SS_MODE_MASK	0xf8
+
 #define PCIE_RC_DL_MDIO_ADDR				0x1100
 #define PCIE_RC_DL_MDIO_WR_DATA				0x1104
 #define PCIE_RC_DL_MDIO_RD_DATA				0x1108
 
+#define PCIE_0_RC_PL_PHY_DBG_CLKREQ2_0			0x1e30
+#define  CLKREQ2_0_CLKREQ_IN_CNT_MASK			0x3f000000
+#define  CLKREQ2_0_CLKREQ_IN_MASK			0x40000000
+
 #define PCIE_MISC_MISC_CTRL				0x4008
 #define  PCIE_MISC_MISC_CTRL_PCIE_RCB_64B_MODE_MASK	0x80
 #define  PCIE_MISC_MISC_CTRL_PCIE_RCB_MPS_MODE_MASK	0x400
@@ -121,9 +128,12 @@
 
 #define PCIE_MISC_HARD_PCIE_HARD_DEBUG					0x4204
 #define  PCIE_MISC_HARD_PCIE_HARD_DEBUG_CLKREQ_DEBUG_ENABLE_MASK	0x2
+#define  PCIE_MISC_HARD_PCIE_HARD_DEBUG_L1SS_ENABLE_MASK		0x200000
 #define  PCIE_MISC_HARD_PCIE_HARD_DEBUG_SERDES_IDDQ_MASK		0x08000000
 #define  PCIE_BMIPS_MISC_HARD_PCIE_HARD_DEBUG_SERDES_IDDQ_MASK		0x00800000
-
+#define  PCIE_CLKREQ_MASK \
+	  (PCIE_MISC_HARD_PCIE_HARD_DEBUG_CLKREQ_DEBUG_ENABLE_MASK | \
+	   PCIE_MISC_HARD_PCIE_HARD_DEBUG_L1SS_ENABLE_MASK)
 
 #define PCIE_INTR2_CPU_BASE		0x4300
 #define PCIE_MSI_INTR2_BASE		0x4500
@@ -1024,13 +1034,58 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
 	return 0;
 }
 
+static void brcm_config_clkreq(struct brcm_pcie *pcie)
+{
+	bool l1ss = of_property_read_bool(pcie->np, "brcm,enable-l1ss");
+	void __iomem *base = pcie->base;
+	u32 clkreq_set, tmp = readl(base + PCIE_0_RC_PL_PHY_DBG_CLKREQ2_0);
+	bool clkreq_in_seen;
+
+	/*
+	 * We have "seen" CLKREQ# if it is asserted or has been in the past.
+	 * Note that the CLKREQ_IN_MASK is 1 if CLKREQ# is asserted.
+	 */
+	clkreq_in_seen = !!(tmp & CLKREQ2_0_CLKREQ_IN_MASK) ||
+		!!FIELD_GET(CLKREQ2_0_CLKREQ_IN_CNT_MASK, tmp);
+
+	/* Start with safest setting where we provide refclk regardless */
+	clkreq_set = readl(pcie->base + PCIE_MISC_HARD_PCIE_HARD_DEBUG) &
+		~PCIE_CLKREQ_MASK;
+
+	if (l1ss && IS_ENABLED(CONFIG_PCIEASPM)) {
+		/*
+		 * Note: This (l1ss) mode may not meet requirement for
+		 * downstream devices that require CLKREQ# assertion to
+		 * clock active within 400ns.
+		 */
+		clkreq_set |= PCIE_MISC_HARD_PCIE_HARD_DEBUG_L1SS_ENABLE_MASK;
+		dev_info(pcie->dev, "bi-dir CLKREQ# for l1ss-capable devs\n");
+		dev_info(pcie->dev, "ASPM policy should be set to \"powersupersave\"\n");
+	} else {
+		if (clkreq_in_seen && IS_ENABLED(CONFIG_PCIEASPM)) {
+			clkreq_set |= PCIE_MISC_HARD_PCIE_HARD_DEBUG_CLKREQ_DEBUG_ENABLE_MASK;
+			dev_info(pcie->dev, "uni-dir CLKREQ# for ASPM\n");
+		} else {
+			dev_info(pcie->dev, "CLKREQ# ignored; no ASPM\n");
+			/* Might as well unadvertise ASPM */
+			tmp = readl(base + PCIE_RC_CFG_PRIV1_LINK_CAPABILITY) &
+				~PCIE_RC_CFG_PRIV1_LINK_CAPABILITY_ASPM_SUPPORT_MASK;
+			writel(tmp, base + PCIE_RC_CFG_PRIV1_LINK_CAPABILITY);
+		}
+		/* Setting the field to 2 unadvertises L1SS support */
+		tmp = readl(base + PCIE_RC_CFG_PRIV1_ROOT_CAP);
+		u32p_replace_bits(&tmp, 2, PCIE_RC_CFG_PRIV1_ROOT_CAP_L1SS_MODE_MASK);
+		writel(tmp, base + PCIE_RC_CFG_PRIV1_ROOT_CAP);
+	}
+	writel(clkreq_set, pcie->base + PCIE_MISC_HARD_PCIE_HARD_DEBUG);
+}
+
 static int brcm_pcie_start_link(struct brcm_pcie *pcie)
 {
 	struct device *dev = pcie->dev;
 	void __iomem *base = pcie->base;
 	u16 nlw, cls, lnksta;
 	bool ssc_good = false;
-	u32 tmp;
 	int ret, i;
 
 	/* Unassert the fundamental reset */
@@ -1055,6 +1110,8 @@ static int brcm_pcie_start_link(struct brcm_pcie *pcie)
 		return -ENODEV;
 	}
 
+	brcm_config_clkreq(pcie);
+
 	if (pcie->gen)
 		brcm_pcie_set_gen(pcie, pcie->gen);
 
@@ -1073,14 +1130,6 @@ static int brcm_pcie_start_link(struct brcm_pcie *pcie)
 		 pci_speed_string(pcie_link_speed[cls]), nlw,
 		 ssc_good ? "(SSC)" : "(!SSC)");
 
-	/*
-	 * Refclk from RC should be gated with CLKREQ# input when ASPM L0s,L1
-	 * is enabled => setting the CLKREQ_DEBUG_ENABLE field to 1.
-	 */
-	tmp = readl(base + PCIE_MISC_HARD_PCIE_HARD_DEBUG);
-	tmp |= PCIE_MISC_HARD_PCIE_HARD_DEBUG_CLKREQ_DEBUG_ENABLE_MASK;
-	writel(tmp, base + PCIE_MISC_HARD_PCIE_HARD_DEBUG);
-
 	return 0;
 }
 
-- 
2.17.1

[PATCH v2 3/3] PCI: brcmstb: Set PCIe transaction completion timeout

[Jim Quinlan]

Since the STB PCIe HW will cause a CPU abort on a PCIe transaction
completion timeout abort, we might as well extend the default timeout
limit.  Further, different devices and systems may requires a larger or
smaller amount commensurate with their L1SS exit time, so the property
"brcm,completion-abort-us" may be used to set a custom timeout value.

Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
---
 drivers/pci/controller/pcie-brcmstb.c | 30 +++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/drivers/pci/controller/pcie-brcmstb.c b/drivers/pci/controller/pcie-brcmstb.c
index 56b96aa02221..9610066f8c80 100644
--- a/drivers/pci/controller/pcie-brcmstb.c
+++ b/drivers/pci/controller/pcie-brcmstb.c
@@ -1080,6 +1080,35 @@ static void brcm_config_clkreq(struct brcm_pcie *pcie)
 	writel(clkreq_set, pcie->base + PCIE_MISC_HARD_PCIE_HARD_DEBUG);
 }
 
+static void brcm_config_completion_timeout(struct brcm_pcie *pcie)
+{
+	/* TIMEOUT register is two registers before RGR1_SW_INIT_1 */
+	const char *fmt = "brcm,completion-timeout-us clamped to region [%u..%u]\n";
+	const unsigned int REG_OFFSET = PCIE_RGR1_SW_INIT_1(pcie) - 8;
+	const u32 timeout_us_min = 16;
+	const u32 timeout_us_max = 19884107;
+	u32 timeout_us = 1000000; /* Our default, 1 second */
+	int rval, ret;
+
+	ret = of_property_read_u32(pcie->np, "brcm,completion-timeout-us",
+				   &timeout_us);
+	if (ret && ret != -EINVAL)
+		dev_err(pcie->dev, "malformed/invalid 'brcm,completion-timeout-us'\n");
+
+	/* If needed, clamp the requested timeout value and issue a warning */
+	if (timeout_us < timeout_us_min) {
+		timeout_us = timeout_us_min;
+		dev_warn(pcie->dev, fmt, timeout_us_min, timeout_us_max);
+	} else if (timeout_us > timeout_us_max) {
+		timeout_us = timeout_us_max;
+		dev_warn(pcie->dev, fmt, timeout_us_min, timeout_us_max);
+	}
+
+	/* Each unit in timeout register is 1/216,000,000 seconds */
+	rval = 216 * timeout_us;
+	writel(rval, pcie->base + REG_OFFSET);
+}
+
 static int brcm_pcie_start_link(struct brcm_pcie *pcie)
 {
 	struct device *dev = pcie->dev;
@@ -1110,6 +1139,7 @@ static int brcm_pcie_start_link(struct brcm_pcie *pcie)
 		return -ENODEV;
 	}
 
+	brcm_config_completion_timeout(pcie);
 	brcm_config_clkreq(pcie);
 
 	if (pcie->gen)
-- 
2.17.1

Re: [PATCH v2 3/3] PCI: brcmstb: Set PCIe transaction completion timeout

[Cyril Brulebois]

[-- Attachment #1: Type: text/plain, Size: 1079 bytes --]

Hi Jim,

Jim Quinlan <jim2101024@gmail.com> (2023-04-11):
> Since the STB PCIe HW will cause a CPU abort on a PCIe transaction
> completion timeout abort, we might as well extend the default timeout
> limit.  Further, different devices and systems may requires a larger or
> smaller amount commensurate with their L1SS exit time, so the property
> "brcm,completion-abort-us" may be used to set a custom timeout value.
   ^^^^^^^^^^^^^^^^^^^^^^^^

> +	ret = of_property_read_u32(pcie->np, "brcm,completion-timeout-us",
> +				   &timeout_us);
> +	if (ret && ret != -EINVAL)
> +		dev_err(pcie->dev, "malformed/invalid 'brcm,completion-timeout-us'\n");

v2 renames brcm,completion-abort-msecs into brcm,completion-timeout-us
but the commit message mentions the half-way brcm,completion-abort-us
property instead.

(Also spotted “immplementation” in 2/3 but I thought I'd spare everyone
an extra mail.)


Cheers,
-- 
Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/3] dt-bindings: PCI: brcmstb: Add two optional props

[Krzysztof Kozlowski]

On 11/04/2023 18:59, Jim Quinlan wrote:
> Regarding "brcm,enable-l1ss":
> 
>   The Broadcom STB/CM PCIe HW -- a core that is also used by RPi SOCs --
>   requires the driver probe() to deliberately place the HW one of three
>   CLKREQ# modes:
> 
>   (a) CLKREQ# driven by the RC unconditionally
>   (b) CLKREQ# driven by the EP for ASPM L0s, L1
>   (c) Bidirectional CLKREQ#, as used for L1 Substates (L1SS).
> 
>   The HW+driver can tell the difference between downstream devices that
>   need (a) and (b), but does not know when to configure (c).  Further, the
>   HW may cause a CPU abort on boot if guesses wrong regarding the need for
>   (c).  So we introduce the boolean "brcm,enable-l1ss" property to indicate
>   that (c) is desired.  Setting this property only makes sense when the
>   downstream device is L1SS-capable and the OS is configured to activate
>   this mode (e.g. policy==superpowersave).
> 
>   This property is already present in the Raspian version of Linux, but the
>   upstream driver implementaion that will follow adds more details and

typo, implementation

>   discerns between (a) and (b).
> 
> Regarding "brcm,completion-timeout-us"
> 
>   Our HW will cause a CPU abort if the L1SS exit time is longer than the
>   PCIe transaction completion abort timeout.  We've been asked to make this
>   configurable, so we are introducing "brcm,completion-timeout-us".
> 
> Signed-off-by: Jim Quinlan <jim2101024@gmail.com>

What happened here? Where is the changelog?

Best regards,
Krzysztof

Re: [PATCH v2 1/3] dt-bindings: PCI: brcmstb: Add two optional props

[Florian Fainelli]

On 4/12/2023 1:09 AM, Krzysztof Kozlowski wrote:
> On 11/04/2023 18:59, Jim Quinlan wrote:
>> Regarding "brcm,enable-l1ss":
>>
>>    The Broadcom STB/CM PCIe HW -- a core that is also used by RPi SOCs --
>>    requires the driver probe() to deliberately place the HW one of three
>>    CLKREQ# modes:
>>
>>    (a) CLKREQ# driven by the RC unconditionally
>>    (b) CLKREQ# driven by the EP for ASPM L0s, L1
>>    (c) Bidirectional CLKREQ#, as used for L1 Substates (L1SS).
>>
>>    The HW+driver can tell the difference between downstream devices that
>>    need (a) and (b), but does not know when to configure (c).  Further, the
>>    HW may cause a CPU abort on boot if guesses wrong regarding the need for
>>    (c).  So we introduce the boolean "brcm,enable-l1ss" property to indicate
>>    that (c) is desired.  Setting this property only makes sense when the
>>    downstream device is L1SS-capable and the OS is configured to activate
>>    this mode (e.g. policy==superpowersave).
>>
>>    This property is already present in the Raspian version of Linux, but the
>>    upstream driver implementaion that will follow adds more details and
> 
> typo, implementation
> 
>>    discerns between (a) and (b).
>>
>> Regarding "brcm,completion-timeout-us"
>>
>>    Our HW will cause a CPU abort if the L1SS exit time is longer than the
>>    PCIe transaction completion abort timeout.  We've been asked to make this
>>    configurable, so we are introducing "brcm,completion-timeout-us".
>>
>> Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> 
> What happened here? Where is the changelog?

It is in the cover letter:

https://lore.kernel.org/all/20230411165919.23955-1-jim2101024@gmail.com/

but it does not look like the cover letter was copied to you or Rob.
-- 
Florian

Re: [PATCH v2 1/3] dt-bindings: PCI: brcmstb: Add two optional props

[Krzysztof Kozlowski]

On 12/04/2023 13:49, Florian Fainelli wrote:
> 
> 
> On 4/12/2023 1:09 AM, Krzysztof Kozlowski wrote:
>> On 11/04/2023 18:59, Jim Quinlan wrote:
>>> Regarding "brcm,enable-l1ss":
>>>
>>>    The Broadcom STB/CM PCIe HW -- a core that is also used by RPi SOCs --
>>>    requires the driver probe() to deliberately place the HW one of three
>>>    CLKREQ# modes:
>>>
>>>    (a) CLKREQ# driven by the RC unconditionally
>>>    (b) CLKREQ# driven by the EP for ASPM L0s, L1
>>>    (c) Bidirectional CLKREQ#, as used for L1 Substates (L1SS).
>>>
>>>    The HW+driver can tell the difference between downstream devices that
>>>    need (a) and (b), but does not know when to configure (c).  Further, the
>>>    HW may cause a CPU abort on boot if guesses wrong regarding the need for
>>>    (c).  So we introduce the boolean "brcm,enable-l1ss" property to indicate
>>>    that (c) is desired.  Setting this property only makes sense when the
>>>    downstream device is L1SS-capable and the OS is configured to activate
>>>    this mode (e.g. policy==superpowersave).
>>>
>>>    This property is already present in the Raspian version of Linux, but the
>>>    upstream driver implementaion that will follow adds more details and
>>
>> typo, implementation
>>
>>>    discerns between (a) and (b).
>>>
>>> Regarding "brcm,completion-timeout-us"
>>>
>>>    Our HW will cause a CPU abort if the L1SS exit time is longer than the
>>>    PCIe transaction completion abort timeout.  We've been asked to make this
>>>    configurable, so we are introducing "brcm,completion-timeout-us".
>>>
>>> Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
>>
>> What happened here? Where is the changelog?
> 
> It is in the cover letter:
> 
> https://lore.kernel.org/all/20230411165919.23955-1-jim2101024@gmail.com/
> 
> but it does not look like the cover letter was copied to you or Rob.

As you said, I did not get it.

Best regards,
Krzysztof

Re: [PATCH v2 1/3] dt-bindings: PCI: brcmstb: Add two optional props

[Jim Quinlan]

On Wed, Apr 12, 2023 at 7:56 AM Krzysztof Kozlowski
<krzysztof.kozlowski@linaro.org> wrote:
>
> On 12/04/2023 13:49, Florian Fainelli wrote:
> >
> >
> > On 4/12/2023 1:09 AM, Krzysztof Kozlowski wrote:
> >> On 11/04/2023 18:59, Jim Quinlan wrote:
> >>> Regarding "brcm,enable-l1ss":
> >>>
> >>>    The Broadcom STB/CM PCIe HW -- a core that is also used by RPi SOCs --
> >>>    requires the driver probe() to deliberately place the HW one of three
> >>>    CLKREQ# modes:
> >>>
> >>>    (a) CLKREQ# driven by the RC unconditionally
> >>>    (b) CLKREQ# driven by the EP for ASPM L0s, L1
> >>>    (c) Bidirectional CLKREQ#, as used for L1 Substates (L1SS).
> >>>
> >>>    The HW+driver can tell the difference between downstream devices that
> >>>    need (a) and (b), but does not know when to configure (c).  Further, the
> >>>    HW may cause a CPU abort on boot if guesses wrong regarding the need for
> >>>    (c).  So we introduce the boolean "brcm,enable-l1ss" property to indicate
> >>>    that (c) is desired.  Setting this property only makes sense when the
> >>>    downstream device is L1SS-capable and the OS is configured to activate
> >>>    this mode (e.g. policy==superpowersave).
> >>>
> >>>    This property is already present in the Raspian version of Linux, but the
> >>>    upstream driver implementaion that will follow adds more details and
> >>
> >> typo, implementation
> >>
> >>>    discerns between (a) and (b).
> >>>
> >>> Regarding "brcm,completion-timeout-us"
> >>>
> >>>    Our HW will cause a CPU abort if the L1SS exit time is longer than the
> >>>    PCIe transaction completion abort timeout.  We've been asked to make this
> >>>    configurable, so we are introducing "brcm,completion-timeout-us".
> >>>
> >>> Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> >>
> >> What happened here? Where is the changelog?
> >
> > It is in the cover letter:
> >
> > https://lore.kernel.org/all/20230411165919.23955-1-jim2101024@gmail.com/
> >
> > but it does not look like the cover letter was copied to you or Rob.
>
> As you said, I did not get it.

Yes, sorry about that; I use a wrapper over the "cocci_cc" script and
I need to modify one or both scripts to send the cover to the
superset of recipients in the constituent commits.

Regards,
Jim Quinan
Broadcom STB
>
> Best regards,
> Krzysztof
>

Re: [PATCH v2 1/3] dt-bindings: PCI: brcmstb: Add two optional props

[Rob Herring]

On Wed, Apr 12, 2023 at 10:14:46AM -0400, Jim Quinlan wrote:
> On Wed, Apr 12, 2023 at 7:56 AM Krzysztof Kozlowski
> <krzysztof.kozlowski@linaro.org> wrote:
> >
> > On 12/04/2023 13:49, Florian Fainelli wrote:
> > >
> > >
> > > On 4/12/2023 1:09 AM, Krzysztof Kozlowski wrote:
> > >> On 11/04/2023 18:59, Jim Quinlan wrote:
> > >>> Regarding "brcm,enable-l1ss":
> > >>>
> > >>>    The Broadcom STB/CM PCIe HW -- a core that is also used by RPi SOCs --
> > >>>    requires the driver probe() to deliberately place the HW one of three
> > >>>    CLKREQ# modes:
> > >>>
> > >>>    (a) CLKREQ# driven by the RC unconditionally
> > >>>    (b) CLKREQ# driven by the EP for ASPM L0s, L1
> > >>>    (c) Bidirectional CLKREQ#, as used for L1 Substates (L1SS).
> > >>>
> > >>>    The HW+driver can tell the difference between downstream devices that
> > >>>    need (a) and (b), but does not know when to configure (c).  Further, the
> > >>>    HW may cause a CPU abort on boot if guesses wrong regarding the need for
> > >>>    (c).  So we introduce the boolean "brcm,enable-l1ss" property to indicate
> > >>>    that (c) is desired.  Setting this property only makes sense when the
> > >>>    downstream device is L1SS-capable and the OS is configured to activate
> > >>>    this mode (e.g. policy==superpowersave).
> > >>>
> > >>>    This property is already present in the Raspian version of Linux, but the
> > >>>    upstream driver implementaion that will follow adds more details and
> > >>
> > >> typo, implementation
> > >>
> > >>>    discerns between (a) and (b).
> > >>>
> > >>> Regarding "brcm,completion-timeout-us"
> > >>>
> > >>>    Our HW will cause a CPU abort if the L1SS exit time is longer than the
> > >>>    PCIe transaction completion abort timeout.  We've been asked to make this
> > >>>    configurable, so we are introducing "brcm,completion-timeout-us".
> > >>>
> > >>> Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> > >>
> > >> What happened here? Where is the changelog?
> > >
> > > It is in the cover letter:
> > >
> > > https://lore.kernel.org/all/20230411165919.23955-1-jim2101024@gmail.com/
> > >
> > > but it does not look like the cover letter was copied to you or Rob.
> >
> > As you said, I did not get it.
> 
> Yes, sorry about that; I use a wrapper over the "cocci_cc" script and
> I need to modify one or both scripts to send the cover to the
> superset of recipients in the constituent commits.

Try out 'b4'. It's much easier.

In any case, I don't read cover letters. Changes to a patch belong with 
the patch.

Rob

Re: [PATCH v2 1/3] dt-bindings: PCI: brcmstb: Add two optional props

[Florian Fainelli]

On 4/12/23 08:37, Rob Herring wrote:
> On Wed, Apr 12, 2023 at 10:14:46AM -0400, Jim Quinlan wrote:
>> On Wed, Apr 12, 2023 at 7:56 AM Krzysztof Kozlowski
>> <krzysztof.kozlowski@linaro.org> wrote:
>>>
>>> On 12/04/2023 13:49, Florian Fainelli wrote:
>>>>
>>>>
>>>> On 4/12/2023 1:09 AM, Krzysztof Kozlowski wrote:
>>>>> On 11/04/2023 18:59, Jim Quinlan wrote:
>>>>>> Regarding "brcm,enable-l1ss":
>>>>>>
>>>>>>     The Broadcom STB/CM PCIe HW -- a core that is also used by RPi SOCs --
>>>>>>     requires the driver probe() to deliberately place the HW one of three
>>>>>>     CLKREQ# modes:
>>>>>>
>>>>>>     (a) CLKREQ# driven by the RC unconditionally
>>>>>>     (b) CLKREQ# driven by the EP for ASPM L0s, L1
>>>>>>     (c) Bidirectional CLKREQ#, as used for L1 Substates (L1SS).
>>>>>>
>>>>>>     The HW+driver can tell the difference between downstream devices that
>>>>>>     need (a) and (b), but does not know when to configure (c).  Further, the
>>>>>>     HW may cause a CPU abort on boot if guesses wrong regarding the need for
>>>>>>     (c).  So we introduce the boolean "brcm,enable-l1ss" property to indicate
>>>>>>     that (c) is desired.  Setting this property only makes sense when the
>>>>>>     downstream device is L1SS-capable and the OS is configured to activate
>>>>>>     this mode (e.g. policy==superpowersave).
>>>>>>
>>>>>>     This property is already present in the Raspian version of Linux, but the
>>>>>>     upstream driver implementaion that will follow adds more details and
>>>>>
>>>>> typo, implementation
>>>>>
>>>>>>     discerns between (a) and (b).
>>>>>>
>>>>>> Regarding "brcm,completion-timeout-us"
>>>>>>
>>>>>>     Our HW will cause a CPU abort if the L1SS exit time is longer than the
>>>>>>     PCIe transaction completion abort timeout.  We've been asked to make this
>>>>>>     configurable, so we are introducing "brcm,completion-timeout-us".
>>>>>>
>>>>>> Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
>>>>>
>>>>> What happened here? Where is the changelog?
>>>>
>>>> It is in the cover letter:
>>>>
>>>> https://lore.kernel.org/all/20230411165919.23955-1-jim2101024@gmail.com/
>>>>
>>>> but it does not look like the cover letter was copied to you or Rob.
>>>
>>> As you said, I did not get it.
>>
>> Yes, sorry about that; I use a wrapper over the "cocci_cc" script and
>> I need to modify one or both scripts to send the cover to the
>> superset of recipients in the constituent commits.
> 
> Try out 'b4'. It's much easier.
> 
> In any case, I don't read cover letters. Changes to a patch belong with
> the patch.

This is not what most other maintainers do, and there does not appear to 
be a general consensus amongst maintainers that the changes belong in 
the individual patches, or in the cover letter. Some trees like the 
networking tree do merge commits of patch sets where the cover letter is 
used as part of the merge commit message. Other maintainers don't, and 
some want the change log after the '---' and some do not.
-- 
Florian

Re: [PATCH v2 2/3] PCI: brcmstb: CLKREQ# accomodations of downstream device

[Cyril Brulebois]

[-- Attachment #1: Type: text/plain, Size: 3517 bytes --]

Hi Jim,

Jim Quinlan <jim2101024@gmail.com> (2023-04-11):
> […]
> This property has already been in use by Raspian Linux, but this
> immplementation adds more details and discerns between (a) and (b)
  ^^^^^^^^^^^^^^^
  implementation

> automatically.
> 
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=217276
> Signed-off-by: Jim Quinlan <jim2101024@gmail.com>

Sorry, it seems like my initial tests with v1 (applied on top of
v6.3-rc5-137-gf2afccfefe7b) weren't thorough enough, and I'm seeing the
same problems with v2 (applied on top of v6.3-rc6-46-gde4664485abb):
 - same setup as in https://bugzilla.kernel.org/show_bug.cgi?id=217276
 - the kernel panic is indeed gone;
 - a USB keyboard connected on that SupaHub PCIe-to-multiple-USB adapter
   isn't seen by the kernel;
 - a USB memory stick connected on the same adapter isn't seen by the
   kernel either;
 - of course both USB devices are confirmed to work fine if they're
   plugged directly on the CM4's USB ports.

Logs with v2:

    root@cm4:~# dmesg|grep -i pci
    [    0.610997] PCI: CLS 0 bytes, default 64
    [    1.664886] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
    [    1.672083] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
    [    1.679125] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
    [    1.688279] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x0603ffffff -> 0x00f8000000
    [    1.696463] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x00ffffffff -> 0x0400000000
    [    1.705282] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
    [    1.711629] pci_bus 0000:00: root bus resource [bus 00-ff]
    [    1.717172] pci_bus 0000:00: root bus resource [mem 0x600000000-0x603ffffff] (bus address [0xf8000000-0xfbffffff])
    [    1.727653] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
    [    1.733768] pci 0000:00:00.0: PME# supported from D0 D3hot
    [    1.740235] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
    [    1.855826] brcm-pcie fd500000.pcie: CLKREQ# ignored; no ASPM
    [    1.863666] brcm-pcie fd500000.pcie: link up, 5.0 GT/s PCIe x1 (SSC)
    [    1.870115] pci 0000:01:00.0: [1912:0014] type 00 class 0x0c0330
    [    1.876205] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00001fff 64bit]
    [    1.883177] pci 0000:01:00.0: PME# supported from D0 D3hot
    [    1.888881] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
    [    1.895581] pci 0000:00:00.0: BAR 14: assigned [mem 0x600000000-0x6000fffff]
    [    1.902707] pci 0000:01:00.0: BAR 0: assigned [mem 0x600000000-0x600001fff 64bit]
    [    1.910279] pci 0000:00:00.0: PCI bridge to [bus 01]
    [    1.915293] pci 0000:00:00.0:   bridge window [mem 0x600000000-0x6000fffff]
    [    1.922412] pcieport 0000:00:00.0: enabling device (0000 -> 0002)
    [    1.928633] pcieport 0000:00:00.0: PME: Signaling with IRQ 23
    [    1.934609] pcieport 0000:00:00.0: AER: enabled with IRQ 23
    [    1.940340] pci 0000:01:00.0: enabling device (0000 -> 0002)
    [    6.946090] pci 0000:01:00.0: xHCI HW not ready after 5 sec (HC bug?) status = 0x1801
    [    6.954026] pci 0000:01:00.0: quirk_usb_early_handoff+0x0/0x968 took 4896180 usecs

Please let me know what I can do to help.


Cheers,
-- 
Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 2/3] PCI: brcmstb: CLKREQ# accomodations of downstream device

[Jim Quinlan]

Hello Cyril,
Can you provide (a) the full boot log prior to applying the patch
series and (b) full boot log after applying the series, using an
IDENTICAL  setup.
If it fails on both then it has little to do with my patch series.

In my last series your testing somehow conflated the effect of an
unrelated MMC interrupt issue so please be precise.

Regards,
Jim  Quinlan
Broadcom STB

On Thu, Apr 13, 2023 at 10:39 AM Cyril Brulebois <kibi@debian.org> wrote:
>
> Hi Jim,
>
> Jim Quinlan <jim2101024@gmail.com> (2023-04-11):
> > […]
> > This property has already been in use by Raspian Linux, but this
> > immplementation adds more details and discerns between (a) and (b)
>   ^^^^^^^^^^^^^^^
>   implementation
>
> > automatically.
> >
> > Link: https://bugzilla.kernel.org/show_bug.cgi?id=217276
> > Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
>
> Sorry, it seems like my initial tests with v1 (applied on top of
> v6.3-rc5-137-gf2afccfefe7b) weren't thorough enough, and I'm seeing the
> same problems with v2 (applied on top of v6.3-rc6-46-gde4664485abb):
>  - same setup as in https://bugzilla.kernel.org/show_bug.cgi?id=217276
>  - the kernel panic is indeed gone;
>  - a USB keyboard connected on that SupaHub PCIe-to-multiple-USB adapter
>    isn't seen by the kernel;
>  - a USB memory stick connected on the same adapter isn't seen by the
>    kernel either;
>  - of course both USB devices are confirmed to work fine if they're
>    plugged directly on the CM4's USB ports.
>
> Logs with v2:
>
>     root@cm4:~# dmesg|grep -i pci
>     [    0.610997] PCI: CLS 0 bytes, default 64
>     [    1.664886] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
>     [    1.672083] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
>     [    1.679125] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
>     [    1.688279] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x0603ffffff -> 0x00f8000000
>     [    1.696463] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x00ffffffff -> 0x0400000000
>     [    1.705282] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
>     [    1.711629] pci_bus 0000:00: root bus resource [bus 00-ff]
>     [    1.717172] pci_bus 0000:00: root bus resource [mem 0x600000000-0x603ffffff] (bus address [0xf8000000-0xfbffffff])
>     [    1.727653] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
>     [    1.733768] pci 0000:00:00.0: PME# supported from D0 D3hot
>     [    1.740235] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
>     [    1.855826] brcm-pcie fd500000.pcie: CLKREQ# ignored; no ASPM
>     [    1.863666] brcm-pcie fd500000.pcie: link up, 5.0 GT/s PCIe x1 (SSC)
>     [    1.870115] pci 0000:01:00.0: [1912:0014] type 00 class 0x0c0330
>     [    1.876205] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00001fff 64bit]
>     [    1.883177] pci 0000:01:00.0: PME# supported from D0 D3hot
>     [    1.888881] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
>     [    1.895581] pci 0000:00:00.0: BAR 14: assigned [mem 0x600000000-0x6000fffff]
>     [    1.902707] pci 0000:01:00.0: BAR 0: assigned [mem 0x600000000-0x600001fff 64bit]
>     [    1.910279] pci 0000:00:00.0: PCI bridge to [bus 01]
>     [    1.915293] pci 0000:00:00.0:   bridge window [mem 0x600000000-0x6000fffff]
>     [    1.922412] pcieport 0000:00:00.0: enabling device (0000 -> 0002)
>     [    1.928633] pcieport 0000:00:00.0: PME: Signaling with IRQ 23
>     [    1.934609] pcieport 0000:00:00.0: AER: enabled with IRQ 23
>     [    1.940340] pci 0000:01:00.0: enabling device (0000 -> 0002)
>     [    6.946090] pci 0000:01:00.0: xHCI HW not ready after 5 sec (HC bug?) status = 0x1801
>     [    6.954026] pci 0000:01:00.0: quirk_usb_early_handoff+0x0/0x968 took 4896180 usecs
>
> Please let me know what I can do to help.
>
>
> Cheers,
> --
> Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
> D-I release manager -- Release team member -- Freelance Consultant

Re: [PATCH v2 2/3] PCI: brcmstb: CLKREQ# accomodations of downstream device

[Florian Fainelli]

On 4/13/2023 7:39 AM, Cyril Brulebois wrote:
> Hi Jim,
> 
> Jim Quinlan <jim2101024@gmail.com> (2023-04-11):
>> […]
>> This property has already been in use by Raspian Linux, but this
>> immplementation adds more details and discerns between (a) and (b)
>    ^^^^^^^^^^^^^^^
>    implementation
> 
>> automatically.
>>
>> Link: https://bugzilla.kernel.org/show_bug.cgi?id=217276
>> Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> 
> Sorry, it seems like my initial tests with v1 (applied on top of
> v6.3-rc5-137-gf2afccfefe7b) weren't thorough enough, and I'm seeing the
> same problems with v2 (applied on top of v6.3-rc6-46-gde4664485abb):
>   - same setup as in https://bugzilla.kernel.org/show_bug.cgi?id=217276
>   - the kernel panic is indeed gone;
>   - a USB keyboard connected on that SupaHub PCIe-to-multiple-USB adapter
>     isn't seen by the kernel;
>   - a USB memory stick connected on the same adapter isn't seen by the
>     kernel either;
>   - of course both USB devices are confirmed to work fine if they're
>     plugged directly on the CM4's USB ports.
> 
> Logs with v2:
> 
>      root@cm4:~# dmesg|grep -i pci
>      [    0.610997] PCI: CLS 0 bytes, default 64
>      [    1.664886] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
>      [    1.672083] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
>      [    1.679125] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
>      [    1.688279] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x0603ffffff -> 0x00f8000000
>      [    1.696463] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x00ffffffff -> 0x0400000000
>      [    1.705282] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
>      [    1.711629] pci_bus 0000:00: root bus resource [bus 00-ff]
>      [    1.717172] pci_bus 0000:00: root bus resource [mem 0x600000000-0x603ffffff] (bus address [0xf8000000-0xfbffffff])
>      [    1.727653] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
>      [    1.733768] pci 0000:00:00.0: PME# supported from D0 D3hot
>      [    1.740235] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
>      [    1.855826] brcm-pcie fd500000.pcie: CLKREQ# ignored; no ASPM
>      [    1.863666] brcm-pcie fd500000.pcie: link up, 5.0 GT/s PCIe x1 (SSC)
>      [    1.870115] pci 0000:01:00.0: [1912:0014] type 00 class 0x0c0330
>      [    1.876205] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00001fff 64bit]
>      [    1.883177] pci 0000:01:00.0: PME# supported from D0 D3hot
>      [    1.888881] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
>      [    1.895581] pci 0000:00:00.0: BAR 14: assigned [mem 0x600000000-0x6000fffff]
>      [    1.902707] pci 0000:01:00.0: BAR 0: assigned [mem 0x600000000-0x600001fff 64bit]
>      [    1.910279] pci 0000:00:00.0: PCI bridge to [bus 01]
>      [    1.915293] pci 0000:00:00.0:   bridge window [mem 0x600000000-0x6000fffff]
>      [    1.922412] pcieport 0000:00:00.0: enabling device (0000 -> 0002)
>      [    1.928633] pcieport 0000:00:00.0: PME: Signaling with IRQ 23
>      [    1.934609] pcieport 0000:00:00.0: AER: enabled with IRQ 23
>      [    1.940340] pci 0000:01:00.0: enabling device (0000 -> 0002)
>      [    6.946090] pci 0000:01:00.0: xHCI HW not ready after 5 sec (HC bug?) status = 0x1801
>      [    6.954026] pci 0000:01:00.0: quirk_usb_early_handoff+0x0/0x968 took 4896180 usecs
> 
> Please let me know what I can do to help.

Could you please attach your .config so we can check a few things?
-- 
Florian

Re: [PATCH v2 0/3] PCI: brcmstb: CLKREQ# accomodations of downstream device

[Florian Fainelli]

On 4/11/23 09:59, Jim Quinlan wrote:
> v2 -- Changed binding property 'brcm,completion-timeout-msec' to
>        'brcm,completion-timeout-us'.  (StefanW for standard suffix).
>     -- Warn when clamping timeout value, and include clamped
>        region in message. Also add min and max in YAML. (StefanW)
>     -- Qualify description of "brcm,completion-timeout-us" so that
>        it refers to PCIe transactions. (StefanW)
>     -- Remvove mention of Linux specifics in binding description. (StefanW)
>     -- s/clkreq#/CLKREQ#/g (Bjorn)
>     -- Refactor completion-timeout-us code to compare max and min to
>        value given by the property (as opposed to the computed value).
> 
> v1 -- The current driver assumes the downstream devices can
>        provide CLKREQ# for ASPM.  These commits accomodate devices
>        w/ or w/o clkreq# and also handle L1SS-capable devices.
> 
>     -- The Raspian Linux folks have already been using a PCIe RC
>        property "brcm,enable-l1ss".  These commits use the same
>        property, in a backward-compatible manner, and the implementaion
>        adds more detail and also automatically identifies devices w/o
>        a clkreq# signal, i.e. most devices plugged into an RPi CM4
>        IO board.
> 
> Jim Quinlan (3):
>    PCI: brcmstb: CLKREQ# accomodations of downstream device
>    PCI: brcmstb: Set PCIe transaction completion timeout
>    blah blah

Tested-by: Florian Fainelli <f.fainelli@gmail.com>

On a 7216 system test with:

01:00.0 Network controller: Intel Corporation Wireless 7260 (rev 73)

and on the CM4 I/O board with:

01:00.0 Network controller: Intel Corporation Wireless 7260 (rev 73)

01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network 
Connection

01:00.0 Network controller: Broadcom Inc. and subsidiaries BCM43224 
802.11a/b/g/n (rev 01)

01:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9125 PCIe 
SATA 6.0 Gb/s controller (rev 11) (prog-if 01 [AHCI 1.0])

01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme 
BCM5751 Gigabit Ethernet PCI Express (rev 21)

01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI 
Express-to-PCI Bridge (rev aa)
02:00.0 Multiport serial controller: Pepperl+Fuchs RocketPort EXPRESS 
8-port w/Octa Cable

01:00.0 Ethernet controller: Qualcomm Atheros AR5008 Wireless Network 
Adapter (rev 01)

01:00.0 Network controller: Broadcom Inc. and subsidiaries BCM4311 
802.11a/b/g (rev 01)

01:00.0 Network controller: Broadcom Inc. and subsidiaries BCM4322 
802.11a/b/g/n Wireless LAN Controller (rev 01)

01:00.0 Network controller: Broadcom Inc. and subsidiaries BCM43602 
802.11ac Wireless LAN SoC (rev 01)

and finally with a 4 port switch:

-[0000:00]---00.0-[01-07]----00.0-[02-07]--+-01.0-[03]----00.0  Intel 
Corporation 82574L Gigabit Network Connection
 
+-03.0-[04-05]----00.0-[05]----00.0  Pepperl+Fuchs RocketPort EXPRESS 
8-port w/Octa Cable
                                            +-05.0-[06]----00.0 
Broadcom Inc. and subsidiaries NetXtreme BCM5751 Gigabit Ethernet PCI 
Express
                                            \-07.0-[07]----00.0  Intel 
Corporation 82574L Gigabit Network Connection

And than I ran out of devices that I could plug, the others were x4, x8 
or x16.

Most (all?) would previously fail, so definitively an improvement!

Thanks!
-- 
-- 
Florian

Re: [PATCH v2 2/3] PCI: brcmstb: CLKREQ# accomodations of downstream device

[Jim Quinlan]

On Thu, Apr 13, 2023 at 4:06 PM Cyril Brulebois <kibi@debian.org> wrote:
>
> Hi Jim,
>
> Jim Quinlan <jim2101024@gmail.com> (2023-04-13):
> > Can you provide (a) the full boot log prior to applying the patch
> > series and (b) full boot log after applying the series, using an
> > IDENTICAL setup. If it fails on both then it has little to do with my
> > patch series.
>
> Just to be clear, the issue I reported was with:
>  - Raspberry Pi Compute Module 4 (Rev 1.1, 4G RAM, 32G storage)
>  - Raspberry Pi Compute Module 4 IO Board
>  - SupaHub PCIe-to-multiple-USB adapter, reference PCE6U1C-R02, VER 006S
>
> This was my minimal reproducer for the kernel panic at boot-up, which
> goes away with either v1 or v2. When I realized I didn't actually check
> whether the SupaHub board was working correctly, I plugged 2 devices to
> obtain this setup:
>  - Raspberry Pi Compute Module 4 (Rev 1.1, 4G RAM, 32G storage)
>  - Raspberry Pi Compute Module 4 IO Board
>  - SupaHub PCIe-to-multiple-USB adapter, reference PCE6U1C-R02, VER 006S
>  - Kingston DataTraveler G4 32GB on USB-A port #1 of the SupaHub board.
>  - Logitech K120 keyboard on USB-A port #2 of the SupaHub board.
>
> It turns out that this particular revision of the SupaHub board isn't
> supported by xhci_hcd directly (failing to probe with error -110) and
> one needs to enable CONFIG_USB_XHCI_PCI_RENESAS=m and also ship its
> accompanying firmware (/lib/firmware/renesas_usb_fw.mem). With this
> updated kernel config, I'm able to use the keyboard and to read data
> from the memory stick without problems (70 MB/s).
>
> > In my last series your testing somehow conflated the effect of an
> > unrelated MMC interrupt issue so please be precise.
>
> I wish things would be simpler and didn't involve combinatorics, let
> alone other bugs/regressions at times, but I'm really trying my best to
> navigate and report issues and test patches when I can spare some time…

Hi Cyril,

I want to encourage you and others doing testing and bug reporting:
everyone wins when a bug or issue is reported, fixed, and tested.
I'm just asking that when you have negative results, that you provide
information on the "before" and "after" test results of
the patch series, and run both on the same test environment.

Regards,
Jim Quinlan
Broadcom STB

>
>
> In my defence, the very similar board PCE6U1C-R02, VER 006 boots without
> a kernel panic, and works fine with xhci_hcd without that extra module
> and its firmware. It's based on the exact same chip (Renesas Technology
> Corp. uPD720201) though, that's why I didn't realize the need for an
> extra module until now for the PCE6U1C-R02, VER 006S one. (Florian,
> thanks for mentioning .config earlier…)
>
> To sum up, it seems both (sub)versions of that SupaHub PCE6U1C-R02 board
> are usable with this patch series: the kernel panic at boot-up is gone,
> and USB devices plugged into those boards are working as expected.
>
>
> But, speaking of combinatorics, while the patch series helps fix
> https://bugzilla.kernel.org/show_bug.cgi?id=217276 on that particular
> initial combination of CM4 and SupaHub PCE6U1C-R02, VER 006S, it also
> generates a regression if I use a different CM4…
>
> Setup:
>  - Raspberry Pi Compute Module 4 (Rev 1.0, 8G RAM, 32G storage)
>  - Raspberry Pi Compute Module 4 IO Board
>  - SupaHub PCIe-to-multiple-USB adapter, reference PCE6U1C-R02, VER 006S
>
> You'll find attached (a) and (b) as requested above, for this setup, as
> well as the kernel configuration file:
>  - (a) = dmesg-unpatched.txt = boots fine (and USB devices work fine if
>    memory sticks or keyboards are plugged in).
>  - (b) = dmesg-patched.txt = kernel panic.
>  - cm4+pcie.config
>
> I'm getting similar results with the other SupaHub board:
>  - Raspberry Pi Compute Module 4 (Rev 1.0, 8G RAM, 32G storage)
>  - Raspberry Pi Compute Module 4 IO Board
>  - SupaHub PCIe-to-multiple-USB adapter, reference PCE6U1C-R02, VER 006
>
>
> This is probably best summarized this way:
>
>                               | unpatched    | patched
> ------------------------------+--------------+--------------
> CM4 Rev 1.1 4G/32G + VER 006  | OK           | OK
> CM4 Rev 1.1 4G/32G + VER 006S | Kernel panic | OK            ← Bugzilla entry
> CM4 Rev 1.0 8G/32G + VER 006  | OK           | Kernel panic
> CM4 Rev 1.0 8G/32G + VER 006S | OK           | Kernel panic
>
>
> (And I'm of course using the same settings regarding the serial console,
> to get traces when needed.)
>
>
> Cheers,
> --
> Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
> D-I release manager -- Release team member -- Freelance Consultant

Re: [PATCH v2 2/3] PCI: brcmstb: CLKREQ# accomodations of downstream device

[Florian Fainelli]

On 4/14/2023 5:14 AM, Jim Quinlan wrote:
> On Thu, Apr 13, 2023 at 4:06 PM Cyril Brulebois <kibi@debian.org> wrote:
>>
>> Hi Jim,
>>
>> Jim Quinlan <jim2101024@gmail.com> (2023-04-13):
>>> Can you provide (a) the full boot log prior to applying the patch
>>> series and (b) full boot log after applying the series, using an
>>> IDENTICAL setup. If it fails on both then it has little to do with my
>>> patch series.
>>
>> Just to be clear, the issue I reported was with:
>>   - Raspberry Pi Compute Module 4 (Rev 1.1, 4G RAM, 32G storage)
>>   - Raspberry Pi Compute Module 4 IO Board
>>   - SupaHub PCIe-to-multiple-USB adapter, reference PCE6U1C-R02, VER 006S
>>
>> This was my minimal reproducer for the kernel panic at boot-up, which
>> goes away with either v1 or v2. When I realized I didn't actually check
>> whether the SupaHub board was working correctly, I plugged 2 devices to
>> obtain this setup:
>>   - Raspberry Pi Compute Module 4 (Rev 1.1, 4G RAM, 32G storage)
>>   - Raspberry Pi Compute Module 4 IO Board
>>   - SupaHub PCIe-to-multiple-USB adapter, reference PCE6U1C-R02, VER 006S
>>   - Kingston DataTraveler G4 32GB on USB-A port #1 of the SupaHub board.
>>   - Logitech K120 keyboard on USB-A port #2 of the SupaHub board.
>>
>> It turns out that this particular revision of the SupaHub board isn't
>> supported by xhci_hcd directly (failing to probe with error -110) and
>> one needs to enable CONFIG_USB_XHCI_PCI_RENESAS=m and also ship its
>> accompanying firmware (/lib/firmware/renesas_usb_fw.mem). With this
>> updated kernel config, I'm able to use the keyboard and to read data
>> from the memory stick without problems (70 MB/s).
>>
>>> In my last series your testing somehow conflated the effect of an
>>> unrelated MMC interrupt issue so please be precise.
>>
>> I wish things would be simpler and didn't involve combinatorics, let
>> alone other bugs/regressions at times, but I'm really trying my best to
>> navigate and report issues and test patches when I can spare some time…
> 
> Hi Cyril,
> 
> I want to encourage you and others doing testing and bug reporting:
> everyone wins when a bug or issue is reported, fixed, and tested.
> I'm just asking that when you have negative results, that you provide
> information on the "before" and "after" test results of
> the patch series, and run both on the same test environment.

Cyril, based upon the table and logs you provided whereby you have used 
the following:

- Raspberry Pi Compute Module 4 (Rev 1.0, 8G RAM, 32G storage)
- Raspberry Pi Compute Module 4 IO Board
- SupaHub PCIe-to-multiple-USB adapter, reference PCE6U1C-R02, VER 006S

in the before/unpatched case we have a PCIe link down and in the 
after/patched we have a PCIe link up but a kernel panic. Neither are 
great nor resulting in a fully functional PCIe device.

Looking at:

https://www.amazon.co.uk/SupaHub-Express-BandWidth-Capable-Expanding/dp/B092ZQWG5B

it would appear that it can accept an external power supply, do you have 
one connected to that USB expansion card by any chance? Are you able to 
boot the kernel before/after if you disconnect any USB peripheral?

This looks like a broader electrical problem than the scope of this 
patch, though it would be neat if we could find a combination that 
works. At least with Jim's patch we have a PCIe link with 
uni-directional CLKREQ# so we could try a variety of things.

Does that SupaHub board plugged to the CM4 1.0 system work fine in the 
Raspberry Pi kernel tree?
-- 
Florian

Re: [PATCH v2 2/3] PCI: brcmstb: CLKREQ# accomodations of downstream device

[Jim Quinlan]

On Fri, Apr 14, 2023 at 8:27 AM Florian Fainelli <f.fainelli@gmail.com> wrote:
>
>
>
> On 4/14/2023 5:14 AM, Jim Quinlan wrote:
> > On Thu, Apr 13, 2023 at 4:06 PM Cyril Brulebois <kibi@debian.org> wrote:
> >>
> >> Hi Jim,
> >>
> >> Jim Quinlan <jim2101024@gmail.com> (2023-04-13):
> >>> Can you provide (a) the full boot log prior to applying the patch
> >>> series and (b) full boot log after applying the series, using an
> >>> IDENTICAL setup. If it fails on both then it has little to do with my
> >>> patch series.
> >>
> >> Just to be clear, the issue I reported was with:
> >>   - Raspberry Pi Compute Module 4 (Rev 1.1, 4G RAM, 32G storage)
> >>   - Raspberry Pi Compute Module 4 IO Board
> >>   - SupaHub PCIe-to-multiple-USB adapter, reference PCE6U1C-R02, VER 006S
> >>
> >> This was my minimal reproducer for the kernel panic at boot-up, which
> >> goes away with either v1 or v2. When I realized I didn't actually check
> >> whether the SupaHub board was working correctly, I plugged 2 devices to
> >> obtain this setup:
> >>   - Raspberry Pi Compute Module 4 (Rev 1.1, 4G RAM, 32G storage)
> >>   - Raspberry Pi Compute Module 4 IO Board
> >>   - SupaHub PCIe-to-multiple-USB adapter, reference PCE6U1C-R02, VER 006S
> >>   - Kingston DataTraveler G4 32GB on USB-A port #1 of the SupaHub board.
> >>   - Logitech K120 keyboard on USB-A port #2 of the SupaHub board.
> >>
> >> It turns out that this particular revision of the SupaHub board isn't
> >> supported by xhci_hcd directly (failing to probe with error -110) and
> >> one needs to enable CONFIG_USB_XHCI_PCI_RENESAS=m and also ship its
> >> accompanying firmware (/lib/firmware/renesas_usb_fw.mem). With this
> >> updated kernel config, I'm able to use the keyboard and to read data
> >> from the memory stick without problems (70 MB/s).
> >>
> >>> In my last series your testing somehow conflated the effect of an
> >>> unrelated MMC interrupt issue so please be precise.
> >>
> >> I wish things would be simpler and didn't involve combinatorics, let
> >> alone other bugs/regressions at times, but I'm really trying my best to
> >> navigate and report issues and test patches when I can spare some time…
> >
> > Hi Cyril,
> >
> > I want to encourage you and others doing testing and bug reporting:
> > everyone wins when a bug or issue is reported, fixed, and tested.
> > I'm just asking that when you have negative results, that you provide
> > information on the "before" and "after" test results of
> > the patch series, and run both on the same test environment.
>
> Cyril, based upon the table and logs you provided whereby you have used
> the following:
>
> - Raspberry Pi Compute Module 4 (Rev 1.0, 8G RAM, 32G storage)
> - Raspberry Pi Compute Module 4 IO Board
> - SupaHub PCIe-to-multiple-USB adapter, reference PCE6U1C-R02, VER 006S
>
> in the before/unpatched case we have a PCIe link down and in the
> after/patched we have a PCIe link up but a kernel panic. Neither are
> great nor resulting in a fully functional PCIe device.

The data do not make sense to me. My new code is executed AFTER pcie
link up/down determination.  Both test results should have the same
link status -- either "link up" or "link down".

If the system is wishy-washy, i.e. it has link-up in 5/10 boots, then
we need to repeat  the experiment to compare the "link up" cases only.
Or discount the test completely

If the system is not wishy-washy, then something has been changed
between the "before" and "after"  tests.


>
> Looking at:
>
> https://www.amazon.co.uk/SupaHub-Express-BandWidth-Capable-Expanding/dp/B092ZQWG5B
>
> it would appear that it can accept an external power supply, do you have
> one connected to that USB expansion card by any chance? Are you able to
> boot the kernel before/after if you disconnect any USB peripheral?
>
> This looks like a broader electrical problem than the scope of this
> patch, though it would be neat if we could find a combination that
> works. At least with Jim's patch we have a PCIe link with
> uni-directional CLKREQ# so we could try a variety of things.
>
> Does that SupaHub board plugged to the CM4 1.0 system work fine in the
> Raspberry Pi kernel tree?
> --
> Florian

Re: [PATCH v2 2/3] PCI: brcmstb: CLKREQ# accomodations of downstream device

[Cyril Brulebois]

[-- Attachment #1.1: Type: text/plain, Size: 3007 bytes --]

Florian Fainelli <f.fainelli@gmail.com> (2023-04-14):
> Cyril, based upon the table and logs you provided whereby you have used the
> following:
> 
> - Raspberry Pi Compute Module 4 (Rev 1.0, 8G RAM, 32G storage)
> - Raspberry Pi Compute Module 4 IO Board
> - SupaHub PCIe-to-multiple-USB adapter, reference PCE6U1C-R02, VER 006S
> 
> in the before/unpatched case we have a PCIe link down and in the
> after/patched we have a PCIe link up but a kernel panic. Neither are great
> nor resulting in a fully functional PCIe device.

Based on Jim's feedback, I'm attaching a new dmesg for the aforementioned
setup, with an unpatched kernel (dmesg-unpatched-pcie-link-up.txt).

I fear that initially I might have not waited enough before power off and
power on when replugging the SupaHub adapter, and I've only seen the PCIe
link down a few times (now that I'm actively paying attention to that
part). I'm indeed seeing PCIe link up consistently (100%) when using the
following method:

    for i in $(seq 1 20); do
      # power on, let the system boot up and settle:
      sispmctl -t 4; sleep 2m
      ssh cm4 sh -c "dmesg > dmesg-$i.txt; poweroff"
      # let the system power down, and power off:
      sleep 30; sispmctl -t 4
      # wait before power cycling:
      sleep 10
    done

> Looking at:
> https://www.amazon.co.uk/SupaHub-Express-BandWidth-Capable-Expanding/dp/B092ZQWG5B
> 
> it would appear that it can accept an external power supply, do you have one
> connected to that USB expansion card by any chance?

With the patched kernel, I have tried several things (leaving the regular
12V adapter plugged in all the time):
 - No external power supply (that's what I've been using in all previous
   attempts).
 - Using a 5V PSU with 2 pins (ground and 5V) plugged on the Ext PSU
   4-pin header on the IO Board.
 - Using a 5V PSU with a SATA connector plugged directly onto the SupaHub
   adapter.

I'm getting the same trace in all cases.

> Are you able to boot the kernel before/after if you disconnect any USB
> peripheral?

The trace is obtained without any USB peripheral (on the extension card or
on the onboard USB slots). Besides the SupaHub adapter on the PCIe slot, I
only have Ethernet and HDMI plugged in (I'm getting traces via serial
console, and operating the system over SSH when it boots fine).

As soon as I unplug the SupaHub board itself, the system boots fine.

> Does that SupaHub board plugged to the CM4 1.0 system work fine in the
> Raspberry Pi kernel tree?

With the Raspberry Pi OS (64-bit) > Raspberry Pi OS Lite image
(2023-02-21-raspios-bullseye-arm64-lite.img.xz), the system at least
boots fine; I haven't tried adding devices to the mix just yet, trying
to stick with that particular setup.

A full dmesg is attached (dmesg-raspios.txt).


Cheers,
-- 
Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[-- Attachment #1.2: dmesg-unpatched-pcie-link-up.txt --]
[-- Type: text/plain, Size: 32861 bytes --]

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 6.3.0-rc6+ (kibi@genova) (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Thu Apr 13 20:03:01 CEST 2023
[    0.000000] random: crng init done
[    0.000000] Machine model: Raspberry Pi Compute Module 4 Rev 1.0
[    0.000000] efi: UEFI not found.
[    0.000000] Reserved memory: created CMA memory pool at 0x0000000037400000, size 64 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x0000000037400000..0x000000003b3fffff (65536 KiB) map reusable linux,cma
[    0.000000] OF: reserved mem: 0x000000003ef625a0..0x000000003ef626d0 (0 KiB) nomap non-reusable nvram@0
[    0.000000] earlycon: bcm2835aux0 at MMIO32 0x00000000fe215040 (options '115200n8')
[    0.000000] printk: bootconsole [bcm2835aux0] enabled
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x00000001ffffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x1ff00c280-0x1ff00efff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000003fffffff]
[    0.000000]   DMA32    [mem 0x0000000040000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000001ffffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000003b3fffff]
[    0.000000]   node   0: [mem 0x0000000040000000-0x00000000fbffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x00000001ffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000001ffffffff]
[    0.000000] On node 0, zone DMA32: 19456 pages in unavailable ranges
[    0.000000] On node 0, zone Normal: 16384 pages in unavailable ranges
[    0.000000] percpu: Embedded 31 pages/cpu s88232 r8192 d30552 u126976
[    0.000000] pcpu-alloc: s88232 r8192 d30552 u126976 alloc=31*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: Spectre-v2
[    0.000000] CPU features: detected: Spectre-v3a
[    0.000000] CPU features: detected: Spectre-v4
[    0.000000] CPU features: detected: Spectre-BHB
[    0.000000] CPU features: kernel page table isolation forced ON by KASLR
[    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[    0.000000] CPU features: detected: ARM erratum 1742098
[    0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
[    0.000000] alternatives: applying boot alternatives
[    0.000000] Fallback order for Node 0: 0 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 2028848
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: video=HDMI-A-1:1920x1080M@60,margin_left=48,margin_right=48,margin_top=48,margin_bottom=48 dma.dmachans=0x37f5 bcm2709.boardrev=0xd03140 bcm2709.serial=0x751d0632 bcm2709.uart_clock=48000000 smsc95xx.macaddr=E4:5F:01:4D:5F:D7 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000  console=ttyS1,115200 root=LABEL=RASPIROOT rw fsck.repair=yes net.ifnames=0  rootwait earlycon
[    0.000000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, linear)
[    0.000000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:on, heap free:off
[    0.000000] software IO TLB: area num 4.
[    0.000000] software IO TLB: mapped [mem 0x0000000033400000-0x0000000037400000] (64MB)
[    0.000000] Memory: 3987300K/8245248K available (13440K kernel code, 2774K rwdata, 8248K rodata, 6528K init, 607K bss, 290288K reserved, 65536K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] ftrace: allocating 43928 entries in 172 pages
[    0.000000] ftrace: allocated 172 pages with 4 groups
[    0.000000] trace event string verifier disabled
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4.
[    0.000000] 	Rude variant of Tasks RCU enabled.
[    0.000000] 	Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GIC: Using split EOI/Deactivate mode
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] arch_timer: cp15 timer(s) running at 54.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xc743ce346, max_idle_ns: 440795203123 ns
[    0.000000] sched_clock: 56 bits at 54MHz, resolution 18ns, wraps every 4398046511102ns
[    0.008293] Console: colour dummy device 80x25
[    0.012831] Calibrating delay loop (skipped), value calculated using timer frequency.. 108.00 BogoMIPS (lpj=216000)
[    0.023383] pid_max: default: 32768 minimum: 301
[    0.028104] LSM: initializing lsm=lockdown,capability,landlock,yama,integrity,apparmor,tomoyo,bpf
[    0.037094] landlock: Up and running.
[    0.040785] Yama: becoming mindful.
[    0.044391] AppArmor: AppArmor initialized
[    0.048529] TOMOYO Linux initialized
[    0.052150] LSM support for eBPF active
[    0.056115] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.063811] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.073125] cblist_init_generic: Setting adjustable number of callback queues.
[    0.080449] cblist_init_generic: Setting shift to 2 and lim to 1.
[    0.086656] cblist_init_generic: Setting shift to 2 and lim to 1.
[    0.092943] rcu: Hierarchical SRCU implementation.
[    0.097782] rcu: 	Max phase no-delay instances is 1000.
[    0.104118] EFI services will not be available.
[    0.109030] smp: Bringing up secondary CPUs ...
[    0.113978] Detected PIPT I-cache on CPU1
[    0.114047] CPU1: Booted secondary processor 0x0000000001 [0x410fd083]
[    0.114421] Detected PIPT I-cache on CPU2
[    0.114457] CPU2: Booted secondary processor 0x0000000002 [0x410fd083]
[    0.114822] Detected PIPT I-cache on CPU3
[    0.114857] CPU3: Booted secondary processor 0x0000000003 [0x410fd083]
[    0.114898] smp: Brought up 1 node, 4 CPUs
[    0.150939] SMP: Total of 4 processors activated.
[    0.155688] CPU features: detected: 32-bit EL0 Support
[    0.160873] CPU features: detected: 32-bit EL1 Support
[    0.166062] CPU features: detected: CRC32 instructions
[    0.171305] CPU: All CPU(s) started at EL2
[    0.175448] alternatives: applying system-wide alternatives
[    0.258167] node 0 deferred pages initialised in 76ms
[    0.264630] devtmpfs: initialized
[    0.273248] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.283148] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    0.291457] pinctrl core: initialized pinctrl subsystem
[    0.297614] DMI not present or invalid.
[    0.301833] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.309143] DMA: preallocated 1024 KiB GFP_KERNEL pool for atomic allocations
[    0.316833] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.325692] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.333817] audit: initializing netlink subsys (disabled)
[    0.339412] audit: type=2000 audit(0.244:1): state=initialized audit_enabled=0 res=1
[    0.339957] thermal_sys: Registered thermal governor 'fair_share'
[    0.347244] thermal_sys: Registered thermal governor 'bang_bang'
[    0.353398] thermal_sys: Registered thermal governor 'step_wise'
[    0.359463] thermal_sys: Registered thermal governor 'user_space'
[    0.365528] thermal_sys: Registered thermal governor 'power_allocator'
[    0.371707] cpuidle: using governor ladder
[    0.382439] cpuidle: using governor menu
[    0.386517] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.393433] ASID allocator initialised with 32768 entries
[    0.399130] Serial: AMBA PL011 UART driver
[    0.410262] KASLR enabled
[    0.413665] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
[    0.420530] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page
[    0.426862] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages
[    0.433717] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page
[    0.440046] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
[    0.446900] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page
[    0.453229] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages
[    0.460083] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page
[    0.467546] ACPI: Interpreter disabled.
[    0.471664] iommu: Default domain type: Translated 
[    0.476593] iommu: DMA domain TLB invalidation policy: strict mode 
[    0.483082] EDAC MC: Ver: 3.0.0
[    0.486519] scmi_core: SCMI protocol bus registered
[    0.492029] NetLabel: Initializing
[    0.495466] NetLabel:  domain hash size = 128
[    0.499863] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    0.505611] NetLabel:  unlabeled traffic allowed by default
[    0.511374] vgaarb: loaded
[    0.514332] clocksource: Switched to clocksource arch_sys_counter
[    0.549785] VFS: Disk quotas dquot_6.6.0
[    0.553806] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.561228] AppArmor: AppArmor Filesystem Enabled
[    0.566034] pnp: PnP ACPI: disabled
[    0.573512] NET: Registered PF_INET protocol family
[    0.578927] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.590470] tcp_listen_portaddr_hash hash table entries: 4096 (order: 4, 65536 bytes, linear)
[    0.599270] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.607308] TCP established hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.616322] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, linear)
[    0.626439] TCP: Hash tables configured (established 65536 bind 65536)
[    0.633407] MPTCP token hash table entries: 8192 (order: 5, 196608 bytes, linear)
[    0.641344] UDP hash table entries: 4096 (order: 5, 131072 bytes, linear)
[    0.648308] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes, linear)
[    0.655786] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.661535] NET: Registered PF_XDP protocol family
[    0.666384] PCI: CLS 0 bytes, default 64
[    0.670609] Trying to unpack rootfs image as initramfs...
[    0.676170] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
[    0.684669] kvm [1]: IPA Size Limit: 44 bits
[    0.690004] kvm [1]: vgic interrupt IRQ9
[    0.694037] kvm [1]: Hyp mode initialized successfully
[    0.700409] Initialise system trusted keyrings
[    0.704985] Key type blacklist registered
[    0.709327] workingset: timestamp_bits=42 max_order=21 bucket_order=0
[    0.716008] zbud: loaded
[    0.719171] integrity: Platform Keyring initialized
[    0.724197] integrity: Machine keyring initialized
[    0.729077] Key type asymmetric registered
[    0.733224] Asymmetric key parser 'x509' registered
[    1.675267] Freeing initrd memory: 31880K
[    1.692974] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)
[    1.700551] io scheduler mq-deadline registered
[    1.707175] irq_brcmstb_l2: registered L2 intc (/soc/interrupt-controller@7ef00100, parent irq: 18)
[    1.720146] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    1.727354] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    1.734395] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    1.743554] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x0603ffffff -> 0x00f8000000
[    1.751740] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x01ffffffff -> 0x0400000000
[    1.760548] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[    1.766887] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.772430] pci_bus 0000:00: root bus resource [mem 0x600000000-0x603ffffff] (bus address [0xf8000000-0xfbffffff])
[    1.782907] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[    1.789018] pci 0000:00:00.0: PME# supported from D0 D3hot
[    1.904371] brcm-pcie fd500000.pcie: link up, 5.0 GT/s PCIe x1 (SSC)
[    1.910823] pci 0000:01:00.0: [1912:0014] type 00 class 0x0c0330
[    1.916913] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00001fff 64bit]
[    1.923886] pci 0000:01:00.0: PME# supported from D0 D3hot
[    1.938371] pci 0000:00:00.0: BAR 14: assigned [mem 0x600000000-0x6000fffff]
[    1.945500] pci 0000:01:00.0: BAR 0: assigned [mem 0x600000000-0x600001fff 64bit]
[    1.953074] pci 0000:00:00.0: PCI bridge to [bus 01]
[    1.958095] pci 0000:00:00.0:   bridge window [mem 0x600000000-0x6000fffff]
[    1.965230] pcieport 0000:00:00.0: enabling device (0000 -> 0002)
[    1.971458] pcieport 0000:00:00.0: PME: Signaling with IRQ 23
[    1.977465] pcieport 0000:00:00.0: AER: enabled with IRQ 23
[    1.983179] pci 0000:01:00.0: enabling device (0000 -> 0002)
[    6.988927] pci 0000:01:00.0: xHCI HW not ready after 5 sec (HC bug?) status = 0x1801
[    6.996862] pci 0000:01:00.0: quirk_usb_early_handoff+0x0/0x968 took 4896177 usecs
[    7.005184] simple-framebuffer 3e4e0000.framebuffer: framebuffer at 0x3e4e0000, 0x6e7000 bytes
[    7.013892] simple-framebuffer 3e4e0000.framebuffer: format=a8r8g8b8, mode=1824x984x32, linelength=7296
[    7.026754] Console: switching to colour frame buffer device 228x61
[    7.036135] simple-framebuffer 3e4e0000.framebuffer: fb0: simplefb registered!
[    7.048696] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    7.056025] printk: console [ttyS1] disabled
[    7.060403] fe215040.serial: ttyS1 at MMIO 0xfe215040 (irq = 31, base_baud = 62499999) is a 16550
[    7.069404] printk: console [ttyS1] enabled
[    7.077886] printk: bootconsole [bcm2835aux0] disabled
[    7.088802] Serial: AMBA driver
[    7.092199] msm_serial: driver initialized
[    7.097592] bcm2835-power bcm2835-power: Broadcom BCM2835 power domains driver
[    7.105367] mousedev: PS/2 mouse device common for all mice
[    7.111592] brcmstb-i2c fef04500.i2c:  @97500hz registered in polling mode
[    7.118682] brcmstb-i2c fef09500.i2c:  @97500hz registered in polling mode
[    7.126399] ledtrig-cpu: registered to indicate activity on CPUs
[    7.133020] bcm2835-mbox fe00b880.mailbox: mailbox enabled
[    7.156951] NET: Registered PF_INET6 protocol family
[    7.174534] Segment Routing with IPv6
[    7.178286] In-situ OAM (IOAM) with IPv6
[    7.182323] mip6: Mobile IPv6
[    7.185345] NET: Registered PF_PACKET protocol family
[    7.190549] mpls_gso: MPLS GSO support
[    7.197394] registered taskstats version 1
[    7.201792] Loading compiled-in X.509 certificates
[    7.244632] Loaded X.509 cert 'Build time autogenerated kernel key: 83983db0a2e3422ed1c09ac9326e286f87dd9c0e'
[    7.256299] zswap: loaded using pool lzo/zbud
[    7.264462] Key type .fscrypt registered
[    7.268454] Key type fscrypt-provisioning registered
[    7.285473] Key type encrypted registered
[    7.289560] AppArmor: AppArmor sha1 policy hashing enabled
[    7.295146] ima: No TPM chip found, activating TPM-bypass!
[    7.300722] ima: Allocated hash algorithm: sha256
[    7.305517] ima: No architecture policies found
[    7.310133] evm: Initialising EVM extended attributes:
[    7.315340] evm: security.selinux
[    7.318697] evm: security.SMACK64 (disabled)
[    7.323022] evm: security.SMACK64EXEC (disabled)
[    7.327699] evm: security.SMACK64TRANSMUTE (disabled)
[    7.332818] evm: security.SMACK64MMAP (disabled)
[    7.337495] evm: security.apparmor
[    7.340939] evm: security.ima
[    7.343942] evm: security.capability
[    7.347562] evm: HMAC attrs: 0x1
[    7.556359] fe201000.serial: ttyAMA0 at MMIO 0xfe201000 (irq = 33, base_baud = 0) is a PL011 rev2
[    7.565687] serial serial0: tty port ttyAMA0 registered
[    7.571649] raspberrypi-firmware soc:firmware: Attached to firmware from 2022-08-26T14:03:16
[    7.581835] gpio-573 (ant1): hogged as output/high
[    7.587958] gpio-577 (ant2): hogged as output/low
[    7.605203] Freeing unused kernel memory: 6528K
[    7.660804] Checked W+X mappings: passed, no W+X pages found
[    7.666581] Run /init as init process
[    7.670288]   with arguments:
[    7.670292]     /init
[    7.670295]   with environment:
[    7.670298]     HOME=/
[    7.670301]     TERM=linux
[    7.940360] usb_phy_generic phy: dummy supplies not allowed for exclusive requests
[    7.957146] sdhci: Secure Digital Host Controller Interface driver
[    7.963452] sdhci: Copyright(c) Pierre Ossman
[    7.972361] usbcore: registered new interface driver usbfs
[    7.981134] usbcore: registered new interface driver hub
[    7.988461] bcmgenet fd580000.ethernet: GENET 5.0 EPHY: 0x0000
[    7.988739] usbcore: registered new device driver usb
[    7.989085] sdhci-pltfm: SDHCI platform and OF driver helper
[    8.002036] sdhci-iproc fe300000.mmc: allocated mmc-pwrseq
[    8.039280] pps_core: LinuxPPS API ver. 1 registered
[    8.042379] mmc1: SDHCI controller on fe300000.mmc [fe300000.mmc] using PIO
[    8.044401] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    8.051555] mmc0: SDHCI controller on fe340000.mmc [fe340000.mmc] using ADMA
[    8.072286] PTP clock support registered
[    8.113638] dwc2 fe980000.usb: supply vusb_d not found, using dummy regulator
[    8.121016] dwc2 fe980000.usb: supply vusb_a not found, using dummy regulator
[    8.180510] dwc2 fe980000.usb: DWC OTG Controller
[    8.185300] dwc2 fe980000.usb: new USB bus registered, assigned bus number 1
[    8.192464] dwc2 fe980000.usb: irq 38, io mem 0xfe980000
[    8.197962] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.03
[    8.206351] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    8.210412] unimac-mdio unimac-mdio.-19: Broadcom UniMAC MDIO bus
[    8.213675] usb usb1: Product: DWC OTG Controller
[    8.224619] usb usb1: Manufacturer: Linux 6.3.0-rc6+ dwc2_hsotg
[    8.230624] usb usb1: SerialNumber: fe980000.usb
[    8.235979] hub 1-0:1.0: USB hub found
[    8.239821] hub 1-0:1.0: 1 port detected
[    8.542339] usb 1-1: new high-speed USB device number 2 using dwc2
[    8.754442] usb 1-1: New USB device found, idVendor=0424, idProduct=2514, bcdDevice= b.b3
[    8.762741] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    8.770307] hub 1-1:1.0: USB hub found
[    8.774175] hub 1-1:1.0: 4 ports detected
[    9.822715] xhci_hcd 0000:01:00.0: Timeout for Set DATAX step: 34
[    9.828909] xhci_hcd 0000:01:00.0: ROM Download Step 34 failed at position 136 bytes with (-110)
[   10.188426] xhci_hcd 0000:01:00.0: xHCI Host Controller
[   10.193749] xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 2
[   10.199301] mmc0: new HS200 MMC card at address 0001
[   10.207166] mmcblk0: mmc0:0001 BJTD4R 29.1 GiB 
[   10.213627]  mmcblk0: p1 p2
[   10.217291] mmcblk0boot0: mmc0:0001 BJTD4R 4.00 MiB 
[   10.225007] mmcblk0boot1: mmc0:0001 BJTD4R 4.00 MiB 
[   10.232142] mmcblk0rpmb: mmc0:0001 BJTD4R 4.00 MiB, chardev (241:0)
[   10.315008] xhci_hcd 0000:01:00.0: hcc params 0x014051cf hci version 0x100 quirks 0x0000001100000410
[   10.324800] xhci_hcd 0000:01:00.0: xHCI Host Controller
[   10.330118] xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 3
[   10.337628] xhci_hcd 0000:01:00.0: Host supports USB 3.0 SuperSpeed
[   10.344165] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.03
[   10.352553] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   10.359932] usb usb2: Product: xHCI Host Controller
[   10.364877] usb usb2: Manufacturer: Linux 6.3.0-rc6+ xhci-hcd
[   10.370703] usb usb2: SerialNumber: 0000:01:00.0
[   10.375841] hub 2-0:1.0: USB hub found
[   10.379725] hub 2-0:1.0: 4 ports detected
[   10.384226] usb usb3: We don't know the algorithms for LPM for this host, disabling LPM.
[   10.392500] usb usb3: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.03
[   10.400886] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   10.408216] usb usb3: Product: xHCI Host Controller
[   10.413161] usb usb3: Manufacturer: Linux 6.3.0-rc6+ xhci-hcd
[   10.418988] usb usb3: SerialNumber: 0000:01:00.0
[   10.424074] hub 3-0:1.0: USB hub found
[   10.427948] hub 3-0:1.0: 4 ports detected
[   10.591348] EXT4-fs (mmcblk0p2): mounted filesystem 76e585ad-07e2-43e0-aae7-43697734a9c4 with ordered data mode. Quota mode: none.
[   10.658391] usb 2-1: new high-speed USB device number 2 using xhci_hcd
[   10.717021] Not activating Mandatory Access Control as /sbin/tomoyo-init does not exist.
[   10.823840] usb 2-1: New USB device found, idVendor=05e3, idProduct=0610, bcdDevice= 6.54
[   10.832181] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   10.839439] usb 2-1: Product: USB2.1 Hub
[   10.843422] usb 2-1: Manufacturer: GenesysLogic
[   10.851736] hub 2-1:1.0: USB hub found
[   10.856225] hub 2-1:1.0: 4 ports detected
[   10.909886] systemd[1]: System time before build time, advancing clock.
[   10.947056] usb 3-1: new SuperSpeed USB device number 2 using xhci_hcd
[   10.955466] systemd[1]: Inserted module 'autofs4'
[   10.975706] usb 3-1: New USB device found, idVendor=05e3, idProduct=0626, bcdDevice= 6.54
[   10.984029] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   10.991283] usb 3-1: Product: USB3.1 Hub
[   10.995270] usb 3-1: Manufacturer: GenesysLogic
[   11.004261] hub 3-1:1.0: USB hub found
[   11.008720] hub 3-1:1.0: 4 ports detected
[   11.013048] systemd[1]: systemd 252.5-2 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[   11.045555] systemd[1]: Detected architecture arm64.
[   11.068093] systemd[1]: Hostname set to <rpi4-20230307>.
[   11.372962] systemd[1]: Queued start job for default target graphical.target.
[   11.404694] systemd[1]: Created slice system-getty.slice - Slice /system/getty.
[   11.431376] systemd[1]: Created slice system-modprobe.slice - Slice /system/modprobe.
[   11.459456] systemd[1]: Created slice system-serial\x2dgetty.slice - Slice /system/serial-getty.
[   11.487308] systemd[1]: Created slice system-systemd\x2dfsck.slice - Slice /system/systemd-fsck.
[   11.515115] systemd[1]: Created slice user.slice - User and Session Slice.
[   11.538596] systemd[1]: Started systemd-ask-password-console.path - Dispatch Password Requests to Console Directory Watch.
[   11.566527] systemd[1]: Started systemd-ask-password-wall.path - Forward Password Requests to Wall Directory Watch.
[   11.594854] systemd[1]: Set up automount proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point.
[   11.626478] systemd[1]: Reached target cryptsetup.target - Local Encrypted Volumes.
[   11.650418] systemd[1]: Reached target integritysetup.target - Local Integrity Protected Volumes.
[   11.678427] systemd[1]: Reached target paths.target - Path Units.
[   11.702417] systemd[1]: Reached target remote-fs.target - Remote File Systems.
[   11.726391] systemd[1]: Reached target slices.target - Slice Units.
[   11.750410] systemd[1]: Reached target swap.target - Swaps.
[   11.770407] systemd[1]: Reached target veritysetup.target - Local Verity Protected Volumes.
[   11.798822] systemd[1]: Listening on systemd-fsckd.socket - fsck to fsckd communication Socket.
[   11.826561] systemd[1]: Listening on systemd-initctl.socket - initctl Compatibility Named Pipe.
[   11.854837] systemd[1]: Listening on systemd-journald-audit.socket - Journal Audit Socket.
[   11.882732] systemd[1]: Listening on systemd-journald-dev-log.socket - Journal Socket (/dev/log).
[   11.910746] systemd[1]: Listening on systemd-journald.socket - Journal Socket.
[   11.935246] systemd[1]: Listening on systemd-udevd-control.socket - udev Control Socket.
[   11.962692] systemd[1]: Listening on systemd-udevd-kernel.socket - udev Kernel Socket.
[   12.006600] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[   12.033105] systemd[1]: Mounting dev-mqueue.mount - POSIX Message Queue File System...
[   12.061091] systemd[1]: Mounting sys-kernel-debug.mount - Kernel Debug File System...
[   12.088991] systemd[1]: Mounting sys-kernel-tracing.mount - Kernel Trace File System...
[   12.117491] systemd[1]: Starting kmod-static-nodes.service - Create List of Static Device Nodes...
[   12.145364] systemd[1]: Starting modprobe@configfs.service - Load Kernel Module configfs...
[   12.173387] systemd[1]: Starting modprobe@drm.service - Load Kernel Module drm...
[   12.201603] systemd[1]: Starting modprobe@efi_pstore.service - Load Kernel Module efi_pstore...
[   12.229518] systemd[1]: Starting modprobe@fuse.service - Load Kernel Module fuse...
[   12.248682] fuse: init (API version 7.38)
[   12.254624] systemd[1]: systemd-fsck-root.service - File System Check on Root Device was skipped because of an unmet condition check (ConditionPathExists=!/run/initramfs/fsck-root).
[   12.290806] systemd[1]: Starting systemd-journald.service - Journal Service...
[   12.318928] systemd[1]: Starting systemd-modules-load.service - Load Kernel Modules...
[   12.346941] systemd[1]: Starting systemd-remount-fs.service - Remount Root and Kernel File Systems...
[   12.369881] EXT4-fs (mmcblk0p2): re-mounted 76e585ad-07e2-43e0-aae7-43697734a9c4. Quota mode: none.
[   12.379420] systemd[1]: systemd-repart.service - Repartition Root Disk was skipped because no trigger condition checks were met.
[   12.394204] systemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices...
[   12.422815] systemd[1]: Started systemd-journald.service - Journal Service.
[   12.825398] systemd-journald[222]: Received client request to flush runtime journal.
[   12.836254] systemd-journald[222]: File /var/log/journal/04dbc87385f34b49af5f5948a7d17809/system.journal corrupted or uncleanly shut down, renaming and replacing.
[   13.546414] vchiq: module is from the staging directory, the quality is unknown, you have been warned.
[   13.559829] iproc-rng200 fe104000.rng: hwrng registered
[   13.613221] bcm2835-wdt bcm2835-wdt: Broadcom BCM2835 watchdog timer
[   13.710060] [drm] Initialized v3d 1.0.0 20180419 for fec00000.gpu on minor 0
[   13.739924] alg: No test for fips(ansi_cprng) (fips_ansi_cprng)
[   13.750638] mc: Linux media interface: v0.10
[   13.785003] snd_bcm2835: module is from the staging directory, the quality is unknown, you have been warned.
[   13.801798] videodev: Linux video capture interface: v2.00
[   13.811527] bcm2835_audio bcm2835_audio: card created with 8 channels
[   13.873899] bcm2835_mmal_vchiq: module is from the staging directory, the quality is unknown, you have been warned.
[   13.898212] Console: switching to colour dummy device 80x25
[   13.898513] bcm2835_v4l2: module is from the staging directory, the quality is unknown, you have been warned.
[   13.935810] audit: type=1400 audit(1675040631.023:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=346 comm="apparmor_parser"
[   13.951158] audit: type=1400 audit(1675040631.027:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=347 comm="apparmor_parser"
[   13.951770] vc4-drm gpu: bound fe400000.hvs (ops vc4_hvs_ops [vc4])
[   13.968187] audit: type=1400 audit(1675040631.027:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=347 comm="apparmor_parser"
[   13.989368] audit: type=1400 audit(1675040631.039:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=349 comm="apparmor_parser"
[   14.007674] audit: type=1400 audit(1675040631.039:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=349 comm="apparmor_parser"
[   14.025945] audit: type=1400 audit(1675040631.039:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=349 comm="apparmor_parser"
[   14.044325] audit: type=1400 audit(1675040631.039:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/{,usr/}sbin/dhclient" pid=349 comm="apparmor_parser"
[   14.053979] Registered IR keymap rc-cec
[   14.065816] rc rc0: vc4-hdmi-0 as /devices/platform/soc/fef00700.hdmi/rc/rc0
[   14.073723] input: vc4-hdmi-0 as /devices/platform/soc/fef00700.hdmi/rc/rc0/input0
[   14.111139] vc4-drm gpu: bound fef00700.hdmi (ops vc4_hdmi_ops [vc4])
[   14.118921] Registered IR keymap rc-cec
[   14.122949] rc rc1: vc4-hdmi-1 as /devices/platform/soc/fef05700.hdmi/rc/rc1
[   14.130483] input: vc4-hdmi-1 as /devices/platform/soc/fef05700.hdmi/rc/rc1/input1
[   14.149200] vc4-drm gpu: bound fef05700.hdmi (ops vc4_hdmi_ops [vc4])
[   14.156226] vc4-drm gpu: bound fe004000.txp (ops vc4_txp_ops [vc4])
[   14.166016] vc4-drm gpu: bound fe206000.pixelvalve (ops vc4_crtc_ops [vc4])
[   14.174576] vc4-drm gpu: bound fe207000.pixelvalve (ops vc4_crtc_ops [vc4])
[   14.182578] vc4-drm gpu: bound fe20a000.pixelvalve (ops vc4_crtc_ops [vc4])
[   14.190243] vc4-drm gpu: bound fe216000.pixelvalve (ops vc4_crtc_ops [vc4])
[   14.199249] [drm] Initialized vc4 0.0.0 20140616 for gpu on minor 1
[   14.226017] Bluetooth: Core ver 2.22
[   14.229811] NET: Registered PF_BLUETOOTH protocol family
[   14.235247] Bluetooth: HCI device and connection manager initialized
[   14.242619] Bluetooth: HCI socket layer initialized
[   14.247632] Bluetooth: L2CAP socket layer initialized
[   14.252788] Bluetooth: SCO socket layer initialized
[   14.275535] systemd-journald[222]: Oldest entry in /var/log/journal/04dbc87385f34b49af5f5948a7d17809/system.journal is older than the configured file retention duration (1month), suggesting rotation.
[   14.275550] systemd-journald[222]: /var/log/journal/04dbc87385f34b49af5f5948a7d17809/system.journal: Journal header limits reached or header out-of-date, rotating.
[   14.283240] Console: switching to colour frame buffer device 240x67
[   14.323550] Bluetooth: HCI UART driver ver 2.3
[   14.335320] vc4-drm gpu: [drm] fb0: vc4drmfb frame buffer device
[   14.341668] Bluetooth: HCI UART protocol H4 registered
[   14.357601] Bluetooth: HCI UART protocol LL registered
[   14.362846] Bluetooth: HCI UART protocol ATH3K registered
[   14.368456] Bluetooth: HCI UART protocol Three-wire (H5) registered
[   14.375594] Bluetooth: HCI UART protocol Intel registered
[   14.381875] Bluetooth: HCI UART protocol Broadcom registered
[   14.383301] hci_uart_bcm serial0-0: supply vbat not found, using dummy regulator
[   14.388107] Bluetooth: HCI UART protocol QCA registered
[   14.396353] hci_uart_bcm serial0-0: supply vddio not found, using dummy regulator
[   14.402219] Bluetooth: HCI UART protocol AG6XX registered
[   14.402257] Bluetooth: HCI UART protocol Marvell registered
[   14.530445] uart-pl011 fe201000.serial: no DMA platform data
[   15.194826] bcmgenet fd580000.ethernet: configuring instance for external RGMII (RX delay)
[   15.263357] bcmgenet fd580000.ethernet eth0: Link is Down
[   16.674349] Bluetooth: hci0: command 0xfc18 tx timeout
[   19.362458] bcmgenet fd580000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[   19.370808] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   24.674343] Bluetooth: hci0: BCM: failed to write update baudrate (-110)
[   24.681157] Bluetooth: hci0: Failed to set baudrate
[   26.690343] Bluetooth: hci0: command 0x0c03 tx timeout
[   30.517817] memfd_create() without MFD_EXEC nor MFD_NOEXEC_SEAL, pid=481 'systemd'
[   34.914342] Bluetooth: hci0: BCM: Reset failed (-110)

[-- Attachment #1.3: dmesg-raspios.txt --]
[-- Type: text/plain, Size: 29624 bytes --]

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.15.84-v8+ (dom@buildbot) (aarch64-linux-gnu-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1613 SMP PREEMPT Thu Jan 5 12:03:08 GMT 2023
[    0.000000] random: crng init done
[    0.000000] Machine model: Raspberry Pi Compute Module 4 Rev 1.0
[    0.000000] efi: UEFI not found.
[    0.000000] Reserved memory: created CMA memory pool at 0x000000001ac00000, size 320 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000003fffffff]
[    0.000000]   DMA32    [mem 0x0000000040000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000001ffffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000003b3fffff]
[    0.000000]   node   0: [mem 0x0000000040000000-0x00000000fbffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x00000001ffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000001ffffffff]
[    0.000000] On node 0, zone DMA32: 19456 pages in unavailable ranges
[    0.000000] On node 0, zone Normal: 16384 pages in unavailable ranges
[    0.000000] percpu: Embedded 28 pages/cpu s77336 r8192 d29160 u114688
[    0.000000] pcpu-alloc: s77336 r8192 d29160 u114688 alloc=28*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: Spectre-v2
[    0.000000] CPU features: detected: Spectre-v3a
[    0.000000] CPU features: detected: Spectre-v4
[    0.000000] CPU features: detected: Spectre-BHB
[    0.000000] CPU features: kernel page table isolation forced ON by KASLR
[    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[    0.000000] CPU features: detected: ARM erratum 1742098
[    0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 2028848
[    0.000000] Kernel command line: coherent_pool=1M 8250.nr_uarts=0 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 video=HDMI-A-1:1920x1080M@60 smsc95xx.macaddr=E4:5F:01:4D:5F:D7 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000  console=ttyS0,115200 console=tty1 root=PARTUUID=2633f715-02 rootfstype=ext4 fsck.repair=yes rootwait
[    0.000000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, linear)
[    0.000000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x0000000037400000-0x000000003b400000] (64MB)
[    0.000000] Memory: 7668384K/8245248K available (11520K kernel code, 1956K rwdata, 4136K rodata, 3776K init, 975K bss, 249184K reserved, 327680K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] ftrace: allocating 38065 entries in 149 pages
[    0.000000] ftrace: allocated 149 pages with 4 groups
[    0.000000] trace event string verifier disabled
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4.
[    0.000000] 	Trampoline variant of Tasks RCU enabled.
[    0.000000] 	Rude variant of Tasks RCU enabled.
[    0.000000] 	Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GIC: Using split EOI/Deactivate mode
[    0.000000] irq_brcmstb_l2: registered L2 intc (/soc/interrupt-controller@7ef00100, parent irq: 10)
[    0.000000] arch_timer: cp15 timer(s) running at 54.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xc743ce346, max_idle_ns: 440795203123 ns
[    0.000001] sched_clock: 56 bits at 54MHz, resolution 18ns, wraps every 4398046511102ns
[    0.000355] Console: colour dummy device 80x25
[    0.000966] printk: console [tty1] enabled
[    0.001048] Calibrating delay loop (skipped), value calculated using timer frequency.. 108.00 BogoMIPS (lpj=216000)
[    0.001089] pid_max: default: 32768 minimum: 301
[    0.001248] LSM: Security Framework initializing
[    0.001546] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.001681] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.003002] cgroup: Disabling memory control group subsystem
[    0.006038] rcu: Hierarchical SRCU implementation.
[    0.007234] EFI services will not be available.
[    0.007806] smp: Bringing up secondary CPUs ...
[    0.009038] Detected PIPT I-cache on CPU1
[    0.009125] CPU1: Booted secondary processor 0x0000000001 [0x410fd083]
[    0.010352] Detected PIPT I-cache on CPU2
[    0.010404] CPU2: Booted secondary processor 0x0000000002 [0x410fd083]
[    0.011592] Detected PIPT I-cache on CPU3
[    0.011642] CPU3: Booted secondary processor 0x0000000003 [0x410fd083]
[    0.011796] smp: Brought up 1 node, 4 CPUs
[    0.011893] SMP: Total of 4 processors activated.
[    0.011963] CPU features: detected: 32-bit EL0 Support
[    0.011986] CPU features: detected: 32-bit EL1 Support
[    0.012010] CPU features: detected: CRC32 instructions
[    0.045452] CPU: All CPU(s) started at EL2
[    0.045557] alternatives: patching kernel code
[    0.047483] devtmpfs: initialized
[    0.059212] Enabled cp15_barrier support
[    0.059287] Enabled setend support
[    0.059328] KASLR enabled
[    0.059550] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.059602] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    0.069809] pinctrl core: initialized pinctrl subsystem
[    0.070734] DMI not present or invalid.
[    0.071477] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.075331] DMA: preallocated 1024 KiB GFP_KERNEL pool for atomic allocations
[    0.075667] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.076633] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.076757] audit: initializing netlink subsys (disabled)
[    0.077092] audit: type=2000 audit(0.076:1): state=initialized audit_enabled=0 res=1
[    0.077702] thermal_sys: Registered thermal governor 'step_wise'
[    0.077948] cpuidle: using governor menu
[    0.078275] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.078489] ASID allocator initialised with 32768 entries
[    0.078673] Serial: AMBA PL011 UART driver
[    0.091640] bcm2835-mbox fe00b880.mailbox: mailbox enabled
[    0.116400] raspberrypi-firmware soc:firmware: Attached to firmware from 2023-01-05T10:46:54, variant start
[    0.120415] raspberrypi-firmware soc:firmware: Firmware hash is 8ba17717fbcedd4c3b6d4bce7e50c7af4155cba9
[    0.173306] bcm2835-dma fe007000.dma: DMA legacy API manager, dmachans=0x1
[    0.177690] vgaarb: loaded
[    0.178161] SCSI subsystem initialized
[    0.178406] usbcore: registered new interface driver usbfs
[    0.178475] usbcore: registered new interface driver hub
[    0.178555] usbcore: registered new device driver usb
[    0.178945] usb_phy_generic phy: supply vcc not found, using dummy regulator
[    0.179178] usb_phy_generic phy: dummy supplies not allowed for exclusive requests
[    0.179503] pps_core: LinuxPPS API ver. 1 registered
[    0.179527] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.179566] PTP clock support registered
[    0.181225] clocksource: Switched to clocksource arch_sys_counter
[    0.273823] VFS: Disk quotas dquot_6.6.0
[    0.273976] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.274169] FS-Cache: Loaded
[    0.274443] CacheFiles: Loaded
[    0.275842] simple-framebuffer 3e3cf000.framebuffer: framebuffer at 0x3e3cf000, 0x7f8000 bytes
[    0.275880] simple-framebuffer 3e3cf000.framebuffer: format=a8r8g8b8, mode=1920x1080x32, linelength=7680
[    0.288079] Console: switching to colour frame buffer device 240x67
[    0.299085] simple-framebuffer 3e3cf000.framebuffer: fb0: simplefb registered!
[    0.309941] NET: Registered PF_INET protocol family
[    0.311043] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.317271] tcp_listen_portaddr_hash hash table entries: 4096 (order: 4, 65536 bytes, linear)
[    0.317476] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.317589] TCP established hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.318158] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    0.319322] TCP: Hash tables configured (established 65536 bind 65536)
[    0.319726] UDP hash table entries: 4096 (order: 5, 131072 bytes, linear)
[    0.319964] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes, linear)
[    0.320498] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.322192] RPC: Registered named UNIX socket transport module.
[    0.322286] RPC: Registered udp transport module.
[    0.322348] RPC: Registered tcp transport module.
[    0.322411] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.322500] PCI: CLS 0 bytes, default 64
[    0.325077] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
[    0.325580] kvm [1]: IPA Size Limit: 44 bits
[    0.326840] kvm [1]: vgic interrupt IRQ9
[    0.327166] kvm [1]: Hyp mode initialized successfully
[    1.165639] Initialise system trusted keyrings
[    1.166141] workingset: timestamp_bits=46 max_order=21 bucket_order=0
[    1.173319] zbud: loaded
[    1.175334] FS-Cache: Netfs 'nfs' registered for caching
[    1.176145] NFS: Registering the id_resolver key type
[    1.176245] Key type id_resolver registered
[    1.176304] Key type id_legacy registered
[    1.176468] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    1.176553] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    1.177919] Key type asymmetric registered
[    1.177987] Asymmetric key parser 'x509' registered
[    1.178129] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    1.178482] io scheduler mq-deadline registered
[    1.178549] io scheduler kyber registered
[    1.187178] gpio-507 (ant1): hogged as output/high
[    1.191521] gpio-511 (ant2): hogged as output/low
[    1.193106] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    1.196194] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    1.199282] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x063fffffff -> 0x00c0000000
[    1.202337] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x01ffffffff -> 0x0400000000
[    1.239327] brcm-pcie fd500000.pcie: link up, 5.0 GT/s PCIe x1 (SSC)
[    1.242607] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[    1.245566] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.248544] pci_bus 0000:00: root bus resource [mem 0x600000000-0x63fffffff] (bus address [0xc0000000-0xffffffff])
[    1.251681] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[    1.254957] pci 0000:00:00.0: PME# supported from D0 D3hot
[    1.261935] pci 0000:01:00.0: [1912:0014] type 00 class 0x0c0330
[    1.265001] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00001fff 64bit]
[    1.268294] pci 0000:01:00.0: PME# supported from D0 D3hot
[    1.274980] pci 0000:00:00.0: BAR 8: assigned [mem 0x600000000-0x6000fffff]
[    1.277979] pci 0000:01:00.0: BAR 0: assigned [mem 0x600000000-0x600001fff 64bit]
[    1.280967] pci 0000:00:00.0: PCI bridge to [bus 01]
[    1.283928] pci 0000:00:00.0:   bridge window [mem 0x600000000-0x6000fffff]
[    1.286965] pci 0000:00:00.0: enabling device (0000 -> 0002)
[    1.289902] pci 0000:01:00.0: enabling device (0000 -> 0002)
[    6.292814] pci 0000:01:00.0: xHCI HW not ready after 5 sec (HC bug?) status = 0x1801
[    6.295779] pci 0000:01:00.0: quirk_usb_early_handoff+0x0/0x8b0 took 4891439 usecs
[    6.311060] iproc-rng200 fe104000.rng: hwrng registered
[    6.314534] vc-mem: phys_addr:0x00000000 mem_base=0x3ec00000 mem_size:0x40000000(1024 MiB)
[    6.319013] gpiomem-bcm2835 fe200000.gpiomem: Initialised: Registers at 0xfe200000
[    6.335069] brd: module loaded
[    6.348156] loop: module loaded
[    6.351963] Loading iSCSI transport class v2.0-870.
[    6.360243] bcmgenet fd580000.ethernet: GENET 5.0 EPHY: 0x0000
[    6.421392] unimac-mdio unimac-mdio.-19: Broadcom UniMAC MDIO bus
[    6.425459] usbcore: registered new interface driver r8152
[    6.428322] usbcore: registered new interface driver lan78xx
[    6.431160] usbcore: registered new interface driver smsc95xx
[    6.434470] xhci_hcd 0000:01:00.0: xHCI Host Controller
[    6.437182] xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 1
[   16.439927] xhci_hcd 0000:01:00.0: can't setup: -110
[   16.442611] xhci_hcd 0000:01:00.0: USB bus 1 deregistered
[   16.445296] xhci_hcd 0000:01:00.0: init 0000:01:00.0 fail, -110
[   16.447938] xhci_hcd: probe of 0000:01:00.0 failed with error -110
[   16.476657] xhci-hcd fe9c0000.xhci: xHCI Host Controller
[   16.479403] xhci-hcd fe9c0000.xhci: new USB bus registered, assigned bus number 1
[   16.482477] xhci-hcd fe9c0000.xhci: hcc params 0x0220fe65 hci version 0x110 quirks 0x0000000000010010
[   16.485318] xhci-hcd fe9c0000.xhci: irq 60, io mem 0xfe9c0000
[   16.488253] xhci-hcd fe9c0000.xhci: xHCI Host Controller
[   16.490882] xhci-hcd fe9c0000.xhci: new USB bus registered, assigned bus number 2
[   16.493588] xhci-hcd fe9c0000.xhci: Host supports USB 3.0 SuperSpeed
[   16.496643] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.15
[   16.499387] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   16.502112] usb usb1: Product: xHCI Host Controller
[   16.504795] usb usb1: Manufacturer: Linux 5.15.84-v8+ xhci-hcd
[   16.507470] usb usb1: SerialNumber: fe9c0000.xhci
[   16.510874] hub 1-0:1.0: USB hub found
[   16.513639] hub 1-0:1.0: 1 port detected
[   16.516948] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[   16.519964] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.15
[   16.522732] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   16.525424] usb usb2: Product: xHCI Host Controller
[   16.528124] usb usb2: Manufacturer: Linux 5.15.84-v8+ xhci-hcd
[   16.530826] usb usb2: SerialNumber: fe9c0000.xhci
[   16.534135] hub 2-0:1.0: USB hub found
[   16.536843] hub 2-0:1.0: config failed, hub doesn't have any ports! (err -19)
[   16.540215] dwc_otg: version 3.00a 10-AUG-2012 (platform bus)
[   16.543346] dwc_otg: FIQ enabled
[   16.543358] dwc_otg: NAK holdoff enabled
[   16.543367] dwc_otg: FIQ split-transaction FSM enabled
[   16.543381] Module dwc_common_port init
[   16.543991] usbcore: registered new interface driver uas
[   16.546859] usbcore: registered new interface driver usb-storage
[   16.549814] mousedev: PS/2 mouse device common for all mice
[   16.557943] sdhci: Secure Digital Host Controller Interface driver
[   16.560691] sdhci: Copyright(c) Pierre Ossman
[   16.564066] sdhci-pltfm: SDHCI platform and OF driver helper
[   16.570498] ledtrig-cpu: registered to indicate activity on CPUs
[   16.573830] hid: raw HID events driver (C) Jiri Kosina
[   16.576692] usbcore: registered new interface driver usbhid
[   16.579387] usbhid: USB HID core driver
[   16.582406] ashmem: initialized
[   16.594310] NET: Registered PF_PACKET protocol family
[   16.597253] Key type dns_resolver registered
[   16.601363] registered taskstats version 1
[   16.604033] Loading compiled-in X.509 certificates
[   16.607327] Key type .fscrypt registered
[   16.610011] Key type fscrypt-provisioning registered
[   16.624033] uart-pl011 fe201000.serial: there is not valid maps for state default
[   16.627121] uart-pl011 fe201000.serial: cts_event_workaround enabled
[   16.629994] fe201000.serial: ttyAMA0 at MMIO 0xfe201000 (irq = 19, base_baud = 0) is a PL011 rev2
[   16.644203] bcm2835-wdt bcm2835-wdt: Broadcom BCM2835 watchdog timer
[   16.647525] bcm2835-power bcm2835-power: Broadcom BCM2835 power domains driver
[   16.651495] mmc-bcm2835 fe300000.mmcnr: mmc_debug:0 mmc_debug2:0
[   16.654242] mmc-bcm2835 fe300000.mmcnr: DMA channel allocated
[   16.688803] of_cfs_init
[   16.691670] of_cfs_init: OK
[   16.722560] mmc0: SDHCI controller on fe340000.mmc [fe340000.mmc] using ADMA
[   16.725724] Waiting for root device PARTUUID=2633f715-02...
[   16.769309] usb 1-1: new high-speed USB device number 2 using xhci-hcd
[   16.796903] mmc0: new DDR MMC card at address 0001
[   16.800837] mmcblk0: mmc0:0001 BJTD4R 29.1 GiB 
[   16.809033]  mmcblk0: p1 p2
[   16.812442] mmcblk0: mmc0:0001 BJTD4R 29.1 GiB
[   16.815836] mmcblk0boot0: mmc0:0001 BJTD4R 4.00 MiB 
[   16.821551] mmcblk0boot1: mmc0:0001 BJTD4R 4.00 MiB 
[   16.826723] mmcblk0rpmb: mmc0:0001 BJTD4R 4.00 MiB, chardev (242:0)
[   16.846435] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[   16.849554] VFS: Mounted root (ext4 filesystem) readonly on device 179:2.
[   16.856110] devtmpfs: mounted
[   16.867885] Freeing unused kernel memory: 3776K
[   16.871018] Run /sbin/init as init process
[   16.873799]   with arguments:
[   16.873812]     /sbin/init
[   16.873824]   with environment:
[   16.873834]     HOME=/
[   16.873844]     TERM=linux
[   16.917625] usb 1-1: New USB device found, idVendor=0424, idProduct=2514, bcdDevice= b.b3
[   16.920517] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[   16.976801] hub 1-1:1.0: USB hub found
[   16.979955] hub 1-1:1.0: 4 ports detected
[   17.115025] systemd[1]: System time before build time, advancing clock.
[   17.281006] NET: Registered PF_INET6 protocol family
[   17.285955] Segment Routing with IPv6
[   17.288759] In-situ OAM (IOAM) with IPv6
[   17.347801] systemd[1]: systemd 247.3-7+deb11u1 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
[   17.355107] systemd[1]: Detected architecture arm64.
[   17.372826] systemd[1]: Set hostname to <raspberrypi>.
[   18.007461] systemd[1]: Queued start job for default target Multi-User System.
[   18.015245] systemd[1]: Created slice system-getty.slice.
[   18.023361] systemd[1]: Created slice system-modprobe.slice.
[   18.031070] systemd[1]: Created slice system-systemd\x2dfsck.slice.
[   18.038628] systemd[1]: Created slice User and Session Slice.
[   18.045695] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[   18.052794] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[   18.060646] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[   18.067819] systemd[1]: Reached target Local Encrypted Volumes.
[   18.074931] systemd[1]: Reached target Paths.
[   18.081989] systemd[1]: Reached target Slices.
[   18.088999] systemd[1]: Reached target Swap.
[   18.098874] systemd[1]: Listening on Syslog Socket.
[   18.107183] systemd[1]: Listening on fsck to fsckd communication Socket.
[   18.114639] systemd[1]: Listening on initctl Compatibility Named Pipe.
[   18.123004] systemd[1]: Listening on Journal Audit Socket.
[   18.130947] systemd[1]: Listening on Journal Socket (/dev/log).
[   18.139130] systemd[1]: Listening on Journal Socket.
[   18.150293] systemd[1]: Listening on udev Control Socket.
[   18.158768] systemd[1]: Listening on udev Kernel Socket.
[   18.167005] systemd[1]: Condition check resulted in Huge Pages File System being skipped.
[   18.174867] systemd[1]: Mounting POSIX Message Queue File System...
[   18.188259] systemd[1]: Mounting RPC Pipe File System...
[   18.203433] systemd[1]: Mounting Kernel Debug File System...
[   18.216228] systemd[1]: Mounting Kernel Trace File System...
[   18.224478] systemd[1]: Condition check resulted in Kernel Module supporting RPCSEC_GSS being skipped.
[   18.233963] systemd[1]: Starting Restore / save the current clock...
[   18.247761] systemd[1]: Starting Set the console keyboard layout...
[   18.263157] systemd[1]: Starting Create list of static device nodes for the current kernel...
[   18.277612] systemd[1]: Starting Load Kernel Module configfs...
[   18.292076] systemd[1]: Starting Load Kernel Module drm...
[   18.310158] systemd[1]: Starting Load Kernel Module fuse...
[   18.322455] systemd[1]: Condition check resulted in Set Up Additional Binary Formats being skipped.
[   18.332214] systemd[1]: Starting File System Check on Root Device...
[   18.351258] systemd[1]: Starting Journal Service...
[   18.366322] fuse: init (API version 7.34)
[   18.370518] systemd[1]: Starting Load Kernel Modules...
[   18.386246] systemd[1]: Starting Coldplug All udev Devices...
[   18.429542] systemd[1]: Mounted POSIX Message Queue File System.
[   18.445993] systemd[1]: Mounted RPC Pipe File System.
[   18.472913] systemd[1]: Mounted Kernel Debug File System.
[   18.480740] systemd[1]: Mounted Kernel Trace File System.
[   18.495347] systemd[1]: Finished Restore / save the current clock.
[   18.526563] systemd[1]: Finished Create list of static device nodes for the current kernel.
[   18.537341] systemd[1]: modprobe@configfs.service: Succeeded.
[   18.542775] systemd[1]: Finished Load Kernel Module configfs.
[   18.552696] systemd[1]: modprobe@drm.service: Succeeded.
[   18.558659] systemd[1]: Finished Load Kernel Module drm.
[   18.568541] systemd[1]: modprobe@fuse.service: Succeeded.
[   18.573880] systemd[1]: Finished Load Kernel Module fuse.
[   18.583801] systemd[1]: Finished File System Check on Root Device.
[   18.594209] systemd[1]: Finished Load Kernel Modules.
[   18.614455] systemd[1]: Mounting FUSE Control File System...
[   18.627736] systemd[1]: Mounting Kernel Configuration File System...
[   18.642651] systemd[1]: Started File System Check Daemon to report status.
[   18.665986] systemd[1]: Starting Remount Root and Kernel File Systems...
[   18.678575] systemd[1]: Starting Apply Kernel Variables...
[   18.693594] systemd[1]: Mounted FUSE Control File System.
[   18.717867] systemd[1]: Mounted Kernel Configuration File System.
[   18.737925] systemd[1]: Started Journal Service.
[   18.843553] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null). Quota mode: none.
[   18.900078] systemd-journald[146]: Received client request to flush runtime journal.
[   18.908223] systemd-journald[146]: File /var/log/journal/5c71b33d54d54b56a83192f2a7dd0c26/system.journal corrupted or uncleanly shut down, renaming and replacing.
[   19.868410] mc: Linux media interface: v0.10
[   20.022952] videodev: Linux video capture interface: v2.00
[   20.085706] vc_sm_cma: module is from the staging directory, the quality is unknown, you have been warned.
[   20.116995] bcm2835_vc_sm_cma_probe: Videocore shared memory driver
[   20.117044] [vc_sm_connected_init]: start
[   20.126068] [vc_sm_connected_init]: installed successfully
[   20.165594] snd_bcm2835: module is from the staging directory, the quality is unknown, you have been warned.
[   20.178502] bcm2835_audio bcm2835_audio: there is not valid maps for state default
[   20.222178] bcm2835_mmal_vchiq: module is from the staging directory, the quality is unknown, you have been warned.
[   20.231807] bcm2835_mmal_vchiq: module is from the staging directory, the quality is unknown, you have been warned.
[   20.236649] brcmstb-i2c fef04500.i2c:  @97500hz registered in polling mode
[   20.237863] brcmstb-i2c fef09500.i2c:  @97500hz registered in polling mode
[   20.238268] bcm2835_v4l2: module is from the staging directory, the quality is unknown, you have been warned.
[   20.251942] bcm2835_isp: module is from the staging directory, the quality is unknown, you have been warned.
[   20.269008] bcm2835-isp bcm2835-isp: Device node output[0] registered as /dev/video13
[   20.270311] bcm2835-isp bcm2835-isp: Device node capture[0] registered as /dev/video14
[   20.296440] bcm2835-isp bcm2835-isp: Device node capture[1] registered as /dev/video15
[   20.333534] rpivid_hevc: module is from the staging directory, the quality is unknown, you have been warned.
[   20.339814] bcm2835_codec: module is from the staging directory, the quality is unknown, you have been warned.
[   20.371042] rpivid feb10000.codec: Device registered as /dev/video19
[   20.383685] bcm2835-isp bcm2835-isp: Device node stats[2] registered as /dev/video16
[   20.383742] bcm2835-isp bcm2835-isp: Register output node 0 with media controller
[   20.383765] bcm2835-isp bcm2835-isp: Register capture node 1 with media controller
[   20.383784] bcm2835-isp bcm2835-isp: Register capture node 2 with media controller
[   20.383826] bcm2835-isp bcm2835-isp: Register capture node 3 with media controller
[   20.425869] bcm2835-isp bcm2835-isp: Device node output[0] registered as /dev/video20
[   20.426721] bcm2835-isp bcm2835-isp: Device node capture[0] registered as /dev/video21
[   20.427593] bcm2835-isp bcm2835-isp: Device node capture[1] registered as /dev/video22
[   20.427623] bcm2835-codec bcm2835-codec: Device registered as /dev/video10
[   20.427682] bcm2835-codec bcm2835-codec: Loaded V4L2 decode
[   20.435536] bcm2835-isp bcm2835-isp: Device node stats[2] registered as /dev/video23
[   20.435595] bcm2835-isp bcm2835-isp: Register output node 0 with media controller
[   20.435638] bcm2835-isp bcm2835-isp: Register capture node 1 with media controller
[   20.435657] bcm2835-isp bcm2835-isp: Register capture node 2 with media controller
[   20.435674] bcm2835-isp bcm2835-isp: Register capture node 3 with media controller
[   20.436031] bcm2835-isp bcm2835-isp: Loaded V4L2 bcm2835-isp
[   20.555668] bcm2835-codec bcm2835-codec: Device registered as /dev/video11
[   20.558412] bcm2835-codec bcm2835-codec: Loaded V4L2 encode
[   20.572482] bcm2835-codec bcm2835-codec: Device registered as /dev/video12
[   20.572563] bcm2835-codec bcm2835-codec: Loaded V4L2 isp
[   20.609906] bcm2835-codec bcm2835-codec: Device registered as /dev/video18
[   20.609978] bcm2835-codec bcm2835-codec: Loaded V4L2 image_fx
[   20.617966] bcm2835-codec bcm2835-codec: Device registered as /dev/video31
[   20.618066] bcm2835-codec bcm2835-codec: Loaded V4L2 encode_image
[   20.700103] [drm] Initialized v3d 1.0.0 20180419 for fec00000.v3d on minor 0
[   21.480722] checking generic (3e3cf000 7f8000) vs hw (0 ffffffffffffffff)
[   21.480772] fb0: switching to vc4 from simple
[   21.490606] Console: switching to colour dummy device 80x25
[   21.538621] vc4-drm gpu: bound fe400000.hvs (ops vc4_hvs_ops [vc4])
[   21.541009] Registered IR keymap rc-cec
[   21.541283] rc rc0: vc4 as /devices/platform/soc/fef00700.hdmi/rc/rc0
[   21.541508] input: vc4 as /devices/platform/soc/fef00700.hdmi/rc/rc0/input0
[   21.549809] vc4-drm gpu: bound fef00700.hdmi (ops vc4_hdmi_ops [vc4])
[   21.560367] Registered IR keymap rc-cec
[   21.560599] rc rc1: vc4 as /devices/platform/soc/fef05700.hdmi/rc/rc1
[   21.560831] input: vc4 as /devices/platform/soc/fef05700.hdmi/rc/rc1/input1
[   21.567421] vc4-drm gpu: bound fef05700.hdmi (ops vc4_hdmi_ops [vc4])
[   21.567959] vc4-drm gpu: bound fe004000.txp (ops vc4_txp_ops [vc4])
[   21.568349] vc4-drm gpu: bound fe206000.pixelvalve (ops vc4_crtc_ops [vc4])
[   21.568694] vc4-drm gpu: bound fe207000.pixelvalve (ops vc4_crtc_ops [vc4])
[   21.569042] vc4-drm gpu: bound fe20a000.pixelvalve (ops vc4_crtc_ops [vc4])
[   21.569502] vc4-drm gpu: bound fe216000.pixelvalve (ops vc4_crtc_ops [vc4])
[   21.569876] vc4-drm gpu: bound fec12000.pixelvalve (ops vc4_crtc_ops [vc4])
[   21.586325] [drm] Initialized vc4 0.0.0 20140616 for gpu on minor 1
[   21.683816] Console: switching to colour frame buffer device 240x67
[   21.712387] vc4-drm gpu: [drm] fb0: vc4drmfb frame buffer device
[   22.009917] 8021q: 802.1Q VLAN Support v1.8
[   22.086183] uart-pl011 fe201000.serial: no DMA platform data
[   22.096980] Adding 102396k swap on /var/swap.  Priority:-2 extents:1 across:102396k SSFS
[   22.360820] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[   22.389520] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[   22.397905] cfg80211: loaded regulatory.db is malformed or signature is missing/invalid
[   22.488754] bcmgenet fd580000.ethernet: configuring instance for external RGMII (RX delay)
[   22.490537] bcmgenet fd580000.ethernet eth0: Link is Down
[   26.589395] bcmgenet fd580000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[   26.589433] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/3] dt-bindings: PCI: brcmstb: Add two optional props

[Bjorn Helgaas]

It'd be nice to mention the property names (maybe omit the "brcm,"
prefix if that helps) in the commit log so "git log --oneline" is more
useful:

  959e000f0463 ("dt-bindings: PCI: brcmstb: Add two optional props")
  ea372f45cfff ("dt-bindings: PCI: Add bindings for Brcmstb EP voltage regulators")
  504253e44a9d ("dt-bindings: PCI: Correct brcmstb interrupts, interrupt-map.")
  145790e55d82 ("dt-bindings: PCI: Add compatible string for Brcmstb 74[23]5 MIPs SOCs")
  5e8a7d26d935 ("dt-bindings: PCI: brcmstb: compatible is required")
  f435ce7ebf8c ("dt-bindings: PCI: brcmstb: add BCM4908 binding")

On Tue, Apr 11, 2023 at 12:59:16PM -0400, Jim Quinlan wrote:
> Regarding "brcm,enable-l1ss":
> 
>   The Broadcom STB/CM PCIe HW -- a core that is also used by RPi SOCs --
>   requires the driver probe() to deliberately place the HW one of three
>   CLKREQ# modes:
> 
>   (a) CLKREQ# driven by the RC unconditionally
>   (b) CLKREQ# driven by the EP for ASPM L0s, L1
>   (c) Bidirectional CLKREQ#, as used for L1 Substates (L1SS).
> 
>   The HW+driver can tell the difference between downstream devices that
>   need (a) and (b), but does not know when to configure (c).  Further, the
>   HW may cause a CPU abort on boot if guesses wrong regarding the need for
>   (c).  So we introduce the boolean "brcm,enable-l1ss" property to indicate
>   that (c) is desired.  Setting this property only makes sense when the
>   downstream device is L1SS-capable and the OS is configured to activate
>   this mode (e.g. policy==superpowersave).
> 
>   This property is already present in the Raspian version of Linux, but the
>   upstream driver implementaion that will follow adds more details and
>   discerns between (a) and (b).
> 
> Regarding "brcm,completion-timeout-us"
> 
>   Our HW will cause a CPU abort if the L1SS exit time is longer than the
>   PCIe transaction completion abort timeout.  We've been asked to make this
>   configurable, so we are introducing "brcm,completion-timeout-us".

Completion Timeout is a generic PCIe concept.  Do we want a generic
(non-brcm) name that would be documented elsewhere?  Rob?

> Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> ---
>  .../devicetree/bindings/pci/brcm,stb-pcie.yaml   | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
> 
> diff --git a/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml b/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
> index 7e15aae7d69e..f7fc2f6561bb 100644
> --- a/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
> +++ b/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
> @@ -64,6 +64,22 @@ properties:
>  
>    aspm-no-l0s: true
>  
> +  brcm,enable-l1ss:
> +    description: Indicates that PCIe L1SS power savings
> +      are desired, the downstream device is L1SS-capable, and the
> +      OS has been configured to enable this mode.  Note that when
> +      in this mode, this particular HW may not meet the requirement
> +      that requires CLKREQ# assertion to clock active to be
> +      within 400ns.

Maybe a pointer to the source of the 400ns requirement?

"requirement that requires" is a little redundant, maybe "... may not
meet the requirement that Refclk be valid within 400ns of CLKREQ#
assertion"?

(I don't actually know whether this refers to Refclk or if that would
be a true statement; this is just a possible sentence structure.)

Re: [PATCH v2 2/3] PCI: brcmstb: CLKREQ# accomodations of downstream device

[Bjorn Helgaas]

This subject line no verb.  Can you add a leading verb to suggest what
this patch does?

s/accomodations/accommodations/

On Tue, Apr 11, 2023 at 12:59:17PM -0400, Jim Quinlan wrote:
> The Broadcom STB/CM PCIe HW core, which is also used in RPi SOCs, must be
> deliberately set by the probe() into one of three mutually exclusive modes:
> 
>   (a) No CLKREQ# expected or required, refclk is always available.
>   (b) CLKREQ# is expected to be driven by downstream device when needed.
>   (c) Bidirectional CLKREQ# for L1SS capable devices.
> 
> Previously, only (b) was supported by the driver, as almost all STB/CM
> boards operate in this mode.  But now there is interest in activating L1SS
> power savings from STB/CM customers, and also interest in accomodating mode
> (a) for designs such as the RPi CM4 with IO board.

accommodating

> The HW+driver is able to tell us when mode (a) mode is needed.  But there
> is no easy way to tell if L1SS mode should be configured.  In certain
> situations, getting this wrong may cause a panic during boot time.  So we
> rely on the DT prop "brcm,enable-l1ss" to tell us when mode (c) is desired.
> Using this mode only makes sense when the downstream device is L1SS-capable
> and the OS has been configured to activate L1SS
> (e.g. policy==powersupersave).

I'm really concerned about the user experience here.  I assume users
do not want to edit the DT based on what device they plug in.  They
shouldn't need to (and probably won't) know whether the device
supports L1SS.

I hate kernel/module parameters, but I think even that would be better
then having to edit the DT.

There's obviously a period of time when L1SS is supported but not yet
enabled, so I'm *guessing* the "OS has been configured to activate
L1SS" is not actually a requirement, and choosing (c) really just
opens the possibility that L1SS can be used?

Would be nice to have a hint (maybe a line or two of the panic
message) to help users find the fix for a problem they're seeing.

Obviously the ideal would be if we could use (c) in all cases, so I
assume that's where a panic might happen.  What situation would that
be?  An endpoint that doesn't support L1SS?  One that supports L1SS
but it's not enabled?  Maybe if L1SS isn't configured correctly, e.g.,
LTR values programmed wrong?

Bjorn

Re: [PATCH v2 2/3] PCI: brcmstb: CLKREQ# accomodations of downstream device

[Florian Fainelli]

On 4/14/23 13:27, Bjorn Helgaas wrote:
> This subject line no verb.  Can you add a leading verb to suggest what
> this patch does?
> 
> s/accomodations/accommodations/
> 
> On Tue, Apr 11, 2023 at 12:59:17PM -0400, Jim Quinlan wrote:
>> The Broadcom STB/CM PCIe HW core, which is also used in RPi SOCs, must be
>> deliberately set by the probe() into one of three mutually exclusive modes:
>>
>>    (a) No CLKREQ# expected or required, refclk is always available.
>>    (b) CLKREQ# is expected to be driven by downstream device when needed.
>>    (c) Bidirectional CLKREQ# for L1SS capable devices.
>>
>> Previously, only (b) was supported by the driver, as almost all STB/CM
>> boards operate in this mode.  But now there is interest in activating L1SS
>> power savings from STB/CM customers, and also interest in accomodating mode
>> (a) for designs such as the RPi CM4 with IO board.
> 
> accommodating
> 
>> The HW+driver is able to tell us when mode (a) mode is needed.  But there
>> is no easy way to tell if L1SS mode should be configured.  In certain
>> situations, getting this wrong may cause a panic during boot time.  So we
>> rely on the DT prop "brcm,enable-l1ss" to tell us when mode (c) is desired.
>> Using this mode only makes sense when the downstream device is L1SS-capable
>> and the OS has been configured to activate L1SS
>> (e.g. policy==powersupersave).
> 
> I'm really concerned about the user experience here.  I assume users
> do not want to edit the DT based on what device they plug in.  They
> shouldn't need to (and probably won't) know whether the device
> supports L1SS.
> 
> I hate kernel/module parameters, but I think even that would be better
> then having to edit the DT.

The problem I see with kernel/module parameters is that it is really 
hard to differentiate whether they should be applied across all 
instances of the device/drivers or just for one in particular.

The Raspberry Pi 4 is a single pcie-brcmstb instance, but we have other 
systems supported by that driver on Set-top-box and Cable Modem chips 
for instance where we may have different types of end-points being 
connected, some of which could be Multi-chip-module (MCM) where 
everything is known ahead of time, and sometimes cards that are plugged 
to full-sized PCIe or mini-PCIe connectors, where some amount of runtime 
discoverability is involved.

Without inventing some custom modular parameter syntax, it may not work 
that well. The Device Tree properties at least easily allow for 
per-instance configuration.

> 
> There's obviously a period of time when L1SS is supported but not yet
> enabled, so I'm *guessing* the "OS has been configured to activate
> L1SS" is not actually a requirement, and choosing (c) really just
> opens the possibility that L1SS can be used?
> 
> Would be nice to have a hint (maybe a line or two of the panic
> message) to help users find the fix for a problem they're seeing.
> 
> Obviously the ideal would be if we could use (c) in all cases, so I
> assume that's where a panic might happen.  What situation would that
> be?  An endpoint that doesn't support L1SS?  One that supports L1SS
> but it's not enabled?  Maybe if L1SS isn't configured correctly, e.g.,
> LTR values programmed wrong?
> 
> Bjorn

-- 
Florian

Re: [PATCH v2 2/3] PCI: brcmstb: CLKREQ# accomodations of downstream device

[Jim Quinlan]

On Fri, Apr 14, 2023 at 4:27 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> This subject line no verb.  Can you add a leading verb to suggest what
> this patch does?
>
> s/accomodations/accommodations/
>
> On Tue, Apr 11, 2023 at 12:59:17PM -0400, Jim Quinlan wrote:
> > The Broadcom STB/CM PCIe HW core, which is also used in RPi SOCs, must be
> > deliberately set by the probe() into one of three mutually exclusive modes:
> >
> >   (a) No CLKREQ# expected or required, refclk is always available.
> >   (b) CLKREQ# is expected to be driven by downstream device when needed.
> >   (c) Bidirectional CLKREQ# for L1SS capable devices.
> >
> > Previously, only (b) was supported by the driver, as almost all STB/CM
> > boards operate in this mode.  But now there is interest in activating L1SS
> > power savings from STB/CM customers, and also interest in accomodating mode
> > (a) for designs such as the RPi CM4 with IO board.
>
> accommodating
>
> > The HW+driver is able to tell us when mode (a) mode is needed.  But there
> > is no easy way to tell if L1SS mode should be configured.  In certain
> > situations, getting this wrong may cause a panic during boot time.  So we
> > rely on the DT prop "brcm,enable-l1ss" to tell us when mode (c) is desired.
> > Using this mode only makes sense when the downstream device is L1SS-capable
> > and the OS has been configured to activate L1SS
> > (e.g. policy==powersupersave).
>
> I'm really concerned about the user experience here.  I assume users
> do not want to edit the DT based on what device they plug in.  They
> shouldn't need to (and probably won't) know whether the device
> supports L1SS.
>
> I hate kernel/module parameters, but I think even that would be better
> then having to edit the DT.
>
> There's obviously a period of time when L1SS is supported but not yet
> enabled, so I'm *guessing* the "OS has been configured to activate
> L1SS" is not actually a requirement, and choosing (c) really just
> opens the possibility that L1SS can be used?

Yes.  Before this patch series we had two panic scenarios:

(a) Endpoint devices with no CLKREQ# connection
(b) Endpoints that are L1SS-capable

Even without  the "brcm,enable-l1ss" property present, both (a) and
(b) should be eliminated.
The reason (b) is eliminated is because the RC driver now unadvertises
 RC L1SS by default; subsequently, Linux does
not turn it on.  So the default setting should be fine for all devices.

For those folks who have L1SS capable devices and desire L1SS power
savings, they can add
the brcm,enable-l1ss property.  But everyone should have functionality
 w/o doing anything.

As I am typing this I realize that my comments and dev_info()s  are not
aligned with what I am saying so I will change them in V3.  Sorry
about the confusion.

>
> Would be nice to have a hint (maybe a line or two of the panic
> message) to help users find the fix for a problem they're seeing.SS
>
> Obviously the ideal would be if we could use (c) in all cases, so I
> assume that's where a panic might happen.  What situation would that
> be?  An endpoint that doesn't support L1SS?  One that supports L1SS
> but it's not enabled?  Maybe if L1SS isn't configured correctly, e.g.,
> LTR values programmed wrong?

Let me test everything on Monday and get back to you before I make any
statements.


Regards,
Jim Quinilan
Broadcom STB

>
> Bjorn

Re: [PATCH v2 2/3] PCI: brcmstb: CLKREQ# accomodations of downstream device

[Bjorn Helgaas]

On Fri, Apr 14, 2023 at 01:33:29PM -0700, Florian Fainelli wrote:
> On 4/14/23 13:27, Bjorn Helgaas wrote:
> > On Tue, Apr 11, 2023 at 12:59:17PM -0400, Jim Quinlan wrote:
> ...

> > > The HW+driver is able to tell us when mode (a) mode is needed.  But there
> > > is no easy way to tell if L1SS mode should be configured.  In certain
> > > situations, getting this wrong may cause a panic during boot time.  So we
> > > rely on the DT prop "brcm,enable-l1ss" to tell us when mode (c) is desired.
> > > Using this mode only makes sense when the downstream device is L1SS-capable
> > > and the OS has been configured to activate L1SS
> > > (e.g. policy==powersupersave).
> > 
> > I'm really concerned about the user experience here.  I assume users
> > do not want to edit the DT based on what device they plug in.  They
> > shouldn't need to (and probably won't) know whether the device
> > supports L1SS.
> > 
> > I hate kernel/module parameters, but I think even that would be better
> > then having to edit the DT.
> 
> The problem I see with kernel/module parameters is that it is really hard to
> differentiate whether they should be applied across all instances of the
> device/drivers or just for one in particular.
> 
> The Raspberry Pi 4 is a single pcie-brcmstb instance, but we have other
> systems supported by that driver on Set-top-box and Cable Modem chips for
> instance where we may have different types of end-points being connected,
> some of which could be Multi-chip-module (MCM) where everything is known
> ahead of time, and sometimes cards that are plugged to full-sized PCIe or
> mini-PCIe connectors, where some amount of runtime discoverability is
> involved.
> 
> Without inventing some custom modular parameter syntax, it may not work that
> well. The Device Tree properties at least easily allow for per-instance
> configuration.

We do have this already (from
Documentation/admin-guide/kernel-parameters.txt):

        pci=option[,option...]  [PCI] various PCI subsystem options.

                                Some options herein operate on a specific device
                                or a set of devices (<pci_dev>). These are
                                specified in one of the following formats:

                                [<domain>:]<bus>:<dev>.<func>[/<dev>.<func>]*
                                pci:<vendor>:<device>[:<subvendor>:<subdevice>]

                                Note: the first format specifies a PCI
                                bus/device/function address which may change
                                if new hardware is inserted, if motherboard
                                firmware changes, or due to changes caused
                                by other kernel parameters. If the
                                domain is left unspecified, it is
                                taken to be zero. Optionally, a path
                                to a device through multiple device/function
                                addresses can be specified after the base
                                address (this is more robust against
                                renumbering issues).  The second format
                                selects devices using IDs from the
                                configuration space which may match multiple
                                devices in the system.

Re: [PATCH v2 1/3] dt-bindings: PCI: brcmstb: Add two optional props

[Rob Herring]

On Wed, Apr 12, 2023 at 09:12:07AM -0700, Florian Fainelli wrote:
> On 4/12/23 08:37, Rob Herring wrote:
> > On Wed, Apr 12, 2023 at 10:14:46AM -0400, Jim Quinlan wrote:
> > > On Wed, Apr 12, 2023 at 7:56 AM Krzysztof Kozlowski
> > > <krzysztof.kozlowski@linaro.org> wrote:
> > > > 
> > > > On 12/04/2023 13:49, Florian Fainelli wrote:
> > > > > 
> > > > > 
> > > > > On 4/12/2023 1:09 AM, Krzysztof Kozlowski wrote:
> > > > > > On 11/04/2023 18:59, Jim Quinlan wrote:
> > > > > > > Regarding "brcm,enable-l1ss":
> > > > > > > 
> > > > > > >     The Broadcom STB/CM PCIe HW -- a core that is also used by RPi SOCs --
> > > > > > >     requires the driver probe() to deliberately place the HW one of three
> > > > > > >     CLKREQ# modes:
> > > > > > > 
> > > > > > >     (a) CLKREQ# driven by the RC unconditionally
> > > > > > >     (b) CLKREQ# driven by the EP for ASPM L0s, L1
> > > > > > >     (c) Bidirectional CLKREQ#, as used for L1 Substates (L1SS).
> > > > > > > 
> > > > > > >     The HW+driver can tell the difference between downstream devices that
> > > > > > >     need (a) and (b), but does not know when to configure (c).  Further, the
> > > > > > >     HW may cause a CPU abort on boot if guesses wrong regarding the need for
> > > > > > >     (c).  So we introduce the boolean "brcm,enable-l1ss" property to indicate
> > > > > > >     that (c) is desired.  Setting this property only makes sense when the
> > > > > > >     downstream device is L1SS-capable and the OS is configured to activate
> > > > > > >     this mode (e.g. policy==superpowersave).
> > > > > > > 
> > > > > > >     This property is already present in the Raspian version of Linux, but the
> > > > > > >     upstream driver implementaion that will follow adds more details and
> > > > > > 
> > > > > > typo, implementation
> > > > > > 
> > > > > > >     discerns between (a) and (b).
> > > > > > > 
> > > > > > > Regarding "brcm,completion-timeout-us"
> > > > > > > 
> > > > > > >     Our HW will cause a CPU abort if the L1SS exit time is longer than the
> > > > > > >     PCIe transaction completion abort timeout.  We've been asked to make this
> > > > > > >     configurable, so we are introducing "brcm,completion-timeout-us".
> > > > > > > 
> > > > > > > Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> > > > > > 
> > > > > > What happened here? Where is the changelog?
> > > > > 
> > > > > It is in the cover letter:
> > > > > 
> > > > > https://lore.kernel.org/all/20230411165919.23955-1-jim2101024@gmail.com/
> > > > > 
> > > > > but it does not look like the cover letter was copied to you or Rob.
> > > > 
> > > > As you said, I did not get it.
> > > 
> > > Yes, sorry about that; I use a wrapper over the "cocci_cc" script and
> > > I need to modify one or both scripts to send the cover to the
> > > superset of recipients in the constituent commits.
> > 
> > Try out 'b4'. It's much easier.
> > 
> > In any case, I don't read cover letters. Changes to a patch belong with
> > the patch.
> 
> This is not what most other maintainers do, and there does not appear to be
> a general consensus amongst maintainers that the changes belong in the
> individual patches, or in the cover letter. 

Well, I stole that phrase from someone else (gregkh). 

> Some trees like the networking
> tree do merge commits of patch sets where the cover letter is used as part
> of the merge commit message. Other maintainers don't, and some want the
> change log after the '---' and some do not.

I'm not aware of anyone except for DRM wanting the changelog in the 
final commits, but that's really a different issue.

I'm pretty sure no one will complain about a changelog in the patches. I 
guess you just have to duplicate it if you think it should be in both. 
b4 could be taught to do that I suppose. IMO, the cover letter should 
have a higher level changelog than the individual patches.

Rob

Re: [PATCH v2 2/3] PCI: brcmstb: CLKREQ# accomodations of downstream device

[Jim Quinlan]

On Fri, Apr 14, 2023 at 12:19 PM Cyril Brulebois <kibi@debian.org> wrote:
>
> Florian Fainelli <f.fainelli@gmail.com> (2023-04-14):
> > Cyril, based upon the table and logs you provided whereby you have used the
> > following:
> >
> > - Raspberry Pi Compute Module 4 (Rev 1.0, 8G RAM, 32G storage)
> > - Raspberry Pi Compute Module 4 IO Board
> > - SupaHub PCIe-to-multiple-USB adapter, reference PCE6U1C-R02, VER 006S
> >
> > in the before/unpatched case we have a PCIe link down and in the
> > after/patched we have a PCIe link up but a kernel panic. Neither are great
> > nor resulting in a fully functional PCIe device.
>
> Based on Jim's feedback, I'm attaching a new dmesg for the aforementioned
> setup, with an unpatched kernel (dmesg-unpatched-pcie-link-up.txt).

Hello Cyril,

I'm still seeing things in the logs that do not make sense to me.

First, the "unpatched" version logs -- including the Raspian case --
all have the following lines:

    [    0.000000] Movable zone start for each node
    /* ... */
    [    0.000000] pcpu-alloc: s88232 r8192 d30552 u126976 alloc=31*4096
    [    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3

The above lines are also in my logs.  But they are missing from your
"after patch" logs -- what did you  change to have these lines
disappear on the patched logs?

Second, you say that you used a different "CM4" from the one used in
the Bugzilla  report -- what do you mean by that?   Are you referring
to the CM4 module proper, whose only change was going from 4GB to 8GB,
or the IO board being used?  My  testing is done with a standard RPi
CM4 and standard RPi IO board.  Before this patch series, the only way
this standard configuration can work for most hobbyist PCI cards (i.e.
floating CLKREQ# pin) is by using Raspian AND adding
"brcm,enable-l1ss" to the DT node.

I'm guessing that you are using the configuration that you described
to me in  a personal email: "[the] chip is embedded directly on the
modified PCB, as opposed to plugged into the PCIe slot on the official
CM4 IO Board".  If true, you are testing on a configuration that (a)
is unique to you and your group and (b) must be doing something with
the CLKREQ# signal so that your "before" case does not panic.  Is this
the case?

Finally, and this is a big one, is the fact that in both of the
"before" and "after" cases the value written to the internal Brcm
CLKREQ# register is the same.
This is evident by this line in the "after" patch:  "brcm-pcie
fd500000.pcie: uni-dir CLKREQ# for ASPM".  This mode is the only mode
supported by the "before"
case.  Now there are  some other things going on in the patch series
-- reading two registers and extending the timeout value of a
completion abort, but I'm having
a hard time imagining how they could cause a panic.

Regards,
Jim

PS  Can you please include in your logs the output of  "sudo lspci
-vvv" for those cases that boot to prompt?
>
> I fear that initially I might have not waited enough before power off and
> power on when replugging the SupaHub adapter, and I've only seen the PCIe
> link down a few times (now that I'm actively paying attention to that
> part). I'm indeed seeing PCIe link up consistently (100%) when using the
> following method:
>
>     for i in $(seq 1 20); do
>       # power on, let the system boot up and settle:
>       sispmctl -t 4; sleep 2m
>       ssh cm4 sh -c "dmesg > dmesg-$i.txt; poweroff"
>       # let the system power down, and power off:
>       sleep 30; sispmctl -t 4
>       # wait before power cycling:
>       sleep 10
>     done
>
> > Looking at:
> > https://www.amazon.co.uk/SupaHub-Express-BandWidth-Capable-Expanding/dp/B092ZQWG5B
> >
> > it would appear that it can accept an external power supply, do you have one
> > connected to that USB expansion card by any chance?
>
> With the patched kernel, I have tried several things (leaving the regular
> 12V adapter plugged in all the time):
>  - No external power supply (that's what I've been using in all previous
>    attempts).
>  - Using a 5V PSU with 2 pins (ground and 5V) plugged on the Ext PSU
>    4-pin header on the IO Board.
>  - Using a 5V PSU with a SATA connector plugged directly onto the SupaHub
>    adapter.
>
> I'm getting the same trace in all cases.
>
> > Are you able to boot the kernel before/after if you disconnect any USB
> > peripheral?
>
> The trace is obtained without any USB peripheral (on the extension card or
> on the onboard USB slots). Besides the SupaHub adapter on the PCIe slot, I
> only have Ethernet and HDMI plugged in (I'm getting traces via serial
> console, and operating the system over SSH when it boots fine).
>
> As soon as I unplug the SupaHub board itself, the system boots fine.
>
> > Does that SupaHub board plugged to the CM4 1.0 system work fine in the
> > Raspberry Pi kernel tree?
>
> With the Raspberry Pi OS (64-bit) > Raspberry Pi OS Lite image
> (2023-02-21-raspios-bullseye-arm64-lite.img.xz), the system at least
> boots fine; I haven't tried adding devices to the mix just yet, trying
> to stick with that particular setup.
>
> A full dmesg is attached (dmesg-raspios.txt).
>
>
> Cheers,
> --
> Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
> D-I release manager -- Release team member -- Freelance Consultant

Re: [PATCH v2 2/3] PCI: brcmstb: CLKREQ# accomodations of downstream device

[Cyril Brulebois]

[-- Attachment #1: Type: text/plain, Size: 2561 bytes --]

Hi Jim,

It might take a few days before getting back to you regarding the
various questions you asked. To be on the safe side, I'll probably run
a cold boot for each setup a number of times (e.g. 10), so that any
possible outlier can be spotted/rejected. And maybe share results off
list once we have a better understanding of what's going on. Does that
make sense to you?

I'll cover a particular topic right away though.

Jim Quinlan <jim2101024@gmail.com> (2023-04-19):
> Second, you say that you used a different "CM4" from the one used in
> the Bugzilla  report -- what do you mean by that?   Are you referring
> to the CM4 module proper, whose only change was going from 4GB to 8GB,
> or the IO board being used?  My  testing is done with a standard RPi
> CM4 and standard RPi IO board.  Before this patch series, the only way
> this standard configuration can work for most hobbyist PCI cards (i.e.
> floating CLKREQ# pin) is by using Raspian AND adding
> "brcm,enable-l1ss" to the DT node.

Regarding the IO Board, I'm using the official Compute Module 4 IO
Board: https://www.raspberrypi.com/products/compute-module-4-io-board/

I've been using the very same IO Board for all my testing, and what I'm
changing is the standard RPi CM4 plugged on it.

> I'm guessing that you are using the configuration that you described
> to me in  a personal email: "[the] chip is embedded directly on the
> modified PCB, as opposed to plugged into the PCIe slot on the official
> CM4 IO Board".  If true, you are testing on a configuration that (a)
> is unique to you and your group and (b) must be doing something with
> the CLKREQ# signal so that your "before" case does not panic.  Is this
> the case?

That's definitely not the case.

True, as mentioned in a personal mail, we've seen problems with a custom
PCB, derived from the CM4 IO Board design, but of course there could be
some faulty design at work there… So we've first researched whether the
same problem could be produced with consumer grade products, and once
we've verified that, I opened #217276 on Bugzilla.


Since Florian's testing seems overwhelmingly positive, and since I'm
seeing definitive improvement with at least one CM4, maybe it would make
sense not to block the patch series on the kernel panic I'm seeing with
the other CM4, and track that particular issue via a separate bug?


Cheers,
-- 
Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/3] dt-bindings: PCI: brcmstb: Add two optional props

[Konstantin Ryabitsev]

April 18, 2023 2:35 PM, "Rob Herring" <robh@kernel.org> wrote:
>> Some trees like the networking
>> tree do merge commits of patch sets where the cover letter is used as part
>> of the merge commit message. Other maintainers don't, and some want the
>> change log after the '---' and some do not.
> 
> I'm not aware of anyone except for DRM wanting the changelog in the
> final commits, but that's really a different issue.

I don't think anyone wants changelogs in actual final commits, they usually go under "---" in patch submissions.

> I'm pretty sure no one will complain about a changelog in the patches. I
> guess you just have to duplicate it if you think it should be in both.
> b4 could be taught to do that I suppose. IMO, the cover letter should
> have a higher level changelog than the individual patches.

b4 doesn't really need to manage per-patch changelogs -- they should just go under "---" in the commit. When you send the series either via "b4 send" or via git-send-email, the changelogs will be properly included in the message, but they won't make it into the tree after the maintainer runs "git am", because git will drop anything under the first "---" in the commit message.

The cover letter changelog is supposed to be higher level than individual patch changelogs, so I don't think it makes sense for b4 to collect them from individual patches.

-K