[PATCH] PCI: pciehp: Simplify Command Completion checking

[PATCH] PCI: pciehp: Simplify Command Completion checking

[Bjorn Helgaas]

From: Bjorn Helgaas <bhelgaas@google.com>

If a device sets the No Command Completed Support bit in the Slot
Capabilities register (PCIe r6.0, sec 7.5.3.9), it does not generate
software notifications when a command is completed, and software can
write all fields of the Slot Control register without any delays.

Since we only need to wait for command completion on devices that do *not*
set the No Command Completed Support bit, there's no need to even set the
ctrl->cmd_busy bit that tracks when the device is busy handling a command.

Set the ctrl->cmd_busy bit only when we need to wait for a command to
complete.  This shouldn't make much functional difference, but does avoid
an smp_mb() for controllers that set PCI_EXP_SLTCAP_NCCS.

Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
---
 drivers/pci/hotplug/pciehp_hpc.c | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)

diff --git a/drivers/pci/hotplug/pciehp_hpc.c b/drivers/pci/hotplug/pciehp_hpc.c
index 1c1ebf3dad43..0ff928693a13 100644
--- a/drivers/pci/hotplug/pciehp_hpc.c
+++ b/drivers/pci/hotplug/pciehp_hpc.c
@@ -114,13 +114,6 @@ static void pcie_wait_cmd(struct controller *ctrl)
 	unsigned long now, timeout;
 	int rc;
 
-	/*
-	 * If the controller does not generate notifications for command
-	 * completions, we never need to wait between writes.
-	 */
-	if (NO_CMD_CMPL(ctrl))
-		return;
-
 	if (!ctrl->cmd_busy)
 		return;
 
@@ -173,8 +166,17 @@ static void pcie_do_write_cmd(struct controller *ctrl, u16 cmd,
 	slot_ctrl_orig = slot_ctrl;
 	slot_ctrl &= ~mask;
 	slot_ctrl |= (cmd & mask);
-	ctrl->cmd_busy = 1;
-	smp_mb();
+
+	/*
+	 * If controller generates Command Completed events, we must wait
+	 * for it to finish one command before sending another, so we need
+	 * to keep track of when the controller is busy.
+	 */
+	if (!NO_CMD_CMPL(ctrl)) {
+		ctrl->cmd_busy = 1;
+		smp_mb();
+	}
+
 	ctrl->slot_ctrl = slot_ctrl;
 	pcie_capability_write_word(pdev, PCI_EXP_SLTCTL, slot_ctrl);
 	ctrl->cmd_started = jiffies;
-- 
2.25.1

Re: [PATCH] PCI: pciehp: Simplify Command Completion checking

[Lukas Wunner]

On Thu, Feb 10, 2022 at 05:38:06PM -0600, Bjorn Helgaas wrote:
> If a device sets the No Command Completed Support bit in the Slot
> Capabilities register (PCIe r6.0, sec 7.5.3.9), it does not generate
> software notifications when a command is completed, and software can
> write all fields of the Slot Control register without any delays.
> 
> Since we only need to wait for command completion on devices that do *not*
> set the No Command Completed Support bit, there's no need to even set the
> ctrl->cmd_busy bit that tracks when the device is busy handling a command.
> 
> Set the ctrl->cmd_busy bit only when we need to wait for a command to
> complete.  This shouldn't make much functional difference, but does avoid
> an smp_mb() for controllers that set PCI_EXP_SLTCAP_NCCS.
[...]
> --- a/drivers/pci/hotplug/pciehp_hpc.c
> +++ b/drivers/pci/hotplug/pciehp_hpc.c
> @@ -114,13 +114,6 @@ static void pcie_wait_cmd(struct controller *ctrl)
>  	unsigned long now, timeout;
>  	int rc;
>  
> -	/*
> -	 * If the controller does not generate notifications for command
> -	 * completions, we never need to wait between writes.
> -	 */
> -	if (NO_CMD_CMPL(ctrl))
> -		return;
> -
>  	if (!ctrl->cmd_busy)
>  		return;
>  
> @@ -173,8 +166,17 @@ static void pcie_do_write_cmd(struct controller *ctrl, u16 cmd,
>  	slot_ctrl_orig = slot_ctrl;
>  	slot_ctrl &= ~mask;
>  	slot_ctrl |= (cmd & mask);
> -	ctrl->cmd_busy = 1;
> -	smp_mb();
> +
> +	/*
> +	 * If controller generates Command Completed events, we must wait
> +	 * for it to finish one command before sending another, so we need
> +	 * to keep track of when the controller is busy.
> +	 */
> +	if (!NO_CMD_CMPL(ctrl)) {
> +		ctrl->cmd_busy = 1;
> +		smp_mb();
> +	}
> +
>  	ctrl->slot_ctrl = slot_ctrl;
>  	pcie_capability_write_word(pdev, PCI_EXP_SLTCTL, slot_ctrl);
>  	ctrl->cmd_started = jiffies;

Looks good in principle, however ctrl->cmd_busy is set to 1 in two
other places, namely pciehp_resume_noirq() and pciehp_runtime_resume()
in pciehp_core.c.  Those two need to be amended as well.

Note that a few lines after the hunk above, the ctrl->cmd_busy flag
is reverted back to 0 on certain controllers:

	if (pdev->broken_cmd_compl &&
	    (slot_ctrl_orig & CC_ERRATUM_MASK) == (slot_ctrl & CC_ERRATUM_MASK))
		ctrl->cmd_busy = 0;

That could be combined with your "if (!NO_CMD_CMPL(ctrl))" check:
Basically, "set cmd_busy to 1 if the controller supports Command
Completed events and it's not affected by this erratum".

Perhaps it makes sense to put that in a pciehp_has_command_completion()
helper (or whatever name you prefer), which could then also be called
from pciehp_resume_noirq() / pciehp_runtime_resume().

Thanks (and sorry for the delay),

Lukas