From: Bjorn Helgaas <helgaas@kernel.org>
To: Vidya Sagar <vidyas@nvidia.com>,
Kuppuswamy Sathyanarayanan
<sathyanarayanan.kuppuswamy@linux.intel.com>,
"David E. Box" <david.e.box@linux.intel.com>
Cc: bhelgaas@google.com, macro@orcam.me.uk, ajayagarwal@google.com,
ilpo.jarvinen@linux.intel.com, hkallweit1@gmail.com,
johan+linaro@kernel.org, xueshuai@linux.alibaba.com,
linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org,
treding@nvidia.com, jonathanh@nvidia.com, kthota@nvidia.com,
mmaddireddy@nvidia.com, sagar.tv@gmail.com
Subject: Re: [PATCH V4] PCI/ASPM: Update saved buffers with latest ASPM
Date: Thu, 7 Mar 2024 16:01:19 -0600 [thread overview]
Message-ID: <20240307220119.GA652393@bhelgaas> (raw)
In-Reply-To: <20240305220342.GA552530@bhelgaas>
On Tue, Mar 05, 2024 at 04:03:42PM -0600, Bjorn Helgaas wrote:
> [+to Sathy, David in case you want to update your Reviewed-by]
>
> On Thu, Feb 22, 2024 at 11:14:36PM +0530, Vidya Sagar wrote:
> > Many PCIe device drivers save the configuration state of their respective
> > devices during probe and restore the same when their 'slot_reset' hook
> > is called through PCIe Error Recovery Handler.
> >
> > If the system has a change in ASPM policy after the driver's probe is
> > called and before error event occurred, 'slot_reset' hook restores the
> > PCIe configuration state to what it was at the time of probe but not to
> > what it was just before the occurrence of the error event.
> > This effectively leads to a mismatch in the ASPM configuration between
> > the device and its upstream parent device.
> >
> > Update the saved configuration state of the device with the latest info
> > whenever there is a change w.r.t ASPM policy.
> >
> > Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
>
> > -void pci_save_aspm_state(struct pci_dev *pdev);
> > +void pci_save_aspm_l1ss_state(struct pci_dev *pdev);
>
> I rebased this again on top of my pci/aspm updates to remove the need
> for the rename above.
>
> > +static void pci_save_aspm_state(struct pci_dev *dev)
> > +{
> > + struct pci_cap_saved_state *save_state;
> > + u16 *cap;
> > +
> > + if (!pci_is_pcie(dev))
> > + return;
> > +
> > + save_state = pci_find_saved_cap(dev, PCI_CAP_ID_EXP);
> > + if (!save_state)
> > + return;
> > +
> > + cap = (u16 *)&save_state->cap.data[0];
> > + pcie_capability_read_word(dev, PCI_EXP_LNKCTL, &cap[1]);
>
> And I changed this part so it only updates the PCI_EXP_LNKCTL_ASPMC
> bits, not the entire LNKCTL.
>
> Updating the entire saved register probably wouldn't *break* anything,
> but it could randomly hide other LNKCTL changes depending on whether
> or not ASPM configuration was changed in the interim. For example:
>
> - driver .probe() saves LNKCTL
> - LNKCTL changes some non-ASPMC thing via setpci or other mechanism
> - save_state updated via pcie_config_aspm_link()
>
> A restore in .slot_reset() would restore different LNKCTL values for
> the non-ASPMC change depending on whether pcie_config_aspm_link() was
> used.
Oops, I blew it here. I think it's good to limit the LNKCTL changes
we put in save_state, but PCI_EXP_LNKCTL_ASPMC is not enough. We
should include PCI_EXP_LNKCTL_CLKREQ_EN as well since that may be
updated in many of the same paths that update PCI_EXP_LNKCTL_ASPMC.
I updated it to the patch at the bottom; here's the interdiff first:
-------------------------------------------------------------------------
diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c
index 46352132bb14..10160d82c10a 100644
--- a/drivers/pci/pcie/aspm.c
+++ b/drivers/pci/pcie/aspm.c
@@ -304,18 +304,25 @@ static int policy_to_clkpm_state(struct pcie_link_state *link)
static void pci_update_aspm_saved_state(struct pci_dev *dev)
{
struct pci_cap_saved_state *save_state;
- u16 *cap, lnkctl, aspmc;
+ u16 *cap, lnkctl, aspm_ctl;
save_state = pci_find_saved_cap(dev, PCI_CAP_ID_EXP);
if (!save_state)
return;
+ pcie_capability_read_word(dev, PCI_EXP_LNKCTL, &lnkctl);
+
+ /*
+ * Update ASPM and CLKREQ bits of LNKCTL in save_state. We only
+ * write PCI_EXP_LNKCTL_CCC during enumeration, so it shouldn't
+ * change after being captured in save_state.
+ */
+ aspm_ctl = lnkctl & (PCI_EXP_LNKCTL_ASPMC | PCI_EXP_LNKCTL_CLKREQ_EN);
+ lnkctl &= ~(PCI_EXP_LNKCTL_ASPMC | PCI_EXP_LNKCTL_CLKREQ_EN);
+
/* Depends on pci_save_pcie_state(): cap[1] is LNKCTL */
cap = (u16 *)&save_state->cap.data[0];
- pcie_capability_read_word(dev, PCI_EXP_LNKCTL, &lnkctl);
- aspmc = FIELD_GET(PCI_EXP_LNKCTL_ASPMC, lnkctl);
- cap[1] = (lnkctl & ~PCI_EXP_LNKCTL_ASPMC) |
- FIELD_PREP(PCI_EXP_LNKCTL_ASPMC, aspmc);
+ cap[1] = lnkctl | aspm_ctl;
}
static void pcie_set_clkpm_nocheck(struct pcie_link_state *link, int enable)
-------------------------------------------------------------------------
And here's the entire patch:
commit ff92b9348534 ("PCI/ASPM: Update save_state when configuration changes")
Author: Vidya Sagar <vidyas@nvidia.com>
Date: Fri Feb 23 13:36:24 2024 -0600
PCI/ASPM: Update save_state when configuration changes
Many PCIe device drivers save the configuration state of their device
during probe and restore it when their .slot_reset() hook is called during
PCIe error recovery.
If the ASPM configuration is changed after the driver's probe is called and
before an error event occurs, .slot_reset() restores the ASPM configuration
to what it was at the time of probe, not to what it was just before the
occurrence of the error event. This leads to a mismatch in ASPM
configuration between the device and its upstream device.
Update the saved configuration of the device when the ASPM configuration
changes.
Link: https://lore.kernel.org/r/20240222174436.3565146-1-vidyas@nvidia.com
Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
[bhelgaas: commit log, rebase to pci/aspm, rename to
pci_update_aspm_saved_state() since it updates only LNKCTL, update only
ASPMC and CLKREQ_EN in LNKCTL]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
Reviewed-by: David E. Box <david.e.box@linux.intel.com>
diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c
index 1379b8decdf1..10160d82c10a 100644
--- a/drivers/pci/pcie/aspm.c
+++ b/drivers/pci/pcie/aspm.c
@@ -301,16 +301,42 @@ static int policy_to_clkpm_state(struct pcie_link_state *link)
return 0;
}
+static void pci_update_aspm_saved_state(struct pci_dev *dev)
+{
+ struct pci_cap_saved_state *save_state;
+ u16 *cap, lnkctl, aspm_ctl;
+
+ save_state = pci_find_saved_cap(dev, PCI_CAP_ID_EXP);
+ if (!save_state)
+ return;
+
+ pcie_capability_read_word(dev, PCI_EXP_LNKCTL, &lnkctl);
+
+ /*
+ * Update ASPM and CLKREQ bits of LNKCTL in save_state. We only
+ * write PCI_EXP_LNKCTL_CCC during enumeration, so it shouldn't
+ * change after being captured in save_state.
+ */
+ aspm_ctl = lnkctl & (PCI_EXP_LNKCTL_ASPMC | PCI_EXP_LNKCTL_CLKREQ_EN);
+ lnkctl &= ~(PCI_EXP_LNKCTL_ASPMC | PCI_EXP_LNKCTL_CLKREQ_EN);
+
+ /* Depends on pci_save_pcie_state(): cap[1] is LNKCTL */
+ cap = (u16 *)&save_state->cap.data[0];
+ cap[1] = lnkctl | aspm_ctl;
+}
+
static void pcie_set_clkpm_nocheck(struct pcie_link_state *link, int enable)
{
struct pci_dev *child;
struct pci_bus *linkbus = link->pdev->subordinate;
u32 val = enable ? PCI_EXP_LNKCTL_CLKREQ_EN : 0;
- list_for_each_entry(child, &linkbus->devices, bus_list)
+ list_for_each_entry(child, &linkbus->devices, bus_list) {
pcie_capability_clear_and_set_word(child, PCI_EXP_LNKCTL,
PCI_EXP_LNKCTL_CLKREQ_EN,
val);
+ pci_update_aspm_saved_state(child);
+ }
link->clkpm_enabled = !!enable;
}
@@ -929,6 +955,12 @@ static void pcie_config_aspm_link(struct pcie_link_state *link, u32 state)
pcie_config_aspm_dev(parent, upstream);
link->aspm_enabled = state;
+
+ /* Update latest ASPM configuration in saved context */
+ pci_save_aspm_l1ss_state(link->downstream);
+ pci_update_aspm_saved_state(link->downstream);
+ pci_save_aspm_l1ss_state(parent);
+ pci_update_aspm_saved_state(parent);
}
static void pcie_config_aspm_path(struct pcie_link_state *link)
prev parent reply other threads:[~2024-03-07 22:01 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-01-25 13:38 [PATCH V1] PCI/ASPM: Update saved buffers with latest ASPM configuration Vidya Sagar
2023-01-25 15:01 ` Wysocki, Rafael J
2023-01-25 17:22 ` Vidya Sagar
2023-01-26 11:43 ` Wysocki, Rafael J
2023-01-26 21:22 ` Sathyanarayanan Kuppuswamy
2023-01-27 20:10 ` Bjorn Helgaas
2023-01-28 18:26 ` Bjorn Helgaas
2024-01-03 10:35 ` [PATCH V2] PCI/ASPM: Update saved buffers with latest ASPM Vidya Sagar
2024-01-03 15:37 ` Kuppuswamy Sathyanarayanan
2024-01-08 12:42 ` [PATCH V3] " Vidya Sagar
2024-02-20 4:07 ` Vidya Sagar
2024-02-20 17:23 ` Bjorn Helgaas
2024-02-22 17:44 ` [PATCH V4] " Vidya Sagar
2024-02-22 18:20 ` Kuppuswamy Sathyanarayanan
2024-02-22 18:54 ` Bjorn Helgaas
2024-02-22 19:54 ` Kuppuswamy Sathyanarayanan
2024-02-22 22:14 ` David E. Box
2024-02-22 20:59 ` Vidya Sagar
2024-03-05 22:03 ` Bjorn Helgaas
2024-03-05 22:35 ` Kuppuswamy Sathyanarayanan
2024-03-06 1:37 ` David E. Box
2024-03-07 22:01 ` Bjorn Helgaas [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240307220119.GA652393@bhelgaas \
--to=helgaas@kernel.org \
--cc=ajayagarwal@google.com \
--cc=bhelgaas@google.com \
--cc=david.e.box@linux.intel.com \
--cc=hkallweit1@gmail.com \
--cc=ilpo.jarvinen@linux.intel.com \
--cc=johan+linaro@kernel.org \
--cc=jonathanh@nvidia.com \
--cc=kthota@nvidia.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=macro@orcam.me.uk \
--cc=mmaddireddy@nvidia.com \
--cc=sagar.tv@gmail.com \
--cc=sathyanarayanan.kuppuswamy@linux.intel.com \
--cc=treding@nvidia.com \
--cc=vidyas@nvidia.com \
--cc=xueshuai@linux.alibaba.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).