* [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT @ 2017-04-08 4:55 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-08 4:55 UTC (permalink / raw) To: linux-pci, timur Cc: mayurkumar.patel, linux-arm-msm, linux-arm-kernel, Sinan Kaya When the operating system is booted with the default ASPM policy (POLICY_DEFAULT), current code is querying the enable/disable states from ASPM registers to determine the policy. For example, a BIOS could set the power saving state to performance and clear all ASPM control registers. A balanced ASPM policy could enable L0s and disable L1. A power conscious BIOS could enable both L0s and L1 to trade off latency and performance vs. power. After hotplug removal, pcie_aspm_exit_link_state() function clears the ASPM registers. An insertion following hotplug removal reads incorrect policy as ASPM disabled even though ASPM was enabled during boot. This is caused by the fact that same function is used for reconfiguring ASPM regardless of the power on state. ------------------------ Changes from v7 (https://www.spinics.net/lists/arm-kernel/msg573034.html) ------------------------ - enable LTSSM as a best effort if the endpoint is not connected during boot Sinan Kaya (5): PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() PCI/ASPM: split pci_aspm_init() into two PCI/ASPM: add init hook to device_add PCI/ASPM: save power on values during bridge init PCI/ASPM: move link_state cleanup to bridge remove drivers/pci/pcie/aspm.c | 149 +++++++++++++++++++++++++++++++++--------------- drivers/pci/probe.c | 3 + drivers/pci/remove.c | 3 +- include/linux/pci.h | 2 + 4 files changed, 110 insertions(+), 47 deletions(-) -- 1.9.1 ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT @ 2017-04-08 4:55 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-08 4:55 UTC (permalink / raw) To: linux-arm-kernel When the operating system is booted with the default ASPM policy (POLICY_DEFAULT), current code is querying the enable/disable states from ASPM registers to determine the policy. For example, a BIOS could set the power saving state to performance and clear all ASPM control registers. A balanced ASPM policy could enable L0s and disable L1. A power conscious BIOS could enable both L0s and L1 to trade off latency and performance vs. power. After hotplug removal, pcie_aspm_exit_link_state() function clears the ASPM registers. An insertion following hotplug removal reads incorrect policy as ASPM disabled even though ASPM was enabled during boot. This is caused by the fact that same function is used for reconfiguring ASPM regardless of the power on state. ------------------------ Changes from v7 (https://www.spinics.net/lists/arm-kernel/msg573034.html) ------------------------ - enable LTSSM as a best effort if the endpoint is not connected during boot Sinan Kaya (5): PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() PCI/ASPM: split pci_aspm_init() into two PCI/ASPM: add init hook to device_add PCI/ASPM: save power on values during bridge init PCI/ASPM: move link_state cleanup to bridge remove drivers/pci/pcie/aspm.c | 149 +++++++++++++++++++++++++++++++++--------------- drivers/pci/probe.c | 3 + drivers/pci/remove.c | 3 +- include/linux/pci.h | 2 + 4 files changed, 110 insertions(+), 47 deletions(-) -- 1.9.1 ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT @ 2017-04-08 4:55 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-08 4:55 UTC (permalink / raw) To: linux-pci, timur Cc: Sinan Kaya, linux-arm-msm, mayurkumar.patel, linux-arm-kernel When the operating system is booted with the default ASPM policy (POLICY_DEFAULT), current code is querying the enable/disable states from ASPM registers to determine the policy. For example, a BIOS could set the power saving state to performance and clear all ASPM control registers. A balanced ASPM policy could enable L0s and disable L1. A power conscious BIOS could enable both L0s and L1 to trade off latency and performance vs. power. After hotplug removal, pcie_aspm_exit_link_state() function clears the ASPM registers. An insertion following hotplug removal reads incorrect policy as ASPM disabled even though ASPM was enabled during boot. This is caused by the fact that same function is used for reconfiguring ASPM regardless of the power on state. ------------------------ Changes from v7 (https://www.spinics.net/lists/arm-kernel/msg573034.html) ------------------------ - enable LTSSM as a best effort if the endpoint is not connected during boot Sinan Kaya (5): PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() PCI/ASPM: split pci_aspm_init() into two PCI/ASPM: add init hook to device_add PCI/ASPM: save power on values during bridge init PCI/ASPM: move link_state cleanup to bridge remove drivers/pci/pcie/aspm.c | 149 +++++++++++++++++++++++++++++++++--------------- drivers/pci/probe.c | 3 + drivers/pci/remove.c | 3 +- include/linux/pci.h | 2 + 4 files changed, 110 insertions(+), 47 deletions(-) -- 1.9.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 1/5] PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() 2017-04-08 4:55 ` Sinan Kaya (?) @ 2017-04-08 4:55 ` Sinan Kaya -1 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-08 4:55 UTC (permalink / raw) To: linux-pci, timur Cc: mayurkumar.patel, linux-arm-msm, linux-arm-kernel, Sinan Kaya, Bjorn Helgaas, Rajat Jain, Julia Lawall, David Daney, Shawn Lin, linux-kernel We need a callback from pci_init_capabilities function for every single new PCI device that is currently being added. pci_aspm_init() will be used to save the power on state of the HW. Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 Signed-off-by: Sinan Kaya <okaya@codeaurora.org> --- drivers/pci/pcie/aspm.c | 10 ++++++++++ drivers/pci/probe.c | 3 +++ include/linux/pci.h | 2 ++ 3 files changed, 15 insertions(+) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index 1dfa10c..dc36717 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -827,6 +827,16 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) } /* + * pci_aspm_init: Initiate PCI express link state. + * It is called from device_add for every single pci device. + * @pdev: all pci devices + */ +int pci_aspm_init(struct pci_dev *pdev) +{ + return 0; +} + +/* * pcie_aspm_init_link_state: Initiate PCI express link state. * It is called after the pcie and its children devices are scanned. * @pdev: the root port or switch downstream port diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index dfc9a27..1e19364 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -1847,6 +1847,9 @@ static void pci_init_capabilities(struct pci_dev *dev) /* Advanced Error Reporting */ pci_aer_init(dev); + + /* Active State Power Management */ + pci_aspm_init(dev); } /* diff --git a/include/linux/pci.h b/include/linux/pci.h index eb3da1a..8828dd7 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -1396,8 +1396,10 @@ static inline int pci_irq_get_node(struct pci_dev *pdev, int vec) #ifdef CONFIG_PCIEASPM bool pcie_aspm_support_enabled(void); +int pci_aspm_init(struct pci_dev *pdev); #else static inline bool pcie_aspm_support_enabled(void) { return false; } +static inline int pci_aspm_init(struct pci_dev *pdev) { return -ENODEV; } #endif #ifdef CONFIG_PCIEAER -- 1.9.1 ^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH V8 1/5] PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() @ 2017-04-08 4:55 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-08 4:55 UTC (permalink / raw) To: linux-arm-kernel We need a callback from pci_init_capabilities function for every single new PCI device that is currently being added. pci_aspm_init() will be used to save the power on state of the HW. Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 Signed-off-by: Sinan Kaya <okaya@codeaurora.org> --- drivers/pci/pcie/aspm.c | 10 ++++++++++ drivers/pci/probe.c | 3 +++ include/linux/pci.h | 2 ++ 3 files changed, 15 insertions(+) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index 1dfa10c..dc36717 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -827,6 +827,16 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) } /* + * pci_aspm_init: Initiate PCI express link state. + * It is called from device_add for every single pci device. + * @pdev: all pci devices + */ +int pci_aspm_init(struct pci_dev *pdev) +{ + return 0; +} + +/* * pcie_aspm_init_link_state: Initiate PCI express link state. * It is called after the pcie and its children devices are scanned. * @pdev: the root port or switch downstream port diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index dfc9a27..1e19364 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -1847,6 +1847,9 @@ static void pci_init_capabilities(struct pci_dev *dev) /* Advanced Error Reporting */ pci_aer_init(dev); + + /* Active State Power Management */ + pci_aspm_init(dev); } /* diff --git a/include/linux/pci.h b/include/linux/pci.h index eb3da1a..8828dd7 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -1396,8 +1396,10 @@ static inline int pci_irq_get_node(struct pci_dev *pdev, int vec) #ifdef CONFIG_PCIEASPM bool pcie_aspm_support_enabled(void); +int pci_aspm_init(struct pci_dev *pdev); #else static inline bool pcie_aspm_support_enabled(void) { return false; } +static inline int pci_aspm_init(struct pci_dev *pdev) { return -ENODEV; } #endif #ifdef CONFIG_PCIEAER -- 1.9.1 ^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH V8 1/5] PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() @ 2017-04-08 4:55 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-08 4:55 UTC (permalink / raw) To: linux-pci, timur Cc: mayurkumar.patel, David Daney, linux-arm-msm, Shawn Lin, linux-kernel, Sinan Kaya, Julia Lawall, Bjorn Helgaas, Rajat Jain, linux-arm-kernel We need a callback from pci_init_capabilities function for every single new PCI device that is currently being added. pci_aspm_init() will be used to save the power on state of the HW. Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 Signed-off-by: Sinan Kaya <okaya@codeaurora.org> --- drivers/pci/pcie/aspm.c | 10 ++++++++++ drivers/pci/probe.c | 3 +++ include/linux/pci.h | 2 ++ 3 files changed, 15 insertions(+) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index 1dfa10c..dc36717 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -827,6 +827,16 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) } /* + * pci_aspm_init: Initiate PCI express link state. + * It is called from device_add for every single pci device. + * @pdev: all pci devices + */ +int pci_aspm_init(struct pci_dev *pdev) +{ + return 0; +} + +/* * pcie_aspm_init_link_state: Initiate PCI express link state. * It is called after the pcie and its children devices are scanned. * @pdev: the root port or switch downstream port diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index dfc9a27..1e19364 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -1847,6 +1847,9 @@ static void pci_init_capabilities(struct pci_dev *dev) /* Advanced Error Reporting */ pci_aer_init(dev); + + /* Active State Power Management */ + pci_aspm_init(dev); } /* diff --git a/include/linux/pci.h b/include/linux/pci.h index eb3da1a..8828dd7 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -1396,8 +1396,10 @@ static inline int pci_irq_get_node(struct pci_dev *pdev, int vec) #ifdef CONFIG_PCIEASPM bool pcie_aspm_support_enabled(void); +int pci_aspm_init(struct pci_dev *pdev); #else static inline bool pcie_aspm_support_enabled(void) { return false; } +static inline int pci_aspm_init(struct pci_dev *pdev) { return -ENODEV; } #endif #ifdef CONFIG_PCIEAER -- 1.9.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 94+ messages in thread
* Re: [PATCH V8 1/5] PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() 2017-04-08 4:55 ` Sinan Kaya @ 2017-04-13 20:51 ` Bjorn Helgaas -1 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-13 20:51 UTC (permalink / raw) To: Sinan Kaya Cc: linux-pci, timur, mayurkumar.patel, David Daney, linux-arm-msm, Shawn Lin, linux-kernel, Julia Lawall, Bjorn Helgaas, Rajat Jain, linux-arm-kernel Hi Sinan, On Sat, Apr 08, 2017 at 12:55:47AM -0400, Sinan Kaya wrote: > We need a callback from pci_init_capabilities function for every > single new PCI device that is currently being added. > > pci_aspm_init() will be used to save the power on state of the HW. > > Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 > Signed-off-by: Sinan Kaya <okaya@codeaurora.org> > --- > drivers/pci/pcie/aspm.c | 10 ++++++++++ > drivers/pci/probe.c | 3 +++ > include/linux/pci.h | 2 ++ > 3 files changed, 15 insertions(+) > > diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c > index 1dfa10c..dc36717 100644 > --- a/drivers/pci/pcie/aspm.c > +++ b/drivers/pci/pcie/aspm.c > @@ -827,6 +827,16 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) > } > > /* > + * pci_aspm_init: Initiate PCI express link state. > + * It is called from device_add for every single pci device. > + * @pdev: all pci devices > + */ > +int pci_aspm_init(struct pci_dev *pdev) > +{ > + return 0; > +} > + > +/* > * pcie_aspm_init_link_state: Initiate PCI express link state. > * It is called after the pcie and its children devices are scanned. > * @pdev: the root port or switch downstream port > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c > index dfc9a27..1e19364 100644 > --- a/drivers/pci/probe.c > +++ b/drivers/pci/probe.c > @@ -1847,6 +1847,9 @@ static void pci_init_capabilities(struct pci_dev *dev) > > /* Advanced Error Reporting */ > pci_aer_init(dev); > + > + /* Active State Power Management */ > + pci_aspm_init(dev); > } > > /* > diff --git a/include/linux/pci.h b/include/linux/pci.h > index eb3da1a..8828dd7 100644 > --- a/include/linux/pci.h > +++ b/include/linux/pci.h > @@ -1396,8 +1396,10 @@ static inline int pci_irq_get_node(struct pci_dev *pdev, int vec) > > #ifdef CONFIG_PCIEASPM > bool pcie_aspm_support_enabled(void); > +int pci_aspm_init(struct pci_dev *pdev); I think these can go in drivers/pci/pci.h instead of the public include/linux/pci.h. There's no ASPM section in drivers/pci/pci.h yet, but I think we should add one. It looks like the following things from include/linux/pci-aspm.h could be moved there as well, since they're only used by the PCI core: void pcie_aspm_init_link_state(struct pci_dev *pdev); void pcie_aspm_exit_link_state(struct pci_dev *pdev); void pcie_aspm_pm_state_change(struct pci_dev *pdev); void pcie_aspm_powersave_config_link(struct pci_dev *pdev); void pcie_aspm_create_sysfs_dev_files(struct pci_dev *pdev); void pcie_aspm_remove_sysfs_dev_files(struct pci_dev *pdev); This would be a separate patch. > #else > static inline bool pcie_aspm_support_enabled(void) { return false; } > +static inline int pci_aspm_init(struct pci_dev *pdev) { return -ENODEV; } > #endif > > #ifdef CONFIG_PCIEAER > -- > 1.9.1 > > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 1/5] PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() @ 2017-04-13 20:51 ` Bjorn Helgaas 0 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-13 20:51 UTC (permalink / raw) To: linux-arm-kernel Hi Sinan, On Sat, Apr 08, 2017 at 12:55:47AM -0400, Sinan Kaya wrote: > We need a callback from pci_init_capabilities function for every > single new PCI device that is currently being added. > > pci_aspm_init() will be used to save the power on state of the HW. > > Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 > Signed-off-by: Sinan Kaya <okaya@codeaurora.org> > --- > drivers/pci/pcie/aspm.c | 10 ++++++++++ > drivers/pci/probe.c | 3 +++ > include/linux/pci.h | 2 ++ > 3 files changed, 15 insertions(+) > > diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c > index 1dfa10c..dc36717 100644 > --- a/drivers/pci/pcie/aspm.c > +++ b/drivers/pci/pcie/aspm.c > @@ -827,6 +827,16 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) > } > > /* > + * pci_aspm_init: Initiate PCI express link state. > + * It is called from device_add for every single pci device. > + * @pdev: all pci devices > + */ > +int pci_aspm_init(struct pci_dev *pdev) > +{ > + return 0; > +} > + > +/* > * pcie_aspm_init_link_state: Initiate PCI express link state. > * It is called after the pcie and its children devices are scanned. > * @pdev: the root port or switch downstream port > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c > index dfc9a27..1e19364 100644 > --- a/drivers/pci/probe.c > +++ b/drivers/pci/probe.c > @@ -1847,6 +1847,9 @@ static void pci_init_capabilities(struct pci_dev *dev) > > /* Advanced Error Reporting */ > pci_aer_init(dev); > + > + /* Active State Power Management */ > + pci_aspm_init(dev); > } > > /* > diff --git a/include/linux/pci.h b/include/linux/pci.h > index eb3da1a..8828dd7 100644 > --- a/include/linux/pci.h > +++ b/include/linux/pci.h > @@ -1396,8 +1396,10 @@ static inline int pci_irq_get_node(struct pci_dev *pdev, int vec) > > #ifdef CONFIG_PCIEASPM > bool pcie_aspm_support_enabled(void); > +int pci_aspm_init(struct pci_dev *pdev); I think these can go in drivers/pci/pci.h instead of the public include/linux/pci.h. There's no ASPM section in drivers/pci/pci.h yet, but I think we should add one. It looks like the following things from include/linux/pci-aspm.h could be moved there as well, since they're only used by the PCI core: void pcie_aspm_init_link_state(struct pci_dev *pdev); void pcie_aspm_exit_link_state(struct pci_dev *pdev); void pcie_aspm_pm_state_change(struct pci_dev *pdev); void pcie_aspm_powersave_config_link(struct pci_dev *pdev); void pcie_aspm_create_sysfs_dev_files(struct pci_dev *pdev); void pcie_aspm_remove_sysfs_dev_files(struct pci_dev *pdev); This would be a separate patch. > #else > static inline bool pcie_aspm_support_enabled(void) { return false; } > +static inline int pci_aspm_init(struct pci_dev *pdev) { return -ENODEV; } > #endif > > #ifdef CONFIG_PCIEAER > -- > 1.9.1 > > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel at lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 1/5] PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() 2017-04-13 20:51 ` Bjorn Helgaas @ 2017-04-14 19:10 ` Sinan Kaya -1 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-14 19:10 UTC (permalink / raw) To: Bjorn Helgaas Cc: linux-pci, timur, mayurkumar.patel, David Daney, linux-arm-msm, Shawn Lin, linux-kernel, Julia Lawall, Bjorn Helgaas, Rajat Jain, linux-arm-kernel On 4/13/2017 4:51 PM, Bjorn Helgaas wrote: > I think these can go in drivers/pci/pci.h instead of the public > include/linux/pci.h. There's no ASPM section in drivers/pci/pci.h > yet, but I think we should add one. > > It looks like the following things from include/linux/pci-aspm.h could > be moved there as well, since they're only used by the PCI core: > > void pcie_aspm_init_link_state(struct pci_dev *pdev); > void pcie_aspm_exit_link_state(struct pci_dev *pdev); > void pcie_aspm_pm_state_change(struct pci_dev *pdev); > void pcie_aspm_powersave_config_link(struct pci_dev *pdev); > void pcie_aspm_create_sysfs_dev_files(struct pci_dev *pdev); > void pcie_aspm_remove_sysfs_dev_files(struct pci_dev *pdev); > > This would be a separate patch. > Sure, let me work on this. >> #else >> static inline bool pcie_aspm_support_enabled(void) { return false; } >> +static inline int pci_aspm_init(struct pci_dev *pdev) { return -ENODEV; } >> #endif -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 1/5] PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() @ 2017-04-14 19:10 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-14 19:10 UTC (permalink / raw) To: linux-arm-kernel On 4/13/2017 4:51 PM, Bjorn Helgaas wrote: > I think these can go in drivers/pci/pci.h instead of the public > include/linux/pci.h. There's no ASPM section in drivers/pci/pci.h > yet, but I think we should add one. > > It looks like the following things from include/linux/pci-aspm.h could > be moved there as well, since they're only used by the PCI core: > > void pcie_aspm_init_link_state(struct pci_dev *pdev); > void pcie_aspm_exit_link_state(struct pci_dev *pdev); > void pcie_aspm_pm_state_change(struct pci_dev *pdev); > void pcie_aspm_powersave_config_link(struct pci_dev *pdev); > void pcie_aspm_create_sysfs_dev_files(struct pci_dev *pdev); > void pcie_aspm_remove_sysfs_dev_files(struct pci_dev *pdev); > > This would be a separate patch. > Sure, let me work on this. >> #else >> static inline bool pcie_aspm_support_enabled(void) { return false; } >> +static inline int pci_aspm_init(struct pci_dev *pdev) { return -ENODEV; } >> #endif -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 2/5] PCI/ASPM: split pci_aspm_init() into two 2017-04-08 4:55 ` Sinan Kaya (?) @ 2017-04-08 4:55 ` Sinan Kaya -1 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-08 4:55 UTC (permalink / raw) To: linux-pci, timur Cc: mayurkumar.patel, linux-arm-msm, linux-arm-kernel, Sinan Kaya, Bjorn Helgaas, Rajat Jain, Shawn Lin, Julia Lawall, David Daney, linux-kernel Split pci_aspm_init() body into pci_aspm_init_upstream() and pci_aspm_init_downstream() for bridge and endpoint specific code behavior. Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 Signed-off-by: Sinan Kaya <okaya@codeaurora.org> --- drivers/pci/pcie/aspm.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index dc36717..a80d64b 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -826,6 +826,16 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) return link; } +static int pci_aspm_init_downstream(struct pci_dev *pdev) +{ + return 0; +} + +static int pci_aspm_init_upstream(struct pci_dev *pdev) +{ + return 0; +} + /* * pci_aspm_init: Initiate PCI express link state. * It is called from device_add for every single pci device. @@ -833,7 +843,10 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) */ int pci_aspm_init(struct pci_dev *pdev) { - return 0; + if (!pdev->has_secondary_link) + return pci_aspm_init_downstream(pdev); + + return pci_aspm_init_upstream(pdev); } /* -- 1.9.1 ^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH V8 2/5] PCI/ASPM: split pci_aspm_init() into two @ 2017-04-08 4:55 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-08 4:55 UTC (permalink / raw) To: linux-arm-kernel Split pci_aspm_init() body into pci_aspm_init_upstream() and pci_aspm_init_downstream() for bridge and endpoint specific code behavior. Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 Signed-off-by: Sinan Kaya <okaya@codeaurora.org> --- drivers/pci/pcie/aspm.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index dc36717..a80d64b 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -826,6 +826,16 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) return link; } +static int pci_aspm_init_downstream(struct pci_dev *pdev) +{ + return 0; +} + +static int pci_aspm_init_upstream(struct pci_dev *pdev) +{ + return 0; +} + /* * pci_aspm_init: Initiate PCI express link state. * It is called from device_add for every single pci device. @@ -833,7 +843,10 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) */ int pci_aspm_init(struct pci_dev *pdev) { - return 0; + if (!pdev->has_secondary_link) + return pci_aspm_init_downstream(pdev); + + return pci_aspm_init_upstream(pdev); } /* -- 1.9.1 ^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH V8 2/5] PCI/ASPM: split pci_aspm_init() into two @ 2017-04-08 4:55 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-08 4:55 UTC (permalink / raw) To: linux-pci, timur Cc: mayurkumar.patel, David Daney, linux-arm-msm, Shawn Lin, linux-kernel, Sinan Kaya, Julia Lawall, Bjorn Helgaas, Rajat Jain, linux-arm-kernel Split pci_aspm_init() body into pci_aspm_init_upstream() and pci_aspm_init_downstream() for bridge and endpoint specific code behavior. Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 Signed-off-by: Sinan Kaya <okaya@codeaurora.org> --- drivers/pci/pcie/aspm.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index dc36717..a80d64b 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -826,6 +826,16 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) return link; } +static int pci_aspm_init_downstream(struct pci_dev *pdev) +{ + return 0; +} + +static int pci_aspm_init_upstream(struct pci_dev *pdev) +{ + return 0; +} + /* * pci_aspm_init: Initiate PCI express link state. * It is called from device_add for every single pci device. @@ -833,7 +843,10 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) */ int pci_aspm_init(struct pci_dev *pdev) { - return 0; + if (!pdev->has_secondary_link) + return pci_aspm_init_downstream(pdev); + + return pci_aspm_init_upstream(pdev); } /* -- 1.9.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 94+ messages in thread
* Re: [PATCH V8 2/5] PCI/ASPM: split pci_aspm_init() into two 2017-04-08 4:55 ` Sinan Kaya @ 2017-04-12 19:16 ` Rajat Jain -1 siblings, 0 replies; 94+ messages in thread From: Rajat Jain @ 2017-04-12 19:16 UTC (permalink / raw) To: Sinan Kaya Cc: linux-pci, timur, Patel, Mayurkumar, linux-arm-msm, linux-arm-kernel, Bjorn Helgaas, Shawn Lin, Julia Lawall, David Daney, linux-kernel, Rajat Jain On Fri, Apr 7, 2017 at 9:55 PM, Sinan Kaya <okaya@codeaurora.org> wrote: > Split pci_aspm_init() body into pci_aspm_init_upstream() > and pci_aspm_init_downstream() for bridge and endpoint > specific code behavior. > > Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 > Signed-off-by: Sinan Kaya <okaya@codeaurora.org> > --- > drivers/pci/pcie/aspm.c | 15 ++++++++++++++- > 1 file changed, 14 insertions(+), 1 deletion(-) > > diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c > index dc36717..a80d64b 100644 > --- a/drivers/pci/pcie/aspm.c > +++ b/drivers/pci/pcie/aspm.c > @@ -826,6 +826,16 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) > return link; > } > > +static int pci_aspm_init_downstream(struct pci_dev *pdev) > +{ > + return 0; > +} > + > +static int pci_aspm_init_upstream(struct pci_dev *pdev) > +{ > + return 0; > +} > + > /* > * pci_aspm_init: Initiate PCI express link state. > * It is called from device_add for every single pci device. > @@ -833,7 +843,10 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) > */ > int pci_aspm_init(struct pci_dev *pdev) > { > - return 0; > + if (!pdev->has_secondary_link) > + return pci_aspm_init_downstream(pdev); > + > + return pci_aspm_init_upstream(pdev); > } Nit: if (x_flag()) return x(); return y(); May be better than if (!x_flag) return y(); return x(); > > /* > -- > 1.9.1 > ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 2/5] PCI/ASPM: split pci_aspm_init() into two @ 2017-04-12 19:16 ` Rajat Jain 0 siblings, 0 replies; 94+ messages in thread From: Rajat Jain @ 2017-04-12 19:16 UTC (permalink / raw) To: linux-arm-kernel On Fri, Apr 7, 2017 at 9:55 PM, Sinan Kaya <okaya@codeaurora.org> wrote: > Split pci_aspm_init() body into pci_aspm_init_upstream() > and pci_aspm_init_downstream() for bridge and endpoint > specific code behavior. > > Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 > Signed-off-by: Sinan Kaya <okaya@codeaurora.org> > --- > drivers/pci/pcie/aspm.c | 15 ++++++++++++++- > 1 file changed, 14 insertions(+), 1 deletion(-) > > diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c > index dc36717..a80d64b 100644 > --- a/drivers/pci/pcie/aspm.c > +++ b/drivers/pci/pcie/aspm.c > @@ -826,6 +826,16 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) > return link; > } > > +static int pci_aspm_init_downstream(struct pci_dev *pdev) > +{ > + return 0; > +} > + > +static int pci_aspm_init_upstream(struct pci_dev *pdev) > +{ > + return 0; > +} > + > /* > * pci_aspm_init: Initiate PCI express link state. > * It is called from device_add for every single pci device. > @@ -833,7 +843,10 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) > */ > int pci_aspm_init(struct pci_dev *pdev) > { > - return 0; > + if (!pdev->has_secondary_link) > + return pci_aspm_init_downstream(pdev); > + > + return pci_aspm_init_upstream(pdev); > } Nit: if (x_flag()) return x(); return y(); May be better than if (!x_flag) return y(); return x(); > > /* > -- > 1.9.1 > ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 2/5] PCI/ASPM: split pci_aspm_init() into two 2017-04-12 19:16 ` Rajat Jain (?) @ 2017-04-13 18:25 ` Bjorn Helgaas -1 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-13 18:25 UTC (permalink / raw) To: Rajat Jain Cc: Sinan Kaya, linux-pci, Timur Tabi, Patel, Mayurkumar, linux-arm-msm, linux-arm, Shawn Lin, Julia Lawall, David Daney, linux-kernel, Rajat Jain On Wed, Apr 12, 2017 at 2:16 PM, Rajat Jain <rajatja@google.com> wrote: > On Fri, Apr 7, 2017 at 9:55 PM, Sinan Kaya <okaya@codeaurora.org> wrote: >> Split pci_aspm_init() body into pci_aspm_init_upstream() >> and pci_aspm_init_downstream() for bridge and endpoint >> specific code behavior. >> >> Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 >> Signed-off-by: Sinan Kaya <okaya@codeaurora.org> >> --- >> drivers/pci/pcie/aspm.c | 15 ++++++++++++++- >> 1 file changed, 14 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c >> index dc36717..a80d64b 100644 >> --- a/drivers/pci/pcie/aspm.c >> +++ b/drivers/pci/pcie/aspm.c >> @@ -826,6 +826,16 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) >> return link; >> } >> >> +static int pci_aspm_init_downstream(struct pci_dev *pdev) >> +{ >> + return 0; >> +} >> + >> +static int pci_aspm_init_upstream(struct pci_dev *pdev) >> +{ >> + return 0; >> +} >> + >> /* >> * pci_aspm_init: Initiate PCI express link state. >> * It is called from device_add for every single pci device. >> @@ -833,7 +843,10 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) >> */ >> int pci_aspm_init(struct pci_dev *pdev) >> { >> - return 0; >> + if (!pdev->has_secondary_link) >> + return pci_aspm_init_downstream(pdev); >> + >> + return pci_aspm_init_upstream(pdev); >> } > > Nit: > > if (x_flag()) > return x(); > return y(); > > May be better than > > if (!x_flag) > return y(); > return x(); I agree, and I made that change on my branch. I also renamed these to pci_aspm_init_downstream_port() (for Root Ports and Switch Downstream Ports) and pci_aspm_init_upstream_port() (for Switch Upstream Ports and Endpoints) to try to match the terminology in the spec. FWIW, your email didn't seem to make it to the list, Rajat. Maybe some gmail weirdness? I don't *see* anything wrong, but ... I'm responding via gmail, so my response probably won't make it to the list either. Bjorn ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 2/5] PCI/ASPM: split pci_aspm_init() into two @ 2017-04-13 18:25 ` Bjorn Helgaas 0 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-13 18:25 UTC (permalink / raw) To: linux-arm-kernel On Wed, Apr 12, 2017 at 2:16 PM, Rajat Jain <rajatja@google.com> wrote: > On Fri, Apr 7, 2017 at 9:55 PM, Sinan Kaya <okaya@codeaurora.org> wrote: >> Split pci_aspm_init() body into pci_aspm_init_upstream() >> and pci_aspm_init_downstream() for bridge and endpoint >> specific code behavior. >> >> Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 >> Signed-off-by: Sinan Kaya <okaya@codeaurora.org> >> --- >> drivers/pci/pcie/aspm.c | 15 ++++++++++++++- >> 1 file changed, 14 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c >> index dc36717..a80d64b 100644 >> --- a/drivers/pci/pcie/aspm.c >> +++ b/drivers/pci/pcie/aspm.c >> @@ -826,6 +826,16 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) >> return link; >> } >> >> +static int pci_aspm_init_downstream(struct pci_dev *pdev) >> +{ >> + return 0; >> +} >> + >> +static int pci_aspm_init_upstream(struct pci_dev *pdev) >> +{ >> + return 0; >> +} >> + >> /* >> * pci_aspm_init: Initiate PCI express link state. >> * It is called from device_add for every single pci device. >> @@ -833,7 +843,10 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) >> */ >> int pci_aspm_init(struct pci_dev *pdev) >> { >> - return 0; >> + if (!pdev->has_secondary_link) >> + return pci_aspm_init_downstream(pdev); >> + >> + return pci_aspm_init_upstream(pdev); >> } > > Nit: > > if (x_flag()) > return x(); > return y(); > > May be better than > > if (!x_flag) > return y(); > return x(); I agree, and I made that change on my branch. I also renamed these to pci_aspm_init_downstream_port() (for Root Ports and Switch Downstream Ports) and pci_aspm_init_upstream_port() (for Switch Upstream Ports and Endpoints) to try to match the terminology in the spec. FWIW, your email didn't seem to make it to the list, Rajat. Maybe some gmail weirdness? I don't *see* anything wrong, but ... I'm responding via gmail, so my response probably won't make it to the list either. Bjorn ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 2/5] PCI/ASPM: split pci_aspm_init() into two @ 2017-04-13 18:25 ` Bjorn Helgaas 0 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-13 18:25 UTC (permalink / raw) To: Rajat Jain Cc: Patel, Mayurkumar, Rajat Jain, David Daney, linux-pci, Timur Tabi, linux-kernel, Sinan Kaya, Julia Lawall, linux-arm-msm, Shawn Lin, linux-arm On Wed, Apr 12, 2017 at 2:16 PM, Rajat Jain <rajatja@google.com> wrote: > On Fri, Apr 7, 2017 at 9:55 PM, Sinan Kaya <okaya@codeaurora.org> wrote: >> Split pci_aspm_init() body into pci_aspm_init_upstream() >> and pci_aspm_init_downstream() for bridge and endpoint >> specific code behavior. >> >> Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 >> Signed-off-by: Sinan Kaya <okaya@codeaurora.org> >> --- >> drivers/pci/pcie/aspm.c | 15 ++++++++++++++- >> 1 file changed, 14 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c >> index dc36717..a80d64b 100644 >> --- a/drivers/pci/pcie/aspm.c >> +++ b/drivers/pci/pcie/aspm.c >> @@ -826,6 +826,16 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) >> return link; >> } >> >> +static int pci_aspm_init_downstream(struct pci_dev *pdev) >> +{ >> + return 0; >> +} >> + >> +static int pci_aspm_init_upstream(struct pci_dev *pdev) >> +{ >> + return 0; >> +} >> + >> /* >> * pci_aspm_init: Initiate PCI express link state. >> * It is called from device_add for every single pci device. >> @@ -833,7 +843,10 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) >> */ >> int pci_aspm_init(struct pci_dev *pdev) >> { >> - return 0; >> + if (!pdev->has_secondary_link) >> + return pci_aspm_init_downstream(pdev); >> + >> + return pci_aspm_init_upstream(pdev); >> } > > Nit: > > if (x_flag()) > return x(); > return y(); > > May be better than > > if (!x_flag) > return y(); > return x(); I agree, and I made that change on my branch. I also renamed these to pci_aspm_init_downstream_port() (for Root Ports and Switch Downstream Ports) and pci_aspm_init_upstream_port() (for Switch Upstream Ports and Endpoints) to try to match the terminology in the spec. FWIW, your email didn't seem to make it to the list, Rajat. Maybe some gmail weirdness? I don't *see* anything wrong, but ... I'm responding via gmail, so my response probably won't make it to the list either. Bjorn _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 2/5] PCI/ASPM: split pci_aspm_init() into two 2017-04-13 18:25 ` Bjorn Helgaas (?) @ 2017-04-14 19:10 ` Sinan Kaya -1 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-14 19:10 UTC (permalink / raw) To: Bjorn Helgaas, Rajat Jain Cc: linux-pci, Timur Tabi, Patel, Mayurkumar, linux-arm-msm, linux-arm, Shawn Lin, Julia Lawall, David Daney, linux-kernel, Rajat Jain On 4/13/2017 2:25 PM, Bjorn Helgaas wrote: > I agree, and I made that change on my branch. I also renamed these to > pci_aspm_init_downstream_port() (for Root Ports and Switch Downstream > Ports) and pci_aspm_init_upstream_port() (for Switch Upstream Ports > and Endpoints) to try to match the terminology in the spec. OK. I reordered and renamed the functions according to your suggestions on my local copy too. -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 2/5] PCI/ASPM: split pci_aspm_init() into two @ 2017-04-14 19:10 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-14 19:10 UTC (permalink / raw) To: linux-arm-kernel On 4/13/2017 2:25 PM, Bjorn Helgaas wrote: > I agree, and I made that change on my branch. I also renamed these to > pci_aspm_init_downstream_port() (for Root Ports and Switch Downstream > Ports) and pci_aspm_init_upstream_port() (for Switch Upstream Ports > and Endpoints) to try to match the terminology in the spec. OK. I reordered and renamed the functions according to your suggestions on my local copy too. -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 2/5] PCI/ASPM: split pci_aspm_init() into two @ 2017-04-14 19:10 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-14 19:10 UTC (permalink / raw) To: Bjorn Helgaas, Rajat Jain Cc: Patel, Mayurkumar, Rajat Jain, David Daney, linux-pci, Timur Tabi, linux-kernel, Julia Lawall, linux-arm-msm, Shawn Lin, linux-arm On 4/13/2017 2:25 PM, Bjorn Helgaas wrote: > I agree, and I made that change on my branch. I also renamed these to > pci_aspm_init_downstream_port() (for Root Ports and Switch Downstream > Ports) and pci_aspm_init_upstream_port() (for Switch Upstream Ports > and Endpoints) to try to match the terminology in the spec. OK. I reordered and renamed the functions according to your suggestions on my local copy too. -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 3/5] PCI/ASPM: add init hook to device_add 2017-04-08 4:55 ` Sinan Kaya (?) @ 2017-04-08 4:55 ` Sinan Kaya -1 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-08 4:55 UTC (permalink / raw) To: linux-pci, timur Cc: mayurkumar.patel, linux-arm-msm, linux-arm-kernel, Sinan Kaya, Bjorn Helgaas, Rajat Jain, David Daney, Julia Lawall, linux-kernel For bridges, have pcie_aspm_init_link_state() allocate a link_state, regardless of whether it currently has any children. pcie_aspm_init_link_state(): Called for bridges (upstream end of link) after all children have been enumerated. No longer needs to check aspm_support_enabled or pdev->has_secondary_link or the VIA quirk: pci_aspm_init() already checked that stuff, so we only need to check pdev->link_state here. Now that we are calling alloc_pcie_link_state() from pci_aspm_init() , get rid of pci_function_0 function and detect downstream link in pci_aspm_init_upstream() instead when the function number is 0. Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 Signed-off-by: Sinan Kaya <okaya@codeaurora.org> --- drivers/pci/pcie/aspm.c | 72 ++++++++++++++++++++++++------------------------- 1 file changed, 36 insertions(+), 36 deletions(-) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index a80d64b..e33f84b 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -422,20 +422,6 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint) } } -/* - * The L1 PM substate capability is only implemented in function 0 in a - * multi function device. - */ -static struct pci_dev *pci_function_0(struct pci_bus *linkbus) -{ - struct pci_dev *child; - - list_for_each_entry(child, &linkbus->devices, bus_list) - if (PCI_FUNC(child->devfn) == 0) - return child; - return NULL; -} - /* Calculate L1.2 PM substate timing parameters */ static void aspm_calc_l1ss_info(struct pcie_link_state *link, struct aspm_register_info *upreg, @@ -798,7 +784,6 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) INIT_LIST_HEAD(&link->children); INIT_LIST_HEAD(&link->link); link->pdev = pdev; - link->downstream = pci_function_0(pdev->subordinate); /* * Root Ports and PCI/PCI-X to PCIe Bridges are roots of PCIe @@ -828,11 +813,33 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) static int pci_aspm_init_downstream(struct pci_dev *pdev) { + struct pcie_link_state *link; + struct pci_dev *parent; + + /* + * The L1 PM substate capability is only implemented in function 0 in a + * multi function device. + */ + if (PCI_FUNC(pdev->devfn) != 0) + return -EINVAL; + + parent = pdev->bus->self; + if (!parent) + return -EINVAL; + + link = parent->link_state; + link->downstream = pdev; return 0; } static int pci_aspm_init_upstream(struct pci_dev *pdev) { + struct pcie_link_state *link; + + link = alloc_pcie_link_state(pdev); + if (!link) + return -ENOMEM; + return 0; } @@ -843,6 +850,17 @@ static int pci_aspm_init_upstream(struct pci_dev *pdev) */ int pci_aspm_init(struct pci_dev *pdev) { + if (!aspm_support_enabled) + return 0; + + if (!pci_is_pcie(pdev)) + return -EINVAL; + + /* VIA has a strange chipset, root port is under a bridge */ + if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT && + pdev->bus->self) + return 0; + if (!pdev->has_secondary_link) return pci_aspm_init_downstream(pdev); @@ -859,33 +877,16 @@ void pcie_aspm_init_link_state(struct pci_dev *pdev) struct pcie_link_state *link; int blacklist = !!pcie_aspm_sanity_check(pdev); - if (!aspm_support_enabled) - return; - - if (pdev->link_state) - return; - - /* - * We allocate pcie_link_state for the component on the upstream - * end of a Link, so there's nothing to do unless this device has a - * Link on its secondary side. - */ - if (!pdev->has_secondary_link) - return; - - /* VIA has a strange chipset, root port is under a bridge */ - if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT && - pdev->bus->self) + if (!pdev->link_state) return; + link = pdev->link_state; down_read(&pci_bus_sem); if (list_empty(&pdev->subordinate->devices)) goto out; mutex_lock(&aspm_lock); - link = alloc_pcie_link_state(pdev); - if (!link) - goto unlock; + /* * Setup initial ASPM state. Note that we need to configure * upstream links also because capable state of them can be @@ -910,7 +911,6 @@ void pcie_aspm_init_link_state(struct pci_dev *pdev) pcie_set_clkpm(link, policy_to_clkpm_state(link)); } -unlock: mutex_unlock(&aspm_lock); out: up_read(&pci_bus_sem); -- 1.9.1 ^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH V8 3/5] PCI/ASPM: add init hook to device_add @ 2017-04-08 4:55 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-08 4:55 UTC (permalink / raw) To: linux-arm-kernel For bridges, have pcie_aspm_init_link_state() allocate a link_state, regardless of whether it currently has any children. pcie_aspm_init_link_state(): Called for bridges (upstream end of link) after all children have been enumerated. No longer needs to check aspm_support_enabled or pdev->has_secondary_link or the VIA quirk: pci_aspm_init() already checked that stuff, so we only need to check pdev->link_state here. Now that we are calling alloc_pcie_link_state() from pci_aspm_init() , get rid of pci_function_0 function and detect downstream link in pci_aspm_init_upstream() instead when the function number is 0. Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 Signed-off-by: Sinan Kaya <okaya@codeaurora.org> --- drivers/pci/pcie/aspm.c | 72 ++++++++++++++++++++++++------------------------- 1 file changed, 36 insertions(+), 36 deletions(-) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index a80d64b..e33f84b 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -422,20 +422,6 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint) } } -/* - * The L1 PM substate capability is only implemented in function 0 in a - * multi function device. - */ -static struct pci_dev *pci_function_0(struct pci_bus *linkbus) -{ - struct pci_dev *child; - - list_for_each_entry(child, &linkbus->devices, bus_list) - if (PCI_FUNC(child->devfn) == 0) - return child; - return NULL; -} - /* Calculate L1.2 PM substate timing parameters */ static void aspm_calc_l1ss_info(struct pcie_link_state *link, struct aspm_register_info *upreg, @@ -798,7 +784,6 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) INIT_LIST_HEAD(&link->children); INIT_LIST_HEAD(&link->link); link->pdev = pdev; - link->downstream = pci_function_0(pdev->subordinate); /* * Root Ports and PCI/PCI-X to PCIe Bridges are roots of PCIe @@ -828,11 +813,33 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) static int pci_aspm_init_downstream(struct pci_dev *pdev) { + struct pcie_link_state *link; + struct pci_dev *parent; + + /* + * The L1 PM substate capability is only implemented in function 0 in a + * multi function device. + */ + if (PCI_FUNC(pdev->devfn) != 0) + return -EINVAL; + + parent = pdev->bus->self; + if (!parent) + return -EINVAL; + + link = parent->link_state; + link->downstream = pdev; return 0; } static int pci_aspm_init_upstream(struct pci_dev *pdev) { + struct pcie_link_state *link; + + link = alloc_pcie_link_state(pdev); + if (!link) + return -ENOMEM; + return 0; } @@ -843,6 +850,17 @@ static int pci_aspm_init_upstream(struct pci_dev *pdev) */ int pci_aspm_init(struct pci_dev *pdev) { + if (!aspm_support_enabled) + return 0; + + if (!pci_is_pcie(pdev)) + return -EINVAL; + + /* VIA has a strange chipset, root port is under a bridge */ + if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT && + pdev->bus->self) + return 0; + if (!pdev->has_secondary_link) return pci_aspm_init_downstream(pdev); @@ -859,33 +877,16 @@ void pcie_aspm_init_link_state(struct pci_dev *pdev) struct pcie_link_state *link; int blacklist = !!pcie_aspm_sanity_check(pdev); - if (!aspm_support_enabled) - return; - - if (pdev->link_state) - return; - - /* - * We allocate pcie_link_state for the component on the upstream - * end of a Link, so there's nothing to do unless this device has a - * Link on its secondary side. - */ - if (!pdev->has_secondary_link) - return; - - /* VIA has a strange chipset, root port is under a bridge */ - if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT && - pdev->bus->self) + if (!pdev->link_state) return; + link = pdev->link_state; down_read(&pci_bus_sem); if (list_empty(&pdev->subordinate->devices)) goto out; mutex_lock(&aspm_lock); - link = alloc_pcie_link_state(pdev); - if (!link) - goto unlock; + /* * Setup initial ASPM state. Note that we need to configure * upstream links also because capable state of them can be @@ -910,7 +911,6 @@ void pcie_aspm_init_link_state(struct pci_dev *pdev) pcie_set_clkpm(link, policy_to_clkpm_state(link)); } -unlock: mutex_unlock(&aspm_lock); out: up_read(&pci_bus_sem); -- 1.9.1 ^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH V8 3/5] PCI/ASPM: add init hook to device_add @ 2017-04-08 4:55 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-08 4:55 UTC (permalink / raw) To: linux-pci, timur Cc: mayurkumar.patel, David Daney, linux-arm-msm, linux-kernel, Sinan Kaya, Julia Lawall, Bjorn Helgaas, Rajat Jain, linux-arm-kernel For bridges, have pcie_aspm_init_link_state() allocate a link_state, regardless of whether it currently has any children. pcie_aspm_init_link_state(): Called for bridges (upstream end of link) after all children have been enumerated. No longer needs to check aspm_support_enabled or pdev->has_secondary_link or the VIA quirk: pci_aspm_init() already checked that stuff, so we only need to check pdev->link_state here. Now that we are calling alloc_pcie_link_state() from pci_aspm_init() , get rid of pci_function_0 function and detect downstream link in pci_aspm_init_upstream() instead when the function number is 0. Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 Signed-off-by: Sinan Kaya <okaya@codeaurora.org> --- drivers/pci/pcie/aspm.c | 72 ++++++++++++++++++++++++------------------------- 1 file changed, 36 insertions(+), 36 deletions(-) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index a80d64b..e33f84b 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -422,20 +422,6 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint) } } -/* - * The L1 PM substate capability is only implemented in function 0 in a - * multi function device. - */ -static struct pci_dev *pci_function_0(struct pci_bus *linkbus) -{ - struct pci_dev *child; - - list_for_each_entry(child, &linkbus->devices, bus_list) - if (PCI_FUNC(child->devfn) == 0) - return child; - return NULL; -} - /* Calculate L1.2 PM substate timing parameters */ static void aspm_calc_l1ss_info(struct pcie_link_state *link, struct aspm_register_info *upreg, @@ -798,7 +784,6 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) INIT_LIST_HEAD(&link->children); INIT_LIST_HEAD(&link->link); link->pdev = pdev; - link->downstream = pci_function_0(pdev->subordinate); /* * Root Ports and PCI/PCI-X to PCIe Bridges are roots of PCIe @@ -828,11 +813,33 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) static int pci_aspm_init_downstream(struct pci_dev *pdev) { + struct pcie_link_state *link; + struct pci_dev *parent; + + /* + * The L1 PM substate capability is only implemented in function 0 in a + * multi function device. + */ + if (PCI_FUNC(pdev->devfn) != 0) + return -EINVAL; + + parent = pdev->bus->self; + if (!parent) + return -EINVAL; + + link = parent->link_state; + link->downstream = pdev; return 0; } static int pci_aspm_init_upstream(struct pci_dev *pdev) { + struct pcie_link_state *link; + + link = alloc_pcie_link_state(pdev); + if (!link) + return -ENOMEM; + return 0; } @@ -843,6 +850,17 @@ static int pci_aspm_init_upstream(struct pci_dev *pdev) */ int pci_aspm_init(struct pci_dev *pdev) { + if (!aspm_support_enabled) + return 0; + + if (!pci_is_pcie(pdev)) + return -EINVAL; + + /* VIA has a strange chipset, root port is under a bridge */ + if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT && + pdev->bus->self) + return 0; + if (!pdev->has_secondary_link) return pci_aspm_init_downstream(pdev); @@ -859,33 +877,16 @@ void pcie_aspm_init_link_state(struct pci_dev *pdev) struct pcie_link_state *link; int blacklist = !!pcie_aspm_sanity_check(pdev); - if (!aspm_support_enabled) - return; - - if (pdev->link_state) - return; - - /* - * We allocate pcie_link_state for the component on the upstream - * end of a Link, so there's nothing to do unless this device has a - * Link on its secondary side. - */ - if (!pdev->has_secondary_link) - return; - - /* VIA has a strange chipset, root port is under a bridge */ - if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT && - pdev->bus->self) + if (!pdev->link_state) return; + link = pdev->link_state; down_read(&pci_bus_sem); if (list_empty(&pdev->subordinate->devices)) goto out; mutex_lock(&aspm_lock); - link = alloc_pcie_link_state(pdev); - if (!link) - goto unlock; + /* * Setup initial ASPM state. Note that we need to configure * upstream links also because capable state of them can be @@ -910,7 +911,6 @@ void pcie_aspm_init_link_state(struct pci_dev *pdev) pcie_set_clkpm(link, policy_to_clkpm_state(link)); } -unlock: mutex_unlock(&aspm_lock); out: up_read(&pci_bus_sem); -- 1.9.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 94+ messages in thread
* Re: [PATCH V8 3/5] PCI/ASPM: add init hook to device_add 2017-04-08 4:55 ` Sinan Kaya (?) @ 2017-04-13 20:48 ` Bjorn Helgaas -1 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-13 20:48 UTC (permalink / raw) To: Sinan Kaya Cc: linux-pci, timur, mayurkumar.patel, David Daney, linux-arm-msm, linux-kernel, Julia Lawall, Bjorn Helgaas, Rajat Jain, linux-arm-kernel Hi Sinan, On Sat, Apr 08, 2017 at 12:55:49AM -0400, Sinan Kaya wrote: > For bridges, have pcie_aspm_init_link_state() allocate a link_state, > regardless of whether it currently has any children. > > pcie_aspm_init_link_state(): Called for bridges (upstream end of > link) after all children have been enumerated. No longer needs to > check aspm_support_enabled or pdev->has_secondary_link or the VIA > quirk: pci_aspm_init() already checked that stuff, so we only need > to check pdev->link_state here. > > Now that we are calling alloc_pcie_link_state() from pci_aspm_init() > , get rid of pci_function_0 function and detect downstream link in > pci_aspm_init_upstream() instead when the function number is 0. > > Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 > Signed-off-by: Sinan Kaya <okaya@codeaurora.org> > --- > drivers/pci/pcie/aspm.c | 72 ++++++++++++++++++++++++------------------------- > 1 file changed, 36 insertions(+), 36 deletions(-) > > diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c > index a80d64b..e33f84b 100644 > --- a/drivers/pci/pcie/aspm.c > +++ b/drivers/pci/pcie/aspm.c > @@ -422,20 +422,6 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint) > } > } > > -/* > - * The L1 PM substate capability is only implemented in function 0 in a > - * multi function device. > - */ > -static struct pci_dev *pci_function_0(struct pci_bus *linkbus) > -{ > - struct pci_dev *child; > - > - list_for_each_entry(child, &linkbus->devices, bus_list) > - if (PCI_FUNC(child->devfn) == 0) > - return child; > - return NULL; > -} > - > /* Calculate L1.2 PM substate timing parameters */ > static void aspm_calc_l1ss_info(struct pcie_link_state *link, > struct aspm_register_info *upreg, > @@ -798,7 +784,6 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) > INIT_LIST_HEAD(&link->children); > INIT_LIST_HEAD(&link->link); > link->pdev = pdev; > - link->downstream = pci_function_0(pdev->subordinate); > > /* > * Root Ports and PCI/PCI-X to PCIe Bridges are roots of PCIe > @@ -828,11 +813,33 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) > > static int pci_aspm_init_downstream(struct pci_dev *pdev) > { > + struct pcie_link_state *link; > + struct pci_dev *parent; > + > + /* > + * The L1 PM substate capability is only implemented in function 0 in a > + * multi function device. > + */ > + if (PCI_FUNC(pdev->devfn) != 0) > + return -EINVAL; > + > + parent = pdev->bus->self; > + if (!parent) > + return -EINVAL; > + > + link = parent->link_state; > + link->downstream = pdev; > return 0; > } > > static int pci_aspm_init_upstream(struct pci_dev *pdev) > { > + struct pcie_link_state *link; > + > + link = alloc_pcie_link_state(pdev); > + if (!link) > + return -ENOMEM; This allocates the link_state when we enumerate a Downstream Port instead of when we enumerate a child device. We want the link_state lifetime to match that of the Downstream Port, so this seems good. But we shouldn't at the same time change the link_state deallocation so it happens when the Downstream Port is removed, not when the child device is removed? > return 0; > } > > @@ -843,6 +850,17 @@ static int pci_aspm_init_upstream(struct pci_dev *pdev) > */ > int pci_aspm_init(struct pci_dev *pdev) > { > + if (!aspm_support_enabled) > + return 0; > + > + if (!pci_is_pcie(pdev)) > + return -EINVAL; > + > + /* VIA has a strange chipset, root port is under a bridge */ > + if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT && > + pdev->bus->self) > + return 0; > + > if (!pdev->has_secondary_link) > return pci_aspm_init_downstream(pdev); > > @@ -859,33 +877,16 @@ void pcie_aspm_init_link_state(struct pci_dev *pdev) > struct pcie_link_state *link; > int blacklist = !!pcie_aspm_sanity_check(pdev); > > - if (!aspm_support_enabled) > - return; > - > - if (pdev->link_state) > - return; > - > - /* > - * We allocate pcie_link_state for the component on the upstream > - * end of a Link, so there's nothing to do unless this device has a > - * Link on its secondary side. > - */ > - if (!pdev->has_secondary_link) > - return; > - > - /* VIA has a strange chipset, root port is under a bridge */ > - if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT && > - pdev->bus->self) > + if (!pdev->link_state) > return; > > + link = pdev->link_state; > down_read(&pci_bus_sem); > if (list_empty(&pdev->subordinate->devices)) > goto out; > > mutex_lock(&aspm_lock); > - link = alloc_pcie_link_state(pdev); > - if (!link) > - goto unlock; > + > /* > * Setup initial ASPM state. Note that we need to configure > * upstream links also because capable state of them can be > @@ -910,7 +911,6 @@ void pcie_aspm_init_link_state(struct pci_dev *pdev) > pcie_set_clkpm(link, policy_to_clkpm_state(link)); > } > > -unlock: > mutex_unlock(&aspm_lock); > out: > up_read(&pci_bus_sem); > -- > 1.9.1 > > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 3/5] PCI/ASPM: add init hook to device_add @ 2017-04-13 20:48 ` Bjorn Helgaas 0 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-13 20:48 UTC (permalink / raw) To: linux-arm-kernel Hi Sinan, On Sat, Apr 08, 2017 at 12:55:49AM -0400, Sinan Kaya wrote: > For bridges, have pcie_aspm_init_link_state() allocate a link_state, > regardless of whether it currently has any children. > > pcie_aspm_init_link_state(): Called for bridges (upstream end of > link) after all children have been enumerated. No longer needs to > check aspm_support_enabled or pdev->has_secondary_link or the VIA > quirk: pci_aspm_init() already checked that stuff, so we only need > to check pdev->link_state here. > > Now that we are calling alloc_pcie_link_state() from pci_aspm_init() > , get rid of pci_function_0 function and detect downstream link in > pci_aspm_init_upstream() instead when the function number is 0. > > Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 > Signed-off-by: Sinan Kaya <okaya@codeaurora.org> > --- > drivers/pci/pcie/aspm.c | 72 ++++++++++++++++++++++++------------------------- > 1 file changed, 36 insertions(+), 36 deletions(-) > > diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c > index a80d64b..e33f84b 100644 > --- a/drivers/pci/pcie/aspm.c > +++ b/drivers/pci/pcie/aspm.c > @@ -422,20 +422,6 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint) > } > } > > -/* > - * The L1 PM substate capability is only implemented in function 0 in a > - * multi function device. > - */ > -static struct pci_dev *pci_function_0(struct pci_bus *linkbus) > -{ > - struct pci_dev *child; > - > - list_for_each_entry(child, &linkbus->devices, bus_list) > - if (PCI_FUNC(child->devfn) == 0) > - return child; > - return NULL; > -} > - > /* Calculate L1.2 PM substate timing parameters */ > static void aspm_calc_l1ss_info(struct pcie_link_state *link, > struct aspm_register_info *upreg, > @@ -798,7 +784,6 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) > INIT_LIST_HEAD(&link->children); > INIT_LIST_HEAD(&link->link); > link->pdev = pdev; > - link->downstream = pci_function_0(pdev->subordinate); > > /* > * Root Ports and PCI/PCI-X to PCIe Bridges are roots of PCIe > @@ -828,11 +813,33 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) > > static int pci_aspm_init_downstream(struct pci_dev *pdev) > { > + struct pcie_link_state *link; > + struct pci_dev *parent; > + > + /* > + * The L1 PM substate capability is only implemented in function 0 in a > + * multi function device. > + */ > + if (PCI_FUNC(pdev->devfn) != 0) > + return -EINVAL; > + > + parent = pdev->bus->self; > + if (!parent) > + return -EINVAL; > + > + link = parent->link_state; > + link->downstream = pdev; > return 0; > } > > static int pci_aspm_init_upstream(struct pci_dev *pdev) > { > + struct pcie_link_state *link; > + > + link = alloc_pcie_link_state(pdev); > + if (!link) > + return -ENOMEM; This allocates the link_state when we enumerate a Downstream Port instead of when we enumerate a child device. We want the link_state lifetime to match that of the Downstream Port, so this seems good. But we shouldn't at the same time change the link_state deallocation so it happens when the Downstream Port is removed, not when the child device is removed? > return 0; > } > > @@ -843,6 +850,17 @@ static int pci_aspm_init_upstream(struct pci_dev *pdev) > */ > int pci_aspm_init(struct pci_dev *pdev) > { > + if (!aspm_support_enabled) > + return 0; > + > + if (!pci_is_pcie(pdev)) > + return -EINVAL; > + > + /* VIA has a strange chipset, root port is under a bridge */ > + if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT && > + pdev->bus->self) > + return 0; > + > if (!pdev->has_secondary_link) > return pci_aspm_init_downstream(pdev); > > @@ -859,33 +877,16 @@ void pcie_aspm_init_link_state(struct pci_dev *pdev) > struct pcie_link_state *link; > int blacklist = !!pcie_aspm_sanity_check(pdev); > > - if (!aspm_support_enabled) > - return; > - > - if (pdev->link_state) > - return; > - > - /* > - * We allocate pcie_link_state for the component on the upstream > - * end of a Link, so there's nothing to do unless this device has a > - * Link on its secondary side. > - */ > - if (!pdev->has_secondary_link) > - return; > - > - /* VIA has a strange chipset, root port is under a bridge */ > - if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT && > - pdev->bus->self) > + if (!pdev->link_state) > return; > > + link = pdev->link_state; > down_read(&pci_bus_sem); > if (list_empty(&pdev->subordinate->devices)) > goto out; > > mutex_lock(&aspm_lock); > - link = alloc_pcie_link_state(pdev); > - if (!link) > - goto unlock; > + > /* > * Setup initial ASPM state. Note that we need to configure > * upstream links also because capable state of them can be > @@ -910,7 +911,6 @@ void pcie_aspm_init_link_state(struct pci_dev *pdev) > pcie_set_clkpm(link, policy_to_clkpm_state(link)); > } > > -unlock: > mutex_unlock(&aspm_lock); > out: > up_read(&pci_bus_sem); > -- > 1.9.1 > > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel at lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 3/5] PCI/ASPM: add init hook to device_add @ 2017-04-13 20:48 ` Bjorn Helgaas 0 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-13 20:48 UTC (permalink / raw) To: Sinan Kaya Cc: mayurkumar.patel, David Daney, linux-pci, timur, linux-kernel, Julia Lawall, linux-arm-msm, Bjorn Helgaas, Rajat Jain, linux-arm-kernel Hi Sinan, On Sat, Apr 08, 2017 at 12:55:49AM -0400, Sinan Kaya wrote: > For bridges, have pcie_aspm_init_link_state() allocate a link_state, > regardless of whether it currently has any children. > > pcie_aspm_init_link_state(): Called for bridges (upstream end of > link) after all children have been enumerated. No longer needs to > check aspm_support_enabled or pdev->has_secondary_link or the VIA > quirk: pci_aspm_init() already checked that stuff, so we only need > to check pdev->link_state here. > > Now that we are calling alloc_pcie_link_state() from pci_aspm_init() > , get rid of pci_function_0 function and detect downstream link in > pci_aspm_init_upstream() instead when the function number is 0. > > Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 > Signed-off-by: Sinan Kaya <okaya@codeaurora.org> > --- > drivers/pci/pcie/aspm.c | 72 ++++++++++++++++++++++++------------------------- > 1 file changed, 36 insertions(+), 36 deletions(-) > > diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c > index a80d64b..e33f84b 100644 > --- a/drivers/pci/pcie/aspm.c > +++ b/drivers/pci/pcie/aspm.c > @@ -422,20 +422,6 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint) > } > } > > -/* > - * The L1 PM substate capability is only implemented in function 0 in a > - * multi function device. > - */ > -static struct pci_dev *pci_function_0(struct pci_bus *linkbus) > -{ > - struct pci_dev *child; > - > - list_for_each_entry(child, &linkbus->devices, bus_list) > - if (PCI_FUNC(child->devfn) == 0) > - return child; > - return NULL; > -} > - > /* Calculate L1.2 PM substate timing parameters */ > static void aspm_calc_l1ss_info(struct pcie_link_state *link, > struct aspm_register_info *upreg, > @@ -798,7 +784,6 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) > INIT_LIST_HEAD(&link->children); > INIT_LIST_HEAD(&link->link); > link->pdev = pdev; > - link->downstream = pci_function_0(pdev->subordinate); > > /* > * Root Ports and PCI/PCI-X to PCIe Bridges are roots of PCIe > @@ -828,11 +813,33 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) > > static int pci_aspm_init_downstream(struct pci_dev *pdev) > { > + struct pcie_link_state *link; > + struct pci_dev *parent; > + > + /* > + * The L1 PM substate capability is only implemented in function 0 in a > + * multi function device. > + */ > + if (PCI_FUNC(pdev->devfn) != 0) > + return -EINVAL; > + > + parent = pdev->bus->self; > + if (!parent) > + return -EINVAL; > + > + link = parent->link_state; > + link->downstream = pdev; > return 0; > } > > static int pci_aspm_init_upstream(struct pci_dev *pdev) > { > + struct pcie_link_state *link; > + > + link = alloc_pcie_link_state(pdev); > + if (!link) > + return -ENOMEM; This allocates the link_state when we enumerate a Downstream Port instead of when we enumerate a child device. We want the link_state lifetime to match that of the Downstream Port, so this seems good. But we shouldn't at the same time change the link_state deallocation so it happens when the Downstream Port is removed, not when the child device is removed? > return 0; > } > > @@ -843,6 +850,17 @@ static int pci_aspm_init_upstream(struct pci_dev *pdev) > */ > int pci_aspm_init(struct pci_dev *pdev) > { > + if (!aspm_support_enabled) > + return 0; > + > + if (!pci_is_pcie(pdev)) > + return -EINVAL; > + > + /* VIA has a strange chipset, root port is under a bridge */ > + if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT && > + pdev->bus->self) > + return 0; > + > if (!pdev->has_secondary_link) > return pci_aspm_init_downstream(pdev); > > @@ -859,33 +877,16 @@ void pcie_aspm_init_link_state(struct pci_dev *pdev) > struct pcie_link_state *link; > int blacklist = !!pcie_aspm_sanity_check(pdev); > > - if (!aspm_support_enabled) > - return; > - > - if (pdev->link_state) > - return; > - > - /* > - * We allocate pcie_link_state for the component on the upstream > - * end of a Link, so there's nothing to do unless this device has a > - * Link on its secondary side. > - */ > - if (!pdev->has_secondary_link) > - return; > - > - /* VIA has a strange chipset, root port is under a bridge */ > - if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT && > - pdev->bus->self) > + if (!pdev->link_state) > return; > > + link = pdev->link_state; > down_read(&pci_bus_sem); > if (list_empty(&pdev->subordinate->devices)) > goto out; > > mutex_lock(&aspm_lock); > - link = alloc_pcie_link_state(pdev); > - if (!link) > - goto unlock; > + > /* > * Setup initial ASPM state. Note that we need to configure > * upstream links also because capable state of them can be > @@ -910,7 +911,6 @@ void pcie_aspm_init_link_state(struct pci_dev *pdev) > pcie_set_clkpm(link, policy_to_clkpm_state(link)); > } > > -unlock: > mutex_unlock(&aspm_lock); > out: > up_read(&pci_bus_sem); > -- > 1.9.1 > > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 3/5] PCI/ASPM: add init hook to device_add 2017-04-13 20:48 ` Bjorn Helgaas (?) @ 2017-04-13 21:02 ` Bjorn Helgaas -1 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-13 21:02 UTC (permalink / raw) To: Sinan Kaya Cc: mayurkumar.patel, David Daney, linux-pci, timur, linux-kernel, Julia Lawall, linux-arm-msm, Bjorn Helgaas, Rajat Jain, linux-arm-kernel On Thu, Apr 13, 2017 at 03:48:00PM -0500, Bjorn Helgaas wrote: > Hi Sinan, > > On Sat, Apr 08, 2017 at 12:55:49AM -0400, Sinan Kaya wrote: > > For bridges, have pcie_aspm_init_link_state() allocate a link_state, > > regardless of whether it currently has any children. > > > > pcie_aspm_init_link_state(): Called for bridges (upstream end of > > link) after all children have been enumerated. No longer needs to > > check aspm_support_enabled or pdev->has_secondary_link or the VIA > > quirk: pci_aspm_init() already checked that stuff, so we only need > > to check pdev->link_state here. > > > > Now that we are calling alloc_pcie_link_state() from pci_aspm_init() > > , get rid of pci_function_0 function and detect downstream link in > > pci_aspm_init_upstream() instead when the function number is 0. > > > > Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 > > Signed-off-by: Sinan Kaya <okaya@codeaurora.org> > > --- > > drivers/pci/pcie/aspm.c | 72 ++++++++++++++++++++++++------------------------- > > 1 file changed, 36 insertions(+), 36 deletions(-) > > > > diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c > > index a80d64b..e33f84b 100644 > > --- a/drivers/pci/pcie/aspm.c > > +++ b/drivers/pci/pcie/aspm.c > > @@ -422,20 +422,6 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint) > > } > > } > > > > -/* > > - * The L1 PM substate capability is only implemented in function 0 in a > > - * multi function device. > > - */ > > -static struct pci_dev *pci_function_0(struct pci_bus *linkbus) > > -{ > > - struct pci_dev *child; > > - > > - list_for_each_entry(child, &linkbus->devices, bus_list) > > - if (PCI_FUNC(child->devfn) == 0) > > - return child; > > - return NULL; > > -} > > - > > /* Calculate L1.2 PM substate timing parameters */ > > static void aspm_calc_l1ss_info(struct pcie_link_state *link, > > struct aspm_register_info *upreg, > > @@ -798,7 +784,6 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) > > INIT_LIST_HEAD(&link->children); > > INIT_LIST_HEAD(&link->link); > > link->pdev = pdev; > > - link->downstream = pci_function_0(pdev->subordinate); > > > > /* > > * Root Ports and PCI/PCI-X to PCIe Bridges are roots of PCIe > > @@ -828,11 +813,33 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) > > > > static int pci_aspm_init_downstream(struct pci_dev *pdev) > > { > > + struct pcie_link_state *link; > > + struct pci_dev *parent; > > + > > + /* > > + * The L1 PM substate capability is only implemented in function 0 in a > > + * multi function device. > > + */ > > + if (PCI_FUNC(pdev->devfn) != 0) > > + return -EINVAL; > > + > > + parent = pdev->bus->self; > > + if (!parent) > > + return -EINVAL; > > + > > + link = parent->link_state; > > + link->downstream = pdev; > > return 0; > > } > > > > static int pci_aspm_init_upstream(struct pci_dev *pdev) > > { > > + struct pcie_link_state *link; > > + > > + link = alloc_pcie_link_state(pdev); > > + if (!link) > > + return -ENOMEM; > > This allocates the link_state when we enumerate a Downstream Port > instead of when we enumerate a child device. We want the link_state > lifetime to match that of the Downstream Port, so this seems good. > > But we shouldn't at the same time change the link_state deallocation > so it happens when the Downstream Port is removed, not when the child > device is removed? I do see that you change the deallocation in patch [5/5], but I think the deallocation change should be in the same patch as the allocation change. Otherwise I think we have a use-after-free problem in this sequence: # initial enumeration pci_device_add(downstream_port) pci_aspm_init(downstream_port) alloc_pcie_link_state pci_device_add(endpoint) pci_aspm_init(endpoint) # hot-remove endpoint pci_stop_dev(endpoint) pcie_aspm_exit_link_state(endpoint) link = parent->link_state free_link_state(link) # hot-add endpoint pci_aspm_init(endpoint) link = parent->link_state <--- use after free ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 3/5] PCI/ASPM: add init hook to device_add @ 2017-04-13 21:02 ` Bjorn Helgaas 0 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-13 21:02 UTC (permalink / raw) To: linux-arm-kernel On Thu, Apr 13, 2017 at 03:48:00PM -0500, Bjorn Helgaas wrote: > Hi Sinan, > > On Sat, Apr 08, 2017 at 12:55:49AM -0400, Sinan Kaya wrote: > > For bridges, have pcie_aspm_init_link_state() allocate a link_state, > > regardless of whether it currently has any children. > > > > pcie_aspm_init_link_state(): Called for bridges (upstream end of > > link) after all children have been enumerated. No longer needs to > > check aspm_support_enabled or pdev->has_secondary_link or the VIA > > quirk: pci_aspm_init() already checked that stuff, so we only need > > to check pdev->link_state here. > > > > Now that we are calling alloc_pcie_link_state() from pci_aspm_init() > > , get rid of pci_function_0 function and detect downstream link in > > pci_aspm_init_upstream() instead when the function number is 0. > > > > Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 > > Signed-off-by: Sinan Kaya <okaya@codeaurora.org> > > --- > > drivers/pci/pcie/aspm.c | 72 ++++++++++++++++++++++++------------------------- > > 1 file changed, 36 insertions(+), 36 deletions(-) > > > > diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c > > index a80d64b..e33f84b 100644 > > --- a/drivers/pci/pcie/aspm.c > > +++ b/drivers/pci/pcie/aspm.c > > @@ -422,20 +422,6 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint) > > } > > } > > > > -/* > > - * The L1 PM substate capability is only implemented in function 0 in a > > - * multi function device. > > - */ > > -static struct pci_dev *pci_function_0(struct pci_bus *linkbus) > > -{ > > - struct pci_dev *child; > > - > > - list_for_each_entry(child, &linkbus->devices, bus_list) > > - if (PCI_FUNC(child->devfn) == 0) > > - return child; > > - return NULL; > > -} > > - > > /* Calculate L1.2 PM substate timing parameters */ > > static void aspm_calc_l1ss_info(struct pcie_link_state *link, > > struct aspm_register_info *upreg, > > @@ -798,7 +784,6 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) > > INIT_LIST_HEAD(&link->children); > > INIT_LIST_HEAD(&link->link); > > link->pdev = pdev; > > - link->downstream = pci_function_0(pdev->subordinate); > > > > /* > > * Root Ports and PCI/PCI-X to PCIe Bridges are roots of PCIe > > @@ -828,11 +813,33 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) > > > > static int pci_aspm_init_downstream(struct pci_dev *pdev) > > { > > + struct pcie_link_state *link; > > + struct pci_dev *parent; > > + > > + /* > > + * The L1 PM substate capability is only implemented in function 0 in a > > + * multi function device. > > + */ > > + if (PCI_FUNC(pdev->devfn) != 0) > > + return -EINVAL; > > + > > + parent = pdev->bus->self; > > + if (!parent) > > + return -EINVAL; > > + > > + link = parent->link_state; > > + link->downstream = pdev; > > return 0; > > } > > > > static int pci_aspm_init_upstream(struct pci_dev *pdev) > > { > > + struct pcie_link_state *link; > > + > > + link = alloc_pcie_link_state(pdev); > > + if (!link) > > + return -ENOMEM; > > This allocates the link_state when we enumerate a Downstream Port > instead of when we enumerate a child device. We want the link_state > lifetime to match that of the Downstream Port, so this seems good. > > But we shouldn't at the same time change the link_state deallocation > so it happens when the Downstream Port is removed, not when the child > device is removed? I do see that you change the deallocation in patch [5/5], but I think the deallocation change should be in the same patch as the allocation change. Otherwise I think we have a use-after-free problem in this sequence: # initial enumeration pci_device_add(downstream_port) pci_aspm_init(downstream_port) alloc_pcie_link_state pci_device_add(endpoint) pci_aspm_init(endpoint) # hot-remove endpoint pci_stop_dev(endpoint) pcie_aspm_exit_link_state(endpoint) link = parent->link_state free_link_state(link) # hot-add endpoint pci_aspm_init(endpoint) link = parent->link_state <--- use after free ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 3/5] PCI/ASPM: add init hook to device_add @ 2017-04-13 21:02 ` Bjorn Helgaas 0 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-13 21:02 UTC (permalink / raw) To: Sinan Kaya Cc: mayurkumar.patel, David Daney, linux-pci, timur, linux-kernel, Julia Lawall, linux-arm-msm, Bjorn Helgaas, Rajat Jain, linux-arm-kernel On Thu, Apr 13, 2017 at 03:48:00PM -0500, Bjorn Helgaas wrote: > Hi Sinan, > > On Sat, Apr 08, 2017 at 12:55:49AM -0400, Sinan Kaya wrote: > > For bridges, have pcie_aspm_init_link_state() allocate a link_state, > > regardless of whether it currently has any children. > > > > pcie_aspm_init_link_state(): Called for bridges (upstream end of > > link) after all children have been enumerated. No longer needs to > > check aspm_support_enabled or pdev->has_secondary_link or the VIA > > quirk: pci_aspm_init() already checked that stuff, so we only need > > to check pdev->link_state here. > > > > Now that we are calling alloc_pcie_link_state() from pci_aspm_init() > > , get rid of pci_function_0 function and detect downstream link in > > pci_aspm_init_upstream() instead when the function number is 0. > > > > Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 > > Signed-off-by: Sinan Kaya <okaya@codeaurora.org> > > --- > > drivers/pci/pcie/aspm.c | 72 ++++++++++++++++++++++++------------------------- > > 1 file changed, 36 insertions(+), 36 deletions(-) > > > > diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c > > index a80d64b..e33f84b 100644 > > --- a/drivers/pci/pcie/aspm.c > > +++ b/drivers/pci/pcie/aspm.c > > @@ -422,20 +422,6 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint) > > } > > } > > > > -/* > > - * The L1 PM substate capability is only implemented in function 0 in a > > - * multi function device. > > - */ > > -static struct pci_dev *pci_function_0(struct pci_bus *linkbus) > > -{ > > - struct pci_dev *child; > > - > > - list_for_each_entry(child, &linkbus->devices, bus_list) > > - if (PCI_FUNC(child->devfn) == 0) > > - return child; > > - return NULL; > > -} > > - > > /* Calculate L1.2 PM substate timing parameters */ > > static void aspm_calc_l1ss_info(struct pcie_link_state *link, > > struct aspm_register_info *upreg, > > @@ -798,7 +784,6 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) > > INIT_LIST_HEAD(&link->children); > > INIT_LIST_HEAD(&link->link); > > link->pdev = pdev; > > - link->downstream = pci_function_0(pdev->subordinate); > > > > /* > > * Root Ports and PCI/PCI-X to PCIe Bridges are roots of PCIe > > @@ -828,11 +813,33 @@ static struct pcie_link_state *alloc_pcie_link_state(struct pci_dev *pdev) > > > > static int pci_aspm_init_downstream(struct pci_dev *pdev) > > { > > + struct pcie_link_state *link; > > + struct pci_dev *parent; > > + > > + /* > > + * The L1 PM substate capability is only implemented in function 0 in a > > + * multi function device. > > + */ > > + if (PCI_FUNC(pdev->devfn) != 0) > > + return -EINVAL; > > + > > + parent = pdev->bus->self; > > + if (!parent) > > + return -EINVAL; > > + > > + link = parent->link_state; > > + link->downstream = pdev; > > return 0; > > } > > > > static int pci_aspm_init_upstream(struct pci_dev *pdev) > > { > > + struct pcie_link_state *link; > > + > > + link = alloc_pcie_link_state(pdev); > > + if (!link) > > + return -ENOMEM; > > This allocates the link_state when we enumerate a Downstream Port > instead of when we enumerate a child device. We want the link_state > lifetime to match that of the Downstream Port, so this seems good. > > But we shouldn't at the same time change the link_state deallocation > so it happens when the Downstream Port is removed, not when the child > device is removed? I do see that you change the deallocation in patch [5/5], but I think the deallocation change should be in the same patch as the allocation change. Otherwise I think we have a use-after-free problem in this sequence: # initial enumeration pci_device_add(downstream_port) pci_aspm_init(downstream_port) alloc_pcie_link_state pci_device_add(endpoint) pci_aspm_init(endpoint) # hot-remove endpoint pci_stop_dev(endpoint) pcie_aspm_exit_link_state(endpoint) link = parent->link_state free_link_state(link) # hot-add endpoint pci_aspm_init(endpoint) link = parent->link_state <--- use after free _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 3/5] PCI/ASPM: add init hook to device_add 2017-04-13 21:02 ` Bjorn Helgaas @ 2017-04-14 1:19 ` Sinan Kaya -1 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-14 1:19 UTC (permalink / raw) To: Bjorn Helgaas Cc: mayurkumar.patel, David Daney, linux-pci, timur, linux-kernel, Julia Lawall, linux-arm-msm, Bjorn Helgaas, Rajat Jain, linux-arm-kernel On 4/13/2017 5:02 PM, Bjorn Helgaas wrote: > I do see that you change the deallocation in patch [5/5], but I think > the deallocation change should be in the same patch as the allocation > change. Otherwise I think we have a use-after-free problem in this > sequence: Sure, I'll reorder. As you can see here, link will be only removed if root port is being removed. Without this, we'll hit the use after free issue you mentioned. if (pdev->has_secondary_link) { link = pdev->link_state; down_read(&pci_bus_sem); mutex_lock(&aspm_lock); list_del(&link->sibling); list_del(&link->link); /* Clock PM is for endpoint device */ free_link_state(link); mutex_unlock(&aspm_lock); up_read(&pci_bus_sem); return; } -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 3/5] PCI/ASPM: add init hook to device_add @ 2017-04-14 1:19 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-14 1:19 UTC (permalink / raw) To: linux-arm-kernel On 4/13/2017 5:02 PM, Bjorn Helgaas wrote: > I do see that you change the deallocation in patch [5/5], but I think > the deallocation change should be in the same patch as the allocation > change. Otherwise I think we have a use-after-free problem in this > sequence: Sure, I'll reorder. As you can see here, link will be only removed if root port is being removed. Without this, we'll hit the use after free issue you mentioned. if (pdev->has_secondary_link) { link = pdev->link_state; down_read(&pci_bus_sem); mutex_lock(&aspm_lock); list_del(&link->sibling); list_del(&link->link); /* Clock PM is for endpoint device */ free_link_state(link); mutex_unlock(&aspm_lock); up_read(&pci_bus_sem); return; } -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 3/5] PCI/ASPM: add init hook to device_add 2017-04-14 1:19 ` Sinan Kaya @ 2017-04-14 1:30 ` Bjorn Helgaas -1 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-14 1:30 UTC (permalink / raw) To: Sinan Kaya Cc: Bjorn Helgaas, Patel, Mayurkumar, David Daney, linux-pci, Timur Tabi, linux-kernel, Julia Lawall, linux-arm-msm, Rajat Jain, linux-arm On Thu, Apr 13, 2017 at 8:19 PM, Sinan Kaya <okaya@codeaurora.org> wrote: > On 4/13/2017 5:02 PM, Bjorn Helgaas wrote: >> I do see that you change the deallocation in patch [5/5], but I think >> the deallocation change should be in the same patch as the allocation >> change. Otherwise I think we have a use-after-free problem in this >> sequence: > > Sure, I'll reorder. As you can see here, link will be only removed if > root port is being removed. > > Without this, we'll hit the use after free issue you mentioned. > > if (pdev->has_secondary_link) { > link = pdev->link_state; > down_read(&pci_bus_sem); > mutex_lock(&aspm_lock); > > list_del(&link->sibling); > list_del(&link->link); > > /* Clock PM is for endpoint device */ > free_link_state(link); > mutex_unlock(&aspm_lock); > up_read(&pci_bus_sem); > return; > } Right, but this "pdev->has_secondary_link" check is in your new code and doesn't show up until patch [5/5]. As of *this* patch [3/5], the link is removed when the endpoint is removed, so we could hit the use-after-free. Granted, we'll only be susceptible to this while bisecting because normally all the patches will be applied. But I think this patch will make more sense and be easier to review if it makes the link_state lifetime match the Port's lifetime. Bjorn ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 3/5] PCI/ASPM: add init hook to device_add @ 2017-04-14 1:30 ` Bjorn Helgaas 0 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-14 1:30 UTC (permalink / raw) To: linux-arm-kernel On Thu, Apr 13, 2017 at 8:19 PM, Sinan Kaya <okaya@codeaurora.org> wrote: > On 4/13/2017 5:02 PM, Bjorn Helgaas wrote: >> I do see that you change the deallocation in patch [5/5], but I think >> the deallocation change should be in the same patch as the allocation >> change. Otherwise I think we have a use-after-free problem in this >> sequence: > > Sure, I'll reorder. As you can see here, link will be only removed if > root port is being removed. > > Without this, we'll hit the use after free issue you mentioned. > > if (pdev->has_secondary_link) { > link = pdev->link_state; > down_read(&pci_bus_sem); > mutex_lock(&aspm_lock); > > list_del(&link->sibling); > list_del(&link->link); > > /* Clock PM is for endpoint device */ > free_link_state(link); > mutex_unlock(&aspm_lock); > up_read(&pci_bus_sem); > return; > } Right, but this "pdev->has_secondary_link" check is in your new code and doesn't show up until patch [5/5]. As of *this* patch [3/5], the link is removed when the endpoint is removed, so we could hit the use-after-free. Granted, we'll only be susceptible to this while bisecting because normally all the patches will be applied. But I think this patch will make more sense and be easier to review if it makes the link_state lifetime match the Port's lifetime. Bjorn ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init 2017-04-08 4:55 ` Sinan Kaya (?) @ 2017-04-08 4:55 ` Sinan Kaya -1 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-08 4:55 UTC (permalink / raw) To: linux-pci, timur Cc: mayurkumar.patel, linux-arm-msm, linux-arm-kernel, Sinan Kaya, Bjorn Helgaas, Rajat Jain, Yinghai Lu, David Daney, Shawn Lin, Julia Lawall, linux-kernel Now that we added a hook to be called from device_add, save the default values from the HW registers early in the boot for further reuse during hot device add/remove operations. If the link is down during boot, assume that we want to enable L0s and L1 following hotplug insertion as well as L1SS if supported. Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 Signed-off-by: Sinan Kaya <okaya@codeaurora.org> --- drivers/pci/pcie/aspm.c | 34 ++++++++++++++++++++++++++++++---- 1 file changed, 30 insertions(+), 4 deletions(-) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index e33f84b..c7da087 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -505,8 +505,10 @@ static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist) */ if (dwreg.support & upreg.support & PCIE_LINK_STATE_L0S) link->aspm_support |= ASPM_STATE_L0S; - if (dwreg.enabled & PCIE_LINK_STATE_L0S) + if (dwreg.enabled & PCIE_LINK_STATE_L0S) { link->aspm_enabled |= ASPM_STATE_L0S_UP; + link->aspm_default |= ASPM_STATE_L0S_UP; + } if (upreg.enabled & PCIE_LINK_STATE_L0S) link->aspm_enabled |= ASPM_STATE_L0S_DW; link->latency_up.l0s = calc_l0s_latency(upreg.latency_encoding_l0s); @@ -542,9 +544,6 @@ static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist) if (link->aspm_support & ASPM_STATE_L1SS) aspm_calc_l1ss_info(link, &upreg, &dwreg); - /* Save default state */ - link->aspm_default = link->aspm_enabled; - /* Setup initial capable state. Will be updated later */ link->aspm_capable = link->aspm_support; /* @@ -835,11 +834,38 @@ static int pci_aspm_init_downstream(struct pci_dev *pdev) static int pci_aspm_init_upstream(struct pci_dev *pdev) { struct pcie_link_state *link; + struct aspm_register_info upreg; + u16 lnk_status; + bool ret; link = alloc_pcie_link_state(pdev); if (!link) return -ENOMEM; + pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &lnk_status); + ret = !!(lnk_status & PCI_EXP_LNKSTA_DLLLA); + + if (ret) { + pcie_get_aspm_reg(pdev, &upreg); + if (upreg.enabled & PCIE_LINK_STATE_L0S) + link->aspm_default |= ASPM_STATE_L0S_DW; + if (upreg.enabled & PCIE_LINK_STATE_L1) + link->aspm_default |= ASPM_STATE_L1; + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_ASPM_L1_1) + link->aspm_default |= ASPM_STATE_L1_1; + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_ASPM_L1_2) + link->aspm_default |= ASPM_STATE_L1_2; + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_PCIPM_L1_1) + link->aspm_default |= ASPM_STATE_L1_1_PCIPM; + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_PCIPM_L1_2) + link->aspm_default |= ASPM_STATE_L1_2_PCIPM; + } else { + if (!pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_L1SS)) + link->aspm_default = ASPM_STATE_L0S | ASPM_STATE_L1; + else + link->aspm_default = ASPM_STATE_ALL; + } + return 0; } -- 1.9.1 ^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-04-08 4:55 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-08 4:55 UTC (permalink / raw) To: linux-arm-kernel Now that we added a hook to be called from device_add, save the default values from the HW registers early in the boot for further reuse during hot device add/remove operations. If the link is down during boot, assume that we want to enable L0s and L1 following hotplug insertion as well as L1SS if supported. Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 Signed-off-by: Sinan Kaya <okaya@codeaurora.org> --- drivers/pci/pcie/aspm.c | 34 ++++++++++++++++++++++++++++++---- 1 file changed, 30 insertions(+), 4 deletions(-) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index e33f84b..c7da087 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -505,8 +505,10 @@ static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist) */ if (dwreg.support & upreg.support & PCIE_LINK_STATE_L0S) link->aspm_support |= ASPM_STATE_L0S; - if (dwreg.enabled & PCIE_LINK_STATE_L0S) + if (dwreg.enabled & PCIE_LINK_STATE_L0S) { link->aspm_enabled |= ASPM_STATE_L0S_UP; + link->aspm_default |= ASPM_STATE_L0S_UP; + } if (upreg.enabled & PCIE_LINK_STATE_L0S) link->aspm_enabled |= ASPM_STATE_L0S_DW; link->latency_up.l0s = calc_l0s_latency(upreg.latency_encoding_l0s); @@ -542,9 +544,6 @@ static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist) if (link->aspm_support & ASPM_STATE_L1SS) aspm_calc_l1ss_info(link, &upreg, &dwreg); - /* Save default state */ - link->aspm_default = link->aspm_enabled; - /* Setup initial capable state. Will be updated later */ link->aspm_capable = link->aspm_support; /* @@ -835,11 +834,38 @@ static int pci_aspm_init_downstream(struct pci_dev *pdev) static int pci_aspm_init_upstream(struct pci_dev *pdev) { struct pcie_link_state *link; + struct aspm_register_info upreg; + u16 lnk_status; + bool ret; link = alloc_pcie_link_state(pdev); if (!link) return -ENOMEM; + pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &lnk_status); + ret = !!(lnk_status & PCI_EXP_LNKSTA_DLLLA); + + if (ret) { + pcie_get_aspm_reg(pdev, &upreg); + if (upreg.enabled & PCIE_LINK_STATE_L0S) + link->aspm_default |= ASPM_STATE_L0S_DW; + if (upreg.enabled & PCIE_LINK_STATE_L1) + link->aspm_default |= ASPM_STATE_L1; + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_ASPM_L1_1) + link->aspm_default |= ASPM_STATE_L1_1; + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_ASPM_L1_2) + link->aspm_default |= ASPM_STATE_L1_2; + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_PCIPM_L1_1) + link->aspm_default |= ASPM_STATE_L1_1_PCIPM; + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_PCIPM_L1_2) + link->aspm_default |= ASPM_STATE_L1_2_PCIPM; + } else { + if (!pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_L1SS)) + link->aspm_default = ASPM_STATE_L0S | ASPM_STATE_L1; + else + link->aspm_default = ASPM_STATE_ALL; + } + return 0; } -- 1.9.1 ^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-04-08 4:55 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-08 4:55 UTC (permalink / raw) To: linux-pci, timur Cc: mayurkumar.patel, David Daney, linux-arm-msm, Shawn Lin, linux-kernel, Sinan Kaya, Julia Lawall, Bjorn Helgaas, Rajat Jain, Yinghai Lu, linux-arm-kernel Now that we added a hook to be called from device_add, save the default values from the HW registers early in the boot for further reuse during hot device add/remove operations. If the link is down during boot, assume that we want to enable L0s and L1 following hotplug insertion as well as L1SS if supported. Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 Signed-off-by: Sinan Kaya <okaya@codeaurora.org> --- drivers/pci/pcie/aspm.c | 34 ++++++++++++++++++++++++++++++---- 1 file changed, 30 insertions(+), 4 deletions(-) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index e33f84b..c7da087 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -505,8 +505,10 @@ static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist) */ if (dwreg.support & upreg.support & PCIE_LINK_STATE_L0S) link->aspm_support |= ASPM_STATE_L0S; - if (dwreg.enabled & PCIE_LINK_STATE_L0S) + if (dwreg.enabled & PCIE_LINK_STATE_L0S) { link->aspm_enabled |= ASPM_STATE_L0S_UP; + link->aspm_default |= ASPM_STATE_L0S_UP; + } if (upreg.enabled & PCIE_LINK_STATE_L0S) link->aspm_enabled |= ASPM_STATE_L0S_DW; link->latency_up.l0s = calc_l0s_latency(upreg.latency_encoding_l0s); @@ -542,9 +544,6 @@ static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist) if (link->aspm_support & ASPM_STATE_L1SS) aspm_calc_l1ss_info(link, &upreg, &dwreg); - /* Save default state */ - link->aspm_default = link->aspm_enabled; - /* Setup initial capable state. Will be updated later */ link->aspm_capable = link->aspm_support; /* @@ -835,11 +834,38 @@ static int pci_aspm_init_downstream(struct pci_dev *pdev) static int pci_aspm_init_upstream(struct pci_dev *pdev) { struct pcie_link_state *link; + struct aspm_register_info upreg; + u16 lnk_status; + bool ret; link = alloc_pcie_link_state(pdev); if (!link) return -ENOMEM; + pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &lnk_status); + ret = !!(lnk_status & PCI_EXP_LNKSTA_DLLLA); + + if (ret) { + pcie_get_aspm_reg(pdev, &upreg); + if (upreg.enabled & PCIE_LINK_STATE_L0S) + link->aspm_default |= ASPM_STATE_L0S_DW; + if (upreg.enabled & PCIE_LINK_STATE_L1) + link->aspm_default |= ASPM_STATE_L1; + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_ASPM_L1_1) + link->aspm_default |= ASPM_STATE_L1_1; + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_ASPM_L1_2) + link->aspm_default |= ASPM_STATE_L1_2; + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_PCIPM_L1_1) + link->aspm_default |= ASPM_STATE_L1_1_PCIPM; + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_PCIPM_L1_2) + link->aspm_default |= ASPM_STATE_L1_2_PCIPM; + } else { + if (!pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_L1SS)) + link->aspm_default = ASPM_STATE_L0S | ASPM_STATE_L1; + else + link->aspm_default = ASPM_STATE_ALL; + } + return 0; } -- 1.9.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 94+ messages in thread
* Re: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init 2017-04-08 4:55 ` Sinan Kaya (?) @ 2017-04-12 19:19 ` Rajat Jain -1 siblings, 0 replies; 94+ messages in thread From: Rajat Jain @ 2017-04-12 19:19 UTC (permalink / raw) To: Sinan Kaya Cc: linux-pci, timur, Patel, Mayurkumar, linux-arm-msm, linux-arm-kernel, Bjorn Helgaas, Yinghai Lu, David Daney, Shawn Lin, Julia Lawall, linux-kernel, Rajat Jain On Fri, Apr 7, 2017 at 9:55 PM, Sinan Kaya <okaya@codeaurora.org> wrote: > Now that we added a hook to be called from device_add, save the > default values from the HW registers early in the boot for further > reuse during hot device add/remove operations. > > If the link is down during boot, assume that we want to enable L0s > and L1 following hotplug insertion as well as L1SS if supported. IIUC, so far POLICY_DEFAULT meant that we'd just use & follow what BIOS has done, and play it safe (never try to be more opportunistic). With this change however, we'd be slightly overstepping and giving ourselves benefit of doubt if the BIOS could not enable ASPM states because the link was not up. This may be good, but I think we should call it out, and add some more elaborate comment on the POLICY_DEFAULT description (what to, and what not to expect in different situations). It is important because existing systems today, that used to boot without cards and later hotplugged them, didn't have ASPM states enabled. They will now suddenly start seeing all ASPM states enabled including L1 substates for the first time (if supported). My system is not hotplug capable (I have the EP soldered on board, so couldn't do much testing, except for sanity. Please feel free to use my Reviewed-by. > > Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 > Signed-off-by: Sinan Kaya <okaya@codeaurora.org> > --- > drivers/pci/pcie/aspm.c | 34 ++++++++++++++++++++++++++++++---- > 1 file changed, 30 insertions(+), 4 deletions(-) > > diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c > index e33f84b..c7da087 100644 > --- a/drivers/pci/pcie/aspm.c > +++ b/drivers/pci/pcie/aspm.c > @@ -505,8 +505,10 @@ static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist) > */ > if (dwreg.support & upreg.support & PCIE_LINK_STATE_L0S) > link->aspm_support |= ASPM_STATE_L0S; > - if (dwreg.enabled & PCIE_LINK_STATE_L0S) > + if (dwreg.enabled & PCIE_LINK_STATE_L0S) { > link->aspm_enabled |= ASPM_STATE_L0S_UP; > + link->aspm_default |= ASPM_STATE_L0S_UP; > + } > if (upreg.enabled & PCIE_LINK_STATE_L0S) > link->aspm_enabled |= ASPM_STATE_L0S_DW; > link->latency_up.l0s = calc_l0s_latency(upreg.latency_encoding_l0s); > @@ -542,9 +544,6 @@ static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist) > if (link->aspm_support & ASPM_STATE_L1SS) > aspm_calc_l1ss_info(link, &upreg, &dwreg); > > - /* Save default state */ > - link->aspm_default = link->aspm_enabled; > - > /* Setup initial capable state. Will be updated later */ > link->aspm_capable = link->aspm_support; > /* > @@ -835,11 +834,38 @@ static int pci_aspm_init_downstream(struct pci_dev *pdev) > static int pci_aspm_init_upstream(struct pci_dev *pdev) > { > struct pcie_link_state *link; > + struct aspm_register_info upreg; > + u16 lnk_status; > + bool ret; > > link = alloc_pcie_link_state(pdev); > if (!link) > return -ENOMEM; > > + pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &lnk_status); > + ret = !!(lnk_status & PCI_EXP_LNKSTA_DLLLA); > + > + if (ret) { > + pcie_get_aspm_reg(pdev, &upreg); > + if (upreg.enabled & PCIE_LINK_STATE_L0S) > + link->aspm_default |= ASPM_STATE_L0S_DW; > + if (upreg.enabled & PCIE_LINK_STATE_L1) > + link->aspm_default |= ASPM_STATE_L1; > + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_ASPM_L1_1) > + link->aspm_default |= ASPM_STATE_L1_1; > + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_ASPM_L1_2) > + link->aspm_default |= ASPM_STATE_L1_2; > + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_PCIPM_L1_1) > + link->aspm_default |= ASPM_STATE_L1_1_PCIPM; > + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_PCIPM_L1_2) > + link->aspm_default |= ASPM_STATE_L1_2_PCIPM; > + } else { > + if (!pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_L1SS)) > + link->aspm_default = ASPM_STATE_L0S | ASPM_STATE_L1; > + else > + link->aspm_default = ASPM_STATE_ALL; > + } Optional: May be consider moving this code (more aptly) to pcie_aspm_cap_init() by adding a check for link-up before we start reading downstream registers there? I guess you'll need to move the call to pcie_aspm_cap_init() a little further up in pcie_aspm_init_link_state(). > + > return 0; > } > > -- > 1.9.1 > ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-04-12 19:19 ` Rajat Jain 0 siblings, 0 replies; 94+ messages in thread From: Rajat Jain @ 2017-04-12 19:19 UTC (permalink / raw) To: linux-arm-kernel On Fri, Apr 7, 2017 at 9:55 PM, Sinan Kaya <okaya@codeaurora.org> wrote: > Now that we added a hook to be called from device_add, save the > default values from the HW registers early in the boot for further > reuse during hot device add/remove operations. > > If the link is down during boot, assume that we want to enable L0s > and L1 following hotplug insertion as well as L1SS if supported. IIUC, so far POLICY_DEFAULT meant that we'd just use & follow what BIOS has done, and play it safe (never try to be more opportunistic). With this change however, we'd be slightly overstepping and giving ourselves benefit of doubt if the BIOS could not enable ASPM states because the link was not up. This may be good, but I think we should call it out, and add some more elaborate comment on the POLICY_DEFAULT description (what to, and what not to expect in different situations). It is important because existing systems today, that used to boot without cards and later hotplugged them, didn't have ASPM states enabled. They will now suddenly start seeing all ASPM states enabled including L1 substates for the first time (if supported). My system is not hotplug capable (I have the EP soldered on board, so couldn't do much testing, except for sanity. Please feel free to use my Reviewed-by. > > Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 > Signed-off-by: Sinan Kaya <okaya@codeaurora.org> > --- > drivers/pci/pcie/aspm.c | 34 ++++++++++++++++++++++++++++++---- > 1 file changed, 30 insertions(+), 4 deletions(-) > > diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c > index e33f84b..c7da087 100644 > --- a/drivers/pci/pcie/aspm.c > +++ b/drivers/pci/pcie/aspm.c > @@ -505,8 +505,10 @@ static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist) > */ > if (dwreg.support & upreg.support & PCIE_LINK_STATE_L0S) > link->aspm_support |= ASPM_STATE_L0S; > - if (dwreg.enabled & PCIE_LINK_STATE_L0S) > + if (dwreg.enabled & PCIE_LINK_STATE_L0S) { > link->aspm_enabled |= ASPM_STATE_L0S_UP; > + link->aspm_default |= ASPM_STATE_L0S_UP; > + } > if (upreg.enabled & PCIE_LINK_STATE_L0S) > link->aspm_enabled |= ASPM_STATE_L0S_DW; > link->latency_up.l0s = calc_l0s_latency(upreg.latency_encoding_l0s); > @@ -542,9 +544,6 @@ static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist) > if (link->aspm_support & ASPM_STATE_L1SS) > aspm_calc_l1ss_info(link, &upreg, &dwreg); > > - /* Save default state */ > - link->aspm_default = link->aspm_enabled; > - > /* Setup initial capable state. Will be updated later */ > link->aspm_capable = link->aspm_support; > /* > @@ -835,11 +834,38 @@ static int pci_aspm_init_downstream(struct pci_dev *pdev) > static int pci_aspm_init_upstream(struct pci_dev *pdev) > { > struct pcie_link_state *link; > + struct aspm_register_info upreg; > + u16 lnk_status; > + bool ret; > > link = alloc_pcie_link_state(pdev); > if (!link) > return -ENOMEM; > > + pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &lnk_status); > + ret = !!(lnk_status & PCI_EXP_LNKSTA_DLLLA); > + > + if (ret) { > + pcie_get_aspm_reg(pdev, &upreg); > + if (upreg.enabled & PCIE_LINK_STATE_L0S) > + link->aspm_default |= ASPM_STATE_L0S_DW; > + if (upreg.enabled & PCIE_LINK_STATE_L1) > + link->aspm_default |= ASPM_STATE_L1; > + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_ASPM_L1_1) > + link->aspm_default |= ASPM_STATE_L1_1; > + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_ASPM_L1_2) > + link->aspm_default |= ASPM_STATE_L1_2; > + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_PCIPM_L1_1) > + link->aspm_default |= ASPM_STATE_L1_1_PCIPM; > + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_PCIPM_L1_2) > + link->aspm_default |= ASPM_STATE_L1_2_PCIPM; > + } else { > + if (!pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_L1SS)) > + link->aspm_default = ASPM_STATE_L0S | ASPM_STATE_L1; > + else > + link->aspm_default = ASPM_STATE_ALL; > + } Optional: May be consider moving this code (more aptly) to pcie_aspm_cap_init() by adding a check for link-up before we start reading downstream registers there? I guess you'll need to move the call to pcie_aspm_cap_init() a little further up in pcie_aspm_init_link_state(). > + > return 0; > } > > -- > 1.9.1 > ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-04-12 19:19 ` Rajat Jain 0 siblings, 0 replies; 94+ messages in thread From: Rajat Jain @ 2017-04-12 19:19 UTC (permalink / raw) To: Sinan Kaya Cc: Patel, Mayurkumar, Rajat Jain, David Daney, linux-pci, timur, linux-kernel, Julia Lawall, linux-arm-msm, Bjorn Helgaas, Yinghai Lu, Shawn Lin, linux-arm-kernel On Fri, Apr 7, 2017 at 9:55 PM, Sinan Kaya <okaya@codeaurora.org> wrote: > Now that we added a hook to be called from device_add, save the > default values from the HW registers early in the boot for further > reuse during hot device add/remove operations. > > If the link is down during boot, assume that we want to enable L0s > and L1 following hotplug insertion as well as L1SS if supported. IIUC, so far POLICY_DEFAULT meant that we'd just use & follow what BIOS has done, and play it safe (never try to be more opportunistic). With this change however, we'd be slightly overstepping and giving ourselves benefit of doubt if the BIOS could not enable ASPM states because the link was not up. This may be good, but I think we should call it out, and add some more elaborate comment on the POLICY_DEFAULT description (what to, and what not to expect in different situations). It is important because existing systems today, that used to boot without cards and later hotplugged them, didn't have ASPM states enabled. They will now suddenly start seeing all ASPM states enabled including L1 substates for the first time (if supported). My system is not hotplug capable (I have the EP soldered on board, so couldn't do much testing, except for sanity. Please feel free to use my Reviewed-by. > > Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 > Signed-off-by: Sinan Kaya <okaya@codeaurora.org> > --- > drivers/pci/pcie/aspm.c | 34 ++++++++++++++++++++++++++++++---- > 1 file changed, 30 insertions(+), 4 deletions(-) > > diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c > index e33f84b..c7da087 100644 > --- a/drivers/pci/pcie/aspm.c > +++ b/drivers/pci/pcie/aspm.c > @@ -505,8 +505,10 @@ static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist) > */ > if (dwreg.support & upreg.support & PCIE_LINK_STATE_L0S) > link->aspm_support |= ASPM_STATE_L0S; > - if (dwreg.enabled & PCIE_LINK_STATE_L0S) > + if (dwreg.enabled & PCIE_LINK_STATE_L0S) { > link->aspm_enabled |= ASPM_STATE_L0S_UP; > + link->aspm_default |= ASPM_STATE_L0S_UP; > + } > if (upreg.enabled & PCIE_LINK_STATE_L0S) > link->aspm_enabled |= ASPM_STATE_L0S_DW; > link->latency_up.l0s = calc_l0s_latency(upreg.latency_encoding_l0s); > @@ -542,9 +544,6 @@ static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist) > if (link->aspm_support & ASPM_STATE_L1SS) > aspm_calc_l1ss_info(link, &upreg, &dwreg); > > - /* Save default state */ > - link->aspm_default = link->aspm_enabled; > - > /* Setup initial capable state. Will be updated later */ > link->aspm_capable = link->aspm_support; > /* > @@ -835,11 +834,38 @@ static int pci_aspm_init_downstream(struct pci_dev *pdev) > static int pci_aspm_init_upstream(struct pci_dev *pdev) > { > struct pcie_link_state *link; > + struct aspm_register_info upreg; > + u16 lnk_status; > + bool ret; > > link = alloc_pcie_link_state(pdev); > if (!link) > return -ENOMEM; > > + pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &lnk_status); > + ret = !!(lnk_status & PCI_EXP_LNKSTA_DLLLA); > + > + if (ret) { > + pcie_get_aspm_reg(pdev, &upreg); > + if (upreg.enabled & PCIE_LINK_STATE_L0S) > + link->aspm_default |= ASPM_STATE_L0S_DW; > + if (upreg.enabled & PCIE_LINK_STATE_L1) > + link->aspm_default |= ASPM_STATE_L1; > + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_ASPM_L1_1) > + link->aspm_default |= ASPM_STATE_L1_1; > + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_ASPM_L1_2) > + link->aspm_default |= ASPM_STATE_L1_2; > + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_PCIPM_L1_1) > + link->aspm_default |= ASPM_STATE_L1_1_PCIPM; > + if (upreg.l1ss_ctl1 & PCI_L1SS_CTL1_PCIPM_L1_2) > + link->aspm_default |= ASPM_STATE_L1_2_PCIPM; > + } else { > + if (!pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_L1SS)) > + link->aspm_default = ASPM_STATE_L0S | ASPM_STATE_L1; > + else > + link->aspm_default = ASPM_STATE_ALL; > + } Optional: May be consider moving this code (more aptly) to pcie_aspm_cap_init() by adding a check for link-up before we start reading downstream registers there? I guess you'll need to move the call to pcie_aspm_cap_init() a little further up in pcie_aspm_init_link_state(). > + > return 0; > } > > -- > 1.9.1 > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init 2017-04-12 19:19 ` Rajat Jain (?) @ 2017-04-14 19:12 ` Sinan Kaya -1 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-14 19:12 UTC (permalink / raw) To: Rajat Jain Cc: linux-pci, timur, Patel, Mayurkumar, linux-arm-msm, linux-arm-kernel, Bjorn Helgaas, Yinghai Lu, David Daney, Shawn Lin, Julia Lawall, linux-kernel, Rajat Jain Bjorn, On 4/12/2017 3:19 PM, Rajat Jain wrote: > On Fri, Apr 7, 2017 at 9:55 PM, Sinan Kaya <okaya@codeaurora.org> wrote: >> Now that we added a hook to be called from device_add, save the >> default values from the HW registers early in the boot for further >> reuse during hot device add/remove operations. >> >> If the link is down during boot, assume that we want to enable L0s >> and L1 following hotplug insertion as well as L1SS if supported. > > IIUC, so far POLICY_DEFAULT meant that we'd just use & follow what > BIOS has done, and play it safe (never try to be more opportunistic). > With this change however, we'd be slightly overstepping and giving > ourselves benefit of doubt if the BIOS could not enable ASPM states > because the link was not up. This may be good, but I think we should > call it out, and add some more elaborate comment on the POLICY_DEFAULT > description (what to, and what not to expect in different situations). > > It is important because existing systems today, that used to boot > without cards and later hotplugged them, didn't have ASPM states > enabled. They will now suddenly start seeing all ASPM states enabled > including L1 substates for the first time (if supported). > Rajat has a good point here. Would you like me to update the ASPM document with this new behavior for hotplug? Do you have another behavior preference when it comes this? Sinan -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-04-14 19:12 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-14 19:12 UTC (permalink / raw) To: linux-arm-kernel Bjorn, On 4/12/2017 3:19 PM, Rajat Jain wrote: > On Fri, Apr 7, 2017 at 9:55 PM, Sinan Kaya <okaya@codeaurora.org> wrote: >> Now that we added a hook to be called from device_add, save the >> default values from the HW registers early in the boot for further >> reuse during hot device add/remove operations. >> >> If the link is down during boot, assume that we want to enable L0s >> and L1 following hotplug insertion as well as L1SS if supported. > > IIUC, so far POLICY_DEFAULT meant that we'd just use & follow what > BIOS has done, and play it safe (never try to be more opportunistic). > With this change however, we'd be slightly overstepping and giving > ourselves benefit of doubt if the BIOS could not enable ASPM states > because the link was not up. This may be good, but I think we should > call it out, and add some more elaborate comment on the POLICY_DEFAULT > description (what to, and what not to expect in different situations). > > It is important because existing systems today, that used to boot > without cards and later hotplugged them, didn't have ASPM states > enabled. They will now suddenly start seeing all ASPM states enabled > including L1 substates for the first time (if supported). > Rajat has a good point here. Would you like me to update the ASPM document with this new behavior for hotplug? Do you have another behavior preference when it comes this? Sinan -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-04-14 19:12 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-14 19:12 UTC (permalink / raw) To: Rajat Jain Cc: Patel, Mayurkumar, Rajat Jain, David Daney, linux-pci, timur, linux-kernel, Julia Lawall, linux-arm-msm, Bjorn Helgaas, Yinghai Lu, Shawn Lin, linux-arm-kernel Bjorn, On 4/12/2017 3:19 PM, Rajat Jain wrote: > On Fri, Apr 7, 2017 at 9:55 PM, Sinan Kaya <okaya@codeaurora.org> wrote: >> Now that we added a hook to be called from device_add, save the >> default values from the HW registers early in the boot for further >> reuse during hot device add/remove operations. >> >> If the link is down during boot, assume that we want to enable L0s >> and L1 following hotplug insertion as well as L1SS if supported. > > IIUC, so far POLICY_DEFAULT meant that we'd just use & follow what > BIOS has done, and play it safe (never try to be more opportunistic). > With this change however, we'd be slightly overstepping and giving > ourselves benefit of doubt if the BIOS could not enable ASPM states > because the link was not up. This may be good, but I think we should > call it out, and add some more elaborate comment on the POLICY_DEFAULT > description (what to, and what not to expect in different situations). > > It is important because existing systems today, that used to boot > without cards and later hotplugged them, didn't have ASPM states > enabled. They will now suddenly start seeing all ASPM states enabled > including L1 substates for the first time (if supported). > Rajat has a good point here. Would you like me to update the ASPM document with this new behavior for hotplug? Do you have another behavior preference when it comes this? Sinan -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init 2017-04-14 19:12 ` Sinan Kaya (?) @ 2017-04-14 21:44 ` Bjorn Helgaas -1 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-14 21:44 UTC (permalink / raw) To: Sinan Kaya Cc: Rajat Jain, Patel, Mayurkumar, Rajat Jain, David Daney, linux-pci, timur, linux-kernel, Julia Lawall, linux-arm-msm, Bjorn Helgaas, Yinghai Lu, Shawn Lin, linux-arm-kernel, Myron Stowe [+cc Myron, lkml] On Fri, Apr 14, 2017 at 03:12:35PM -0400, Sinan Kaya wrote: > Bjorn, > > On 4/12/2017 3:19 PM, Rajat Jain wrote: > > On Fri, Apr 7, 2017 at 9:55 PM, Sinan Kaya <okaya@codeaurora.org> wrote: > >> Now that we added a hook to be called from device_add, save the > >> default values from the HW registers early in the boot for further > >> reuse during hot device add/remove operations. > >> > >> If the link is down during boot, assume that we want to enable L0s > >> and L1 following hotplug insertion as well as L1SS if supported. > > > > IIUC, so far POLICY_DEFAULT meant that we'd just use & follow what > > BIOS has done, and play it safe (never try to be more opportunistic). > > With this change however, we'd be slightly overstepping and giving > > ourselves benefit of doubt if the BIOS could not enable ASPM states > > because the link was not up. This may be good, but I think we should > > call it out, and add some more elaborate comment on the POLICY_DEFAULT > > description (what to, and what not to expect in different situations). > > > > It is important because existing systems today, that used to boot > > without cards and later hotplugged them, didn't have ASPM states > > enabled. They will now suddenly start seeing all ASPM states enabled > > including L1 substates for the first time (if supported). > > > > Rajat has a good point here. Would you like me to update the ASPM document > with this new behavior for hotplug? > > Do you have another behavior preference when it comes this? That *is* a very good point. I think the change in behavior could be surprising. I wonder if we should revise our understanding of what POLICY_DEFAULT means. If we decided it means "the kernel never changes any ASPM config", it would be clear that we keep the BIOS configuration for everything present at boot, and we don't enable ASPM for any hot-added devices. I think the motivation for this series is to apply the BIOS's power management policy to hot-added devices. There's no direct way to know the BIOS's policy, so we're trying to infer it from the boot-time link configurations. Should we even *try* to apply the BIOS's policy? I don't know. If a platform really wanted to maintain control over ASPM and apply its policy consistently, I think it could do that by setting ACPI_FADT_NO_ASPM and using acpiphp instead of pciehp. Then the OS would keep its mitts off ASPM, and the BIOS would have a chance to configure ASPM for hot-added devices before giving them to the OS. Here are the possibilities I see for POLICY_DEFAULT: 1) Never touch ASPM config (what we have today) Boot-present devices: ASPM config retained from BIOS Hot-added devices: ASPM disabled (poweron state) 2) Linux maintains BIOS policy (conservative) Boot-present devices: ASPM config retained from BIOS Hot-added devices (slot occupied at boot): use boot-time ASPM config Hot-added devices (slot empty at boot): ASPM disabled 3) Linux maintains BIOS policy (aggressive, your current patch) Boot-present devices: ASPM config retained from BIOS Hot-added devices (slot occupied at boot): use boot-time ASPM config Hot-added devices (slot empty at boot): ASPM enabled I'm becoming less convinced that options 2 or 3 make sense. For one thing, they're both hard to describe concisely because there are too many special cases, and that's always a red flag for me. Even for a given BIOS power management policy, the ASPM configuration may depend on the particular device; for example, a balanced policy might enable ASPM for USB devices but not for NICs. So I'm not sure it really makes sense to remember what BIOS did for the card that was in the slot at boot-time and apply that to a possibly different card hot-added later. I think there's an argument to be made that if we care about ASPM configuration, we should be using a non-POLICY_DEFAULT setting. And I think there's value in having POLICY_DEFAULT be the absolute lowest- risk setting, which I think means option 1. What do you think? Bjorn ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-04-14 21:44 ` Bjorn Helgaas 0 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-14 21:44 UTC (permalink / raw) To: linux-arm-kernel [+cc Myron, lkml] On Fri, Apr 14, 2017 at 03:12:35PM -0400, Sinan Kaya wrote: > Bjorn, > > On 4/12/2017 3:19 PM, Rajat Jain wrote: > > On Fri, Apr 7, 2017 at 9:55 PM, Sinan Kaya <okaya@codeaurora.org> wrote: > >> Now that we added a hook to be called from device_add, save the > >> default values from the HW registers early in the boot for further > >> reuse during hot device add/remove operations. > >> > >> If the link is down during boot, assume that we want to enable L0s > >> and L1 following hotplug insertion as well as L1SS if supported. > > > > IIUC, so far POLICY_DEFAULT meant that we'd just use & follow what > > BIOS has done, and play it safe (never try to be more opportunistic). > > With this change however, we'd be slightly overstepping and giving > > ourselves benefit of doubt if the BIOS could not enable ASPM states > > because the link was not up. This may be good, but I think we should > > call it out, and add some more elaborate comment on the POLICY_DEFAULT > > description (what to, and what not to expect in different situations). > > > > It is important because existing systems today, that used to boot > > without cards and later hotplugged them, didn't have ASPM states > > enabled. They will now suddenly start seeing all ASPM states enabled > > including L1 substates for the first time (if supported). > > > > Rajat has a good point here. Would you like me to update the ASPM document > with this new behavior for hotplug? > > Do you have another behavior preference when it comes this? That *is* a very good point. ?I think the change in behavior could be surprising. I wonder if we should revise our understanding of what POLICY_DEFAULT means. ?If we decided it means "the kernel never changes any ASPM config", it would be clear that we keep the BIOS configuration for everything present at boot, and we don't enable ASPM for any hot-added devices. I think the motivation for this series is to apply the BIOS's power management policy to hot-added devices. ?There's no direct way to know the BIOS's policy, so we're trying to infer it from the boot-time link configurations. Should we even *try* to apply the BIOS's policy? ?I don't know. ?If a platform really wanted to maintain control over ASPM and apply its policy consistently, I think it could do that by setting ACPI_FADT_NO_ASPM and using acpiphp instead of pciehp. ?Then the OS would keep its mitts off ASPM, and the BIOS would have a chance to configure ASPM for hot-added devices before giving them to the OS. Here are the possibilities I see for POLICY_DEFAULT: 1) Never touch ASPM config (what we have today) Boot-present devices: ASPM config retained from BIOS Hot-added devices: ASPM disabled (poweron state) 2) Linux maintains BIOS policy (conservative) Boot-present devices: ASPM config retained from BIOS Hot-added devices (slot occupied at boot): use boot-time ASPM config Hot-added devices (slot empty at boot): ASPM disabled 3) Linux maintains BIOS policy (aggressive, your current patch) Boot-present devices: ASPM config retained from BIOS Hot-added devices (slot occupied at boot): use boot-time ASPM config Hot-added devices (slot empty at boot): ASPM enabled I'm becoming less convinced that options 2 or 3 make sense. For one thing, they're both hard to describe concisely because there are too many special cases, and that's always a red flag for me. Even for a given BIOS power management policy, the ASPM configuration may depend on the particular device; for example, a balanced policy might enable ASPM for USB devices but not for NICs. So I'm not sure it really makes sense to remember what BIOS did for the card that was in the slot at boot-time and apply that to a possibly different card hot-added later. I think there's an argument to be made that if we care about ASPM configuration, we should be using a non-POLICY_DEFAULT setting. And I think there's value in having POLICY_DEFAULT be the absolute lowest- risk setting, which I think means option 1. What do you think? Bjorn ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-04-14 21:44 ` Bjorn Helgaas 0 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-14 21:44 UTC (permalink / raw) To: Sinan Kaya Cc: Patel, Mayurkumar, Rajat Jain, Myron Stowe, David Daney, linux-pci, timur, linux-kernel, Julia Lawall, linux-arm-msm, Bjorn Helgaas, Rajat Jain, Yinghai Lu, Shawn Lin, linux-arm-kernel [+cc Myron, lkml] On Fri, Apr 14, 2017 at 03:12:35PM -0400, Sinan Kaya wrote: > Bjorn, > = > On 4/12/2017 3:19 PM, Rajat Jain wrote: > > On Fri, Apr 7, 2017 at 9:55 PM, Sinan Kaya <okaya@codeaurora.org> wrote: > >> Now that we added a hook to be called from device_add, save the > >> default values from the HW registers early in the boot for further > >> reuse during hot device add/remove operations. > >> > >> If the link is down during boot, assume that we want to enable L0s > >> and L1 following hotplug insertion as well as L1SS if supported. > > = > > IIUC, so far POLICY_DEFAULT meant that we'd just use & follow what > > BIOS has done, and play it safe (never try to be more opportunistic). > > With this change however, we'd be slightly overstepping and giving > > ourselves benefit of doubt if the BIOS could not enable ASPM states > > because the link was not up. This may be good, but I think we should > > call it out, and add some more elaborate comment on the POLICY_DEFAULT > > description (what to, and what not to expect in different situations). > > = > > It is important because existing systems today, that used to boot > > without cards and later hotplugged them, didn't have ASPM states > > enabled. They will now suddenly start seeing all ASPM states enabled > > including L1 substates for the first time (if supported). > > = > = > Rajat has a good point here. Would you like me to update the ASPM document > with this new behavior for hotplug? > = > Do you have another behavior preference when it comes this? That *is* a very good point. =A0I think the change in behavior could be surprising. I wonder if we should revise our understanding of what POLICY_DEFAULT means. =A0If we decided it means "the kernel never changes any ASPM config", it would be clear that we keep the BIOS configuration for everything present at boot, and we don't enable ASPM for any hot-added devices. I think the motivation for this series is to apply the BIOS's power management policy to hot-added devices. =A0There's no direct way to know the BIOS's policy, so we're trying to infer it from the boot-time link configurations. Should we even *try* to apply the BIOS's policy? =A0I don't know. =A0If a platform really wanted to maintain control over ASPM and apply its policy consistently, I think it could do that by setting ACPI_FADT_NO_ASPM and using acpiphp instead of pciehp. =A0Then the OS would keep its mitts off ASPM, and the BIOS would have a chance to configure ASPM for hot-added devices before giving them to the OS. Here are the possibilities I see for POLICY_DEFAULT: 1) Never touch ASPM config (what we have today) Boot-present devices: ASPM config retained from BIOS Hot-added devices: ASPM disabled (poweron state) 2) Linux maintains BIOS policy (conservative) Boot-present devices: ASPM config retained from BIOS Hot-added devices (slot occupied at boot): use boot-time ASPM config Hot-added devices (slot empty at boot): ASPM disabled 3) Linux maintains BIOS policy (aggressive, your current patch) Boot-present devices: ASPM config retained from BIOS Hot-added devices (slot occupied at boot): use boot-time ASPM config Hot-added devices (slot empty at boot): ASPM enabled I'm becoming less convinced that options 2 or 3 make sense. For one thing, they're both hard to describe concisely because there are too many special cases, and that's always a red flag for me. Even for a given BIOS power management policy, the ASPM configuration may depend on the particular device; for example, a balanced policy might enable ASPM for USB devices but not for NICs. So I'm not sure it really makes sense to remember what BIOS did for the card that was in the slot at boot-time and apply that to a possibly different card hot-added later. I think there's an argument to be made that if we care about ASPM configuration, we should be using a non-POLICY_DEFAULT setting. And I think there's value in having POLICY_DEFAULT be the absolute lowest- risk setting, which I think means option 1. What do you think? Bjorn _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init 2017-04-14 21:44 ` Bjorn Helgaas @ 2017-04-14 22:17 ` Sinan Kaya -1 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-14 22:17 UTC (permalink / raw) To: Bjorn Helgaas Cc: Rajat Jain, Patel, Mayurkumar, Rajat Jain, David Daney, linux-pci, timur, linux-kernel, Julia Lawall, linux-arm-msm, Bjorn Helgaas, Yinghai Lu, Shawn Lin, linux-arm-kernel, Myron Stowe On 4/14/2017 5:44 PM, Bjorn Helgaas wrote: > I think there's an argument to be made that if we care about ASPM > configuration, we should be using a non-POLICY_DEFAULT setting. And I > think there's value in having POLICY_DEFAULT be the absolute lowest- > risk setting, which I think means option 1. > > What do you think? I see your point. The counter argument is that most of the users do not know what an ASPM kernel command line is unless they understand PCI language. I have been using the powersave policy option until now. I recently realized that nobody except me is using this option. Therefore, we are wasting power by default following a hotplug insertion. This is the case where I'm trying to avoid. With the introduction of NVMe u.2 drives, hotplug is becoming more and more mainstream. I decided to take the matters into my hand with this series for this very reason. Like you said, BIOS is out of the picture with pciehp. There is nobody to configure ASPM following a hotplug insertion. I can also claim that If user wants performance, they should boot with the performance policy or pcie_aspm=off parameters. I saw this recommendation in multiple DPDK tuning documents. Like you said, what do we do by default is the question. Should we opt for safe like we are doing, or try to save some power. Maybe, we are missing a HPP option from the PCI spec. -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-04-14 22:17 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-14 22:17 UTC (permalink / raw) To: linux-arm-kernel On 4/14/2017 5:44 PM, Bjorn Helgaas wrote: > I think there's an argument to be made that if we care about ASPM > configuration, we should be using a non-POLICY_DEFAULT setting. And I > think there's value in having POLICY_DEFAULT be the absolute lowest- > risk setting, which I think means option 1. > > What do you think? I see your point. The counter argument is that most of the users do not know what an ASPM kernel command line is unless they understand PCI language. I have been using the powersave policy option until now. I recently realized that nobody except me is using this option. Therefore, we are wasting power by default following a hotplug insertion. This is the case where I'm trying to avoid. With the introduction of NVMe u.2 drives, hotplug is becoming more and more mainstream. I decided to take the matters into my hand with this series for this very reason. Like you said, BIOS is out of the picture with pciehp. There is nobody to configure ASPM following a hotplug insertion. I can also claim that If user wants performance, they should boot with the performance policy or pcie_aspm=off parameters. I saw this recommendation in multiple DPDK tuning documents. Like you said, what do we do by default is the question. Should we opt for safe like we are doing, or try to save some power. Maybe, we are missing a HPP option from the PCI spec. -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init 2017-04-14 22:17 ` Sinan Kaya @ 2017-04-17 16:38 ` Bjorn Helgaas -1 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-17 16:38 UTC (permalink / raw) To: Sinan Kaya Cc: Bjorn Helgaas, Rajat Jain, Patel, Mayurkumar, Rajat Jain, David Daney, linux-pci, Timur Tabi, linux-kernel, Julia Lawall, linux-arm-msm, Yinghai Lu, Shawn Lin, linux-arm, Myron Stowe On Fri, Apr 14, 2017 at 5:17 PM, Sinan Kaya <okaya@codeaurora.org> wrote: > On 4/14/2017 5:44 PM, Bjorn Helgaas wrote: >> I think there's an argument to be made that if we care about ASPM >> configuration, we should be using a non-POLICY_DEFAULT setting. And I >> think there's value in having POLICY_DEFAULT be the absolute lowest- >> risk setting, which I think means option 1. >> >> What do you think? > > I see your point. The counter argument is that most of the users do not > know what an ASPM kernel command line is unless they understand PCI > language. I don't think the answer is using the "pcie_aspm.policy=" boot argument. I certainly don't want users to have to deal with that. I wish we didn't even have that parameter. I think we need runtime knobs instead (and I guess we already have /sys/module/pcie_aspm/parameters/policy and /sys/.../link_state), and distro userspace should use them. I'm envisioning something in "System Settings / Power" or similar. Basically I think the policy doesn't *have* to be dictated by a kernel boot-time parameter, so it should not be. > I have been using the powersave policy option until now. I recently realized > that nobody except me is using this option. Therefore, we are wasting > power by default following a hotplug insertion. > > This is the case where I'm trying to avoid. With the introduction of NVMe > u.2 drives, hotplug is becoming more and more mainstream. I decided to > take the matters into my hand with this series for this very reason. > > Like you said, BIOS is out of the picture with pciehp. There is nobody > to configure ASPM following a hotplug insertion. > > I can also claim that If user wants performance, they should boot with > the performance policy or pcie_aspm=off parameters. > > I saw this recommendation in multiple DPDK tuning documents. > > Like you said, what do we do by default is the question. Should we opt > for safe like we are doing, or try to save some power. I think safety is paramount. Every user should be able to boot safely without any kernel parameters. We don't want users to have a problem booting and then have to search for a workaround like booting with "pcie_aspm=off". Most users will never do that. Here's a long-term strawman proposal, see what you think: - Deprecate CONFIG_PCIEASPM_DEFAULT, CONFIG_PCIEASPM_POWERSAVE, etc. - Default aspm_policy is POLICY_DEFAULT always. - POLICY_DEFAULT means Linux doesn't touch anything: if BIOS enabled ASPM, we leave it that way; we leave ASPM disabled on hot-added devices. - Deprecate kernel boot parameters (possibly keep pcie_aspm=off for debugging use). - Use /sys/module/pcie_aspm/parameters/policy for run-time system-wide control, including for future hot-added devices. - Remove CONFIG_PCIEASPM_DEBUG and enable that code always, so we have fine-grained run-time control. > Maybe, we are missing a HPP option from the PCI spec. That's an interesting idea. _HPX does have provision for manipulating Link Control bits (see ACPI r5.0, sec 6.2.8.3), but I don't think very many systems implement it. And there's currently no connection between program_hpp_type2() and aspm.c, so I'm a little worried that we might have issues if a system did implement an _HPX that sets any of the ASPM bits. Bjorn ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-04-17 16:38 ` Bjorn Helgaas 0 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-17 16:38 UTC (permalink / raw) To: linux-arm-kernel On Fri, Apr 14, 2017 at 5:17 PM, Sinan Kaya <okaya@codeaurora.org> wrote: > On 4/14/2017 5:44 PM, Bjorn Helgaas wrote: >> I think there's an argument to be made that if we care about ASPM >> configuration, we should be using a non-POLICY_DEFAULT setting. And I >> think there's value in having POLICY_DEFAULT be the absolute lowest- >> risk setting, which I think means option 1. >> >> What do you think? > > I see your point. The counter argument is that most of the users do not > know what an ASPM kernel command line is unless they understand PCI > language. I don't think the answer is using the "pcie_aspm.policy=" boot argument. I certainly don't want users to have to deal with that. I wish we didn't even have that parameter. I think we need runtime knobs instead (and I guess we already have /sys/module/pcie_aspm/parameters/policy and /sys/.../link_state), and distro userspace should use them. I'm envisioning something in "System Settings / Power" or similar. Basically I think the policy doesn't *have* to be dictated by a kernel boot-time parameter, so it should not be. > I have been using the powersave policy option until now. I recently realized > that nobody except me is using this option. Therefore, we are wasting > power by default following a hotplug insertion. > > This is the case where I'm trying to avoid. With the introduction of NVMe > u.2 drives, hotplug is becoming more and more mainstream. I decided to > take the matters into my hand with this series for this very reason. > > Like you said, BIOS is out of the picture with pciehp. There is nobody > to configure ASPM following a hotplug insertion. > > I can also claim that If user wants performance, they should boot with > the performance policy or pcie_aspm=off parameters. > > I saw this recommendation in multiple DPDK tuning documents. > > Like you said, what do we do by default is the question. Should we opt > for safe like we are doing, or try to save some power. I think safety is paramount. Every user should be able to boot safely without any kernel parameters. We don't want users to have a problem booting and then have to search for a workaround like booting with "pcie_aspm=off". Most users will never do that. Here's a long-term strawman proposal, see what you think: - Deprecate CONFIG_PCIEASPM_DEFAULT, CONFIG_PCIEASPM_POWERSAVE, etc. - Default aspm_policy is POLICY_DEFAULT always. - POLICY_DEFAULT means Linux doesn't touch anything: if BIOS enabled ASPM, we leave it that way; we leave ASPM disabled on hot-added devices. - Deprecate kernel boot parameters (possibly keep pcie_aspm=off for debugging use). - Use /sys/module/pcie_aspm/parameters/policy for run-time system-wide control, including for future hot-added devices. - Remove CONFIG_PCIEASPM_DEBUG and enable that code always, so we have fine-grained run-time control. > Maybe, we are missing a HPP option from the PCI spec. That's an interesting idea. _HPX does have provision for manipulating Link Control bits (see ACPI r5.0, sec 6.2.8.3), but I don't think very many systems implement it. And there's currently no connection between program_hpp_type2() and aspm.c, so I'm a little worried that we might have issues if a system did implement an _HPX that sets any of the ASPM bits. Bjorn ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init 2017-04-17 16:38 ` Bjorn Helgaas @ 2017-04-17 17:50 ` Sinan Kaya -1 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-17 17:50 UTC (permalink / raw) To: Bjorn Helgaas Cc: Bjorn Helgaas, Rajat Jain, Patel, Mayurkumar, Rajat Jain, David Daney, linux-pci, Timur Tabi, linux-kernel, Julia Lawall, linux-arm-msm, Yinghai Lu, Shawn Lin, linux-arm, Myron Stowe On 4/17/2017 12:38 PM, Bjorn Helgaas wrote: >> Like you said, what do we do by default is the question. Should we opt >> for safe like we are doing, or try to save some power. > I think safety is paramount. Every user should be able to boot safely > without any kernel parameters. We don't want users to have a problem > booting and then have to search for a workaround like booting with > "pcie_aspm=off". Most users will never do that. > OK, no problem with leaving the behavior as it is. My initial approach was #2. We knew this way that user had full control over the ASPM policy by changing the BIOS option. Then, Mayurkumar complained that ASPM is not enabled following a hotplug insertion to an empty slot. That's when I switched to #3 as it sounded like a good thing to have for us. > Here's a long-term strawman proposal, see what you think: > > - Deprecate CONFIG_PCIEASPM_DEFAULT, CONFIG_PCIEASPM_POWERSAVE, etc. > - Default aspm_policy is POLICY_DEFAULT always. > - POLICY_DEFAULT means Linux doesn't touch anything: if BIOS enabled > ASPM, we leave it that way; we leave ASPM disabled on hot-added > devices. I can easily see people complaining the other way around. There could be some boot FW that doesn't know what ASPM is and this particular system could rely on the compile time option for enabling power options. Maybe, a command line option will be required for them to keep the existing behavior. > - Deprecate kernel boot parameters (possibly keep pcie_aspm=off for > debugging use). > - Use /sys/module/pcie_aspm/parameters/policy for run-time > system-wide control, including for future hot-added devices. > - Remove CONFIG_PCIEASPM_DEBUG and enable that code always, so we > have fine-grained run-time control. > Runtime control sounds like a better plan. We need hooks into the system power management policy. >> Maybe, we are missing a HPP option from the PCI spec. > That's an interesting idea. _HPX does have provision for manipulating > Link Control bits (see ACPI r5.0, sec 6.2.8.3), but I don't think very > many systems implement it. And there's currently no connection > between program_hpp_type2() and aspm.c, so I'm a little worried that > we might have issues if a system did implement an _HPX that sets any > of the ASPM bits. I looked at the spec some more. These are there to restore the register settings following hotplug insertion. I agree it won't play nice with ASPM as the control bits need to be enabled in coordination with the upstream device. I think the right approach is to let the userspace feed the required policy to the kernel like you suggested. Then, it needs to be per port via link_state to have the most flexibility. -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-04-17 17:50 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-17 17:50 UTC (permalink / raw) To: linux-arm-kernel On 4/17/2017 12:38 PM, Bjorn Helgaas wrote: >> Like you said, what do we do by default is the question. Should we opt >> for safe like we are doing, or try to save some power. > I think safety is paramount. Every user should be able to boot safely > without any kernel parameters. We don't want users to have a problem > booting and then have to search for a workaround like booting with > "pcie_aspm=off". Most users will never do that. > OK, no problem with leaving the behavior as it is. My initial approach was #2. We knew this way that user had full control over the ASPM policy by changing the BIOS option. Then, Mayurkumar complained that ASPM is not enabled following a hotplug insertion to an empty slot. That's when I switched to #3 as it sounded like a good thing to have for us. > Here's a long-term strawman proposal, see what you think: > > - Deprecate CONFIG_PCIEASPM_DEFAULT, CONFIG_PCIEASPM_POWERSAVE, etc. > - Default aspm_policy is POLICY_DEFAULT always. > - POLICY_DEFAULT means Linux doesn't touch anything: if BIOS enabled > ASPM, we leave it that way; we leave ASPM disabled on hot-added > devices. I can easily see people complaining the other way around. There could be some boot FW that doesn't know what ASPM is and this particular system could rely on the compile time option for enabling power options. Maybe, a command line option will be required for them to keep the existing behavior. > - Deprecate kernel boot parameters (possibly keep pcie_aspm=off for > debugging use). > - Use /sys/module/pcie_aspm/parameters/policy for run-time > system-wide control, including for future hot-added devices. > - Remove CONFIG_PCIEASPM_DEBUG and enable that code always, so we > have fine-grained run-time control. > Runtime control sounds like a better plan. We need hooks into the system power management policy. >> Maybe, we are missing a HPP option from the PCI spec. > That's an interesting idea. _HPX does have provision for manipulating > Link Control bits (see ACPI r5.0, sec 6.2.8.3), but I don't think very > many systems implement it. And there's currently no connection > between program_hpp_type2() and aspm.c, so I'm a little worried that > we might have issues if a system did implement an _HPX that sets any > of the ASPM bits. I looked at the spec some more. These are there to restore the register settings following hotplug insertion. I agree it won't play nice with ASPM as the control bits need to be enabled in coordination with the upstream device. I think the right approach is to let the userspace feed the required policy to the kernel like you suggested. Then, it needs to be per port via link_state to have the most flexibility. -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 94+ messages in thread
* RE: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init 2017-04-17 17:50 ` Sinan Kaya (?) (?) @ 2017-04-21 7:46 ` Patel, Mayurkumar -1 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-04-21 7:46 UTC (permalink / raw) To: Sinan Kaya, Bjorn Helgaas Cc: Rajat Jain, Myron Stowe, David Daney, linux-pci, Timur Tabi, linux-kernel, Julia Lawall, Bjorn Helgaas, linux-arm-msm, Rajat Jain, Yinghai Lu, Shawn Lin, linux-arm Hi Bjorn/Kaya, > >On 4/17/2017 12:38 PM, Bjorn Helgaas wrote: >>> Like you said, what do we do by default is the question. Should we opt >>> for safe like we are doing, or try to save some power. >> I think safety is paramount. Every user should be able to boot safely >> without any kernel parameters. We don't want users to have a problem >> booting and then have to search for a workaround like booting with >> "pcie_aspm=off". Most users will never do that. >> > >OK, no problem with leaving the behavior as it is. > >My initial approach was #2. We knew this way that user had full control >over the ASPM policy by changing the BIOS option. Then, Mayurkumar >complained that ASPM is not enabled following a hotplug insertion to an >empty slot. That's when I switched to #3 as it sounded like a good thing >to have for us. > >> Here's a long-term strawman proposal, see what you think: >> >> - Deprecate CONFIG_PCIEASPM_DEFAULT, CONFIG_PCIEASPM_POWERSAVE, etc. >> - Default aspm_policy is POLICY_DEFAULT always. >> - POLICY_DEFAULT means Linux doesn't touch anything: if BIOS enabled >> ASPM, we leave it that way; we leave ASPM disabled on hot-added >> devices. > I am also ok with leaving the same behavior as now. But still following is something open I feel besides, Which may be there in your comments redundantly. The current problem is, pcie_aspm_exit_link_state() disables the ASPM configuration even if POLICY_DEFAULT was set. I am seeing already following problem(or may be influence) with it. The Endpoint I have does not have does not have "Presence detect change" mechanism. Hot plug is working with Link status events. When link is in L1 or L1SS and if EP is powered off, no Link status change event are triggered (It might be the expected behavior in L1 or L1SS). When next time EP is powered on there are link down and link up events coming one after other. BIOS enables ASPM on Root port and Endpoint, but while processing link status down, pcie_aspm_exit_link_state() clears the ASPM already which were enabled by BIOS. If we want to follow above approach then shall we consider having something similar as following? diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index 1dfa10c..bf5be6d 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -940,7 +940,8 @@ void pcie_aspm_exit_link_state(struct pci_dev *pdev) parent_link = link->parent; /* All functions are removed, so just disable ASPM for the link */ - pcie_config_aspm_link(link, 0); + if (aspm_policy != POLICY_DEFAULT) + pcie_config_aspm_link(link, 0); list_del(&link->sibling); list_del(&link->link); /* Clock PM is for endpoint device */ >I can easily see people complaining the other way around. There >could be some boot FW that doesn't know what ASPM is and this particular >system could rely on the compile time option for enabling power options. >Maybe, a command line option will be required for them to keep the existing >behavior. > >> - Deprecate kernel boot parameters (possibly keep pcie_aspm=off for >> debugging use). >> - Use /sys/module/pcie_aspm/parameters/policy for run-time >> system-wide control, including for future hot-added devices. >> - Remove CONFIG_PCIEASPM_DEBUG and enable that code always, so we >> have fine-grained run-time control. >> > >Runtime control sounds like a better plan. We need hooks into the system >power management policy. > >>> Maybe, we are missing a HPP option from the PCI spec. >> That's an interesting idea. _HPX does have provision for manipulating >> Link Control bits (see ACPI r5.0, sec 6.2.8.3), but I don't think very >> many systems implement it. And there's currently no connection >> between program_hpp_type2() and aspm.c, so I'm a little worried that >> we might have issues if a system did implement an _HPX that sets any >> of the ASPM bits. > >I looked at the spec some more. These are there to restore the register >settings following hotplug insertion. I agree it won't play nice with ASPM >as the control bits need to be enabled in coordination with the upstream >device. > >I think the right approach is to let the userspace feed the required >policy to the kernel like you suggested. Then, it needs to be per port >via link_state to have the most flexibility. > > >-- >Sinan Kaya >Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. >Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Christian Lamprechter Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 ^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-04-21 7:46 ` Patel, Mayurkumar 0 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-04-21 7:46 UTC (permalink / raw) To: linux-arm-kernel Hi Bjorn/Kaya, > >On 4/17/2017 12:38 PM, Bjorn Helgaas wrote: >>> Like you said, what do we do by default is the question. Should we opt >>> for safe like we are doing, or try to save some power. >> I think safety is paramount. Every user should be able to boot safely >> without any kernel parameters. We don't want users to have a problem >> booting and then have to search for a workaround like booting with >> "pcie_aspm=off". Most users will never do that. >> > >OK, no problem with leaving the behavior as it is. > >My initial approach was #2. We knew this way that user had full control >over the ASPM policy by changing the BIOS option. Then, Mayurkumar >complained that ASPM is not enabled following a hotplug insertion to an >empty slot. That's when I switched to #3 as it sounded like a good thing >to have for us. > >> Here's a long-term strawman proposal, see what you think: >> >> - Deprecate CONFIG_PCIEASPM_DEFAULT, CONFIG_PCIEASPM_POWERSAVE, etc. >> - Default aspm_policy is POLICY_DEFAULT always. >> - POLICY_DEFAULT means Linux doesn't touch anything: if BIOS enabled >> ASPM, we leave it that way; we leave ASPM disabled on hot-added >> devices. > I am also ok with leaving the same behavior as now. But still following is something open I feel besides, Which may be there in your comments redundantly. The current problem is, pcie_aspm_exit_link_state() disables the ASPM configuration even if POLICY_DEFAULT was set. I am seeing already following problem(or may be influence) with it. The Endpoint I have does not have does not have "Presence detect change" mechanism. Hot plug is working with Link status events. When link is in L1 or L1SS and if EP is powered off, no Link status change event are triggered (It might be the expected behavior in L1 or L1SS). When next time EP is powered on there are link down and link up events coming one after other. BIOS enables ASPM on Root port and Endpoint, but while processing link status down, pcie_aspm_exit_link_state() clears the ASPM already which were enabled by BIOS. If we want to follow above approach then shall we consider having something similar as following? diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index 1dfa10c..bf5be6d 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -940,7 +940,8 @@ void pcie_aspm_exit_link_state(struct pci_dev *pdev) parent_link = link->parent; /* All functions are removed, so just disable ASPM for the link */ - pcie_config_aspm_link(link, 0); + if (aspm_policy != POLICY_DEFAULT) + pcie_config_aspm_link(link, 0); list_del(&link->sibling); list_del(&link->link); /* Clock PM is for endpoint device */ >I can easily see people complaining the other way around. There >could be some boot FW that doesn't know what ASPM is and this particular >system could rely on the compile time option for enabling power options. >Maybe, a command line option will be required for them to keep the existing >behavior. > >> - Deprecate kernel boot parameters (possibly keep pcie_aspm=off for >> debugging use). >> - Use /sys/module/pcie_aspm/parameters/policy for run-time >> system-wide control, including for future hot-added devices. >> - Remove CONFIG_PCIEASPM_DEBUG and enable that code always, so we >> have fine-grained run-time control. >> > >Runtime control sounds like a better plan. We need hooks into the system >power management policy. > >>> Maybe, we are missing a HPP option from the PCI spec. >> That's an interesting idea. _HPX does have provision for manipulating >> Link Control bits (see ACPI r5.0, sec 6.2.8.3), but I don't think very >> many systems implement it. And there's currently no connection >> between program_hpp_type2() and aspm.c, so I'm a little worried that >> we might have issues if a system did implement an _HPX that sets any >> of the ASPM bits. > >I looked at the spec some more. These are there to restore the register >settings following hotplug insertion. I agree it won't play nice with ASPM >as the control bits need to be enabled in coordination with the upstream >device. > >I think the right approach is to let the userspace feed the required >policy to the kernel like you suggested. Then, it needs to be per port >via link_state to have the most flexibility. > > >-- >Sinan Kaya >Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. >Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Christian Lamprechter Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 ^ permalink raw reply related [flat|nested] 94+ messages in thread
* RE: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-04-21 7:46 ` Patel, Mayurkumar 0 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-04-21 7:46 UTC (permalink / raw) To: Sinan Kaya, Bjorn Helgaas Cc: Bjorn Helgaas, Rajat Jain, Rajat Jain, David Daney, linux-pci, Timur Tabi, linux-kernel, Julia Lawall, linux-arm-msm, Yinghai Lu, Shawn Lin, linux-arm, Myron Stowe SGkgQmpvcm4vS2F5YSwNCg0KDQo+DQo+T24gNC8xNy8yMDE3IDEyOjM4IFBNLCBCam9ybiBIZWxn YWFzIHdyb3RlOg0KPj4+IExpa2UgeW91IHNhaWQsIHdoYXQgZG8gd2UgZG8gYnkgZGVmYXVsdCBp cyB0aGUgcXVlc3Rpb24uIFNob3VsZCB3ZSBvcHQNCj4+PiBmb3Igc2FmZSBsaWtlIHdlIGFyZSBk b2luZywgb3IgdHJ5IHRvIHNhdmUgc29tZSBwb3dlci4NCj4+IEkgdGhpbmsgc2FmZXR5IGlzIHBh cmFtb3VudC4gIEV2ZXJ5IHVzZXIgc2hvdWxkIGJlIGFibGUgdG8gYm9vdCBzYWZlbHkNCj4+IHdp dGhvdXQgYW55IGtlcm5lbCBwYXJhbWV0ZXJzLiAgV2UgZG9uJ3Qgd2FudCB1c2VycyB0byBoYXZl IGEgcHJvYmxlbQ0KPj4gYm9vdGluZyBhbmQgdGhlbiBoYXZlIHRvIHNlYXJjaCBmb3IgYSB3b3Jr YXJvdW5kIGxpa2UgYm9vdGluZyB3aXRoDQo+PiAicGNpZV9hc3BtPW9mZiIuICBNb3N0IHVzZXJz IHdpbGwgbmV2ZXIgZG8gdGhhdC4NCj4+DQo+DQo+T0ssIG5vIHByb2JsZW0gd2l0aCBsZWF2aW5n IHRoZSBiZWhhdmlvciBhcyBpdCBpcy4NCj4NCj5NeSBpbml0aWFsIGFwcHJvYWNoIHdhcyAjMi4g V2Uga25ldyB0aGlzIHdheSB0aGF0IHVzZXIgaGFkIGZ1bGwgY29udHJvbA0KPm92ZXIgdGhlIEFT UE0gcG9saWN5IGJ5IGNoYW5naW5nIHRoZSBCSU9TIG9wdGlvbi4gVGhlbiwgTWF5dXJrdW1hcg0K PmNvbXBsYWluZWQgdGhhdCBBU1BNIGlzIG5vdCBlbmFibGVkIGZvbGxvd2luZyBhIGhvdHBsdWcg aW5zZXJ0aW9uIHRvIGFuDQo+ZW1wdHkgc2xvdC4gVGhhdCdzIHdoZW4gSSBzd2l0Y2hlZCB0byAj MyBhcyBpdCBzb3VuZGVkIGxpa2UgYSBnb29kIHRoaW5nDQo+dG8gaGF2ZSBmb3IgdXMuDQo+DQo+ PiBIZXJlJ3MgYSBsb25nLXRlcm0gc3RyYXdtYW4gcHJvcG9zYWwsIHNlZSB3aGF0IHlvdSB0aGlu azoNCj4+DQo+PiAgIC0gRGVwcmVjYXRlIENPTkZJR19QQ0lFQVNQTV9ERUZBVUxULCBDT05GSUdf UENJRUFTUE1fUE9XRVJTQVZFLCBldGMuDQo+PiAgIC0gRGVmYXVsdCBhc3BtX3BvbGljeSBpcyBQ T0xJQ1lfREVGQVVMVCBhbHdheXMuDQo+PiAgIC0gUE9MSUNZX0RFRkFVTFQgbWVhbnMgTGludXgg ZG9lc24ndCB0b3VjaCBhbnl0aGluZzogaWYgQklPUyBlbmFibGVkDQo+PiBBU1BNLCB3ZSBsZWF2 ZSBpdCB0aGF0IHdheTsgd2UgbGVhdmUgQVNQTSBkaXNhYmxlZCBvbiBob3QtYWRkZWQNCj4+IGRl dmljZXMuDQo+DQpJIGFtIGFsc28gb2sgd2l0aCBsZWF2aW5nIHRoZSBzYW1lIGJlaGF2aW9yIGFz IG5vdy4NCkJ1dCBzdGlsbCBmb2xsb3dpbmcgaXMgc29tZXRoaW5nIG9wZW4gSSBmZWVsIGJlc2lk ZXMsIFdoaWNoIG1heSBiZSB0aGVyZSBpbiB5b3VyIGNvbW1lbnRzIHJlZHVuZGFudGx5Lg0KVGhl IGN1cnJlbnQgcHJvYmxlbSBpcywgcGNpZV9hc3BtX2V4aXRfbGlua19zdGF0ZSgpIGRpc2FibGVz IHRoZSBBU1BNIGNvbmZpZ3VyYXRpb24gZXZlbg0KaWYgUE9MSUNZX0RFRkFVTFQgd2FzIHNldC4N CkkgYW0gc2VlaW5nIGFscmVhZHkgZm9sbG93aW5nIHByb2JsZW0ob3IgbWF5IGJlIGluZmx1ZW5j ZSkgd2l0aCBpdC4gVGhlIEVuZHBvaW50IEkgaGF2ZSBkb2VzIG5vdCBoYXZlDQpkb2VzIG5vdCBo YXZlICJQcmVzZW5jZSBkZXRlY3QgY2hhbmdlIiBtZWNoYW5pc20uIEhvdCBwbHVnIGlzIHdvcmtp bmcgd2l0aCBMaW5rIHN0YXR1cyBldmVudHMuDQpXaGVuIGxpbmsgaXMgaW4gTDEgb3IgTDFTUyBh bmQgaWYgRVAgaXMgcG93ZXJlZCBvZmYsIG5vIExpbmsgc3RhdHVzIGNoYW5nZSBldmVudCBhcmUg dHJpZ2dlcmVkIChJdCBtaWdodCBiZQ0KdGhlIGV4cGVjdGVkIGJlaGF2aW9yIGluIEwxIG9yIEwx U1MpLiAgV2hlbiBuZXh0IHRpbWUgRVAgaXMgcG93ZXJlZCBvbiB0aGVyZSBhcmUgbGluayBkb3du IGFuZA0KbGluayB1cCBldmVudHMgY29taW5nIG9uZSBhZnRlciBvdGhlci4gQklPUyBlbmFibGVz IEFTUE0gb24gUm9vdCBwb3J0IGFuZCBFbmRwb2ludCwgYnV0IHdoaWxlDQpwcm9jZXNzaW5nIGxp bmsgc3RhdHVzIGRvd24sIHBjaWVfYXNwbV9leGl0X2xpbmtfc3RhdGUoKSBjbGVhcnMgdGhlIEFT UE0gYWxyZWFkeSB3aGljaCB3ZXJlIGVuYWJsZWQgYnkgQklPUy4gDQpJZiB3ZSB3YW50IHRvIGZv bGxvdyBhYm92ZSBhcHByb2FjaCB0aGVuIHNoYWxsIHdlIGNvbnNpZGVyIGhhdmluZyBzb21ldGhp bmcgc2ltaWxhciBhcyBmb2xsb3dpbmc/DQoNCmRpZmYgLS1naXQgYS9kcml2ZXJzL3BjaS9wY2ll L2FzcG0uYyBiL2RyaXZlcnMvcGNpL3BjaWUvYXNwbS5jDQppbmRleCAxZGZhMTBjLi5iZjViZTZk IDEwMDY0NA0KLS0tIGEvZHJpdmVycy9wY2kvcGNpZS9hc3BtLmMNCisrKyBiL2RyaXZlcnMvcGNp L3BjaWUvYXNwbS5jDQpAQCAtOTQwLDcgKzk0MCw4IEBAIHZvaWQgcGNpZV9hc3BtX2V4aXRfbGlu a19zdGF0ZShzdHJ1Y3QgcGNpX2RldiAqcGRldikNCiAgICAgICAgcGFyZW50X2xpbmsgPSBsaW5r LT5wYXJlbnQ7DQoNCiAgICAgICAgLyogQWxsIGZ1bmN0aW9ucyBhcmUgcmVtb3ZlZCwgc28ganVz dCBkaXNhYmxlIEFTUE0gZm9yIHRoZSBsaW5rICovDQotICAgICAgIHBjaWVfY29uZmlnX2FzcG1f bGluayhsaW5rLCAwKTsNCisgICAgICAgaWYgKGFzcG1fcG9saWN5ICE9IFBPTElDWV9ERUZBVUxU KQ0KKyAgICAgICAgICAgICAgIHBjaWVfY29uZmlnX2FzcG1fbGluayhsaW5rLCAwKTsNCiAgICAg ICAgbGlzdF9kZWwoJmxpbmstPnNpYmxpbmcpOw0KICAgICAgICBsaXN0X2RlbCgmbGluay0+bGlu ayk7DQogICAgICAgIC8qIENsb2NrIFBNIGlzIGZvciBlbmRwb2ludCBkZXZpY2UgKi8NCiANCg0K PkkgY2FuIGVhc2lseSBzZWUgcGVvcGxlIGNvbXBsYWluaW5nIHRoZSBvdGhlciB3YXkgYXJvdW5k LiBUaGVyZQ0KPmNvdWxkIGJlIHNvbWUgYm9vdCBGVyB0aGF0IGRvZXNuJ3Qga25vdyB3aGF0IEFT UE0gaXMgYW5kIHRoaXMgcGFydGljdWxhcg0KPnN5c3RlbSBjb3VsZCByZWx5IG9uIHRoZSBjb21w aWxlIHRpbWUgb3B0aW9uIGZvciBlbmFibGluZyBwb3dlciBvcHRpb25zLg0KPk1heWJlLCBhIGNv bW1hbmQgbGluZSBvcHRpb24gd2lsbCBiZSByZXF1aXJlZCBmb3IgdGhlbSB0byBrZWVwIHRoZSBl eGlzdGluZw0KPmJlaGF2aW9yLg0KPg0KPj4gICAtIERlcHJlY2F0ZSBrZXJuZWwgYm9vdCBwYXJh bWV0ZXJzIChwb3NzaWJseSBrZWVwIHBjaWVfYXNwbT1vZmYgZm9yDQo+PiBkZWJ1Z2dpbmcgdXNl KS4NCj4+ICAgLSBVc2UgL3N5cy9tb2R1bGUvcGNpZV9hc3BtL3BhcmFtZXRlcnMvcG9saWN5IGZv ciBydW4tdGltZQ0KPj4gc3lzdGVtLXdpZGUgIGNvbnRyb2wsIGluY2x1ZGluZyBmb3IgZnV0dXJl IGhvdC1hZGRlZCBkZXZpY2VzLg0KPj4gICAtIFJlbW92ZSBDT05GSUdfUENJRUFTUE1fREVCVUcg YW5kIGVuYWJsZSB0aGF0IGNvZGUgYWx3YXlzLCBzbyB3ZQ0KPj4gaGF2ZSBmaW5lLWdyYWluZWQg cnVuLXRpbWUgY29udHJvbC4NCj4+DQo+DQo+UnVudGltZSBjb250cm9sIHNvdW5kcyBsaWtlIGEg YmV0dGVyIHBsYW4uIFdlIG5lZWQgaG9va3MgaW50byB0aGUgc3lzdGVtDQo+cG93ZXIgbWFuYWdl bWVudCBwb2xpY3kuDQo+DQo+Pj4gTWF5YmUsIHdlIGFyZSBtaXNzaW5nIGEgSFBQIG9wdGlvbiBm cm9tIHRoZSBQQ0kgc3BlYy4NCj4+IFRoYXQncyBhbiBpbnRlcmVzdGluZyBpZGVhLiAgX0hQWCBk b2VzIGhhdmUgcHJvdmlzaW9uIGZvciBtYW5pcHVsYXRpbmcNCj4+IExpbmsgQ29udHJvbCBiaXRz IChzZWUgQUNQSSByNS4wLCBzZWMgNi4yLjguMyksIGJ1dCBJIGRvbid0IHRoaW5rIHZlcnkNCj4+ IG1hbnkgc3lzdGVtcyBpbXBsZW1lbnQgaXQuICBBbmQgdGhlcmUncyBjdXJyZW50bHkgbm8gY29u bmVjdGlvbg0KPj4gYmV0d2VlbiBwcm9ncmFtX2hwcF90eXBlMigpIGFuZCBhc3BtLmMsIHNvIEkn bSBhIGxpdHRsZSB3b3JyaWVkIHRoYXQNCj4+IHdlIG1pZ2h0IGhhdmUgaXNzdWVzIGlmIGEgc3lz dGVtIGRpZCBpbXBsZW1lbnQgYW4gX0hQWCB0aGF0IHNldHMgYW55DQo+PiBvZiB0aGUgQVNQTSBi aXRzLg0KPg0KPkkgbG9va2VkIGF0IHRoZSBzcGVjIHNvbWUgbW9yZS4gVGhlc2UgYXJlIHRoZXJl IHRvIHJlc3RvcmUgdGhlIHJlZ2lzdGVyDQo+c2V0dGluZ3MgZm9sbG93aW5nIGhvdHBsdWcgaW5z ZXJ0aW9uLiBJIGFncmVlIGl0IHdvbid0IHBsYXkgbmljZSB3aXRoIEFTUE0NCj5hcyB0aGUgY29u dHJvbCBiaXRzIG5lZWQgdG8gYmUgZW5hYmxlZCBpbiBjb29yZGluYXRpb24gd2l0aCB0aGUgdXBz dHJlYW0NCj5kZXZpY2UuDQo+DQo+SSB0aGluayB0aGUgcmlnaHQgYXBwcm9hY2ggaXMgdG8gbGV0 IHRoZSB1c2Vyc3BhY2UgZmVlZCB0aGUgcmVxdWlyZWQNCj5wb2xpY3kgdG8gdGhlIGtlcm5lbCBs aWtlIHlvdSBzdWdnZXN0ZWQuIFRoZW4sIGl0IG5lZWRzIHRvIGJlIHBlciBwb3J0DQo+dmlhIGxp bmtfc3RhdGUgdG8gaGF2ZSB0aGUgbW9zdCBmbGV4aWJpbGl0eS4NCj4NCj4NCj4tLQ0KPlNpbmFu IEtheWENCj5RdWFsY29tbSBEYXRhY2VudGVyIFRlY2hub2xvZ2llcywgSW5jLiBhcyBhbiBhZmZp bGlhdGUgb2YgUXVhbGNvbW0gVGVjaG5vbG9naWVzLCBJbmMuDQo+UXVhbGNvbW0gVGVjaG5vbG9n aWVzLCBJbmMuIGlzIGEgbWVtYmVyIG9mIHRoZSBDb2RlIEF1cm9yYSBGb3J1bSwgYSBMaW51eCBG b3VuZGF0aW9uIENvbGxhYm9yYXRpdmUgUHJvamVjdC4NCkludGVsIERldXRzY2hsYW5kIEdtYkgK UmVnaXN0ZXJlZCBBZGRyZXNzOiBBbSBDYW1wZW9uIDEwLTEyLCA4NTU3OSBOZXViaWJlcmcsIEdl cm1hbnkKVGVsOiArNDkgODkgOTkgODg1My0wLCB3d3cuaW50ZWwuZGUKTWFuYWdpbmcgRGlyZWN0 b3JzOiBDaHJpc3RpbiBFaXNlbnNjaG1pZCwgQ2hyaXN0aWFuIExhbXByZWNodGVyCkNoYWlycGVy c29uIG9mIHRoZSBTdXBlcnZpc29yeSBCb2FyZDogTmljb2xlIExhdQpSZWdpc3RlcmVkIE9mZmlj ZTogTXVuaWNoCkNvbW1lcmNpYWwgUmVnaXN0ZXI6IEFtdHNnZXJpY2h0IE11ZW5jaGVuIEhSQiAx ODY5MjgK ^ permalink raw reply [flat|nested] 94+ messages in thread
* RE: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-04-21 7:46 ` Patel, Mayurkumar 0 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-04-21 7:46 UTC (permalink / raw) To: Sinan Kaya, Bjorn Helgaas Cc: Bjorn Helgaas, Rajat Jain, Rajat Jain, David Daney, linux-pci, Timur Tabi, linux-kernel, Julia Lawall, linux-arm-msm, Yinghai Lu, Shawn Lin, linux-arm, Myron Stowe Hi Bjorn/Kaya, > >On 4/17/2017 12:38 PM, Bjorn Helgaas wrote: >>> Like you said, what do we do by default is the question. Should we opt >>> for safe like we are doing, or try to save some power. >> I think safety is paramount. Every user should be able to boot safely >> without any kernel parameters. We don't want users to have a problem >> booting and then have to search for a workaround like booting with >> "pcie_aspm=off". Most users will never do that. >> > >OK, no problem with leaving the behavior as it is. > >My initial approach was #2. We knew this way that user had full control >over the ASPM policy by changing the BIOS option. Then, Mayurkumar >complained that ASPM is not enabled following a hotplug insertion to an >empty slot. That's when I switched to #3 as it sounded like a good thing >to have for us. > >> Here's a long-term strawman proposal, see what you think: >> >> - Deprecate CONFIG_PCIEASPM_DEFAULT, CONFIG_PCIEASPM_POWERSAVE, etc. >> - Default aspm_policy is POLICY_DEFAULT always. >> - POLICY_DEFAULT means Linux doesn't touch anything: if BIOS enabled >> ASPM, we leave it that way; we leave ASPM disabled on hot-added >> devices. > I am also ok with leaving the same behavior as now. But still following is something open I feel besides, Which may be there in your comments redundantly. The current problem is, pcie_aspm_exit_link_state() disables the ASPM configuration even if POLICY_DEFAULT was set. I am seeing already following problem(or may be influence) with it. The Endpoint I have does not have does not have "Presence detect change" mechanism. Hot plug is working with Link status events. When link is in L1 or L1SS and if EP is powered off, no Link status change event are triggered (It might be the expected behavior in L1 or L1SS). When next time EP is powered on there are link down and link up events coming one after other. BIOS enables ASPM on Root port and Endpoint, but while processing link status down, pcie_aspm_exit_link_state() clears the ASPM already which were enabled by BIOS. If we want to follow above approach then shall we consider having something similar as following? diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index 1dfa10c..bf5be6d 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -940,7 +940,8 @@ void pcie_aspm_exit_link_state(struct pci_dev *pdev) parent_link = link->parent; /* All functions are removed, so just disable ASPM for the link */ - pcie_config_aspm_link(link, 0); + if (aspm_policy != POLICY_DEFAULT) + pcie_config_aspm_link(link, 0); list_del(&link->sibling); list_del(&link->link); /* Clock PM is for endpoint device */ >I can easily see people complaining the other way around. There >could be some boot FW that doesn't know what ASPM is and this particular >system could rely on the compile time option for enabling power options. >Maybe, a command line option will be required for them to keep the existing >behavior. > >> - Deprecate kernel boot parameters (possibly keep pcie_aspm=off for >> debugging use). >> - Use /sys/module/pcie_aspm/parameters/policy for run-time >> system-wide control, including for future hot-added devices. >> - Remove CONFIG_PCIEASPM_DEBUG and enable that code always, so we >> have fine-grained run-time control. >> > >Runtime control sounds like a better plan. We need hooks into the system >power management policy. > >>> Maybe, we are missing a HPP option from the PCI spec. >> That's an interesting idea. _HPX does have provision for manipulating >> Link Control bits (see ACPI r5.0, sec 6.2.8.3), but I don't think very >> many systems implement it. And there's currently no connection >> between program_hpp_type2() and aspm.c, so I'm a little worried that >> we might have issues if a system did implement an _HPX that sets any >> of the ASPM bits. > >I looked at the spec some more. These are there to restore the register >settings following hotplug insertion. I agree it won't play nice with ASPM >as the control bits need to be enabled in coordination with the upstream >device. > >I think the right approach is to let the userspace feed the required >policy to the kernel like you suggested. Then, it needs to be per port >via link_state to have the most flexibility. > > >-- >Sinan Kaya >Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. >Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Christian Lamprechter Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 ^ permalink raw reply related [flat|nested] 94+ messages in thread
* Re: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init 2017-04-21 7:46 ` Patel, Mayurkumar @ 2017-04-21 13:50 ` Sinan Kaya -1 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-21 13:50 UTC (permalink / raw) To: Patel, Mayurkumar, Bjorn Helgaas Cc: Bjorn Helgaas, Rajat Jain, Rajat Jain, David Daney, linux-pci, Timur Tabi, linux-kernel, Julia Lawall, linux-arm-msm, Yinghai Lu, Shawn Lin, linux-arm, Myron Stowe On 4/21/2017 3:46 AM, Patel, Mayurkumar wrote: > If we want to follow above approach then shall we consider having something similar as following? Do you see this problem if you boot with pcie_aspm.policy=powersave option? -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-04-21 13:50 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-21 13:50 UTC (permalink / raw) To: linux-arm-kernel On 4/21/2017 3:46 AM, Patel, Mayurkumar wrote: > If we want to follow above approach then shall we consider having something similar as following? Do you see this problem if you boot with pcie_aspm.policy=powersave option? -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 94+ messages in thread
* RE: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init 2017-04-21 13:50 ` Sinan Kaya (?) (?) @ 2017-04-21 14:13 ` Patel, Mayurkumar -1 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-04-21 14:13 UTC (permalink / raw) To: Sinan Kaya, Bjorn Helgaas Cc: Rajat Jain, Myron Stowe, David Daney, linux-pci, Timur Tabi, linux-kernel, Julia Lawall, Bjorn Helgaas, linux-arm-msm, Rajat Jain, Yinghai Lu, Shawn Lin, linux-arm > >On 4/21/2017 3:46 AM, Patel, Mayurkumar wrote: >> If we want to follow above approach then shall we consider having something similar as following? > >Do you see this problem if you boot with pcie_aspm.policy=powersave option? > No problems. with pcie_aspm.policy=powersave(L1SS are not enabled in this case but L1 stays ok all the time after many Power(hotplug) cycles but I think that is expected with this policy) and pcie_aspm.policy=powersupersave (L1/L1SS both stays ok all the time). > >-- >Sinan Kaya >Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. >Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Christian Lamprechter Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-04-21 14:13 ` Patel, Mayurkumar 0 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-04-21 14:13 UTC (permalink / raw) To: linux-arm-kernel > >On 4/21/2017 3:46 AM, Patel, Mayurkumar wrote: >> If we want to follow above approach then shall we consider having something similar as following? > >Do you see this problem if you boot with pcie_aspm.policy=powersave option? > No problems. with pcie_aspm.policy=powersave(L1SS are not enabled in this case but L1 stays ok all the time after many Power(hotplug) cycles but I think that is expected with this policy) and pcie_aspm.policy=powersupersave (L1/L1SS both stays ok all the time). > >-- >Sinan Kaya >Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. >Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Christian Lamprechter Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 ^ permalink raw reply [flat|nested] 94+ messages in thread
* RE: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-04-21 14:13 ` Patel, Mayurkumar 0 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-04-21 14:13 UTC (permalink / raw) To: Sinan Kaya, Bjorn Helgaas Cc: Bjorn Helgaas, Rajat Jain, Rajat Jain, David Daney, linux-pci, Timur Tabi, linux-kernel, Julia Lawall, linux-arm-msm, Yinghai Lu, Shawn Lin, linux-arm, Myron Stowe Pg0KPk9uIDQvMjEvMjAxNyAzOjQ2IEFNLCBQYXRlbCwgTWF5dXJrdW1hciB3cm90ZToNCj4+IElm IHdlIHdhbnQgdG8gZm9sbG93IGFib3ZlIGFwcHJvYWNoIHRoZW4gc2hhbGwgd2UgY29uc2lkZXIg aGF2aW5nIHNvbWV0aGluZyBzaW1pbGFyIGFzIGZvbGxvd2luZz8NCj4NCj5EbyB5b3Ugc2VlIHRo aXMgcHJvYmxlbSBpZiB5b3UgYm9vdCB3aXRoIHBjaWVfYXNwbS5wb2xpY3k9cG93ZXJzYXZlIG9w dGlvbj8NCj4NCg0KTm8gcHJvYmxlbXMuIHdpdGggcGNpZV9hc3BtLnBvbGljeT1wb3dlcnNhdmUo TDFTUyBhcmUgbm90IGVuYWJsZWQgaW4gdGhpcyBjYXNlDQpidXQgTDEgc3RheXMgb2sgYWxsIHRo ZSB0aW1lIGFmdGVyIG1hbnkgUG93ZXIoaG90cGx1ZykgY3ljbGVzIGJ1dCBJIHRoaW5rIHRoYXQg aXMgZXhwZWN0ZWQgd2l0aCB0aGlzIHBvbGljeSkNCmFuZCBwY2llX2FzcG0ucG9saWN5PXBvd2Vy c3VwZXJzYXZlIChMMS9MMVNTIGJvdGggc3RheXMgb2sgYWxsIHRoZSB0aW1lKS4NCg0KPg0KPi0t DQo+U2luYW4gS2F5YQ0KPlF1YWxjb21tIERhdGFjZW50ZXIgVGVjaG5vbG9naWVzLCBJbmMuIGFz IGFuIGFmZmlsaWF0ZSBvZiBRdWFsY29tbSBUZWNobm9sb2dpZXMsIEluYy4NCj5RdWFsY29tbSBU ZWNobm9sb2dpZXMsIEluYy4gaXMgYSBtZW1iZXIgb2YgdGhlIENvZGUgQXVyb3JhIEZvcnVtLCBh IExpbnV4IEZvdW5kYXRpb24gQ29sbGFib3JhdGl2ZSBQcm9qZWN0Lg0KSW50ZWwgRGV1dHNjaGxh bmQgR21iSApSZWdpc3RlcmVkIEFkZHJlc3M6IEFtIENhbXBlb24gMTAtMTIsIDg1NTc5IE5ldWJp YmVyZywgR2VybWFueQpUZWw6ICs0OSA4OSA5OSA4ODUzLTAsIHd3dy5pbnRlbC5kZQpNYW5hZ2lu ZyBEaXJlY3RvcnM6IENocmlzdGluIEVpc2Vuc2NobWlkLCBDaHJpc3RpYW4gTGFtcHJlY2h0ZXIK Q2hhaXJwZXJzb24gb2YgdGhlIFN1cGVydmlzb3J5IEJvYXJkOiBOaWNvbGUgTGF1ClJlZ2lzdGVy ZWQgT2ZmaWNlOiBNdW5pY2gKQ29tbWVyY2lhbCBSZWdpc3RlcjogQW10c2dlcmljaHQgTXVlbmNo ZW4gSFJCIDE4NjkyOAo= ^ permalink raw reply [flat|nested] 94+ messages in thread
* RE: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-04-21 14:13 ` Patel, Mayurkumar 0 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-04-21 14:13 UTC (permalink / raw) To: Sinan Kaya, Bjorn Helgaas Cc: Bjorn Helgaas, Rajat Jain, Rajat Jain, David Daney, linux-pci, Timur Tabi, linux-kernel, Julia Lawall, linux-arm-msm, Yinghai Lu, Shawn Lin, linux-arm, Myron Stowe > >On 4/21/2017 3:46 AM, Patel, Mayurkumar wrote: >> If we want to follow above approach then shall we consider having something similar as following? > >Do you see this problem if you boot with pcie_aspm.policy=powersave option? > No problems. with pcie_aspm.policy=powersave(L1SS are not enabled in this case but L1 stays ok all the time after many Power(hotplug) cycles but I think that is expected with this policy) and pcie_aspm.policy=powersupersave (L1/L1SS both stays ok all the time). > >-- >Sinan Kaya >Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. >Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Christian Lamprechter Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init 2017-04-21 7:46 ` Patel, Mayurkumar @ 2017-04-25 18:45 ` Bjorn Helgaas -1 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-25 18:45 UTC (permalink / raw) To: Patel, Mayurkumar Cc: Sinan Kaya, Bjorn Helgaas, Rajat Jain, Rajat Jain, David Daney, linux-pci, Timur Tabi, linux-kernel, Julia Lawall, linux-arm-msm, Yinghai Lu, Shawn Lin, linux-arm, Myron Stowe On Fri, Apr 21, 2017 at 2:46 AM, Patel, Mayurkumar <mayurkumar.patel@intel.com> wrote: > Hi Bjorn/Kaya, > > >> >>On 4/17/2017 12:38 PM, Bjorn Helgaas wrote: >>>> Like you said, what do we do by default is the question. Should we opt >>>> for safe like we are doing, or try to save some power. >>> I think safety is paramount. Every user should be able to boot safely >>> without any kernel parameters. We don't want users to have a problem >>> booting and then have to search for a workaround like booting with >>> "pcie_aspm=off". Most users will never do that. >>> >> >>OK, no problem with leaving the behavior as it is. >> >>My initial approach was #2. We knew this way that user had full control >>over the ASPM policy by changing the BIOS option. Then, Mayurkumar >>complained that ASPM is not enabled following a hotplug insertion to an >>empty slot. That's when I switched to #3 as it sounded like a good thing >>to have for us. >> >>> Here's a long-term strawman proposal, see what you think: >>> >>> - Deprecate CONFIG_PCIEASPM_DEFAULT, CONFIG_PCIEASPM_POWERSAVE, etc. >>> - Default aspm_policy is POLICY_DEFAULT always. >>> - POLICY_DEFAULT means Linux doesn't touch anything: if BIOS enabled >>> ASPM, we leave it that way; we leave ASPM disabled on hot-added >>> devices. >> > I am also ok with leaving the same behavior as now. > But still following is something open I feel besides, Which may be there in your comments redundantly. > The current problem is, pcie_aspm_exit_link_state() disables the ASPM configuration even > if POLICY_DEFAULT was set. We call pcie_aspm_exit_link_state() when removing an endpoint. When we remove an endpoint, I think disabling ASPM is the right thing to do. The spec (PCIe r3.1, sec 5.4.1.3) says "Software must not enable L0s in either direction on a given Link unless components on both sides of the Link each support L0s; otherwise, the result is undefined." > I am seeing already following problem(or may be influence) with it. The Endpoint I have does not have > does not have "Presence detect change" mechanism. Hot plug is working with Link status events. > When link is in L1 or L1SS and if EP is powered off, no Link status change event are triggered (It might be > the expected behavior in L1 or L1SS). When next time EP is powered on there are link down and > link up events coming one after other. BIOS enables ASPM on Root port and Endpoint, but while > processing link status down, pcie_aspm_exit_link_state() clears the ASPM already which were enabled by BIOS. > If we want to follow above approach then shall we consider having something similar as following? The proposal was to leave ASPM disabled on hot-added devices. If the endpoint was powered off and powered back on again, I think that device looks like a hot-added device, doesn't it? Bjorn ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-04-25 18:45 ` Bjorn Helgaas 0 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-25 18:45 UTC (permalink / raw) To: linux-arm-kernel On Fri, Apr 21, 2017 at 2:46 AM, Patel, Mayurkumar <mayurkumar.patel@intel.com> wrote: > Hi Bjorn/Kaya, > > >> >>On 4/17/2017 12:38 PM, Bjorn Helgaas wrote: >>>> Like you said, what do we do by default is the question. Should we opt >>>> for safe like we are doing, or try to save some power. >>> I think safety is paramount. Every user should be able to boot safely >>> without any kernel parameters. We don't want users to have a problem >>> booting and then have to search for a workaround like booting with >>> "pcie_aspm=off". Most users will never do that. >>> >> >>OK, no problem with leaving the behavior as it is. >> >>My initial approach was #2. We knew this way that user had full control >>over the ASPM policy by changing the BIOS option. Then, Mayurkumar >>complained that ASPM is not enabled following a hotplug insertion to an >>empty slot. That's when I switched to #3 as it sounded like a good thing >>to have for us. >> >>> Here's a long-term strawman proposal, see what you think: >>> >>> - Deprecate CONFIG_PCIEASPM_DEFAULT, CONFIG_PCIEASPM_POWERSAVE, etc. >>> - Default aspm_policy is POLICY_DEFAULT always. >>> - POLICY_DEFAULT means Linux doesn't touch anything: if BIOS enabled >>> ASPM, we leave it that way; we leave ASPM disabled on hot-added >>> devices. >> > I am also ok with leaving the same behavior as now. > But still following is something open I feel besides, Which may be there in your comments redundantly. > The current problem is, pcie_aspm_exit_link_state() disables the ASPM configuration even > if POLICY_DEFAULT was set. We call pcie_aspm_exit_link_state() when removing an endpoint. When we remove an endpoint, I think disabling ASPM is the right thing to do. The spec (PCIe r3.1, sec 5.4.1.3) says "Software must not enable L0s in either direction on a given Link unless components on both sides of the Link each support L0s; otherwise, the result is undefined." > I am seeing already following problem(or may be influence) with it. The Endpoint I have does not have > does not have "Presence detect change" mechanism. Hot plug is working with Link status events. > When link is in L1 or L1SS and if EP is powered off, no Link status change event are triggered (It might be > the expected behavior in L1 or L1SS). When next time EP is powered on there are link down and > link up events coming one after other. BIOS enables ASPM on Root port and Endpoint, but while > processing link status down, pcie_aspm_exit_link_state() clears the ASPM already which were enabled by BIOS. > If we want to follow above approach then shall we consider having something similar as following? The proposal was to leave ASPM disabled on hot-added devices. If the endpoint was powered off and powered back on again, I think that device looks like a hot-added device, doesn't it? Bjorn ^ permalink raw reply [flat|nested] 94+ messages in thread
* RE: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init 2017-04-25 18:45 ` Bjorn Helgaas (?) @ 2017-05-02 12:02 ` Patel, Mayurkumar -1 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-05-02 12:02 UTC (permalink / raw) To: Bjorn Helgaas Cc: Sinan Kaya, Bjorn Helgaas, Rajat Jain, Rajat Jain, David Daney, linux-pci, Timur Tabi, linux-kernel, Julia Lawall, linux-arm-msm, Yinghai Lu, Shawn Lin, linux-arm, Myron Stowe Hi Bjorn > >On Fri, Apr 21, 2017 at 2:46 AM, Patel, Mayurkumar ><mayurkumar.patel@intel.com> wrote: >> Hi Bjorn/Kaya, >> >> >>> >>>On 4/17/2017 12:38 PM, Bjorn Helgaas wrote: >>>>> Like you said, what do we do by default is the question. Should we opt >>>>> for safe like we are doing, or try to save some power. >>>> I think safety is paramount. Every user should be able to boot safely >>>> without any kernel parameters. We don't want users to have a problem >>>> booting and then have to search for a workaround like booting with >>>> "pcie_aspm=off". Most users will never do that. >>>> >>> >>>OK, no problem with leaving the behavior as it is. >>> >>>My initial approach was #2. We knew this way that user had full control >>>over the ASPM policy by changing the BIOS option. Then, Mayurkumar >>>complained that ASPM is not enabled following a hotplug insertion to an >>>empty slot. That's when I switched to #3 as it sounded like a good thing >>>to have for us. >>> >>>> Here's a long-term strawman proposal, see what you think: >>>> >>>> - Deprecate CONFIG_PCIEASPM_DEFAULT, CONFIG_PCIEASPM_POWERSAVE, etc. >>>> - Default aspm_policy is POLICY_DEFAULT always. >>>> - POLICY_DEFAULT means Linux doesn't touch anything: if BIOS enabled >>>> ASPM, we leave it that way; we leave ASPM disabled on hot-added >>>> devices. >>> >> I am also ok with leaving the same behavior as now. >> But still following is something open I feel besides, Which may be there in your comments redundantly. >> The current problem is, pcie_aspm_exit_link_state() disables the ASPM configuration even >> if POLICY_DEFAULT was set. > >We call pcie_aspm_exit_link_state() when removing an endpoint. When >we remove an endpoint, I think disabling ASPM is the right thing to >do. The spec (PCIe r3.1, sec 5.4.1.3) says "Software must not enable >L0s in either direction on a given Link unless components on both >sides of the Link each support L0s; otherwise, the result is >undefined." > Yes, you are right and per spec also it makes sense that ASPM needs to be disabled. But, if POLICY_DEFAULT is set then, shouldn't BIOS take care of disabling ASPM? >> I am seeing already following problem(or may be influence) with it. The Endpoint I have does not have >> does not have "Presence detect change" mechanism. Hot plug is working with Link status events. >> When link is in L1 or L1SS and if EP is powered off, no Link status change event are triggered (It might be >> the expected behavior in L1 or L1SS). When next time EP is powered on there are link down and >> link up events coming one after other. BIOS enables ASPM on Root port and Endpoint, but while >> processing link status down, pcie_aspm_exit_link_state() clears the ASPM already which were enabled by BIOS. >> If we want to follow above approach then shall we consider having something similar as following? > >The proposal was to leave ASPM disabled on hot-added devices. If the >endpoint was powered off and powered back on again, I think that >device looks like a hot-added device, doesn't it? > Yes, it is hot-added device. Also, I understand, for POLICY_DEFAULT, OS would/should not touch ASPM(enable/disable), but BIOS could still (enable/disable), right? Currently, what happens in my system is as following, (each 2nd power cycle/hotplug of Endpoint disables ASPM): First Power cycle (When ASPM L1 is already enabled): device gets powered off -> there are no Link status events, so no pcie hotplug interrupt and pcie_aspm_exit_link_state() triggered. When the device gets powered on again -> Link down/Link up events are coming back to back. First Link down is served. (BIOS checks for the Link status and enables ASPM already, as the device is actually powered back). OS calls pcie_aspm_exit_link_state() and ASPM gets disabled by OS. Second Power cycle (When ASPM L1 is disabled after above): device gets powered off -> there are link status events, pcie hotplug interrupt is triggered and pcie_aspm_exit_link_state() triggered. OS disables ASPM. BIOS checks Link status and disables ASPM too. When the device gets powered on -> BIOS enables ASPM and as this is pcie hotplug insertion, OS does not interfere and we have ASPM enabled. The above sequence happens each 2nd power cycle of the hotplug device. So One could still argue if POLICY_DEFAULT is set, then why OS disables ASPM if it is not meant to touch configuration. This is why I proposed following kind of change, so that OS would not touch ASPM, if POLICY_DEFAULT is set. Also, With the below change, everything relies on BIOS for ASPM when POLICY_DEFAULT is set and I see above problem gets resolved. Also, the existing ASPM behavior does not have impact, unless specific BIOS does not disable ASPM on Root Port when device gets removed. >Bjorn Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Christian Lamprechter Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-05-02 12:02 ` Patel, Mayurkumar 0 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-05-02 12:02 UTC (permalink / raw) To: linux-arm-kernel Hi Bjorn > >On Fri, Apr 21, 2017 at 2:46 AM, Patel, Mayurkumar ><mayurkumar.patel@intel.com> wrote: >> Hi Bjorn/Kaya, >> >> >>> >>>On 4/17/2017 12:38 PM, Bjorn Helgaas wrote: >>>>> Like you said, what do we do by default is the question. Should we opt >>>>> for safe like we are doing, or try to save some power. >>>> I think safety is paramount. Every user should be able to boot safely >>>> without any kernel parameters. We don't want users to have a problem >>>> booting and then have to search for a workaround like booting with >>>> "pcie_aspm=off". Most users will never do that. >>>> >>> >>>OK, no problem with leaving the behavior as it is. >>> >>>My initial approach was #2. We knew this way that user had full control >>>over the ASPM policy by changing the BIOS option. Then, Mayurkumar >>>complained that ASPM is not enabled following a hotplug insertion to an >>>empty slot. That's when I switched to #3 as it sounded like a good thing >>>to have for us. >>> >>>> Here's a long-term strawman proposal, see what you think: >>>> >>>> - Deprecate CONFIG_PCIEASPM_DEFAULT, CONFIG_PCIEASPM_POWERSAVE, etc. >>>> - Default aspm_policy is POLICY_DEFAULT always. >>>> - POLICY_DEFAULT means Linux doesn't touch anything: if BIOS enabled >>>> ASPM, we leave it that way; we leave ASPM disabled on hot-added >>>> devices. >>> >> I am also ok with leaving the same behavior as now. >> But still following is something open I feel besides, Which may be there in your comments redundantly. >> The current problem is, pcie_aspm_exit_link_state() disables the ASPM configuration even >> if POLICY_DEFAULT was set. > >We call pcie_aspm_exit_link_state() when removing an endpoint. When >we remove an endpoint, I think disabling ASPM is the right thing to >do. The spec (PCIe r3.1, sec 5.4.1.3) says "Software must not enable >L0s in either direction on a given Link unless components on both >sides of the Link each support L0s; otherwise, the result is >undefined." > Yes, you are right and per spec also it makes sense that ASPM needs to be disabled. But, if POLICY_DEFAULT is set then, shouldn't BIOS take care of disabling ASPM? >> I am seeing already following problem(or may be influence) with it. The Endpoint I have does not have >> does not have "Presence detect change" mechanism. Hot plug is working with Link status events. >> When link is in L1 or L1SS and if EP is powered off, no Link status change event are triggered (It might be >> the expected behavior in L1 or L1SS). When next time EP is powered on there are link down and >> link up events coming one after other. BIOS enables ASPM on Root port and Endpoint, but while >> processing link status down, pcie_aspm_exit_link_state() clears the ASPM already which were enabled by BIOS. >> If we want to follow above approach then shall we consider having something similar as following? > >The proposal was to leave ASPM disabled on hot-added devices. If the >endpoint was powered off and powered back on again, I think that >device looks like a hot-added device, doesn't it? > Yes, it is hot-added device. Also, I understand, for POLICY_DEFAULT, OS would/should not touch ASPM(enable/disable), but BIOS could still (enable/disable), right? Currently, what happens in my system is as following, (each 2nd power cycle/hotplug of Endpoint disables ASPM): First Power cycle (When ASPM L1 is already enabled): device gets powered off -> there are no Link status events, so no pcie hotplug interrupt and pcie_aspm_exit_link_state() triggered. When the device gets powered on again -> Link down/Link up events are coming back to back. First Link down is served. (BIOS checks for the Link status and enables ASPM already, as the device is actually powered back). OS calls pcie_aspm_exit_link_state() and ASPM gets disabled by OS. Second Power cycle (When ASPM L1 is disabled after above): device gets powered off -> there are link status events, pcie hotplug interrupt is triggered and pcie_aspm_exit_link_state() triggered. OS disables ASPM. BIOS checks Link status and disables ASPM too. When the device gets powered on -> BIOS enables ASPM and as this is pcie hotplug insertion, OS does not interfere and we have ASPM enabled. The above sequence happens each 2nd power cycle of the hotplug device. So One could still argue if POLICY_DEFAULT is set, then why OS disables ASPM if it is not meant to touch configuration. This is why I proposed following kind of change, so that OS would not touch ASPM, if POLICY_DEFAULT is set. Also, With the below change, everything relies on BIOS for ASPM when POLICY_DEFAULT is set and I see above problem gets resolved. Also, the existing ASPM behavior does not have impact, unless specific BIOS does not disable ASPM on Root Port when device gets removed. >Bjorn Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Christian Lamprechter Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 ^ permalink raw reply [flat|nested] 94+ messages in thread
* RE: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-05-02 12:02 ` Patel, Mayurkumar 0 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-05-02 12:02 UTC (permalink / raw) To: Bjorn Helgaas Cc: Sinan Kaya, Bjorn Helgaas, Rajat Jain, Rajat Jain, David Daney, linux-pci, Timur Tabi, linux-kernel, Julia Lawall, linux-arm-msm, Yinghai Lu, Shawn Lin, linux-arm, Myron Stowe SGkgQmpvcm4NCg0KPg0KPk9uIEZyaSwgQXByIDIxLCAyMDE3IGF0IDI6NDYgQU0sIFBhdGVsLCBN YXl1cmt1bWFyDQo+PG1heXVya3VtYXIucGF0ZWxAaW50ZWwuY29tPiB3cm90ZToNCj4+IEhpIEJq b3JuL0theWEsDQo+Pg0KPj4NCj4+Pg0KPj4+T24gNC8xNy8yMDE3IDEyOjM4IFBNLCBCam9ybiBI ZWxnYWFzIHdyb3RlOg0KPj4+Pj4gTGlrZSB5b3Ugc2FpZCwgd2hhdCBkbyB3ZSBkbyBieSBkZWZh dWx0IGlzIHRoZSBxdWVzdGlvbi4gU2hvdWxkIHdlIG9wdA0KPj4+Pj4gZm9yIHNhZmUgbGlrZSB3 ZSBhcmUgZG9pbmcsIG9yIHRyeSB0byBzYXZlIHNvbWUgcG93ZXIuDQo+Pj4+IEkgdGhpbmsgc2Fm ZXR5IGlzIHBhcmFtb3VudC4gIEV2ZXJ5IHVzZXIgc2hvdWxkIGJlIGFibGUgdG8gYm9vdCBzYWZl bHkNCj4+Pj4gd2l0aG91dCBhbnkga2VybmVsIHBhcmFtZXRlcnMuICBXZSBkb24ndCB3YW50IHVz ZXJzIHRvIGhhdmUgYSBwcm9ibGVtDQo+Pj4+IGJvb3RpbmcgYW5kIHRoZW4gaGF2ZSB0byBzZWFy Y2ggZm9yIGEgd29ya2Fyb3VuZCBsaWtlIGJvb3Rpbmcgd2l0aA0KPj4+PiAicGNpZV9hc3BtPW9m ZiIuICBNb3N0IHVzZXJzIHdpbGwgbmV2ZXIgZG8gdGhhdC4NCj4+Pj4NCj4+Pg0KPj4+T0ssIG5v IHByb2JsZW0gd2l0aCBsZWF2aW5nIHRoZSBiZWhhdmlvciBhcyBpdCBpcy4NCj4+Pg0KPj4+TXkg aW5pdGlhbCBhcHByb2FjaCB3YXMgIzIuIFdlIGtuZXcgdGhpcyB3YXkgdGhhdCB1c2VyIGhhZCBm dWxsIGNvbnRyb2wNCj4+Pm92ZXIgdGhlIEFTUE0gcG9saWN5IGJ5IGNoYW5naW5nIHRoZSBCSU9T IG9wdGlvbi4gVGhlbiwgTWF5dXJrdW1hcg0KPj4+Y29tcGxhaW5lZCB0aGF0IEFTUE0gaXMgbm90 IGVuYWJsZWQgZm9sbG93aW5nIGEgaG90cGx1ZyBpbnNlcnRpb24gdG8gYW4NCj4+PmVtcHR5IHNs b3QuIFRoYXQncyB3aGVuIEkgc3dpdGNoZWQgdG8gIzMgYXMgaXQgc291bmRlZCBsaWtlIGEgZ29v ZCB0aGluZw0KPj4+dG8gaGF2ZSBmb3IgdXMuDQo+Pj4NCj4+Pj4gSGVyZSdzIGEgbG9uZy10ZXJt IHN0cmF3bWFuIHByb3Bvc2FsLCBzZWUgd2hhdCB5b3UgdGhpbms6DQo+Pj4+DQo+Pj4+ICAgLSBE ZXByZWNhdGUgQ09ORklHX1BDSUVBU1BNX0RFRkFVTFQsIENPTkZJR19QQ0lFQVNQTV9QT1dFUlNB VkUsIGV0Yy4NCj4+Pj4gICAtIERlZmF1bHQgYXNwbV9wb2xpY3kgaXMgUE9MSUNZX0RFRkFVTFQg YWx3YXlzLg0KPj4+PiAgIC0gUE9MSUNZX0RFRkFVTFQgbWVhbnMgTGludXggZG9lc24ndCB0b3Vj aCBhbnl0aGluZzogaWYgQklPUyBlbmFibGVkDQo+Pj4+IEFTUE0sIHdlIGxlYXZlIGl0IHRoYXQg d2F5OyB3ZSBsZWF2ZSBBU1BNIGRpc2FibGVkIG9uIGhvdC1hZGRlZA0KPj4+PiBkZXZpY2VzLg0K Pj4+DQo+PiBJIGFtIGFsc28gb2sgd2l0aCBsZWF2aW5nIHRoZSBzYW1lIGJlaGF2aW9yIGFzIG5v dy4NCj4+IEJ1dCBzdGlsbCBmb2xsb3dpbmcgaXMgc29tZXRoaW5nIG9wZW4gSSBmZWVsIGJlc2lk ZXMsIFdoaWNoIG1heSBiZSB0aGVyZSBpbiB5b3VyIGNvbW1lbnRzIHJlZHVuZGFudGx5Lg0KPj4g VGhlIGN1cnJlbnQgcHJvYmxlbSBpcywgcGNpZV9hc3BtX2V4aXRfbGlua19zdGF0ZSgpIGRpc2Fi bGVzIHRoZSBBU1BNIGNvbmZpZ3VyYXRpb24gZXZlbg0KPj4gaWYgUE9MSUNZX0RFRkFVTFQgd2Fz IHNldC4NCj4NCj5XZSBjYWxsIHBjaWVfYXNwbV9leGl0X2xpbmtfc3RhdGUoKSB3aGVuIHJlbW92 aW5nIGFuIGVuZHBvaW50LiAgV2hlbg0KPndlIHJlbW92ZSBhbiBlbmRwb2ludCwgSSB0aGluayBk aXNhYmxpbmcgQVNQTSBpcyB0aGUgcmlnaHQgdGhpbmcgdG8NCj5kby4gIFRoZSBzcGVjIChQQ0ll IHIzLjEsIHNlYyA1LjQuMS4zKSBzYXlzICJTb2Z0d2FyZSBtdXN0IG5vdCBlbmFibGUNCj5MMHMg aW4gZWl0aGVyIGRpcmVjdGlvbiBvbiBhIGdpdmVuIExpbmsgdW5sZXNzIGNvbXBvbmVudHMgb24g Ym90aA0KPnNpZGVzIG9mIHRoZSBMaW5rIGVhY2ggc3VwcG9ydCBMMHM7IG90aGVyd2lzZSwgdGhl IHJlc3VsdCBpcw0KPnVuZGVmaW5lZC4iDQo+DQoNClllcywgeW91IGFyZSByaWdodCBhbmQgcGVy IHNwZWMgYWxzbyBpdCBtYWtlcyBzZW5zZSB0aGF0IEFTUE0gbmVlZHMgdG8gYmUgZGlzYWJsZWQu DQpCdXQsIGlmIFBPTElDWV9ERUZBVUxUIGlzIHNldCB0aGVuLCBzaG91bGRuJ3QgQklPUyB0YWtl IGNhcmUgb2YgZGlzYWJsaW5nIEFTUE0/DQoNCg0KDQo+PiBJIGFtIHNlZWluZyBhbHJlYWR5IGZv bGxvd2luZyBwcm9ibGVtKG9yIG1heSBiZSBpbmZsdWVuY2UpIHdpdGggaXQuIFRoZSBFbmRwb2lu dCBJIGhhdmUgZG9lcyBub3QgaGF2ZQ0KPj4gZG9lcyBub3QgaGF2ZSAiUHJlc2VuY2UgZGV0ZWN0 IGNoYW5nZSIgbWVjaGFuaXNtLiBIb3QgcGx1ZyBpcyB3b3JraW5nIHdpdGggTGluayBzdGF0dXMg ZXZlbnRzLg0KPj4gV2hlbiBsaW5rIGlzIGluIEwxIG9yIEwxU1MgYW5kIGlmIEVQIGlzIHBvd2Vy ZWQgb2ZmLCBubyBMaW5rIHN0YXR1cyBjaGFuZ2UgZXZlbnQgYXJlIHRyaWdnZXJlZCAoSXQgbWln aHQgYmUNCj4+IHRoZSBleHBlY3RlZCBiZWhhdmlvciBpbiBMMSBvciBMMVNTKS4gIFdoZW4gbmV4 dCB0aW1lIEVQIGlzIHBvd2VyZWQgb24gdGhlcmUgYXJlIGxpbmsgZG93biBhbmQNCj4+IGxpbmsg dXAgZXZlbnRzIGNvbWluZyBvbmUgYWZ0ZXIgb3RoZXIuIEJJT1MgZW5hYmxlcyBBU1BNIG9uIFJv b3QgcG9ydCBhbmQgRW5kcG9pbnQsIGJ1dCB3aGlsZQ0KPj4gcHJvY2Vzc2luZyBsaW5rIHN0YXR1 cyBkb3duLCBwY2llX2FzcG1fZXhpdF9saW5rX3N0YXRlKCkgY2xlYXJzIHRoZSBBU1BNIGFscmVh ZHkgd2hpY2ggd2VyZSBlbmFibGVkIGJ5IEJJT1MuDQo+PiBJZiB3ZSB3YW50IHRvIGZvbGxvdyBh Ym92ZSBhcHByb2FjaCB0aGVuIHNoYWxsIHdlIGNvbnNpZGVyIGhhdmluZyBzb21ldGhpbmcgc2lt aWxhciBhcyBmb2xsb3dpbmc/DQo+DQo+VGhlIHByb3Bvc2FsIHdhcyB0byBsZWF2ZSBBU1BNIGRp c2FibGVkIG9uIGhvdC1hZGRlZCBkZXZpY2VzLiAgSWYgdGhlDQo+ZW5kcG9pbnQgd2FzIHBvd2Vy ZWQgb2ZmIGFuZCBwb3dlcmVkIGJhY2sgb24gYWdhaW4sIEkgdGhpbmsgdGhhdA0KPmRldmljZSBs b29rcyBsaWtlIGEgaG90LWFkZGVkIGRldmljZSwgZG9lc24ndCBpdD8NCj4NCg0KWWVzLCBpdCBp cyBob3QtYWRkZWQgZGV2aWNlLiBBbHNvLCBJIHVuZGVyc3RhbmQsIGZvciBQT0xJQ1lfREVGQVVM VCwgT1Mgd291bGQvc2hvdWxkIG5vdCB0b3VjaCBBU1BNKGVuYWJsZS9kaXNhYmxlKSwNCmJ1dCBC SU9TIGNvdWxkIHN0aWxsIChlbmFibGUvZGlzYWJsZSksIHJpZ2h0Pw0KDQpDdXJyZW50bHksIHdo YXQgaGFwcGVucyBpbiBteSBzeXN0ZW0gaXMgYXMgZm9sbG93aW5nLCAoZWFjaCAybmQgcG93ZXIg Y3ljbGUvaG90cGx1ZyBvZiBFbmRwb2ludCBkaXNhYmxlcyBBU1BNKToNCg0KDQpGaXJzdCBQb3dl ciBjeWNsZSAoV2hlbiBBU1BNIEwxIGlzIGFscmVhZHkgZW5hYmxlZCk6DQpkZXZpY2UgZ2V0cyBw b3dlcmVkIG9mZiAtPiB0aGVyZSBhcmUgbm8gTGluayBzdGF0dXMgZXZlbnRzLCBzbyBubyBwY2ll IGhvdHBsdWcgaW50ZXJydXB0IGFuZCBwY2llX2FzcG1fZXhpdF9saW5rX3N0YXRlKCkgdHJpZ2dl cmVkLg0KV2hlbiB0aGUgZGV2aWNlIGdldHMgcG93ZXJlZCBvbiBhZ2FpbiAtPiBMaW5rIGRvd24v TGluayB1cCBldmVudHMgYXJlIGNvbWluZyBiYWNrIHRvIGJhY2suIA0KRmlyc3QgTGluayBkb3du IGlzIHNlcnZlZC4gKEJJT1MgY2hlY2tzIGZvciB0aGUgTGluayBzdGF0dXMgYW5kIGVuYWJsZXMg QVNQTSBhbHJlYWR5LCBhcyB0aGUgZGV2aWNlIGlzDQphY3R1YWxseSBwb3dlcmVkIGJhY2spLiBP UyBjYWxscyBwY2llX2FzcG1fZXhpdF9saW5rX3N0YXRlKCkgYW5kIEFTUE0gZ2V0cyBkaXNhYmxl ZCBieSBPUy4NCg0KU2Vjb25kIFBvd2VyIGN5Y2xlIChXaGVuIEFTUE0gTDEgaXMgZGlzYWJsZWQg YWZ0ZXIgYWJvdmUpOg0KZGV2aWNlIGdldHMgcG93ZXJlZCBvZmYgLT4gdGhlcmUgYXJlIGxpbmsg c3RhdHVzIGV2ZW50cywgcGNpZSBob3RwbHVnIGludGVycnVwdCBpcyB0cmlnZ2VyZWQgYW5kIHBj aWVfYXNwbV9leGl0X2xpbmtfc3RhdGUoKSB0cmlnZ2VyZWQuDQpPUyBkaXNhYmxlcyBBU1BNLiBC SU9TIGNoZWNrcyBMaW5rIHN0YXR1cyBhbmQgZGlzYWJsZXMgQVNQTSB0b28uDQpXaGVuIHRoZSBk ZXZpY2UgZ2V0cyBwb3dlcmVkIG9uIC0+IEJJT1MgZW5hYmxlcyBBU1BNIGFuZCBhcyB0aGlzIGlz IHBjaWUgaG90cGx1ZyBpbnNlcnRpb24sIE9TDQpkb2VzIG5vdCBpbnRlcmZlcmUgYW5kIHdlIGhh dmUgQVNQTSBlbmFibGVkLg0KDQpUaGUgYWJvdmUgc2VxdWVuY2UgaGFwcGVucyBlYWNoIDJuZCBw b3dlciBjeWNsZSBvZiB0aGUgaG90cGx1ZyBkZXZpY2UuDQoNClNvIE9uZSBjb3VsZCBzdGlsbCBh cmd1ZSBpZiBQT0xJQ1lfREVGQVVMVCBpcyBzZXQsIHRoZW4gd2h5IE9TIGRpc2FibGVzIEFTUE0g aWYgaXQgaXMgbm90IG1lYW50IHRvIHRvdWNoIGNvbmZpZ3VyYXRpb24uDQpUaGlzIGlzIHdoeSBJ IHByb3Bvc2VkIGZvbGxvd2luZyBraW5kIG9mIGNoYW5nZSwgc28gdGhhdCBPUyB3b3VsZCBub3Qg dG91Y2ggQVNQTSwgaWYgUE9MSUNZX0RFRkFVTFQgaXMgc2V0Lg0KQWxzbywgV2l0aCB0aGUgYmVs b3cgY2hhbmdlLCBldmVyeXRoaW5nIHJlbGllcyBvbiBCSU9TIGZvciBBU1BNIHdoZW4gUE9MSUNZ X0RFRkFVTFQgaXMgc2V0IGFuZCBJIHNlZSBhYm92ZSBwcm9ibGVtDQpnZXRzIHJlc29sdmVkLiBB bHNvLCB0aGUgZXhpc3RpbmcgQVNQTSBiZWhhdmlvciBkb2VzIG5vdCBoYXZlIGltcGFjdCwgdW5s ZXNzIHNwZWNpZmljIEJJT1MgZG9lcyBub3QgZGlzYWJsZSBBU1BNIG9uDQpSb290IFBvcnQgd2hl biBkZXZpY2UgZ2V0cyByZW1vdmVkLg0KDQoNCg0KPkJqb3JuDQpJbnRlbCBEZXV0c2NobGFuZCBH bWJIClJlZ2lzdGVyZWQgQWRkcmVzczogQW0gQ2FtcGVvbiAxMC0xMiwgODU1NzkgTmV1YmliZXJn LCBHZXJtYW55ClRlbDogKzQ5IDg5IDk5IDg4NTMtMCwgd3d3LmludGVsLmRlCk1hbmFnaW5nIERp cmVjdG9yczogQ2hyaXN0aW4gRWlzZW5zY2htaWQsIENocmlzdGlhbiBMYW1wcmVjaHRlcgpDaGFp cnBlcnNvbiBvZiB0aGUgU3VwZXJ2aXNvcnkgQm9hcmQ6IE5pY29sZSBMYXUKUmVnaXN0ZXJlZCBP ZmZpY2U6IE11bmljaApDb21tZXJjaWFsIFJlZ2lzdGVyOiBBbXRzZ2VyaWNodCBNdWVuY2hlbiBI UkIgMTg2OTI4Cg== ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init 2017-05-02 12:02 ` Patel, Mayurkumar (?) @ 2017-05-03 21:10 ` Bjorn Helgaas -1 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-05-03 21:10 UTC (permalink / raw) To: Patel, Mayurkumar Cc: Bjorn Helgaas, Sinan Kaya, Rajat Jain, Rajat Jain, David Daney, linux-pci, Timur Tabi, linux-kernel, Julia Lawall, linux-arm-msm, Yinghai Lu, Shawn Lin, linux-arm, Myron Stowe On Tue, May 02, 2017 at 12:02:53PM +0000, Patel, Mayurkumar wrote: > >On Fri, Apr 21, 2017 at 2:46 AM, Patel, Mayurkumar > ><mayurkumar.patel@intel.com> wrote: > >>>On 4/17/2017 12:38 PM, Bjorn Helgaas wrote: > >>>>> Like you said, what do we do by default is the question. Should we opt > >>>>> for safe like we are doing, or try to save some power. > >>>> I think safety is paramount. Every user should be able to boot safely > >>>> without any kernel parameters. We don't want users to have a problem > >>>> booting and then have to search for a workaround like booting with > >>>> "pcie_aspm=off". Most users will never do that. > >>> > >>>OK, no problem with leaving the behavior as it is. > >>> > >>>My initial approach was #2. We knew this way that user had full control > >>>over the ASPM policy by changing the BIOS option. Then, Mayurkumar > >>>complained that ASPM is not enabled following a hotplug insertion to an > >>>empty slot. That's when I switched to #3 as it sounded like a good thing > >>>to have for us. > >>> > >>>> Here's a long-term strawman proposal, see what you think: > >>>> > >>>> - Deprecate CONFIG_PCIEASPM_DEFAULT, CONFIG_PCIEASPM_POWERSAVE, etc. > >>>> - Default aspm_policy is POLICY_DEFAULT always. > >>>> - POLICY_DEFAULT means Linux doesn't touch anything: if BIOS enabled > >>>> ASPM, we leave it that way; we leave ASPM disabled on hot-added > >>>> devices. > >>> > >> I am also ok with leaving the same behavior as now. But still > >> following is something open I feel besides, Which may be there in > >> your comments redundantly. The current problem is, > >> pcie_aspm_exit_link_state() disables the ASPM configuration even > >> if POLICY_DEFAULT was set. > > > >We call pcie_aspm_exit_link_state() when removing an endpoint. When > >we remove an endpoint, I think disabling ASPM is the right thing to > >do. The spec (PCIe r3.1, sec 5.4.1.3) says "Software must not enable > >L0s in either direction on a given Link unless components on both > >sides of the Link each support L0s; otherwise, the result is > >undefined." > > Yes, you are right and per spec also it makes sense that ASPM needs > to be disabled. But, if POLICY_DEFAULT is set then, shouldn't BIOS > take care of disabling ASPM? No, I don't think so. POLICY_DEFAULT is a Linux thing and BIOS doesn't know anything about it. ASPM can be configured by BIOS before handoff to Linux, but after handoff it should be managed either entirely by BIOS or entirely by Linux. If BIOS wants to retain ASPM control, it would have to tell the OS *not* to use ASPM, and it would have to use ACPI hotplug. In this case, POLICY_DEFAULT is irrelevant because Linux shouldn't do anything with ASPM. But normally BIOS allows Linux to control ASPM, and we would use native PCIe hotplug (pciehp) instead of ACPI hotplug, and BIOS has no opportunity to enable or disable ASPM on hotplug events. > >> I am seeing already following problem(or may be influence) with > >> it. The Endpoint I have does not have does not have "Presence > >> detect change" mechanism. Hot plug is working with Link status > >> events. When link is in L1 or L1SS and if EP is powered off, no > >> Link status change event are triggered (It might be the expected > >> behavior in L1 or L1SS). When next time EP is powered on there > >> are link down and link up events coming one after other. BIOS > >> enables ASPM on Root port and Endpoint, but while processing link > >> status down, pcie_aspm_exit_link_state() clears the ASPM already > >> which were enabled by BIOS. If we want to follow above approach > >> then shall we consider having something similar as following? > > > >The proposal was to leave ASPM disabled on hot-added devices. If > >the endpoint was powered off and powered back on again, I think > >that device looks like a hot-added device, doesn't it? > > Yes, it is hot-added device. Also, I understand, for POLICY_DEFAULT, > OS would/should not touch ASPM(enable/disable), but BIOS could still > (enable/disable), right? Maybe I'm misunderstanding your question. There are two questions here: 1) Does the BIOS allow Linux to manage ASPM? 2) If Linux does manage ASPM, what policy does it use? If BIOS doesn't allow Linux to manage ASPM, POLICY_DEFAULT is irrelevant. If BIOS does allow Linux to manage ASPM, POLICY_DEFAULT means Linux should use the settings made by BIOS. The user could select a different policy, and then Linux would change the ASPM configuration accordingly. > Currently, what happens in my system is as following, (each 2nd > power cycle/hotplug of Endpoint disables ASPM): > > > First Power cycle (When ASPM L1 is already enabled): device gets > powered off -> there are no Link status events, so no pcie hotplug > interrupt and pcie_aspm_exit_link_state() triggered. If the Downstream Port leading to your Endpoint is hotplug capable, doesn't the spec require that it can report link state changes (PCIe r3.1, sec 7.8.6, 7.8.10, 7.8.11)? > When the device gets powered on again -> Link down/Link up events > are coming back to back. First Link down is served. (BIOS checks > for the Link status and enables ASPM already, as the device is > actually powered back). OS calls pcie_aspm_exit_link_state() and > ASPM gets disabled by OS. > > Second Power cycle (When ASPM L1 is disabled after above): device > gets powered off -> there are link status events, pcie hotplug > interrupt is triggered and pcie_aspm_exit_link_state() triggered. > OS disables ASPM. BIOS checks Link status and disables ASPM too. > When the device gets powered on -> BIOS enables ASPM and as this is > pcie hotplug insertion, OS does not interfere and we have ASPM > enabled. I don't understand this sequence. If we're using native PCIe hotplug, BIOS should not be involved to enable ASPM when the device is powered on. > The above sequence happens each 2nd power cycle of the hotplug > device. > > So One could still argue if POLICY_DEFAULT is set, then why OS > disables ASPM if it is not meant to touch configuration. This is > why I proposed following kind of change, so that OS would not touch > ASPM, if POLICY_DEFAULT is set. Also, With the below change, > everything relies on BIOS for ASPM when POLICY_DEFAULT is set and I > see above problem gets resolved. Also, the existing ASPM behavior > does not have impact, unless specific BIOS does not disable ASPM on > Root Port when device gets removed. ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-05-03 21:10 ` Bjorn Helgaas 0 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-05-03 21:10 UTC (permalink / raw) To: linux-arm-kernel On Tue, May 02, 2017 at 12:02:53PM +0000, Patel, Mayurkumar wrote: > >On Fri, Apr 21, 2017 at 2:46 AM, Patel, Mayurkumar > ><mayurkumar.patel@intel.com> wrote: > >>>On 4/17/2017 12:38 PM, Bjorn Helgaas wrote: > >>>>> Like you said, what do we do by default is the question. Should we opt > >>>>> for safe like we are doing, or try to save some power. > >>>> I think safety is paramount. Every user should be able to boot safely > >>>> without any kernel parameters. We don't want users to have a problem > >>>> booting and then have to search for a workaround like booting with > >>>> "pcie_aspm=off". Most users will never do that. > >>> > >>>OK, no problem with leaving the behavior as it is. > >>> > >>>My initial approach was #2. We knew this way that user had full control > >>>over the ASPM policy by changing the BIOS option. Then, Mayurkumar > >>>complained that ASPM is not enabled following a hotplug insertion to an > >>>empty slot. That's when I switched to #3 as it sounded like a good thing > >>>to have for us. > >>> > >>>> Here's a long-term strawman proposal, see what you think: > >>>> > >>>> - Deprecate CONFIG_PCIEASPM_DEFAULT, CONFIG_PCIEASPM_POWERSAVE, etc. > >>>> - Default aspm_policy is POLICY_DEFAULT always. > >>>> - POLICY_DEFAULT means Linux doesn't touch anything: if BIOS enabled > >>>> ASPM, we leave it that way; we leave ASPM disabled on hot-added > >>>> devices. > >>> > >> I am also ok with leaving the same behavior as now. But still > >> following is something open I feel besides, Which may be there in > >> your comments redundantly. The current problem is, > >> pcie_aspm_exit_link_state() disables the ASPM configuration even > >> if POLICY_DEFAULT was set. > > > >We call pcie_aspm_exit_link_state() when removing an endpoint. When > >we remove an endpoint, I think disabling ASPM is the right thing to > >do. The spec (PCIe r3.1, sec 5.4.1.3) says "Software must not enable > >L0s in either direction on a given Link unless components on both > >sides of the Link each support L0s; otherwise, the result is > >undefined." > > Yes, you are right and per spec also it makes sense that ASPM needs > to be disabled. But, if POLICY_DEFAULT is set then, shouldn't BIOS > take care of disabling ASPM? No, I don't think so. POLICY_DEFAULT is a Linux thing and BIOS doesn't know anything about it. ASPM can be configured by BIOS before handoff to Linux, but after handoff it should be managed either entirely by BIOS or entirely by Linux. If BIOS wants to retain ASPM control, it would have to tell the OS *not* to use ASPM, and it would have to use ACPI hotplug. In this case, POLICY_DEFAULT is irrelevant because Linux shouldn't do anything with ASPM. But normally BIOS allows Linux to control ASPM, and we would use native PCIe hotplug (pciehp) instead of ACPI hotplug, and BIOS has no opportunity to enable or disable ASPM on hotplug events. > >> I am seeing already following problem(or may be influence) with > >> it. The Endpoint I have does not have does not have "Presence > >> detect change" mechanism. Hot plug is working with Link status > >> events. When link is in L1 or L1SS and if EP is powered off, no > >> Link status change event are triggered (It might be the expected > >> behavior in L1 or L1SS). When next time EP is powered on there > >> are link down and link up events coming one after other. BIOS > >> enables ASPM on Root port and Endpoint, but while processing link > >> status down, pcie_aspm_exit_link_state() clears the ASPM already > >> which were enabled by BIOS. If we want to follow above approach > >> then shall we consider having something similar as following? > > > >The proposal was to leave ASPM disabled on hot-added devices. If > >the endpoint was powered off and powered back on again, I think > >that device looks like a hot-added device, doesn't it? > > Yes, it is hot-added device. Also, I understand, for POLICY_DEFAULT, > OS would/should not touch ASPM(enable/disable), but BIOS could still > (enable/disable), right? Maybe I'm misunderstanding your question. There are two questions here: 1) Does the BIOS allow Linux to manage ASPM? 2) If Linux does manage ASPM, what policy does it use? If BIOS doesn't allow Linux to manage ASPM, POLICY_DEFAULT is irrelevant. If BIOS does allow Linux to manage ASPM, POLICY_DEFAULT means Linux should use the settings made by BIOS. The user could select a different policy, and then Linux would change the ASPM configuration accordingly. > Currently, what happens in my system is as following, (each 2nd > power cycle/hotplug of Endpoint disables ASPM): > > > First Power cycle (When ASPM L1 is already enabled): device gets > powered off -> there are no Link status events, so no pcie hotplug > interrupt and pcie_aspm_exit_link_state() triggered. If the Downstream Port leading to your Endpoint is hotplug capable, doesn't the spec require that it can report link state changes (PCIe r3.1, sec 7.8.6, 7.8.10, 7.8.11)? > When the device gets powered on again -> Link down/Link up events > are coming back to back. First Link down is served. (BIOS checks > for the Link status and enables ASPM already, as the device is > actually powered back). OS calls pcie_aspm_exit_link_state() and > ASPM gets disabled by OS. > > Second Power cycle (When ASPM L1 is disabled after above): device > gets powered off -> there are link status events, pcie hotplug > interrupt is triggered and pcie_aspm_exit_link_state() triggered. > OS disables ASPM. BIOS checks Link status and disables ASPM too. > When the device gets powered on -> BIOS enables ASPM and as this is > pcie hotplug insertion, OS does not interfere and we have ASPM > enabled. I don't understand this sequence. If we're using native PCIe hotplug, BIOS should not be involved to enable ASPM when the device is powered on. > The above sequence happens each 2nd power cycle of the hotplug > device. > > So One could still argue if POLICY_DEFAULT is set, then why OS > disables ASPM if it is not meant to touch configuration. This is > why I proposed following kind of change, so that OS would not touch > ASPM, if POLICY_DEFAULT is set. Also, With the below change, > everything relies on BIOS for ASPM when POLICY_DEFAULT is set and I > see above problem gets resolved. Also, the existing ASPM behavior > does not have impact, unless specific BIOS does not disable ASPM on > Root Port when device gets removed. ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-05-03 21:10 ` Bjorn Helgaas 0 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-05-03 21:10 UTC (permalink / raw) To: Patel, Mayurkumar Cc: Rajat Jain, Myron Stowe, David Daney, linux-pci, Timur Tabi, linux-kernel, Sinan Kaya, Julia Lawall, linux-arm-msm, Bjorn Helgaas, Rajat Jain, Yinghai Lu, Shawn Lin, linux-arm On Tue, May 02, 2017 at 12:02:53PM +0000, Patel, Mayurkumar wrote: > >On Fri, Apr 21, 2017 at 2:46 AM, Patel, Mayurkumar > ><mayurkumar.patel@intel.com> wrote: > >>>On 4/17/2017 12:38 PM, Bjorn Helgaas wrote: > >>>>> Like you said, what do we do by default is the question. Should we opt > >>>>> for safe like we are doing, or try to save some power. > >>>> I think safety is paramount. Every user should be able to boot safely > >>>> without any kernel parameters. We don't want users to have a problem > >>>> booting and then have to search for a workaround like booting with > >>>> "pcie_aspm=off". Most users will never do that. > >>> > >>>OK, no problem with leaving the behavior as it is. > >>> > >>>My initial approach was #2. We knew this way that user had full control > >>>over the ASPM policy by changing the BIOS option. Then, Mayurkumar > >>>complained that ASPM is not enabled following a hotplug insertion to an > >>>empty slot. That's when I switched to #3 as it sounded like a good thing > >>>to have for us. > >>> > >>>> Here's a long-term strawman proposal, see what you think: > >>>> > >>>> - Deprecate CONFIG_PCIEASPM_DEFAULT, CONFIG_PCIEASPM_POWERSAVE, etc. > >>>> - Default aspm_policy is POLICY_DEFAULT always. > >>>> - POLICY_DEFAULT means Linux doesn't touch anything: if BIOS enabled > >>>> ASPM, we leave it that way; we leave ASPM disabled on hot-added > >>>> devices. > >>> > >> I am also ok with leaving the same behavior as now. But still > >> following is something open I feel besides, Which may be there in > >> your comments redundantly. The current problem is, > >> pcie_aspm_exit_link_state() disables the ASPM configuration even > >> if POLICY_DEFAULT was set. > > > >We call pcie_aspm_exit_link_state() when removing an endpoint. When > >we remove an endpoint, I think disabling ASPM is the right thing to > >do. The spec (PCIe r3.1, sec 5.4.1.3) says "Software must not enable > >L0s in either direction on a given Link unless components on both > >sides of the Link each support L0s; otherwise, the result is > >undefined." > > Yes, you are right and per spec also it makes sense that ASPM needs > to be disabled. But, if POLICY_DEFAULT is set then, shouldn't BIOS > take care of disabling ASPM? No, I don't think so. POLICY_DEFAULT is a Linux thing and BIOS doesn't know anything about it. ASPM can be configured by BIOS before handoff to Linux, but after handoff it should be managed either entirely by BIOS or entirely by Linux. If BIOS wants to retain ASPM control, it would have to tell the OS *not* to use ASPM, and it would have to use ACPI hotplug. In this case, POLICY_DEFAULT is irrelevant because Linux shouldn't do anything with ASPM. But normally BIOS allows Linux to control ASPM, and we would use native PCIe hotplug (pciehp) instead of ACPI hotplug, and BIOS has no opportunity to enable or disable ASPM on hotplug events. > >> I am seeing already following problem(or may be influence) with > >> it. The Endpoint I have does not have does not have "Presence > >> detect change" mechanism. Hot plug is working with Link status > >> events. When link is in L1 or L1SS and if EP is powered off, no > >> Link status change event are triggered (It might be the expected > >> behavior in L1 or L1SS). When next time EP is powered on there > >> are link down and link up events coming one after other. BIOS > >> enables ASPM on Root port and Endpoint, but while processing link > >> status down, pcie_aspm_exit_link_state() clears the ASPM already > >> which were enabled by BIOS. If we want to follow above approach > >> then shall we consider having something similar as following? > > > >The proposal was to leave ASPM disabled on hot-added devices. If > >the endpoint was powered off and powered back on again, I think > >that device looks like a hot-added device, doesn't it? > > Yes, it is hot-added device. Also, I understand, for POLICY_DEFAULT, > OS would/should not touch ASPM(enable/disable), but BIOS could still > (enable/disable), right? Maybe I'm misunderstanding your question. There are two questions here: 1) Does the BIOS allow Linux to manage ASPM? 2) If Linux does manage ASPM, what policy does it use? If BIOS doesn't allow Linux to manage ASPM, POLICY_DEFAULT is irrelevant. If BIOS does allow Linux to manage ASPM, POLICY_DEFAULT means Linux should use the settings made by BIOS. The user could select a different policy, and then Linux would change the ASPM configuration accordingly. > Currently, what happens in my system is as following, (each 2nd > power cycle/hotplug of Endpoint disables ASPM): > > > First Power cycle (When ASPM L1 is already enabled): device gets > powered off -> there are no Link status events, so no pcie hotplug > interrupt and pcie_aspm_exit_link_state() triggered. If the Downstream Port leading to your Endpoint is hotplug capable, doesn't the spec require that it can report link state changes (PCIe r3.1, sec 7.8.6, 7.8.10, 7.8.11)? > When the device gets powered on again -> Link down/Link up events > are coming back to back. First Link down is served. (BIOS checks > for the Link status and enables ASPM already, as the device is > actually powered back). OS calls pcie_aspm_exit_link_state() and > ASPM gets disabled by OS. > > Second Power cycle (When ASPM L1 is disabled after above): device > gets powered off -> there are link status events, pcie hotplug > interrupt is triggered and pcie_aspm_exit_link_state() triggered. > OS disables ASPM. BIOS checks Link status and disables ASPM too. > When the device gets powered on -> BIOS enables ASPM and as this is > pcie hotplug insertion, OS does not interfere and we have ASPM > enabled. I don't understand this sequence. If we're using native PCIe hotplug, BIOS should not be involved to enable ASPM when the device is powered on. > The above sequence happens each 2nd power cycle of the hotplug > device. > > So One could still argue if POLICY_DEFAULT is set, then why OS > disables ASPM if it is not meant to touch configuration. This is > why I proposed following kind of change, so that OS would not touch > ASPM, if POLICY_DEFAULT is set. Also, With the below change, > everything relies on BIOS for ASPM when POLICY_DEFAULT is set and I > see above problem gets resolved. Also, the existing ASPM behavior > does not have impact, unless specific BIOS does not disable ASPM on > Root Port when device gets removed. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 94+ messages in thread
* RE: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init 2017-05-03 21:10 ` Bjorn Helgaas (?) (?) @ 2017-05-15 9:10 ` Patel, Mayurkumar -1 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-05-15 9:10 UTC (permalink / raw) To: 'Bjorn Helgaas' Cc: Rajat Jain, Myron Stowe, David Daney, linux-pci, Timur Tabi, linux-kernel, Sinan Kaya, Julia Lawall, linux-arm-msm, Bjorn Helgaas, Rajat Jain, Yinghai Lu, Shawn Lin, linux-arm Hi Bjorn Thanks a lot for your reply and explanations. Sorry for my late reply due to some other emergencies. > >On Tue, May 02, 2017 at 12:02:53PM +0000, Patel, Mayurkumar wrote: >> >On Fri, Apr 21, 2017 at 2:46 AM, Patel, Mayurkumar >> ><mayurkumar.patel@intel.com> wrote: >> >>>On 4/17/2017 12:38 PM, Bjorn Helgaas wrote: >> >>>>> Like you said, what do we do by default is the question. Should we opt >> >>>>> for safe like we are doing, or try to save some power. >> >>>> I think safety is paramount. Every user should be able to boot safely >> >>>> without any kernel parameters. We don't want users to have a problem >> >>>> booting and then have to search for a workaround like booting with >> >>>> "pcie_aspm=off". Most users will never do that. >> >>> >> >>>OK, no problem with leaving the behavior as it is. >> >>> >> >>>My initial approach was #2. We knew this way that user had full control >> >>>over the ASPM policy by changing the BIOS option. Then, Mayurkumar >> >>>complained that ASPM is not enabled following a hotplug insertion to an >> >>>empty slot. That's when I switched to #3 as it sounded like a good thing >> >>>to have for us. >> >>> >> >>>> Here's a long-term strawman proposal, see what you think: >> >>>> >> >>>> - Deprecate CONFIG_PCIEASPM_DEFAULT, CONFIG_PCIEASPM_POWERSAVE, etc. >> >>>> - Default aspm_policy is POLICY_DEFAULT always. >> >>>> - POLICY_DEFAULT means Linux doesn't touch anything: if BIOS enabled >> >>>> ASPM, we leave it that way; we leave ASPM disabled on hot-added >> >>>> devices. >> >>> >> >> I am also ok with leaving the same behavior as now. But still >> >> following is something open I feel besides, Which may be there in >> >> your comments redundantly. The current problem is, >> >> pcie_aspm_exit_link_state() disables the ASPM configuration even >> >> if POLICY_DEFAULT was set. >> > >> >We call pcie_aspm_exit_link_state() when removing an endpoint. When >> >we remove an endpoint, I think disabling ASPM is the right thing to >> >do. The spec (PCIe r3.1, sec 5.4.1.3) says "Software must not enable >> >L0s in either direction on a given Link unless components on both >> >sides of the Link each support L0s; otherwise, the result is >> >undefined." >> >> Yes, you are right and per spec also it makes sense that ASPM needs >> to be disabled. But, if POLICY_DEFAULT is set then, shouldn't BIOS >> take care of disabling ASPM? > >No, I don't think so. POLICY_DEFAULT is a Linux thing and BIOS >doesn't know anything about it. > >ASPM can be configured by BIOS before handoff to Linux, but after >handoff it should be managed either entirely by BIOS or entirely by >Linux. If BIOS wants to retain ASPM control, it would have to tell >the OS *not* to use ASPM, and it would have to use ACPI hotplug. In >this case, POLICY_DEFAULT is irrelevant because Linux shouldn't do >anything with ASPM. > >But normally BIOS allows Linux to control ASPM, and we would use >native PCIe hotplug (pciehp) instead of ACPI hotplug, and BIOS has no >opportunity to enable or disable ASPM on hotplug events. > BIOS that I am having, has an SMI handler Which gets triggered upon Hotplug (Data Link Layer State Changed) Interrupt Which configures ASPM L1/L1SS in BIOS and We are still using Native Hotplug driver. Sounds like BIOS we have in our System, does not inform OS that it wants control ASPM. >> >> I am seeing already following problem(or may be influence) with >> >> it. The Endpoint I have does not have does not have "Presence >> >> detect change" mechanism. Hot plug is working with Link status >> >> events. When link is in L1 or L1SS and if EP is powered off, no >> >> Link status change event are triggered (It might be the expected >> >> behavior in L1 or L1SS). When next time EP is powered on there >> >> are link down and link up events coming one after other. BIOS >> >> enables ASPM on Root port and Endpoint, but while processing link >> >> status down, pcie_aspm_exit_link_state() clears the ASPM already >> >> which were enabled by BIOS. If we want to follow above approach >> >> then shall we consider having something similar as following? >> > >> >The proposal was to leave ASPM disabled on hot-added devices. If >> >the endpoint was powered off and powered back on again, I think >> >that device looks like a hot-added device, doesn't it? >> >> Yes, it is hot-added device. Also, I understand, for POLICY_DEFAULT, >> OS would/should not touch ASPM(enable/disable), but BIOS could still >> (enable/disable), right? > >Maybe I'm misunderstanding your question. There are two questions >here: > > 1) Does the BIOS allow Linux to manage ASPM? > > 2) If Linux does manage ASPM, what policy does it use? > >If BIOS doesn't allow Linux to manage ASPM, POLICY_DEFAULT is >irrelevant. If BIOS does allow Linux to manage ASPM, POLICY_DEFAULT >means Linux should use the settings made by BIOS. The user could >select a different policy, and then Linux would change the ASPM >configuration accordingly. > Ok understood. >> Currently, what happens in my system is as following, (each 2nd >> power cycle/hotplug of Endpoint disables ASPM): >> >> >> First Power cycle (When ASPM L1 is already enabled): device gets >> powered off -> there are no Link status events, so no pcie hotplug >> interrupt and pcie_aspm_exit_link_state() triggered. > >If the Downstream Port leading to your Endpoint is hotplug capable, >doesn't the spec require that it can report link state changes (PCIe >r3.1, sec 7.8.6, 7.8.10, 7.8.11)? > >> When the device gets powered on again -> Link down/Link up events >> are coming back to back. First Link down is served. (BIOS checks >> for the Link status and enables ASPM already, as the device is >> actually powered back). OS calls pcie_aspm_exit_link_state() and >> ASPM gets disabled by OS. >> >> Second Power cycle (When ASPM L1 is disabled after above): device >> gets powered off -> there are link status events, pcie hotplug >> interrupt is triggered and pcie_aspm_exit_link_state() triggered. >> OS disables ASPM. BIOS checks Link status and disables ASPM too. >> When the device gets powered on -> BIOS enables ASPM and as this is >> pcie hotplug insertion, OS does not interfere and we have ASPM >> enabled. > >I don't understand this sequence. If we're using native PCIe hotplug, >BIOS should not be involved to enable ASPM when the device is powered >on. > >> The above sequence happens each 2nd power cycle of the hotplug >> device. >> >> So One could still argue if POLICY_DEFAULT is set, then why OS >> disables ASPM if it is not meant to touch configuration. This is >> why I proposed following kind of change, so that OS would not touch >> ASPM, if POLICY_DEFAULT is set. Also, With the below change, >> everything relies on BIOS for ASPM when POLICY_DEFAULT is set and I >> see above problem gets resolved. Also, the existing ASPM behavior >> does not have impact, unless specific BIOS does not disable ASPM on >> Root Port when device gets removed. Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Christian Lamprechter Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-05-15 9:10 ` Patel, Mayurkumar 0 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-05-15 9:10 UTC (permalink / raw) To: linux-arm-kernel Hi Bjorn Thanks a lot for your reply and explanations. Sorry for my late reply due to some other emergencies. > >On Tue, May 02, 2017 at 12:02:53PM +0000, Patel, Mayurkumar wrote: >> >On Fri, Apr 21, 2017 at 2:46 AM, Patel, Mayurkumar >> ><mayurkumar.patel@intel.com> wrote: >> >>>On 4/17/2017 12:38 PM, Bjorn Helgaas wrote: >> >>>>> Like you said, what do we do by default is the question. Should we opt >> >>>>> for safe like we are doing, or try to save some power. >> >>>> I think safety is paramount. Every user should be able to boot safely >> >>>> without any kernel parameters. We don't want users to have a problem >> >>>> booting and then have to search for a workaround like booting with >> >>>> "pcie_aspm=off". Most users will never do that. >> >>> >> >>>OK, no problem with leaving the behavior as it is. >> >>> >> >>>My initial approach was #2. We knew this way that user had full control >> >>>over the ASPM policy by changing the BIOS option. Then, Mayurkumar >> >>>complained that ASPM is not enabled following a hotplug insertion to an >> >>>empty slot. That's when I switched to #3 as it sounded like a good thing >> >>>to have for us. >> >>> >> >>>> Here's a long-term strawman proposal, see what you think: >> >>>> >> >>>> - Deprecate CONFIG_PCIEASPM_DEFAULT, CONFIG_PCIEASPM_POWERSAVE, etc. >> >>>> - Default aspm_policy is POLICY_DEFAULT always. >> >>>> - POLICY_DEFAULT means Linux doesn't touch anything: if BIOS enabled >> >>>> ASPM, we leave it that way; we leave ASPM disabled on hot-added >> >>>> devices. >> >>> >> >> I am also ok with leaving the same behavior as now. But still >> >> following is something open I feel besides, Which may be there in >> >> your comments redundantly. The current problem is, >> >> pcie_aspm_exit_link_state() disables the ASPM configuration even >> >> if POLICY_DEFAULT was set. >> > >> >We call pcie_aspm_exit_link_state() when removing an endpoint. When >> >we remove an endpoint, I think disabling ASPM is the right thing to >> >do. The spec (PCIe r3.1, sec 5.4.1.3) says "Software must not enable >> >L0s in either direction on a given Link unless components on both >> >sides of the Link each support L0s; otherwise, the result is >> >undefined." >> >> Yes, you are right and per spec also it makes sense that ASPM needs >> to be disabled. But, if POLICY_DEFAULT is set then, shouldn't BIOS >> take care of disabling ASPM? > >No, I don't think so. POLICY_DEFAULT is a Linux thing and BIOS >doesn't know anything about it. > >ASPM can be configured by BIOS before handoff to Linux, but after >handoff it should be managed either entirely by BIOS or entirely by >Linux. If BIOS wants to retain ASPM control, it would have to tell >the OS *not* to use ASPM, and it would have to use ACPI hotplug. In >this case, POLICY_DEFAULT is irrelevant because Linux shouldn't do >anything with ASPM. > >But normally BIOS allows Linux to control ASPM, and we would use >native PCIe hotplug (pciehp) instead of ACPI hotplug, and BIOS has no >opportunity to enable or disable ASPM on hotplug events. > BIOS that I am having, has an SMI handler Which gets triggered upon Hotplug (Data Link Layer State Changed) Interrupt Which configures ASPM L1/L1SS in BIOS and We are still using Native Hotplug driver. Sounds like BIOS we have in our System, does not inform OS that it wants control ASPM. >> >> I am seeing already following problem(or may be influence) with >> >> it. The Endpoint I have does not have does not have "Presence >> >> detect change" mechanism. Hot plug is working with Link status >> >> events. When link is in L1 or L1SS and if EP is powered off, no >> >> Link status change event are triggered (It might be the expected >> >> behavior in L1 or L1SS). When next time EP is powered on there >> >> are link down and link up events coming one after other. BIOS >> >> enables ASPM on Root port and Endpoint, but while processing link >> >> status down, pcie_aspm_exit_link_state() clears the ASPM already >> >> which were enabled by BIOS. If we want to follow above approach >> >> then shall we consider having something similar as following? >> > >> >The proposal was to leave ASPM disabled on hot-added devices. If >> >the endpoint was powered off and powered back on again, I think >> >that device looks like a hot-added device, doesn't it? >> >> Yes, it is hot-added device. Also, I understand, for POLICY_DEFAULT, >> OS would/should not touch ASPM(enable/disable), but BIOS could still >> (enable/disable), right? > >Maybe I'm misunderstanding your question. There are two questions >here: > > 1) Does the BIOS allow Linux to manage ASPM? > > 2) If Linux does manage ASPM, what policy does it use? > >If BIOS doesn't allow Linux to manage ASPM, POLICY_DEFAULT is >irrelevant. If BIOS does allow Linux to manage ASPM, POLICY_DEFAULT >means Linux should use the settings made by BIOS. The user could >select a different policy, and then Linux would change the ASPM >configuration accordingly. > Ok understood. >> Currently, what happens in my system is as following, (each 2nd >> power cycle/hotplug of Endpoint disables ASPM): >> >> >> First Power cycle (When ASPM L1 is already enabled): device gets >> powered off -> there are no Link status events, so no pcie hotplug >> interrupt and pcie_aspm_exit_link_state() triggered. > >If the Downstream Port leading to your Endpoint is hotplug capable, >doesn't the spec require that it can report link state changes (PCIe >r3.1, sec 7.8.6, 7.8.10, 7.8.11)? > >> When the device gets powered on again -> Link down/Link up events >> are coming back to back. First Link down is served. (BIOS checks >> for the Link status and enables ASPM already, as the device is >> actually powered back). OS calls pcie_aspm_exit_link_state() and >> ASPM gets disabled by OS. >> >> Second Power cycle (When ASPM L1 is disabled after above): device >> gets powered off -> there are link status events, pcie hotplug >> interrupt is triggered and pcie_aspm_exit_link_state() triggered. >> OS disables ASPM. BIOS checks Link status and disables ASPM too. >> When the device gets powered on -> BIOS enables ASPM and as this is >> pcie hotplug insertion, OS does not interfere and we have ASPM >> enabled. > >I don't understand this sequence. If we're using native PCIe hotplug, >BIOS should not be involved to enable ASPM when the device is powered >on. > >> The above sequence happens each 2nd power cycle of the hotplug >> device. >> >> So One could still argue if POLICY_DEFAULT is set, then why OS >> disables ASPM if it is not meant to touch configuration. This is >> why I proposed following kind of change, so that OS would not touch >> ASPM, if POLICY_DEFAULT is set. Also, With the below change, >> everything relies on BIOS for ASPM when POLICY_DEFAULT is set and I >> see above problem gets resolved. Also, the existing ASPM behavior >> does not have impact, unless specific BIOS does not disable ASPM on >> Root Port when device gets removed. Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Christian Lamprechter Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 ^ permalink raw reply [flat|nested] 94+ messages in thread
* RE: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-05-15 9:10 ` Patel, Mayurkumar 0 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-05-15 9:10 UTC (permalink / raw) To: 'Bjorn Helgaas' Cc: Bjorn Helgaas, Sinan Kaya, Rajat Jain, Rajat Jain, David Daney, linux-pci, Timur Tabi, linux-kernel, Julia Lawall, linux-arm-msm, Yinghai Lu, Shawn Lin, linux-arm, Myron Stowe Hi Bjorn Thanks a lot for your reply and explanations. Sorry for my late reply due t= o some other emergencies. > >On Tue, May 02, 2017 at 12:02:53PM +0000, Patel, Mayurkumar wrote: >> >On Fri, Apr 21, 2017 at 2:46 AM, Patel, Mayurkumar >> ><mayurkumar.patel@intel.com> wrote: >> >>>On 4/17/2017 12:38 PM, Bjorn Helgaas wrote: >> >>>>> Like you said, what do we do by default is the question. Should we= opt >> >>>>> for safe like we are doing, or try to save some power. >> >>>> I think safety is paramount. Every user should be able to boot saf= ely >> >>>> without any kernel parameters. We don't want users to have a probl= em >> >>>> booting and then have to search for a workaround like booting with >> >>>> "pcie_aspm=3Doff". Most users will never do that. >> >>> >> >>>OK, no problem with leaving the behavior as it is. >> >>> >> >>>My initial approach was #2. We knew this way that user had full contr= ol >> >>>over the ASPM policy by changing the BIOS option. Then, Mayurkumar >> >>>complained that ASPM is not enabled following a hotplug insertion to = an >> >>>empty slot. That's when I switched to #3 as it sounded like a good th= ing >> >>>to have for us. >> >>> >> >>>> Here's a long-term strawman proposal, see what you think: >> >>>> >> >>>> - Deprecate CONFIG_PCIEASPM_DEFAULT, CONFIG_PCIEASPM_POWERSAVE, e= tc. >> >>>> - Default aspm_policy is POLICY_DEFAULT always. >> >>>> - POLICY_DEFAULT means Linux doesn't touch anything: if BIOS enab= led >> >>>> ASPM, we leave it that way; we leave ASPM disabled on hot-added >> >>>> devices. >> >>> >> >> I am also ok with leaving the same behavior as now. But still >> >> following is something open I feel besides, Which may be there in >> >> your comments redundantly. The current problem is, >> >> pcie_aspm_exit_link_state() disables the ASPM configuration even >> >> if POLICY_DEFAULT was set. >> > >> >We call pcie_aspm_exit_link_state() when removing an endpoint. When >> >we remove an endpoint, I think disabling ASPM is the right thing to >> >do. The spec (PCIe r3.1, sec 5.4.1.3) says "Software must not enable >> >L0s in either direction on a given Link unless components on both >> >sides of the Link each support L0s; otherwise, the result is >> >undefined." >> >> Yes, you are right and per spec also it makes sense that ASPM needs >> to be disabled. But, if POLICY_DEFAULT is set then, shouldn't BIOS >> take care of disabling ASPM? > >No, I don't think so. POLICY_DEFAULT is a Linux thing and BIOS >doesn't know anything about it. > >ASPM can be configured by BIOS before handoff to Linux, but after >handoff it should be managed either entirely by BIOS or entirely by >Linux. If BIOS wants to retain ASPM control, it would have to tell >the OS *not* to use ASPM, and it would have to use ACPI hotplug. In >this case, POLICY_DEFAULT is irrelevant because Linux shouldn't do >anything with ASPM. > >But normally BIOS allows Linux to control ASPM, and we would use >native PCIe hotplug (pciehp) instead of ACPI hotplug, and BIOS has no >opportunity to enable or disable ASPM on hotplug events. > BIOS that I am having, has an SMI handler Which gets triggered upon Hotplug (Data Link Layer State Changed) Interrupt Which configures ASPM L1/= L1SS in BIOS and We are still using Native Hotplug driver. Sounds like BIOS we have in o= ur System, does not inform OS that it wants control ASPM. >> >> I am seeing already following problem(or may be influence) with >> >> it. The Endpoint I have does not have does not have "Presence >> >> detect change" mechanism. Hot plug is working with Link status >> >> events. When link is in L1 or L1SS and if EP is powered off, no >> >> Link status change event are triggered (It might be the expected >> >> behavior in L1 or L1SS). When next time EP is powered on there >> >> are link down and link up events coming one after other. BIOS >> >> enables ASPM on Root port and Endpoint, but while processing link >> >> status down, pcie_aspm_exit_link_state() clears the ASPM already >> >> which were enabled by BIOS. If we want to follow above approach >> >> then shall we consider having something similar as following? >> > >> >The proposal was to leave ASPM disabled on hot-added devices. If >> >the endpoint was powered off and powered back on again, I think >> >that device looks like a hot-added device, doesn't it? >> >> Yes, it is hot-added device. Also, I understand, for POLICY_DEFAULT, >> OS would/should not touch ASPM(enable/disable), but BIOS could still >> (enable/disable), right? > >Maybe I'm misunderstanding your question. There are two questions >here: > > 1) Does the BIOS allow Linux to manage ASPM? > > 2) If Linux does manage ASPM, what policy does it use? > >If BIOS doesn't allow Linux to manage ASPM, POLICY_DEFAULT is >irrelevant. If BIOS does allow Linux to manage ASPM, POLICY_DEFAULT >means Linux should use the settings made by BIOS. The user could >select a different policy, and then Linux would change the ASPM >configuration accordingly. > Ok understood. >> Currently, what happens in my system is as following, (each 2nd >> power cycle/hotplug of Endpoint disables ASPM): >> >> >> First Power cycle (When ASPM L1 is already enabled): device gets >> powered off -> there are no Link status events, so no pcie hotplug >> interrupt and pcie_aspm_exit_link_state() triggered. > >If the Downstream Port leading to your Endpoint is hotplug capable, >doesn't the spec require that it can report link state changes (PCIe >r3.1, sec 7.8.6, 7.8.10, 7.8.11)? > >> When the device gets powered on again -> Link down/Link up events >> are coming back to back. First Link down is served. (BIOS checks >> for the Link status and enables ASPM already, as the device is >> actually powered back). OS calls pcie_aspm_exit_link_state() and >> ASPM gets disabled by OS. >> >> Second Power cycle (When ASPM L1 is disabled after above): device >> gets powered off -> there are link status events, pcie hotplug >> interrupt is triggered and pcie_aspm_exit_link_state() triggered. >> OS disables ASPM. BIOS checks Link status and disables ASPM too. >> When the device gets powered on -> BIOS enables ASPM and as this is >> pcie hotplug insertion, OS does not interfere and we have ASPM >> enabled. > >I don't understand this sequence. If we're using native PCIe hotplug, >BIOS should not be involved to enable ASPM when the device is powered >on. > >> The above sequence happens each 2nd power cycle of the hotplug >> device. >> >> So One could still argue if POLICY_DEFAULT is set, then why OS >> disables ASPM if it is not meant to touch configuration. This is >> why I proposed following kind of change, so that OS would not touch >> ASPM, if POLICY_DEFAULT is set. Also, With the below change, >> everything relies on BIOS for ASPM when POLICY_DEFAULT is set and I >> see above problem gets resolved. Also, the existing ASPM behavior >> does not have impact, unless specific BIOS does not disable ASPM on >> Root Port when device gets removed. Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Christian Lamprechter Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 ^ permalink raw reply [flat|nested] 94+ messages in thread
* RE: [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init @ 2017-05-15 9:10 ` Patel, Mayurkumar 0 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-05-15 9:10 UTC (permalink / raw) To: 'Bjorn Helgaas' Cc: Bjorn Helgaas, Sinan Kaya, Rajat Jain, Rajat Jain, David Daney, linux-pci, Timur Tabi, linux-kernel, Julia Lawall, linux-arm-msm, Yinghai Lu, Shawn Lin, linux-arm, Myron Stowe Hi Bjorn Thanks a lot for your reply and explanations. Sorry for my late reply due to some other emergencies. > >On Tue, May 02, 2017 at 12:02:53PM +0000, Patel, Mayurkumar wrote: >> >On Fri, Apr 21, 2017 at 2:46 AM, Patel, Mayurkumar >> ><mayurkumar.patel@intel.com> wrote: >> >>>On 4/17/2017 12:38 PM, Bjorn Helgaas wrote: >> >>>>> Like you said, what do we do by default is the question. Should we opt >> >>>>> for safe like we are doing, or try to save some power. >> >>>> I think safety is paramount. Every user should be able to boot safely >> >>>> without any kernel parameters. We don't want users to have a problem >> >>>> booting and then have to search for a workaround like booting with >> >>>> "pcie_aspm=off". Most users will never do that. >> >>> >> >>>OK, no problem with leaving the behavior as it is. >> >>> >> >>>My initial approach was #2. We knew this way that user had full control >> >>>over the ASPM policy by changing the BIOS option. Then, Mayurkumar >> >>>complained that ASPM is not enabled following a hotplug insertion to an >> >>>empty slot. That's when I switched to #3 as it sounded like a good thing >> >>>to have for us. >> >>> >> >>>> Here's a long-term strawman proposal, see what you think: >> >>>> >> >>>> - Deprecate CONFIG_PCIEASPM_DEFAULT, CONFIG_PCIEASPM_POWERSAVE, etc. >> >>>> - Default aspm_policy is POLICY_DEFAULT always. >> >>>> - POLICY_DEFAULT means Linux doesn't touch anything: if BIOS enabled >> >>>> ASPM, we leave it that way; we leave ASPM disabled on hot-added >> >>>> devices. >> >>> >> >> I am also ok with leaving the same behavior as now. But still >> >> following is something open I feel besides, Which may be there in >> >> your comments redundantly. The current problem is, >> >> pcie_aspm_exit_link_state() disables the ASPM configuration even >> >> if POLICY_DEFAULT was set. >> > >> >We call pcie_aspm_exit_link_state() when removing an endpoint. When >> >we remove an endpoint, I think disabling ASPM is the right thing to >> >do. The spec (PCIe r3.1, sec 5.4.1.3) says "Software must not enable >> >L0s in either direction on a given Link unless components on both >> >sides of the Link each support L0s; otherwise, the result is >> >undefined." >> >> Yes, you are right and per spec also it makes sense that ASPM needs >> to be disabled. But, if POLICY_DEFAULT is set then, shouldn't BIOS >> take care of disabling ASPM? > >No, I don't think so. POLICY_DEFAULT is a Linux thing and BIOS >doesn't know anything about it. > >ASPM can be configured by BIOS before handoff to Linux, but after >handoff it should be managed either entirely by BIOS or entirely by >Linux. If BIOS wants to retain ASPM control, it would have to tell >the OS *not* to use ASPM, and it would have to use ACPI hotplug. In >this case, POLICY_DEFAULT is irrelevant because Linux shouldn't do >anything with ASPM. > >But normally BIOS allows Linux to control ASPM, and we would use >native PCIe hotplug (pciehp) instead of ACPI hotplug, and BIOS has no >opportunity to enable or disable ASPM on hotplug events. > BIOS that I am having, has an SMI handler Which gets triggered upon Hotplug (Data Link Layer State Changed) Interrupt Which configures ASPM L1/L1SS in BIOS and We are still using Native Hotplug driver. Sounds like BIOS we have in our System, does not inform OS that it wants control ASPM. >> >> I am seeing already following problem(or may be influence) with >> >> it. The Endpoint I have does not have does not have "Presence >> >> detect change" mechanism. Hot plug is working with Link status >> >> events. When link is in L1 or L1SS and if EP is powered off, no >> >> Link status change event are triggered (It might be the expected >> >> behavior in L1 or L1SS). When next time EP is powered on there >> >> are link down and link up events coming one after other. BIOS >> >> enables ASPM on Root port and Endpoint, but while processing link >> >> status down, pcie_aspm_exit_link_state() clears the ASPM already >> >> which were enabled by BIOS. If we want to follow above approach >> >> then shall we consider having something similar as following? >> > >> >The proposal was to leave ASPM disabled on hot-added devices. If >> >the endpoint was powered off and powered back on again, I think >> >that device looks like a hot-added device, doesn't it? >> >> Yes, it is hot-added device. Also, I understand, for POLICY_DEFAULT, >> OS would/should not touch ASPM(enable/disable), but BIOS could still >> (enable/disable), right? > >Maybe I'm misunderstanding your question. There are two questions >here: > > 1) Does the BIOS allow Linux to manage ASPM? > > 2) If Linux does manage ASPM, what policy does it use? > >If BIOS doesn't allow Linux to manage ASPM, POLICY_DEFAULT is >irrelevant. If BIOS does allow Linux to manage ASPM, POLICY_DEFAULT >means Linux should use the settings made by BIOS. The user could >select a different policy, and then Linux would change the ASPM >configuration accordingly. > Ok understood. >> Currently, what happens in my system is as following, (each 2nd >> power cycle/hotplug of Endpoint disables ASPM): >> >> >> First Power cycle (When ASPM L1 is already enabled): device gets >> powered off -> there are no Link status events, so no pcie hotplug >> interrupt and pcie_aspm_exit_link_state() triggered. > >If the Downstream Port leading to your Endpoint is hotplug capable, >doesn't the spec require that it can report link state changes (PCIe >r3.1, sec 7.8.6, 7.8.10, 7.8.11)? > >> When the device gets powered on again -> Link down/Link up events >> are coming back to back. First Link down is served. (BIOS checks >> for the Link status and enables ASPM already, as the device is >> actually powered back). OS calls pcie_aspm_exit_link_state() and >> ASPM gets disabled by OS. >> >> Second Power cycle (When ASPM L1 is disabled after above): device >> gets powered off -> there are link status events, pcie hotplug >> interrupt is triggered and pcie_aspm_exit_link_state() triggered. >> OS disables ASPM. BIOS checks Link status and disables ASPM too. >> When the device gets powered on -> BIOS enables ASPM and as this is >> pcie hotplug insertion, OS does not interfere and we have ASPM >> enabled. > >I don't understand this sequence. If we're using native PCIe hotplug, >BIOS should not be involved to enable ASPM when the device is powered >on. > >> The above sequence happens each 2nd power cycle of the hotplug >> device. >> >> So One could still argue if POLICY_DEFAULT is set, then why OS >> disables ASPM if it is not meant to touch configuration. This is >> why I proposed following kind of change, so that OS would not touch >> ASPM, if POLICY_DEFAULT is set. Also, With the below change, >> everything relies on BIOS for ASPM when POLICY_DEFAULT is set and I >> see above problem gets resolved. Also, the existing ASPM behavior >> does not have impact, unless specific BIOS does not disable ASPM on >> Root Port when device gets removed. Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Christian Lamprechter Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 5/5] PCI/ASPM: move link_state cleanup to bridge remove 2017-04-08 4:55 ` Sinan Kaya (?) @ 2017-04-08 4:55 ` Sinan Kaya -1 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-08 4:55 UTC (permalink / raw) To: linux-pci, timur Cc: mayurkumar.patel, linux-arm-msm, linux-arm-kernel, Sinan Kaya, Bjorn Helgaas, Rajat Jain, Yinghai Lu, David Daney, Shawn Lin, Julia Lawall, linux-kernel For endpoints, change pcie_aspm_exit_link_state() so it cleans up the device's own state and disables ASPM if necessary, but doesn't remove the parent's link_state. For bridges, change pcie_aspm_exit_link_state() so it frees the bridge's own link_state. Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 Signed-off-by: Sinan Kaya <okaya@codeaurora.org> --- drivers/pci/pcie/aspm.c | 20 +++++++++++++++----- drivers/pci/remove.c | 3 +-- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index c7da087..d99fa3f 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -972,6 +972,21 @@ void pcie_aspm_exit_link_state(struct pci_dev *pdev) struct pci_dev *parent = pdev->bus->self; struct pcie_link_state *link, *root, *parent_link; + if (pdev->has_secondary_link) { + link = pdev->link_state; + down_read(&pci_bus_sem); + mutex_lock(&aspm_lock); + + list_del(&link->sibling); + list_del(&link->link); + + /* Clock PM is for endpoint device */ + free_link_state(link); + mutex_unlock(&aspm_lock); + up_read(&pci_bus_sem); + return; + } + if (!parent || !parent->link_state) return; @@ -990,11 +1005,6 @@ void pcie_aspm_exit_link_state(struct pci_dev *pdev) /* All functions are removed, so just disable ASPM for the link */ pcie_config_aspm_link(link, 0); - list_del(&link->sibling); - list_del(&link->link); - /* Clock PM is for endpoint device */ - free_link_state(link); - /* Recheck latencies and configure upstream links */ if (parent_link) { pcie_update_aspm_capable(root); diff --git a/drivers/pci/remove.c b/drivers/pci/remove.c index 73a03d3..7e14ebd 100644 --- a/drivers/pci/remove.c +++ b/drivers/pci/remove.c @@ -25,8 +25,7 @@ static void pci_stop_dev(struct pci_dev *dev) dev->is_added = 0; } - if (dev->bus->self) - pcie_aspm_exit_link_state(dev); + pcie_aspm_exit_link_state(dev); } static void pci_destroy_dev(struct pci_dev *dev) -- 1.9.1 ^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH V8 5/5] PCI/ASPM: move link_state cleanup to bridge remove @ 2017-04-08 4:55 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-08 4:55 UTC (permalink / raw) To: linux-arm-kernel For endpoints, change pcie_aspm_exit_link_state() so it cleans up the device's own state and disables ASPM if necessary, but doesn't remove the parent's link_state. For bridges, change pcie_aspm_exit_link_state() so it frees the bridge's own link_state. Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 Signed-off-by: Sinan Kaya <okaya@codeaurora.org> --- drivers/pci/pcie/aspm.c | 20 +++++++++++++++----- drivers/pci/remove.c | 3 +-- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index c7da087..d99fa3f 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -972,6 +972,21 @@ void pcie_aspm_exit_link_state(struct pci_dev *pdev) struct pci_dev *parent = pdev->bus->self; struct pcie_link_state *link, *root, *parent_link; + if (pdev->has_secondary_link) { + link = pdev->link_state; + down_read(&pci_bus_sem); + mutex_lock(&aspm_lock); + + list_del(&link->sibling); + list_del(&link->link); + + /* Clock PM is for endpoint device */ + free_link_state(link); + mutex_unlock(&aspm_lock); + up_read(&pci_bus_sem); + return; + } + if (!parent || !parent->link_state) return; @@ -990,11 +1005,6 @@ void pcie_aspm_exit_link_state(struct pci_dev *pdev) /* All functions are removed, so just disable ASPM for the link */ pcie_config_aspm_link(link, 0); - list_del(&link->sibling); - list_del(&link->link); - /* Clock PM is for endpoint device */ - free_link_state(link); - /* Recheck latencies and configure upstream links */ if (parent_link) { pcie_update_aspm_capable(root); diff --git a/drivers/pci/remove.c b/drivers/pci/remove.c index 73a03d3..7e14ebd 100644 --- a/drivers/pci/remove.c +++ b/drivers/pci/remove.c @@ -25,8 +25,7 @@ static void pci_stop_dev(struct pci_dev *dev) dev->is_added = 0; } - if (dev->bus->self) - pcie_aspm_exit_link_state(dev); + pcie_aspm_exit_link_state(dev); } static void pci_destroy_dev(struct pci_dev *dev) -- 1.9.1 ^ permalink raw reply related [flat|nested] 94+ messages in thread
* [PATCH V8 5/5] PCI/ASPM: move link_state cleanup to bridge remove @ 2017-04-08 4:55 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-08 4:55 UTC (permalink / raw) To: linux-pci, timur Cc: mayurkumar.patel, David Daney, linux-arm-msm, Shawn Lin, linux-kernel, Sinan Kaya, Julia Lawall, Bjorn Helgaas, Rajat Jain, Yinghai Lu, linux-arm-kernel For endpoints, change pcie_aspm_exit_link_state() so it cleans up the device's own state and disables ASPM if necessary, but doesn't remove the parent's link_state. For bridges, change pcie_aspm_exit_link_state() so it frees the bridge's own link_state. Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=194895 Signed-off-by: Sinan Kaya <okaya@codeaurora.org> --- drivers/pci/pcie/aspm.c | 20 +++++++++++++++----- drivers/pci/remove.c | 3 +-- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index c7da087..d99fa3f 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -972,6 +972,21 @@ void pcie_aspm_exit_link_state(struct pci_dev *pdev) struct pci_dev *parent = pdev->bus->self; struct pcie_link_state *link, *root, *parent_link; + if (pdev->has_secondary_link) { + link = pdev->link_state; + down_read(&pci_bus_sem); + mutex_lock(&aspm_lock); + + list_del(&link->sibling); + list_del(&link->link); + + /* Clock PM is for endpoint device */ + free_link_state(link); + mutex_unlock(&aspm_lock); + up_read(&pci_bus_sem); + return; + } + if (!parent || !parent->link_state) return; @@ -990,11 +1005,6 @@ void pcie_aspm_exit_link_state(struct pci_dev *pdev) /* All functions are removed, so just disable ASPM for the link */ pcie_config_aspm_link(link, 0); - list_del(&link->sibling); - list_del(&link->link); - /* Clock PM is for endpoint device */ - free_link_state(link); - /* Recheck latencies and configure upstream links */ if (parent_link) { pcie_update_aspm_capable(root); diff --git a/drivers/pci/remove.c b/drivers/pci/remove.c index 73a03d3..7e14ebd 100644 --- a/drivers/pci/remove.c +++ b/drivers/pci/remove.c @@ -25,8 +25,7 @@ static void pci_stop_dev(struct pci_dev *dev) dev->is_added = 0; } - if (dev->bus->self) - pcie_aspm_exit_link_state(dev); + pcie_aspm_exit_link_state(dev); } static void pci_destroy_dev(struct pci_dev *dev) -- 1.9.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 94+ messages in thread
* RE: [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT 2017-04-08 4:55 ` Sinan Kaya (?) @ 2017-04-10 11:37 ` Patel, Mayurkumar -1 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-04-10 11:37 UTC (permalink / raw) To: Sinan Kaya, linux-pci, timur; +Cc: linux-arm-msm, linux-arm-kernel > >When the operating system is booted with the default ASPM policy >(POLICY_DEFAULT), current code is querying the enable/disable >states from ASPM registers to determine the policy. > >For example, a BIOS could set the power saving state to performance >and clear all ASPM control registers. A balanced ASPM policy could >enable L0s and disable L1. A power conscious BIOS could enable both >L0s and L1 to trade off latency and performance vs. power. > >After hotplug removal, pcie_aspm_exit_link_state() function clears >the ASPM registers. An insertion following hotplug removal reads >incorrect policy as ASPM disabled even though ASPM was enabled >during boot. > >This is caused by the fact that same function is used for reconfiguring >ASPM regardless of the power on state. > >------------------------ >Changes from v7 (https://www.spinics.net/lists/arm-kernel/msg573034.html) >------------------------ >- enable LTSSM as a best effort if the endpoint is not connected during > boot > > >Sinan Kaya (5): > PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() > PCI/ASPM: split pci_aspm_init() into two > PCI/ASPM: add init hook to device_add > PCI/ASPM: save power on values during bridge init > PCI/ASPM: move link_state cleanup to bridge remove > Thanks Sinan for new patches. No issues seen so far in my testing for L1 and L1SS both. May you please add my tested-by in the patches? I will keep on testing it and let you know if something goes wrong for any corner case. > drivers/pci/pcie/aspm.c | 149 +++++++++++++++++++++++++++++++++--------------- > drivers/pci/probe.c | 3 + > drivers/pci/remove.c | 3 +- > include/linux/pci.h | 2 + > 4 files changed, 110 insertions(+), 47 deletions(-) > >-- >1.9.1 Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Christian Lamprechter Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT @ 2017-04-10 11:37 ` Patel, Mayurkumar 0 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-04-10 11:37 UTC (permalink / raw) To: linux-arm-kernel > >When the operating system is booted with the default ASPM policy >(POLICY_DEFAULT), current code is querying the enable/disable >states from ASPM registers to determine the policy. > >For example, a BIOS could set the power saving state to performance >and clear all ASPM control registers. A balanced ASPM policy could >enable L0s and disable L1. A power conscious BIOS could enable both >L0s and L1 to trade off latency and performance vs. power. > >After hotplug removal, pcie_aspm_exit_link_state() function clears >the ASPM registers. An insertion following hotplug removal reads >incorrect policy as ASPM disabled even though ASPM was enabled >during boot. > >This is caused by the fact that same function is used for reconfiguring >ASPM regardless of the power on state. > >------------------------ >Changes from v7 (https://www.spinics.net/lists/arm-kernel/msg573034.html) >------------------------ >- enable LTSSM as a best effort if the endpoint is not connected during > boot > > >Sinan Kaya (5): > PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() > PCI/ASPM: split pci_aspm_init() into two > PCI/ASPM: add init hook to device_add > PCI/ASPM: save power on values during bridge init > PCI/ASPM: move link_state cleanup to bridge remove > Thanks Sinan for new patches. No issues seen so far in my testing for L1 and L1SS both. May you please add my tested-by in the patches? I will keep on testing it and let you know if something goes wrong for any corner case. > drivers/pci/pcie/aspm.c | 149 +++++++++++++++++++++++++++++++++--------------- > drivers/pci/probe.c | 3 + > drivers/pci/remove.c | 3 +- > include/linux/pci.h | 2 + > 4 files changed, 110 insertions(+), 47 deletions(-) > >-- >1.9.1 Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Christian Lamprechter Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 ^ permalink raw reply [flat|nested] 94+ messages in thread
* RE: [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT @ 2017-04-10 11:37 ` Patel, Mayurkumar 0 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-04-10 11:37 UTC (permalink / raw) To: Sinan Kaya, linux-pci, timur; +Cc: linux-arm-msm, linux-arm-kernel > >When the operating system is booted with the default ASPM policy >(POLICY_DEFAULT), current code is querying the enable/disable >states from ASPM registers to determine the policy. > >For example, a BIOS could set the power saving state to performance >and clear all ASPM control registers. A balanced ASPM policy could >enable L0s and disable L1. A power conscious BIOS could enable both >L0s and L1 to trade off latency and performance vs. power. > >After hotplug removal, pcie_aspm_exit_link_state() function clears >the ASPM registers. An insertion following hotplug removal reads >incorrect policy as ASPM disabled even though ASPM was enabled >during boot. > >This is caused by the fact that same function is used for reconfiguring >ASPM regardless of the power on state. > >------------------------ >Changes from v7 (https://www.spinics.net/lists/arm-kernel/msg573034.html) >------------------------ >- enable LTSSM as a best effort if the endpoint is not connected during > boot > > >Sinan Kaya (5): > PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() > PCI/ASPM: split pci_aspm_init() into two > PCI/ASPM: add init hook to device_add > PCI/ASPM: save power on values during bridge init > PCI/ASPM: move link_state cleanup to bridge remove > Thanks Sinan for new patches. No issues seen so far in my testing for L1 and L1SS both. May you please add my tested-by in the patches? I will keep on testing it and let you know if something goes wrong for any corner case. > drivers/pci/pcie/aspm.c | 149 +++++++++++++++++++++++++++++++++--------------- > drivers/pci/probe.c | 3 + > drivers/pci/remove.c | 3 +- > include/linux/pci.h | 2 + > 4 files changed, 110 insertions(+), 47 deletions(-) > >-- >1.9.1 Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Christian Lamprechter Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT 2017-04-10 11:37 ` Patel, Mayurkumar (?) @ 2017-04-10 13:07 ` Sinan Kaya -1 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-10 13:07 UTC (permalink / raw) To: Patel, Mayurkumar, linux-pci, timur; +Cc: linux-arm-msm, linux-arm-kernel On 4/10/2017 7:37 AM, Patel, Mayurkumar wrote: >> PCI/ASPM: save power on values during bridge init >> PCI/ASPM: move link_state cleanup to bridge remove >> > Thanks Sinan for new patches. No issues seen so far in my testing for L1 and L1SS both. > May you please add my tested-by in the patches? > > I will keep on testing it and let you know if something goes wrong for any > corner case. > Thanks for testing and providing your tested-by. -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT @ 2017-04-10 13:07 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-10 13:07 UTC (permalink / raw) To: linux-arm-kernel On 4/10/2017 7:37 AM, Patel, Mayurkumar wrote: >> PCI/ASPM: save power on values during bridge init >> PCI/ASPM: move link_state cleanup to bridge remove >> > Thanks Sinan for new patches. No issues seen so far in my testing for L1 and L1SS both. > May you please add my tested-by in the patches? > > I will keep on testing it and let you know if something goes wrong for any > corner case. > Thanks for testing and providing your tested-by. -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT @ 2017-04-10 13:07 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-10 13:07 UTC (permalink / raw) To: Patel, Mayurkumar, linux-pci, timur; +Cc: linux-arm-msm, linux-arm-kernel On 4/10/2017 7:37 AM, Patel, Mayurkumar wrote: >> PCI/ASPM: save power on values during bridge init >> PCI/ASPM: move link_state cleanup to bridge remove >> > Thanks Sinan for new patches. No issues seen so far in my testing for L1 and L1SS both. > May you please add my tested-by in the patches? > > I will keep on testing it and let you know if something goes wrong for any > corner case. > Thanks for testing and providing your tested-by. -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 94+ messages in thread
* RE: [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT 2017-04-10 13:07 ` Sinan Kaya (?) @ 2017-04-10 13:11 ` Patel, Mayurkumar -1 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-04-10 13:11 UTC (permalink / raw) To: Sinan Kaya, linux-pci, timur; +Cc: linux-arm-msm, linux-arm-kernel > >On 4/10/2017 7:37 AM, Patel, Mayurkumar wrote: >>> PCI/ASPM: save power on values during bridge init >>> PCI/ASPM: move link_state cleanup to bridge remove >>> >> Thanks Sinan for new patches. No issues seen so far in my testing for L1 and L1SS both. >> May you please add my tested-by in the patches? >> >> I will keep on testing it and let you know if something goes wrong for any >> corner case. >> > >Thanks for testing and providing your tested-by. Tested-by: Mayurkumar Patel <mayurkumar.patel@intel.com> > >-- >Sinan Kaya >Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. >Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Christian Lamprechter Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT @ 2017-04-10 13:11 ` Patel, Mayurkumar 0 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-04-10 13:11 UTC (permalink / raw) To: linux-arm-kernel > >On 4/10/2017 7:37 AM, Patel, Mayurkumar wrote: >>> PCI/ASPM: save power on values during bridge init >>> PCI/ASPM: move link_state cleanup to bridge remove >>> >> Thanks Sinan for new patches. No issues seen so far in my testing for L1 and L1SS both. >> May you please add my tested-by in the patches? >> >> I will keep on testing it and let you know if something goes wrong for any >> corner case. >> > >Thanks for testing and providing your tested-by. Tested-by: Mayurkumar Patel <mayurkumar.patel@intel.com> > >-- >Sinan Kaya >Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. >Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Christian Lamprechter Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 ^ permalink raw reply [flat|nested] 94+ messages in thread
* RE: [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT @ 2017-04-10 13:11 ` Patel, Mayurkumar 0 siblings, 0 replies; 94+ messages in thread From: Patel, Mayurkumar @ 2017-04-10 13:11 UTC (permalink / raw) To: Sinan Kaya, linux-pci, timur; +Cc: linux-arm-msm, linux-arm-kernel > >On 4/10/2017 7:37 AM, Patel, Mayurkumar wrote: >>> PCI/ASPM: save power on values during bridge init >>> PCI/ASPM: move link_state cleanup to bridge remove >>> >> Thanks Sinan for new patches. No issues seen so far in my testing for L1 and L1SS both. >> May you please add my tested-by in the patches? >> >> I will keep on testing it and let you know if something goes wrong for any >> corner case. >> > >Thanks for testing and providing your tested-by. Tested-by: Mayurkumar Patel <mayurkumar.patel@intel.com> > >-- >Sinan Kaya >Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. >Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Christian Lamprechter Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT 2017-04-08 4:55 ` Sinan Kaya (?) @ 2017-04-11 21:19 ` Bjorn Helgaas -1 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-11 21:19 UTC (permalink / raw) To: Sinan Kaya Cc: linux-pci, timur, linux-arm-msm, mayurkumar.patel, linux-arm-kernel On Sat, Apr 08, 2017 at 12:55:46AM -0400, Sinan Kaya wrote: > When the operating system is booted with the default ASPM policy > (POLICY_DEFAULT), current code is querying the enable/disable > states from ASPM registers to determine the policy. > > For example, a BIOS could set the power saving state to performance > and clear all ASPM control registers. A balanced ASPM policy could > enable L0s and disable L1. A power conscious BIOS could enable both > L0s and L1 to trade off latency and performance vs. power. > > After hotplug removal, pcie_aspm_exit_link_state() function clears > the ASPM registers. An insertion following hotplug removal reads > incorrect policy as ASPM disabled even though ASPM was enabled > during boot. > > This is caused by the fact that same function is used for reconfiguring > ASPM regardless of the power on state. > > ------------------------ > Changes from v7 (https://www.spinics.net/lists/arm-kernel/msg573034.html) > ------------------------ > - enable LTSSM as a best effort if the endpoint is not connected during > boot > > > Sinan Kaya (5): > PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() > PCI/ASPM: split pci_aspm_init() into two > PCI/ASPM: add init hook to device_add > PCI/ASPM: save power on values during bridge init > PCI/ASPM: move link_state cleanup to bridge remove > > drivers/pci/pcie/aspm.c | 149 +++++++++++++++++++++++++++++++++--------------- > drivers/pci/probe.c | 3 + > drivers/pci/remove.c | 3 +- > include/linux/pci.h | 2 + > 4 files changed, 110 insertions(+), 47 deletions(-) Hi Sinan, What are these patches based on? I normally apply things to my "master" branch, which is typically -rc1 or -rc2. Right now it's v4.11-rc1, and these don't apply cleanly: 04:17:59 ~/linux (master)$ stg branch --create pci/aspm-sk-v8 v4.11-rc1 Checking for changes in the working directory ... done Don't know how to determine parent branch from "v4.11-rc1" Branch "pci/aspm-sk-v8" created 04:18:18 ~/linux (pci/aspm-sk-v8)$ stg import -M --sign m/sk-v8 Checking for changes in the working directory ... done Importing patch "pci-aspm-introduce" ... done Importing patch "pci-aspm-split-pci_aspm_init" ... done Importing patch "pci-aspm-add-init-hook-to" ... error: patch failed: drivers/pci/pcie/aspm.c:798 error: drivers/pci/pcie/aspm.c: patch does not apply stg import: Diff does not apply cleanly ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT @ 2017-04-11 21:19 ` Bjorn Helgaas 0 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-11 21:19 UTC (permalink / raw) To: linux-arm-kernel On Sat, Apr 08, 2017 at 12:55:46AM -0400, Sinan Kaya wrote: > When the operating system is booted with the default ASPM policy > (POLICY_DEFAULT), current code is querying the enable/disable > states from ASPM registers to determine the policy. > > For example, a BIOS could set the power saving state to performance > and clear all ASPM control registers. A balanced ASPM policy could > enable L0s and disable L1. A power conscious BIOS could enable both > L0s and L1 to trade off latency and performance vs. power. > > After hotplug removal, pcie_aspm_exit_link_state() function clears > the ASPM registers. An insertion following hotplug removal reads > incorrect policy as ASPM disabled even though ASPM was enabled > during boot. > > This is caused by the fact that same function is used for reconfiguring > ASPM regardless of the power on state. > > ------------------------ > Changes from v7 (https://www.spinics.net/lists/arm-kernel/msg573034.html) > ------------------------ > - enable LTSSM as a best effort if the endpoint is not connected during > boot > > > Sinan Kaya (5): > PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() > PCI/ASPM: split pci_aspm_init() into two > PCI/ASPM: add init hook to device_add > PCI/ASPM: save power on values during bridge init > PCI/ASPM: move link_state cleanup to bridge remove > > drivers/pci/pcie/aspm.c | 149 +++++++++++++++++++++++++++++++++--------------- > drivers/pci/probe.c | 3 + > drivers/pci/remove.c | 3 +- > include/linux/pci.h | 2 + > 4 files changed, 110 insertions(+), 47 deletions(-) Hi Sinan, What are these patches based on? I normally apply things to my "master" branch, which is typically -rc1 or -rc2. Right now it's v4.11-rc1, and these don't apply cleanly: 04:17:59 ~/linux (master)$ stg branch --create pci/aspm-sk-v8 v4.11-rc1 Checking for changes in the working directory ... done Don't know how to determine parent branch from "v4.11-rc1" Branch "pci/aspm-sk-v8" created 04:18:18 ~/linux (pci/aspm-sk-v8)$ stg import -M --sign m/sk-v8 Checking for changes in the working directory ... done Importing patch "pci-aspm-introduce" ... done Importing patch "pci-aspm-split-pci_aspm_init" ... done Importing patch "pci-aspm-add-init-hook-to" ... error: patch failed: drivers/pci/pcie/aspm.c:798 error: drivers/pci/pcie/aspm.c: patch does not apply stg import: Diff does not apply cleanly ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT @ 2017-04-11 21:19 ` Bjorn Helgaas 0 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-11 21:19 UTC (permalink / raw) To: Sinan Kaya Cc: linux-pci, timur, mayurkumar.patel, linux-arm-kernel, linux-arm-msm On Sat, Apr 08, 2017 at 12:55:46AM -0400, Sinan Kaya wrote: > When the operating system is booted with the default ASPM policy > (POLICY_DEFAULT), current code is querying the enable/disable > states from ASPM registers to determine the policy. > > For example, a BIOS could set the power saving state to performance > and clear all ASPM control registers. A balanced ASPM policy could > enable L0s and disable L1. A power conscious BIOS could enable both > L0s and L1 to trade off latency and performance vs. power. > > After hotplug removal, pcie_aspm_exit_link_state() function clears > the ASPM registers. An insertion following hotplug removal reads > incorrect policy as ASPM disabled even though ASPM was enabled > during boot. > > This is caused by the fact that same function is used for reconfiguring > ASPM regardless of the power on state. > > ------------------------ > Changes from v7 (https://www.spinics.net/lists/arm-kernel/msg573034.html) > ------------------------ > - enable LTSSM as a best effort if the endpoint is not connected during > boot > > > Sinan Kaya (5): > PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() > PCI/ASPM: split pci_aspm_init() into two > PCI/ASPM: add init hook to device_add > PCI/ASPM: save power on values during bridge init > PCI/ASPM: move link_state cleanup to bridge remove > > drivers/pci/pcie/aspm.c | 149 +++++++++++++++++++++++++++++++++--------------- > drivers/pci/probe.c | 3 + > drivers/pci/remove.c | 3 +- > include/linux/pci.h | 2 + > 4 files changed, 110 insertions(+), 47 deletions(-) Hi Sinan, What are these patches based on? I normally apply things to my "master" branch, which is typically -rc1 or -rc2. Right now it's v4.11-rc1, and these don't apply cleanly: 04:17:59 ~/linux (master)$ stg branch --create pci/aspm-sk-v8 v4.11-rc1 Checking for changes in the working directory ... done Don't know how to determine parent branch from "v4.11-rc1" Branch "pci/aspm-sk-v8" created 04:18:18 ~/linux (pci/aspm-sk-v8)$ stg import -M --sign m/sk-v8 Checking for changes in the working directory ... done Importing patch "pci-aspm-introduce" ... done Importing patch "pci-aspm-split-pci_aspm_init" ... done Importing patch "pci-aspm-add-init-hook-to" ... error: patch failed: drivers/pci/pcie/aspm.c:798 error: drivers/pci/pcie/aspm.c: patch does not apply stg import: Diff does not apply cleanly _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT 2017-04-11 21:19 ` Bjorn Helgaas @ 2017-04-11 21:27 ` Sinan Kaya -1 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-11 21:27 UTC (permalink / raw) To: Bjorn Helgaas Cc: linux-pci, timur, linux-arm-msm, mayurkumar.patel, linux-arm-kernel On 4/11/2017 5:19 PM, Bjorn Helgaas wrote: > Hi Sinan, > > What are these patches based on? I normally apply things to my "master" > branch, which is typically -rc1 or -rc2. Right now it's v4.11-rc1, and > these don't apply cleanly: > > 04:17:59 ~/linux (master)$ stg branch --create pci/aspm-sk-v8 v4.11-rc1 > Checking for changes in the working directory ... done > Don't know how to determine parent branch from "v4.11-rc1" > Branch "pci/aspm-sk-v8" created > 04:18:18 ~/linux (pci/aspm-sk-v8)$ stg import -M --sign m/sk-v8 > Checking for changes in the working directory ... done > Importing patch "pci-aspm-introduce" ... done > Importing patch "pci-aspm-split-pci_aspm_init" ... done > Importing patch "pci-aspm-add-init-hook-to" ... error: patch failed: drivers/pci/pcie/aspm.c:798 > error: drivers/pci/pcie/aspm.c: patch does not apply > stg import: Diff does not apply cleanly This is what I have. 41a7175 PCI/ASPM: move link_state cleanup to bridge remove 3b646e2 PCI/ASPM: save power on values during bridge init 181510d PCI/ASPM: add init hook to device_add 282d008 PCI/ASPM: split pci_aspm_init() into two 8e89fc2 PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() 97da385 Linux 4.11-rc3 -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT @ 2017-04-11 21:27 ` Sinan Kaya 0 siblings, 0 replies; 94+ messages in thread From: Sinan Kaya @ 2017-04-11 21:27 UTC (permalink / raw) To: linux-arm-kernel On 4/11/2017 5:19 PM, Bjorn Helgaas wrote: > Hi Sinan, > > What are these patches based on? I normally apply things to my "master" > branch, which is typically -rc1 or -rc2. Right now it's v4.11-rc1, and > these don't apply cleanly: > > 04:17:59 ~/linux (master)$ stg branch --create pci/aspm-sk-v8 v4.11-rc1 > Checking for changes in the working directory ... done > Don't know how to determine parent branch from "v4.11-rc1" > Branch "pci/aspm-sk-v8" created > 04:18:18 ~/linux (pci/aspm-sk-v8)$ stg import -M --sign m/sk-v8 > Checking for changes in the working directory ... done > Importing patch "pci-aspm-introduce" ... done > Importing patch "pci-aspm-split-pci_aspm_init" ... done > Importing patch "pci-aspm-add-init-hook-to" ... error: patch failed: drivers/pci/pcie/aspm.c:798 > error: drivers/pci/pcie/aspm.c: patch does not apply > stg import: Diff does not apply cleanly This is what I have. 41a7175 PCI/ASPM: move link_state cleanup to bridge remove 3b646e2 PCI/ASPM: save power on values during bridge init 181510d PCI/ASPM: add init hook to device_add 282d008 PCI/ASPM: split pci_aspm_init() into two 8e89fc2 PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() 97da385 Linux 4.11-rc3 -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT 2017-04-11 21:27 ` Sinan Kaya (?) @ 2017-04-11 22:41 ` Bjorn Helgaas -1 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-11 22:41 UTC (permalink / raw) To: Sinan Kaya Cc: linux-pci, timur, linux-arm-msm, mayurkumar.patel, linux-arm-kernel On Tue, Apr 11, 2017 at 05:27:59PM -0400, Sinan Kaya wrote: > On 4/11/2017 5:19 PM, Bjorn Helgaas wrote: > > Hi Sinan, > > > > What are these patches based on? I normally apply things to my "master" > > branch, which is typically -rc1 or -rc2. Right now it's v4.11-rc1, and > > these don't apply cleanly: > > > > 04:17:59 ~/linux (master)$ stg branch --create pci/aspm-sk-v8 v4.11-rc1 > > Checking for changes in the working directory ... done > > Don't know how to determine parent branch from "v4.11-rc1" > > Branch "pci/aspm-sk-v8" created > > 04:18:18 ~/linux (pci/aspm-sk-v8)$ stg import -M --sign m/sk-v8 > > Checking for changes in the working directory ... done > > Importing patch "pci-aspm-introduce" ... done > > Importing patch "pci-aspm-split-pci_aspm_init" ... done > > Importing patch "pci-aspm-add-init-hook-to" ... error: patch failed: drivers/pci/pcie/aspm.c:798 > > error: drivers/pci/pcie/aspm.c: patch does not apply > > stg import: Diff does not apply cleanly > > This is what I have. > > 41a7175 PCI/ASPM: move link_state cleanup to bridge remove > 3b646e2 PCI/ASPM: save power on values during bridge init > 181510d PCI/ASPM: add init hook to device_add > 282d008 PCI/ASPM: split pci_aspm_init() into two > 8e89fc2 PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() > 97da385 Linux 4.11-rc3 Thanks, I guess I should have figured that out since my for-linus branch includes an ASPM patch. ^ permalink raw reply [flat|nested] 94+ messages in thread
* [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT @ 2017-04-11 22:41 ` Bjorn Helgaas 0 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-11 22:41 UTC (permalink / raw) To: linux-arm-kernel On Tue, Apr 11, 2017 at 05:27:59PM -0400, Sinan Kaya wrote: > On 4/11/2017 5:19 PM, Bjorn Helgaas wrote: > > Hi Sinan, > > > > What are these patches based on? I normally apply things to my "master" > > branch, which is typically -rc1 or -rc2. Right now it's v4.11-rc1, and > > these don't apply cleanly: > > > > 04:17:59 ~/linux (master)$ stg branch --create pci/aspm-sk-v8 v4.11-rc1 > > Checking for changes in the working directory ... done > > Don't know how to determine parent branch from "v4.11-rc1" > > Branch "pci/aspm-sk-v8" created > > 04:18:18 ~/linux (pci/aspm-sk-v8)$ stg import -M --sign m/sk-v8 > > Checking for changes in the working directory ... done > > Importing patch "pci-aspm-introduce" ... done > > Importing patch "pci-aspm-split-pci_aspm_init" ... done > > Importing patch "pci-aspm-add-init-hook-to" ... error: patch failed: drivers/pci/pcie/aspm.c:798 > > error: drivers/pci/pcie/aspm.c: patch does not apply > > stg import: Diff does not apply cleanly > > This is what I have. > > 41a7175 PCI/ASPM: move link_state cleanup to bridge remove > 3b646e2 PCI/ASPM: save power on values during bridge init > 181510d PCI/ASPM: add init hook to device_add > 282d008 PCI/ASPM: split pci_aspm_init() into two > 8e89fc2 PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() > 97da385 Linux 4.11-rc3 Thanks, I guess I should have figured that out since my for-linus branch includes an ASPM patch. ^ permalink raw reply [flat|nested] 94+ messages in thread
* Re: [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT @ 2017-04-11 22:41 ` Bjorn Helgaas 0 siblings, 0 replies; 94+ messages in thread From: Bjorn Helgaas @ 2017-04-11 22:41 UTC (permalink / raw) To: Sinan Kaya Cc: linux-pci, timur, mayurkumar.patel, linux-arm-kernel, linux-arm-msm On Tue, Apr 11, 2017 at 05:27:59PM -0400, Sinan Kaya wrote: > On 4/11/2017 5:19 PM, Bjorn Helgaas wrote: > > Hi Sinan, > > > > What are these patches based on? I normally apply things to my "master" > > branch, which is typically -rc1 or -rc2. Right now it's v4.11-rc1, and > > these don't apply cleanly: > > > > 04:17:59 ~/linux (master)$ stg branch --create pci/aspm-sk-v8 v4.11-rc1 > > Checking for changes in the working directory ... done > > Don't know how to determine parent branch from "v4.11-rc1" > > Branch "pci/aspm-sk-v8" created > > 04:18:18 ~/linux (pci/aspm-sk-v8)$ stg import -M --sign m/sk-v8 > > Checking for changes in the working directory ... done > > Importing patch "pci-aspm-introduce" ... done > > Importing patch "pci-aspm-split-pci_aspm_init" ... done > > Importing patch "pci-aspm-add-init-hook-to" ... error: patch failed: drivers/pci/pcie/aspm.c:798 > > error: drivers/pci/pcie/aspm.c: patch does not apply > > stg import: Diff does not apply cleanly > > This is what I have. > > 41a7175 PCI/ASPM: move link_state cleanup to bridge remove > 3b646e2 PCI/ASPM: save power on values during bridge init > 181510d PCI/ASPM: add init hook to device_add > 282d008 PCI/ASPM: split pci_aspm_init() into two > 8e89fc2 PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() > 97da385 Linux 4.11-rc3 Thanks, I guess I should have figured that out since my for-linus branch includes an ASPM patch. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 94+ messages in thread
end of thread, other threads:[~2017-05-15 9:10 UTC | newest] Thread overview: 94+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-04-08 4:55 [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT Sinan Kaya 2017-04-08 4:55 ` Sinan Kaya 2017-04-08 4:55 ` Sinan Kaya 2017-04-08 4:55 ` [PATCH V8 1/5] PCI/ASPM: introduce pci_aspm_init() and add to pci_init_capabilities() Sinan Kaya 2017-04-08 4:55 ` Sinan Kaya 2017-04-08 4:55 ` Sinan Kaya 2017-04-13 20:51 ` Bjorn Helgaas 2017-04-13 20:51 ` Bjorn Helgaas 2017-04-14 19:10 ` Sinan Kaya 2017-04-14 19:10 ` Sinan Kaya 2017-04-08 4:55 ` [PATCH V8 2/5] PCI/ASPM: split pci_aspm_init() into two Sinan Kaya 2017-04-08 4:55 ` Sinan Kaya 2017-04-08 4:55 ` Sinan Kaya 2017-04-12 19:16 ` Rajat Jain 2017-04-12 19:16 ` Rajat Jain 2017-04-13 18:25 ` Bjorn Helgaas 2017-04-13 18:25 ` Bjorn Helgaas 2017-04-13 18:25 ` Bjorn Helgaas 2017-04-14 19:10 ` Sinan Kaya 2017-04-14 19:10 ` Sinan Kaya 2017-04-14 19:10 ` Sinan Kaya 2017-04-08 4:55 ` [PATCH V8 3/5] PCI/ASPM: add init hook to device_add Sinan Kaya 2017-04-08 4:55 ` Sinan Kaya 2017-04-08 4:55 ` Sinan Kaya 2017-04-13 20:48 ` Bjorn Helgaas 2017-04-13 20:48 ` Bjorn Helgaas 2017-04-13 20:48 ` Bjorn Helgaas 2017-04-13 21:02 ` Bjorn Helgaas 2017-04-13 21:02 ` Bjorn Helgaas 2017-04-13 21:02 ` Bjorn Helgaas 2017-04-14 1:19 ` Sinan Kaya 2017-04-14 1:19 ` Sinan Kaya 2017-04-14 1:30 ` Bjorn Helgaas 2017-04-14 1:30 ` Bjorn Helgaas 2017-04-08 4:55 ` [PATCH V8 4/5] PCI/ASPM: save power on values during bridge init Sinan Kaya 2017-04-08 4:55 ` Sinan Kaya 2017-04-08 4:55 ` Sinan Kaya 2017-04-12 19:19 ` Rajat Jain 2017-04-12 19:19 ` Rajat Jain 2017-04-12 19:19 ` Rajat Jain 2017-04-14 19:12 ` Sinan Kaya 2017-04-14 19:12 ` Sinan Kaya 2017-04-14 19:12 ` Sinan Kaya 2017-04-14 21:44 ` Bjorn Helgaas 2017-04-14 21:44 ` Bjorn Helgaas 2017-04-14 21:44 ` Bjorn Helgaas 2017-04-14 22:17 ` Sinan Kaya 2017-04-14 22:17 ` Sinan Kaya 2017-04-17 16:38 ` Bjorn Helgaas 2017-04-17 16:38 ` Bjorn Helgaas 2017-04-17 17:50 ` Sinan Kaya 2017-04-17 17:50 ` Sinan Kaya 2017-04-21 7:46 ` Patel, Mayurkumar 2017-04-21 7:46 ` Patel, Mayurkumar 2017-04-21 7:46 ` Patel, Mayurkumar 2017-04-21 7:46 ` Patel, Mayurkumar 2017-04-21 13:50 ` Sinan Kaya 2017-04-21 13:50 ` Sinan Kaya 2017-04-21 14:13 ` Patel, Mayurkumar 2017-04-21 14:13 ` Patel, Mayurkumar 2017-04-21 14:13 ` Patel, Mayurkumar 2017-04-21 14:13 ` Patel, Mayurkumar 2017-04-25 18:45 ` Bjorn Helgaas 2017-04-25 18:45 ` Bjorn Helgaas 2017-05-02 12:02 ` Patel, Mayurkumar 2017-05-02 12:02 ` Patel, Mayurkumar 2017-05-02 12:02 ` Patel, Mayurkumar 2017-05-03 21:10 ` Bjorn Helgaas 2017-05-03 21:10 ` Bjorn Helgaas 2017-05-03 21:10 ` Bjorn Helgaas 2017-05-15 9:10 ` Patel, Mayurkumar 2017-05-15 9:10 ` Patel, Mayurkumar 2017-05-15 9:10 ` Patel, Mayurkumar 2017-05-15 9:10 ` Patel, Mayurkumar 2017-04-08 4:55 ` [PATCH V8 5/5] PCI/ASPM: move link_state cleanup to bridge remove Sinan Kaya 2017-04-08 4:55 ` Sinan Kaya 2017-04-08 4:55 ` Sinan Kaya 2017-04-10 11:37 ` [PATCH V8 0/5] PCI/ASPM: reconfigure ASPM following hotplug for POLICY_DEFAULT Patel, Mayurkumar 2017-04-10 11:37 ` Patel, Mayurkumar 2017-04-10 11:37 ` Patel, Mayurkumar 2017-04-10 13:07 ` Sinan Kaya 2017-04-10 13:07 ` Sinan Kaya 2017-04-10 13:07 ` Sinan Kaya 2017-04-10 13:11 ` Patel, Mayurkumar 2017-04-10 13:11 ` Patel, Mayurkumar 2017-04-10 13:11 ` Patel, Mayurkumar 2017-04-11 21:19 ` Bjorn Helgaas 2017-04-11 21:19 ` Bjorn Helgaas 2017-04-11 21:19 ` Bjorn Helgaas 2017-04-11 21:27 ` Sinan Kaya 2017-04-11 21:27 ` Sinan Kaya 2017-04-11 22:41 ` Bjorn Helgaas 2017-04-11 22:41 ` Bjorn Helgaas 2017-04-11 22:41 ` Bjorn Helgaas
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.