* [RFC PATCH 1/2] powerpc/powernv: Add support for CXL mode switch that need PHB reset
2019-01-25 5:11 [RFC PATCH 0/2] cxl: Add support for disabling CAPP when unloading CXL Vaibhav Jain
@ 2019-01-25 5:11 ` Vaibhav Jain
2019-01-28 13:34 ` Frederic Barrat
2019-01-28 16:06 ` christophe lombard
2019-01-25 5:11 ` [RFC PATCH 2/2] cxl: Force a CAPP reset when unloading CXL module Vaibhav Jain
2019-01-28 13:34 ` [RFC PATCH 0/2] cxl: Add support for disabling CAPP when unloading CXL Frederic Barrat
2 siblings, 2 replies; 9+ messages in thread
From: Vaibhav Jain @ 2019-01-25 5:11 UTC (permalink / raw)
To: linuxppc-dev, Frederic Barrat
Cc: Philippe Bergheaud, Vaibhav Jain, Alastair D'Silva,
Christophe Lombard, Andrew Donnellan
Recent updates to OPAL [1] have provided support for new CXL modes on
PHB that need to force a cold reset on the bridge (CRESET). However
PHB CRESET is a multi step process and cannot be completed
synchronously as expected by current kernel implementation that issues
opal call opal_pci_set_phb_cxl_mode().
Hence this patch updates pnv_phb_to_cxl_mode() to implement a polling
loop that handles specific error codes (OPAL_BUSY) returned from
opal_pci_set_phb_cxl_mode() and drive the OPAL pci-state machine, if the
requested CXL mode needs a CRESET.
The patch also updates pnv_phb_to_cxl_mode() to convert and return
OPAL error codes into kernel error codes. This removes a previous
issue where callers to this function would have to include
'opal-api.h' to check for specific OPAL error codes.
References:
[1]: https://lists.ozlabs.org/pipermail/skiboot/2019-January/013063.html
Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com>
---
arch/powerpc/platforms/powernv/pci-cxl.c | 71 +++++++++++++++++++++---
1 file changed, 63 insertions(+), 8 deletions(-)
diff --git a/arch/powerpc/platforms/powernv/pci-cxl.c b/arch/powerpc/platforms/powernv/pci-cxl.c
index 1b18111453d7..d33d662c6212 100644
--- a/arch/powerpc/platforms/powernv/pci-cxl.c
+++ b/arch/powerpc/platforms/powernv/pci-cxl.c
@@ -10,6 +10,7 @@
#include <linux/module.h>
#include <asm/pnv-pci.h>
#include <asm/opal.h>
+#include <linux/delay.h>
#include "pci.h"
@@ -18,21 +19,75 @@ int pnv_phb_to_cxl_mode(struct pci_dev *dev, uint64_t mode)
struct pci_controller *hose = pci_bus_to_host(dev->bus);
struct pnv_phb *phb = hose->private_data;
struct pnv_ioda_pe *pe;
+ unsigned long starttime, endtime;
int rc;
pe = pnv_ioda_get_pe(dev);
if (!pe)
- return -ENODEV;
+ return -ENOENT;
+
+ pe_info(pe, "Switching PHB to CXL mode=%d\n", mode);
+
+ /*
+ * Use a 15 second timeout for mode switch. Value arrived after
+ * limited testing and may need more tweaking.
+ */
+ starttime = jiffies;
+ endtime = starttime + HZ * 15;
+
+ do {
+ rc = opal_pci_set_phb_cxl_mode(phb->opal_id, mode,
+ pe->pe_number);
+
+ /* Wait until mode transistion done */
+ if (rc != OPAL_BUSY && rc != OPAL_BUSY_EVENT)
+ break;
+
+ /* Check if we timedout */
+ if (time_after(jiffies, endtime)) {
+ rc = OPAL_TIMEOUT;
+ break;
+ }
- pe_info(pe, "Switching PHB to CXL\n");
+ /* Opal Busy with mode switch. Run pci state-machine */
+ rc = opal_pci_poll(phb->opal_id);
+ if (rc >= 0) {
+ /* wait for some time */
+ if (rc > 0)
+ msleep(rc);
+ opal_poll_events(NULL);
+ rc = OPAL_BUSY;
+ /* Continue with the mode switch */
+ }
+ } while (rc == OPAL_BUSY || rc == OPAL_BUSY_EVENT);
+
+ pe_level_printk(pe, KERN_DEBUG, "CXL mode switch finished in %u-msecs.",
+ jiffies_to_msecs(jiffies - starttime));
- rc = opal_pci_set_phb_cxl_mode(phb->opal_id, mode, pe->pe_number);
- if (rc == OPAL_UNSUPPORTED)
- dev_err(&dev->dev, "Required cxl mode not supported by firmware - update skiboot\n");
- else if (rc)
- dev_err(&dev->dev, "opal_pci_set_phb_cxl_mode failed: %i\n", rc);
+ /* Check OPAL errors and convert them to kernel error codes */
+ switch (rc) {
+ case OPAL_SUCCESS:
+ return 0;
- return rc;
+ case OPAL_PARAMETER:
+ dev_err(&dev->dev, "CXL not supported on this PHB\n");
+ return -ENOENT;
+
+ case OPAL_UNSUPPORTED:
+ dev_err(&dev->dev,
+ "Required cxl mode not supported by firmware"
+ " - update skiboot\n");
+ return -ENODEV;
+
+ case OPAL_TIMEOUT:
+ dev_err(&dev->dev, "opal_pci_set_phb_cxl_mode Timedout\n");
+ return -ETIME;
+
+ default:
+ dev_err(&dev->dev,
+ "opal_pci_set_phb_cxl_mode failed: %i\n", rc);
+ return -EIO;
+ };
}
EXPORT_SYMBOL(pnv_phb_to_cxl_mode);
--
2.20.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [RFC PATCH 1/2] powerpc/powernv: Add support for CXL mode switch that need PHB reset
2019-01-25 5:11 ` [RFC PATCH 1/2] powerpc/powernv: Add support for CXL mode switch that need PHB reset Vaibhav Jain
@ 2019-01-28 13:34 ` Frederic Barrat
2019-01-29 7:00 ` Vaibhav Jain
2019-01-28 16:06 ` christophe lombard
1 sibling, 1 reply; 9+ messages in thread
From: Frederic Barrat @ 2019-01-28 13:34 UTC (permalink / raw)
To: Vaibhav Jain, linuxppc-dev
Cc: Philippe Bergheaud, Alastair D'Silva, Christophe Lombard,
Andrew Donnellan
Le 25/01/2019 à 06:11, Vaibhav Jain a écrit :
> Recent updates to OPAL [1] have provided support for new CXL modes on
> PHB that need to force a cold reset on the bridge (CRESET). However
> PHB CRESET is a multi step process and cannot be completed
> synchronously as expected by current kernel implementation that issues
> opal call opal_pci_set_phb_cxl_mode().
>
> Hence this patch updates pnv_phb_to_cxl_mode() to implement a polling
> loop that handles specific error codes (OPAL_BUSY) returned from
> opal_pci_set_phb_cxl_mode() and drive the OPAL pci-state machine, if the
> requested CXL mode needs a CRESET.
>
> The patch also updates pnv_phb_to_cxl_mode() to convert and return
> OPAL error codes into kernel error codes. This removes a previous
> issue where callers to this function would have to include
> 'opal-api.h' to check for specific OPAL error codes.
>
> References:
> [1]: https://lists.ozlabs.org/pipermail/skiboot/2019-January/013063.html
>
> Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com>
> ---
> arch/powerpc/platforms/powernv/pci-cxl.c | 71 +++++++++++++++++++++---
> 1 file changed, 63 insertions(+), 8 deletions(-)
>
> diff --git a/arch/powerpc/platforms/powernv/pci-cxl.c b/arch/powerpc/platforms/powernv/pci-cxl.c
> index 1b18111453d7..d33d662c6212 100644
> --- a/arch/powerpc/platforms/powernv/pci-cxl.c
> +++ b/arch/powerpc/platforms/powernv/pci-cxl.c
> @@ -10,6 +10,7 @@
> #include <linux/module.h>
> #include <asm/pnv-pci.h>
> #include <asm/opal.h>
> +#include <linux/delay.h>
>
> #include "pci.h"
>
> @@ -18,21 +19,75 @@ int pnv_phb_to_cxl_mode(struct pci_dev *dev, uint64_t mode)
> struct pci_controller *hose = pci_bus_to_host(dev->bus);
> struct pnv_phb *phb = hose->private_data;
> struct pnv_ioda_pe *pe;
> + unsigned long starttime, endtime;
> int rc;
>
> pe = pnv_ioda_get_pe(dev);
> if (!pe)
> - return -ENODEV;
> + return -ENOENT;
> +
> + pe_info(pe, "Switching PHB to CXL mode=%d\n", mode);
> +
> + /*
> + * Use a 15 second timeout for mode switch. Value arrived after
> + * limited testing and may need more tweaking.
> + */
> + starttime = jiffies;
> + endtime = starttime + HZ * 15;
> +
> + do {
> + rc = opal_pci_set_phb_cxl_mode(phb->opal_id, mode,
> + pe->pe_number);
> +
> + /* Wait until mode transistion done */
> + if (rc != OPAL_BUSY && rc != OPAL_BUSY_EVENT)
> + break;
> +
> + /* Check if we timedout */
> + if (time_after(jiffies, endtime)) {
> + rc = OPAL_TIMEOUT;
> + break;
> + }
>
> - pe_info(pe, "Switching PHB to CXL\n");
> + /* Opal Busy with mode switch. Run pci state-machine */
> + rc = opal_pci_poll(phb->opal_id);
> + if (rc >= 0) {
> + /* wait for some time */
> + if (rc > 0)
> + msleep(rc);
> + opal_poll_events(NULL);
Why is a call to opal_poll_events() needed?
Fred
> + rc = OPAL_BUSY;
> + /* Continue with the mode switch */
> + }
> + } while (rc == OPAL_BUSY || rc == OPAL_BUSY_EVENT);
> +
> + pe_level_printk(pe, KERN_DEBUG, "CXL mode switch finished in %u-msecs.",
> + jiffies_to_msecs(jiffies - starttime));
>
> - rc = opal_pci_set_phb_cxl_mode(phb->opal_id, mode, pe->pe_number);
> - if (rc == OPAL_UNSUPPORTED)
> - dev_err(&dev->dev, "Required cxl mode not supported by firmware - update skiboot\n");
> - else if (rc)
> - dev_err(&dev->dev, "opal_pci_set_phb_cxl_mode failed: %i\n", rc);
> + /* Check OPAL errors and convert them to kernel error codes */
> + switch (rc) {
> + case OPAL_SUCCESS:
> + return 0;
>
> - return rc;
> + case OPAL_PARAMETER:
> + dev_err(&dev->dev, "CXL not supported on this PHB\n");
> + return -ENOENT;
> +
> + case OPAL_UNSUPPORTED:
> + dev_err(&dev->dev,
> + "Required cxl mode not supported by firmware"
> + " - update skiboot\n");
> + return -ENODEV;
> +
> + case OPAL_TIMEOUT:
> + dev_err(&dev->dev, "opal_pci_set_phb_cxl_mode Timedout\n");
> + return -ETIME;
> +
> + default:
> + dev_err(&dev->dev,
> + "opal_pci_set_phb_cxl_mode failed: %i\n", rc);
> + return -EIO;
> + };
> }
> EXPORT_SYMBOL(pnv_phb_to_cxl_mode);
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RFC PATCH 1/2] powerpc/powernv: Add support for CXL mode switch that need PHB reset
2019-01-28 13:34 ` Frederic Barrat
@ 2019-01-29 7:00 ` Vaibhav Jain
0 siblings, 0 replies; 9+ messages in thread
From: Vaibhav Jain @ 2019-01-29 7:00 UTC (permalink / raw)
To: Frederic Barrat, linuxppc-dev
Cc: Philippe Bergheaud, Alastair D'Silva, Christophe Lombard,
Andrew Donnellan
Frederic Barrat <fbarrat@linux.ibm.com> writes:
>> + opal_poll_events(NULL);
>
> Why is a call to opal_poll_events() needed?
Trying to make sure that opal pollers are run on the current CPU and any
opal timer are executed.
--
Vaibhav Jain <vaibhav@linux.ibm.com>
Linux Technology Center, IBM India Pvt. Ltd.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RFC PATCH 1/2] powerpc/powernv: Add support for CXL mode switch that need PHB reset
2019-01-25 5:11 ` [RFC PATCH 1/2] powerpc/powernv: Add support for CXL mode switch that need PHB reset Vaibhav Jain
2019-01-28 13:34 ` Frederic Barrat
@ 2019-01-28 16:06 ` christophe lombard
2019-01-29 5:13 ` Vaibhav Jain
1 sibling, 1 reply; 9+ messages in thread
From: christophe lombard @ 2019-01-28 16:06 UTC (permalink / raw)
To: Vaibhav Jain, linuxppc-dev, Frederic Barrat
Cc: Philippe Bergheaud, Alastair D'Silva, Christophe Lombard,
Andrew Donnellan
On 25/01/2019 06:11, Vaibhav Jain wrote:
> Recent updates to OPAL [1] have provided support for new CXL modes on
> PHB that need to force a cold reset on the bridge (CRESET). However
> PHB CRESET is a multi step process and cannot be completed
> synchronously as expected by current kernel implementation that issues
> opal call opal_pci_set_phb_cxl_mode().
>
> Hence this patch updates pnv_phb_to_cxl_mode() to implement a polling
> loop that handles specific error codes (OPAL_BUSY) returned from
> opal_pci_set_phb_cxl_mode() and drive the OPAL pci-state machine, if the
> requested CXL mode needs a CRESET.
>
> The patch also updates pnv_phb_to_cxl_mode() to convert and return
> OPAL error codes into kernel error codes. This removes a previous
> issue where callers to this function would have to include
> 'opal-api.h' to check for specific OPAL error codes.
>
> References:
> [1]: https://lists.ozlabs.org/pipermail/skiboot/2019-January/013063.html
>
> Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com>
> ---
> arch/powerpc/platforms/powernv/pci-cxl.c | 71 +++++++++++++++++++++---
> 1 file changed, 63 insertions(+), 8 deletions(-)
>
> diff --git a/arch/powerpc/platforms/powernv/pci-cxl.c b/arch/powerpc/platforms/powernv/pci-cxl.c
> index 1b18111453d7..d33d662c6212 100644
> --- a/arch/powerpc/platforms/powernv/pci-cxl.c
> +++ b/arch/powerpc/platforms/powernv/pci-cxl.c
> @@ -10,6 +10,7 @@
> #include <linux/module.h>
> #include <asm/pnv-pci.h>
> #include <asm/opal.h>
> +#include <linux/delay.h>
>
> #include "pci.h"
>
> @@ -18,21 +19,75 @@ int pnv_phb_to_cxl_mode(struct pci_dev *dev, uint64_t mode)
> struct pci_controller *hose = pci_bus_to_host(dev->bus);
> struct pnv_phb *phb = hose->private_data;
> struct pnv_ioda_pe *pe;
> + unsigned long starttime, endtime;
> int rc;
>
> pe = pnv_ioda_get_pe(dev);
> if (!pe)
> - return -ENODEV;
> + return -ENOENT;
The return code of pnv_phb_to_cxl_mode() is also returned by an api in
the cxllib librarie. So, hoping that nobody test the value !!
> +
> + pe_info(pe, "Switching PHB to CXL mode=%d\n", mode);
> +
> + /*
> + * Use a 15 second timeout for mode switch. Value arrived after
> + * limited testing and may need more tweaking.
> + */
> + starttime = jiffies;
> + endtime = starttime + HZ * 15;
> +
> + do {
> + rc = opal_pci_set_phb_cxl_mode(phb->opal_id, mode,
> + pe->pe_number);
> +
> + /* Wait until mode transistion done */
> + if (rc != OPAL_BUSY && rc != OPAL_BUSY_EVENT)
> + break;
> +
> + /* Check if we timedout */
> + if (time_after(jiffies, endtime)) {
> + rc = OPAL_TIMEOUT;
> + break;
> + }
>
> - pe_info(pe, "Switching PHB to CXL\n");
> + /* Opal Busy with mode switch. Run pci state-machine */
> + rc = opal_pci_poll(phb->opal_id);
> + if (rc >= 0) {
> + /* wait for some time */
> + if (rc > 0)
> + msleep(rc);
> + opal_poll_events(NULL);
> + rc = OPAL_BUSY;
> + /* Continue with the mode switch */
> + }
> + } while (rc == OPAL_BUSY || rc == OPAL_BUSY_EVENT);
> +
> + pe_level_printk(pe, KERN_DEBUG, "CXL mode switch finished in %u-msecs.",
> + jiffies_to_msecs(jiffies - starttime));
>
> - rc = opal_pci_set_phb_cxl_mode(phb->opal_id, mode, pe->pe_number);
> - if (rc == OPAL_UNSUPPORTED)
> - dev_err(&dev->dev, "Required cxl mode not supported by firmware - update skiboot\n");
> - else if (rc)
> - dev_err(&dev->dev, "opal_pci_set_phb_cxl_mode failed: %i\n", rc);
> + /* Check OPAL errors and convert them to kernel error codes */
> + switch (rc) {
> + case OPAL_SUCCESS:
> + return 0;
>
> - return rc;
> + case OPAL_PARAMETER:
> + dev_err(&dev->dev, "CXL not supported on this PHB\n");
> + return -ENOENT;
> +
> + case OPAL_UNSUPPORTED:
> + dev_err(&dev->dev,
> + "Required cxl mode not supported by firmware"
> + " - update skiboot\n");
> + return -ENODEV;
> +
> + case OPAL_TIMEOUT:
> + dev_err(&dev->dev, "opal_pci_set_phb_cxl_mode Timedout\n");
> + return -ETIME;
> +
> + default:
> + dev_err(&dev->dev,
> + "opal_pci_set_phb_cxl_mode failed: %i\n", rc);
> + return -EIO;
> + };
> }
> EXPORT_SYMBOL(pnv_phb_to_cxl_mode);
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RFC PATCH 1/2] powerpc/powernv: Add support for CXL mode switch that need PHB reset
2019-01-28 16:06 ` christophe lombard
@ 2019-01-29 5:13 ` Vaibhav Jain
0 siblings, 0 replies; 9+ messages in thread
From: Vaibhav Jain @ 2019-01-29 5:13 UTC (permalink / raw)
To: christophe lombard, linuxppc-dev, Frederic Barrat
Cc: Philippe Bergheaud, Alastair D'Silva, Christophe Lombard,
Andrew Donnellan
Thanks for reviewing this patch Christophe,
christophe lombard <clombard@linux.vnet.ibm.com> writes:
>>
>> pe = pnv_ioda_get_pe(dev);
>> if (!pe)
>> - return -ENODEV;
>> + return -ENOENT;
>
> The return code of pnv_phb_to_cxl_mode() is also returned by an api in
> the cxllib librarie. So, hoping that nobody test the value !!
Agreed. I did peek into cxllib_switch_phb_mode() before sending the
patch and saw two conflicting cases. While switching to CXL_MODE_PCI we
make sure that we return kernel error codes and while switching to
CXL_MODE_CXL we return OPAL error codes.
I havent seen how CX5 handles return values from this function but I am
betting thats its the usual zero & non-zero return value check, which
then should work with the proposed change.
--
Vaibhav Jain <vaibhav@linux.ibm.com>
Linux Technology Center, IBM India Pvt. Ltd.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH 2/2] cxl: Force a CAPP reset when unloading CXL module
2019-01-25 5:11 [RFC PATCH 0/2] cxl: Add support for disabling CAPP when unloading CXL Vaibhav Jain
2019-01-25 5:11 ` [RFC PATCH 1/2] powerpc/powernv: Add support for CXL mode switch that need PHB reset Vaibhav Jain
@ 2019-01-25 5:11 ` Vaibhav Jain
2019-01-28 13:34 ` Frederic Barrat
2019-01-28 13:34 ` [RFC PATCH 0/2] cxl: Add support for disabling CAPP when unloading CXL Frederic Barrat
2 siblings, 1 reply; 9+ messages in thread
From: Vaibhav Jain @ 2019-01-25 5:11 UTC (permalink / raw)
To: linuxppc-dev, Frederic Barrat
Cc: Philippe Bergheaud, Vaibhav Jain, Alastair D'Silva,
Christophe Lombard, Andrew Donnellan
This patch forces shutdown of CAPP when CXL module is unloaded. This
is accomplished via a call to pnv_phb_to_cxl_mode() with mode ==
OPAL_PHB_CAPI_MODE_PCIE.
Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com>
---
drivers/misc/cxl/cxl.h | 1 +
drivers/misc/cxl/main.c | 3 +++
drivers/misc/cxl/pci.c | 25 ++++++++++++++++++++++++-
3 files changed, 28 insertions(+), 1 deletion(-)
diff --git a/drivers/misc/cxl/cxl.h b/drivers/misc/cxl/cxl.h
index d1d927ccb589..e545c2b81faf 100644
--- a/drivers/misc/cxl/cxl.h
+++ b/drivers/misc/cxl/cxl.h
@@ -1136,4 +1136,5 @@ void cxl_context_mm_count_get(struct cxl_context *ctx);
/* Decrements the reference count to "struct mm_struct" */
void cxl_context_mm_count_put(struct cxl_context *ctx);
+void cxl_pci_shutdown_capp(void);
#endif
diff --git a/drivers/misc/cxl/main.c b/drivers/misc/cxl/main.c
index f35406be465a..f14ff0dcf231 100644
--- a/drivers/misc/cxl/main.c
+++ b/drivers/misc/cxl/main.c
@@ -372,6 +372,9 @@ static void exit_cxl(void)
if (cxl_is_power8())
unregister_cxl_calls(&cxl_calls);
idr_destroy(&cxl_adapter_idr);
+
+ if (cpu_has_feature(CPU_FTR_HVMODE))
+ cxl_pci_shutdown_capp();
}
module_init(init_cxl);
diff --git a/drivers/misc/cxl/pci.c b/drivers/misc/cxl/pci.c
index c79ba1c699ad..01be2e2d1069 100644
--- a/drivers/misc/cxl/pci.c
+++ b/drivers/misc/cxl/pci.c
@@ -25,7 +25,7 @@
#include "cxl.h"
#include <misc/cxl.h>
-
+#include <misc/cxllib.h>
#define CXL_PCI_VSEC_ID 0x1280
#define CXL_VSEC_MIN_SIZE 0x80
@@ -2065,6 +2065,29 @@ static void cxl_pci_resume(struct pci_dev *pdev)
}
}
+void cxl_pci_shutdown_capp(void)
+{
+ struct pci_dev *pdev;
+ struct pci_bus *root_bus;
+ int rc;
+
+ /* Iterate over all CAPP supported PHB's and force them to PCI mode */
+ list_for_each_entry(root_bus, &pci_root_buses, node) {
+ for_each_pci_bridge(pdev, root_bus) {
+
+ if (!cxllib_slot_is_supported(pdev, 0))
+ continue;
+
+ rc = pnv_phb_to_cxl_mode(pdev,
+ OPAL_PHB_CAPI_MODE_PCIE);
+ if (rc)
+ dev_err(&pdev->dev,
+ "cxl: Error resetting CAPP. Err=%d\n",
+ rc);
+ }
+ }
+}
+
static const struct pci_error_handlers cxl_err_handler = {
.error_detected = cxl_pci_error_detected,
.slot_reset = cxl_pci_slot_reset,
--
2.20.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [RFC PATCH 2/2] cxl: Force a CAPP reset when unloading CXL module
2019-01-25 5:11 ` [RFC PATCH 2/2] cxl: Force a CAPP reset when unloading CXL module Vaibhav Jain
@ 2019-01-28 13:34 ` Frederic Barrat
0 siblings, 0 replies; 9+ messages in thread
From: Frederic Barrat @ 2019-01-28 13:34 UTC (permalink / raw)
To: Vaibhav Jain, linuxppc-dev
Cc: Philippe Bergheaud, Alastair D'Silva, Christophe Lombard,
Andrew Donnellan
Le 25/01/2019 à 06:11, Vaibhav Jain a écrit :
> This patch forces shutdown of CAPP when CXL module is unloaded. This
> is accomplished via a call to pnv_phb_to_cxl_mode() with mode ==
> OPAL_PHB_CAPI_MODE_PCIE.
>
> Signed-off-by: Vaibhav Jain <vaibhav@linux.ibm.com>
> ---
> drivers/misc/cxl/cxl.h | 1 +
> drivers/misc/cxl/main.c | 3 +++
> drivers/misc/cxl/pci.c | 25 ++++++++++++++++++++++++-
> 3 files changed, 28 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/misc/cxl/cxl.h b/drivers/misc/cxl/cxl.h
> index d1d927ccb589..e545c2b81faf 100644
> --- a/drivers/misc/cxl/cxl.h
> +++ b/drivers/misc/cxl/cxl.h
> @@ -1136,4 +1136,5 @@ void cxl_context_mm_count_get(struct cxl_context *ctx);
> /* Decrements the reference count to "struct mm_struct" */
> void cxl_context_mm_count_put(struct cxl_context *ctx);
>
> +void cxl_pci_shutdown_capp(void);
> #endif
> diff --git a/drivers/misc/cxl/main.c b/drivers/misc/cxl/main.c
> index f35406be465a..f14ff0dcf231 100644
> --- a/drivers/misc/cxl/main.c
> +++ b/drivers/misc/cxl/main.c
> @@ -372,6 +372,9 @@ static void exit_cxl(void)
> if (cxl_is_power8())
> unregister_cxl_calls(&cxl_calls);
> idr_destroy(&cxl_adapter_idr);
> +
> + if (cpu_has_feature(CPU_FTR_HVMODE))
> + cxl_pci_shutdown_capp();
> }
>
> module_init(init_cxl);
> diff --git a/drivers/misc/cxl/pci.c b/drivers/misc/cxl/pci.c
> index c79ba1c699ad..01be2e2d1069 100644
> --- a/drivers/misc/cxl/pci.c
> +++ b/drivers/misc/cxl/pci.c
> @@ -25,7 +25,7 @@
>
> #include "cxl.h"
> #include <misc/cxl.h>
> -
> +#include <misc/cxllib.h>
>
> #define CXL_PCI_VSEC_ID 0x1280
> #define CXL_VSEC_MIN_SIZE 0x80
> @@ -2065,6 +2065,29 @@ static void cxl_pci_resume(struct pci_dev *pdev)
> }
> }
>
> +void cxl_pci_shutdown_capp(void)
> +{
> + struct pci_dev *pdev;
> + struct pci_bus *root_bus;
> + int rc;
> +
> + /* Iterate over all CAPP supported PHB's and force them to PCI mode */
> + list_for_each_entry(root_bus, &pci_root_buses, node) {
> + for_each_pci_bridge(pdev, root_bus) {
> +
> + if (!cxllib_slot_is_supported(pdev, 0))
> + continue;
> +
> + rc = pnv_phb_to_cxl_mode(pdev,
> + OPAL_PHB_CAPI_MODE_PCIE);
> + if (rc)
> + dev_err(&pdev->dev,
> + "cxl: Error resetting CAPP. Err=%d\n",
> + rc);
> + }
That's the part I don't like. We're iterating over quite a few PCI
devices, we basically don't know the ones we need to reset.
If we have a per-adapter property on /sys to activate the
reset-on-unload, then we could move the call to
pnv_phb_to_cxl_mode(OPAL_PHB_CAPI_MODE_PCIE) on the cxl_remove()
callback, and only do it for the adapters we've been asked.
Fred
> + }
> +}
> +
> static const struct pci_error_handlers cxl_err_handler = {
> .error_detected = cxl_pci_error_detected,
> .slot_reset = cxl_pci_slot_reset,
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RFC PATCH 0/2] cxl: Add support for disabling CAPP when unloading CXL
2019-01-25 5:11 [RFC PATCH 0/2] cxl: Add support for disabling CAPP when unloading CXL Vaibhav Jain
2019-01-25 5:11 ` [RFC PATCH 1/2] powerpc/powernv: Add support for CXL mode switch that need PHB reset Vaibhav Jain
2019-01-25 5:11 ` [RFC PATCH 2/2] cxl: Force a CAPP reset when unloading CXL module Vaibhav Jain
@ 2019-01-28 13:34 ` Frederic Barrat
2 siblings, 0 replies; 9+ messages in thread
From: Frederic Barrat @ 2019-01-28 13:34 UTC (permalink / raw)
To: Vaibhav Jain, linuxppc-dev
Cc: Philippe Bergheaud, Alastair D'Silva, Christophe Lombard,
Andrew Donnellan
Hi Vaibhav,
I think there's value in there, as I'm hearing Mellanox would prefer to
start from a "clean" state, as in pci mode, when they load their driver
after a card FW update.
However, I'm pretty reluctant to make it the default behaviour. It's not
like we haven't had our share of problems on reset before :-)
Furthermore, for non-mlx5 case, we don't really have a reason for this.
I'm thinking we could activate it through a per-adapter property, on
/sys. See comment in one of the patches.
Fred
Le 25/01/2019 à 06:11, Vaibhav Jain a écrit :
> Recent updates to OPAL have implemented necessary infrastructure [1] to disable
> CAPP and switch PHB back to PCIE mode during fast reset. This small patch-set
> uses the same OPAL infrastructure to force disable of CAPP when CXL module is
> unloaded via rmmod.
>
> References:
> [1]: https://lists.ozlabs.org/pipermail/skiboot/2019-January/013063.html
>
> Vaibhav Jain (2):
> powerpc/powernv: Add support for CXL mode switch that need PHB reset
> cxl: Force a CAPP reset when unloading CXL module
>
> arch/powerpc/platforms/powernv/pci-cxl.c | 71 +++++++++++++++++++++---
> drivers/misc/cxl/cxl.h | 1 +
> drivers/misc/cxl/main.c | 3 +
> drivers/misc/cxl/pci.c | 25 ++++++++-
> 4 files changed, 91 insertions(+), 9 deletions(-)
>
^ permalink raw reply [flat|nested] 9+ messages in thread