From: Dan Williams <dan.j.williams@intel.com>
To: Terry Bowman <terry.bowman@amd.com>, <alison.schofield@intel.com>,
<vishal.l.verma@intel.com>, <ira.weiny@intel.com>,
<bwidawsk@kernel.org>, <dan.j.williams@intel.com>,
<dave.jiang@intel.com>, <Jonathan.Cameron@huawei.com>,
<linux-cxl@vger.kernel.org>
Cc: <terry.bowman@amd.com>, <rrichter@amd.com>,
<linux-kernel@vger.kernel.org>, <bhelgaas@google.com>
Subject: RE: [PATCH v5 11/26] cxl/pci: Early setup RCH dport component registers from RCRB
Date: Fri, 9 Jun 2023 18:36:48 -0700 [thread overview]
Message-ID: <6483d3b06bef6_e067a294ea@dwillia2-xfh.jf.intel.com.notmuch> (raw)
In-Reply-To: <20230607221651.2454764-12-terry.bowman@amd.com>
Terry Bowman wrote:
> From: Robert Richter <rrichter@amd.com>
>
> CXL RAS capabilities must be enabled and accessible as soon as the CXL
> endpoint is detected in the PCI hierarchy and bound to the cxl_pci
> driver. This needs to be independent of other modules such as cxl_port
> or cxl_mem.
>
> CXL RAS capabilities reside in the Component Registers. For an RCH
> this is determined by probing RCRB which is implemented very late once
> the CXL Memory Device is created.
>
> Change this by moving the RCRB probe to the cxl_pci driver. Do this by
> using a new introduced function cxl_pci_find_port() similar to
> cxl_mem_find_port() to determine the involved dport by the endpoint's
> PCI handle. Plug this into the existing cxl_pci_setup_regs() function
> to setup Component Registers. Probe the RCRB in case the Component
> Registers cannot be located through the CXL Register Locator
> capability.
>
> This unifies code and early sets up the Component Registers at the
> same time for both, VH and RCH mode. Only the cxl_pci driver is
> involved for this. This allows an early mapping of the CXL RAS
> capability registers.
This is problematic because it creates an implicit dependency between
cxl_acpi and cxl_pci. More comments below:
>
> Signed-off-by: Robert Richter <rrichter@amd.com>
> Signed-off-by: Terry Bowman <terry.bowman@amd.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> ---
> drivers/cxl/core/port.c | 7 +++++++
> drivers/cxl/cxl.h | 2 ++
> drivers/cxl/mem.c | 10 ----------
> drivers/cxl/pci.c | 37 ++++++++++++++++++++++++++++++++++++-
> 4 files changed, 45 insertions(+), 11 deletions(-)
>
> diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
> index 82de858506c7..8b688ac506ca 100644
> --- a/drivers/cxl/core/port.c
> +++ b/drivers/cxl/core/port.c
> @@ -1476,6 +1476,13 @@ int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd)
> }
> EXPORT_SYMBOL_NS_GPL(devm_cxl_enumerate_ports, CXL);
>
> +struct cxl_port *cxl_pci_find_port(struct pci_dev *pdev,
> + struct cxl_dport **dport)
> +{
> + return find_cxl_port(pdev->dev.parent, dport);
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_pci_find_port, CXL);
> +
> struct cxl_port *cxl_mem_find_port(struct cxl_memdev *cxlmd,
> struct cxl_dport **dport)
> {
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index 1c6fe53e9dc7..e5ae5f4e6669 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -670,6 +670,8 @@ struct cxl_port *find_cxl_root(struct cxl_port *port);
> int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd);
> void cxl_bus_rescan(void);
> void cxl_bus_drain(void);
> +struct cxl_port *cxl_pci_find_port(struct pci_dev *pdev,
> + struct cxl_dport **dport);
> struct cxl_port *cxl_mem_find_port(struct cxl_memdev *cxlmd,
> struct cxl_dport **dport);
> bool schedule_cxl_memdev_detach(struct cxl_memdev *cxlmd);
> diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> index a34d6560c25c..0643852444f3 100644
> --- a/drivers/cxl/mem.c
> +++ b/drivers/cxl/mem.c
> @@ -65,16 +65,6 @@ static int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd,
> ep->next = down;
> }
>
> - /*
> - * The component registers for an RCD might come from the
> - * host-bridge RCRB if they are not already mapped via the
> - * typical register locator mechanism.
> - */
> - if (parent_dport->rch && cxlds->component_reg_phys == CXL_RESOURCE_NONE)
> - cxlds->component_reg_phys =
> - cxl_probe_rcrb(&cxlmd->dev, parent_dport->rcrb.base,
> - NULL, CXL_RCRB_UPSTREAM);
> -
> endpoint = devm_cxl_add_port(host, &cxlmd->dev,
> cxlds->component_reg_phys,
> parent_dport);
> diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
> index 945ca0304d68..2975b232fcd1 100644
> --- a/drivers/cxl/pci.c
> +++ b/drivers/cxl/pci.c
> @@ -274,13 +274,48 @@ static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds)
> return 0;
> }
>
> +/* Extract RCRB, use same function interface as cxl_find_regblock(). */
> +static int cxl_rcrb_get_comp_regs(struct pci_dev *pdev,
> + enum cxl_regloc_type type,
> + struct cxl_register_map *map)
> +{
> + struct cxl_dport *dport;
> + resource_size_t component_reg_phys;
> +
> + memset(map, 0, sizeof(*map));
> + map->dev = &pdev->dev;
> + map->resource = CXL_RESOURCE_NONE;
> +
> + if (type != CXL_REGLOC_RBI_COMPONENT)
> + return -ENODEV;
Why would a function called cxl_rcrb_get_comp_regs() need a type
parameter that only takes one acceptable value. I would drop the
parameter and move this distinction to the caller.
> +
> + if (!cxl_pci_find_port(pdev, &dport) || !dport->rch)
> + return -ENXIO;
This should return -EPROBE_DEFER in the !cxl_pci_find_port() case to try
to await cxl_acpi initialization.
> +
> + component_reg_phys = cxl_probe_rcrb(&pdev->dev, dport->rcrb.base,
> + NULL, CXL_RCRB_UPSTREAM);
> + if (component_reg_phys == CXL_RESOURCE_NONE)
> + return -ENXIO;
> +
> + map->resource = component_reg_phys;
> + map->reg_type = type;
> + map->max_size = CXL_COMPONENT_REG_BLOCK_SIZE;
> +
> + return 0;
> +}
> +
> static int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,
> struct cxl_register_map *map)
> {
> int rc;
>
> + /*
> + * If the Register Locator DVSEC does not contain the
> + * Component Registers, assume it is an RCH and try to extract
> + * them from an RCRB.
> + */
> rc = cxl_find_regblock(pdev, type, map);
> - if (rc)
> + if (rc && cxl_rcrb_get_comp_regs(pdev, type, map))
> return rc;
I am not a big fan of just assuming this is an RCD, especially when
there is the is_cxl_restricted() helper just beneath this.
next prev parent reply other threads:[~2023-06-10 1:37 UTC|newest]
Thread overview: 69+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-06-07 22:16 [PATCH v5 00/26] cxl/pci: Add support for RCH RAS error handling Terry Bowman
2023-06-07 22:16 ` [PATCH v5 01/26] cxl/acpi: Probe RCRB later during RCH downstream port creation Terry Bowman
2023-06-08 1:03 ` Dan Williams
2023-06-08 1:11 ` Dan Williams
2023-06-07 22:16 ` [PATCH v5 02/26] cxl/rch: Prepare for caching the MMIO mapped PCIe AER capability Terry Bowman
2023-06-08 4:53 ` Dan Williams
2023-06-07 22:16 ` [PATCH v5 03/26] cxl: Rename member @dport of struct cxl_dport to @dev Terry Bowman
2023-06-08 6:42 ` Dan Williams
2023-06-08 14:36 ` Terry Bowman
2023-06-08 19:08 ` Dan Williams
2023-06-08 19:22 ` Terry Bowman
2023-06-07 22:16 ` [PATCH v5 04/26] cxl/core/regs: Rename phys_addr in cxl_map_component_regs() Terry Bowman
2023-06-08 6:47 ` Dan Williams
2023-06-07 22:16 ` [PATCH v5 05/26] cxl/core/regs: Add @dev to cxl_register_map Terry Bowman
2023-06-08 19:29 ` Dan Williams
2023-06-08 21:50 ` Terry Bowman
2023-06-07 22:16 ` [PATCH v5 06/26] cxl/pci: Refactor component register discovery for reuse Terry Bowman
2023-06-08 19:57 ` Dan Williams
2023-06-07 22:16 ` [PATCH v5 07/26] cxl/acpi: Moving add_host_bridge_uport() around Terry Bowman
2023-06-08 20:02 ` Dan Williams
2023-06-08 21:50 ` Terry Bowman
2023-06-07 22:16 ` [PATCH v5 08/26] cxl/acpi: Directly bind the CEDT detected CHBCR to the Host Bridge's port Terry Bowman
2023-06-09 4:24 ` Dan Williams
2023-06-07 22:16 ` [PATCH v5 09/26] cxl/regs: Remove early capability checks in Component Register setup Terry Bowman
2023-06-10 0:18 ` Dan Williams
2023-06-07 22:16 ` [PATCH v5 10/26] cxl/mem: Prepare for early RCH dport component register setup Terry Bowman
2023-06-10 0:26 ` Dan Williams
2023-06-07 22:16 ` [PATCH v5 11/26] cxl/pci: Early setup RCH dport component registers from RCRB Terry Bowman
2023-06-10 1:36 ` Dan Williams [this message]
2023-06-10 1:44 ` Dan Williams
2023-06-12 20:39 ` Dan Williams
2023-06-07 22:16 ` [PATCH v5 12/26] cxl/port: Store the port's Component Register mappings in struct cxl_port Terry Bowman
2023-06-10 2:18 ` Dan Williams
2023-06-07 22:16 ` [PATCH v5 13/26] cxl/port: Store the downstream port's Component Register mappings in struct cxl_dport Terry Bowman
2023-06-10 2:23 ` Dan Williams
2023-06-07 22:16 ` [PATCH v5 14/26] cxl/pci: Store the endpoint's Component Register mappings in struct cxl_dev_state Terry Bowman
2023-06-10 2:29 ` Dan Williams
2023-06-07 22:16 ` [PATCH v5 15/26] cxl/hdm: Use stored Component Register mappings to map HDM decoder capability Terry Bowman
2023-06-10 2:34 ` Dan Williams
2023-06-07 22:16 ` [PATCH v5 16/26] cxl/port: Remove Component Register base address from struct cxl_port Terry Bowman
2023-06-10 2:36 ` Dan Williams
2023-06-07 22:16 ` [PATCH v5 17/26] cxl/port: Remove Component Register base address from struct cxl_dport Terry Bowman
2023-06-10 2:37 ` Dan Williams
2023-06-07 22:16 ` [PATCH v5 18/26] cxl/pci: Remove Component Register base address from struct cxl_dev_state Terry Bowman
2023-06-10 2:38 ` Dan Williams
2023-06-07 22:16 ` [PATCH v5 19/26] cxl/pci: Add RCH downstream port AER register discovery Terry Bowman
2023-06-10 3:09 ` Dan Williams
2023-06-12 14:41 ` Terry Bowman
2023-06-07 22:16 ` [PATCH v5 20/26] PCI/AER: Refactor cper_print_aer() for use by CXL driver module Terry Bowman
2023-06-10 3:11 ` Dan Williams
2023-06-07 22:16 ` [PATCH v5 21/26] cxl/pci: Update CXL error logging to use RAS register address Terry Bowman
2023-06-10 3:12 ` Dan Williams
2023-06-12 21:12 ` Dan Williams
2023-06-07 22:16 ` [PATCH v5 22/26] cxl/pci: Map RCH downstream AER registers for logging protocol errors Terry Bowman
2023-06-10 3:23 ` Dan Williams
2023-06-12 18:19 ` Terry Bowman
2023-06-07 22:16 ` [PATCH v5 23/26] cxl/pci: Disable root port interrupts in RCH mode Terry Bowman
2023-06-12 20:29 ` Dan Williams
2023-06-13 15:28 ` Terry Bowman
2023-06-07 22:16 ` [PATCH v5 24/26] cxl/pci: Add RCH downstream port error logging Terry Bowman
2023-06-12 21:38 ` Dan Williams
2023-06-16 16:17 ` Terry Bowman
2023-06-16 16:28 ` Terry Bowman
2023-06-07 22:16 ` [PATCH v5 25/26] PCI/AER: Forward RCH downstream port-detected errors to the CXL.mem dev handler Terry Bowman
2023-06-08 22:54 ` kernel test robot
2023-06-12 22:49 ` Dan Williams
2023-06-07 22:16 ` [PATCH v5 26/26] PCI/AER: Unmask RCEC internal errors to enable RCH downstream port error handling Terry Bowman
2023-06-08 19:21 ` Bjorn Helgaas
2023-06-12 22:57 ` Dan Williams
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6483d3b06bef6_e067a294ea@dwillia2-xfh.jf.intel.com.notmuch \
--to=dan.j.williams@intel.com \
--cc=Jonathan.Cameron@huawei.com \
--cc=alison.schofield@intel.com \
--cc=bhelgaas@google.com \
--cc=bwidawsk@kernel.org \
--cc=dave.jiang@intel.com \
--cc=ira.weiny@intel.com \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=rrichter@amd.com \
--cc=terry.bowman@amd.com \
--cc=vishal.l.verma@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).