From: Ira Weiny <ira.weiny@intel.com>
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: Dan Williams <dan.j.williams@intel.com>,
Bjorn Helgaas <bhelgaas@google.com>,
Jonathan Cameron <Jonathan.Cameron@huawei.com>,
"Alison Schofield" <alison.schofield@intel.com>,
Vishal Verma <vishal.l.verma@intel.com>,
Dave Jiang <dave.jiang@intel.com>,
Ben Widawsky <bwidawsk@kernel.org>,
<linux-kernel@vger.kernel.org>, <linux-cxl@vger.kernel.org>,
<linux-pci@vger.kernel.org>
Subject: Re: [PATCH V11 3/8] PCI: Create PCI library functions in support of DOE mailboxes.
Date: Wed, 22 Jun 2022 09:38:31 -0700 [thread overview]
Message-ID: <YrNFh9CfwxmO+rsy@iweiny-desk3> (raw)
In-Reply-To: <20220617224019.GA1208614@bhelgaas>
On Fri, Jun 17, 2022 at 05:40:19PM -0500, Bjorn Helgaas wrote:
> On Fri, Jun 10, 2022 at 01:22:54PM -0700, ira.weiny@intel.com wrote:
> > From: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> >
> > Introduced in a PCI r6.0, sec 6.30, DOE provides a config space based
> > mailbox with standard protocol discovery. Each mailbox is accessed
> > through a DOE Extended Capability.
>
> > +/* Timeout of 1 second from 6.30.2 Operation, PCI Spec r6.0 */
>
> s/PCI/PCIe/ (up in commit log, too, I guess :))
>
> Not that there will ever be a conventional PCI r6.0 spec, but there
> was a PCI r3.0 well as a PCIe r3.0, so might as well keep them
> straight.
Done.
>
> > +struct pci_doe_mb {
> > + struct pci_dev *pdev;
>
> Trivial, but I would put cap_offset here next to pdev because the
> (pdev, cap_offset) tuple is basically the identifier for the DOE
> instance.
Done.
>
> > + struct completion abort_c;
> > + int irq;
> > + struct pci_doe_protocol *prots;
> > + int num_prots;
> > + u16 cap_offset;
>
> > +static void pci_doe_abort_start(struct pci_doe_mb *doe_mb)
> > +{
> > + struct pci_dev *pdev = doe_mb->pdev;
> > + int offset = doe_mb->cap_offset;
> > + u32 val;
> > +
> > + val = PCI_DOE_CTRL_ABORT;
> > + if (doe_mb->irq >= 0)
>
> Is zero a valid IRQ? In general, I don't think it is, but maybe this
> is a special case. Or maybe this is actually the "Interrupt Message
> Number" mentioned in sec 6.30.3? If so maybe something other than
> "irq" would be a better name here.
Yes I think irq is a bad name. I think 0 is valid here because this is the
Interrupt Message Number" from the DOE Capabilities Register (7.9.24.2).
At least with Qemu 0 is returned for the 1st mailbox. I'm not sure if that is
valid or not but I think it is.
But reading that in detail I think there is even more complexity than Jonathan
or I realized with regard to MSI vs MSI-X.
I'm going to leave the irq support in this layer (changing 'irq' to
'irq_msg_num'?) but I think the callers will need to resolve what support they
enable.
>
> Possibly relevant: a85a6c86c25b ("driver core: platform: Clarify that
> IRQ 0 is invalid")
>
> > + pci_err(pdev,
> > + "DOE [%x] expected [VID, Protocol] = [%04x, %02x], got [%04x, %02x]\n",
>
> Wouldn't make a big difference, but could consider something like this
> for enforced consistency:
>
> #define dev_fmt(fmt) "DOE: " fmt
Good idea.
>
> > + case DOE_WAIT_ABORT:
> > + case DOE_WAIT_ABORT_ON_ERR:
> > + prev_state = doe_mb->state;
> > +
> > + pci_read_config_dword(pdev, offset + PCI_DOE_STATUS, &val);
> > +
> > + if (!FIELD_GET(PCI_DOE_STATUS_ERROR, val) &&
> > + !FIELD_GET(PCI_DOE_STATUS_BUSY, val)) {
> > + doe_mb->state = DOE_IDLE;
> > + /* Back to normal state - carry on */
> > + retire_cur_task(doe_mb);
> > + } else if (time_after(jiffies, doe_mb->timeout_jiffies)) {
> > + /* Task has timed out and is dead - abort */
> > + pci_err(pdev, "DOE [%x] ABORT timed out\n",
> > + doe_mb->cap_offset);
> > + set_bit(PCI_DOE_FLAG_DEAD, &doe_mb->flags);
> > + retire_cur_task(doe_mb);
> > + }
> > +
> > + /*
> > + * For deliberately triggered abort, someone is
> > + * waiting.
> > + */
> > + if (prev_state == DOE_WAIT_ABORT) {
> > + if (task)
> > + signal_task_complete(task, -EFAULT);
> > + complete(&doe_mb->abort_c);
> > + }
> > +
> > + return;
> > + }
>
> The "return" in each case is perfectly correct, but it feels a little
> more conventional to make them "break" and return once here after the
> switch to make it clear that the only way to get to the labels is via
> an error path "goto".
Done.
>
> > +err_abort:
> > + doe_mb->state = DOE_WAIT_ABORT_ON_ERR;
> > + pci_doe_abort_start(doe_mb);
> > +err_busy:
> > + signal_task_complete(task, rc);
> > + if (doe_mb->state == DOE_IDLE)
> > + retire_cur_task(doe_mb);
> > +}
>
> > + * Enabling bus mastering is required for MSI/MSIx. It is safe to call
>
> s/MSIx/MSI-X/ (typical spelling in spec)
>
> > + * this multiple times and thus is called here to ensure that mastering
> > + * is enabled even if the driver has done so.
> > + */
> > + pci_set_master(pdev);
> > + rc = pci_request_irq(pdev, irq, pci_doe_irq_handler, NULL, doe_mb,
> > + "DOE[%d:%s]", irq, pci_name(pdev));
>
> I assume the "DOE[%d:%s]" part appears in /proc/interrupts?
Yes
> Is it
> redundant to include "irq", since /proc/interrupts already prints it,
> or is there somewhere else where "irq" is useful?
As you pointed out irq is the wrong name here. This is just the message
number.
>
> How does the user associate this IRQ in /proc/interrupts with a
> specific DOE capability? Should we include the cap_offset along with
> the pci_name()?
Good idea, cap_offset is much more useful. In my testing the irq's were all
unique but as Dan pointed out I did not realize that the message number could
be shared.
>
> > + * pci_doe_get_irq_num() - Return the irq number for the mailbox at offset
> > + *
> > + * @pdev: The PCI device
> > + * @offset: Offset of the DOE mailbox
> > + *
> > + * Returns: irq number on success
> > + * -errno if irqs are not supported on this mailbox
>
> I normally capitalize IRQ/IRQs in comments. There are probably others
> throughout the file. I notice some are already capitalized but not all.
Done.
>
> > + */
> > +int pci_doe_get_irq_num(struct pci_dev *pdev, int offset)
> > +{
> > + u32 val;
> > +
> > + pci_read_config_dword(pdev, offset + PCI_DOE_CAP, &val);
> > + if (!FIELD_GET(PCI_DOE_CAP_INT, val))
> > + return -EOPNOTSUPP;
> > +
> > + return FIELD_GET(PCI_DOE_CAP_IRQ, val);
> > +}
> > +EXPORT_SYMBOL_GPL(pci_doe_get_irq_num);
>
> Confusing function name (and comment) since PCI_DOE_CAP_IRQ is an
> Interrupt Message Number that has nothing to do with Linux IRQ
> numbers.
Agreed. Changed to pci_doe_get_irq_msg_num(); With corresponding fixups to
the kdoc.
>
> I see we already have PCI_EXP_FLAGS_IRQ, PCI_ERR_ROOT_AER_IRQ,
> PCI_EXP_DPC_IRQ, so I guess you're in good company.
>
> At least maybe update the comment to say "Interrupt Message Number"
> instead of "irq".
Yea I did that too.
>
> > + * pci_doe_supports_prot() - Return if the DOE instance supports the given
> > + * protocol
> > + * @doe_mb: DOE mailbox capability to query
> > + * @vid: Protocol Vendor ID
> > + * @type: Protocol type
> > + *
> > + * RETURNS: True if the DOE mailbox supports the protocol specified
>
> Is the typical use that the caller has a few specific protocols it
> cares about?
That is how CXL needs it right now yes.
> There's no case where a caller might want to enumerate
> them all?
Not at this time.
> I guess they're all in prots[], but that's supposed to be
> opaque to users.
Agreed. Something else would be needed in that use case.
>
> > + */
> > +bool pci_doe_supports_prot(struct pci_doe_mb *doe_mb, u16 vid, u8 type)
> > +{
> > + int i;
> > +
> > + /* The discovery protocol must always be supported */
> > + if (vid == PCI_VENDOR_ID_PCI_SIG && type == PCI_DOE_PROTOCOL_DISCOVERY)
> > + return true;
> > +
> > + for (i = 0; i < doe_mb->num_prots; i++)
> > + if ((doe_mb->prots[i].vid == vid) &&
> > + (doe_mb->prots[i].type == type))
> > + return true;
> > +
> > + return false;
> > +}
> > +EXPORT_SYMBOL_GPL(pci_doe_supports_prot);
>
> > + * struct pci_doe_task - represents a single query/response
> > + *
> > + * @prot: DOE Protocol
> > + * @request_pl: The request payload
> > + * @request_pl_sz: Size of the request payload
>
> Size is in dwords, not bytes, I guess?
No. Those are in bytes and the DOE layer takes care of the DW conversion.
I'll update the kdoc to make that clear.
Thanks again for the review,
Ira
>
> > + * @response_pl: The response payload
> > + * @response_pl_sz: Size of the response payload
> > + * @rv: Return value. Length of received response or error
> > + * @complete: Called when task is complete
> > + * @private: Private data for the consumer
> > + */
> > +struct pci_doe_task {
> > + struct pci_doe_protocol prot;
> > + u32 *request_pl;
> > + size_t request_pl_sz;
> > + u32 *response_pl;
> > + size_t response_pl_sz;
> > + int rv;
> > + void (*complete)(struct pci_doe_task *task);
> > + void *private;
> > +};
next prev parent reply other threads:[~2022-06-22 16:38 UTC|newest]
Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-06-10 20:22 [PATCH V11 0/8] CXL: Read CDAT and DSMAS data ira.weiny
2022-06-10 20:22 ` [PATCH V11 1/8] PCI: Add vendor ID for the PCI SIG ira.weiny
2022-06-10 20:22 ` [PATCH V11 2/8] PCI: Replace magic constant for PCI Sig Vendor ID ira.weiny
2022-06-10 20:22 ` [PATCH V11 3/8] PCI: Create PCI library functions in support of DOE mailboxes ira.weiny
2022-06-14 3:53 ` Li, Ming
2022-06-15 4:18 ` Ira Weiny
2022-06-17 22:40 ` Bjorn Helgaas
2022-06-18 16:39 ` Bjorn Helgaas
2022-06-22 16:46 ` Ira Weiny
2022-06-20 9:24 ` Jonathan Cameron
2022-06-22 23:06 ` Ira Weiny
2022-06-22 16:38 ` Ira Weiny [this message]
2022-06-17 22:56 ` Dan Williams
2022-06-20 10:23 ` Jonathan Cameron
2022-06-22 22:57 ` Ira Weiny
2022-06-23 18:03 ` Dan Williams
2022-06-22 22:37 ` Ira Weiny
2022-06-22 22:45 ` Ira Weiny
2022-06-22 22:57 ` Dan Williams
2022-06-23 0:25 ` Ira Weiny
2022-06-23 10:24 ` Jonathan Cameron
2022-06-23 18:14 ` Dan Williams
2022-06-23 18:07 ` Dan Williams
2022-06-10 20:22 ` [PATCH V11 4/8] cxl/pci: Create PCI DOE mailbox's for memory devices ira.weiny
2022-06-17 20:40 ` [PATCH v11 " Davidlohr Bueso
2022-06-17 20:51 ` Davidlohr Bueso
2022-06-21 18:24 ` Ira Weiny
2022-06-17 23:44 ` [PATCH V11 " Dan Williams
2022-06-21 18:29 ` Ira Weiny
2022-06-22 23:18 ` Ira Weiny
2022-06-21 20:37 ` Bjorn Helgaas
2022-06-10 20:22 ` [PATCH V11 5/8] cxl/port: Read CDAT table ira.weiny
2022-06-18 0:43 ` Dan Williams
2022-06-21 19:10 ` Dan Williams
2022-06-21 19:34 ` Lukas Wunner
2022-06-21 19:41 ` Dan Williams
2022-06-21 20:38 ` Ira Weiny
2022-06-21 21:14 ` Ira Weiny
2022-06-21 21:48 ` Dan Williams
2022-06-28 3:24 ` Ira Weiny
2022-06-10 20:22 ` [PATCH V11 6/8] cxl/port: Introduce cxl_cdat_valid() ira.weiny
2022-06-10 20:22 ` [PATCH V11 7/8] cxl/port: Retry reading CDAT on failure ira.weiny
2022-06-28 3:32 ` Alison Schofield
2022-06-10 20:22 ` [PATCH V11 8/8] cxl/port: Parse out DSMAS data from CDAT table ira.weiny
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YrNFh9CfwxmO+rsy@iweiny-desk3 \
--to=ira.weiny@intel.com \
--cc=Jonathan.Cameron@huawei.com \
--cc=alison.schofield@intel.com \
--cc=bhelgaas@google.com \
--cc=bwidawsk@kernel.org \
--cc=dan.j.williams@intel.com \
--cc=dave.jiang@intel.com \
--cc=helgaas@kernel.org \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=vishal.l.verma@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).