From: Dan Williams <dan.j.williams@intel.com>
To: Ben Widawsky <ben.widawsky@intel.com>
Cc: linux-cxl@vger.kernel.org,
Jonathan Cameron <Jonathan.Cameron@huawei.com>,
Linux PCI <linux-pci@vger.kernel.org>,
Linux NVDIMM <nvdimm@lists.linux.dev>
Subject: Re: [PATCH v3 02/40] cxl/pci: Implement Interface Ready Timeout
Date: Mon, 31 Jan 2022 15:11:05 -0800 [thread overview]
Message-ID: <CAPcyv4gGMsUbJ=Cu8U0B2Znz6+jE6PcDCALNfxJzvAHZZccVvQ@mail.gmail.com> (raw)
In-Reply-To: <20220131222101.tckwbcxheuuorkiq@intel.com>
On Mon, Jan 31, 2022 at 2:21 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
>
> On 22-01-23 16:28:49, Dan Williams wrote:
> > From: Ben Widawsky <ben.widawsky@intel.com>
> >
> > The original driver implementation used the doorbell timeout for the
> > Mailbox Interface Ready bit to piggy back off of, since the latter does
> > not have a defined timeout. This functionality, introduced in commit
> > 8adaf747c9f0 ("cxl/mem: Find device capabilities"), needs improvement as
> > the recent "Add Mailbox Ready Time" ECN timeout indicates that the
> > mailbox ready time can be significantly longer that 2 seconds.
> >
> > While the specification limits the maximum timeout to 256s, the cxl_pci
> > driver gives up on the mailbox after 60s. This value corresponds with
> > important timeout values already present in the kernel. A module
> > parameter is provided as an emergency override and represents the
> > default Linux policy for all devices.
> >
> > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> > Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > [djbw: add modparam, drop check_device_status()]
> > Co-developed-by: Dan Williams <dan.j.williams@intel.com>
> > Signed-off-by: Dan Williams <dan.j.williams@intel.com>
> > ---
> > drivers/cxl/pci.c | 35 +++++++++++++++++++++++++++++++++++
> > 1 file changed, 35 insertions(+)
> >
> > diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
> > index 8dc91fd3396a..ed8de9eac970 100644
> > --- a/drivers/cxl/pci.c
> > +++ b/drivers/cxl/pci.c
> > @@ -1,7 +1,9 @@
> > // SPDX-License-Identifier: GPL-2.0-only
> > /* Copyright(c) 2020 Intel Corporation. All rights reserved. */
> > #include <linux/io-64-nonatomic-lo-hi.h>
> > +#include <linux/moduleparam.h>
> > #include <linux/module.h>
> > +#include <linux/delay.h>
> > #include <linux/sizes.h>
> > #include <linux/mutex.h>
> > #include <linux/list.h>
> > @@ -35,6 +37,20 @@
> > /* CXL 2.0 - 8.2.8.4 */
> > #define CXL_MAILBOX_TIMEOUT_MS (2 * HZ)
> >
> > +/*
> > + * CXL 2.0 ECN "Add Mailbox Ready Time" defines a capability field to
> > + * dictate how long to wait for the mailbox to become ready. The new
> > + * field allows the device to tell software the amount of time to wait
> > + * before mailbox ready. This field per the spec theoretically allows
> > + * for up to 255 seconds. 255 seconds is unreasonably long, its longer
> > + * than the maximum SATA port link recovery wait. Default to 60 seconds
> > + * until someone builds a CXL device that needs more time in practice.
> > + */
> > +static unsigned short mbox_ready_timeout = 60;
> > +module_param(mbox_ready_timeout, ushort, 0600);
>
> Any reason not to make it 0644?
>
Are there any tooling scenarios where this information is usable by non-root?
> > +MODULE_PARM_DESC(mbox_ready_timeout,
> > + "seconds to wait for mailbox ready status");
> > +
> > static int cxl_pci_mbox_wait_for_doorbell(struct cxl_dev_state *cxlds)
> > {
> > const unsigned long start = jiffies;
> > @@ -281,6 +297,25 @@ static int cxl_pci_mbox_send(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *c
> > static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds)
> > {
> > const int cap = readl(cxlds->regs.mbox + CXLDEV_MBOX_CAPS_OFFSET);
> > + unsigned long timeout;
> > + u64 md_status;
> > +
> > + timeout = jiffies + mbox_ready_timeout * HZ;
> > + do {
> > + md_status = readq(cxlds->regs.memdev + CXLMDEV_STATUS_OFFSET);
> > + if (md_status & CXLMDEV_MBOX_IF_READY)
> > + break;
> > + if (msleep_interruptible(100))
> > + break;
> > + } while (!time_after(jiffies, timeout));
>
> Just pointing out the [probably] obvious. If the user specifies a zero second
> timeout, the code will still wait 100ms.
Sure, is that going to be a problem in practice? I expect the
overwhelming common case is that the mailbox is already ready by this
point, so it's a zero-wait.
>
> > +
> > + if (!(md_status & CXLMDEV_MBOX_IF_READY)) {
> > + dev_err(cxlds->dev,
> > + "timeout awaiting mailbox ready, device state:%s%s\n",
> > + md_status & CXLMDEV_DEV_FATAL ? " fatal" : "",
> > + md_status & CXLMDEV_FW_HALT ? " firmware-halt" : "");
> > + return -EIO;
> > + }
> >
> > cxlds->mbox_send = cxl_pci_mbox_send;
> > cxlds->payload_size =
> >
next prev parent reply other threads:[~2022-01-31 23:11 UTC|newest]
Thread overview: 163+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-01-24 0:28 [PATCH v3 00/40] CXL.mem Topology Discovery and Hotplug Support Dan Williams
2022-01-24 0:28 ` [PATCH v3 01/40] cxl: Rename CXL_MEM to CXL_PCI Dan Williams
2022-01-24 0:28 ` [PATCH v3 02/40] cxl/pci: Implement Interface Ready Timeout Dan Williams
2022-01-31 22:21 ` Ben Widawsky
2022-01-31 23:11 ` Dan Williams [this message]
2022-01-31 23:25 ` Ben Widawsky
2022-01-31 23:47 ` Dan Williams
2022-01-31 23:51 ` [PATCH v4 " Dan Williams
2022-01-24 0:28 ` [PATCH v3 03/40] cxl/pci: Defer mailbox status checks to command timeouts Dan Williams
2022-01-31 22:28 ` Ben Widawsky
2022-01-24 0:29 ` [PATCH v3 04/40] cxl: Flesh out register names Dan Williams
2022-01-24 0:29 ` [PATCH v3 05/40] cxl/pci: Add new DVSEC definitions Dan Williams
2022-01-24 0:29 ` [PATCH v3 06/40] cxl/acpi: Map component registers for Root Ports Dan Williams
2022-01-24 0:29 ` [PATCH v3 07/40] cxl: Introduce module_cxl_driver Dan Williams
2022-01-24 0:29 ` [PATCH v3 08/40] cxl/core/port: Rename bus.c to port.c Dan Williams
2022-01-31 22:34 ` Ben Widawsky
2022-01-24 0:29 ` [PATCH v3 09/40] cxl/decoder: Hide physical address information from non-root Dan Williams
2022-01-31 14:14 ` Jonathan Cameron
2022-01-31 22:34 ` Ben Widawsky
2022-01-24 0:29 ` [PATCH v3 10/40] cxl/core: Convert decoder range to resource Dan Williams
2022-01-24 0:29 ` [PATCH v3 11/40] cxl/core/port: Clarify decoder creation Dan Williams
2022-01-31 14:46 ` Jonathan Cameron
2022-01-31 21:17 ` Dan Williams
2022-01-31 21:33 ` [PATCH v4 " Dan Williams
2022-02-01 10:49 ` Jonathan Cameron
2022-01-24 0:29 ` [PATCH v3 12/40] cxl/core: Fix cxl_probe_component_regs() error message Dan Williams
2022-01-31 14:53 ` Jonathan Cameron
2022-01-31 22:29 ` Dan Williams
2022-01-31 22:39 ` Ben Widawsky
2022-01-24 0:29 ` [PATCH v3 13/40] cxl/core/port: Make passthrough decoder init implicit Dan Williams
2022-01-31 14:56 ` Jonathan Cameron
2022-01-24 0:29 ` [PATCH v3 14/40] cxl/core: Track port depth Dan Williams
2022-01-31 14:57 ` Jonathan Cameron
2022-01-24 0:29 ` [PATCH v3 15/40] cxl: Prove CXL locking Dan Williams
2022-01-31 15:48 ` Jonathan Cameron
2022-01-31 19:43 ` Dan Williams
2022-01-31 19:50 ` [PATCH v4 " Dan Williams
2022-01-31 23:23 ` Ben Widawsky
2022-01-24 0:30 ` [PATCH v3 16/40] cxl/core/port: Use dedicated lock for decoder target list Dan Williams
2022-01-24 0:30 ` [PATCH v3 17/40] cxl/port: Introduce cxl_port_to_pci_bus() Dan Williams
2022-01-31 16:04 ` Jonathan Cameron
2022-01-31 16:44 ` [PATCH v4 " Dan Williams
2022-01-31 23:41 ` Ben Widawsky
2022-01-24 0:30 ` [PATCH v3 18/40] cxl/pmem: Introduce a find_cxl_root() helper Dan Williams
2022-01-26 18:55 ` [PATCH v4 " Dan Williams
2022-01-26 23:59 ` [PATCH v5 " Dan Williams
2022-01-31 16:18 ` Jonathan Cameron
2022-02-01 0:22 ` Dan Williams
2022-02-01 10:58 ` Jonathan Cameron
2022-02-01 0:34 ` [PATCH v6 " Dan Williams
2022-02-01 10:59 ` Jonathan Cameron
2022-01-24 0:30 ` [PATCH v3 19/40] cxl/port: Up-level cxl_add_dport() locking requirements to the caller Dan Williams
2022-01-31 16:20 ` Jonathan Cameron
2022-01-31 23:47 ` Ben Widawsky
2022-02-01 0:43 ` Dan Williams
2022-02-01 1:07 ` [PATCH v4 " Dan Williams
2022-02-01 11:00 ` Jonathan Cameron
2022-01-24 0:30 ` [PATCH v3 20/40] cxl/pci: Rename pci.h to cxlpci.h Dan Williams
2022-01-31 16:22 ` Jonathan Cameron
2022-02-01 0:00 ` Dan Williams
2022-01-31 23:48 ` Ben Widawsky
2022-01-24 0:30 ` [PATCH v3 21/40] cxl/core: Generalize dport enumeration in the core Dan Williams
2022-01-31 17:02 ` Jonathan Cameron
2022-02-01 1:58 ` Dan Williams
2022-02-01 2:10 ` [PATCH v4 " Dan Williams
2022-02-01 11:03 ` Jonathan Cameron
2022-01-24 0:30 ` [PATCH v3 22/40] cxl/core/hdm: Add CXL standard decoder enumeration to " Dan Williams
2022-01-26 3:09 ` [PATCH v4 " Dan Williams
2022-01-31 14:26 ` Jonathan Cameron
2022-01-31 17:51 ` Jonathan Cameron
2022-02-01 5:10 ` Dan Williams
2022-02-01 20:24 ` [PATCH v5 " Dan Williams
2022-02-02 9:31 ` Jonathan Cameron
2022-02-01 0:24 ` [PATCH v3 " Ben Widawsky
2022-02-01 4:58 ` Dan Williams
2022-01-24 0:30 ` [PATCH v3 23/40] cxl/core: Emit modalias for CXL devices Dan Williams
2022-01-31 17:57 ` Jonathan Cameron
2022-02-01 15:11 ` Ben Widawsky
2022-01-24 0:30 ` [PATCH v3 24/40] cxl/port: Add a driver for 'struct cxl_port' objects Dan Williams
2022-01-26 20:16 ` [PATCH v4 " Dan Williams
2022-01-31 18:11 ` Jonathan Cameron
2022-02-01 20:43 ` Dan Williams
2022-02-02 9:33 ` Jonathan Cameron
2022-02-01 21:07 ` [PATCH v5 " Dan Williams
2022-01-24 0:30 ` [PATCH v3 25/40] cxl/core/port: Remove @host argument for dport + decoder enumeration Dan Williams
2022-01-31 14:32 ` Jonathan Cameron
2022-01-31 18:14 ` Jonathan Cameron
2022-02-01 15:17 ` Ben Widawsky
2022-02-01 21:09 ` Dan Williams
2022-02-01 21:23 ` [PATCH v4 " Dan Williams
2022-01-24 0:30 ` [PATCH v3 26/40] cxl/pci: Store component register base in cxlds Dan Williams
2022-01-31 18:15 ` Jonathan Cameron
2022-02-01 21:28 ` [PATCH v4 " Dan Williams
2022-01-24 0:31 ` [PATCH v3 27/40] cxl/pci: Cache device DVSEC offset Dan Williams
2022-01-31 18:19 ` Jonathan Cameron
2022-02-01 15:24 ` Ben Widawsky
2022-02-01 21:41 ` Dan Williams
2022-02-01 22:11 ` Ben Widawsky
2022-02-01 22:15 ` Dan Williams
2022-02-01 22:20 ` Ben Widawsky
2022-02-01 22:24 ` Dan Williams
2022-02-02 9:36 ` Jonathan Cameron
2022-02-01 22:06 ` [PATCH v4 " Dan Williams
2022-02-02 9:36 ` Jonathan Cameron
2022-01-24 0:31 ` [PATCH v3 28/40] cxl/pci: Retrieve CXL DVSEC memory info Dan Williams
2022-01-31 18:25 ` Jonathan Cameron
2022-02-01 22:52 ` Dan Williams
2022-02-01 23:48 ` [PATCH v4 " Dan Williams
2022-02-02 9:39 ` Jonathan Cameron
2022-01-24 0:31 ` [PATCH v3 29/40] cxl/pci: Implement wait for media active Dan Williams
2022-01-31 18:29 ` Jonathan Cameron
2022-02-01 23:56 ` Dan Williams
2022-01-24 0:31 ` [PATCH v3 30/40] cxl/pci: Emit device serial number Dan Williams
2022-01-31 18:33 ` Jonathan Cameron
2022-01-31 21:43 ` Dan Williams
2022-01-31 21:56 ` [PATCH v4 " Dan Williams
2022-01-24 0:31 ` [PATCH v3 31/40] cxl/memdev: Add numa_node attribute Dan Williams
2022-01-31 18:41 ` Jonathan Cameron
2022-02-01 23:57 ` Dan Williams
2022-02-02 9:44 ` Jonathan Cameron
2022-02-02 15:44 ` Dan Williams
2022-02-03 9:41 ` Jonathan Cameron
2022-02-03 16:59 ` Dan Williams
2022-02-03 18:05 ` Jonathan Cameron
2022-02-04 4:25 ` Dan Williams
2022-02-01 15:31 ` Ben Widawsky
2022-02-01 15:49 ` Jonathan Cameron
2022-02-01 16:35 ` Ben Widawsky
2022-02-01 17:38 ` Jonathan Cameron
2022-02-01 23:59 ` Dan Williams
2022-02-02 1:18 ` Dan Williams
2022-01-24 0:31 ` [PATCH v3 32/40] cxl/core/port: Add switch port enumeration Dan Williams
2022-02-01 12:13 ` Jonathan Cameron
2022-02-02 5:26 ` Dan Williams
2022-02-01 17:37 ` Ben Widawsky
2022-02-02 6:03 ` Dan Williams
2022-02-02 17:07 ` [PATCH v4 " Dan Williams
2022-02-03 9:55 ` Jonathan Cameron
2022-02-04 15:08 ` [PATCH v5 " Dan Williams
2022-01-24 0:31 ` [PATCH v3 33/40] cxl/mem: Add the cxl_mem driver Dan Williams
2022-01-26 3:16 ` [PATCH v4 " Dan Williams
2022-02-01 12:45 ` Jonathan Cameron
2022-02-01 17:44 ` Ben Widawsky
2022-02-03 2:49 ` Dan Williams
2022-02-03 9:59 ` Jonathan Cameron
2022-02-04 14:54 ` Dan Williams
2022-02-03 3:56 ` [PATCH v5 " Dan Williams
2022-02-03 12:07 ` Jonathan Cameron
2022-02-04 15:18 ` [PATCH v6 " Dan Williams
2022-01-24 0:31 ` [PATCH v3 34/40] cxl/core: Move target_list out of base decoder attributes Dan Williams
2022-01-31 18:45 ` Jonathan Cameron
2022-02-01 17:45 ` Ben Widawsky
2022-01-24 0:31 ` [PATCH v3 35/40] cxl/core/port: Add endpoint decoders Dan Williams
2022-02-01 12:47 ` Jonathan Cameron
2022-02-03 4:02 ` [PATCH v4 " Dan Williams
2022-02-14 17:45 ` Jonathan Cameron
2022-02-14 19:14 ` Dan Williams
2022-01-24 0:31 ` [PATCH v3 36/40] tools/testing/cxl: Mock dvsec_ranges() Dan Williams
2022-01-24 0:31 ` [PATCH v3 37/40] tools/testing/cxl: Fix root port to host bridge assignment Dan Williams
2022-01-24 0:32 ` [PATCH v3 38/40] tools/testing/cxl: Mock one level of switches Dan Williams
2022-01-24 0:32 ` [PATCH v3 39/40] tools/testing/cxl: Enumerate mock decoders Dan Williams
2022-01-24 0:32 ` [PATCH v3 40/40] tools/testing/cxl: Add a physical_node link Dan Williams
2022-02-01 12:53 ` Jonathan Cameron
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAPcyv4gGMsUbJ=Cu8U0B2Znz6+jE6PcDCALNfxJzvAHZZccVvQ@mail.gmail.com' \
--to=dan.j.williams@intel.com \
--cc=Jonathan.Cameron@huawei.com \
--cc=ben.widawsky@intel.com \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=nvdimm@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).