From: Stephen Boyd <swboyd@chromium.org>
To: Douglas Anderson <dianders@chromium.org>,
Mark Brown <broonie@kernel.org>
Cc: Alok Chauhan <alokc@codeaurora.org>,
skakit@codeaurora.org, Douglas Anderson <dianders@chromium.org>,
Andy Gross <agross@kernel.org>,
Bjorn Andersson <bjorn.andersson@linaro.org>,
Dilip Kota <dkota@codeaurora.org>,
linux-arm-msm@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-spi@vger.kernel.org
Subject: Re: [PATCH v3 2/5] spi: spi-geni-qcom: Mo' betta locking
Date: Wed, 17 Jun 2020 13:53:08 -0700 [thread overview]
Message-ID: <159242718864.62212.18160698526818943096@swboyd.mtv.corp.google.com> (raw)
In-Reply-To: <20200616034044.v3.2.I752ebdcfd5e8bf0de06d66e767b8974932b3620e@changeid>
Quoting Douglas Anderson (2020-06-16 03:40:47)
> If you added a bit of a delay (like a trace_printk) into the ISR for
> the spi-geni-qcom driver, you would suddenly start seeing some errors
> spit out. The problem was that, though the ISR itself held a lock,
> other parts of the driver didn't always grab the lock.
>
> One example race was this:
> a) Driver queues off a command to set a Chip Select (CS).
> b) ISR fires indicating the CS is done.
> c) Done bit is set, so we complete().
> d) Second CPU gallops off and starts a transfer.
> e) Second CPU starts messing with hardware / software state (not under
> spinlock).
> f) ISR now does things like set "mas->cur_mcmd" to CMD_NONE, prints
> errors if "tx_rem_bytes" / "rx_rem_bytes" have been set, and also
> Acks all interrupts it handled.
Can we get a CPU0/CPU1 diagram here? At point e) I got sort of lost. And
maybe it's not even a dual CPU problem? i.e. it can happen on one CPU?
CPU0
----
a) spi_geni_set_cs()
mas->cur_mcmd = CMD_CS
wait_for_completion_timeout(&xfer_done)
b) <INTERRUPT>
geni_spi_isr()
c) complete(&xfer_done);
<END INTERRUPT>
pm_runtime_put(mas->dev);
d) galloping?
I got lost... Sorry!
>
> Let's fix this. Before we start messing with hardware, we'll grab the
> lock to make sure that the IRQ handler from some previous command has
> really finished. We don't need to hold the lock unless we're in a
> state where more interrupts can come in, but we at least need to make
> sure the previous IRQ is done. This lock is used exclusively to
> prevent the IRQ handler and non-IRQ from stomping on each other. The
> SPI core handles all other mutual exclusion.
Ok maybe the problem is the completion at c) never happens until the
wait_for_completion_timeout() times out?
>
> As part of this, we change the way that the IRQ handler detects
> spurious interrupts. Previously we checked for our state variable
> being set to IRQ_NONE, but that was done outside the spinlock. We
> could move it into the spinlock, but instead let's just change it to
> look for the lack of any IRQ status bits being set. This can be done
> outside the lock--the hardware certainly isn't grabbing or looking at
> the spinlock when it updates its status register.
>
> It's possible that this will fix real (but very rare) errors seen in
> the field that look like:
> irq ...: nobody cared (try booting with the "irqpoll" option)
>
> NOTE: an alternate strategy considered here was to always make the
> complete() / spi_finalize_current_transfer() the very last thing in
> our IRQ handler. With such a change you could consider that we could
> be "lockless". In that case, though, we'd have to be very careful w/
> memory barriers so we made sure we didn't have any bugs with weakly
> ordered memory. Using spinlocks makes the driver much easier to
> understand.
>
> Fixes: 561de45f72bd ("spi: spi-geni-qcom: Add SPI driver support for GENI based QUP")
> Signed-off-by: Douglas Anderson <dianders@chromium.org>
> ---
>
> Changes in v3:
> - Split out some lock cleanup to previous patch.
> - Don't need to read IRQ status register inside spinlock.
> - Don't check for state CMD_NONE; later patch is removing state var.
> - Don't hold the lock for all of setup_fifo_xfer().
> - Comment about why it's safe to Ack interrupts at the end.
> - Subject/desc changed since race is definitely there.
>
> Changes in v2:
> - Detect true spurious interrupt.
> - Still return IRQ_NONE for state machine mismatch, but print warn.
>
> drivers/spi/spi-geni-qcom.c | 45 ++++++++++++++++++++++++++++++++++---
> 1 file changed, 42 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/spi/spi-geni-qcom.c b/drivers/spi/spi-geni-qcom.c
> index c7d2c7e45b3f..e0f0e5c241f3 100644
> --- a/drivers/spi/spi-geni-qcom.c
> +++ b/drivers/spi/spi-geni-qcom.c
> @@ -367,6 +384,12 @@ static void setup_fifo_xfer(struct spi_transfer *xfer,
> }
> writel(spi_tx_cfg, se->base + SE_SPI_TRANS_CFG);
> mas->cur_mcmd = CMD_XFER;
> +
> + /*
> + * Lock around right before we start the transfer since our
> + * interrupt controller could come in at any time now.
drop 'controller'?
> + */
> + spin_lock_irq(&mas->lock);
> geni_se_setup_m_cmd(se, m_cmd, FRAGMENTATION);
>
> /*
next prev parent reply other threads:[~2020-06-17 20:53 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-06-16 10:40 [PATCH v3 0/5] spi: spi-geni-qcom: Fixes / perf improvements Douglas Anderson
2020-06-16 10:40 ` [PATCH v3 1/5] spi: spi-geni-qcom: No need for irqsave variant of spinlock calls Douglas Anderson
2020-06-17 20:24 ` Stephen Boyd
2020-06-16 10:40 ` [PATCH v3 2/5] spi: spi-geni-qcom: Mo' betta locking Douglas Anderson
2020-06-17 20:53 ` Stephen Boyd [this message]
2020-06-17 21:19 ` Doug Anderson
2020-06-18 0:47 ` Stephen Boyd
2020-06-16 10:40 ` [PATCH v3 3/5] spi: spi-geni-qcom: Check for error IRQs Douglas Anderson
2020-06-17 20:54 ` Stephen Boyd
2020-06-16 10:40 ` [PATCH v3 4/5] spi: spi-geni-qcom: Actually use our FIFO Douglas Anderson
2020-06-17 20:56 ` Stephen Boyd
2020-06-16 10:40 ` [PATCH v3 5/5] spi: spi-geni-qcom: Don't keep a local state variable Douglas Anderson
2020-06-17 21:16 ` Stephen Boyd
2020-06-18 15:05 ` Doug Anderson
2020-06-18 16:48 ` [PATCH v3 0/5] spi: spi-geni-qcom: Fixes / perf improvements Mark Brown
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=159242718864.62212.18160698526818943096@swboyd.mtv.corp.google.com \
--to=swboyd@chromium.org \
--cc=agross@kernel.org \
--cc=alokc@codeaurora.org \
--cc=bjorn.andersson@linaro.org \
--cc=broonie@kernel.org \
--cc=dianders@chromium.org \
--cc=dkota@codeaurora.org \
--cc=linux-arm-msm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-spi@vger.kernel.org \
--cc=skakit@codeaurora.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).