From: Jerry Snitselaar <jsnitsel@redhat.com>
To: Jarkko Sakkinen <jarkko@kernel.org>
Cc: Jani Nikula <jani.nikula@linux.intel.com>,
linux-kernel@vger.kernel.org, linux-integrity@vger.kernel.org,
Jason Gunthorpe <jgg@ziepe.ca>, "Peter Huewe" <peterhuewe@gmx.de>,
James Bottomley <James.Bottomley@hansenpartnership.com>,
Matthew Garrett <mjg59@google.com>,
Hans de Goede <hdegoede@redhat.com>
Subject: Re: [PATCH v2] tpm_tis: Disable interrupts if interrupt storm detected
Date: Thu, 03 Dec 2020 13:14:34 -0700 [thread overview]
Message-ID: <87ft4mpryt.fsf@redhat.com> (raw)
In-Reply-To: <87lfefe7vm.fsf@redhat.com>
Jerry Snitselaar @ 2020-12-02 23:11 MST:
> Jerry Snitselaar @ 2020-12-02 17:02 MST:
>
>> Jarkko Sakkinen @ 2020-12-02 09:49 MST:
>>
>>> On Tue, Dec 01, 2020 at 12:59:23PM -0700, Jerry Snitselaar wrote:
>>>>
>>>> Jerry Snitselaar @ 2020-11-30 20:26 MST:
>>>>
>>>> > Jerry Snitselaar @ 2020-11-30 19:58 MST:
>>>> >
>>>> >> When enabling the interrupt code for the tpm_tis driver we have
>>>> >> noticed some systems have a bios issue causing an interrupt storm to
>>>> >> occur. The issue isn't limited to a single tpm or system manufacturer
>>>> >> so keeping a denylist of systems with the issue isn't optimal. Instead
>>>> >> try to detect the problem occurring, disable interrupts, and revert to
>>>> >> polling when it happens.
>>>> >>
>>>> >> Cc: Jarkko Sakkinen <jarkko@kernel.org>
>>>> >> Cc: Jason Gunthorpe <jgg@ziepe.ca>
>>>> >> Cc: Peter Huewe <peterhuewe@gmx.de>
>>>> >> Cc: James Bottomley <James.Bottomley@HansenPartnership.com>
>>>> >> Cc: Matthew Garrett <mjg59@google.com>
>>>> >> Cc: Hans de Goede <hdegoede@redhat.com>
>>>> >> Signed-off-by: Jerry Snitselaar <jsnitsel@redhat.com>
>>>> >> ---
>>>> >> v2: drop tpm_tis specific workqueue and use just system_wq
>>>> >>
>>>> >> drivers/char/tpm/tpm_tis_core.c | 27 +++++++++++++++++++++++++++
>>>> >> drivers/char/tpm/tpm_tis_core.h | 2 ++
>>>> >> 2 files changed, 29 insertions(+)
>>>> >>
>>>> >> diff --git a/drivers/char/tpm/tpm_tis_core.c b/drivers/char/tpm/tpm_tis_core.c
>>>> >> index 23b60583928b..72cc8a5a152c 100644
>>>> >> --- a/drivers/char/tpm/tpm_tis_core.c
>>>> >> +++ b/drivers/char/tpm/tpm_tis_core.c
>>>> >> @@ -24,6 +24,8 @@
>>>> >> #include <linux/wait.h>
>>>> >> #include <linux/acpi.h>
>>>> >> #include <linux/freezer.h>
>>>> >> +#include <linux/workqueue.h>
>>>> >> +#include <linux/kernel_stat.h>
>>>> >> #include "tpm.h"
>>>> >> #include "tpm_tis_core.h"
>>>> >>
>>>> >> @@ -745,9 +747,23 @@ static irqreturn_t tis_int_handler(int dummy, void *dev_id)
>>>> >> {
>>>> >> struct tpm_chip *chip = dev_id;
>>>> >> struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev);
>>>> >> + static bool check_storm = true;
>>>> >> + static unsigned int check_start;
>>>> >> u32 interrupt;
>>>> >> int i, rc;
>>>> >>
>>>> >> + if (unlikely(check_storm)) {
>>>> >> + if (!check_start) {
>>>> >> + check_start = jiffies_to_msecs(jiffies);
>>>> >> + } else if ((kstat_irqs(priv->irq) > 1000) &&
>>>> >> + (jiffies_to_msecs(jiffies) - check_start < 500)) {
>>>> >> + check_storm = false;
>>>> >> + schedule_work(&priv->storm_work);
>>>> >> + } else if (jiffies_to_msecs(jiffies) - check_start >= 500) {
>>>> >> + check_storm = false;
>>>> >> + }
>>>> >> + }
>>>> >> +
>>>> >> rc = tpm_tis_read32(priv, TPM_INT_STATUS(priv->locality), &interrupt);
>>>> >> if (rc < 0)
>>>> >> return IRQ_NONE;
>>>> >> @@ -987,6 +1003,14 @@ static const struct tpm_class_ops tpm_tis = {
>>>> >> .clk_enable = tpm_tis_clkrun_enable,
>>>> >> };
>>>> >>
>>>> >> +static void tpm_tis_storm_work(struct work_struct *work)
>>>> >> +{
>>>> >> + struct tpm_tis_data *priv = container_of(work, struct tpm_tis_data, storm_work);
>>>> >> +
>>>> >> + disable_interrupts(priv->chip);
>>>> >> + dev_warn(&priv->chip->dev, "Interrupt storm detected, using polling.\n");
>>>> >> +}
>>>> >> +
>>>> >> int tpm_tis_core_init(struct device *dev, struct tpm_tis_data *priv, int irq,
>>>> >> const struct tpm_tis_phy_ops *phy_ops,
>>>> >> acpi_handle acpi_dev_handle)
>>>> >> @@ -1003,6 +1027,9 @@ int tpm_tis_core_init(struct device *dev, struct tpm_tis_data *priv, int irq,
>>>> >> if (IS_ERR(chip))
>>>> >> return PTR_ERR(chip);
>>>> >>
>>>> >> + priv->chip = chip;
>>>> >> + INIT_WORK(&priv->storm_work, tpm_tis_storm_work);
>>>> >> +
>>>> >> #ifdef CONFIG_ACPI
>>>> >> chip->acpi_dev_handle = acpi_dev_handle;
>>>> >> #endif
>>>> >> diff --git a/drivers/char/tpm/tpm_tis_core.h b/drivers/char/tpm/tpm_tis_core.h
>>>> >> index edeb5dc61c95..5630f294dc0c 100644
>>>> >> --- a/drivers/char/tpm/tpm_tis_core.h
>>>> >> +++ b/drivers/char/tpm/tpm_tis_core.h
>>>> >> @@ -95,6 +95,8 @@ struct tpm_tis_data {
>>>> >> u16 clkrun_enabled;
>>>> >> wait_queue_head_t int_queue;
>>>> >> wait_queue_head_t read_queue;
>>>> >> + struct work_struct storm_work;
>>>> >> + struct tpm_chip *chip;
>>>> >> const struct tpm_tis_phy_ops *phy_ops;
>>>> >> unsigned short rng_quality;
>>>> >> };
>>>> >
>>>> > I've tested this with the Intel platform that has an Infineon chip that
>>>> > I found the other week. It works, but isn't the complete fix. With this
>>>> > on top of James' patchset I sometimes see the message "Lost Interrupt
>>>> > waiting for TPM stat", so I guess there needs to be a check in
>>>> > wait_for_tpm_stat and request_locality to see if interrupts were
>>>> > disabled when the wait_event_interruptible_timeout call times out.
>>>>
>>>> As kernel test robot pointed out. kstat_irqs isn't visible when tpm_tis
>>>> builds as a module. It looks like it is only called by kstat_irq_usrs,
>>>> and that is only by the fs/proc code. I have a patch to export it, but
>>>> the i915 driver open codes their own version instead of using it. Is
>>>> there any reason not to export it?
>>>
>>> If you add a patch that exports it, then for coherency it'd be better to
>>> also patch i915 driver. Jani?
>>>
>>> /Jarkko
>>
>> It looks like this might not solve all cases. I'm having Lenovo test
>> another build to make sure I gave them the right code, but they reported
>> with the L490 that the system hangs right when it is initializing
>> tpm_tis. I'm working on getting a build on the T490s I have to try there
>> as well. With the Intel system it spits out that it detects the
>> interrupt storm, and continues on its way.
>
> The interrupt storm detection code works on the T490s. I'm not sure what
> is going on with the L490. I will see if I can get access to one.
>
> Jerry
Lenovo verified that the L490 hangs.
next prev parent reply other threads:[~2020-12-03 20:16 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-30 23:23 [PATCH] tpm_tis: Disable interrupts if interrupt storm detected Jerry Snitselaar
2020-12-01 2:58 ` [PATCH v2] " Jerry Snitselaar
2020-12-01 3:26 ` Jerry Snitselaar
2020-12-01 19:59 ` Jerry Snitselaar
2020-12-02 16:49 ` Jarkko Sakkinen
2020-12-03 0:02 ` Jerry Snitselaar
2020-12-03 6:11 ` Jerry Snitselaar
2020-12-03 20:14 ` Jerry Snitselaar [this message]
2020-12-03 21:05 ` James Bottomley
2020-12-04 21:51 ` Jerry Snitselaar
2020-12-05 0:06 ` James Bottomley
2020-12-04 1:59 ` Jarkko Sakkinen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87ft4mpryt.fsf@redhat.com \
--to=jsnitsel@redhat.com \
--cc=James.Bottomley@hansenpartnership.com \
--cc=hdegoede@redhat.com \
--cc=jani.nikula@linux.intel.com \
--cc=jarkko@kernel.org \
--cc=jgg@ziepe.ca \
--cc=linux-integrity@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mjg59@google.com \
--cc=peterhuewe@gmx.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).