linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Bjorn Helgaas <helgaas@kernel.org>
To: Kar Hin Ong <kar.hin.ong@ni.com>
Cc: linux-rt-users@vger.kernel.org, linux-kernel@vger.kernel.org,
	linux-x86_64@vger.kernel.org, linux-pci@vger.kernel.org,
	Thomas Gleixner <tglx@linutronix.de>
Subject: Re: "oneshot" interrupt causes another interrupt to be fired erroneously in Haswell system
Date: Thu, 31 Oct 2019 18:05:32 -0500	[thread overview]
Message-ID: <20191031230532.GA170712@google.com> (raw)
In-Reply-To: <MN2PR04MB625541BF4ADC84690B5C45E9C3630@MN2PR04MB6255.namprd04.prod.outlook.com>

[+cc Thomas, IRQ maintainer]

On Thu, Oct 31, 2019 at 03:53:50AM +0000, Kar Hin Ong wrote:
> Hi,
> 
> I've an Intel Haswell system running Linux kernel v4.14 with
> preempt_rt patch. The system contain 2 IOAPICs: IOAPIC 1 is on the
> PCH where IOAPIC 2 is on the CPU.
> 
> I observed that whenever a PCI device is firing interrupt (INTx) to
> Pin 20 of IOAPIC 2 (GSI 44); the kernel will receives 2 interrupts: 
>    1. Interrupt from Pin 20 of IOAPIC 2  -> Expected
>    2. Interrupt from Pin 19 of IOAPIC 1  -> UNEXPECTED, erroneously
>       triggered
> 
> The unexpected interrupt is unhandled eventually. When this scenario
> happen more than 99,000 times, kernel disables the interrupt line
> (Pin 19 of IOAPIC 1) and causing device that has requested it become
> malfunction.
> 
> I managed to also reproduced this issue on RHEL 8 and Ubuntu 19-04
> (without preempt_rt patch) after added "threadirqs" to the kernel
> command line.
> 
> After digging further, I noticed that the said issue is happened
> whenever an interrupt pin on IOAPIC 2 is masked:
>  - Masking Pin 20 of IOAPIC 2 triggers Pin 19 of IOAPIC 1  
>  - Masking Pin 22 of IOAPIC 2 triggers Pin 18 of IOAPIC 1  
> 
> I also noticed that kernel will explicitly mask a specific interrupt
> pin before execute its handler, if the interrupt is configured as
> "oneshot" (i.e. threaded). See
> https://elixir.bootlin.com/linux/v4.14/source/kernel/irq/chip.c#L695
> This explained why it only happened on RTOS and Desktop Linux with
> "threadirqs" flag, because these configurations force the interrupt
> handler to be threaded.
> 
> From Intel Xeon Processor E5/E7 v3 Product Family External Design
> Specification (EDS), Volume One: Architecture, section 13.1 (Legacy
> PCI Interrupt Handling), it mention: "If the I/OxAPIC entry is
> masked (via the 'mask' bit in the corresponding Redirection Table
> Entry), then the corresponding PCI Express interrupt(s) is forwarded
> to the legacy PCH"
> 
> My interpretation is: when kernel receive a "oneshot" interrupt, it
> mask the line before start handling it (or sending the eoi signal).
> At this moment, if the interrupt line is still asserting, then the
> interrupt signal will be routed to the IOAPIC in PCH, and hence
> causing another interrupt to be fired erroneously.  
> 
> I would like to understand if my interpretation is make sense. If
> yes, should the "oneshot" algorithm need to be updated to support
> Haswell system?

Just to make sure this hasn't already been fixed, can you reproduce
the problem on a current kernel, e.g., v5.3 or v5.4-rc5?

Bjorn

  reply	other threads:[~2019-10-31 23:05 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-10-31  3:53 "oneshot" interrupt causes another interrupt to be fired erroneously in Haswell system Kar Hin Ong
2019-10-31 23:05 ` Bjorn Helgaas [this message]
2019-11-01  7:46   ` Kar Hin Ong
2019-11-04 23:41   ` Thomas Gleixner
2019-11-07  9:38     ` Kar Hin Ong
2019-11-21 11:22       ` Kar Hin Ong
2020-01-15 23:05       ` Thomas Gleixner
2020-01-16  7:32         ` Kar Hin Ong
2020-01-16 10:01           ` Thomas Gleixner
2020-01-23  1:37             ` Sean V Kelley
2020-01-23 13:08               ` Thomas Gleixner
2020-02-09 15:37                 ` Thomas Gleixner
2020-02-10 16:32                   ` Sean V Kelley

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191031230532.GA170712@google.com \
    --to=helgaas@kernel.org \
    --cc=kar.hin.ong@ni.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=linux-rt-users@vger.kernel.org \
    --cc=linux-x86_64@vger.kernel.org \
    --cc=tglx@linutronix.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).