Re: What are the real ioapic rte programming constraints?

From: ebiederm@xmission.com (Eric W. Biederman)
To: "Natalie Protasevich" <protasnb@gmail.com>
Cc: "Zwane Mwaikambo" <zwane@infradead.org>,
	"Ashok Raj" <ashok.raj@intel.com>, "Ingo Molnar" <mingo@elte.hu>,
	"Andrew Morton" <akpm@osdl.org>,
	linux-kernel@vger.kernel.org, "Lu, Yinghai" <yinghai.lu@amd.com>,
	"Andi Kleen" <ak@suse.de>, "Coywolf Qi Hunt" <coywolf@lovecn.org>
Subject: Re: What are the real ioapic rte programming constraints?
Date: Sun, 11 Feb 2007 14:36:47 -0700	[thread overview]
Message-ID: <m1ejowz89c.fsf@ebiederm.dsl.xmission.com> (raw)
In-Reply-To: <32209efe0702111212j77f5011xe2430cb13c13686@mail.gmail.com> (Natalie Protasevich's message of "Sun, 11 Feb 2007 12:12:19 -0800")

"Natalie Protasevich" <protasnb@gmail.com> writes:

> On 2/11/07, Eric W. Biederman <ebiederm@xmission.com> wrote:
>
>     The code currently in the kernel does:
>
>     pending
>     mask
>     read io_apic
>     ack
>     reprogram vector and destination
>     unmask
>
>     So I guess it does retain the bug fix.
>
>     What I am looking at doing is:
>
>     mask
>     read io_apic
>     -- Past this point no more irqs are expected from the io_apic
>     -- Now I work to drain any inflight/pending instances of the irq
>     send ipi to all irq destinations cpus and wait for it to return
>     read lapic
>     disable local irqs
>     take irq lock
>     -- Now no more irqs are expected to arrive
>     reprogram vector and destination
>     enable local irqs
>     unmask
>
>     What I need to ensure is that I have a point where I will not receive any
>     new messages from an ioapic about a particular irq anymore.  Even if
>     everything is working perfectly setting the disable bit is not enough
>     because there could be an irq message in flight. So I need to give any
>     in flight irqs a chance to complete.
>
>
> It is probably safer to have "interlapping" time in having second rte
> programmed, then both willl have the same vector and receive the eoi. And if
> you have interrupts disabled for a little while before you dismantle the old
> entry it should insure that all the in-flight ones got services and acked...
> --Natalie

I am assuming you mean interrupts disabled at the ioapic.  If that
is true mostly I agree with your assertion.

If you look above the mask is where I disable that interrupt source on
the ioapic.  I then send an ipi to all of the other possible cpus and
read the local apic to ensure that the window is over in which I need
to handle interrupts at the old destination. 

If I don't know when the window ends I can never dismantle the data
structures for receiving the irq at the old destination.  Ouch!

If I know how long I have to wait until I can clean up the data
structure at the old destination I don't have to worry about
overlapping interrupts.

Therefore it doesn't by me anything but complexity to setup the data
structures on the new cpu before the window for receiving the irqs
has completed.

Make sense?

Eric