From: Jeremy Fitzhardinge <jeremy@goop.org> To: "Eric W. Biederman" <ebiederm@xmission.com> Cc: Ingo Molnar <mingo@redhat.com>, Thomas Gleixner <tglx@linutronix.de>, "H. Peter Anvin" <hpa@zytor.com>, the arch/x86 maintainers <x86@kernel.org>, Linux Kernel Mailing List <linux-kernel@vger.kernel.org>, Xen-devel <xen-devel@lists.xensource.com> Subject: Re: [PATCH RFC] x86/acpi: don't ignore I/O APICs just because there's no local APIC Date: Mon, 15 Jun 2009 13:49:48 -0700 [thread overview] Message-ID: <4A36B3EC.7010004@goop.org> (raw) In-Reply-To: <m1k53dbwo2.fsf@fess.ebiederm.org> On 06/15/09 03:47, Eric W. Biederman wrote: > For code reuse and maintainability that is a horrible separation of > responsibility. Things looks similar to the existing cases until you > get up close and you discover all of the fundamental assumptions are > different so none of the existing code actually works unmodified. > The I/O APIC code is used exactly as normal, routing from device->pin->vector; the whole interrupt emission path is unchanged. The local APIC code doesn't get used at all, because we have a different interrupt catcher operating at the irq_chip level. In terms of system architecture its a reasonable place to make the split; the local APICs and I/O APICs are distinct entities which communicate via fairly well-defined path. Xen puts the hypervisor/control domain split at the same place. This is mainly because Xen itself cares about managing CPUs (and memory), but doesn't really care about the rest of the system hardware much - it leaves that up to the control domain. > The only clean way I can see to handle this is to make xen dom0 it's own > weird separate subarch that does all of the table parsing of the > firmware tables in completely separate code. Then once we have something > that works factoring out the commonalities into a helper library for > better long term maintenance. > That seems like overkill. We can get things working under Xen with 3 changes: 1. make sure I/O APICs are discovered via ACPI properly (or MPTABLE if ACPI isn't present) 2. get Xen to allocate a vector and bind that vector to an event channel 3. make sure I/O APIC register writes get to the appropriate I/O APIC in hardware (the normal pin->vector routing) These points already have fairly well-defined interfaces; there are no subtle interactions with the core of the APIC code. This patch achieves the first of these, in a fairly minimal way. I'm still investigating better ways of achieving 2 & 3. > As it stands right now what Xen wants and what we need to do for normal > hardware are radically different, to the point of painful. Things like > irq migration, and cpu hotplug require completely different algorithms. > The control domain, being a virtual machine, has no access or visibility of physical CPUs in the system; all its CPUs are virtual (this is why a "local APIC" doesn't make much sense for it, since they're an inherent property of a physical CPU, and are not virtualized). The hypervisor is responsible for all management of physical CPUs, and is therefore responsible for physical-CPU things like hotplug and interrupt migration. The kernel doesn't need new algorithms to handle these because it simply doesn't know or care about them. As far as the kernel is concerned, the interrupts look like events on event channels, like IPIs, timers, etc, and can be handled accordingly. The irq_chip machinery is already in place for them. > I think Xen dom0 has picked the wrong abstraction for this one. There > seems to be no gain and a lot of pain asking the slave kernel to > program the ioapics for it, when Xen presents a wildly different > abstraction at the cpu level. > Well, the bulk of the code is already present. We avoid the local APIC part of the kernel completely, by installing a new irq_chip to handle incoming interrupts and deliver them into the core interrupt handling accordingly. The control domain patches simply add the ability to bind a hardware-originated interrupt to an event channel to be delivered via this mechanism. And, as Xen contains no device drivers or real hardware knowledge of busses, interrupt routing, etc, it falls to the control domain to work out those aspects. The I/O APIC side of the setup is the same as it would be in the native case (program a vector corresponding to a pin on an I/O APIC). > If what xen was provided looked like an ioapic semantically I would > suggest setting cpu_has_apic in a different fashion. cpu_has_apic has the specific meaning of "this CPU has a local APIC". It doesn't say anything about the presence or absence of I/O APICs; conflating the two notions doesn't seem like a good idea. I'm clearing cpu_has_apic to indicate this specific fact: the CPU has no usable local APIC, and there's no point pretending it does - but that doesn't mean the I/O APICs aren't functional. > We already have two local apic variants after all so a 3rd should not be too nasty. > We currently avoid any need to have, or pretend to have, a local APIC by taking control of the interrupt delivery subsystem at the irq_chip level. I don't think there's much to be gained by adding a Xen-specific lapic abstraction for this case. > Except the Xen appears to have totally moved the responsibility around > in ways that over constrain the problem by taking, making the > existing code useless. > I don't think that's true at at all. The split is along hardware lines, and so puts the same constraints on kernel development that the hardware does. > Please put the Xen dom0 insanity somewhere off in a corner where the rest > of x86 can ignore it. > Yep, trying to. J
WARNING: multiple messages have this Message-ID (diff)
From: Jeremy Fitzhardinge <jeremy@goop.org> To: "Eric W. Biederman" <ebiederm@xmission.com> Cc: Xen-devel <xen-devel@lists.xensource.com>, the arch/x86 maintainers <x86@kernel.org>, Linux Kernel Mailing List <linux-kernel@vger.kernel.org>, Ingo Molnar <mingo@redhat.com>, "H. Peter Anvin" <hpa@zytor.com>, Thomas Gleixner <tglx@linutronix.de> Subject: Re: [PATCH RFC] x86/acpi: don't ignore I/O APICs just because there's no local APIC Date: Mon, 15 Jun 2009 13:49:48 -0700 [thread overview] Message-ID: <4A36B3EC.7010004@goop.org> (raw) In-Reply-To: <m1k53dbwo2.fsf@fess.ebiederm.org> On 06/15/09 03:47, Eric W. Biederman wrote: > For code reuse and maintainability that is a horrible separation of > responsibility. Things looks similar to the existing cases until you > get up close and you discover all of the fundamental assumptions are > different so none of the existing code actually works unmodified. > The I/O APIC code is used exactly as normal, routing from device->pin->vector; the whole interrupt emission path is unchanged. The local APIC code doesn't get used at all, because we have a different interrupt catcher operating at the irq_chip level. In terms of system architecture its a reasonable place to make the split; the local APICs and I/O APICs are distinct entities which communicate via fairly well-defined path. Xen puts the hypervisor/control domain split at the same place. This is mainly because Xen itself cares about managing CPUs (and memory), but doesn't really care about the rest of the system hardware much - it leaves that up to the control domain. > The only clean way I can see to handle this is to make xen dom0 it's own > weird separate subarch that does all of the table parsing of the > firmware tables in completely separate code. Then once we have something > that works factoring out the commonalities into a helper library for > better long term maintenance. > That seems like overkill. We can get things working under Xen with 3 changes: 1. make sure I/O APICs are discovered via ACPI properly (or MPTABLE if ACPI isn't present) 2. get Xen to allocate a vector and bind that vector to an event channel 3. make sure I/O APIC register writes get to the appropriate I/O APIC in hardware (the normal pin->vector routing) These points already have fairly well-defined interfaces; there are no subtle interactions with the core of the APIC code. This patch achieves the first of these, in a fairly minimal way. I'm still investigating better ways of achieving 2 & 3. > As it stands right now what Xen wants and what we need to do for normal > hardware are radically different, to the point of painful. Things like > irq migration, and cpu hotplug require completely different algorithms. > The control domain, being a virtual machine, has no access or visibility of physical CPUs in the system; all its CPUs are virtual (this is why a "local APIC" doesn't make much sense for it, since they're an inherent property of a physical CPU, and are not virtualized). The hypervisor is responsible for all management of physical CPUs, and is therefore responsible for physical-CPU things like hotplug and interrupt migration. The kernel doesn't need new algorithms to handle these because it simply doesn't know or care about them. As far as the kernel is concerned, the interrupts look like events on event channels, like IPIs, timers, etc, and can be handled accordingly. The irq_chip machinery is already in place for them. > I think Xen dom0 has picked the wrong abstraction for this one. There > seems to be no gain and a lot of pain asking the slave kernel to > program the ioapics for it, when Xen presents a wildly different > abstraction at the cpu level. > Well, the bulk of the code is already present. We avoid the local APIC part of the kernel completely, by installing a new irq_chip to handle incoming interrupts and deliver them into the core interrupt handling accordingly. The control domain patches simply add the ability to bind a hardware-originated interrupt to an event channel to be delivered via this mechanism. And, as Xen contains no device drivers or real hardware knowledge of busses, interrupt routing, etc, it falls to the control domain to work out those aspects. The I/O APIC side of the setup is the same as it would be in the native case (program a vector corresponding to a pin on an I/O APIC). > If what xen was provided looked like an ioapic semantically I would > suggest setting cpu_has_apic in a different fashion. cpu_has_apic has the specific meaning of "this CPU has a local APIC". It doesn't say anything about the presence or absence of I/O APICs; conflating the two notions doesn't seem like a good idea. I'm clearing cpu_has_apic to indicate this specific fact: the CPU has no usable local APIC, and there's no point pretending it does - but that doesn't mean the I/O APICs aren't functional. > We already have two local apic variants after all so a 3rd should not be too nasty. > We currently avoid any need to have, or pretend to have, a local APIC by taking control of the interrupt delivery subsystem at the irq_chip level. I don't think there's much to be gained by adding a Xen-specific lapic abstraction for this case. > Except the Xen appears to have totally moved the responsibility around > in ways that over constrain the problem by taking, making the > existing code useless. > I don't think that's true at at all. The split is along hardware lines, and so puts the same constraints on kernel development that the hardware does. > Please put the Xen dom0 insanity somewhere off in a corner where the rest > of x86 can ignore it. > Yep, trying to. J
next prev parent reply other threads:[~2009-06-15 20:50 UTC|newest] Thread overview: 79+ messages / expand[flat|nested] mbox.gz Atom feed top 2009-06-12 18:22 [PATCH RFC] x86/acpi: don't ignore I/O APICs just because there's no local APIC Jeremy Fitzhardinge 2009-06-12 18:22 ` Jeremy Fitzhardinge 2009-06-12 18:28 ` Alan Cox 2009-06-12 18:28 ` Alan Cox 2009-06-12 18:33 ` Jeremy Fitzhardinge 2009-06-12 18:33 ` Jeremy Fitzhardinge 2009-06-12 20:11 ` Cyrill Gorcunov 2009-06-15 2:01 ` Jeremy Fitzhardinge 2009-06-12 20:35 ` Eric W. Biederman 2009-06-12 20:35 ` Eric W. Biederman 2009-06-15 2:06 ` Jeremy Fitzhardinge 2009-06-15 10:47 ` Eric W. Biederman 2009-06-15 10:47 ` Eric W. Biederman 2009-06-15 20:49 ` Jeremy Fitzhardinge [this message] 2009-06-15 20:49 ` Jeremy Fitzhardinge 2009-06-15 21:58 ` Eric W. Biederman 2009-06-15 21:58 ` Eric W. Biederman 2009-06-16 19:38 ` Jeremy Fitzhardinge 2009-06-16 19:38 ` Jeremy Fitzhardinge 2009-06-17 5:10 ` Eric W. Biederman 2009-06-17 5:10 ` Eric W. Biederman 2009-06-17 12:02 ` Eric W. Biederman 2009-06-17 12:02 ` Eric W. Biederman 2009-06-17 17:32 ` Jeremy Fitzhardinge 2009-06-17 17:32 ` Jeremy Fitzhardinge 2009-06-18 2:58 ` Eric W. Biederman 2009-06-18 2:58 ` Eric W. Biederman 2009-06-18 19:34 ` Jeremy Fitzhardinge 2009-06-18 19:34 ` Jeremy Fitzhardinge 2009-06-18 20:28 ` Eric W. Biederman 2009-06-18 21:09 ` Jeremy Fitzhardinge 2009-06-18 21:09 ` Jeremy Fitzhardinge 2009-06-19 1:38 ` Eric W. Biederman 2009-06-19 1:38 ` Eric W. Biederman 2009-06-19 3:10 ` [Xen-devel] " Jiang, Yunhong 2009-06-19 3:10 ` Jiang, Yunhong 2009-06-18 12:26 ` Eric W. Biederman 2009-06-15 10:51 ` Eric W. Biederman 2009-06-15 10:51 ` Eric W. Biederman 2009-06-18 16:08 ` Len Brown 2009-06-18 19:14 ` Jeremy Fitzhardinge 2009-06-18 19:14 ` Jeremy Fitzhardinge 2009-06-18 19:27 ` Eric W. Biederman 2009-06-18 19:48 ` Jeremy Fitzhardinge 2009-06-18 19:48 ` Jeremy Fitzhardinge 2009-06-18 20:39 ` Eric W. Biederman 2009-06-18 22:33 ` Jeremy Fitzhardinge 2009-06-18 22:33 ` Jeremy Fitzhardinge 2009-06-19 2:42 ` Eric W. Biederman 2009-06-19 2:42 ` Eric W. Biederman 2009-06-19 19:58 ` Jeremy Fitzhardinge 2009-06-19 19:58 ` Jeremy Fitzhardinge 2009-06-19 23:44 ` [Xen-devel] " Nakajima, Jun 2009-06-19 23:44 ` Nakajima, Jun 2009-06-20 7:39 ` [Xen-devel] " Keir Fraser 2009-06-20 7:39 ` Keir Fraser 2009-06-20 8:21 ` [Xen-devel] " Eric W. Biederman 2009-06-20 8:21 ` Eric W. Biederman 2009-06-20 8:57 ` [Xen-devel] " Tian, Kevin 2009-06-20 8:57 ` Tian, Kevin 2009-06-20 10:22 ` [Xen-devel] " Keir Fraser 2009-06-20 10:22 ` Keir Fraser 2009-06-20 8:18 ` [Xen-devel] " Eric W. Biederman 2009-06-20 8:18 ` Eric W. Biederman 2009-06-19 5:32 ` Yinghai Lu 2009-06-19 5:32 ` Yinghai Lu 2009-06-19 5:50 ` Eric W. Biederman 2009-06-19 5:50 ` Eric W. Biederman 2009-06-19 7:52 ` [Xen-devel] Re: [PATCH RFC] x86/acpi: don't ignore I/O APICs justbecause " Jan Beulich 2009-06-19 7:52 ` Jan Beulich 2009-06-19 8:16 ` [Xen-devel] " Eric W. Biederman 2009-06-19 8:16 ` Eric W. Biederman 2009-06-20 3:58 ` [Xen-devel] " Yinghai Lu 2009-06-20 3:58 ` Yinghai Lu 2009-06-20 5:40 ` [Xen-devel] " Eric W. Biederman 2009-06-20 5:40 ` Eric W. Biederman 2009-06-20 5:58 ` [Xen-devel] " Yinghai Lu 2009-06-20 5:58 ` Yinghai Lu 2009-06-18 22:51 ` [PATCH RFC] x86/acpi: don't ignore I/O APICs just because " Maciej W. Rozycki
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=4A36B3EC.7010004@goop.org \ --to=jeremy@goop.org \ --cc=ebiederm@xmission.com \ --cc=hpa@zytor.com \ --cc=linux-kernel@vger.kernel.org \ --cc=mingo@redhat.com \ --cc=tglx@linutronix.de \ --cc=x86@kernel.org \ --cc=xen-devel@lists.xensource.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.