Re: [PATCH] x86/hyperv: Enable 15-bit APIC ID if the hypervisor supports it

From: David Woodhouse <dwmw2@infradead.org>
To: Dexuan Cui <decui@microsoft.com>,
	tglx@linutronix.de, x86@kernel.org, mikelley@microsoft.com,
	linux-hyperv@vger.kernel.org
Cc: linux-kernel@vger.kernel.org, Tianyu.Lan@microsoft.com,
	vkuznets@redhat.com, kys@microsoft.com, haiyangz@microsoft.com,
	sthemmin@microsoft.com, wei.liu@kernel.org, mingo@redhat.com,
	bp@alien8.de, hpa@zytor.com
Subject: Re: [PATCH] x86/hyperv: Enable 15-bit APIC ID if the hypervisor supports it
Date: Tue, 03 Nov 2020 08:02:32 +0000	[thread overview]
Message-ID: <6b08181e5520355ac17da8ea376972ab82dca200.camel@infradead.org> (raw)
In-Reply-To: <20201103011136.59108-1-decui@microsoft.com>

[-- Attachment #1: Type: text/plain, Size: 3079 bytes --]

On Mon, 2020-11-02 at 17:11 -0800, Dexuan Cui wrote:
> When a Linux VM runs on Hyper-V, if the VM has CPUs with >255 APIC IDs,
> the CPUs can't be the destination of IOAPIC interrupts, because the
> IOAPIC RTE's Dest Field has only 8 bits. Currently the hackery driver
> drivers/iommu/hyperv-iommu.c is used to ensure IOAPIC interrupts are
> only routed to CPUs that don't have >255 APIC IDs. However, there is
> an issue with kdump, because the kdump kernel can run on any CPU, and
> hence IOAPIC interrupts can't work if the kdump kernel run on a CPU
> with a >255 APIC ID.
> 
> The kdump issue can be fixed by the Extended Dest ID, which is introduced
> recently by David Woodhouse (for IOAPIC, see the field virt_destid_8_14 in
> struct IO_APIC_route_entry). Of course, the Extended Dest ID needs the
> support of the underlying hypervisor. The latest Hyper-V has added the
> support recently: with this commit, on such a Hyper-V host, Linux VM
> does not use hyperv-iommu.c because hyperv_prepare_irq_remapping()
> returns -ENODEV; instead, Linux kernel's generic support of Extended Dest
> ID from David is used, meaning that Linux VM is able to support up to
> 32K CPUs, and IOAPIC interrupts can be routed to all the CPUs.
> 
> On an old Hyper-V host that doesn't support the Extended Dest ID, nothing
> changes with this commit: Linux VM is still able to bring up the CPUs with
> > 255 APIC IDs with the help of hyperv-iommu.c, but IOAPIC interrupts still
> 
> can not go to such CPUs, and the kdump kernel still can not work properly
> on such CPUs.
> 
> Signed-off-by: Dexuan Cui <decui@microsoft.com>

Acked-by: David Woodhouse <dwmw@amazon.co.uk>

> +/*
> + * If ms_hyperv_msi_ext_dest_id() returns true, hyperv_prepare_irq_remapping()
> + * returns -ENODEV and the Hyper-V IOMMU driver is not used; instead, the
> + * generic support of the 15-bit APIC ID is used: see __irq_msi_compose_msg().
> + *
> + * Note: For a VM on Hyper-V, no emulated legacy device supports PCI MSI/MSI-X,
> + * and PCI MSI/MSI-X only come from the assigned physical PCIe device, and the
> + * PCI MSI/MSI-X interrupts are handled by the pci-hyperv driver. Here despite
> + * the word "msi" in the name "msi_ext_dest_id", actually the callback only
> + * affects how IOAPIC interrupts are routed.
> + */

I named it like that on purpose to make the point that the I/OAPIC is
just a device for turning line interrupts into MSIs. Some VMMs, just
like real hardware, really do implement their I/OAPIC emulation that
way. It makes a lot of sense to do so if you support interrupt
remapping.

FWIW I might have phrased your last paragraph in that comment as

  Note: for a VM on Hyper-V, the I/OAPIC is the only device which
  (logically) generates MSIs directly to the system APIC irq domain.
  There is no HPET, and PCI MSI/MSI-X interrupts are remapped by the
  pci-hyperv host bridge.

But don't bother to change it; I think I've made my point quite well
enough with https://git.kernel.org/tip/tip/c/5d5a97133 :)

-- 
dwmw2

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5174 bytes --]