* [PATCH linux 1/8] x86/xen: update cpuid.h from Xen-4.7
2016-06-28 16:47 [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs Vitaly Kuznetsov
@ 2016-06-28 16:47 ` Vitaly Kuznetsov
2016-06-28 16:47 ` [PATCH linux 2/8] xen: introduce xen_vcpu_id mapping Vitaly Kuznetsov
` (7 subsequent siblings)
8 siblings, 0 replies; 26+ messages in thread
From: Vitaly Kuznetsov @ 2016-06-28 16:47 UTC (permalink / raw)
To: xen-devel
Cc: linux-kernel, x86, Stefano Stabellini, Boris Ostrovsky,
David Vrabel, Juergen Gross, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Julien Grall
Update cpuid.h header from xen hypervisor tree to get
XEN_HVM_CPUID_VCPU_ID_PRESENT definition.
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
arch/x86/include/asm/xen/cpuid.h | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/xen/cpuid.h b/arch/x86/include/asm/xen/cpuid.h
index 0d809e9..3bdd10d 100644
--- a/arch/x86/include/asm/xen/cpuid.h
+++ b/arch/x86/include/asm/xen/cpuid.h
@@ -76,15 +76,18 @@
/*
* Leaf 5 (0x40000x04)
* HVM-specific features
+ * EAX: Features
+ * EBX: vcpu id (iff EAX has XEN_HVM_CPUID_VCPU_ID_PRESENT flag)
*/
-/* EAX Features */
/* Virtualized APIC registers */
#define XEN_HVM_CPUID_APIC_ACCESS_VIRT (1u << 0)
/* Virtualized x2APIC accesses */
#define XEN_HVM_CPUID_X2APIC_VIRT (1u << 1)
/* Memory mapped from other domains has valid IOMMU entries */
#define XEN_HVM_CPUID_IOMMU_MAPPINGS (1u << 2)
+/* vcpu id is present in EBX */
+#define XEN_HVM_CPUID_VCPU_ID_PRESENT (1u << 3)
#define XEN_CPUID_MAX_NUM_LEAVES 4
--
2.5.5
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH linux 2/8] xen: introduce xen_vcpu_id mapping
2016-06-28 16:47 [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs Vitaly Kuznetsov
2016-06-28 16:47 ` [PATCH linux 1/8] x86/xen: update cpuid.h from Xen-4.7 Vitaly Kuznetsov
@ 2016-06-28 16:47 ` Vitaly Kuznetsov
2016-06-28 17:28 ` [Xen-devel] " Andrew Cooper
2016-06-28 16:47 ` [PATCH linux 3/8] x86/xen: use xen_vcpu_id mapping for HYPERVISOR_vcpu_op Vitaly Kuznetsov
` (6 subsequent siblings)
8 siblings, 1 reply; 26+ messages in thread
From: Vitaly Kuznetsov @ 2016-06-28 16:47 UTC (permalink / raw)
To: xen-devel
Cc: linux-kernel, x86, Stefano Stabellini, Boris Ostrovsky,
David Vrabel, Juergen Gross, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Julien Grall
It may happen that Xen's and Linux's ideas of vCPU id diverge. In
particular, when we crash on a secondary vCPU we may want to do kdump
and unlike plain kexec where we do migrate_to_reboot_cpu() we try booting
on the vCPU which crashed. This doesn't work very well for PVHVM guests as
we have a number of hypercalls where we pass vCPU id as a parameter. These
hypercalls either fail or do something unexpected. To solve the issue
introduce percpu xen_vcpu_id mapping. ARM and PV guests get direct mapping
for now. Boot CPU for PVHVM guest gets its id from CPUID. With secondary
CPUs it is a bit more trickier. Currently, we initialize IPI vectors
before these CPUs boot so we can't use CPUID. However, we know that
physical CPU id (vLAPIC id) is Xen's vCPU id * 2, we can piggyback on
that. Alternatively, we could have disabled all secondary CPUs once we
detect that Xen's and Linux's ideas of vCPU id diverged.
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
arch/arm/xen/enlighten.c | 10 ++++++++++
arch/x86/xen/enlighten.c | 18 +++++++++++++++++-
include/xen/xen-ops.h | 1 +
3 files changed, 28 insertions(+), 1 deletion(-)
diff --git a/arch/arm/xen/enlighten.c b/arch/arm/xen/enlighten.c
index 75cd734..ea99ca2 100644
--- a/arch/arm/xen/enlighten.c
+++ b/arch/arm/xen/enlighten.c
@@ -46,6 +46,10 @@ struct shared_info *HYPERVISOR_shared_info = (void *)&xen_dummy_shared_info;
DEFINE_PER_CPU(struct vcpu_info *, xen_vcpu);
static struct vcpu_info __percpu *xen_vcpu_info;
+/* Linux <-> Xen vCPU id mapping */
+DEFINE_PER_CPU(int, xen_vcpu_id) = -1;
+EXPORT_SYMBOL_GPL(xen_vcpu_id);
+
/* These are unused until we support booting "pre-ballooned" */
unsigned long xen_released_pages;
struct xen_memory_region xen_extra_mem[XEN_EXTRA_MEM_MAX_REGIONS] __initdata;
@@ -179,6 +183,9 @@ static void xen_percpu_init(void)
pr_info("Xen: initializing cpu%d\n", cpu);
vcpup = per_cpu_ptr(xen_vcpu_info, cpu);
+ /* Direct vCPU id mapping for ARM guests. */
+ per_cpu(xen_vcpu_id, cpu) = cpu;
+
info.mfn = virt_to_gfn(vcpup);
info.offset = xen_offset_in_page(vcpup);
@@ -328,6 +335,9 @@ static int __init xen_guest_init(void)
if (xen_vcpu_info == NULL)
return -ENOMEM;
+ /* Direct vCPU id mapping for ARM guests. */
+ per_cpu(xen_vcpu_id, 0) = 0;
+
if (gnttab_setup_auto_xlat_frames(grant_frames)) {
free_percpu(xen_vcpu_info);
return -ENOMEM;
diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 760789a..69f4c0c 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -59,6 +59,7 @@
#include <asm/xen/pci.h>
#include <asm/xen/hypercall.h>
#include <asm/xen/hypervisor.h>
+#include <asm/xen/cpuid.h>
#include <asm/fixmap.h>
#include <asm/processor.h>
#include <asm/proto.h>
@@ -118,6 +119,10 @@ DEFINE_PER_CPU(struct vcpu_info *, xen_vcpu);
*/
DEFINE_PER_CPU(struct vcpu_info, xen_vcpu_info);
+/* Linux <-> Xen vCPU id mapping */
+DEFINE_PER_CPU(int, xen_vcpu_id) = -1;
+EXPORT_SYMBOL_GPL(xen_vcpu_id);
+
enum xen_domain_type xen_domain_type = XEN_NATIVE;
EXPORT_SYMBOL_GPL(xen_domain_type);
@@ -1137,8 +1142,11 @@ void xen_setup_vcpu_info_placement(void)
{
int cpu;
- for_each_possible_cpu(cpu)
+ for_each_possible_cpu(cpu) {
+ /* Set up direct vCPU id mapping for PV guests. */
+ per_cpu(xen_vcpu_id, cpu) = cpu;
xen_vcpu_setup(cpu);
+ }
/* xen_vcpu_setup managed to place the vcpu_info within the
* percpu area for all cpus, so make use of it. Note that for
@@ -1797,6 +1805,12 @@ static void __init init_hvm_pv_info(void)
xen_setup_features();
+ cpuid(base + 4, &eax, &ebx, &ecx, &edx);
+ if (eax & XEN_HVM_CPUID_VCPU_ID_PRESENT)
+ this_cpu_write(xen_vcpu_id, ebx);
+ else
+ this_cpu_write(xen_vcpu_id, smp_processor_id());
+
pv_info.name = "Xen HVM";
xen_domain_type = XEN_HVM_DOMAIN;
@@ -1808,6 +1822,8 @@ static int xen_hvm_cpu_notify(struct notifier_block *self, unsigned long action,
int cpu = (long)hcpu;
switch (action) {
case CPU_UP_PREPARE:
+ /* vLAPIC_ID == Xen's vCPU_ID * 2 for HVM guests */
+ per_cpu(xen_vcpu_id, cpu) = cpu_physical_id(cpu) / 2;
xen_vcpu_setup(cpu);
if (xen_have_vector_callback) {
if (xen_feature(XENFEAT_hvm_safe_pvclock))
diff --git a/include/xen/xen-ops.h b/include/xen/xen-ops.h
index 86abe07..b02a343 100644
--- a/include/xen/xen-ops.h
+++ b/include/xen/xen-ops.h
@@ -8,6 +8,7 @@
#include <xen/interface/vcpu.h>
DECLARE_PER_CPU(struct vcpu_info *, xen_vcpu);
+DECLARE_PER_CPU(int, xen_vcpu_id);
void xen_arch_pre_suspend(void);
void xen_arch_post_suspend(int suspend_cancelled);
--
2.5.5
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [Xen-devel] [PATCH linux 2/8] xen: introduce xen_vcpu_id mapping
2016-06-28 16:47 ` [PATCH linux 2/8] xen: introduce xen_vcpu_id mapping Vitaly Kuznetsov
@ 2016-06-28 17:28 ` Andrew Cooper
2016-06-29 12:16 ` Vitaly Kuznetsov
0 siblings, 1 reply; 26+ messages in thread
From: Andrew Cooper @ 2016-06-28 17:28 UTC (permalink / raw)
To: Vitaly Kuznetsov, xen-devel
Cc: Juergen Gross, Stefano Stabellini, x86, linux-kernel,
Julien Grall, Ingo Molnar, David Vrabel, H. Peter Anvin,
Boris Ostrovsky, Thomas Gleixner
On 28/06/16 17:47, Vitaly Kuznetsov wrote:
> @@ -1808,6 +1822,8 @@ static int xen_hvm_cpu_notify(struct notifier_block *self, unsigned long action,
> int cpu = (long)hcpu;
> switch (action) {
> case CPU_UP_PREPARE:
> + /* vLAPIC_ID == Xen's vCPU_ID * 2 for HVM guests */
> + per_cpu(xen_vcpu_id, cpu) = cpu_physical_id(cpu) / 2;
Please do not assume or propagate this brokenness. It is incorrect in
the general case, and I will be fixing in the hypervisor in due course.
Always read the APIC_ID from the LAPIC, per regular hardware.
~Andrew
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Xen-devel] [PATCH linux 2/8] xen: introduce xen_vcpu_id mapping
2016-06-28 17:28 ` [Xen-devel] " Andrew Cooper
@ 2016-06-29 12:16 ` Vitaly Kuznetsov
2016-06-29 12:30 ` Andrew Cooper
0 siblings, 1 reply; 26+ messages in thread
From: Vitaly Kuznetsov @ 2016-06-29 12:16 UTC (permalink / raw)
To: Andrew Cooper
Cc: xen-devel, Juergen Gross, Stefano Stabellini, x86, linux-kernel,
Julien Grall, Ingo Molnar, David Vrabel, H. Peter Anvin,
Boris Ostrovsky, Thomas Gleixner
Andrew Cooper <andrew.cooper3@citrix.com> writes:
> On 28/06/16 17:47, Vitaly Kuznetsov wrote:
>> @@ -1808,6 +1822,8 @@ static int xen_hvm_cpu_notify(struct notifier_block *self, unsigned long action,
>> int cpu = (long)hcpu;
>> switch (action) {
>> case CPU_UP_PREPARE:
>> + /* vLAPIC_ID == Xen's vCPU_ID * 2 for HVM guests */
>> + per_cpu(xen_vcpu_id, cpu) = cpu_physical_id(cpu) / 2;
>
> Please do not assume or propagate this brokenness. It is incorrect in
> the general case, and I will be fixing in the hypervisor in due course.
>
> Always read the APIC_ID from the LAPIC, per regular hardware.
(I'm probbaly missing something important - please bear with me)
The problem here is that I need to get _other_ CPU's id before any code
is executed on that CPU (or, at least, this is the current state of
affairs if you look at xen_hvm_cpu_up()) so I can't use CPUID/do MSR
reads/... The only option I see here is to rely on ACPI (MADT) data
which is stored in x86_cpu_to_apicid (and that's what cpu_physical_id()
gives us). MADT also has processor id which connects it to DSDT but I'm
not sure Linux keeps this data. But this is something fixable I guess.
--
Vitaly
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Xen-devel] [PATCH linux 2/8] xen: introduce xen_vcpu_id mapping
2016-06-29 12:16 ` Vitaly Kuznetsov
@ 2016-06-29 12:30 ` Andrew Cooper
2016-06-29 12:50 ` Vitaly Kuznetsov
0 siblings, 1 reply; 26+ messages in thread
From: Andrew Cooper @ 2016-06-29 12:30 UTC (permalink / raw)
To: Vitaly Kuznetsov
Cc: xen-devel, Juergen Gross, Stefano Stabellini, x86, linux-kernel,
Julien Grall, Ingo Molnar, David Vrabel, H. Peter Anvin,
Boris Ostrovsky, Thomas Gleixner, Jan Beulich
On 29/06/16 13:16, Vitaly Kuznetsov wrote:
> Andrew Cooper <andrew.cooper3@citrix.com> writes:
>
>> On 28/06/16 17:47, Vitaly Kuznetsov wrote:
>>> @@ -1808,6 +1822,8 @@ static int xen_hvm_cpu_notify(struct notifier_block *self, unsigned long action,
>>> int cpu = (long)hcpu;
>>> switch (action) {
>>> case CPU_UP_PREPARE:
>>> + /* vLAPIC_ID == Xen's vCPU_ID * 2 for HVM guests */
>>> + per_cpu(xen_vcpu_id, cpu) = cpu_physical_id(cpu) / 2;
>> Please do not assume or propagate this brokenness. It is incorrect in
>> the general case, and I will be fixing in the hypervisor in due course.
>>
>> Always read the APIC_ID from the LAPIC, per regular hardware.
> (I'm probbaly missing something important - please bear with me)
>
> The problem here is that I need to get _other_ CPU's id before any code
> is executed on that CPU (or, at least, this is the current state of
> affairs if you look at xen_hvm_cpu_up()) so I can't use CPUID/do MSR
> reads/... The only option I see here is to rely on ACPI (MADT) data
> which is stored in x86_cpu_to_apicid (and that's what cpu_physical_id()
> gives us). MADT also has processor id which connects it to DSDT but I'm
> not sure Linux keeps this data. But this is something fixable I guess.
Hmm yes - that is a tricky issue.
It is not safe or correct to assume that xen_vcpu_id is APICID / 2.
This is currently the case for most modern versions of Xen, but isn't
the case for older versions, and won't be the case in the future when I
(or someone else) fixes topology representation for guests.
For this to work, we need one or more of:
1) to provide the guest a full mapping from APIC_ID to vcpu id at boot time.
2) add a new interface where the guest can explicitly query "what is the
vcpu id for the entity with this APIC_ID".
3) Allow HVM guests to identify a vcpu in a hypercall by APIC_ID.
3 is the cleaner approach, but given that vcpu ids have already leaked
into an HVM domains idea of the world, 1 or 2 is probably a better
ladder to dig us out of this hole.
~Andrew.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Xen-devel] [PATCH linux 2/8] xen: introduce xen_vcpu_id mapping
2016-06-29 12:30 ` Andrew Cooper
@ 2016-06-29 12:50 ` Vitaly Kuznetsov
2016-06-29 16:19 ` Vitaly Kuznetsov
0 siblings, 1 reply; 26+ messages in thread
From: Vitaly Kuznetsov @ 2016-06-29 12:50 UTC (permalink / raw)
To: Andrew Cooper
Cc: xen-devel, Juergen Gross, Stefano Stabellini, x86, linux-kernel,
Julien Grall, Ingo Molnar, David Vrabel, H. Peter Anvin,
Boris Ostrovsky, Thomas Gleixner, Jan Beulich
Andrew Cooper <andrew.cooper3@citrix.com> writes:
> On 29/06/16 13:16, Vitaly Kuznetsov wrote:
>> Andrew Cooper <andrew.cooper3@citrix.com> writes:
>>
>>> On 28/06/16 17:47, Vitaly Kuznetsov wrote:
>>>> @@ -1808,6 +1822,8 @@ static int xen_hvm_cpu_notify(struct notifier_block *self, unsigned long action,
>>>> int cpu = (long)hcpu;
>>>> switch (action) {
>>>> case CPU_UP_PREPARE:
>>>> + /* vLAPIC_ID == Xen's vCPU_ID * 2 for HVM guests */
>>>> + per_cpu(xen_vcpu_id, cpu) = cpu_physical_id(cpu) / 2;
>>> Please do not assume or propagate this brokenness. It is incorrect in
>>> the general case, and I will be fixing in the hypervisor in due course.
>>>
>>> Always read the APIC_ID from the LAPIC, per regular hardware.
>> (I'm probbaly missing something important - please bear with me)
>>
>> The problem here is that I need to get _other_ CPU's id before any code
>> is executed on that CPU (or, at least, this is the current state of
>> affairs if you look at xen_hvm_cpu_up()) so I can't use CPUID/do MSR
>> reads/... The only option I see here is to rely on ACPI (MADT) data
>> which is stored in x86_cpu_to_apicid (and that's what cpu_physical_id()
>> gives us). MADT also has processor id which connects it to DSDT but I'm
>> not sure Linux keeps this data. But this is something fixable I guess.
>
> Hmm yes - that is a tricky issue.
>
> It is not safe or correct to assume that xen_vcpu_id is APICID / 2.
>
> This is currently the case for most modern versions of Xen, but isn't
> the case for older versions, and won't be the case in the future when I
> (or someone else) fixes topology representation for guests.
>
> For this to work, we need one or more of:
>
> 1) to provide the guest a full mapping from APIC_ID to vcpu id at boot
> time.
So can we rely on ACPI data? Especially on MADT and processor ids there?
I think we can always guarantee that processor ids there match vCPU
ids. If yes I can try saving this data when we parse MADT.
> 2) add a new interface where the guest can explicitly query "what is the
> vcpu id for the entity with this APIC_ID".
> 3) Allow HVM guests to identify a vcpu in a hypercall by APIC_ID.
>
> 3 is the cleaner approach, but given that vcpu ids have already leaked
> into an HVM domains idea of the world, 1 or 2 is probably a better
> ladder to dig us out of this hole.
It would be nice to avoid hypervisor changes but if we have to modify it
we can fail all secondary CPUs for now when we detect that CPU0's vCPU
id is not 0 (and CPU0 gets its id with CPUID).
--
Vitaly
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [Xen-devel] [PATCH linux 2/8] xen: introduce xen_vcpu_id mapping
2016-06-29 12:50 ` Vitaly Kuznetsov
@ 2016-06-29 16:19 ` Vitaly Kuznetsov
[not found] ` <5040c916-279f-c350-383a-583ec1700686@citrix.com>
0 siblings, 1 reply; 26+ messages in thread
From: Vitaly Kuznetsov @ 2016-06-29 16:19 UTC (permalink / raw)
To: Andrew Cooper
Cc: xen-devel, Juergen Gross, Stefano Stabellini, x86, linux-kernel,
Ingo Molnar, David Vrabel, H. Peter Anvin, Boris Ostrovsky,
Thomas Gleixner, Jan Beulich
[-- Attachment #1: Type: text/plain, Size: 2286 bytes --]
Vitaly Kuznetsov <vkuznets@redhat.com> writes:
> Andrew Cooper <andrew.cooper3@citrix.com> writes:
>
>> On 29/06/16 13:16, Vitaly Kuznetsov wrote:
>>> Andrew Cooper <andrew.cooper3@citrix.com> writes:
>>>
>>>> On 28/06/16 17:47, Vitaly Kuznetsov wrote:
>>>>> @@ -1808,6 +1822,8 @@ static int xen_hvm_cpu_notify(struct notifier_block *self, unsigned long action,
>>>>> int cpu = (long)hcpu;
>>>>> switch (action) {
>>>>> case CPU_UP_PREPARE:
>>>>> + /* vLAPIC_ID == Xen's vCPU_ID * 2 for HVM guests */
>>>>> + per_cpu(xen_vcpu_id, cpu) = cpu_physical_id(cpu) / 2;
>>>> Please do not assume or propagate this brokenness. It is incorrect in
>>>> the general case, and I will be fixing in the hypervisor in due course.
>>>>
>>>> Always read the APIC_ID from the LAPIC, per regular hardware.
>>> (I'm probbaly missing something important - please bear with me)
>>>
>>> The problem here is that I need to get _other_ CPU's id before any code
>>> is executed on that CPU (or, at least, this is the current state of
>>> affairs if you look at xen_hvm_cpu_up()) so I can't use CPUID/do MSR
>>> reads/... The only option I see here is to rely on ACPI (MADT) data
>>> which is stored in x86_cpu_to_apicid (and that's what cpu_physical_id()
>>> gives us). MADT also has processor id which connects it to DSDT but I'm
>>> not sure Linux keeps this data. But this is something fixable I guess.
>>
>> Hmm yes - that is a tricky issue.
>>
>> It is not safe or correct to assume that xen_vcpu_id is APICID / 2.
>>
>> This is currently the case for most modern versions of Xen, but isn't
>> the case for older versions, and won't be the case in the future when I
>> (or someone else) fixes topology representation for guests.
>>
>> For this to work, we need one or more of:
>>
>> 1) to provide the guest a full mapping from APIC_ID to vcpu id at boot
>> time.
>
> So can we rely on ACPI data? Especially on MADT and processor ids there?
> I think we can always guarantee that processor ids there match vCPU
> ids. If yes I can try saving this data when we parse MADT.
>
To explain better what I'm trying to suggest here please take a look at
the attached patch. If we can guarantee long term that ACPI id always
equals to Xen's idea of vCPU id this is probably the easiest way.
--
Vitaly
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-x86-acpi-store-APIC-ids-for-future-usage-use-it-in-x.patch --]
[-- Type: text/x-patch, Size: 5294 bytes --]
>From 7c9cd25bdcd3e6ee866aa49550c9a4160194c3ba Mon Sep 17 00:00:00 2001
From: Vitaly Kuznetsov <vkuznets@redhat.com>
Date: Wed, 29 Jun 2016 18:13:01 +0200
Subject: [PATCH RFC/WIP] x86/acpi: store APIC ids for future usage (+use it in
xen_hvm_cpu_notify)
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
arch/x86/include/asm/smp.h | 1 +
arch/x86/kernel/acpi/boot.c | 18 ++++++++++++++----
arch/x86/kernel/setup_percpu.c | 3 +++
arch/x86/xen/enlighten.c | 8 ++++++--
4 files changed, 24 insertions(+), 6 deletions(-)
diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h
index 66b0573..c68b56a 100644
--- a/arch/x86/include/asm/smp.h
+++ b/arch/x86/include/asm/smp.h
@@ -33,6 +33,7 @@ static inline struct cpumask *cpu_llc_shared_mask(int cpu)
}
DECLARE_EARLY_PER_CPU_READ_MOSTLY(u16, x86_cpu_to_apicid);
+DECLARE_EARLY_PER_CPU_READ_MOSTLY(int, x86_cpu_to_acpiid);
DECLARE_EARLY_PER_CPU_READ_MOSTLY(u16, x86_bios_cpu_apicid);
#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86_32)
DECLARE_EARLY_PER_CPU_READ_MOSTLY(int, x86_cpu_to_logical_apicid);
diff --git a/arch/x86/kernel/acpi/boot.c b/arch/x86/kernel/acpi/boot.c
index 9414f84..3bbf0ab 100644
--- a/arch/x86/kernel/acpi/boot.c
+++ b/arch/x86/kernel/acpi/boot.c
@@ -110,6 +110,9 @@ static u32 isa_irq_to_gsi[NR_IRQS_LEGACY] __read_mostly = {
#define ACPI_INVALID_GSI INT_MIN
+DEFINE_EARLY_PER_CPU_READ_MOSTLY(int, x86_cpu_to_acpiid, -1);
+EXPORT_EARLY_PER_CPU_SYMBOL(x86_cpu_to_acpiid);
+
/*
* This is just a simple wrapper around early_ioremap(),
* with sanity checks for phys == 0 and size == 0.
@@ -165,9 +168,10 @@ static int __init acpi_parse_madt(struct acpi_table_header *table)
*
* Returns the logic cpu number which maps to the local apic
*/
-static int acpi_register_lapic(int id, u8 enabled)
+static int acpi_register_lapic(int id, int acpiid, u8 enabled)
{
unsigned int ver = 0;
+ int cpu;
if (id >= MAX_LOCAL_APIC) {
printk(KERN_INFO PREFIX "skipped apicid that is too big\n");
@@ -182,7 +186,11 @@ static int acpi_register_lapic(int id, u8 enabled)
if (boot_cpu_physical_apicid != -1U)
ver = apic_version[boot_cpu_physical_apicid];
- return generic_processor_info(id, ver);
+ cpu = generic_processor_info(id, ver);
+ if (cpu >= 0)
+ early_per_cpu(x86_cpu_to_acpiid, cpu) = acpiid;
+
+ return cpu;
}
static int __init
@@ -212,7 +220,7 @@ acpi_parse_x2apic(struct acpi_subtable_header *header, const unsigned long end)
if (!apic->apic_id_valid(apic_id) && enabled)
printk(KERN_WARNING PREFIX "x2apic entry ignored\n");
else
- acpi_register_lapic(apic_id, enabled);
+ acpi_register_lapic(apic_id, processor->uid, enabled);
#else
printk(KERN_WARNING PREFIX "x2apic entry ignored\n");
#endif
@@ -240,6 +248,7 @@ acpi_parse_lapic(struct acpi_subtable_header * header, const unsigned long end)
* when we use CPU hotplug.
*/
acpi_register_lapic(processor->id, /* APIC ID */
+ processor->processor_id, /* ACPI ID */
processor->lapic_flags & ACPI_MADT_ENABLED);
return 0;
@@ -258,6 +267,7 @@ acpi_parse_sapic(struct acpi_subtable_header *header, const unsigned long end)
acpi_table_print_madt_entry(header);
acpi_register_lapic((processor->id << 8) | processor->eid,/* APIC ID */
+ processor->uid, /* ACPI ID */
processor->lapic_flags & ACPI_MADT_ENABLED);
return 0;
@@ -714,7 +724,7 @@ int acpi_map_cpu(acpi_handle handle, phys_cpuid_t physid, int *pcpu)
{
int cpu;
- cpu = acpi_register_lapic(physid, ACPI_MADT_ENABLED);
+ cpu = acpi_register_lapic(physid, -1, ACPI_MADT_ENABLED);
if (cpu < 0) {
pr_info(PREFIX "Unable to map lapic to logical cpu number\n");
return cpu;
diff --git a/arch/x86/kernel/setup_percpu.c b/arch/x86/kernel/setup_percpu.c
index e4fcb87..7a40e06 100644
--- a/arch/x86/kernel/setup_percpu.c
+++ b/arch/x86/kernel/setup_percpu.c
@@ -236,6 +236,8 @@ void __init setup_per_cpu_areas(void)
early_per_cpu_map(x86_cpu_to_apicid, cpu);
per_cpu(x86_bios_cpu_apicid, cpu) =
early_per_cpu_map(x86_bios_cpu_apicid, cpu);
+ per_cpu(x86_cpu_to_acpiid, cpu) =
+ early_per_cpu_map(x86_cpu_to_acpiid, cpu);
#endif
#ifdef CONFIG_X86_32
per_cpu(x86_cpu_to_logical_apicid, cpu) =
@@ -271,6 +273,7 @@ void __init setup_per_cpu_areas(void)
#ifdef CONFIG_X86_LOCAL_APIC
early_per_cpu_ptr(x86_cpu_to_apicid) = NULL;
early_per_cpu_ptr(x86_bios_cpu_apicid) = NULL;
+ early_per_cpu_ptr(x86_cpu_to_acpiid) = NULL;
#endif
#ifdef CONFIG_X86_32
early_per_cpu_ptr(x86_cpu_to_logical_apicid) = NULL;
diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 0be29a0..8ae4871 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -1825,10 +1825,14 @@ static int xen_hvm_cpu_notify(struct notifier_block *self, unsigned long action,
void *hcpu)
{
int cpu = (long)hcpu;
+ int acpiid = per_cpu(x86_cpu_to_acpiid, cpu);
+
switch (action) {
case CPU_UP_PREPARE:
- /* vLAPIC_ID == Xen's vCPU_ID * 2 for HVM guests */
- per_cpu(xen_vcpu_id, cpu) = cpu_physical_id(cpu) / 2;
+ if (acpiid != -1)
+ per_cpu(xen_vcpu_id, cpu) = acpiid;
+ else
+ per_cpu(xen_vcpu_id, cpu) = cpu;
xen_vcpu_setup(cpu);
if (xen_have_vector_callback) {
if (xen_feature(XENFEAT_hvm_safe_pvclock))
--
2.5.5
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH linux 3/8] x86/xen: use xen_vcpu_id mapping for HYPERVISOR_vcpu_op
2016-06-28 16:47 [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs Vitaly Kuznetsov
2016-06-28 16:47 ` [PATCH linux 1/8] x86/xen: update cpuid.h from Xen-4.7 Vitaly Kuznetsov
2016-06-28 16:47 ` [PATCH linux 2/8] xen: introduce xen_vcpu_id mapping Vitaly Kuznetsov
@ 2016-06-28 16:47 ` Vitaly Kuznetsov
2016-06-29 12:26 ` [Xen-devel] " David Vrabel
2016-06-28 16:47 ` [PATCH linux 4/8] x86/xen: use xen_vcpu_id mapping when pointing vcpu_info to the shared_info page Vitaly Kuznetsov
` (5 subsequent siblings)
8 siblings, 1 reply; 26+ messages in thread
From: Vitaly Kuznetsov @ 2016-06-28 16:47 UTC (permalink / raw)
To: xen-devel
Cc: linux-kernel, x86, Stefano Stabellini, Boris Ostrovsky,
David Vrabel, Juergen Gross, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Julien Grall
HYPERVISOR_vcpu_op passes Linux's idea of vCPU id as a parameter while
Xen's idea is expected. In some cases these ideas diverge so we need to
do remapping.
There is an issue, however. PV guests do VCPUOP_is_up very early
(see xen_fill_possible_map() and xen_filter_cpu_maps()) when we don't have
perpu areas initialized. While it could be solved with switching to
early_percpu for xen_vcpu_id I think it's not worth it: PV guests will
probably never get to the point where their idea of vCPU id diverges from
Xen's.
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
arch/x86/include/asm/xen/hypercall.h | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/xen/hypercall.h b/arch/x86/include/asm/xen/hypercall.h
index a12a047..20750c8 100644
--- a/arch/x86/include/asm/xen/hypercall.h
+++ b/arch/x86/include/asm/xen/hypercall.h
@@ -49,6 +49,7 @@
#include <xen/interface/physdev.h>
#include <xen/interface/platform.h>
#include <xen/interface/xen-mca.h>
+#include <xen/xen-ops.h>
/*
* The hypercall asms have to meet several constraints:
@@ -422,7 +423,15 @@ HYPERVISOR_vm_assist(unsigned int cmd, unsigned int type)
static inline int
HYPERVISOR_vcpu_op(int cmd, int vcpuid, void *extra_args)
{
- return _hypercall3(int, vcpu_op, cmd, vcpuid, extra_args);
+ /*
+ * PV guests call HYPERVISOR_vcpu_op before percpu areas are
+ * initialized. As we always use direct mapping for vCPU ids
+ * for them we can simply use Linux vcpuid here.
+ */
+ return _hypercall3(int, vcpu_op, cmd,
+ per_cpu(xen_vcpu_id, vcpuid) != -1 ?
+ per_cpu(xen_vcpu_id, vcpuid) : vcpuid,
+ extra_args);
}
#ifdef CONFIG_X86_64
--
2.5.5
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [Xen-devel] [PATCH linux 3/8] x86/xen: use xen_vcpu_id mapping for HYPERVISOR_vcpu_op
2016-06-28 16:47 ` [PATCH linux 3/8] x86/xen: use xen_vcpu_id mapping for HYPERVISOR_vcpu_op Vitaly Kuznetsov
@ 2016-06-29 12:26 ` David Vrabel
0 siblings, 0 replies; 26+ messages in thread
From: David Vrabel @ 2016-06-29 12:26 UTC (permalink / raw)
To: Vitaly Kuznetsov, xen-devel
Cc: Juergen Gross, Stefano Stabellini, x86, linux-kernel,
Julien Grall, Ingo Molnar, David Vrabel, H. Peter Anvin,
Boris Ostrovsky, Thomas Gleixner
On 28/06/16 17:47, Vitaly Kuznetsov wrote:
> HYPERVISOR_vcpu_op passes Linux's idea of vCPU id as a parameter while
> Xen's idea is expected. In some cases these ideas diverge so we need to
> do remapping.
>
> There is an issue, however. PV guests do VCPUOP_is_up very early
> (see xen_fill_possible_map() and xen_filter_cpu_maps()) when we don't have
> perpu areas initialized. While it could be solved with switching to
> early_percpu for xen_vcpu_id I think it's not worth it: PV guests will
> probably never get to the point where their idea of vCPU id diverges from
> Xen's.
[...]
> static inline int
> HYPERVISOR_vcpu_op(int cmd, int vcpuid, void *extra_args)
> {
> - return _hypercall3(int, vcpu_op, cmd, vcpuid, extra_args);
> + /*
> + * PV guests call HYPERVISOR_vcpu_op before percpu areas are
> + * initialized. As we always use direct mapping for vCPU ids
> + * for them we can simply use Linux vcpuid here.
> + */
> + return _hypercall3(int, vcpu_op, cmd,
> + per_cpu(xen_vcpu_id, vcpuid) != -1 ?
> + per_cpu(xen_vcpu_id, vcpuid) : vcpuid,
> + extra_args);
> }
HYPERVISOR_vcpu_op() should take Xen VCPUs, with the callers doing the
mapping.
David
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH linux 4/8] x86/xen: use xen_vcpu_id mapping when pointing vcpu_info to the shared_info page
2016-06-28 16:47 [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs Vitaly Kuznetsov
` (2 preceding siblings ...)
2016-06-28 16:47 ` [PATCH linux 3/8] x86/xen: use xen_vcpu_id mapping for HYPERVISOR_vcpu_op Vitaly Kuznetsov
@ 2016-06-28 16:47 ` Vitaly Kuznetsov
2016-06-29 12:32 ` [Xen-devel] " David Vrabel
2016-06-28 16:47 ` [PATCH linux 5/8] xen/events: use xen_vcpu_id mapping in events_base Vitaly Kuznetsov
` (4 subsequent siblings)
8 siblings, 1 reply; 26+ messages in thread
From: Vitaly Kuznetsov @ 2016-06-28 16:47 UTC (permalink / raw)
To: xen-devel
Cc: linux-kernel, x86, Stefano Stabellini, Boris Ostrovsky,
David Vrabel, Juergen Gross, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Julien Grall
shared_info page has space for 32 vcpu info slots for first 32 vCPUs but
these are the first 32 vCPUs from Xen's perspective and we should map them
accordingly with the newly introduced xen_vcpu_id mapping.
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
arch/x86/xen/enlighten.c | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 69f4c0c..1596626 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -189,6 +189,7 @@ static void xen_vcpu_setup(int cpu)
struct vcpu_register_vcpu_info info;
int err;
struct vcpu_info *vcpup;
+ int xen_cpu = per_cpu(xen_vcpu_id, cpu);
BUG_ON(HYPERVISOR_shared_info == &xen_dummy_shared_info);
@@ -207,8 +208,9 @@ static void xen_vcpu_setup(int cpu)
if (per_cpu(xen_vcpu, cpu) == &per_cpu(xen_vcpu_info, cpu))
return;
}
- if (cpu < MAX_VIRT_CPUS)
- per_cpu(xen_vcpu,cpu) = &HYPERVISOR_shared_info->vcpu_info[cpu];
+ if (xen_cpu < MAX_VIRT_CPUS)
+ per_cpu(xen_vcpu, cpu) =
+ &HYPERVISOR_shared_info->vcpu_info[xen_cpu];
if (!have_vcpu_info_placement) {
if (cpu >= MAX_VIRT_CPUS)
@@ -1752,7 +1754,7 @@ asmlinkage __visible void __init xen_start_kernel(void)
void __ref xen_hvm_init_shared_info(void)
{
- int cpu;
+ int cpu, xen_cpu;
struct xen_add_to_physmap xatp;
static struct shared_info *shared_info_page = 0;
@@ -1777,10 +1779,13 @@ void __ref xen_hvm_init_shared_info(void)
* online but xen_hvm_init_shared_info is run at resume time too and
* in that case multiple vcpus might be online. */
for_each_online_cpu(cpu) {
+ xen_cpu = per_cpu(xen_vcpu_id, cpu);
+
/* Leave it to be NULL. */
- if (cpu >= MAX_VIRT_CPUS)
+ if (xen_cpu >= MAX_VIRT_CPUS)
continue;
- per_cpu(xen_vcpu, cpu) = &HYPERVISOR_shared_info->vcpu_info[cpu];
+ per_cpu(xen_vcpu, cpu) =
+ &HYPERVISOR_shared_info->vcpu_info[xen_cpu];
}
}
--
2.5.5
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [Xen-devel] [PATCH linux 4/8] x86/xen: use xen_vcpu_id mapping when pointing vcpu_info to the shared_info page
2016-06-28 16:47 ` [PATCH linux 4/8] x86/xen: use xen_vcpu_id mapping when pointing vcpu_info to the shared_info page Vitaly Kuznetsov
@ 2016-06-29 12:32 ` David Vrabel
0 siblings, 0 replies; 26+ messages in thread
From: David Vrabel @ 2016-06-29 12:32 UTC (permalink / raw)
To: Vitaly Kuznetsov, xen-devel
Cc: Juergen Gross, Stefano Stabellini, x86, linux-kernel,
Ingo Molnar, David Vrabel, H. Peter Anvin, Boris Ostrovsky,
Thomas Gleixner
On 28/06/16 17:47, Vitaly Kuznetsov wrote:
> shared_info page has space for 32 vcpu info slots for first 32 vCPUs but
> these are the first 32 vCPUs from Xen's perspective and we should map them
> accordingly with the newly introduced xen_vcpu_id mapping.
>
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
> arch/x86/xen/enlighten.c | 15 ++++++++++-----
> 1 file changed, 10 insertions(+), 5 deletions(-)
>
> diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
> index 69f4c0c..1596626 100644
> --- a/arch/x86/xen/enlighten.c
> +++ b/arch/x86/xen/enlighten.c
> @@ -189,6 +189,7 @@ static void xen_vcpu_setup(int cpu)
> struct vcpu_register_vcpu_info info;
> int err;
> struct vcpu_info *vcpup;
> + int xen_cpu = per_cpu(xen_vcpu_id, cpu);
I think there should be a
static inline int xen_vcpu_nr(int cpu)
{
return per_cpu(xen_vcpu_id, cpu);
}
helper function.
David
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH linux 5/8] xen/events: use xen_vcpu_id mapping in events_base
2016-06-28 16:47 [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs Vitaly Kuznetsov
` (3 preceding siblings ...)
2016-06-28 16:47 ` [PATCH linux 4/8] x86/xen: use xen_vcpu_id mapping when pointing vcpu_info to the shared_info page Vitaly Kuznetsov
@ 2016-06-28 16:47 ` Vitaly Kuznetsov
2016-06-28 16:47 ` [PATCH linux 6/8] xen/events: fifo: use xen_vcpu_id mapping Vitaly Kuznetsov
` (3 subsequent siblings)
8 siblings, 0 replies; 26+ messages in thread
From: Vitaly Kuznetsov @ 2016-06-28 16:47 UTC (permalink / raw)
To: xen-devel
Cc: linux-kernel, x86, Stefano Stabellini, Boris Ostrovsky,
David Vrabel, Juergen Gross, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Julien Grall
EVTCHNOP_bind_ipi and EVTCHNOP_bind_virq pass vCPU id as a parameter and
Xen's idea of vCPU id should be used. Use the newly introduced xen_vcpu_id
mapping to convert it from Linux's id.
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
drivers/xen/events/events_base.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/drivers/xen/events/events_base.c b/drivers/xen/events/events_base.c
index 71d49a9..73b8b65 100644
--- a/drivers/xen/events/events_base.c
+++ b/drivers/xen/events/events_base.c
@@ -895,7 +895,7 @@ static int bind_ipi_to_irq(unsigned int ipi, unsigned int cpu)
irq_set_chip_and_handler_name(irq, &xen_percpu_chip,
handle_percpu_irq, "ipi");
- bind_ipi.vcpu = cpu;
+ bind_ipi.vcpu = per_cpu(xen_vcpu_id, cpu);
if (HYPERVISOR_event_channel_op(EVTCHNOP_bind_ipi,
&bind_ipi) != 0)
BUG();
@@ -991,7 +991,7 @@ int bind_virq_to_irq(unsigned int virq, unsigned int cpu, bool percpu)
handle_edge_irq, "virq");
bind_virq.virq = virq;
- bind_virq.vcpu = cpu;
+ bind_virq.vcpu = per_cpu(xen_vcpu_id, cpu);
ret = HYPERVISOR_event_channel_op(EVTCHNOP_bind_virq,
&bind_virq);
if (ret == 0)
@@ -1318,7 +1318,7 @@ static int rebind_irq_to_cpu(unsigned irq, unsigned tcpu)
/* Send future instances of this interrupt to other vcpu. */
bind_vcpu.port = evtchn;
- bind_vcpu.vcpu = tcpu;
+ bind_vcpu.vcpu = per_cpu(xen_vcpu_id, tcpu);
/*
* Mask the event while changing the VCPU binding to prevent
@@ -1458,7 +1458,7 @@ static void restore_cpu_virqs(unsigned int cpu)
/* Get a new binding from Xen. */
bind_virq.virq = virq;
- bind_virq.vcpu = cpu;
+ bind_virq.vcpu = per_cpu(xen_vcpu_id, cpu);
if (HYPERVISOR_event_channel_op(EVTCHNOP_bind_virq,
&bind_virq) != 0)
BUG();
@@ -1482,7 +1482,7 @@ static void restore_cpu_ipis(unsigned int cpu)
BUG_ON(ipi_from_irq(irq) != ipi);
/* Get a new binding from Xen. */
- bind_ipi.vcpu = cpu;
+ bind_ipi.vcpu = per_cpu(xen_vcpu_id, cpu);
if (HYPERVISOR_event_channel_op(EVTCHNOP_bind_ipi,
&bind_ipi) != 0)
BUG();
--
2.5.5
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH linux 6/8] xen/events: fifo: use xen_vcpu_id mapping
2016-06-28 16:47 [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs Vitaly Kuznetsov
` (4 preceding siblings ...)
2016-06-28 16:47 ` [PATCH linux 5/8] xen/events: use xen_vcpu_id mapping in events_base Vitaly Kuznetsov
@ 2016-06-28 16:47 ` Vitaly Kuznetsov
2016-06-28 16:47 ` [PATCH linux 7/8] xen/evtchn: " Vitaly Kuznetsov
` (2 subsequent siblings)
8 siblings, 0 replies; 26+ messages in thread
From: Vitaly Kuznetsov @ 2016-06-28 16:47 UTC (permalink / raw)
To: xen-devel
Cc: linux-kernel, x86, Stefano Stabellini, Boris Ostrovsky,
David Vrabel, Juergen Gross, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Julien Grall
EVTCHNOP_init_control has vCPU id as a parameter and Xen's idea of vCPU id
should be used. Use the newly introduced xen_vcpu_id mapping to convert
it from Linux's id.
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
drivers/xen/events/events_fifo.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/xen/events/events_fifo.c b/drivers/xen/events/events_fifo.c
index 9289a17..e3406cd 100644
--- a/drivers/xen/events/events_fifo.c
+++ b/drivers/xen/events/events_fifo.c
@@ -113,7 +113,7 @@ static int init_control_block(int cpu,
init_control.control_gfn = virt_to_gfn(control_block);
init_control.offset = 0;
- init_control.vcpu = cpu;
+ init_control.vcpu = per_cpu(xen_vcpu_id, cpu);
return HYPERVISOR_event_channel_op(EVTCHNOP_init_control, &init_control);
}
--
2.5.5
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH linux 7/8] xen/evtchn: use xen_vcpu_id mapping
2016-06-28 16:47 [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs Vitaly Kuznetsov
` (5 preceding siblings ...)
2016-06-28 16:47 ` [PATCH linux 6/8] xen/events: fifo: use xen_vcpu_id mapping Vitaly Kuznetsov
@ 2016-06-28 16:47 ` Vitaly Kuznetsov
2016-06-28 16:47 ` [PATCH linux 8/8] xen/pvhvm: run xen_vcpu_setup() for the boot CPU Vitaly Kuznetsov
2016-06-28 17:32 ` [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs David Vrabel
8 siblings, 0 replies; 26+ messages in thread
From: Vitaly Kuznetsov @ 2016-06-28 16:47 UTC (permalink / raw)
To: xen-devel
Cc: linux-kernel, x86, Stefano Stabellini, Boris Ostrovsky,
David Vrabel, Juergen Gross, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Julien Grall
Use the newly introduced xen_vcpu_id mapping to get Xen's idea of vCPU id
for CPU0.
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
drivers/xen/evtchn.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/xen/evtchn.c b/drivers/xen/evtchn.c
index f4edd6df..0fc6be4 100644
--- a/drivers/xen/evtchn.c
+++ b/drivers/xen/evtchn.c
@@ -448,7 +448,7 @@ static long evtchn_ioctl(struct file *file,
break;
bind_virq.virq = bind.virq;
- bind_virq.vcpu = 0;
+ bind_virq.vcpu = per_cpu(xen_vcpu_id, 0);
rc = HYPERVISOR_event_channel_op(EVTCHNOP_bind_virq,
&bind_virq);
if (rc != 0)
--
2.5.5
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH linux 8/8] xen/pvhvm: run xen_vcpu_setup() for the boot CPU
2016-06-28 16:47 [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs Vitaly Kuznetsov
` (6 preceding siblings ...)
2016-06-28 16:47 ` [PATCH linux 7/8] xen/evtchn: " Vitaly Kuznetsov
@ 2016-06-28 16:47 ` Vitaly Kuznetsov
2016-06-28 17:32 ` [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs David Vrabel
8 siblings, 0 replies; 26+ messages in thread
From: Vitaly Kuznetsov @ 2016-06-28 16:47 UTC (permalink / raw)
To: xen-devel
Cc: linux-kernel, x86, Stefano Stabellini, Boris Ostrovsky,
David Vrabel, Juergen Gross, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Julien Grall
Historically we didn't call VCPUOP_register_vcpu_info for CPU0 for PVHVM
guests (while we had it for PV and ARM guests). This is usually fine as we
can use vcpu info in the sared_info page but when we try booting on a vCPU
with Xen's vCPU id > 31 (e.g. when we try to kdump after crashing on this
CPU) we're not able to boot. Switch to always doing
VCPUOP_register_vcpu_info for the boot CPU.
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
arch/x86/xen/enlighten.c | 2 +-
arch/x86/xen/smp.c | 7 +++++++
arch/x86/xen/xen-ops.h | 1 +
3 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 1596626..0be29a0 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -184,7 +184,7 @@ static void clamp_max_cpus(void)
#endif
}
-static void xen_vcpu_setup(int cpu)
+void xen_vcpu_setup(int cpu)
{
struct vcpu_register_vcpu_info info;
int err;
diff --git a/arch/x86/xen/smp.c b/arch/x86/xen/smp.c
index 719cf29..b9b31a7 100644
--- a/arch/x86/xen/smp.c
+++ b/arch/x86/xen/smp.c
@@ -322,6 +322,13 @@ static void __init xen_smp_prepare_boot_cpu(void)
xen_filter_cpu_maps();
xen_setup_vcpu_info_placement();
}
+
+ /*
+ * Setup vcpu_info for boot CPU.
+ */
+ if (xen_hvm_domain())
+ xen_vcpu_setup(0);
+
/*
* The alternative logic (which patches the unlock/lock) runs before
* the smp bootup up code is activated. Hence we need to set this up
diff --git a/arch/x86/xen/xen-ops.h b/arch/x86/xen/xen-ops.h
index 4140b07..3cbce3b 100644
--- a/arch/x86/xen/xen-ops.h
+++ b/arch/x86/xen/xen-ops.h
@@ -76,6 +76,7 @@ irqreturn_t xen_debug_interrupt(int irq, void *dev_id);
bool xen_vcpu_stolen(int vcpu);
+void xen_vcpu_setup(int cpu);
void xen_setup_vcpu_info_placement(void);
#ifdef CONFIG_SMP
--
2.5.5
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs
2016-06-28 16:47 [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs Vitaly Kuznetsov
` (7 preceding siblings ...)
2016-06-28 16:47 ` [PATCH linux 8/8] xen/pvhvm: run xen_vcpu_setup() for the boot CPU Vitaly Kuznetsov
@ 2016-06-28 17:32 ` David Vrabel
2016-06-29 9:16 ` Vitaly Kuznetsov
8 siblings, 1 reply; 26+ messages in thread
From: David Vrabel @ 2016-06-28 17:32 UTC (permalink / raw)
To: Vitaly Kuznetsov, xen-devel
Cc: linux-kernel, x86, Stefano Stabellini, Boris Ostrovsky,
Juergen Gross, Thomas Gleixner, Ingo Molnar, H. Peter Anvin,
Julien Grall
On 28/06/16 17:47, Vitaly Kuznetsov wrote:
> It may happen that Xen's and Linux's ideas of vCPU id diverge. In
> particular, when we crash on a secondary vCPU we may want to do kdump
> and unlike plain kexec where we do migrate_to_reboot_cpu() we try booting
> on the vCPU which crashed. This doesn't work very well for PVHVM guests as
> we have a number of hypercalls where we pass vCPU id as a parameter. These
> hypercalls either fail or do something unexpected. To solve the issue we
> need to have a mapping between Linux's and Xen's vCPU ids.
Could the soft-reboot hypercall (optionally) return on vcpu 0?
David
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs
2016-06-28 17:32 ` [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs David Vrabel
@ 2016-06-29 9:16 ` Vitaly Kuznetsov
2016-06-29 12:36 ` David Vrabel
0 siblings, 1 reply; 26+ messages in thread
From: Vitaly Kuznetsov @ 2016-06-29 9:16 UTC (permalink / raw)
To: David Vrabel
Cc: xen-devel, linux-kernel, x86, Stefano Stabellini,
Boris Ostrovsky, Juergen Gross, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Julien Grall
David Vrabel <david.vrabel@citrix.com> writes:
> On 28/06/16 17:47, Vitaly Kuznetsov wrote:
>> It may happen that Xen's and Linux's ideas of vCPU id diverge. In
>> particular, when we crash on a secondary vCPU we may want to do kdump
>> and unlike plain kexec where we do migrate_to_reboot_cpu() we try booting
>> on the vCPU which crashed. This doesn't work very well for PVHVM guests as
>> we have a number of hypercalls where we pass vCPU id as a parameter. These
>> hypercalls either fail or do something unexpected. To solve the issue we
>> need to have a mapping between Linux's and Xen's vCPU ids.
>
> Could the soft-reboot hypercall (optionally) return on vcpu 0?
>
In theory, yes, I think we can re-arrange vCPUs inside the hypervisor so
Linux will get them in the natural order after soft reset.
--
Vitaly
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs
2016-06-29 9:16 ` Vitaly Kuznetsov
@ 2016-06-29 12:36 ` David Vrabel
2016-07-01 9:24 ` Vitaly Kuznetsov
0 siblings, 1 reply; 26+ messages in thread
From: David Vrabel @ 2016-06-29 12:36 UTC (permalink / raw)
To: Vitaly Kuznetsov
Cc: xen-devel, linux-kernel, x86, Stefano Stabellini,
Boris Ostrovsky, Juergen Gross, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Julien Grall
On 29/06/16 10:16, Vitaly Kuznetsov wrote:
> David Vrabel <david.vrabel@citrix.com> writes:
>
>> On 28/06/16 17:47, Vitaly Kuznetsov wrote:
>>> It may happen that Xen's and Linux's ideas of vCPU id diverge. In
>>> particular, when we crash on a secondary vCPU we may want to do kdump
>>> and unlike plain kexec where we do migrate_to_reboot_cpu() we try booting
>>> on the vCPU which crashed. This doesn't work very well for PVHVM guests as
>>> we have a number of hypercalls where we pass vCPU id as a parameter. These
>>> hypercalls either fail or do something unexpected. To solve the issue we
>>> need to have a mapping between Linux's and Xen's vCPU ids.
>>
>> Could the soft-reboot hypercall (optionally) return on vcpu 0?
>>
>
> In theory, yes, I think we can re-arrange vCPUs inside the hypervisor so
> Linux will get them in the natural order after soft reset.
The series is straight forwards and the concept of the guest having to
map its idea of CPU to VCPU is fine, so unless you think a hypervisor
based solution is better we can take this series once it's fixed up.
David
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH linux 0/8] xen: pvhvm: support bootup on secondary vCPUs
2016-06-29 12:36 ` David Vrabel
@ 2016-07-01 9:24 ` Vitaly Kuznetsov
0 siblings, 0 replies; 26+ messages in thread
From: Vitaly Kuznetsov @ 2016-07-01 9:24 UTC (permalink / raw)
To: David Vrabel
Cc: xen-devel, linux-kernel, x86, Stefano Stabellini,
Boris Ostrovsky, Juergen Gross, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Julien Grall
David Vrabel <david.vrabel@citrix.com> writes:
> On 29/06/16 10:16, Vitaly Kuznetsov wrote:
>> David Vrabel <david.vrabel@citrix.com> writes:
>>
>>> On 28/06/16 17:47, Vitaly Kuznetsov wrote:
>>>> It may happen that Xen's and Linux's ideas of vCPU id diverge. In
>>>> particular, when we crash on a secondary vCPU we may want to do kdump
>>>> and unlike plain kexec where we do migrate_to_reboot_cpu() we try booting
>>>> on the vCPU which crashed. This doesn't work very well for PVHVM guests as
>>>> we have a number of hypercalls where we pass vCPU id as a parameter. These
>>>> hypercalls either fail or do something unexpected. To solve the issue we
>>>> need to have a mapping between Linux's and Xen's vCPU ids.
>>>
>>> Could the soft-reboot hypercall (optionally) return on vcpu 0?
>>>
>>
>> In theory, yes, I think we can re-arrange vCPUs inside the hypervisor so
>> Linux will get them in the natural order after soft reset.
>
> The series is straight forwards and the concept of the guest having to
> map its idea of CPU to VCPU is fine, so unless you think a hypervisor
> based solution is better we can take this series once it's fixed up.
I'm afraid that kdump is not the only case where the mapping may come
handy. Linux CPU ids are not guaranteed to always match the order of
CPUs in MADT, e.g. if we fail to add a CPU ids for the rest will get
shifted. This could easily be emulated by adding 'disable_cpu_apicid='
kernel parameter. I didn't try but I'm pretty sure that disabling any
CPU will break guest bootup. That said, I think maintaining the
Xen-Linux vCPU id mapping is preferred till we introduce an option to
address vCPUs in hypercalls by some other ids (vLAPIC id, for example).
--
Vitaly
^ permalink raw reply [flat|nested] 26+ messages in thread