[PATCH 00/10] x86: Add support for running as secondary Jailhouse guest

[PATCH 00/10] x86: Add support for running as secondary Jailhouse guest

[Jan Kiszka]

This series paves the way to run Linux in so-called non-root cells
(guest partitions) of the Jailhouse hypervisor.

Jailhouse [1] was started 4 years ago as an open-source (GPL) leight-
weight hypervisor that statically partitions SMP systems. It's unique in
that it uses one Linux instance, the root cell, as boot loader and
management console. Jailhouse targets use cases for hard real-time and
safety-critical systems that KVM cannot cater due to its inherent
complexity.

Jaihouse can run bare-metal, free and closed-source RTOSes as secondary
guests and, with this series, also x86 Linux instances. While ARM and
ARM64 non-root Linux guests are feasible without extra patches, thanks
to the high configurability via device trees, x86 requires special
platform support, mostly to step away from non-existing resources in a
non-root Jailhouse cell.

This series ensures that Linux can boot in a non-root cell, including
SMP cells, has working timekeeping and can use the platform UARTs and
PCI devices as assigned to it. In follow-up series, we will propose
optimizations and enhancements for the PCI support, a simplistic debug
console, and some improvement for Linux guests on ARM.

What is not yet in upstream-ready state is a driver for inter-cell
communication. The current implementation of virtual peer-to-peer
network [2] uses an enhanced version of the QEMU ivshmem shared memory
device. However we still need to finish the evaluation of virtio /
vhost-pci options prior to settling over the final interface.

This patch series is also available at

git://git.kiszka.org/linux.git d0036688b2da

Jan

[1] http://jailhouse-project.org
[2] http://git.kiszka.org/?p=linux.git;a=shortlog;h=refs/heads/queues/jailhouse

Jan Kiszka (10):
  x86/apic: Install an empty physflat_init_apic_ldr
  x86: jailhouse: Add infrastructure for running in non-root cell
  x86: jailhouse: Enable APIC and SMP support
  x86: jailhouse: Enable PMTIMER
  x86: jailhouse: Set up timekeeping
  x86: jailhouse: Avoid access of unsupported platform resources
  x86: jailhouse: Silence ACPI warning
  x86: jailhouse: Halt instead of failing to restart
  x86: jailhouse: Wire up IOAPIC for legacy UART ports
  x86: jailhouse: Initialize PCI support

 arch/x86/Kconfig                      |  13 ++
 arch/x86/include/asm/hypervisor.h     |   1 +
 arch/x86/include/asm/jailhouse_para.h |  27 ++++
 arch/x86/include/asm/tsc.h            |   3 +
 arch/x86/kernel/Makefile              |   2 +
 arch/x86/kernel/apic/apic_flat_64.c   |  12 +-
 arch/x86/kernel/cpu/hypervisor.c      |   4 +
 arch/x86/kernel/jailhouse.c           | 252 ++++++++++++++++++++++++++++++++++
 arch/x86/kernel/smpboot.c             |   7 +-
 arch/x86/kernel/tsc.c                 |  14 +-
 drivers/acpi/Kconfig                  |  32 ++---
 11 files changed, 339 insertions(+), 28 deletions(-)
 create mode 100644 arch/x86/include/asm/jailhouse_para.h
 create mode 100644 arch/x86/kernel/jailhouse.c

-- 
2.12.3

[PATCH 01/10] x86/apic: Install an empty physflat_init_apic_ldr

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

As the comment already stated, there is no need for setting up LDR in
physflat mode as it remains unused. flat_init_apic_ldr only served as a
placeholder for a nop operation so far, causing no harm.

That will change when running over the Jailhouse hypervisor. Here we
must not touch LDR in a way that destroys the mapping originally set up
by the Linux root cell. Jailhouse enforces this setting in order to
efficiently validate any IPI requests sent by a cell.

Avoid a needless clash caused by flat_init_apic_ldr by installing a true
nop handler.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 arch/x86/kernel/apic/apic_flat_64.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/apic/apic_flat_64.c b/arch/x86/kernel/apic/apic_flat_64.c
index aa85690e9b64..34bfdfe29a04 100644
--- a/arch/x86/kernel/apic/apic_flat_64.c
+++ b/arch/x86/kernel/apic/apic_flat_64.c
@@ -218,6 +218,11 @@ static int physflat_acpi_madt_oem_check(char *oem_id, char *oem_table_id)
 	return 0;
 }
 
+void physflat_init_apic_ldr(void)
+{
+	/* LDR is not used in physflat mode. */
+}
+
 static void physflat_send_IPI_allbutself(int vector)
 {
 	default_send_IPI_mask_allbutself_phys(cpu_online_mask, vector);
@@ -251,8 +256,7 @@ static struct apic apic_physflat __ro_after_init = {
 	.dest_logical			= 0,
 	.check_apicid_used		= NULL,
 
-	/* not needed, but shouldn't hurt: */
-	.init_apic_ldr			= flat_init_apic_ldr,
+	.init_apic_ldr			= physflat_init_apic_ldr,
 
 	.ioapic_phys_id_map		= NULL,
 	.setup_apic_routing		= NULL,
-- 
2.12.3

[PATCH 02/10] x86: jailhouse: Add infrastructure for running in non-root cell

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

The Jailhouse hypervisor is able to statically partition a multicore
system into multiple so-called cells. Linux is used as boot loader and
continues to run in the root cell after Jailhouse is enabled. Linux can
also run in non-root cells.

Jailhouse does not emulate usual x86 devices. It also provides no
complex ACPI but basic platform information that the boot loader
forwards via setup data. This adds the infrastructure to detect when
running in a non-root cell so that the platform can be configured as
required in succeeding steps.

Support is limited to x86-64 so far, primarily because no boot loader
stub exists for i386 and, thus, we wouldn't be able to test the 32-bit
path.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 arch/x86/Kconfig                      | 12 ++++++
 arch/x86/include/asm/hypervisor.h     |  1 +
 arch/x86/include/asm/jailhouse_para.h | 27 +++++++++++++
 arch/x86/kernel/Makefile              |  2 +
 arch/x86/kernel/cpu/hypervisor.c      |  4 ++
 arch/x86/kernel/jailhouse.c           | 76 +++++++++++++++++++++++++++++++++++
 6 files changed, 122 insertions(+)
 create mode 100644 arch/x86/include/asm/jailhouse_para.h
 create mode 100644 arch/x86/kernel/jailhouse.c

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index df3276d6bfe3..c5f4f4683b51 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -782,6 +782,18 @@ config KVM_DEBUG_FS
 	  Statistics are displayed in debugfs filesystem. Enabling this option
 	  may incur significant overhead.
 
+config JAILHOUSE_GUEST
+	bool "Jailhouse non-root cell support"
+	depends on PARAVIRT && X86_64
+	---help---
+	  This option allows to run Linux as guest in a Jailhouse non-root
+	  cell. You can leave this option disabled if you only want to start
+	  Jailhouse and run Linux afterwards in the root cell.
+
+	  You likely also want to disable CONFIG_SUSPEND and CONFIG_SERIO to
+	  avoid access to I/O resources that are usually not assigned to the
+	  non-root cell.
+
 config PARAVIRT_TIME_ACCOUNTING
 	bool "Paravirtual steal time accounting"
 	depends on PARAVIRT
diff --git a/arch/x86/include/asm/hypervisor.h b/arch/x86/include/asm/hypervisor.h
index 1b0a5abcd8ae..376085cb6244 100644
--- a/arch/x86/include/asm/hypervisor.h
+++ b/arch/x86/include/asm/hypervisor.h
@@ -37,6 +37,7 @@ enum x86_hypervisor_type {
 	X86_HYPER_XEN_PV,
 	X86_HYPER_XEN_HVM,
 	X86_HYPER_KVM,
+	X86_HYPER_JAILHOUSE,
 };
 
 struct hypervisor_x86 {
diff --git a/arch/x86/include/asm/jailhouse_para.h b/arch/x86/include/asm/jailhouse_para.h
new file mode 100644
index 000000000000..06a5f41d5451
--- /dev/null
+++ b/arch/x86/include/asm/jailhouse_para.h
@@ -0,0 +1,27 @@
+/*
+ * Jailhouse paravirt_ops implementation
+ *
+ * Copyright (c) Siemens AG, 2015-2017
+ *
+ * Authors:
+ *  Jan Kiszka <jan.kiszka@siemens.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ */
+
+#ifndef _ASM_X86_JAILHOUSE_PARA_H
+#define _ASM_X86_JAILHOUSE_PARA_H
+
+#include <linux/types.h>
+
+#ifdef CONFIG_JAILHOUSE_GUEST
+bool jailhouse_paravirt(void);
+#else
+static inline bool jailhouse_paravirt(void)
+{
+	return false;
+}
+#endif
+
+#endif /* _ASM_X86_JAILHOUSE_PARA_H */
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index 81bb565f4497..aed9296dccd3 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -112,6 +112,8 @@ obj-$(CONFIG_PARAVIRT_SPINLOCKS)+= paravirt-spinlocks.o
 obj-$(CONFIG_PARAVIRT_CLOCK)	+= pvclock.o
 obj-$(CONFIG_X86_PMEM_LEGACY_DEVICE) += pmem.o
 
+obj-$(CONFIG_JAILHOUSE_GUEST)	+= jailhouse.o
+
 obj-$(CONFIG_EISA)		+= eisa.o
 obj-$(CONFIG_PCSPKR_PLATFORM)	+= pcspeaker.o
 
diff --git a/arch/x86/kernel/cpu/hypervisor.c b/arch/x86/kernel/cpu/hypervisor.c
index bea8d3e24f50..479ca4728de0 100644
--- a/arch/x86/kernel/cpu/hypervisor.c
+++ b/arch/x86/kernel/cpu/hypervisor.c
@@ -31,6 +31,7 @@ extern const struct hypervisor_x86 x86_hyper_ms_hyperv;
 extern const struct hypervisor_x86 x86_hyper_xen_pv;
 extern const struct hypervisor_x86 x86_hyper_xen_hvm;
 extern const struct hypervisor_x86 x86_hyper_kvm;
+extern const struct hypervisor_x86 x86_hyper_jailhouse;
 
 static const __initconst struct hypervisor_x86 * const hypervisors[] =
 {
@@ -45,6 +46,9 @@ static const __initconst struct hypervisor_x86 * const hypervisors[] =
 #ifdef CONFIG_KVM_GUEST
 	&x86_hyper_kvm,
 #endif
+#ifdef CONFIG_JAILHOUSE_GUEST
+	&x86_hyper_jailhouse,
+#endif
 };
 
 enum x86_hypervisor_type x86_hyper_type;
diff --git a/arch/x86/kernel/jailhouse.c b/arch/x86/kernel/jailhouse.c
new file mode 100644
index 000000000000..bc0f49a6172d
--- /dev/null
+++ b/arch/x86/kernel/jailhouse.c
@@ -0,0 +1,76 @@
+/*
+ * Jailhouse paravirt_ops implementation
+ *
+ * Copyright (c) Siemens AG, 2015-2017
+ *
+ * Authors:
+ *  Jan Kiszka <jan.kiszka@siemens.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <linux/kernel.h>
+#include <asm/cpu.h>
+#include <asm/hypervisor.h>
+#include <asm/setup.h>
+
+#define SETUP_JAILHOUSE		0x53484c4a /* "JLHS" */
+
+#define SETUP_REQUIRED_VERSION	1
+
+/*
+ * The boot loader is passing platform information via this Jailhouse-specific
+ * setup data structure.
+ */
+struct jailhouse_setup_data {
+	struct setup_data header;
+	u16 version;
+	u16 compatible_version;
+	u16 pm_timer_address;
+	u16 num_cpus;
+	u64 pci_mmconfig_base;
+	u8 standard_ioapic;
+	u8 cpu_ids[255];
+};
+
+static uint32_t jailhouse_cpuid_base(void)
+{
+	if (boot_cpu_data.cpuid_level < 0 ||
+	    !boot_cpu_has(X86_FEATURE_HYPERVISOR))
+		return 0;
+
+	return hypervisor_cpuid_base("Jailhouse\0\0\0", 0);
+}
+
+static uint32_t __init jailhouse_detect(void)
+{
+	return jailhouse_cpuid_base();
+}
+
+static void __init jailhouse_init_platform(void)
+{
+	u64 pa_data = boot_params.hdr.setup_data;
+	struct jailhouse_setup_data *data;
+
+	data = early_memremap(pa_data, sizeof(*data));
+
+	if (data->header.type != SETUP_JAILHOUSE ||
+	    data->compatible_version > SETUP_REQUIRED_VERSION)
+		panic("Jailhouse: Unsupported setup data structure");
+
+	early_memunmap(data, sizeof(*data));
+}
+
+bool jailhouse_paravirt(void)
+{
+	return jailhouse_cpuid_base() != 0;
+}
+
+const struct hypervisor_x86 x86_hyper_jailhouse __refconst = {
+	.name = "Jailhouse",
+	.detect = jailhouse_detect,
+	.init = {
+		.init_platform = jailhouse_init_platform,
+	},
+};
-- 
2.12.3

[PATCH 03/10] x86: jailhouse: Enable APIC and SMP support

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Register the APIC which Jailhouse always exposes at 0xfee00000 if in
xAPIC mode or via MSRs as x2APIC. The latter is only available if it was
already activated because there is no virtualization to switch its mode
during runtime.

Jailhouse requires the APIC to be operated in phys-flat mode. Ensure
that this mode is selected by Linux.

The available CPUs are taken from the setup data structure that the
loader filled and registered with the kernel.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 arch/x86/kernel/apic/apic_flat_64.c |  4 +++-
 arch/x86/kernel/jailhouse.c         | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 37 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/apic/apic_flat_64.c b/arch/x86/kernel/apic/apic_flat_64.c
index 34bfdfe29a04..56d22b4f9ffd 100644
--- a/arch/x86/kernel/apic/apic_flat_64.c
+++ b/arch/x86/kernel/apic/apic_flat_64.c
@@ -19,6 +19,7 @@
 #include <asm/smp.h>
 #include <asm/apic.h>
 #include <asm/ipi.h>
+#include <asm/jailhouse_para.h>
 
 #include <linux/acpi.h>
 
@@ -235,7 +236,8 @@ static void physflat_send_IPI_all(int vector)
 
 static int physflat_probe(void)
 {
-	if (apic == &apic_physflat || num_possible_cpus() > 8)
+	if (apic == &apic_physflat || num_possible_cpus() > 8 ||
+	    jailhouse_paravirt())
 		return 1;
 
 	return 0;
diff --git a/arch/x86/kernel/jailhouse.c b/arch/x86/kernel/jailhouse.c
index bc0f49a6172d..ce9416c70656 100644
--- a/arch/x86/kernel/jailhouse.c
+++ b/arch/x86/kernel/jailhouse.c
@@ -11,6 +11,7 @@
  */
 
 #include <linux/kernel.h>
+#include <asm/apic.h>
 #include <asm/cpu.h>
 #include <asm/hypervisor.h>
 #include <asm/setup.h>
@@ -48,10 +49,16 @@ static uint32_t __init jailhouse_detect(void)
 	return jailhouse_cpuid_base();
 }
 
+static unsigned int x2apic_get_apic_id(unsigned long id)
+{
+        return id;
+}
+
 static void __init jailhouse_init_platform(void)
 {
 	u64 pa_data = boot_params.hdr.setup_data;
 	struct jailhouse_setup_data *data;
+	unsigned int cpu;
 
 	data = early_memremap(pa_data, sizeof(*data));
 
@@ -59,6 +66,23 @@ static void __init jailhouse_init_platform(void)
 	    data->compatible_version > SETUP_REQUIRED_VERSION)
 		panic("Jailhouse: Unsupported setup data structure");
 
+#ifdef CONFIG_X86_X2APIC
+	/*
+	 * Register x2APIC handlers early. We need them when running
+	 * register_lapic_address.
+	 */
+	if (x2apic_enabled()) {
+		apic->read = native_apic_msr_read;
+		apic->write = native_apic_msr_write;
+		apic->get_apic_id = x2apic_get_apic_id;
+	}
+#endif
+	register_lapic_address(0xfee00000);
+	for (cpu = 0; cpu < data->num_cpus; cpu++)
+		generic_processor_info(data->cpu_ids[cpu],
+				       boot_cpu_apic_version);
+	smp_found_config = 1;
+
 	early_memunmap(data, sizeof(*data));
 }
 
@@ -67,10 +91,20 @@ bool jailhouse_paravirt(void)
 	return jailhouse_cpuid_base() != 0;
 }
 
+static bool jailhouse_x2apic_available(void)
+{
+	/*
+	 * The x2APIC is only available if the root cell enabled it. Jailhouse
+	 * does not support switching between xAPIC and x2APIC.
+	 */
+	return x2apic_enabled();
+}
+
 const struct hypervisor_x86 x86_hyper_jailhouse __refconst = {
 	.name = "Jailhouse",
 	.detect = jailhouse_detect,
 	.init = {
 		.init_platform = jailhouse_init_platform,
+		.x2apic_available = jailhouse_x2apic_available,
 	},
 };
-- 
2.12.3

[PATCH 04/10] x86: jailhouse: Enable PMTIMER

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Jailhouse exposes the PMTIMER as only reference clock to all cells. Pick
up its address from the setup data. Allow to enable the Linux support of
it by relaxing its strict dependency on ACPI.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 arch/x86/Kconfig            |  1 +
 arch/x86/kernel/jailhouse.c |  4 ++++
 drivers/acpi/Kconfig        | 32 ++++++++++++++++----------------
 3 files changed, 21 insertions(+), 16 deletions(-)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index c5f4f4683b51..6976c035ea63 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -785,6 +785,7 @@ config KVM_DEBUG_FS
 config JAILHOUSE_GUEST
 	bool "Jailhouse non-root cell support"
 	depends on PARAVIRT && X86_64
+	select X86_PM_TIMER
 	---help---
 	  This option allows to run Linux as guest in a Jailhouse non-root
 	  cell. You can leave this option disabled if you only want to start
diff --git a/arch/x86/kernel/jailhouse.c b/arch/x86/kernel/jailhouse.c
index ce9416c70656..f7e99f7a8873 100644
--- a/arch/x86/kernel/jailhouse.c
+++ b/arch/x86/kernel/jailhouse.c
@@ -10,6 +10,7 @@
  * the COPYING file in the top-level directory.
  */
 
+#include <linux/acpi_pmtmr.h>
 #include <linux/kernel.h>
 #include <asm/apic.h>
 #include <asm/cpu.h>
@@ -66,6 +67,9 @@ static void __init jailhouse_init_platform(void)
 	    data->compatible_version > SETUP_REQUIRED_VERSION)
 		panic("Jailhouse: Unsupported setup data structure");
 
+	pmtmr_ioport = data->pm_timer_address;
+	pr_debug("Jailhouse: PM-Timer IO Port: %#x\n", pmtmr_ioport);
+
 #ifdef CONFIG_X86_X2APIC
 	/*
 	 * Register x2APIC handlers early. We need them when running
diff --git a/drivers/acpi/Kconfig b/drivers/acpi/Kconfig
index 91477d5ab422..c054f9b4f1eb 100644
--- a/drivers/acpi/Kconfig
+++ b/drivers/acpi/Kconfig
@@ -361,22 +361,6 @@ config ACPI_PCI_SLOT
 	  i.e., segment/bus/device/function tuples, with physical slots in
 	  the system.  If you are unsure, say N.
 
-config X86_PM_TIMER
-	bool "Power Management Timer Support" if EXPERT
-	depends on X86
-	default y
-	help
-	  The Power Management Timer is available on all ACPI-capable,
-	  in most cases even if ACPI is unusable or blacklisted.
-
-	  This timing source is not affected by power management features
-	  like aggressive processor idling, throttling, frequency and/or
-	  voltage scaling, unlike the commonly used Time Stamp Counter
-	  (TSC) timing source.
-
-	  You should nearly always say Y here because many modern
-	  systems require this timer. 
-
 config ACPI_CONTAINER
 	bool "Container and Module Devices"
 	default (ACPI_HOTPLUG_MEMORY || ACPI_HOTPLUG_CPU)
@@ -558,3 +542,19 @@ config TPS68470_PMIC_OPREGION
 	  using this, are probed.
 
 endif	# ACPI
+
+config X86_PM_TIMER
+	bool "Power Management Timer Support" if EXPERT
+	depends on X86 && (ACPI || JAILHOUSE_GUEST)
+	default y
+	help
+	  The Power Management Timer is available on all ACPI-capable,
+	  in most cases even if ACPI is unusable or blacklisted.
+
+	  This timing source is not affected by power management features
+	  like aggressive processor idling, throttling, frequency and/or
+	  voltage scaling, unlike the commonly used Time Stamp Counter
+	  (TSC) timing source.
+
+	  You should nearly always say Y here because many modern
+	  systems require this timer.
-- 
2.12.3

[PATCH 05/10] x86: jailhouse: Set up timekeeping

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Calibrate the TSC and, where necessary, the APIC timer against the
TMTIMER. We need our own implementation as neither the PIC nor the HPET
are available, and the standard calibration routines try to make use of
them.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 arch/x86/include/asm/tsc.h  |  3 ++
 arch/x86/kernel/jailhouse.c | 82 +++++++++++++++++++++++++++++++++++++++++++++
 arch/x86/kernel/tsc.c       | 14 ++++----
 3 files changed, 92 insertions(+), 7 deletions(-)

diff --git a/arch/x86/include/asm/tsc.h b/arch/x86/include/asm/tsc.h
index cf5d53c3f9ea..be92e0c8ac17 100644
--- a/arch/x86/include/asm/tsc.h
+++ b/arch/x86/include/asm/tsc.h
@@ -71,4 +71,7 @@ extern void tsc_restore_sched_clock_state(void);
 
 unsigned long cpu_khz_from_msr(void);
 
+u64 tsc_read_refs(u64 *ref, int hpet);
+unsigned long tsc_calc_pmtimer_ref(u64 deltatsc, u64 pm1, u64 pm2);
+
 #endif /* _ASM_X86_TSC_H */
diff --git a/arch/x86/kernel/jailhouse.c b/arch/x86/kernel/jailhouse.c
index f7e99f7a8873..8e5b2f0c8a34 100644
--- a/arch/x86/kernel/jailhouse.c
+++ b/arch/x86/kernel/jailhouse.c
@@ -50,6 +50,83 @@ static uint32_t __init jailhouse_detect(void)
 	return jailhouse_cpuid_base();
 }
 
+#define MAX_RETRIES	5
+#define SMI_TRESHOLD	50000
+
+static unsigned long apic_timer_access(u64 *pmt, bool setup)
+{
+	unsigned long ret = 0;
+	unsigned int n;
+	u64 t1, t2;
+
+	for (n = 0; n < MAX_RETRIES; n++) {
+		t1 = get_cycles();
+		*pmt = acpi_pm_read_early();
+		if (setup)
+			apic_write(APIC_TMICT, 0xffffffff);
+		else
+			ret = apic_read(APIC_TMCCT);
+		t2 = get_cycles();
+
+		if ((t2 - t1) < SMI_TRESHOLD * 2)
+			return ret;
+	}
+
+	panic("Jailhouse: SMI disturbed APIC timer calibration");
+}
+
+static void jailhouse_timer_init(void)
+{
+	u64 divided_apic_freq;
+	unsigned long tmr;
+	u64 start, end;
+
+	if (boot_cpu_has(X86_FEATURE_TSC_DEADLINE_TIMER))
+		return;
+
+	apic_write(APIC_LVTT, APIC_LVT_MASKED);
+	apic_write(APIC_TDCR, APIC_TDR_DIV_16);
+
+	apic_timer_access(&start, true);
+	while ((acpi_pm_read_early() - start) < 100000)
+		cpu_relax();
+	tmr = apic_timer_access(&end, false);
+
+	divided_apic_freq =
+		tsc_calc_pmtimer_ref((0xffffffffU - tmr) * 1000000, start, end);
+
+	lapic_timer_frequency = divided_apic_freq * 16;
+	apic_write(APIC_TMICT, 0);
+}
+
+static unsigned long jailhouse_calibrate_cpu(void)
+{
+	u64 tsc1, tsc2, pm1, pm2;
+	unsigned long flags;
+
+	local_irq_save(flags);
+
+	tsc1 = tsc_read_refs(&pm1, 0);
+	while ((get_cycles() - tsc1) < 50000000)
+		cpu_relax();
+	tsc2 = tsc_read_refs(&pm2, 0);
+
+	local_irq_restore(flags);
+
+	/* Check, whether the sampling succeeded (SMI?) */
+	if (tsc1 == ULLONG_MAX || tsc2 == ULLONG_MAX) {
+		pr_err("Jailhouse: TSC calibration against PMTIMER failed\n");
+		return 0;
+	}
+
+	return tsc_calc_pmtimer_ref((tsc2 - tsc1) * 1000000, pm1, pm2);
+}
+
+static unsigned long jailhouse_calibrate_tsc(void)
+{
+	return 0;
+}
+
 static unsigned int x2apic_get_apic_id(unsigned long id)
 {
         return id;
@@ -61,6 +138,11 @@ static void __init jailhouse_init_platform(void)
 	struct jailhouse_setup_data *data;
 	unsigned int cpu;
 
+	x86_init.timers.timer_init	= jailhouse_timer_init;
+
+	x86_platform.calibrate_cpu = jailhouse_calibrate_cpu;
+	x86_platform.calibrate_tsc = jailhouse_calibrate_tsc;
+
 	data = early_memremap(pa_data, sizeof(*data));
 
 	if (data->header.type != SETUP_JAILHOUSE ||
diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c
index 8ea117f8142e..a61000cc4f21 100644
--- a/arch/x86/kernel/tsc.c
+++ b/arch/x86/kernel/tsc.c
@@ -286,7 +286,7 @@ __setup("tsc=", tsc_setup);
 /*
  * Read TSC and the reference counters. Take care of SMI disturbance
  */
-static u64 tsc_read_refs(u64 *p, int hpet)
+u64 tsc_read_refs(u64 *p, int hpet)
 {
 	u64 t1, t2;
 	int i;
@@ -307,7 +307,7 @@ static u64 tsc_read_refs(u64 *p, int hpet)
 /*
  * Calculate the TSC frequency from HPET reference
  */
-static unsigned long calc_hpet_ref(u64 deltatsc, u64 hpet1, u64 hpet2)
+static unsigned long tsc_calc_hpet_ref(u64 deltatsc, u64 hpet1, u64 hpet2)
 {
 	u64 tmp;
 
@@ -324,7 +324,7 @@ static unsigned long calc_hpet_ref(u64 deltatsc, u64 hpet1, u64 hpet2)
 /*
  * Calculate the TSC frequency from PMTimer reference
  */
-static unsigned long calc_pmtimer_ref(u64 deltatsc, u64 pm1, u64 pm2)
+unsigned long tsc_calc_pmtimer_ref(u64 deltatsc, u64 pm1, u64 pm2)
 {
 	u64 tmp;
 
@@ -728,9 +728,9 @@ unsigned long native_calibrate_cpu(void)
 
 		tsc2 = (tsc2 - tsc1) * 1000000LL;
 		if (hpet)
-			tsc2 = calc_hpet_ref(tsc2, ref1, ref2);
+			tsc2 = tsc_calc_hpet_ref(tsc2, ref1, ref2);
 		else
-			tsc2 = calc_pmtimer_ref(tsc2, ref1, ref2);
+			tsc2 = tsc_calc_pmtimer_ref(tsc2, ref1, ref2);
 
 		tsc_ref_min = min(tsc_ref_min, (unsigned long) tsc2);
 
@@ -1200,9 +1200,9 @@ static void tsc_refine_calibration_work(struct work_struct *work)
 	delta = tsc_stop - tsc_start;
 	delta *= 1000000LL;
 	if (hpet)
-		freq = calc_hpet_ref(delta, ref_start, ref_stop);
+		freq = tsc_calc_hpet_ref(delta, ref_start, ref_stop);
 	else
-		freq = calc_pmtimer_ref(delta, ref_start, ref_stop);
+		freq = tsc_calc_pmtimer_ref(delta, ref_start, ref_stop);
 
 	/* Make sure we're within 1% */
 	if (abs(tsc_khz - freq) > tsc_khz/100)
-- 
2.12.3

[PATCH 06/10] x86: jailhouse: Avoid access of unsupported platform resources

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

We don't have CMOS access, thus we can't set the warm-reset vectors in
do_boot_cpu. There is no RTC, thus also no wall clock. Furthermore,
there are no ISA IRQs and no PIC. So fill the platform callbacks
accordingly.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 arch/x86/kernel/jailhouse.c | 10 ++++++++++
 arch/x86/kernel/smpboot.c   |  7 +++++--
 2 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/jailhouse.c b/arch/x86/kernel/jailhouse.c
index 8e5b2f0c8a34..cfe8ae0c33a2 100644
--- a/arch/x86/kernel/jailhouse.c
+++ b/arch/x86/kernel/jailhouse.c
@@ -15,6 +15,7 @@
 #include <asm/apic.h>
 #include <asm/cpu.h>
 #include <asm/hypervisor.h>
+#include <asm/i8259.h>
 #include <asm/setup.h>
 
 #define SETUP_JAILHOUSE		0x53484c4a /* "JLHS" */
@@ -50,6 +51,11 @@ static uint32_t __init jailhouse_detect(void)
 	return jailhouse_cpuid_base();
 }
 
+static void jailhouse_get_wallclock(struct timespec *now)
+{
+	memset(now, 0, sizeof(*now));
+}
+
 #define MAX_RETRIES	5
 #define SMI_TRESHOLD	50000
 
@@ -139,7 +145,11 @@ static void __init jailhouse_init_platform(void)
 	unsigned int cpu;
 
 	x86_init.timers.timer_init	= jailhouse_timer_init;
+	x86_init.irqs.pre_vector_init	= x86_init_noop;
+	legacy_pic			= &null_legacy_pic;
 
+	x86_platform.legacy.rtc = 0;
+	x86_platform.get_wallclock = jailhouse_get_wallclock;
 	x86_platform.calibrate_cpu = jailhouse_calibrate_cpu;
 	x86_platform.calibrate_tsc = jailhouse_calibrate_tsc;
 
diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index 5f59e6bee123..81339dbafeba 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -78,6 +78,7 @@
 #include <asm/realmode.h>
 #include <asm/misc.h>
 #include <asm/qspinlock.h>
+#include <asm/jailhouse_para.h>
 
 /* Number of siblings per CPU package */
 int smp_num_siblings = 1;
@@ -1006,7 +1007,8 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle,
 	 * the targeted processor.
 	 */
 
-	if (get_uv_system_type() != UV_NON_UNIQUE_APIC) {
+	if (get_uv_system_type() != UV_NON_UNIQUE_APIC &&
+	    !jailhouse_paravirt()) {
 
 		pr_debug("Setting warm reset code and vector.\n");
 
@@ -1078,7 +1080,8 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle,
 	/* mark "stuck" area as not stuck */
 	*trampoline_status = 0;
 
-	if (get_uv_system_type() != UV_NON_UNIQUE_APIC) {
+	if (get_uv_system_type() != UV_NON_UNIQUE_APIC &&
+	    !jailhouse_paravirt()) {
 		/*
 		 * Cleanup possible dangling ends...
 		 */
-- 
2.12.3

[PATCH 07/10] x86: jailhouse: Silence ACPI warning

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Jailhouse support does not depend on ACPI, and does not even use it. But
if it should be enabled, avoid warning about its absence in the
platform.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 arch/x86/kernel/jailhouse.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/arch/x86/kernel/jailhouse.c b/arch/x86/kernel/jailhouse.c
index cfe8ae0c33a2..c1ed7fcd7bfa 100644
--- a/arch/x86/kernel/jailhouse.c
+++ b/arch/x86/kernel/jailhouse.c
@@ -180,6 +180,12 @@ static void __init jailhouse_init_platform(void)
 	smp_found_config = 1;
 
 	early_memunmap(data, sizeof(*data));
+
+	/*
+	 * Avoid that the kernel complains about missing ACPI tables - there
+	 * are none in a non-root cell.
+	 */
+	disable_acpi();
 }
 
 bool jailhouse_paravirt(void)
-- 
2.12.3

[PATCH 08/10] x86: jailhouse: Halt instead of failing to restart

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Jailhouse provides no guest-initiated restart. So, do not even try to.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 arch/x86/kernel/jailhouse.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/x86/kernel/jailhouse.c b/arch/x86/kernel/jailhouse.c
index c1ed7fcd7bfa..0bcca175c35e 100644
--- a/arch/x86/kernel/jailhouse.c
+++ b/arch/x86/kernel/jailhouse.c
@@ -12,10 +12,12 @@
 
 #include <linux/acpi_pmtmr.h>
 #include <linux/kernel.h>
+#include <linux/reboot.h>
 #include <asm/apic.h>
 #include <asm/cpu.h>
 #include <asm/hypervisor.h>
 #include <asm/i8259.h>
+#include <asm/reboot.h>
 #include <asm/setup.h>
 
 #define SETUP_JAILHOUSE		0x53484c4a /* "JLHS" */
@@ -133,6 +135,12 @@ static unsigned long jailhouse_calibrate_tsc(void)
 	return 0;
 }
 
+static void jailhouse_no_restart(void)
+{
+	pr_notice("Jailhouse: Restart not supported, halting\n");
+	machine_halt();
+}
+
 static unsigned int x2apic_get_apic_id(unsigned long id)
 {
         return id;
@@ -153,6 +161,8 @@ static void __init jailhouse_init_platform(void)
 	x86_platform.calibrate_cpu = jailhouse_calibrate_cpu;
 	x86_platform.calibrate_tsc = jailhouse_calibrate_tsc;
 
+	machine_ops.emergency_restart = jailhouse_no_restart;
+
 	data = early_memremap(pa_data, sizeof(*data));
 
 	if (data->header.type != SETUP_JAILHOUSE ||
-- 
2.12.3

[PATCH 09/10] x86: jailhouse: Wire up IOAPIC for legacy UART ports

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

The typical I/O interrupts in non-root cells are MSI-based. However, the
platform UARTs do not support MSI. In order to run a non-root cell that
shall be use one of them, we need to register the standard IOAPIC and
set 1:1 routing for IRQ 3 and 4.

If an IOAPIC is not available, the boot loader clears standard_ioapic in
the setup data, and we skip the registration. If we should not be
allowed to use one of those pins, Jailhouse will simply ignore our
accesses.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 arch/x86/kernel/jailhouse.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/arch/x86/kernel/jailhouse.c b/arch/x86/kernel/jailhouse.c
index 0bcca175c35e..05459ea0ecc7 100644
--- a/arch/x86/kernel/jailhouse.c
+++ b/arch/x86/kernel/jailhouse.c
@@ -17,6 +17,7 @@
 #include <asm/cpu.h>
 #include <asm/hypervisor.h>
 #include <asm/i8259.h>
+#include <asm/irqdomain.h>
 #include <asm/reboot.h>
 #include <asm/setup.h>
 
@@ -148,6 +149,14 @@ static unsigned int x2apic_get_apic_id(unsigned long id)
 
 static void __init jailhouse_init_platform(void)
 {
+	struct ioapic_domain_cfg ioapic_cfg = {
+		.type = IOAPIC_DOMAIN_STRICT,
+		.ops = &mp_ioapic_irqdomain_ops,
+	};
+	struct mpc_intsrc mp_irq = {
+		.type = MP_INTSRC,
+		.irqtype = mp_INT,
+	};
 	u64 pa_data = boot_params.hdr.setup_data;
 	struct jailhouse_setup_data *data;
 	unsigned int cpu;
@@ -189,6 +198,17 @@ static void __init jailhouse_init_platform(void)
 				       boot_cpu_apic_version);
 	smp_found_config = 1;
 
+	if (data->standard_ioapic) {
+		mp_register_ioapic(0, 0xfec00000, gsi_top, &ioapic_cfg);
+
+		/* Register 1:1 mapping for legacy UART IRQs 3 and 4 */
+		mp_irq.srcbusirq = mp_irq.dstirq = 3;
+		mp_save_irq(&mp_irq);
+
+		mp_irq.srcbusirq = mp_irq.dstirq = 4;
+		mp_save_irq(&mp_irq);
+	}
+
 	early_memunmap(data, sizeof(*data));
 
 	/*
-- 
2.12.3

[PATCH 10/10] x86: jailhouse: Initialize PCI support

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

With this change, PCI devices can be detected and used inside a non-root
cell.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 arch/x86/kernel/jailhouse.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/x86/kernel/jailhouse.c b/arch/x86/kernel/jailhouse.c
index 05459ea0ecc7..e573ea8b0a5f 100644
--- a/arch/x86/kernel/jailhouse.c
+++ b/arch/x86/kernel/jailhouse.c
@@ -18,6 +18,7 @@
 #include <asm/hypervisor.h>
 #include <asm/i8259.h>
 #include <asm/irqdomain.h>
+#include <asm/pci_x86.h>
 #include <asm/reboot.h>
 #include <asm/setup.h>
 
@@ -211,6 +212,15 @@ static void __init jailhouse_init_platform(void)
 
 	early_memunmap(data, sizeof(*data));
 
+	pci_probe = 0;
+	pci_direct_init(1);
+
+	/*
+	 * There are no bridges on the virtual PCI root bus under Jailhouse,
+	 * thus no other way to discover all devices than a full scan.
+	 */
+	pcibios_last_bus = 0xff;
+
 	/*
 	 * Avoid that the kernel complains about missing ACPI tables - there
 	 * are none in a non-root cell.
-- 
2.12.3

Re: [PATCH 01/10] x86/apic: Install an empty physflat_init_apic_ldr

[Thomas Gleixner]

On Thu, 16 Nov 2017, Jan Kiszka wrote:

> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> As the comment already stated, there is no need for setting up LDR in

I prefer to have a proper reference to documentation why this is not
needed than to a comment which is technically dubious.

> physflat mode as it remains unused. flat_init_apic_ldr only served as a
> placeholder for a nop operation so far, causing no harm.
> 
> That will change when running over the Jailhouse hypervisor. Here we
> must not touch LDR in a way that destroys the mapping originally set up
> by the Linux root cell. Jailhouse enforces this setting in order to
> efficiently validate any IPI requests sent by a cell.
> 
> Avoid a needless clash caused by flat_init_apic_ldr by installing a true
> nop handler.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  arch/x86/kernel/apic/apic_flat_64.c | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/x86/kernel/apic/apic_flat_64.c b/arch/x86/kernel/apic/apic_flat_64.c
> index aa85690e9b64..34bfdfe29a04 100644
> --- a/arch/x86/kernel/apic/apic_flat_64.c
> +++ b/arch/x86/kernel/apic/apic_flat_64.c
> @@ -218,6 +218,11 @@ static int physflat_acpi_madt_oem_check(char *oem_id, char *oem_table_id)
>  	return 0;
>  }
>  
> +void physflat_init_apic_ldr(void)

static ?

> +{
> +	/* LDR is not used in physflat mode. */

flat_init_apic_ldr() also fiddles with DFR ...

> +}
> +
>  static void physflat_send_IPI_allbutself(int vector)
>  {
>  	default_send_IPI_mask_allbutself_phys(cpu_online_mask, vector);
> @@ -251,8 +256,7 @@ static struct apic apic_physflat __ro_after_init = {
>  	.dest_logical			= 0,
>  	.check_apicid_used		= NULL,
>  
> -	/* not needed, but shouldn't hurt: */
> -	.init_apic_ldr			= flat_init_apic_ldr,
> +	.init_apic_ldr			= physflat_init_apic_ldr,
>  
>  	.ioapic_phys_id_map		= NULL,
>  	.setup_apic_routing		= NULL,
> -- 
> 2.12.3
> 
> 

Re: [PATCH 02/10] x86: jailhouse: Add infrastructure for running in non-root cell

[Thomas Gleixner]

On Thu, 16 Nov 2017, Jan Kiszka wrote:
  
> +config JAILHOUSE_GUEST
> +	bool "Jailhouse non-root cell support"
> +	depends on PARAVIRT && X86_64
> +	---help---
> +	  This option allows to run Linux as guest in a Jailhouse non-root
> +	  cell. You can leave this option disabled if you only want to start
> +	  Jailhouse and run Linux afterwards in the root cell.
> +
> +	  You likely also want to disable CONFIG_SUSPEND and CONFIG_SERIO to
> +	  avoid access to I/O resources that are usually not assigned to the
> +	  non-root cell.

That should be prevented programatically.

> +#include <linux/kernel.h>
> +#include <asm/cpu.h>
> +#include <asm/hypervisor.h>
> +#include <asm/setup.h>
> +
> +#define SETUP_JAILHOUSE		0x53484c4a /* "JLHS" */
> +
> +#define SETUP_REQUIRED_VERSION	1
> +
> +/*
> + * The boot loader is passing platform information via this Jailhouse-specific
> + * setup data structure.
> + */
> +struct jailhouse_setup_data {
> +	struct setup_data header;
> +	u16 version;
> +	u16 compatible_version;
> +	u16 pm_timer_address;
> +	u16 num_cpus;
> +	u64 pci_mmconfig_base;
> +	u8 standard_ioapic;
> +	u8 cpu_ids[255];

Shouldn't this structure and SETUP_JAILHOUSE be defined in a header file
which can be exported to boot loaders?

Aside of that please write the struct in tabular fashion:

struct jailhouse_setup_data {
	struct setup_data	header;
	u16			version;
	....

> +};
> +
> +bool jailhouse_paravirt(void)
> +{
> +	return jailhouse_cpuid_base() != 0;
> +}
> +
> +const struct hypervisor_x86 x86_hyper_jailhouse __refconst = {
> +	.name = "Jailhouse",
> +	.detect = jailhouse_detect,
> +	.init = {
> +		.init_platform = jailhouse_init_platform,

See above.

Thanks,

	tglx

Re: [PATCH 03/10] x86: jailhouse: Enable APIC and SMP support

[Thomas Gleixner]

On Thu, 16 Nov 2017, Jan Kiszka wrote:

Just noticed, that all prefixes should be 'x86/jailhouse:' please

> +static unsigned int x2apic_get_apic_id(unsigned long id)
> +{
> +        return id;
> +}
> +
>  static void __init jailhouse_init_platform(void)
>  {
>  	u64 pa_data = boot_params.hdr.setup_data;
>  	struct jailhouse_setup_data *data;
> +	unsigned int cpu;
>  
>  	data = early_memremap(pa_data, sizeof(*data));
>  
> @@ -59,6 +66,23 @@ static void __init jailhouse_init_platform(void)
>  	    data->compatible_version > SETUP_REQUIRED_VERSION)
>  		panic("Jailhouse: Unsupported setup data structure");
>  
> +#ifdef CONFIG_X86_X2APIC
> +	/*
> +	 * Register x2APIC handlers early. We need them when running
> +	 * register_lapic_address.
> +	 */
> +	if (x2apic_enabled()) {
> +		apic->read = native_apic_msr_read;
> +		apic->write = native_apic_msr_write;
> +		apic->get_apic_id = x2apic_get_apic_id;

Uurgh. That looks hacky.

> +	}
> +#endif
> +	register_lapic_address(0xfee00000);

No need to do all of this here. The right thing to do is to override

   x86_init.mpparse.get_smp_config

which is invoked after generic_apic_probe() which should select
x2apic_phys, but I might be wrong.


> +	for (cpu = 0; cpu < data->num_cpus; cpu++)
> +		generic_processor_info(data->cpu_ids[cpu],
> +				       boot_cpu_apic_version);

Please add brackets around multi line statements.

Thanks,

	tglx

Re: [PATCH 04/10] x86: jailhouse: Enable PMTIMER

[Thomas Gleixner]

On Thu, 16 Nov 2017, Jan Kiszka wrote:
> Jailhouse exposes the PMTIMER as only reference clock to all cells. Pick
> up its address from the setup data. Allow to enable the Linux support of
> it by relaxing its strict dependency on ACPI.

Yes, that should work w/o problems.

Reviewed-by: Thomas Gleixner <tglx@linutronix.de>

Thanks,

	tglx

Re: [PATCH 05/10] x86: jailhouse: Set up timekeeping

[Thomas Gleixner]

On Thu, 16 Nov 2017, Jan Kiszka wrote:
> Calibrate the TSC and, where necessary, the APIC timer against the
> TMTIMER. We need our own implementation as neither the PIC nor the HPET
> are available, and the standard calibration routines try to make use of
> them.

Why is this needed at all?

The host the frequency already. So this can be done w/o pmtimer and extra
calibration routine.

Thanks,

	tglx

Re: [PATCH 06/10] x86: jailhouse: Avoid access of unsupported platform resources

[Thomas Gleixner]

On Thu, 16 Nov 2017, Jan Kiszka wrote:

> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> We don't have CMOS access, thus we can't set the warm-reset vectors in

s/We/A jailhouse cell/ or s/We/A jailhouse guest/

> do_boot_cpu. There is no RTC, thus also no wall clock. Furthermore,
> there are no ISA IRQs and no PIC. So fill the platform callbacks
> accordingly.
  
> +static void jailhouse_get_wallclock(struct timespec *now)
> +{
> +	memset(now, 0, sizeof(*now));
> +}

> @@ -1006,7 +1007,8 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle,
>  	 * the targeted processor.
>  	 */
>  
> -	if (get_uv_system_type() != UV_NON_UNIQUE_APIC) {
> +	if (get_uv_system_type() != UV_NON_UNIQUE_APIC &&
> +	    !jailhouse_paravirt()) {
>  
>  		pr_debug("Setting warm reset code and vector.\n");

Adding more here might justify a function pointer in x86_platform_ops. Not
sure though, but we have an increasing amount of xen()/kvm()/whatever
conditionals popping up all over the place. And now you add jailhouse()
ones :)

Thanks,

	tglx

Re: [PATCH 06/10] x86: jailhouse: Avoid access of unsupported platform resources

[Thomas Gleixner]

On Fri, 17 Nov 2017, Thomas Gleixner wrote:
> On Thu, 16 Nov 2017, Jan Kiszka wrote:
> 
> > From: Jan Kiszka <jan.kiszka@siemens.com>
> > 
> > We don't have CMOS access, thus we can't set the warm-reset vectors in
> 
> s/We/A jailhouse cell/ or s/We/A jailhouse guest/
> 
> > do_boot_cpu. There is no RTC, thus also no wall clock. Furthermore,
> > there are no ISA IRQs and no PIC. So fill the platform callbacks
> > accordingly.
>   
> > +static void jailhouse_get_wallclock(struct timespec *now)
> > +{
> > +	memset(now, 0, sizeof(*now));
> > +}
> 
> > @@ -1006,7 +1007,8 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle,
> >  	 * the targeted processor.
> >  	 */
> >  
> > -	if (get_uv_system_type() != UV_NON_UNIQUE_APIC) {
> > +	if (get_uv_system_type() != UV_NON_UNIQUE_APIC &&
> > +	    !jailhouse_paravirt()) {
> >  
> >  		pr_debug("Setting warm reset code and vector.\n");
> 
> Adding more here might justify a function pointer in x86_platform_ops. Not
> sure though, but we have an increasing amount of xen()/kvm()/whatever
> conditionals popping up all over the place. And now you add jailhouse()
> ones :)

Actually we don't want a function pointer as we would need two. A simple
flag would be sufficient.

Thanks,

	tglx

Re: [PATCH 09/10] x86: jailhouse: Wire up IOAPIC for legacy UART ports

[Thomas Gleixner]

On Thu, 16 Nov 2017, Jan Kiszka wrote:
>  
>  static void __init jailhouse_init_platform(void)
>  {
> +	struct ioapic_domain_cfg ioapic_cfg = {
> +		.type = IOAPIC_DOMAIN_STRICT,
> +		.ops = &mp_ioapic_irqdomain_ops,
> +	};
> +	struct mpc_intsrc mp_irq = {
> +		.type = MP_INTSRC,
> +		.irqtype = mp_INT,
> +	};
>  	u64 pa_data = boot_params.hdr.setup_data;
>  	struct jailhouse_setup_data *data;
>  	unsigned int cpu;
> @@ -189,6 +198,17 @@ static void __init jailhouse_init_platform(void)
>  				       boot_cpu_apic_version);
>  	smp_found_config = 1;
>  
> +	if (data->standard_ioapic) {
> +		mp_register_ioapic(0, 0xfec00000, gsi_top, &ioapic_cfg);
> +
> +		/* Register 1:1 mapping for legacy UART IRQs 3 and 4 */
> +		mp_irq.srcbusirq = mp_irq.dstirq = 3;
> +		mp_save_irq(&mp_irq);
> +
> +		mp_irq.srcbusirq = mp_irq.dstirq = 4;
> +		mp_save_irq(&mp_irq);

Again, that can run later. And please split out initialization for
particular features into separate functions. Otherwise the parse function
will become a kitchen sink of all sorts.

Thanks,

	tglx

Re: [PATCH 02/10] x86: jailhouse: Add infrastructure for running in non-root cell

[Jan Kiszka]

On 2017-11-17 22:54, Thomas Gleixner wrote:
> On Thu, 16 Nov 2017, Jan Kiszka wrote:
>   
>> +config JAILHOUSE_GUEST
>> +	bool "Jailhouse non-root cell support"
>> +	depends on PARAVIRT && X86_64
>> +	---help---
>> +	  This option allows to run Linux as guest in a Jailhouse non-root
>> +	  cell. You can leave this option disabled if you only want to start
>> +	  Jailhouse and run Linux afterwards in the root cell.
>> +
>> +	  You likely also want to disable CONFIG_SUSPEND and CONFIG_SERIO to
>> +	  avoid access to I/O resources that are usually not assigned to the
>> +	  non-root cell.
> 
> That should be prevented programatically.

Theoretically, serio access could also be assigned to a non-root cell.
But excluding SUSPEND may make sense unconditionally, will check again.

> 
>> +#include <linux/kernel.h>
>> +#include <asm/cpu.h>
>> +#include <asm/hypervisor.h>
>> +#include <asm/setup.h>
>> +
>> +#define SETUP_JAILHOUSE		0x53484c4a /* "JLHS" */
>> +
>> +#define SETUP_REQUIRED_VERSION	1
>> +
>> +/*
>> + * The boot loader is passing platform information via this Jailhouse-specific
>> + * setup data structure.
>> + */
>> +struct jailhouse_setup_data {
>> +	struct setup_data header;
>> +	u16 version;
>> +	u16 compatible_version;
>> +	u16 pm_timer_address;
>> +	u16 num_cpus;
>> +	u64 pci_mmconfig_base;
>> +	u8 standard_ioapic;
>> +	u8 cpu_ids[255];
> 
> Shouldn't this structure and SETUP_JAILHOUSE be defined in a header file
> which can be exported to boot loaders?

Something like arch/x86/include/uapi/asm/jailhouse_setup.h?

Jan

Re: [PATCH 05/10] x86: jailhouse: Set up timekeeping

[Jan Kiszka]

On 2017-11-17 23:49, Thomas Gleixner wrote:
> On Thu, 16 Nov 2017, Jan Kiszka wrote:
>> Calibrate the TSC and, where necessary, the APIC timer against the
>> TMTIMER. We need our own implementation as neither the PIC nor the HPET
>> are available, and the standard calibration routines try to make use of
>> them.
> 
> Why is this needed at all?
> 
> The host the frequency already. So this can be done w/o pmtimer and extra
> calibration routine.

The hypervisor does not have the frequencies. It will never use the APIC
timer (it's owned by the guests), and it has no use case for the TSC so
far. Only the root cell (the Linux that booted the system) has that
data. Now we could

- trust the root cell to provide the right values and export them during
  startup to the hypervisor and from there to the non-root cells.

- calculate the frequencies once and store them in the hyperivsor
  config, just like other system-specific information, for re-export to
  the cells.

But I don't think option 1 will be ok for all use cases. Maybe a
combination of both, falling back to the root cell data if nothing is
defined in the config. Let me think about this.

Jan

Re: [PATCH 00/10] x86: Add support for running as secondary Jailhouse guest

[H. Peter Anvin]

On 11/15/17 23:26, Jan Kiszka wrote:
> This series paves the way to run Linux in so-called non-root cells
> (guest partitions) of the Jailhouse hypervisor.
> 
> Jailhouse [1] was started 4 years ago as an open-source (GPL) leight-
> weight hypervisor that statically partitions SMP systems. It's unique in
> that it uses one Linux instance, the root cell, as boot loader and
> management console. Jailhouse targets use cases for hard real-time and
> safety-critical systems that KVM cannot cater due to its inherent
> complexity.
> 
> Jaihouse can run bare-metal, free and closed-source RTOSes as secondary
> guests and, with this series, also x86 Linux instances. While ARM and
> ARM64 non-root Linux guests are feasible without extra patches, thanks
> to the high configurability via device trees, x86 requires special
> platform support, mostly to step away from non-existing resources in a
> non-root Jailhouse cell.
> 

Could you please write a single summary about the virtualization holes
in Jailhouse that you are papering over?

	-hpa

Re: [PATCH 00/10] x86: Add support for running as secondary Jailhouse guest

[Jan Kiszka]

On 2017-11-18 22:15, H. Peter Anvin wrote:
> On 11/15/17 23:26, Jan Kiszka wrote:
>> This series paves the way to run Linux in so-called non-root cells
>> (guest partitions) of the Jailhouse hypervisor.
>>
>> Jailhouse [1] was started 4 years ago as an open-source (GPL) leight-
>> weight hypervisor that statically partitions SMP systems. It's unique in
>> that it uses one Linux instance, the root cell, as boot loader and
>> management console. Jailhouse targets use cases for hard real-time and
>> safety-critical systems that KVM cannot cater due to its inherent
>> complexity.
>>
>> Jaihouse can run bare-metal, free and closed-source RTOSes as secondary
>> guests and, with this series, also x86 Linux instances. While ARM and
>> ARM64 non-root Linux guests are feasible without extra patches, thanks
>> to the high configurability via device trees, x86 requires special
>> platform support, mostly to step away from non-existing resources in a
>> non-root Jailhouse cell.
>>
> 
> Could you please write a single summary about the virtualization holes
> in Jailhouse that you are papering over?

Actually, we are not virtualizing any full device in Jailhouse. That is
an architectural decision that allows to keep the critical code base
very small (9200 LOC on Intel right now). So, anything that is not there
multiple times is not exposed or problematic state modifications are
blocked.

That leaves the non-root cells with:
- local CPU resources (processor, LAPIC etc.)
- exclusive memory regions
- exclusive PCI devices (or functions)
- read access to the PM timer (as clocksource)
- exclusively assigned pins on the IOAPIC (if any - it's not
  recommended to partition it)
- virtual shared memory devices for inter-cell communication

Thus, the list of non-existing x86 resources and features:
- LPC with all its legacy devices like PIT, PIC, RTC, SMBus etc.
  (unless you decide to pass one through exclusively)
- HPET
- normal PCI bus topology
- BIOS / UEFI firmware services, including ACPI enumeration (that lacks
  means to describe absence of PC platform devices anyway)
- mass storage or network virtualization - sharing such devices is not
  in the scope of the hypervisor
- no restart interface

Deviations and restrictions:
- CPU start address is configurable, typically set to 0 - but that's
  handled by the boot loader (comes with Jailhouse)
- LAPIC can only be operated in flat mode, and the content of LDR is
  frozen

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 02/10] x86: jailhouse: Add infrastructure for running in non-root cell

[Thomas Gleixner]

On Sat, 18 Nov 2017, Jan Kiszka wrote:
> On 2017-11-17 22:54, Thomas Gleixner wrote:
> > Shouldn't this structure and SETUP_JAILHOUSE be defined in a header file
> > which can be exported to boot loaders?
> 
> Something like arch/x86/include/uapi/asm/jailhouse_setup.h?

Something like that.

Thanks,

	tglx

Re: [PATCH 05/10] x86: jailhouse: Set up timekeeping

[Thomas Gleixner]

On Sat, 18 Nov 2017, Jan Kiszka wrote:
> On 2017-11-17 23:49, Thomas Gleixner wrote:
> > On Thu, 16 Nov 2017, Jan Kiszka wrote:
> >> Calibrate the TSC and, where necessary, the APIC timer against the
> >> TMTIMER. We need our own implementation as neither the PIC nor the HPET
> >> are available, and the standard calibration routines try to make use of
> >> them.
> > 
> > Why is this needed at all?
> > 
> > The host the frequency already. So this can be done w/o pmtimer and extra
> > calibration routine.
> 
> The hypervisor does not have the frequencies. It will never use the APIC
> timer (it's owned by the guests), and it has no use case for the TSC so
> far. Only the root cell (the Linux that booted the system) has that
> data. Now we could
> 
> - trust the root cell to provide the right values and export them during
>   startup to the hypervisor and from there to the non-root cells.
> 
> - calculate the frequencies once and store them in the hyperivsor
>   config, just like other system-specific information, for re-export to
>   the cells.
> 
> But I don't think option 1 will be ok for all use cases. Maybe a
> combination of both, falling back to the root cell data if nothing is
> defined in the config. Let me think about this.

Another question is whether systems which can support jailhouse, have the
frequencies available via cpuid/msr and can avoid that calibration thing
completely.

Thanks,

	tglx

Re: [PATCH 05/10] x86: jailhouse: Set up timekeeping

[Jan Kiszka]

On 2017-11-20 12:24, Thomas Gleixner wrote:
> On Sat, 18 Nov 2017, Jan Kiszka wrote:
>> On 2017-11-17 23:49, Thomas Gleixner wrote:
>>> On Thu, 16 Nov 2017, Jan Kiszka wrote:
>>>> Calibrate the TSC and, where necessary, the APIC timer against the
>>>> TMTIMER. We need our own implementation as neither the PIC nor the HPET
>>>> are available, and the standard calibration routines try to make use of
>>>> them.
>>>
>>> Why is this needed at all?
>>>
>>> The host the frequency already. So this can be done w/o pmtimer and extra
>>> calibration routine.
>>
>> The hypervisor does not have the frequencies. It will never use the APIC
>> timer (it's owned by the guests), and it has no use case for the TSC so
>> far. Only the root cell (the Linux that booted the system) has that
>> data. Now we could
>>
>> - trust the root cell to provide the right values and export them during
>>   startup to the hypervisor and from there to the non-root cells.
>>
>> - calculate the frequencies once and store them in the hyperivsor
>>   config, just like other system-specific information, for re-export to
>>   the cells.
>>
>> But I don't think option 1 will be ok for all use cases. Maybe a
>> combination of both, falling back to the root cell data if nothing is
>> defined in the config. Let me think about this.
> 
> Another question is whether systems which can support jailhouse, have the
> frequencies available via cpuid/msr and can avoid that calibration thing
> completely.

OK, some may (not Xeons, though), and we would not exploit it with this
approach.

Jan

Re: [PATCH 03/10] x86: jailhouse: Enable APIC and SMP support

[Konrad Rzeszutek Wilk]

On Fri, Nov 17, 2017 at 11:42:02PM +0100, Thomas Gleixner wrote:
> On Thu, 16 Nov 2017, Jan Kiszka wrote:
> 
> Just noticed, that all prefixes should be 'x86/jailhouse:' please
> 
> > +static unsigned int x2apic_get_apic_id(unsigned long id)
> > +{
> > +        return id;
> > +}
> > +
> >  static void __init jailhouse_init_platform(void)
> >  {
> >  	u64 pa_data = boot_params.hdr.setup_data;
> >  	struct jailhouse_setup_data *data;
> > +	unsigned int cpu;
> >  
> >  	data = early_memremap(pa_data, sizeof(*data));
> >  
> > @@ -59,6 +66,23 @@ static void __init jailhouse_init_platform(void)
> >  	    data->compatible_version > SETUP_REQUIRED_VERSION)
> >  		panic("Jailhouse: Unsupported setup data structure");
> >  
> > +#ifdef CONFIG_X86_X2APIC
> > +	/*
> > +	 * Register x2APIC handlers early. We need them when running
> > +	 * register_lapic_address.
> > +	 */
> > +	if (x2apic_enabled()) {
> > +		apic->read = native_apic_msr_read;
> > +		apic->write = native_apic_msr_write;
> > +		apic->get_apic_id = x2apic_get_apic_id;
> 
> Uurgh. That looks hacky.

Could just create a new APIC driver (which would call most of these)?

> 
> > +	}
> > +#endif
> > +	register_lapic_address(0xfee00000);
> 
> No need to do all of this here. The right thing to do is to override
> 
>    x86_init.mpparse.get_smp_config
> 
> which is invoked after generic_apic_probe() which should select
> x2apic_phys, but I might be wrong.
> 
> 
> > +	for (cpu = 0; cpu < data->num_cpus; cpu++)
> > +		generic_processor_info(data->cpu_ids[cpu],
> > +				       boot_cpu_apic_version);
> 
> Please add brackets around multi line statements.
> 
> Thanks,
> 
> 	tglx

Re: [PATCH 03/10] x86: jailhouse: Enable APIC and SMP support

[Thomas Gleixner]

On Mon, 20 Nov 2017, Konrad Rzeszutek Wilk wrote:

> On Fri, Nov 17, 2017 at 11:42:02PM +0100, Thomas Gleixner wrote:
> > On Thu, 16 Nov 2017, Jan Kiszka wrote:
> > 
> > Just noticed, that all prefixes should be 'x86/jailhouse:' please
> > 
> > > +static unsigned int x2apic_get_apic_id(unsigned long id)
> > > +{
> > > +        return id;
> > > +}
> > > +
> > >  static void __init jailhouse_init_platform(void)
> > >  {
> > >  	u64 pa_data = boot_params.hdr.setup_data;
> > >  	struct jailhouse_setup_data *data;
> > > +	unsigned int cpu;
> > >  
> > >  	data = early_memremap(pa_data, sizeof(*data));
> > >  
> > > @@ -59,6 +66,23 @@ static void __init jailhouse_init_platform(void)
> > >  	    data->compatible_version > SETUP_REQUIRED_VERSION)
> > >  		panic("Jailhouse: Unsupported setup data structure");
> > >  
> > > +#ifdef CONFIG_X86_X2APIC
> > > +	/*
> > > +	 * Register x2APIC handlers early. We need them when running
> > > +	 * register_lapic_address.
> > > +	 */
> > > +	if (x2apic_enabled()) {
> > > +		apic->read = native_apic_msr_read;
> > > +		apic->write = native_apic_msr_write;
> > > +		apic->get_apic_id = x2apic_get_apic_id;
> > 
> > Uurgh. That looks hacky.
> 
> Could just create a new APIC driver (which would call most of these)?

Why so? See further down.

> > 
> > > +	}
> > > +#endif
> > > +	register_lapic_address(0xfee00000);
> > 
> > No need to do all of this here. The right thing to do is to override
> > 
> >    x86_init.mpparse.get_smp_config
> > 
> > which is invoked after generic_apic_probe() which should select
> > x2apic_phys, but I might be wrong.
> > 
> > 
> > > +	for (cpu = 0; cpu < data->num_cpus; cpu++)
> > > +		generic_processor_info(data->cpu_ids[cpu],
> > > +				       boot_cpu_apic_version);
> > 
> > Please add brackets around multi line statements.
> > 
> > Thanks,
> > 
> > 	tglx
> 

Re: [PATCH 05/10] x86: jailhouse: Set up timekeeping

[Thomas Gleixner]

On Mon, 20 Nov 2017, Jan Kiszka wrote:
> On 2017-11-20 12:24, Thomas Gleixner wrote:
> > On Sat, 18 Nov 2017, Jan Kiszka wrote:
> >> On 2017-11-17 23:49, Thomas Gleixner wrote:
> >>> On Thu, 16 Nov 2017, Jan Kiszka wrote:
> >>>> Calibrate the TSC and, where necessary, the APIC timer against the
> >>>> TMTIMER. We need our own implementation as neither the PIC nor the HPET
> >>>> are available, and the standard calibration routines try to make use of
> >>>> them.
> >>>
> >>> Why is this needed at all?
> >>>
> >>> The host the frequency already. So this can be done w/o pmtimer and extra
> >>> calibration routine.
> >>
> >> The hypervisor does not have the frequencies. It will never use the APIC
> >> timer (it's owned by the guests), and it has no use case for the TSC so
> >> far. Only the root cell (the Linux that booted the system) has that
> >> data. Now we could
> >>
> >> - trust the root cell to provide the right values and export them during
> >>   startup to the hypervisor and from there to the non-root cells.
> >>
> >> - calculate the frequencies once and store them in the hyperivsor
> >>   config, just like other system-specific information, for re-export to
> >>   the cells.
> >>
> >> But I don't think option 1 will be ok for all use cases. Maybe a
> >> combination of both, falling back to the root cell data if nothing is
> >> defined in the config. Let me think about this.
> > 
> > Another question is whether systems which can support jailhouse, have the
> > frequencies available via cpuid/msr and can avoid that calibration thing
> > completely.
> 
> OK, some may (not Xeons, though), and we would not exploit it with this
> approach.

Ok. Let's expose pmtimer it's not a big issue.

Thanks,

	tglx

Re: [PATCH 05/10] x86: jailhouse: Set up timekeeping

[Jan Kiszka]

On 2017-11-23 19:36, Thomas Gleixner wrote:
> On Mon, 20 Nov 2017, Jan Kiszka wrote:
>> On 2017-11-20 12:24, Thomas Gleixner wrote:
>>> On Sat, 18 Nov 2017, Jan Kiszka wrote:
>>>> On 2017-11-17 23:49, Thomas Gleixner wrote:
>>>>> On Thu, 16 Nov 2017, Jan Kiszka wrote:
>>>>>> Calibrate the TSC and, where necessary, the APIC timer against the
>>>>>> TMTIMER. We need our own implementation as neither the PIC nor the HPET
>>>>>> are available, and the standard calibration routines try to make use of
>>>>>> them.
>>>>>
>>>>> Why is this needed at all?
>>>>>
>>>>> The host the frequency already. So this can be done w/o pmtimer and extra
>>>>> calibration routine.
>>>>
>>>> The hypervisor does not have the frequencies. It will never use the APIC
>>>> timer (it's owned by the guests), and it has no use case for the TSC so
>>>> far. Only the root cell (the Linux that booted the system) has that
>>>> data. Now we could
>>>>
>>>> - trust the root cell to provide the right values and export them during
>>>>   startup to the hypervisor and from there to the non-root cells.
>>>>
>>>> - calculate the frequencies once and store them in the hyperivsor
>>>>   config, just like other system-specific information, for re-export to
>>>>   the cells.
>>>>
>>>> But I don't think option 1 will be ok for all use cases. Maybe a
>>>> combination of both, falling back to the root cell data if nothing is
>>>> defined in the config. Let me think about this.
>>>
>>> Another question is whether systems which can support jailhouse, have the
>>> frequencies available via cpuid/msr and can avoid that calibration thing
>>> completely.
>>
>> OK, some may (not Xeons, though), and we would not exploit it with this
>> approach.
> 
> Ok. Let's expose pmtimer it's not a big issue.

I think PM timer should to be exposed independently of the question
wheather calibrate or not. If - for whatever reason - TSC is/becomes
unstable, we should have a fallback clock.

I've started to hack on forwarding calibration results from (primary)
Linux to the other guests with the option to alternatively hard-code the
values in the configuration. Seems to work find, just requires to export
lapic_timer_frequency. I'll push updates the next days that exploit this
and simplifies this patch.

Jan