* [PATCH 1/4] powerpc/include: Override unneeded early ioremap functions
2019-04-02 18:15 [PATCH 0/4] Enabling secure boot on PowerNV systems Claudio Carvalho
@ 2019-04-02 18:15 ` Claudio Carvalho
2019-04-02 18:15 ` [PATCH 2/4] powerpc/powernv: Add support for OPAL secure variables Claudio Carvalho
` (4 subsequent siblings)
5 siblings, 0 replies; 18+ messages in thread
From: Claudio Carvalho @ 2019-04-02 18:15 UTC (permalink / raw)
To: linuxppc-dev, linux-efi, linux-integrity, linux-kernel
Cc: Michael Ellerman, Paul Mackerras, Benjamin Herrenschmidt,
Ard Biesheuvel, Jeremy Kerr, Matthew Garret, Claudio Carvalho,
Nayna Jain
When CONFIG_EFI is enabled, the EFI driver includes the generic
early_ioremap header, which assumes that architectures may want to
provide their own early ioremap functions.
This patch overrides the ioremap functions in powerpc because they are
not required for secure boot on powerpc systems.
Signed-off-by: Claudio Carvalho <cclaudio@linux.ibm.com>
---
arch/powerpc/include/asm/early_ioremap.h | 41 ++++++++++++++++++++++++
1 file changed, 41 insertions(+)
create mode 100644 arch/powerpc/include/asm/early_ioremap.h
diff --git a/arch/powerpc/include/asm/early_ioremap.h b/arch/powerpc/include/asm/early_ioremap.h
new file mode 100644
index 000000000000..a86a06e9f3b9
--- /dev/null
+++ b/arch/powerpc/include/asm/early_ioremap.h
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Early ioremap definitions
+ *
+ * Copyright (C) 2019 IBM Corporation
+ * Author: Claudio Carvalho <cclaudio@linux.ibm.com>
+ *
+ */
+#ifndef _ASM_POWERPC_EARLY_IOREMAP_H
+#define _ASM_POWERPC_EARLY_IOREMAP_H
+
+static inline void __iomem *early_ioremap(resource_size_t phys_addr,
+ unsigned long size)
+{
+ return NULL;
+}
+
+static inline void *early_memremap(resource_size_t phys_addr,
+ unsigned long size)
+{
+ return NULL;
+}
+
+static inline void *early_memremap_ro(resource_size_t phys_addr,
+ unsigned long size)
+{
+ return NULL;
+}
+
+static inline void *early_memremap_prot(resource_size_t phys_addr,
+ unsigned long size,
+ unsigned long prot_val)
+{
+ return NULL;
+}
+
+static inline void early_iounmap(void __iomem *addr, unsigned long size) { }
+static inline void early_memunmap(void *addr, unsigned long size) { }
+static inline void early_ioremap_shutdown(void) { }
+
+#endif
--
2.20.1
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH 2/4] powerpc/powernv: Add support for OPAL secure variables
2019-04-02 18:15 [PATCH 0/4] Enabling secure boot on PowerNV systems Claudio Carvalho
2019-04-02 18:15 ` [PATCH 1/4] powerpc/include: Override unneeded early ioremap functions Claudio Carvalho
@ 2019-04-02 18:15 ` Claudio Carvalho
2019-04-02 18:15 ` [PATCH 3/4] powerpc/powernv: Detect the secure boot mode of the system Claudio Carvalho
` (3 subsequent siblings)
5 siblings, 0 replies; 18+ messages in thread
From: Claudio Carvalho @ 2019-04-02 18:15 UTC (permalink / raw)
To: linuxppc-dev, linux-efi, linux-integrity, linux-kernel
Cc: Michael Ellerman, Paul Mackerras, Benjamin Herrenschmidt,
Ard Biesheuvel, Jeremy Kerr, Matthew Garret, Claudio Carvalho,
Nayna Jain
The X.509 certificates trusted by the platform and other information
required to secure boot the host OS kernel are wrapped in secure
variables, which are controlled by OPAL.
The OPAL secure variables can be handled through the following OPAL
calls.
OPAL_SECVAR_GET:
Returns the data for a given secure variable name and vendor GUID.
OPAL_SECVAR_GET_NEXT:
For a given secure variable, it returns the name and vendor GUID
of the next variable.
OPAL_SECVAR_ENQUEUE:
Enqueue the supplied secure variable update so that it can be processed
by OPAL in the next boot. Variable updates cannot be be processed right
away because the variable storage is write locked at runtime.
OPAL_SECVAR_INFO:
Returns size information about the variable.
This patch adds support for OPAL secure variables by setting up the EFI
runtime variable services to make OPAL calls.
This patch also introduces CONFIG_OPAL_SECVAR for enabling the OPAL
secure variables support in the kernel. Since CONFIG_OPAL_SECVAR selects
CONFIG_EFI, it also allow us to manage the OPAL secure variables from
userspace via efivarfs.
Signed-off-by: Claudio Carvalho <cclaudio@linux.ibm.com>
---
This patch depends on new OPAL calls that are being added to skiboot.
The patch set that implements the new calls has been posted to
https://patchwork.ozlabs.org/project/skiboot/list/?series=99805
---
arch/powerpc/include/asm/opal-api.h | 6 +-
arch/powerpc/include/asm/opal.h | 10 ++
arch/powerpc/platforms/Kconfig | 3 +
arch/powerpc/platforms/powernv/Kconfig | 9 +
arch/powerpc/platforms/powernv/Makefile | 1 +
arch/powerpc/platforms/powernv/opal-call.c | 4 +
arch/powerpc/platforms/powernv/opal-secvar.c | 179 +++++++++++++++++++
7 files changed, 211 insertions(+), 1 deletion(-)
create mode 100644 arch/powerpc/platforms/powernv/opal-secvar.c
diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h
index 870fb7b239ea..d3066f29cb7a 100644
--- a/arch/powerpc/include/asm/opal-api.h
+++ b/arch/powerpc/include/asm/opal-api.h
@@ -210,7 +210,11 @@
#define OPAL_PCI_GET_PBCQ_TUNNEL_BAR 164
#define OPAL_PCI_SET_PBCQ_TUNNEL_BAR 165
#define OPAL_NX_COPROC_INIT 167
-#define OPAL_LAST 167
+#define OPAL_SECVAR_GET 170
+#define OPAL_SECVAR_GET_NEXT 171
+#define OPAL_SECVAR_ENQUEUE 172
+#define OPAL_SECVAR_INFO 173
+#define OPAL_LAST 173
#define QUIESCE_HOLD 1 /* Spin all calls at entry */
#define QUIESCE_REJECT 2 /* Fail all calls with OPAL_BUSY */
diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
index a55b01c90bb1..fdfd8dd7b326 100644
--- a/arch/powerpc/include/asm/opal.h
+++ b/arch/powerpc/include/asm/opal.h
@@ -385,6 +385,16 @@ void opal_powercap_init(void);
void opal_psr_init(void);
void opal_sensor_groups_init(void);
+extern int opal_secvar_get(uint64_t name, uint64_t vendor, uint64_t attr,
+ uint64_t data_size, uint64_t data);
+extern int opal_secvar_get_next(uint64_t name_size, uint64_t name,
+ uint64_t vendor);
+extern int opal_secvar_enqueue(uint64_t name, uint64_t vendor, uint64_t attr,
+ uint64_t data_size, uint64_t data);
+extern int opal_secvar_info(uint64_t attr, uint64_t storage_space,
+ uint64_t remaining_space,
+ uint64_t max_variable_size);
+
#endif /* __ASSEMBLY__ */
#endif /* _ASM_POWERPC_OPAL_H */
diff --git a/arch/powerpc/platforms/Kconfig b/arch/powerpc/platforms/Kconfig
index f3fb79fccc72..8e30510bc0c1 100644
--- a/arch/powerpc/platforms/Kconfig
+++ b/arch/powerpc/platforms/Kconfig
@@ -326,4 +326,7 @@ config XILINX_PCI
bool "Xilinx PCI host bridge support"
depends on PCI && XILINX_VIRTEX
+config EFI
+ bool
+
endmenu
diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig
index 850eee860cf2..879f8e766098 100644
--- a/arch/powerpc/platforms/powernv/Kconfig
+++ b/arch/powerpc/platforms/powernv/Kconfig
@@ -47,3 +47,12 @@ config PPC_VAS
VAS adapters are found in POWER9 based systems.
If unsure, say N.
+
+config OPAL_SECVAR
+ bool "OPAL Secure Variables"
+ depends on PPC_POWERNV && !CPU_BIG_ENDIAN
+ select UCS2_STRING
+ select EFI
+ help
+ This enables the kernel to access OPAL secure variables via EFI
+ runtime variable services.
diff --git a/arch/powerpc/platforms/powernv/Makefile b/arch/powerpc/platforms/powernv/Makefile
index da2e99efbd04..1511d836fd19 100644
--- a/arch/powerpc/platforms/powernv/Makefile
+++ b/arch/powerpc/platforms/powernv/Makefile
@@ -16,3 +16,4 @@ obj-$(CONFIG_PERF_EVENTS) += opal-imc.o
obj-$(CONFIG_PPC_MEMTRACE) += memtrace.o
obj-$(CONFIG_PPC_VAS) += vas.o vas-window.o vas-debug.o
obj-$(CONFIG_OCXL_BASE) += ocxl.o
+obj-$(CONFIG_OPAL_SECVAR) += opal-secvar.o
diff --git a/arch/powerpc/platforms/powernv/opal-call.c b/arch/powerpc/platforms/powernv/opal-call.c
index daad8c45c8e7..8c791c4c187b 100644
--- a/arch/powerpc/platforms/powernv/opal-call.c
+++ b/arch/powerpc/platforms/powernv/opal-call.c
@@ -282,3 +282,7 @@ OPAL_CALL(opal_pci_set_pbcq_tunnel_bar, OPAL_PCI_SET_PBCQ_TUNNEL_BAR);
OPAL_CALL(opal_sensor_read_u64, OPAL_SENSOR_READ_U64);
OPAL_CALL(opal_sensor_group_enable, OPAL_SENSOR_GROUP_ENABLE);
OPAL_CALL(opal_nx_coproc_init, OPAL_NX_COPROC_INIT);
+OPAL_CALL(opal_secvar_get, OPAL_SECVAR_GET);
+OPAL_CALL(opal_secvar_get_next, OPAL_SECVAR_GET_NEXT);
+OPAL_CALL(opal_secvar_enqueue, OPAL_SECVAR_ENQUEUE);
+OPAL_CALL(opal_secvar_info, OPAL_SECVAR_INFO)
diff --git a/arch/powerpc/platforms/powernv/opal-secvar.c b/arch/powerpc/platforms/powernv/opal-secvar.c
new file mode 100644
index 000000000000..e333828bd0bc
--- /dev/null
+++ b/arch/powerpc/platforms/powernv/opal-secvar.c
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * PowerNV code for secure variables
+ *
+ * Copyright (C) 2019 IBM Corporation
+ * Author: Claudio Carvalho <cclaudio@linux.ibm.com>
+ *
+ */
+#define pr_fmt(fmt) "secvar: "fmt
+
+#include <linux/efi.h>
+#include <asm/machdep.h>
+#include <asm/opal.h>
+
+static bool opal_secvar_supported;
+
+static efi_status_t opal_to_efi_status_log(int rc, const char *func_name)
+{
+ efi_status_t status;
+
+ switch (rc) {
+ case OPAL_EMPTY:
+ status = EFI_NOT_FOUND;
+ break;
+ case OPAL_HARDWARE:
+ status = EFI_DEVICE_ERROR;
+ break;
+ case OPAL_NO_MEM:
+ pr_err("%s: No space in the volatile storage\n", func_name);
+ status = EFI_OUT_OF_RESOURCES;
+ break;
+ case OPAL_PARAMETER:
+ status = EFI_INVALID_PARAMETER;
+ break;
+ case OPAL_PARTIAL:
+ status = EFI_BUFFER_TOO_SMALL;
+ break;
+ case OPAL_PERMISSION:
+ status = EFI_WRITE_PROTECTED;
+ break;
+ case OPAL_RESOURCE:
+ pr_err("%s: No space in the non-volatile storage\n", func_name);
+ status = EFI_OUT_OF_RESOURCES;
+ break;
+ case OPAL_SUCCESS:
+ status = EFI_SUCCESS;
+ break;
+ default:
+ pr_err("%s: Unknown OPAL error %d\n", func_name, rc);
+ status = EFI_DEVICE_ERROR;
+ break;
+ }
+
+ return status;
+}
+
+#define opal_to_efi_status(rc) opal_to_efi_status_log(rc, __func__)
+
+static efi_status_t
+opal_get_variable(efi_char16_t *name, efi_guid_t *vendor, u32 *attr,
+ unsigned long *data_size, void *data)
+{
+ int rc;
+
+ if (!opal_secvar_supported)
+ return EFI_UNSUPPORTED;
+
+ *data_size = cpu_to_be64(*data_size);
+
+ rc = opal_secvar_get(__pa(name), __pa(vendor), __pa(attr),
+ __pa(data_size), __pa(data));
+ /*
+ * The @attr is an optional output parameter. It is returned in
+ * big-endian.
+ */
+ if (attr)
+ *attr = be32_to_cpup(attr);
+ *data_size = be64_to_cpu(*data_size);
+
+ return opal_to_efi_status(rc);
+}
+
+static efi_status_t
+opal_get_next_variable(unsigned long *name_size, efi_char16_t *name,
+ efi_guid_t *vendor)
+{
+ int rc;
+
+ if (!opal_secvar_supported)
+ return EFI_UNSUPPORTED;
+
+ *name_size = cpu_to_be64(*name_size);
+
+ rc = opal_secvar_get_next(__pa(name_size), __pa(name),
+ __pa(vendor));
+
+ *name_size = be64_to_cpu(*name_size);
+
+ return opal_to_efi_status(rc);
+}
+
+static efi_status_t
+opal_set_variable(efi_char16_t *name, efi_guid_t *vendor, u32 attr,
+ unsigned long data_size, void *data)
+{
+ int rc;
+
+ if (!opal_secvar_supported)
+ return EFI_UNSUPPORTED;
+ /*
+ * The secure variable update must be enqueued in order to be processed
+ * in the next boot by firmware. The secure variable storage is write
+ * locked at runtime.
+ */
+ rc = opal_secvar_enqueue(__pa(name), __pa(vendor), attr,
+ data_size, __pa(data));
+ return opal_to_efi_status(rc);
+}
+
+static efi_status_t
+opal_query_variable_info(u32 attr, u64 *storage_space, u64 *remaining_space,
+ u64 *max_variable_size)
+{
+ int rc;
+
+ if (!opal_secvar_supported)
+ return EFI_UNSUPPORTED;
+
+ *storage_space = cpu_to_be64p(storage_space);
+ *remaining_space = cpu_to_be64p(remaining_space);
+ *max_variable_size = cpu_to_be64p(max_variable_size);
+
+ rc = opal_secvar_info(attr, __pa(storage_space), __pa(remaining_space),
+ __pa(max_variable_size));
+
+ *storage_space = be64_to_cpup(storage_space);
+ *remaining_space = be64_to_cpup(remaining_space);
+ *max_variable_size = be64_to_cpup(max_variable_size);
+
+ return opal_to_efi_status(rc);
+}
+
+static void pnv_efi_runtime_setup(void)
+{
+ /*
+ * The opal wrappers below treat the @name, @vendor, and @data
+ * parameters as little endian blobs.
+ * @name is a ucs2 string
+ * @vendor is the vendor GUID. It is converted to LE in the kernel
+ * @data variable data, which layout may be different for each variable
+ */
+ efi.get_variable = opal_get_variable;
+ efi.get_next_variable = opal_get_next_variable;
+ efi.set_variable = opal_set_variable;
+ efi.query_variable_info = opal_query_variable_info;
+
+ if (!opal_check_token(OPAL_SECVAR_GET) ||
+ !opal_check_token(OPAL_SECVAR_GET_NEXT) ||
+ !opal_check_token(OPAL_SECVAR_ENQUEUE) ||
+ !opal_check_token(OPAL_SECVAR_INFO)) {
+ pr_err("OPAL doesn't support secure variables\n");
+ opal_secvar_supported = false;
+ } else {
+ opal_secvar_supported = true;
+ }
+}
+
+static int __init pnv_efi_init(void)
+{
+ set_bit(EFI_RUNTIME_SERVICES, &efi.flags);
+ set_bit(EFI_BOOT, &efi.flags);
+
+ if (IS_ENABLED(CONFIG_64BIT))
+ set_bit(EFI_64BIT, &efi.flags);
+
+ pnv_efi_runtime_setup();
+ return 0;
+}
+machine_arch_initcall(powernv, pnv_efi_init);
--
2.20.1
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH 3/4] powerpc/powernv: Detect the secure boot mode of the system
2019-04-02 18:15 [PATCH 0/4] Enabling secure boot on PowerNV systems Claudio Carvalho
2019-04-02 18:15 ` [PATCH 1/4] powerpc/include: Override unneeded early ioremap functions Claudio Carvalho
2019-04-02 18:15 ` [PATCH 2/4] powerpc/powernv: Add support for OPAL secure variables Claudio Carvalho
@ 2019-04-02 18:15 ` Claudio Carvalho
2019-04-02 18:15 ` [PATCH 4/4] powerpc: Add support to initialize ima policy rules Claudio Carvalho
` (2 subsequent siblings)
5 siblings, 0 replies; 18+ messages in thread
From: Claudio Carvalho @ 2019-04-02 18:15 UTC (permalink / raw)
To: linuxppc-dev, linux-efi, linux-integrity, linux-kernel
Cc: Michael Ellerman, Paul Mackerras, Benjamin Herrenschmidt,
Ard Biesheuvel, Jeremy Kerr, Matthew Garret, Claudio Carvalho,
Nayna Jain
From: Nayna Jain <nayna@linux.ibm.com>
PowerNV secure boot defines different IMA policies based on the secure
boot state of the system.
This patch defines a function to detect the secure boot state of the
system.
Signed-off-by: Nayna Jain <nayna@linux.ibm.com>
---
arch/powerpc/include/asm/secboot.h | 21 +++++++++
arch/powerpc/platforms/powernv/Makefile | 2 +-
arch/powerpc/platforms/powernv/secboot.c | 54 ++++++++++++++++++++++++
3 files changed, 76 insertions(+), 1 deletion(-)
create mode 100644 arch/powerpc/include/asm/secboot.h
create mode 100644 arch/powerpc/platforms/powernv/secboot.c
diff --git a/arch/powerpc/include/asm/secboot.h b/arch/powerpc/include/asm/secboot.h
new file mode 100644
index 000000000000..1904fb4a3352
--- /dev/null
+++ b/arch/powerpc/include/asm/secboot.h
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * PowerPC secure boot definitions
+ *
+ * Copyright (C) 2019 IBM Corporation
+ * Author: Nayna Jain <nayna@linux.ibm.com>
+ *
+ */
+#ifndef POWERPC_SECBOOT_H
+#define POWERPC_SECBOOT_H
+
+#if defined(CONFIG_OPAL_SECVAR)
+extern bool get_powerpc_sb_mode(void);
+#else
+static inline bool get_powerpc_sb_mode(void)
+{
+ return false;
+}
+#endif
+
+#endif
diff --git a/arch/powerpc/platforms/powernv/Makefile b/arch/powerpc/platforms/powernv/Makefile
index 1511d836fd19..a36e22f8ecf8 100644
--- a/arch/powerpc/platforms/powernv/Makefile
+++ b/arch/powerpc/platforms/powernv/Makefile
@@ -16,4 +16,4 @@ obj-$(CONFIG_PERF_EVENTS) += opal-imc.o
obj-$(CONFIG_PPC_MEMTRACE) += memtrace.o
obj-$(CONFIG_PPC_VAS) += vas.o vas-window.o vas-debug.o
obj-$(CONFIG_OCXL_BASE) += ocxl.o
-obj-$(CONFIG_OPAL_SECVAR) += opal-secvar.o
+obj-$(CONFIG_OPAL_SECVAR) += opal-secvar.o secboot.o
diff --git a/arch/powerpc/platforms/powernv/secboot.c b/arch/powerpc/platforms/powernv/secboot.c
new file mode 100644
index 000000000000..afb1552636c5
--- /dev/null
+++ b/arch/powerpc/platforms/powernv/secboot.c
@@ -0,0 +1,54 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2019 IBM Corporation
+ * Author: Nayna Jain <nayna@linux.ibm.com>
+ *
+ * secboot.c
+ * - util functions to get powerpc secboot state
+ *
+ */
+#include <linux/efi.h>
+#include <asm/secboot.h>
+
+bool get_powerpc_sb_mode(void)
+{
+ efi_char16_t efi_SecureBoot_name[] = L"SecureBoot";
+ efi_char16_t efi_SetupMode_name[] = L"SetupMode";
+ efi_guid_t efi_variable_guid = EFI_GLOBAL_VARIABLE_GUID;
+ efi_status_t status;
+ u8 secboot, setupmode;
+ unsigned long size = sizeof(secboot);
+
+ status = efi.get_variable(efi_SecureBoot_name, &efi_variable_guid,
+ NULL, &size, &secboot);
+
+ /*
+ * For now assume all failures reading the SecureBoot variable implies
+ * secure boot is not enabled. Later differentiate failure types.
+ */
+ if (status != EFI_SUCCESS) {
+ secboot = 0;
+ setupmode = 0;
+ goto out;
+ }
+
+ size = sizeof(setupmode);
+ status = efi.get_variable(efi_SetupMode_name, &efi_variable_guid,
+ NULL, &size, &setupmode);
+
+ /*
+ * Failure to read the SetupMode variable does not prevent
+ * secure boot mode
+ */
+ if (status != EFI_SUCCESS)
+ setupmode = 0;
+
+out:
+ if ((secboot == 0) || (setupmode == 1)) {
+ pr_info("ima: secureboot mode disabled\n");
+ return false;
+ }
+
+ pr_info("ima: secureboot mode enabled\n");
+ return true;
+}
--
2.20.1
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH 4/4] powerpc: Add support to initialize ima policy rules
2019-04-02 18:15 [PATCH 0/4] Enabling secure boot on PowerNV systems Claudio Carvalho
` (2 preceding siblings ...)
2019-04-02 18:15 ` [PATCH 3/4] powerpc/powernv: Detect the secure boot mode of the system Claudio Carvalho
@ 2019-04-02 18:15 ` Claudio Carvalho
2019-04-02 19:36 ` [PATCH 0/4] Enabling secure boot on PowerNV systems Matthew Garrett
2019-04-03 13:21 ` Michael Ellerman
5 siblings, 0 replies; 18+ messages in thread
From: Claudio Carvalho @ 2019-04-02 18:15 UTC (permalink / raw)
To: linuxppc-dev, linux-efi, linux-integrity, linux-kernel
Cc: Michael Ellerman, Paul Mackerras, Benjamin Herrenschmidt,
Ard Biesheuvel, Jeremy Kerr, Matthew Garret, Claudio Carvalho,
Nayna Jain
From: Nayna Jain <nayna@linux.ibm.com>
PowerNV secure boot relies on the kernel IMA security subsystem to
perform the OS kernel image signature verification. Since each secure
boot mode has different IMA policy requirements, dynamic definition of
the policy rules based on the runtime secure boot mode of the system is
required. On systems that support secure boot, but have it disabled,
only measurement policy rules of the kernel image and modules are
defined.
This patch defines the arch-specific implementation to retrieve the
secure boot mode of the system and accordingly configures the IMA policy
rules.
This patch will provide arch-specific IMA policies if PPC_SECURE_BOOT
config is enabled.
Signed-off-by: Nayna Jain <nayna@linux.ibm.com>
---
arch/powerpc/Kconfig | 12 ++++++++
arch/powerpc/kernel/Makefile | 1 +
arch/powerpc/kernel/ima_arch.c | 54 ++++++++++++++++++++++++++++++++++
include/linux/ima.h | 3 +-
4 files changed, 69 insertions(+), 1 deletion(-)
create mode 100644 arch/powerpc/kernel/ima_arch.c
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 2d0be82c3061..e0ba9a9114b3 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -901,6 +901,18 @@ config PPC_MEM_KEYS
If unsure, say y.
+config PPC_SECURE_BOOT
+ prompt "Enable PowerPC Secure Boot"
+ bool
+ default n
+ depends on IMA
+ depends on IMA_ARCH_POLICY
+ help
+ Linux on POWER with firmware secure boot enabled needs to define
+ security policies to extend secure boot to the OS.
+ This config allows user to enable OS Secure Boot on PowerPC systems
+ that have firmware secure boot support.
+
endmenu
config ISA_DMA_API
diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile
index cddadccf551d..0f08ed7dfd1b 100644
--- a/arch/powerpc/kernel/Makefile
+++ b/arch/powerpc/kernel/Makefile
@@ -119,6 +119,7 @@ ifdef CONFIG_IMA
obj-y += ima_kexec.o
endif
endif
+obj-$(CONFIG_IMA) += ima_arch.o
obj-$(CONFIG_AUDIT) += audit.o
obj64-$(CONFIG_AUDIT) += compat_audit.o
diff --git a/arch/powerpc/kernel/ima_arch.c b/arch/powerpc/kernel/ima_arch.c
new file mode 100644
index 000000000000..871b321656fb
--- /dev/null
+++ b/arch/powerpc/kernel/ima_arch.c
@@ -0,0 +1,54 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2019 IBM Corporation
+ * Author: Nayna Jain <nayna@linux.ibm.com>
+ *
+ * ima_arch.c
+ * - initialize ima policies for PowerPC Secure Boot
+ */
+
+#include <linux/ima.h>
+#include <asm/secboot.h>
+
+bool arch_ima_get_secureboot(void)
+{
+ bool sb_mode;
+
+ sb_mode = get_powerpc_sb_mode();
+ if (sb_mode)
+ return true;
+ else
+ return false;
+}
+
+/*
+ * File signature verification is not needed, include only measurements
+ */
+static const char *const default_arch_rules[] = {
+ "measure func=KEXEC_KERNEL_CHECK",
+ "measure func=MODULE_CHECK",
+ NULL
+};
+
+/* Both file signature verification and measurements are needed */
+static const char *const sb_arch_rules[] = {
+ "measure func=KEXEC_KERNEL_CHECK",
+ "measure func=MODULE_CHECK",
+ "appraise func=KEXEC_KERNEL_CHECK appraise_type=imasig",
+#if !IS_ENABLED(CONFIG_MODULE_SIG)
+ "appraise func=MODULE_CHECK appraise_type=imasig",
+#endif
+ NULL
+};
+
+/*
+ * On PowerPC, file measurements are to be added to the IMA measurement list
+ * irrespective of the secure boot state of the system. Signature verification
+ * is conditionally enabled based on the secure boot state.
+ */
+const char *const *arch_get_ima_policy(void)
+{
+ if (IS_ENABLED(CONFIG_IMA_ARCH_POLICY) && arch_ima_get_secureboot())
+ return sb_arch_rules;
+ return default_arch_rules;
+}
diff --git a/include/linux/ima.h b/include/linux/ima.h
index dc12fbcf484c..32f46d69ebd7 100644
--- a/include/linux/ima.h
+++ b/include/linux/ima.h
@@ -31,7 +31,8 @@ extern void ima_post_path_mknod(struct dentry *dentry);
extern void ima_add_kexec_buffer(struct kimage *image);
#endif
-#if defined(CONFIG_X86) && defined(CONFIG_EFI)
+#if defined(CONFIG_X86) && defined(CONFIG_EFI) \
+ || defined(CONFIG_PPC_SECURE_BOOT)
extern bool arch_ima_get_secureboot(void);
extern const char * const *arch_get_ima_policy(void);
#else
--
2.20.1
^ permalink raw reply related [flat|nested] 18+ messages in thread* Re: [PATCH 0/4] Enabling secure boot on PowerNV systems
2019-04-02 18:15 [PATCH 0/4] Enabling secure boot on PowerNV systems Claudio Carvalho
` (3 preceding siblings ...)
2019-04-02 18:15 ` [PATCH 4/4] powerpc: Add support to initialize ima policy rules Claudio Carvalho
@ 2019-04-02 19:36 ` Matthew Garrett
2019-04-02 21:11 ` Claudio Carvalho
2019-04-03 13:21 ` Michael Ellerman
5 siblings, 1 reply; 18+ messages in thread
From: Matthew Garrett @ 2019-04-02 19:36 UTC (permalink / raw)
To: Claudio Carvalho
Cc: linuxppc-dev, linux-efi, linux-integrity,
Linux Kernel Mailing List, Michael Ellerman, Paul Mackerras,
Benjamin Herrenschmidt, Ard Biesheuvel, Jeremy Kerr,
Matthew Garret, Nayna Jain
On Tue, Apr 2, 2019 at 11:15 AM Claudio Carvalho <cclaudio@linux.ibm.com> wrote:
> 1. Enable efivarfs by selecting CONFIG_EFI in the CONFIG_OPAL_SECVAR
> introduced in this patch set. With CONFIG_EFIVAR_FS, userspace tools can
> be used to manage the secure variables.
efivarfs has some pretty significant behavioural semantics that
directly reflect the EFI specification. Using it to expose non-EFI
variable data feels like it's going to increase fragility - there's a
risk that we'll change things in a way that makes sense for the EFI
spec but breaks your use case. Is the desire to use efivarfs to
maintain consistency with existing userland tooling, or just to avoid
having a separate filesystem?
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [PATCH 0/4] Enabling secure boot on PowerNV systems
2019-04-02 19:36 ` [PATCH 0/4] Enabling secure boot on PowerNV systems Matthew Garrett
@ 2019-04-02 21:11 ` Claudio Carvalho
2019-04-02 21:51 ` Matthew Garrett
0 siblings, 1 reply; 18+ messages in thread
From: Claudio Carvalho @ 2019-04-02 21:11 UTC (permalink / raw)
To: Matthew Garrett
Cc: linuxppc-dev, linux-efi, linux-integrity,
Linux Kernel Mailing List, Michael Ellerman, Paul Mackerras,
Benjamin Herrenschmidt, Ard Biesheuvel, Jeremy Kerr,
Matthew Garret, Nayna Jain
On 4/2/19 4:36 PM, Matthew Garrett wrote:
> On Tue, Apr 2, 2019 at 11:15 AM Claudio Carvalho <cclaudio@linux.ibm.com> wrote:
>> 1. Enable efivarfs by selecting CONFIG_EFI in the CONFIG_OPAL_SECVAR
>> introduced in this patch set. With CONFIG_EFIVAR_FS, userspace tools can
>> be used to manage the secure variables.
> efivarfs has some pretty significant behavioural semantics that
> directly reflect the EFI specification. Using it to expose non-EFI
> variable data feels like it's going to increase fragility - there's a
> risk that we'll change things in a way that makes sense for the EFI
> spec but breaks your use case. Is the desire to use efivarfs to
> maintain consistency with existing userland tooling, or just to avoid
> having a separate filesystem?
>
We want to use the efivarfs for compatibility with existing userspace
tools. We will track and match any EFI changes that affect us.
Our use case is restricted to secure boot - this is not going to be a
general purpose EFI variable implementation.
Claudio
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 0/4] Enabling secure boot on PowerNV systems
2019-04-02 21:11 ` Claudio Carvalho
@ 2019-04-02 21:51 ` Matthew Garrett
2019-04-02 23:31 ` Claudio Carvalho
0 siblings, 1 reply; 18+ messages in thread
From: Matthew Garrett @ 2019-04-02 21:51 UTC (permalink / raw)
To: Claudio Carvalho
Cc: linuxppc-dev, linux-efi, linux-integrity,
Linux Kernel Mailing List, Michael Ellerman, Paul Mackerras,
Benjamin Herrenschmidt, Ard Biesheuvel, Jeremy Kerr,
Matthew Garret, Nayna Jain
On Tue, Apr 2, 2019 at 2:11 PM Claudio Carvalho <cclaudio@linux.ibm.com> wrote:
> We want to use the efivarfs for compatibility with existing userspace
> tools. We will track and match any EFI changes that affect us.
So you implement the full PK/KEK/db/dbx/dbt infrastructure, and
updates are signed in the same way?
> Our use case is restricted to secure boot - this is not going to be a
> general purpose EFI variable implementation.
In that case we might be better off with a generic interface for this
purpose that we can expose on all platforms that implement a secure
boot key hierarchy. Having an efivarfs that doesn't allow the creation
of arbitrary attributes may break other existing userland
expectations.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 0/4] Enabling secure boot on PowerNV systems
2019-04-02 21:51 ` Matthew Garrett
@ 2019-04-02 23:31 ` Claudio Carvalho
2019-04-03 22:27 ` Matthew Garrett
0 siblings, 1 reply; 18+ messages in thread
From: Claudio Carvalho @ 2019-04-02 23:31 UTC (permalink / raw)
To: Matthew Garrett
Cc: linuxppc-dev, linux-efi, linux-integrity,
Linux Kernel Mailing List, Michael Ellerman, Paul Mackerras,
Benjamin Herrenschmidt, Ard Biesheuvel, Jeremy Kerr,
Matthew Garret, Nayna Jain
On 4/2/19 6:51 PM, Matthew Garrett wrote:
> On Tue, Apr 2, 2019 at 2:11 PM Claudio Carvalho <cclaudio@linux.ibm.com> wrote:
>> We want to use the efivarfs for compatibility with existing userspace
>> tools. We will track and match any EFI changes that affect us.
> So you implement the full PK/KEK/db/dbx/dbt infrastructure, and
> updates are signed in the same way?
For the first version, our firmware will implement a simplistic PK, KEK and
db infrastructure (without dbx and dbt) where only the Setup and User modes
will be supported.
PK, KEK and db updates will be signed the same way, that is, using
userspace tooling like efitools in PowerNV. As for the authentication
descriptors, only the EFI_VARIABLE_AUTHENTICATION_2 descriptor will be
supported.
>> Our use case is restricted to secure boot - this is not going to be a
>> general purpose EFI variable implementation.
> In that case we might be better off with a generic interface for this
> purpose that we can expose on all platforms that implement a secure
> boot key hierarchy. Having an efivarfs that doesn't allow the creation
> of arbitrary attributes may break other existing userland
> expectations.
>
For what it's worth, gsmi uses the efivars infrastructure for EFI-like
variables.
What might a generic interface look like? It would have to work for
existing secure boot solutions - including EFI - which would seem to imply
changes to userspace tools.
Claudio
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 0/4] Enabling secure boot on PowerNV systems
2019-04-02 23:31 ` Claudio Carvalho
@ 2019-04-03 22:27 ` Matthew Garrett
2019-04-05 21:11 ` Claudio Carvalho
0 siblings, 1 reply; 18+ messages in thread
From: Matthew Garrett @ 2019-04-03 22:27 UTC (permalink / raw)
To: Claudio Carvalho
Cc: linuxppc-dev, linux-efi, linux-integrity,
Linux Kernel Mailing List, Michael Ellerman, Paul Mackerras,
Benjamin Herrenschmidt, Ard Biesheuvel, Jeremy Kerr,
Matthew Garret, Nayna Jain
On Tue, Apr 2, 2019 at 4:31 PM Claudio Carvalho <cclaudio@linux.ibm.com> wrote:
>
>
> On 4/2/19 6:51 PM, Matthew Garrett wrote:
> > So you implement the full PK/KEK/db/dbx/dbt infrastructure, and
> > updates are signed in the same way?
>
> For the first version, our firmware will implement a simplistic PK, KEK and
> db infrastructure (without dbx and dbt) where only the Setup and User modes
> will be supported.
Not supporting dbx seems like a pretty significant shortcoming. How
are signatures meant to be revoked?
> PK, KEK and db updates will be signed the same way, that is, using
> userspace tooling like efitools in PowerNV. As for the authentication
> descriptors, only the EFI_VARIABLE_AUTHENTICATION_2 descriptor will be
> supported.
Is this API documented?
> > In that case we might be better off with a generic interface for this
> > purpose that we can expose on all platforms that implement a secure
> > boot key hierarchy. Having an efivarfs that doesn't allow the creation
> > of arbitrary attributes may break other existing userland
> > expectations.
> >
> For what it's worth, gsmi uses the efivars infrastructure for EFI-like
> variables.
My recollection is that at the time the Chromebook firmware still had
EFI underpinnings and the gsmi code was largely just an alternate
mechanism for calling into something that was fundamentally the EFI
variable store. With hindsight I don't think layering this was the
right move - we've adjusted the semantics of efivarfs on more than one
occasion to deal with the behaviour of real-world EFI platforms, and I
don't think it's helpful to end up in a situation where we're trying
to keep behaviour consistent among entirely different firmware
interfaces.
> What might a generic interface look like? It would have to work for
> existing secure boot solutions - including EFI - which would seem to imply
> changes to userspace tools.
I think that depends on exactly what problem you're trying to solve.
Some aspects of the EFI secure boot design end up mirroring the
economics of the PC ecosystem rather than being inherently good design
goals, so it'd be helpful to know whether you're taking this solution
because you want the same three-level key infrastructure or because
that just leaves you compatible with the tooling.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 0/4] Enabling secure boot on PowerNV systems
2019-04-03 22:27 ` Matthew Garrett
@ 2019-04-05 21:11 ` Claudio Carvalho
2019-04-05 22:19 ` Matthew Garrett
0 siblings, 1 reply; 18+ messages in thread
From: Claudio Carvalho @ 2019-04-05 21:11 UTC (permalink / raw)
To: Matthew Garrett
Cc: linuxppc-dev, linux-efi, linux-integrity,
Linux Kernel Mailing List, Michael Ellerman, Paul Mackerras,
Benjamin Herrenschmidt, Ard Biesheuvel, Jeremy Kerr,
Matthew Garret, Nayna Jain
On 4/3/19 7:27 PM, Matthew Garrett wrote:
> On Tue, Apr 2, 2019 at 4:31 PM Claudio Carvalho <cclaudio@linux.ibm.com> wrote:
>>
>> On 4/2/19 6:51 PM, Matthew Garrett wrote:
>>> So you implement the full PK/KEK/db/dbx/dbt infrastructure, and
>>> updates are signed in the same way?
>> For the first version, our firmware will implement a simplistic PK, KEK and
>> db infrastructure (without dbx and dbt) where only the Setup and User modes
>> will be supported.
> Not supporting dbx seems like a pretty significant shortcoming. How
> are signatures meant to be revoked?
We began by focusing on certificates for keys that can be added at
runtime. Before adding support for revocation, we plan to gather
additional use cases. In the meantime, unwanted certificates can be
removed by the administrator.
>
>> PK, KEK and db updates will be signed the same way, that is, using
>> userspace tooling like efitools in PowerNV. As for the authentication
>> descriptors, only the EFI_VARIABLE_AUTHENTICATION_2 descriptor will be
>> supported.
> Is this API documented?
The API is still a work in progress. We are planning to publish a document
describing the current API and overall design shortly.
Perhaps the biggest departure is that the secure variables are stored in
flash memory that is not lockable. In order to protect the secure
variables, hashes of the flash regions where they're stored are written to
TPM NVRAM indices. The TPM NVRAM indices we use are write locked at
runtime. The sysadmin enqueues update commands in flash. During the next
boot, the firmware verifies and processes the commands to update the
certificate store and accompanying integrity hashes in the TPM NVRAM
indices and write locks them. Before certificates read from flash are
used, the certificate store is hashed and compared against the hashes
stored from the TPM. The one exception is the PK. We store it in a TPM
NVRAM index by itself rather than flash because updates to it must be
guaranteed to be atomic.
>>> In that case we might be better off with a generic interface for this
>>> purpose that we can expose on all platforms that implement a secure
>>> boot key hierarchy. Having an efivarfs that doesn't allow the creation
>>> of arbitrary attributes may break other existing userland
>>> expectations.
>>>
>> For what it's worth, gsmi uses the efivars infrastructure for EFI-like
>> variables.
> My recollection is that at the time the Chromebook firmware still had
> EFI underpinnings and the gsmi code was largely just an alternate
> mechanism for calling into something that was fundamentally the EFI
> variable store. With hindsight I don't think layering this was the
> right move - we've adjusted the semantics of efivarfs on more than one
> occasion to deal with the behaviour of real-world EFI platforms, and I
> don't think it's helpful to end up in a situation where we're trying
> to keep behaviour consistent among entirely different firmware
> interfaces.
>
>> What might a generic interface look like? It would have to work for
>> existing secure boot solutions - including EFI - which would seem to imply
>> changes to userspace tools.
> I think that depends on exactly what problem you're trying to solve.
> Some aspects of the EFI secure boot design end up mirroring the
> economics of the PC ecosystem rather than being inherently good design
> goals, so it'd be helpful to know whether you're taking this solution
> because you want the same three-level key infrastructure or because
> that just leaves you compatible with the tooling.
In our use case, the three-level key hierarchy conveniently supports the
concept of (1) an administrator authority, who authorizes (2) other
organizations, e.g., distros, to provide (3) certificates for their code
signing keys. By using efivars, we leverage pre-existing userspace EFI
tools to generate authenticated updates and certificates. Additionally,
pre-existing kernel infrastructure simplifies efivars processing.
Thanks,
Claudio
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 0/4] Enabling secure boot on PowerNV systems
2019-04-05 21:11 ` Claudio Carvalho
@ 2019-04-05 22:19 ` Matthew Garrett
2019-04-09 22:55 ` Claudio Carvalho
0 siblings, 1 reply; 18+ messages in thread
From: Matthew Garrett @ 2019-04-05 22:19 UTC (permalink / raw)
To: Claudio Carvalho
Cc: linuxppc-dev, linux-efi, linux-integrity,
Linux Kernel Mailing List, Michael Ellerman, Paul Mackerras,
Benjamin Herrenschmidt, Ard Biesheuvel, Jeremy Kerr,
Matthew Garret, Nayna Jain
On Fri, Apr 5, 2019 at 2:11 PM Claudio Carvalho <cclaudio@linux.ibm.com> wrote:
>
>
> On 4/3/19 7:27 PM, Matthew Garrett wrote:
> > Not supporting dbx seems like a pretty significant shortcoming. How
> > are signatures meant to be revoked?
>
>
> We began by focusing on certificates for keys that can be added at
> runtime. Before adding support for revocation, we plan to gather
> additional use cases. In the meantime, unwanted certificates can be
> removed by the administrator.
Based on our experience doing this in UEFI, that's insufficient - you
want to be able to block individual binaries or leaf certificates
without dropping trust in an intermediate certificate entirely.
>
> >
> >> PK, KEK and db updates will be signed the same way, that is, using
> >> userspace tooling like efitools in PowerNV. As for the authentication
> >> descriptors, only the EFI_VARIABLE_AUTHENTICATION_2 descriptor will be
> >> supported.
> > Is this API documented?
>
>
> The API is still a work in progress. We are planning to publish a document
> describing the current API and overall design shortly.
Ok. How are the attributes interpreted by the API?
> Perhaps the biggest departure is that the secure variables are stored in
> flash memory that is not lockable. In order to protect the secure
> variables, hashes of the flash regions where they're stored are written to
> TPM NVRAM indices. The TPM NVRAM indices we use are write locked at
> runtime. The sysadmin enqueues update commands in flash. During the next
> boot, the firmware verifies and processes the commands to update the
> certificate store and accompanying integrity hashes in the TPM NVRAM
> indices and write locks them. Before certificates read from flash are
> used, the certificate store is hashed and compared against the hashes
> stored from the TPM. The one exception is the PK. We store it in a TPM
> NVRAM index by itself rather than flash because updates to it must be
> guaranteed to be atomic.
What's the behaviour if multiple updates are enqueued? Does reading
back show a mocked up updated variable or the original state?
> > I think that depends on exactly what problem you're trying to solve.
> > Some aspects of the EFI secure boot design end up mirroring the
> > economics of the PC ecosystem rather than being inherently good design
> > goals, so it'd be helpful to know whether you're taking this solution
> > because you want the same three-level key infrastructure or because
> > that just leaves you compatible with the tooling.
>
>
> In our use case, the three-level key hierarchy conveniently supports the
> concept of (1) an administrator authority, who authorizes (2) other
> organizations, e.g., distros, to provide (3) certificates for their code
> signing keys. By using efivars, we leverage pre-existing userspace EFI
> tools to generate authenticated updates and certificates. Additionally,
> pre-existing kernel infrastructure simplifies efivars processing.
I'm not really clear on the workflow here. Who's the administrator
authority? When would they be updating the second level of keys? If
there's no support for revocation, why would distributions need two
levels of key in the system database rather than just distributing a
single intermediate and signing their actual signing certs with that?
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 0/4] Enabling secure boot on PowerNV systems
2019-04-05 22:19 ` Matthew Garrett
@ 2019-04-09 22:55 ` Claudio Carvalho
2019-04-10 17:36 ` Matthew Garrett
0 siblings, 1 reply; 18+ messages in thread
From: Claudio Carvalho @ 2019-04-09 22:55 UTC (permalink / raw)
To: Matthew Garrett
Cc: linuxppc-dev, linux-efi, linux-integrity,
Linux Kernel Mailing List, Michael Ellerman, Paul Mackerras,
Benjamin Herrenschmidt, Ard Biesheuvel, Jeremy Kerr,
Matthew Garret, Nayna Jain
On 4/5/19 7:19 PM, Matthew Garrett wrote:
> On Fri, Apr 5, 2019 at 2:11 PM Claudio Carvalho <cclaudio@linux.ibm.com> wrote:
>>
>> On 4/3/19 7:27 PM, Matthew Garrett wrote:
>>> Not supporting dbx seems like a pretty significant shortcoming. How
>>> are signatures meant to be revoked?
>>
>> We began by focusing on certificates for keys that can be added at
>> runtime. Before adding support for revocation, we plan to gather
>> additional use cases. In the meantime, unwanted certificates can be
>> removed by the administrator.
> Based on our experience doing this in UEFI, that's insufficient - you
> want to be able to block individual binaries or leaf certificates
> without dropping trust in an intermediate certificate entirely.
We agree that a dbx would be useful for blacklisting particular kernels
signed with given certificate. However, we have been avoiding doing so for
the initial release of secure boot on OpenPOWER. We don't have individual
firmware binaries in OpenPOWER. Kernels are currently the only concern for
the OS secure boot certificates we're discussing here. Also, we have a very
limited keystore space in POWER9.
Petitboot doesn't have standardized OS kernel verification at all right
now. Having the capability even without dbx seems valuable.
>
>>>> PK, KEK and db updates will be signed the same way, that is, using
>>>> userspace tooling like efitools in PowerNV. As for the authentication
>>>> descriptors, only the EFI_VARIABLE_AUTHENTICATION_2 descriptor will be
>>>> supported.
>>> Is this API documented?
>>
>> The API is still a work in progress. We are planning to publish a document
>> describing the current API and overall design shortly.
> Ok. How are the attributes interpreted by the API?
We support a subset of standard EFI variable attributes, and we only use
EFI variables that relate to secure boot. Our goal is not to implement
UEFI. However, we do seek to be compatible with user space tooling and
reuse as much existing infrastructure as possible. We don’t support the
following: EFI_VARIABLE_HARDWARE_ERROR_RECORD,
EFI_VARIABLE_AUTHENTICATED_WRITE_ACCESS and
EFI_VARIABLE_ENHANCED_AUTHENTICATED_ACCESS.
>
>> Perhaps the biggest departure is that the secure variables are stored in
>> flash memory that is not lockable. In order to protect the secure
>> variables, hashes of the flash regions where they're stored are written to
>> TPM NVRAM indices. The TPM NVRAM indices we use are write locked at
>> runtime. The sysadmin enqueues update commands in flash. During the next
>> boot, the firmware verifies and processes the commands to update the
>> certificate store and accompanying integrity hashes in the TPM NVRAM
>> indices and write locks them. Before certificates read from flash are
>> used, the certificate store is hashed and compared against the hashes
>> stored from the TPM. The one exception is the PK. We store it in a TPM
>> NVRAM index by itself rather than flash because updates to it must be
>> guaranteed to be atomic.
> What's the behaviour if multiple updates are enqueued? Does reading
> back show a mocked up updated variable or the original state?
Our secure variable updates are only applied at boot time. If any one of
them fails, they all fail.
>
>>> I think that depends on exactly what problem you're trying to solve.
>>> Some aspects of the EFI secure boot design end up mirroring the
>>> economics of the PC ecosystem rather than being inherently good design
>>> goals, so it'd be helpful to know whether you're taking this solution
>>> because you want the same three-level key infrastructure or because
>>> that just leaves you compatible with the tooling.
>>
>> In our use case, the three-level key hierarchy conveniently supports the
>> concept of (1) an administrator authority, who authorizes (2) other
>> organizations, e.g., distros, to provide (3) certificates for their code
>> signing keys. By using efivars, we leverage pre-existing userspace EFI
>> tools to generate authenticated updates and certificates. Additionally,
>> pre-existing kernel infrastructure simplifies efivars processing.
> I'm not really clear on the workflow here. Who's the administrator
> authority? When would they be updating the second level of keys? If
> there's no support for revocation, why would distributions need two
> levels of key in the system database rather than just distributing a
> single intermediate and signing their actual signing certs with that?
In OpenPOWER systems, we enable our customers and business partners to
establish and manage the platform key certificate, which is the root of our
key hierarchy. From there, through the KEK, they can delegate authority to
intermediate level organizations, e.g. distros or IT departments or
business operations. Those intermediate level organizations then manage the
code signing certificates in the DB. If this answer doesn’t address your
question, can you please rephrase?
Thanks,
Claudio
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 0/4] Enabling secure boot on PowerNV systems
2019-04-09 22:55 ` Claudio Carvalho
@ 2019-04-10 17:36 ` Matthew Garrett
2019-05-10 21:31 ` Claudio Carvalho
0 siblings, 1 reply; 18+ messages in thread
From: Matthew Garrett @ 2019-04-10 17:36 UTC (permalink / raw)
To: Claudio Carvalho
Cc: linuxppc-dev, linux-efi, linux-integrity,
Linux Kernel Mailing List, Michael Ellerman, Paul Mackerras,
Benjamin Herrenschmidt, Ard Biesheuvel, Jeremy Kerr,
Matthew Garret, Nayna Jain, Peter Jones
(Cc:ing Peter Jones)
On Tue, Apr 9, 2019 at 3:55 PM Claudio Carvalho <cclaudio@linux.ibm.com> wrote:
>
>
> On 4/5/19 7:19 PM, Matthew Garrett wrote:
> > Based on our experience doing this in UEFI, that's insufficient - you
> > want to be able to block individual binaries or leaf certificates
> > without dropping trust in an intermediate certificate entirely.
>
>
> We agree that a dbx would be useful for blacklisting particular kernels
> signed with given certificate. However, we have been avoiding doing so for
> the initial release of secure boot on OpenPOWER. We don't have individual
> firmware binaries in OpenPOWER. Kernels are currently the only concern for
> the OS secure boot certificates we're discussing here. Also, we have a very
> limited keystore space in POWER9.
>
> Petitboot doesn't have standardized OS kernel verification at all right
> now. Having the capability even without dbx seems valuable.
I don't see the benefit in attempting to maintain compatibility with
existing tooling unless you're going to be *completely* compatible
with existing tooling. That means supporting dbx and dbt.
> >> The API is still a work in progress. We are planning to publish a document
> >> describing the current API and overall design shortly.
> > Ok. How are the attributes interpreted by the API?
>
>
> We support a subset of standard EFI variable attributes, and we only use
> EFI variables that relate to secure boot. Our goal is not to implement
> UEFI. However, we do seek to be compatible with user space tooling and
> reuse as much existing infrastructure as possible. We don’t support the
> following: EFI_VARIABLE_HARDWARE_ERROR_RECORD,
> EFI_VARIABLE_AUTHENTICATED_WRITE_ACCESS and
> EFI_VARIABLE_ENHANCED_AUTHENTICATED_ACCESS.
Ok. I think that's realistically fine.
>
> >
> >> Perhaps the biggest departure is that the secure variables are stored in
> >> flash memory that is not lockable. In order to protect the secure
> >> variables, hashes of the flash regions where they're stored are written to
> >> TPM NVRAM indices. The TPM NVRAM indices we use are write locked at
> >> runtime. The sysadmin enqueues update commands in flash. During the next
> >> boot, the firmware verifies and processes the commands to update the
> >> certificate store and accompanying integrity hashes in the TPM NVRAM
> >> indices and write locks them. Before certificates read from flash are
> >> used, the certificate store is hashed and compared against the hashes
> >> stored from the TPM. The one exception is the PK. We store it in a TPM
> >> NVRAM index by itself rather than flash because updates to it must be
> >> guaranteed to be atomic.
> > What's the behaviour if multiple updates are enqueued? Does reading
> > back show a mocked up updated variable or the original state?
>
>
> Our secure variable updates are only applied at boot time. If any one of
> them fails, they all fail.
So I do the following:
1) Boot
2) Extend the contents of db
3) Extend the contents of db again
4) Read back the contents of db through efivarfs
5) Reboot
6) Read back the contents of db through efivarfs
Is what I see in (4) and (6) the same? Does it contain the values form
both extensions?
> > I'm not really clear on the workflow here. Who's the administrator
> > authority? When would they be updating the second level of keys? If
> > there's no support for revocation, why would distributions need two
> > levels of key in the system database rather than just distributing a
> > single intermediate and signing their actual signing certs with that?
>
>
> In OpenPOWER systems, we enable our customers and business partners to
> establish and manage the platform key certificate, which is the root of our
> key hierarchy. From there, through the KEK, they can delegate authority to
> intermediate level organizations, e.g. distros or IT departments or
> business operations. Those intermediate level organizations then manage the
> code signing certificates in the DB. If this answer doesn’t address your
> question, can you please rephrase?
Why would the intermediate level organisations not just have entries
in db? The main reason we don't do it this way in UEFI is because we
need to support dbx, and if you're not supporting dbx I'm not sure I
see the benefit.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 0/4] Enabling secure boot on PowerNV systems
2019-04-10 17:36 ` Matthew Garrett
@ 2019-05-10 21:31 ` Claudio Carvalho
2019-05-13 22:06 ` Matthew Garrett
0 siblings, 1 reply; 18+ messages in thread
From: Claudio Carvalho @ 2019-05-10 21:31 UTC (permalink / raw)
To: Matthew Garrett
Cc: linuxppc-dev, linux-efi, linux-integrity,
Linux Kernel Mailing List, Michael Ellerman, Paul Mackerras,
Ard Biesheuvel, Jeremy Kerr, Matthew Garret, Nayna Jain,
Peter Jones
Hi Matthew,
Thanks for the feedback and sorry for the delay in responding.
On 4/10/19 2:36 PM, Matthew Garrett wrote:
> (Cc:ing Peter Jones)
>
> On Tue, Apr 9, 2019 at 3:55 PM Claudio Carvalho <cclaudio@linux.ibm.com> wrote:
>>
>> On 4/5/19 7:19 PM, Matthew Garrett wrote:
>>> Based on our experience doing this in UEFI, that's insufficient - you
>>> want to be able to block individual binaries or leaf certificates
>>> without dropping trust in an intermediate certificate entirely.
>>
>> We agree that a dbx would be useful for blacklisting particular kernels
>> signed with given certificate. However, we have been avoiding doing so for
>> the initial release of secure boot on OpenPOWER. We don't have individual
>> firmware binaries in OpenPOWER. Kernels are currently the only concern for
>> the OS secure boot certificates we're discussing here. Also, we have a very
>> limited keystore space in POWER9.
>>
>> Petitboot doesn't have standardized OS kernel verification at all right
>> now. Having the capability even without dbx seems valuable.
> I don't see the benefit in attempting to maintain compatibility with
> existing tooling unless you're going to be *completely* compatible
> with existing tooling. That means supporting dbx and dbt.
Before addressing that, I'd like to share some of the current OpenPOWER
secure boot design.
Different from UEFI, secure boot in OpenPOWER systems have two distinct
domains. Each one has its own key hierarchy and signing and signature
verification mechanisms.
In the firmware secure boot domain (work already upstream):
- Every image loaded up to skiroot is wrapped in a secure boot container.
Skiroot is a Linux zimage with petitboot (kexec bootloader) embedded in the
initramfs.
- Within the secure boot container, the payload image is protected by a
chain of signatures anchored in the root ECDSA keys, also known as hardware
keys.
- All public keys required to verify the container are stored in the
container itself, but a hash of the trusted public hardware keys is stored
in a protected SEEPROM region outside of the container. Firmware uses it to
check if the container is anchored in the trusted hardware keys. If not,
the container payload is not executed and the boot is aborted.
- The hash of the hardware keys is set by the firmware supplier, for
instance, the platform manufacturer.
In OS secure boot domain (work in progress):
- The skiroot container is verified as part of firmware secure boot.
- Skiroot uses UEFI-like secure variables (PK, KEK and db) to verify OS
kernels. Only X.509 certificates will be supported for these secure variables.
- OS kernels are signed using the Linux kernel sign-file tool, as if they
were kernel modules.
- In the skiroot kernel, if secure boot is enabled, the db certificates
will be loaded into the platform keyring and IMA-appraisal will verify the
kexec image against the platform keyring.
- The PK is set by whoever controls the platform, for instance, the
manufacturer or the end customer.
How about dbx and dbt?
The db keys will be used to verify only OS kernels via kexecs initiated by
petitboot. So we only need the dbx to revoke kernel images, either via
certs or hashes. Currently, the kernel loads certs and hashes from the dbx
to the system blacklist keyring. The revoked certs are checked during pkcs7
signature verification and loading of keys. However, there doesn't appear
to be any verification against blacklisted hashes. Should kernel images be
revoked only by keys and not hashes? We tried to find published revoked
kernel lists but couldn't find any. How is kernel image revocation handled
in practice?
Also, we didn't see the shim or kernel loading anything from dbt.
In general, how do you think the kernel ought to support blacklists?
>
>>>> The API is still a work in progress. We are planning to publish a document
>>>> describing the current API and overall design shortly.
>>> Ok. How are the attributes interpreted by the API?
>>
>> We support a subset of standard EFI variable attributes, and we only use
>> EFI variables that relate to secure boot. Our goal is not to implement
>> UEFI. However, we do seek to be compatible with user space tooling and
>> reuse as much existing infrastructure as possible. We don’t support the
>> following: EFI_VARIABLE_HARDWARE_ERROR_RECORD,
>> EFI_VARIABLE_AUTHENTICATED_WRITE_ACCESS and
>> EFI_VARIABLE_ENHANCED_AUTHENTICATED_ACCESS.
> Ok. I think that's realistically fine.
>
>>>> Perhaps the biggest departure is that the secure variables are stored in
>>>> flash memory that is not lockable. In order to protect the secure
>>>> variables, hashes of the flash regions where they're stored are written to
>>>> TPM NVRAM indices. The TPM NVRAM indices we use are write locked at
>>>> runtime. The sysadmin enqueues update commands in flash. During the next
>>>> boot, the firmware verifies and processes the commands to update the
>>>> certificate store and accompanying integrity hashes in the TPM NVRAM
>>>> indices and write locks them. Before certificates read from flash are
>>>> used, the certificate store is hashed and compared against the hashes
>>>> stored from the TPM. The one exception is the PK. We store it in a TPM
>>>> NVRAM index by itself rather than flash because updates to it must be
>>>> guaranteed to be atomic.
>>> What's the behaviour if multiple updates are enqueued? Does reading
>>> back show a mocked up updated variable or the original state?
>>
>> Our secure variable updates are only applied at boot time. If any one of
>> them fails, they all fail.
> So I do the following:
>
> 1) Boot
> 2) Extend the contents of db
> 3) Extend the contents of db again
> 4) Read back the contents of db through efivarfs
> 5) Reboot
> 6) Read back the contents of db through efivarfs
>
> Is what I see in (4) and (6) the same? Does it contain the values form
> both extensions?
In (2) and (3) the extensions are added to the update queue, which is
processed only in (5) by firmware. So, in (4) you should see the db content
without the extensions.
In (5), firmware (skiboot) will process the update queue. The extensions
will be applied only if *all* of them are valid and pass signature
verification. Only in this case should you be able to see the extensions in
(6). If any of the extensions fail, firmware will discard all of them,
clear the queue, and do the proper logging.
>>> I'm not really clear on the workflow here. Who's the administrator
>>> authority? When would they be updating the second level of keys? If
>>> there's no support for revocation, why would distributions need two
>>> levels of key in the system database rather than just distributing a
>>> single intermediate and signing their actual signing certs with that?
>>
>> In OpenPOWER systems, we enable our customers and business partners to
>> establish and manage the platform key certificate, which is the root of our
>> key hierarchy. From there, through the KEK, they can delegate authority to
>> intermediate level organizations, e.g. distros or IT departments or
>> business operations. Those intermediate level organizations then manage the
>> code signing certificates in the DB. If this answer doesn’t address your
>> question, can you please rephrase?
> Why would the intermediate level organisations not just have entries
> in db?
Because that seems to add more complexity than having three levels (PK, KEK
and db).
Typically, the intermediate level organisations (or KEK) are used to
authorize new additions to db. However, if we also have them in the db, who
would authorize the new additions to db. If that would be the intermediate
level organisation entries now in the db, it seems we would need to
implement a mechanism to determine which entries are for authorizing new
additions and which are for kernel signature verification. If that would be
the PK, we'd be burdening the PK owner to sign every new db addition if the
platform is owned by a company that has intermediate level organizations.
> The main reason we don't do it this way in UEFI is because we
> need to support dbx, and if you're not supporting dbx I'm not sure I
> see the benefit.
I'm not sure I understand your question. We would be using dbx to prevent
kernels from being loaded. How is that related to having three levels in
the key hierarchy (PK, KEK and db)?
Thanks,
Claudio
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 0/4] Enabling secure boot on PowerNV systems
2019-05-10 21:31 ` Claudio Carvalho
@ 2019-05-13 22:06 ` Matthew Garrett
0 siblings, 0 replies; 18+ messages in thread
From: Matthew Garrett @ 2019-05-13 22:06 UTC (permalink / raw)
To: Claudio Carvalho
Cc: linuxppc-dev, linux-efi, linux-integrity,
Linux Kernel Mailing List, Michael Ellerman, Paul Mackerras,
Ard Biesheuvel, Jeremy Kerr, Matthew Garret, Nayna Jain,
Peter Jones
On Fri, May 10, 2019 at 2:31 PM Claudio Carvalho <cclaudio@linux.ibm.com> wrote:
> On 4/10/19 2:36 PM, Matthew Garrett wrote:
> > I don't see the benefit in attempting to maintain compatibility with
> > existing tooling unless you're going to be *completely* compatible
> > with existing tooling. That means supporting dbx and dbt.
(snip)
> In OS secure boot domain (work in progress):
> - The skiroot container is verified as part of firmware secure boot.
> - Skiroot uses UEFI-like secure variables (PK, KEK and db) to verify OS
> kernels. Only X.509 certificates will be supported for these secure variables.
You don't support hashes? If so, this isn't compatible with UEFI
Secure Boot and we shouldn't try to make it look like UEFI Secure
Boot.
> How about dbx and dbt?
>
> The db keys will be used to verify only OS kernels via kexecs initiated by
> petitboot. So we only need the dbx to revoke kernel images, either via
> certs or hashes. Currently, the kernel loads certs and hashes from the dbx
> to the system blacklist keyring. The revoked certs are checked during pkcs7
> signature verification and loading of keys. However, there doesn't appear
> to be any verification against blacklisted hashes. Should kernel images be
> revoked only by keys and not hashes? We tried to find published revoked
> kernel lists but couldn't find any. How is kernel image revocation handled
> in practice?
Hash-based revocation is in active use in the UEFI world - to the best
of my knowledge, all existing dbx entries are hashes with the
exception of the invalidation of the Microsoft Windows 2010 CA.
> Also, we didn't see the shim or kernel loading anything from dbt.
dbt is currently only used for validation at the firmware level - the
way grub and kernel signatures are currently managed means it doesn't
make a huge amount of sense to use it in shim, but it would probably
be reasonable to extend shim's validation to include dbt.
> > So I do the following:
> >
> > 1) Boot
> > 2) Extend the contents of db
> > 3) Extend the contents of db again
> > 4) Read back the contents of db through efivarfs
> > 5) Reboot
> > 6) Read back the contents of db through efivarfs
> >
> > Is what I see in (4) and (6) the same? Does it contain the values form
> > both extensions?
>
> In (2) and (3) the extensions are added to the update queue, which is
> processed only in (5) by firmware. So, in (4) you should see the db content
> without the extensions.
Ok, this is not what we expect from UEFI systems. I'm strongly against
providing what looks like the same ABI on multiple platforms but
carrying subtle differences between those platforms - it's guaranteed
to break tooling in unexpected ways.
> In (5), firmware (skiboot) will process the update queue. The extensions
> will be applied only if *all* of them are valid and pass signature
> verification. Only in this case should you be able to see the extensions in
> (6). If any of the extensions fail, firmware will discard all of them,
> clear the queue, and do the proper logging.
I believe that this is also a violation of expectations.
> > Why would the intermediate level organisations not just have entries
> > in db?
>
> Because that seems to add more complexity than having three levels (PK, KEK
> and db).
>
> Typically, the intermediate level organisations (or KEK) are used to
> authorize new additions to db. However, if we also have them in the db, who
> would authorize the new additions to db. If that would be the intermediate
> level organisation entries now in the db, it seems we would need to
> implement a mechanism to determine which entries are for authorizing new
> additions and which are for kernel signature verification. If that would be
> the PK, we'd be burdening the PK owner to sign every new db addition if the
> platform is owned by a company that has intermediate level organizations.
Ok, in this scenario I don't understand why you wouldn't just want the
intermediates in PK. Or, put another way - if you have a business
justification for three layers of hierarchy, what do you do when
someone has a business justification for four? The three layer
hierarchy represents the weirdness of the PC industry where you have
Microsoft needing to be in KEK (because they need to be able to issue
updates to machines from multiple vendors) but not wanting to be in PK
(because vendors don't want Microsoft to have ultimate control over
their systems). If it weren't for this conflict, we'd just have a two
layer hierarchy, and if some other aspect of the market had evolved
over time we'd have a four layer hierarchy.
>
> > The main reason we don't do it this way in UEFI is because we
> > need to support dbx, and if you're not supporting dbx I'm not sure I
> > see the benefit.
>
> I'm not sure I understand your question. We would be using dbx to prevent
> kernels from being loaded. How is that related to having three levels in
> the key hierarchy (PK, KEK and db)?
dbx entries come from Microsoft, so we need the KEK layer so Microsoft
can update dbx. If Microsoft didn't need to update dbx then we'd leave
Microsoft out of KEK, and then KEK and PK would be the same and we'd
be able to get rid of KEK.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 0/4] Enabling secure boot on PowerNV systems
2019-04-02 18:15 [PATCH 0/4] Enabling secure boot on PowerNV systems Claudio Carvalho
` (4 preceding siblings ...)
2019-04-02 19:36 ` [PATCH 0/4] Enabling secure boot on PowerNV systems Matthew Garrett
@ 2019-04-03 13:21 ` Michael Ellerman
2019-04-03 21:48 ` Claudio Carvalho
5 siblings, 1 reply; 18+ messages in thread
From: Michael Ellerman @ 2019-04-03 13:21 UTC (permalink / raw)
To: Claudio Carvalho, linuxppc-dev, linux-efi, linux-integrity, linux-kernel
Cc: Paul Mackerras, Benjamin Herrenschmidt, Ard Biesheuvel,
Jeremy Kerr, Matthew Garret, Claudio Carvalho, Nayna Jain
Hi Claudio,
Thanks for posting this.
Claudio Carvalho <cclaudio@linux.ibm.com> writes:
> This patch set is part of a series that implements secure boot on
> PowerNV systems.
>
> In order to verify the OS kernel on PowerNV, secure boot requires X.509
> certificates trusted by the platform, the secure boot modes, and several
> other pieces of information. These are stored in secure variables
> controlled by OPAL, also known as OPAL secure variables.
>
> This patch set adds the following features:
>
> 1. Enable efivarfs by selecting CONFIG_EFI in the CONFIG_OPAL_SECVAR
> introduced in this patch set. With CONFIG_EFIVAR_FS, userspace tools can
> be used to manage the secure variables.
> 2. Add support for OPAL secure variables by overwriting the EFI hooks
> (get_variable, get_next_variable, set_variable and query_variable_info)
> with OPAL call wrappers. There is probably a better way to add this
> support, for example, we are investigating if we could register the
> efivar_operations rather than overwriting the EFI hooks. In this patch
> set, CONFIG_OPAL_SECVAR selects CONFIG_EFI. If, instead, we registered
> efivar_operations, CONFIG_EFIVAR_FS would need to depend on
> CONFIG_EFI|| CONFIG_OPAL_SECVAR. Comments or suggestions on the
> preferred technique would be greatly appreciated.
I am *very* reluctant to start selecting CONFIG_EFI on powerpc.
Simply because we don't actually have EFI, and I worry we're going to
both break assumptions in the EFI code as well as impose requirements on
the powerpc code that aren't really necessary.
So I'd definitely prefer we go the route of enabling efivarfs with an
alternate backend.
Better still would be a generic secure variable interface as Matt
suggests, if the userspace tools can be relatively easily adapted to use
that interface.
cheers
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [PATCH 0/4] Enabling secure boot on PowerNV systems
2019-04-03 13:21 ` Michael Ellerman
@ 2019-04-03 21:48 ` Claudio Carvalho
0 siblings, 0 replies; 18+ messages in thread
From: Claudio Carvalho @ 2019-04-03 21:48 UTC (permalink / raw)
To: Michael Ellerman, linuxppc-dev, linux-efi, linux-integrity, linux-kernel
Cc: Paul Mackerras, Benjamin Herrenschmidt, Ard Biesheuvel,
Jeremy Kerr, Matthew Garret, Nayna Jain
On 4/3/19 10:21 AM, Michael Ellerman wrote:
> Hi Claudio,
>
> Thanks for posting this.
>
> Claudio Carvalho <cclaudio@linux.ibm.com> writes:
>> This patch set is part of a series that implements secure boot on
>> PowerNV systems.
>>
>> In order to verify the OS kernel on PowerNV, secure boot requires X.509
>> certificates trusted by the platform, the secure boot modes, and several
>> other pieces of information. These are stored in secure variables
>> controlled by OPAL, also known as OPAL secure variables.
>>
>> This patch set adds the following features:
>>
>> 1. Enable efivarfs by selecting CONFIG_EFI in the CONFIG_OPAL_SECVAR
>> introduced in this patch set. With CONFIG_EFIVAR_FS, userspace tools can
>> be used to manage the secure variables.
>> 2. Add support for OPAL secure variables by overwriting the EFI hooks
>> (get_variable, get_next_variable, set_variable and query_variable_info)
>> with OPAL call wrappers. There is probably a better way to add this
>> support, for example, we are investigating if we could register the
>> efivar_operations rather than overwriting the EFI hooks. In this patch
>> set, CONFIG_OPAL_SECVAR selects CONFIG_EFI. If, instead, we registered
>> efivar_operations, CONFIG_EFIVAR_FS would need to depend on
>> CONFIG_EFI|| CONFIG_OPAL_SECVAR. Comments or suggestions on the
>> preferred technique would be greatly appreciated.
> I am *very* reluctant to start selecting CONFIG_EFI on powerpc.
>
> Simply because we don't actually have EFI, and I worry we're going to
> both break assumptions in the EFI code as well as impose requirements on
> the powerpc code that aren't really necessary.
Yes, we agree. We are working on the v2 and it is not going to depend on
CONFIG_EFI. Rather, the IMA arch policies will make the OPAL calls directly.
>
> So I'd definitely prefer we go the route of enabling efivarfs with an
> alternate backend.
Right, I'm investigating how we can do that, but it looks like we should
post that as a separate patchset to avoid delaying upstreaming signature
verification based on the secure boot variables.
Thanks,
Claudio
>
> Better still would be a generic secure variable interface as Matt
> suggests, if the userspace tools can be relatively easily adapted to use
> that interface.
>
> cheers
>
^ permalink raw reply [flat|nested] 18+ messages in thread