[PATCH RFC] Microcode late loading feature identification

[PATCH RFC] Microcode late loading feature identification

[Mihai Carabas]

This RFC patch set aims to provide a way to identify the modifications
brought in by the new microcode updated at runtime (aka microcode late
loading). This was debated last year and this patch set implements
point #1 from Thomas Gleixner's idea:
https://lore.kernel.org/lkml/alpine.DEB.2.21.1909062237580.1902@nanos.tec.linutronix.de/

This patch set has the following patches:

- patch 1 is introducing a new metadata file that comes with the microcode
(provided by the CPU manufacture) that describes what modifications are
done by loading the new microcode

- patch 2 parses the metadata file and is verifying it against kernel
policy. In this patch, as an RFC, as a kernel policy, it was imposed
the rule of not allowing to remove any feature. If so, it won't be
loaded a new microcode. The policy can be further extended and describe
in different ways

- patch 3 adds the documentation of the metadata file format


How to test:

- place metadata file in /lib/firmware/intel-ucode/ together with the
microcode blob:

[root@ovs108 ~]# ls -l /lib/firmware/intel-ucode
total 96
-rw-r--r--.   1 root root 34816 Mar 11 00:27 06-55-04
-rw-r--r--.   1 root root    84 Mar 25 03:13 06-55-04.metadata

The microcode blob can be taken from the microcode_ctl package.

- after installing the kernel and rebooting the machine run "dracut -f
--no-early-microcode" to create an initramfs without the microcode (and
avoid early loading)

- reboot

- after rebooting issue: echo 1 > /sys/devices/system/cpu/microcode/reload

[root@ovs108 ~]# cat /lib/firmware/intel-ucode/06-55-04.metadata
m - 0x00000122
c + 0x00000007 0x00 0x00000000 0x021cbfbb 0x00000000 0x00000000

[root@ovs108 ~]# echo 1 > /sys/devices/system/cpu/microcode/reload
[root@ovs108 ~]# dmesg | tail -2
[ 1285.729841] microcode: Kernel policy does not allow to remove MSR: 122
[ 1285.737144] microcode: kernel does not support the new microcode: intel-ucode/06-55-04

[root@ovs108 ~]# cat /lib/firmware/intel-ucode/06-55-04.metadata
m + 0x00000122
c + 0x00000007 0x00 0x00000000 0x021cbfbb 0x00000000 0x00000000
[root@ovs108 ~]# echo 1 > /sys/devices/system/cpu/microcode/reload
[root@ovs108 ~]# dmesg | tail -10
[ 1220.212415] microcode: updated to revision 0x2000065, date = 2019-09-05
[ 1220.212645] microcode: Reload completed, microcode revision: 0x2000065

Mihai Carabas (3):
  x86: microcode: intel: read microcode metadata file
  x86: microcode: intel: process microcode metadata
  Documentation: x86: microcode: add description for metadata file

 Documentation/x86/microcode.rst       | 36 +++++++++++++
 arch/x86/kernel/cpu/microcode/intel.c | 97 +++++++++++++++++++++++++++++++++++
 2 files changed, 133 insertions(+)

-- 
1.8.3.1

[PATCH RFC 1/3] x86: microcode: intel: read microcode metadata file

[Mihai Carabas]

Try to read the microcode metadata file in order to see what features
are added or remove by the new microcode blob. If the metadata file
does not exists passthrough normal loading with a warning message.

Signed-off-by: Mihai Carabas <mihai.carabas@oracle.com>
---
 arch/x86/kernel/cpu/microcode/intel.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/arch/x86/kernel/cpu/microcode/intel.c b/arch/x86/kernel/cpu/microcode/intel.c
index 6a99535..a54d5e6 100644
--- a/arch/x86/kernel/cpu/microcode/intel.c
+++ b/arch/x86/kernel/cpu/microcode/intel.c
@@ -970,10 +970,12 @@ static enum ucode_state request_microcode_fw(int cpu, struct device *device,
 					     bool refresh_fw)
 {
 	struct cpuinfo_x86 *c = &cpu_data(cpu);
+	const struct firmware *meta_firmware;
 	const struct firmware *firmware;
 	struct iov_iter iter;
 	enum ucode_state ret;
 	struct kvec kvec;
+	char meta_name[40];
 	char name[30];
 
 	if (is_blacklisted(cpu))
@@ -982,11 +984,21 @@ static enum ucode_state request_microcode_fw(int cpu, struct device *device,
 	sprintf(name, "intel-ucode/%02x-%02x-%02x",
 		c->x86, c->x86_model, c->x86_stepping);
 
+	sprintf(meta_name, "intel-ucode/%02x-%02x-%02x.metadata",
+		c->x86, c->x86_model, c->x86_stepping);
+
 	if (request_firmware_direct(&firmware, name, device)) {
 		pr_debug("data file %s load failed\n", name);
 		return UCODE_NFOUND;
 	}
 
+	if (request_firmware_direct(&meta_firmware, meta_name, device)) {
+		pr_debug("metadata file %s load failed\n", name);
+		pr_debug("no feature check will be done pre-loading the microcode\n");
+	} else {
+		release_firmware(meta_firmware);
+	}
+
 	kvec.iov_base = (void *)firmware->data;
 	kvec.iov_len = firmware->size;
 	iov_iter_kvec(&iter, WRITE, &kvec, 1, firmware->size);
-- 
1.8.3.1

Re: [PATCH RFC 1/3] x86: microcode: intel: read microcode metadata file

[Borislav Petkov]

> Subject: Re: [PATCH RFC 1/3] x86: microcode: intel: read microcode metadata file

For the future, do:

git log <file I'm changing under arch/x86/>

to figure out what commit title prefix to use for the tip tree.

On Mon, Apr 27, 2020 at 10:27:57AM +0300, Mihai Carabas wrote:
> Try to read the microcode metadata file in order to see what features
> are added or remove by the new microcode blob. If the metadata file
> does not exists passthrough normal loading with a warning message.

So this file must be signed by the microcode vendors and that signature
then must be verified by the loader before we even look at the metadata.
We don't trust luserspace.

Also, I don't like it being a separate file - it could just as well be
appended to the microcode blob and parsed properly by the loader.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

[PATCH RFC 2/3] x86: microcode: intel: process microcode metadata

[Mihai Carabas]

Process the microcode metadata to see what changes are done on
CPU feature bits.

The default kernel policy imposed by is_microcode_allowed() is
to abort any microcode late loading if there will be removed a
feature and will continue the loading otherwise. Complex policies
can be created in the future.

Signed-off-by: Mihai Carabas <mihai.carabas@oracle.com>
---
 arch/x86/kernel/cpu/microcode/intel.c | 85 +++++++++++++++++++++++++++++++++++
 1 file changed, 85 insertions(+)

diff --git a/arch/x86/kernel/cpu/microcode/intel.c b/arch/x86/kernel/cpu/microcode/intel.c
index a54d5e6..2ef4338 100644
--- a/arch/x86/kernel/cpu/microcode/intel.c
+++ b/arch/x86/kernel/cpu/microcode/intel.c
@@ -23,6 +23,7 @@
 #include <linux/firmware.h>
 #include <linux/uaccess.h>
 #include <linux/vmalloc.h>
+#include <linux/string.h>
 #include <linux/initrd.h>
 #include <linux/kernel.h>
 #include <linux/slab.h>
@@ -965,6 +966,81 @@ static bool is_blacklisted(unsigned int cpu)
 
 	return false;
 }
+/**
+ * is_microcode_allowed - parse and enforce policies
+ * @firmware - metadata blob
+ *
+ * Parses the metadata file provided with the microcode in order to see the
+ * changes brought in by the new blob.
+ *
+ * Based on the type of changes, currently it enforces the following policy: if
+ * there is a feature being removed by the new microcode, reject the loading.
+ *
+ * RETURNS:
+ * false if it rejects the microcode or true if it accepts the microcode.
+ */
+static bool is_microcode_allowed(const struct firmware *firmware)
+{
+	int ret, index, leaf, eax, ebx, ecx, edx, line_nr = 0;
+	char *data = (char *) firmware->data;
+	char *line;
+	char op;
+
+	/*
+	 * Get each line and match it with regular expression:
+	 * {m|c} {+|-} u32 [u32]*
+	 *
+	 * See section Late loading metadata file from
+	 * Documentation/x86/microcode.rst
+	 */
+	while (data && (data - ((char *)(firmware->data)) < firmware->size)) {
+		line = strsep(&data, "\n");
+		line_nr++;
+
+		/*
+		 * For each line check the first letter to see the type of
+		 * operation done by the microcode and create a sscanf call
+		 * the would match the rest of the line of that operation.
+		 * If it is not a match, exist with parsing error. If it is
+		 * a match and is a remove action (-) do not allow microcode
+		 * to be loaded.
+		 */
+		switch (line[0]) {
+		case 'c':
+			ret = sscanf(&line[1], " %c %x %x %x %x %x %x", &op,
+			    &index, &leaf, &eax, &ebx, &ecx, &edx);
+			if (ret != 7)
+				goto out_error;
+			if (op == '-')
+				goto out_cpuid_rem_notsupp;
+			break;
+		case 'm':
+			ret = sscanf(&line[1], " %c %x", &op, &index);
+			if (ret != 2)
+				goto out_error;
+			if (op == '-')
+				goto out_msr_rem_notsupp;
+			break;
+		default:
+			goto out_error;
+		}
+	}
+
+	return true;
+
+out_error:
+	pr_warn("Microcode metadata failed parsing at line %d\n", line_nr);
+	return false;
+
+out_msr_rem_notsupp:
+	pr_warn("Kernel policy does not allow to remove MSR: %x\n", index);
+	return false;
+
+out_cpuid_rem_notsupp:
+	pr_warn("Kernel policy does not allow to remove CPUID: %x leaf: %x, eax: %x, ebx: %x, ecx: %x, edx: %x\n",
+	    index, leaf, eax, ebx, ecx, edx);
+	return false;
+}
 
 static enum ucode_state request_microcode_fw(int cpu, struct device *device,
 					     bool refresh_fw)
@@ -992,10 +1068,19 @@ static enum ucode_state request_microcode_fw(int cpu, struct device *device,
 		return UCODE_NFOUND;
 	}
 
+	/* Load the metadata file in memory. */
 	if (request_firmware_direct(&meta_firmware, meta_name, device)) {
 		pr_debug("metadata file %s load failed\n", name);
 		pr_debug("no feature check will be done pre-loading the microcode\n");
 	} else {
+		/* Check the policy based on the metadata file. */
+		if (!is_microcode_allowed(meta_firmware)) {
+			pr_warn("kernel does not support the new microcode: %s\n",
+			    name);
+			release_firmware(meta_firmware);
+			release_firmware(firmware);
+			return UCODE_ERROR;
+		}
 		release_firmware(meta_firmware);
 	}
 
-- 
1.8.3.1

[PATCH RFC 3/3] Documentation: x86: microcode: add description for metadata file

[Mihai Carabas]

Microcode nowadays may remove of modify certain CPU feature bits. Prior
to this patch the kernel was blindly loading any microcode blob which might
have caused an unrecoverable error (e.g. the kernel was executing an
instruction that was removed). The following patches will process the
metadata file and will know what features are being added/removed/modified
and can take a decision on loading or not the new microcode blob.

Signed-off-by: Mihai Carabas <mihai.carabas@oracle.com>
---
 Documentation/x86/microcode.rst | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/Documentation/x86/microcode.rst b/Documentation/x86/microcode.rst
index a320d37..45e3ae8 100644
--- a/Documentation/x86/microcode.rst
+++ b/Documentation/x86/microcode.rst
@@ -110,6 +110,42 @@ The loading mechanism looks for microcode blobs in
 /lib/firmware/{intel-ucode,amd-ucode}. The default distro installation
 packages already put them there.
 
+Late loading metadata file
+==========================
+
+New microcode blobs may remove or modify CPU feature bits. Prior to this
+metadata file, the microcode was blindly loaded and might have created an
+unrecoverable error (e.g. remove an instruction used currently in the kernel).
+
+In order to improve visibility on what features a new microcode that is being
+loaded at runtime (late loading) brings in, a new metadata file is created
+together with the microcode blob. The metadata file has the same name as the
+microcode blob with a suffix of ".metadata". The metadata file respects the
+following regular expression: "{m|c} {+|-} u32 [u32]*", where "m" means MSR
+feature and "c" means a CPUID exposed feature.
+
+Here is an example of content for the metadata file::
+   m + 0x00000122
+   m - 0x00000120
+   c + 0x00000007 0x00 0x00000000 0x021cbfbb 0x00000000 0x00000000
+   c - 0x00000007 0x00 0x00000000 0x021cbfbb 0x00000000 0x00000000
+
+The definition of the file format is as follows::
+   - each line contains an action on a CPU feature that the microcode will do
+   - the first letter specify the type of the feature
+   - the second letter specify the operation:
+   -- + - adds the feature
+   -- - - removes the feature
+   - the third letter specifies the index of the CPUID or the MSR
+   - for the CPUID case all the others parameters specifies the
+     leaf, eax, ebx, ecx and edx values
+
+Using this metadata file, the kernel, based on its internal policies, may
+deny a microcode update in order to ensure system stability (e.g. if an
+instruction is removed by the microcode and that instruction is still being
+used by the current code, we would drop the update as it would brake the
+system).
+
 Builtin microcode
 =================
 
-- 
1.8.3.1

Re: [PATCH RFC 3/3] Documentation: x86: microcode: add description for metadata file

[Borislav Petkov]

On Mon, Apr 27, 2020 at 10:27:59AM +0300, Mihai Carabas wrote:
> +Here is an example of content for the metadata file::
> +   m + 0x00000122
> +   m - 0x00000120

That's not enough. Imagine a blob adds the MSR and a subsequent blob
adds *another* bit in that MSR. You need to be able express that.

IOW, I think it would be easier if each line describes exactly *one*
software-visible change brought by the microcode.

Also, what is the use case to say that you're adding a new MSR?

This would require to patch all the code that *potentially* might use
that MSR, to be able to handle a *change* in that MSR or it appearing
all of a sudden. Yuck.

I mean, an easy way to handle it is to say, "Hmm, nope, won't load that
ucode."

> +   c + 0x00000007 0x00 0x00000000 0x021cbfbb 0x00000000 0x00000000
> +   c - 0x00000007 0x00 0x00000000 0x021cbfbb 0x00000000 0x00000000

I don't think this'll work with the vendors as depending on the
configuration, CPUID on the different platforms could be different.

And then you might gonna have to specify CPUID for this particular
{family,model,stepping, ... } tuple which identifies the platform
uniquely so the c-line is insufficient.

IOW, I think it would be easier to be able to specify one CPUID bit per
line of being added/removed so that the post-load callback can handle
this properly. And this specification needs to be complete: i.e.

"I'm adding this CPUID bit for this family,model,stepping configurations
under this and that conditions".

And that needs to be describable by the "language" of the metadata.

So think of actual examples and then try to represent them with this
format.

But all of this is moot if you don't get a vendor buy-in into this. IOW,
microcode vendors need to agree to this format and adhere to the format
when allowing microcode to late-load.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH RFC] Microcode late loading feature identification

[Mihai Carabas]

La 27.04.2020 10:27, Mihai Carabas a scris:
> This RFC patch set aims to provide a way to identify the modifications
> brought in by the new microcode updated at runtime (aka microcode late
> loading). This was debated last year and this patch set implements
> point #1 from Thomas Gleixner's idea:
> https://lore.kernel.org/lkml/alpine.DEB.2.21.1909062237580.1902@nanos.tec.linutronix.de/
> 

+Ashok and Thomas to get a feedback from vendor side on file 
format/integration in the microcode blob and signature.

Thank you,
Mihai

> This patch set has the following patches:
> 
> - patch 1 is introducing a new metadata file that comes with the microcode
> (provided by the CPU manufacture) that describes what modifications are
> done by loading the new microcode
> 
> - patch 2 parses the metadata file and is verifying it against kernel
> policy. In this patch, as an RFC, as a kernel policy, it was imposed
> the rule of not allowing to remove any feature. If so, it won't be
> loaded a new microcode. The policy can be further extended and describe
> in different ways
> 
> - patch 3 adds the documentation of the metadata file format
> 
> 
> How to test:
> 
> - place metadata file in /lib/firmware/intel-ucode/ together with the
> microcode blob:
> 
> [root@ovs108 ~]# ls -l /lib/firmware/intel-ucode
> total 96
> -rw-r--r--.   1 root root 34816 Mar 11 00:27 06-55-04
> -rw-r--r--.   1 root root    84 Mar 25 03:13 06-55-04.metadata
> 
> The microcode blob can be taken from the microcode_ctl package.
> 
> - after installing the kernel and rebooting the machine run "dracut -f
> --no-early-microcode" to create an initramfs without the microcode (and
> avoid early loading)
> 
> - reboot
> 
> - after rebooting issue: echo 1 > /sys/devices/system/cpu/microcode/reload
> 
> [root@ovs108 ~]# cat /lib/firmware/intel-ucode/06-55-04.metadata
> m - 0x00000122
> c + 0x00000007 0x00 0x00000000 0x021cbfbb 0x00000000 0x00000000
> 
> [root@ovs108 ~]# echo 1 > /sys/devices/system/cpu/microcode/reload
> [root@ovs108 ~]# dmesg | tail -2
> [ 1285.729841] microcode: Kernel policy does not allow to remove MSR: 122
> [ 1285.737144] microcode: kernel does not support the new microcode: intel-ucode/06-55-04
> 
> [root@ovs108 ~]# cat /lib/firmware/intel-ucode/06-55-04.metadata
> m + 0x00000122
> c + 0x00000007 0x00 0x00000000 0x021cbfbb 0x00000000 0x00000000
> [root@ovs108 ~]# echo 1 > /sys/devices/system/cpu/microcode/reload
> [root@ovs108 ~]# dmesg | tail -10
> [ 1220.212415] microcode: updated to revision 0x2000065, date = 2019-09-05
> [ 1220.212645] microcode: Reload completed, microcode revision: 0x2000065
> 
> Mihai Carabas (3):
>    x86: microcode: intel: read microcode metadata file
>    x86: microcode: intel: process microcode metadata
>    Documentation: x86: microcode: add description for metadata file
> 
>   Documentation/x86/microcode.rst       | 36 +++++++++++++
>   arch/x86/kernel/cpu/microcode/intel.c | 97 +++++++++++++++++++++++++++++++++++
>   2 files changed, 133 insertions(+)
> 

Re: [PATCH RFC] Microcode late loading feature identification

[Raj, Ashok]

Hi Mihai

Thanks for an attempt to find a fix. To solve this for real there
are several other factors to consider and I'm afraid its not as simple
as you have articulated here. There are lots of practical limitations that
prevent us from solving this completely. But we haven't given up :-)

In order to be successful, this needs to be factored in by the vendor either
as part of the development process or somehow generated automatically. Both
have pitfalls, and as you read below some of it might become clear.

On Mon, May 11, 2020 at 05:11:23PM +0300, Mihai Carabas wrote:
> La 27.04.2020 10:27, Mihai Carabas a scris:
> >This RFC patch set aims to provide a way to identify the modifications
> >brought in by the new microcode updated at runtime (aka microcode late
> >loading). This was debated last year and this patch set implements
> >point #1 from Thomas Gleixner's idea:
> >https://lore.kernel.org/lkml/alpine.DEB.2.21.1909062237580.1902@nanos.tec.linutronix.de/
> >
> 
> +Ashok and Thomas to get a feedback from vendor side on file
> format/integration in the microcode blob and signature.

To understand the complications of microcode there are a few things to consider.

We have been working on this internally, and here is why its difficult to 
simplify it as +msr/-msr, +cpuid etc. Yes these are things that possibly
controlled by microcode, but microcode has several parts, not just CPU microcode.

The revision you see is just one big running number. There are other parts of the 
microcode that you don't see its internal version. In addition some parts of the
microcode are effective only when deployed by FIT. That cpu picks up right after
reset. During late load, even though you see the version updated, it doesn't
mean all the internal versions of ucode are latched and effective.

In addition, there are differences how some mitigations are deployed, as you know
some have MSR's that OS can find, there are others that need BIOS/early boot to 
make the mitigations effective. In order to get a full picture of weather a microcode 
file is late-loadable you need to know a lot more about how we got here to this
version loaded on the CPU.
> 
> Thank you,
> Mihai
> 
> >This patch set has the following patches:
> >
> >- patch 1 is introducing a new metadata file that comes with the microcode
> >(provided by the CPU manufacture) that describes what modifications are
> >done by loading the new microcode
> >
> >- patch 2 parses the metadata file and is verifying it against kernel
> >policy. In this patch, as an RFC, as a kernel policy, it was imposed
> >the rule of not allowing to remove any feature. If so, it won't be
> >loaded a new microcode. The policy can be further extended and describe
> >in different ways

Haven't read the individual patches yet. but you would need every interim
patch metadata to be always available. 

Since if you move from patch x->y you can find that an msr was removed.
But say you go from patch x->z, but there was no msr removed in patch y-z.
You need to process and collate all the msr/cpuid to comprehend what changing
between x->z since that also removes the msr for e.g.

To add more complications CPU has fuses and each patch content can be effective
on some but not all SKU's. So the exact same microcode can have different behavior
depending on which SKU its loaded into. So generating the common meta-data file
is also not a trivial process.

Cheers,
Ashok