[MODERATED] ucode packaging

[MODERATED] ucode packaging

[Andrew Cooper]

[-- Attachment #1: Type: text/plain, Size: 2087 bytes --]

Hello,

I realise this is perhaps not totally on topic, but I thought I'd
solicit feedback here first before expanding to a wider audience, where
some details might need to be moderated.

tl;dr The MDS microcode for SandyBridge has issues, and causing fatal
MCEs on previously-stable systems.  While I don't wish to criticise, and
am certain that Intel do their best to avoid regressions, it has to be
said that newer microcode is a bit like newer software, in that has a
non-zero chance of containing bugs, and the severity of bugs tends to be
catastrophic far as stability goes.

Anyway, customers and management have expressed concerns about the ease
of rolling back microcode versions in the field, to stabilise impacted
systems.

Currently, /lib/firmware/$VENDOR-ucode/ contains exactly one version of
any particular piece of microcode for the part.  Altering this either
requires finding a different microcode binary, or finding a different
version of the package, and typically force installing it.  Neither are
great options for users.

It is already common to have the past $N kernels available for rollback
in the face of problems, and this seems like a sensible approach to take
for microcode as well.

Expressing this via separate packages, as the kernel typically is, is
most likely too complicated to work sensibly, but luckily ucode blobs
are tiny.  How about a scheme whereby a single package has files in the
form of /lib/firmware/$VENDOR-ucode/FF-MM-SS-VERSION and a symlink of
the latest version to FF-MM-SS, for compatibility with existing initrd
generation tools?

This way, if a user finds themselves in need of reverting to an older
version, they can relink FF-MM-SS to an older version, rebuild their
initrd and reboot.  This has the advantage that it will stay in effect
until a newer microcode package is installed, which will hopefully
retract (or better yet, superseded) the problematic version.

Thoughts?  Is there any interest from other distros for a scheme such as
this?

Thanks in advance,

~Andrew

[MODERATED] Re: ucode packaging

[Borislav Petkov]

On Thu, Oct 03, 2019 at 02:46:41AM +0100, speck for Andrew Cooper wrote:
> Expressing this via separate packages, as the kernel typically is, is
> most likely too complicated to work sensibly, but luckily ucode blobs
> are tiny.  How about a scheme whereby a single package has files in the
> form of /lib/firmware/$VENDOR-ucode/FF-MM-SS-VERSION and a symlink of
> the latest version to FF-MM-SS, for compatibility with existing initrd
> generation tools?

I don't see a problem with that since no changes will be needed to the
existing tools - they would only need to be tested whether they can
handle the symlink thing properly.

Some distros, for example, already call some ucode blobs
FF-MM-SS.initramfs, for example, to enforce only early loading of those
due to bugs when loaded late. I believe we did that in our dracut too...

-- 
Regards/Gruss,
    Boris.

SUSE Software Solutions Germany GmbH, GF: Felix Imendörffer, HRB 247165, AG München
-- 

[MODERATED] Re: ucode packaging

[Greg KH]

On Thu, Oct 03, 2019 at 02:46:41AM +0100, speck for Andrew Cooper wrote:
> Hello,
> 
> I realise this is perhaps not totally on topic, but I thought I'd
> solicit feedback here first before expanding to a wider audience, where
> some details might need to be moderated.

The issue of microcode fall-back isn't something that needs to be
handled here in a private list.  No need to bring up the recent
microcode failures, just talk about it as a "what happens if this goes
wrong" type of thing.

thanks,

greg k-h

[MODERATED] Re: ***UNCHECKED*** ucode packaging

[Jiri Kosina]

On Thu, 3 Oct 2019, speck for Andrew Cooper wrote:

> Anyway, customers and management have expressed concerns about the ease
> of rolling back microcode versions in the field, to stabilise impacted
> systems.

As most (all?) distros are shipping intel-ucode as a standard package that 
can be easily downgraded, is there really any value in trying to solve 
this outside of standard distro package management system?

Thanks,

-- 
Jiri Kosina
SUSE Labs