[MODERATED] Re: [PATCH v2 2/8] MDSv2 1

[Borislav Petkov <bp@suse.de>]

On Thu, Dec 13, 2018 at 07:15:29AM -0800, speck for Andrew Cooper wrote:
> No - not if they can possibly avoid it.

Well, from the engineers of the cloud company I know - probably the
biggest one - I hear a completely different story and they *do* have to
reboot.

And in the cases where they reboot, they can update microcode too. I'm
willing to bet a lot of beers that microcode updates come a lot seldomly
than kernel updates.

And if it weren't for the year of mitigations 2018, late loading
wouldnt've even been a topic at all. Because microcode updates are
*really* that seldom. Patch memory being really limited and already
carrying some other patches, comes to mind.

So the normal flow on the boxes out there is, at some point you get the
updated microcode package which gets installed onto the system.

The next time you update the kernel or you have to reboot the box, you
regenerate the initrd and you have your microcode update ready too.
Done. Problem solved and then some.

> Within the constraints of "quiesce the system with stop_machine()", yes.

Except that is not enough in every case:

static bool is_blacklisted(unsigned int cpu)
{
        struct cpuinfo_x86 *c = &cpu_data(cpu);

        /*
         * Late loading on model 79 with microcode revision less than 0x0b000021
         * and LLC size per core bigger than 2.5MB may result in a system hang.
         * This behavior is documented in item BDF90, #334165 (Intel Xeon
         * Processor E7-8800/4800 v4 Product Family).
         */
        if (c->x86 == 6 &&
            c->x86_model == INTEL_FAM6_BROADWELL_X &&
            c->x86_stepping == 0x01 &&
            llc_size_per_core > 2621440 &&
            c->microcode < 0x0b000021) {
                pr_err_once("Erratum BDF90: late loading with revision < 0x0b000021 (0x%x) disabled.\n", c->microcode);
                pr_err_once("Please consider either early loading through initrd/built-in or a potential BIOS update.\n");

Yap, that's an erratum for late loading. Those customers can't do late
loading because shit freezes the box. Good luck!

And why are we doing such a complex conditional, you ask? Well, we're
trying to salvage the situation because, yeah, late loading is just a
*bad* *bad* idea.

And this is the first erratum I'm aware of. I betcha more beers there
are other problems with late loading. The debian people did blacklist a
couple of microcode revisions for that reason too, for example.

> I've got some customers to have said in no uncertain terms that they
> want to reboot to get mitigations for these forthcoming issues.  Said
> customers are large enough that Intel is currently engaged, and have
> tentatively said that it is fine to load ucode in parallel on every
> core during stop machine, rather than serially.

Who said that, Intel or the customers? If it is Intel, see above. If
it is the customers, they are more than free to support their own late
loading.

> I'm expecting Intel to propose this change in Linux as well as Xen.

I have seen the "proposal".

> I personally think everyone should reboot and call it done,

Exactly.

> but at the end of the day, I'm beholden to my customers, and they
> really really do want late load microcode to work.

Huh? They want to reboot but they still want late microcode loading? I
have no idea what your argument here is.

-- 
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
--