Re: [PATCH v3 0/9] Parallel CPU bringup for x86_64

[Paul Menzel <pmenzel@molgen.mpg.de>]

Dear David,


Am 28.12.21 um 15:18 schrieb David Woodhouse:
> On Tue, 2021-12-28 at 12:34 +0100, Paul Menzel wrote:
>> Same on the ASUS F2A85-M PRO with AMD A6-6400K. Without serial console,
>> the messages below are printed below to the monitor after nine seconds.
>>
>>        [    1.078879] smp: Bringing up secondary CPUs ...
>>        [    1.080950] x86: Booting SMP configuration:
>>
>> Please find the serial log attached.
> 
> Thanks for testing. That looks like the same triple-fault on bringup
> that we have been seeing, and that I reproduced without my patches
> using kexec all the way back to a 5.0 kernel.
> 
> Out of interest, are you also able to reproduce it with kexec and
> without the parallel bringup?

No, I am not able to reproduce that with Debian’s 
*linux-image-5.15.0-2-686*, and kexec. With this board, 
`module_blacklist=radeon` is needed, as the driver *radeon* is not able 
to deal with kexec – and amdgpu neither [1].

```
[    3.349911] [drm] Found VCE firmware/feedback version 50.0.1 / 17!
[    3.365259] clocksource: Switched to clocksource tsc
[    3.365284] [drm] GART: num cpu pages 262144, num gpu pages 262144
[    3.405159] random: fast init done
[    3.420492] [drm] PCIE GART of 1024M enabled (table at 
0x00000000001D6000).
[    3.427634] radeon 0000:00:01.0: WB enabled
[    3.431828] radeon 0000:00:01.0: fence driver on ring 0 use gpu addr 
0x0000000020000c00
[    3.440100] radeon 0000:00:01.0: fence driver on ring 5 use gpu addr 
0x0000000000075a18
[    3.458182] radeon 0000:00:01.0: failed VCE resume (-22).
[    3.463591] radeon 0000:00:01.0: fence driver on ring 1 use gpu addr 
0x0000000020000c04
[    3.471615] radeon 0000:00:01.0: fence driver on ring 2 use gpu addr 
0x0000000020000c08
[    3.479636] radeon 0000:00:01.0: fence driver on ring 3 use gpu addr 
0x0000000020000c0c
[    3.487650] radeon 0000:00:01.0: fence driver on ring 4 use gpu addr 
0x0000000020000c10
[    3.495990] radeon 0000:00:01.0: radeon: MSI limited to 32-bit
[    3.502008] radeon 0000:00:01.0: radeon: using MSI.
[    3.506906] ata7: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    3.506918] [drm] radeon: irq initialized.
[
```

> And with that patch I sent Tom in
> https://lore.kernel.org/lkml/721484e0fa719e99f9b8f13e67de05033dd7cc86.camel@infradead.org/
>   to expand the bitlock exclusion and stop the bringup being truly in
> parallel at all?

No, this does not help, and the Linux kernel resets at the same spot.

```
[    1.036036] smpboot: smpboot: After 
x86_init.timers.setup_percpu_clockev()
[    1.037031] smpboot: smp_get_logical_apicid()
[    1.038031] smpboot: CPU0: AMD A6-6400K APU with Radeon(tm) HD 
Graphics (family: 0x15, model: 0x13, stepping: 0x1)
[    1.039366] Performance Events: Fam15h core perfctr, AMD PMU driver.
[    1.040033] ... version:                0
[    1.041031] ... bit width:              48
[    1.042031] ... generic registers:      6
[    1.043033] ... value mask:             0000ffffffffffff
[    1.044031] ... max period:             00007fffffffffff
[    1.045031] ... fixed-purpose events:   0
[    1.046031] ... event mask:             000000000000003f
[    1.048065] rcu: Hierarchical SRCU implementation.
[    1.050642] NMI watchdog: Enabled. Permanently consumes one hw-PMU 
counter.
[    1.051133] smp: Bringing up secondary CPUs ...
[    1.053202] x86: Booting SMP configuration:
```

> Or the one in
> https://lore.kernel.org/lkml/d4cde50b4aab24612823714dfcbe69bc4bb63b60.camel@infradead.org
> which makes it do nothing except prepare all the CPUs before bringing
> them up one at a time?

I applied it on top the other one, and it made no difference either.

> My current theory (not that I've spent that much time thinking about it
> in the last week) is that there's something about the existing CPU
> bringup, possibly a CPU bug or something special about the AMD CPUs,
> which is triggered by just making it a little bit *faster*, which is
> why bringing them up from kexec (especially in qemu) can cause it too?

Would having the serial console enabled make a difference?

> Tom seemed to find that it was in load_TR_desc(), so if you could try
> this hack on a machine that doesn't magically wink out of existence on
> a triplefault before even flushing its serial output, that would be
> much appreciated...
> 
> diff --git a/arch/x86/include/asm/desc.h b/arch/x86/include/asm/desc.h
> index ab97b22ac04a..cc6590712ff4 100644
> --- a/arch/x86/include/asm/desc.h
> +++ b/arch/x86/include/asm/desc.h
> @@ -8,7 +8,7 @@
>   #include <asm/fixmap.h>
>   #include <asm/irq_vectors.h>
>   #include <asm/cpu_entry_area.h>
> -
> +#include <asm/io.h>
>   #include <linux/debug_locks.h>
>   #include <linux/smp.h>
>   #include <linux/percpu.h>
> @@ -265,11 +265,16 @@ static inline void native_load_tr_desc(void)
>   	 * If the current GDT is the read-only fixmap, swap to the original
>   	 * writeable version. Swap back at the end.
>   	 */
> +	outb('d', 0x3f8);
>   	if (gdt.address == (unsigned long)fixmap_gdt) {
> +	outb('e', 0x3f8);
>   		load_direct_gdt(cpu);
>   		restore = 1;
> +	outb('f', 0x3f8);
>   	}
> +	outb('g', 0x3f8);
>   	asm volatile("ltr %w0"::"q" (GDT_ENTRY_TSS*8));
> +	outb('h', 0x3f8);
>   	if (restore)
>   		load_fixmap_gdt(cpu);
>   }
> diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
> index 0083464de5e3..5bc8f30c3283 100644
> --- a/arch/x86/kernel/cpu/common.c
> +++ b/arch/x86/kernel/cpu/common.c
> @@ -1716,7 +1716,9 @@ void identify_secondary_cpu(struct cpuinfo_x86 *c)
>   	enable_sep_cpu();
>   #endif
>   	mtrr_ap_init();
> +outb('A', 0x3f8);
>   	validate_apic_and_package_id(c);
> +outb('B', 0x3f8);
>   	x86_spec_ctrl_setup_ap();
>   	update_srbds_msr();
>   }
> @@ -1957,6 +1959,7 @@ static inline void tss_setup_io_bitmap(struct tss_struct *tss)
>   	tss->io_bitmap.mapall[IO_BITMAP_LONGS] = ~0UL;
>   #endif
>   }
> +#include <asm/realmode.h>
>   
>   /*
>    * Setup everything needed to handle exceptions from the IDT, including the IST
> @@ -1969,16 +1972,24 @@ void cpu_init_exception_handling(void)
>   
>   	/* paranoid_entry() gets the CPU number from the GDT */
>   	setup_getcpu(cpu);
> -
> +	outb('\n', 0x3f8);
> +	outb('0' + cpu / 100, 0x3f8);
> +	outb('0' + (cpu % 100) / 10, 0x3f8);
> +	outb('0' + (cpu % 10), 0x3f8);
> +	
>   	/* IST vectors need TSS to be set up. */
>   	tss_setup_ist(tss);
> +	outb('a', 0x3f8);
>   	tss_setup_io_bitmap(tss);
>   	set_tss_desc(cpu, &get_cpu_entry_area(cpu)->tss.x86_tss);
> -
> +	outb('b', 0x3f8);
>   	load_TR_desc();
> +	outb('c', 0x3f8);
>   
>   	/* Finally load the IDT */
>   	load_current_idt();
> +	outb('z', 0x3f8);
> +
>   }
>   
>   /*

Unfortunately, no more messages were printed on the serial console.


Kind regards,

Paul


[1]: https://gitlab.freedesktop.org/drm/amd/-/issues/1597