All of lore.kernel.org
 help / color / mirror / Atom feed
From: David Woodhouse <dwmw2@infradead.org>
To: Tom Lendacky <thomas.lendacky@amd.com>,
	Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
	Dave Hansen <dave.hansen@linux.intel.com>,
	"x86@kernel.org" <x86@kernel.org>,
	"H . Peter Anvin" <hpa@zytor.com>,
	Paolo Bonzini <pbonzini@redhat.com>,
	"Paul E . McKenney" <paulmck@kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"rcu@vger.kernel.org" <rcu@vger.kernel.org>,
	"mimoja@mimoja.de" <mimoja@mimoja.de>,
	"hewenliang4@huawei.com" <hewenliang4@huawei.com>,
	"hushiyuan@huawei.com" <hushiyuan@huawei.com>,
	"luolongjun@huawei.com" <luolongjun@huawei.com>,
	"hejingxian@huawei.com" <hejingxian@huawei.com>
Subject: Re: [PATCH v3 0/9] Parallel CPU bringup for x86_64
Date: Fri, 17 Dec 2021 19:11:12 +0000	[thread overview]
Message-ID: <2bfb13ed5d565ab09bd794f69a6ef2b1b75e507a.camel@infradead.org> (raw)
In-Reply-To: <1401c5a1-c8a2-cca1-e548-cab143f59d8f@amd.com>

[-- Attachment #1: Type: text/plain, Size: 8626 bytes --]

On Fri, 2021-12-17 at 11:48 -0600, Tom Lendacky wrote:
> On 12/16/21 6:13 PM, David Woodhouse wrote:
> > On Thu, 2021-12-16 at 16:52 -0600, Tom Lendacky wrote:
> > > On baremetal, I haven't seen an issue. This only seems to have a problem
> > > with Qemu/KVM.
> > > 
> > > With 191f08997577 I could boot without issues with and without the
> > > no_parallel_bringup. Only after I applied e78fa57dd642 did the failure happen.
> > > 
> > > With e78fa57dd642 I could boot 64 vCPUs pretty consistently, but when I
> > > jumped to 128 vCPUs it failed again. When I moved the series to
> > > df9726cb7178, then 64 vCPUs also failed pretty consistently.
> > > 
> > > Strange thing is it is random. Sometimes (rarely) it works on the first
> > > boot and then sometimes it doesn't, at which point it will reset and
> > > reboot 3 or 4 times and then make it past the failure and fully boot.
> > 
> > Hm, some of that is just artifacts of timing, I'm sure. But now I'm
> > staring at the way that early_setup_idt() can run in parallel on all
> > CPUs, rewriting bringup_idt_descr and loading it.
> > 
> > To start with, let's try unlocking the trampoline_lock much later,
> > after cpu_init_exception_handling() has loaded the real IDT.
> > 
> > I think we can probably make secondaries load the real IDT early and
> > never use bringup_idt_descr at all, can't we? But let's see if this
> > makes it go away, to start with...
> > 
> 
> This still fails. I ran with -d cpu_reset on the command line and will
> forward the full log to you. I ran "grep "[ER]IP=" stderr.log | uniq -c"
> and got:
> 
>      128 EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=0 SMM=0 HLT=0
>      128 EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> These are before running any of the vCPUs.
>
>        1 RIP=ffffffff810705c6 RFL=00000206 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> This is where vCPU0 is at the time of the reset. This address tends to
> be different all the time and so I think it is just where it happens to
> be when the reset occurs and isn't contributing to the reset.

I note that one is in native_write_msr() though. I wonder what it's writing?

Do you have console output (perhaps with earlyprintk=ttyS0) to go with this?

>        5 RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
>        1 RIP=ffffffff8104af06 RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
>       15 RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> These are some of the APs and all are in wait_for_master_cpu().

As is right and proper. They should be coming up to that point and
waiting for the... erm... controlling CPU to tell them to go any
further.


>        1 EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0
> This seems ok because: CS =9900 00099000 0000ffff 00009b00
> So likely in the trampoline code.

Yeah, that'll be in the bitlock waiting for its turn through the real
mode stack.

    1010:       66 0f ba 26 18          btw    $0x18,(%esi)
    1015:       40                      inc    %eax
    1016:       00 73 04                add    %dh,0x4(%ebx)
    1019:       f3 90                   pause  
    101b:       eb f3                   jmp    1010 <trampoline_start+0x10>
>        1 EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> This one seems odd... could it be the one causing the reset?
> CS =f000 ffff0000 0000ffff 00009a00


Yeah. I'm finding it slightly easier without the 'uniq'...

> CPU Reset (CPU 0)
> RIP=ffffffff810705c6 RFL=00000206 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 1)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 2)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 3)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 4)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 5)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 6)
> RIP=ffffffff8104af06 RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 7)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 8)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 9)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 10)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 11)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 12)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 13)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 14)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 15)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 16)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 17)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 18)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 19)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 20)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 21)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0

All those came up and are waiting in wait_for_master_cpu() as they
should.


> CPU Reset (CPU 22)
> EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0

That one's in the bitlock, also waiting.

> CPU Reset (CPU 23)
> EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0

This one we suspect. Is this what a triple-fault would look like? Not
if it's *already* at f000:fff0, surely? 

CPU Reset (CPU 23)
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00800f12
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009a00
SS =0000 00000000 0000ffff 00009200
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008300
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=00000000 CCD=00000000 CCO=DYNAMIC 
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=0000000000000000 0000000000000000 XMM01=0000000000000000 0000000000000000
XMM02=0000000000000000 0000000000000000 XMM03=0000000000000000 0000000000000000
XMM04=0000000000000000 0000000000000000 XMM05=0000000000000000 0000000000000000
XMM06=0000000000000000 0000000000000000 XMM07=0000000000000000 0000000000000000


> CPU Reset (CPU 24)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 25)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 26)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0

These ones made it through the real mode first and are also waiting.

> CPU Reset (CPU 27)
> EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 28)
> EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 29)

Still in the real mode bitlock. And after this point they are still
halted in presumably 32-bit BIOS code because the BSP hasn't even
touched them yet.

> EIP=3f36e11b EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1
> CPU Reset (CPU 30)
> EIP=3f36e11b EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1
> CPU Reset (CPU 31)
> EIP=3f36e11b EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1
> CPU Reset (CPU 32)
> ...
> CPU Reset (CPU 127)
> EIP=3f36e11b EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1



[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]

  reply	other threads:[~2021-12-17 19:11 UTC|newest]

Thread overview: 57+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-12-15 14:56 [PATCH v3 0/9] Parallel CPU bringup for x86_64 David Woodhouse
2021-12-15 14:56 ` [PATCH v3 1/9] x86/apic/x2apic: Fix parallel handling of cluster_mask David Woodhouse
2021-12-15 14:56 ` [PATCH v3 2/9] cpu/hotplug: Move idle_thread_get() to <linux/smpboot.h> David Woodhouse
2021-12-15 14:56 ` [PATCH v3 3/9] cpu/hotplug: Add dynamic parallel bringup states before CPUHP_BRINGUP_CPU David Woodhouse
2021-12-15 14:56 ` [PATCH v3 4/9] x86/smpboot: Reference count on smpboot_setup_warm_reset_vector() David Woodhouse
2021-12-15 14:56 ` [PATCH v3 5/9] x86/smpboot: Split up native_cpu_up into separate phases and document them David Woodhouse
2021-12-15 14:56 ` [PATCH v3 6/9] x86/smpboot: Support parallel startup of secondary CPUs David Woodhouse
2021-12-16 14:24   ` Tom Lendacky
2021-12-16 18:24     ` David Woodhouse
2021-12-16 19:00       ` Tom Lendacky
2021-12-16 19:20         ` David Woodhouse
2022-01-29 12:04           ` David Woodhouse
2022-01-31 13:59             ` Borislav Petkov
2022-02-01 10:25               ` David Woodhouse
2022-02-01 10:56                 ` Borislav Petkov
2022-02-01 12:39                   ` David Woodhouse
2022-02-01 12:56                     ` Borislav Petkov
2022-02-01 13:02                       ` David Woodhouse
2021-12-15 14:56 ` [PATCH v3 7/9] x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel David Woodhouse
2021-12-15 14:56 ` [PATCH v3 8/9] x86/mtrr: Avoid repeated save of MTRRs on boot-time CPU bringup David Woodhouse
2021-12-15 14:56 ` [PATCH v3 9/9] x86/smpboot: Serialize topology updates for secondary bringup David Woodhouse
2021-12-16 16:27 ` [PATCH v3 0/9] Parallel CPU bringup for x86_64 Tom Lendacky
2021-12-16 19:24   ` David Woodhouse
2021-12-16 22:52     ` Tom Lendacky
2021-12-17  0:13       ` David Woodhouse
2021-12-17 10:09         ` Igor Mammedov
2021-12-17 15:40           ` David Woodhouse
2021-12-20 17:10           ` David Woodhouse
2021-12-20 18:54             ` Tom Lendacky
2021-12-20 21:29               ` David Woodhouse
2021-12-20 21:47                 ` Tom Lendacky
2021-12-21 22:25                   ` Tom Lendacky
2021-12-21 22:33                     ` David Woodhouse
2021-12-17 17:48         ` Tom Lendacky
2021-12-17 19:11           ` David Woodhouse [this message]
2021-12-17 19:26             ` David Woodhouse
2021-12-17 20:15               ` Tom Lendacky
2021-12-17 19:46             ` Tom Lendacky
2021-12-17 20:13               ` David Woodhouse
2021-12-17 20:55                 ` Tom Lendacky
2021-12-17 22:48                   ` David Woodhouse
2022-01-28  9:54                   ` David Woodhouse
2022-01-28 21:40                     ` Sean Christopherson
2022-01-28 21:48                       ` David Woodhouse
2022-01-29  9:22                       ` David Woodhouse
2021-12-16 19:52   ` David Woodhouse
2021-12-16 19:55     ` Tom Lendacky
2021-12-16 19:59       ` David Woodhouse
2021-12-27 16:57 ` Paul Menzel
2021-12-28 11:34   ` Paul Menzel
2021-12-28 14:18     ` David Woodhouse
2021-12-29 13:18       ` Paul Menzel
2021-12-29 13:54         ` David Woodhouse
2022-02-14 13:45           ` Paul Menzel
2022-04-21 10:00             ` Mimoja
2022-04-22 21:19               ` Tom Lendacky
2022-06-01  8:30                 ` David Woodhouse

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=2bfb13ed5d565ab09bd794f69a6ef2b1b75e507a.camel@infradead.org \
    --to=dwmw2@infradead.org \
    --cc=bp@alien8.de \
    --cc=dave.hansen@linux.intel.com \
    --cc=hejingxian@huawei.com \
    --cc=hewenliang4@huawei.com \
    --cc=hpa@zytor.com \
    --cc=hushiyuan@huawei.com \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=luolongjun@huawei.com \
    --cc=mimoja@mimoja.de \
    --cc=mingo@redhat.com \
    --cc=paulmck@kernel.org \
    --cc=pbonzini@redhat.com \
    --cc=rcu@vger.kernel.org \
    --cc=tglx@linutronix.de \
    --cc=thomas.lendacky@amd.com \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.