kvm.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: David Woodhouse <dwmw2@infradead.org>
To: Tom Lendacky <thomas.lendacky@amd.com>,
	Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
	Dave Hansen <dave.hansen@linux.intel.com>,
	"x86@kernel.org" <x86@kernel.org>,
	"H . Peter Anvin" <hpa@zytor.com>,
	Paolo Bonzini <pbonzini@redhat.com>,
	"Paul E . McKenney" <paulmck@kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"rcu@vger.kernel.org" <rcu@vger.kernel.org>,
	"mimoja@mimoja.de" <mimoja@mimoja.de>,
	"hewenliang4@huawei.com" <hewenliang4@huawei.com>,
	"hushiyuan@huawei.com" <hushiyuan@huawei.com>,
	"luolongjun@huawei.com" <luolongjun@huawei.com>,
	"hejingxian@huawei.com" <hejingxian@huawei.com>
Subject: Re: [PATCH v3 0/9] Parallel CPU bringup for x86_64
Date: Fri, 17 Dec 2021 19:11:12 +0000	[thread overview]
Message-ID: <2bfb13ed5d565ab09bd794f69a6ef2b1b75e507a.camel@infradead.org> (raw)
In-Reply-To: <1401c5a1-c8a2-cca1-e548-cab143f59d8f@amd.com>

[-- Attachment #1: Type: text/plain, Size: 8626 bytes --]

On Fri, 2021-12-17 at 11:48 -0600, Tom Lendacky wrote:
> On 12/16/21 6:13 PM, David Woodhouse wrote:
> > On Thu, 2021-12-16 at 16:52 -0600, Tom Lendacky wrote:
> > > On baremetal, I haven't seen an issue. This only seems to have a problem
> > > with Qemu/KVM.
> > > 
> > > With 191f08997577 I could boot without issues with and without the
> > > no_parallel_bringup. Only after I applied e78fa57dd642 did the failure happen.
> > > 
> > > With e78fa57dd642 I could boot 64 vCPUs pretty consistently, but when I
> > > jumped to 128 vCPUs it failed again. When I moved the series to
> > > df9726cb7178, then 64 vCPUs also failed pretty consistently.
> > > 
> > > Strange thing is it is random. Sometimes (rarely) it works on the first
> > > boot and then sometimes it doesn't, at which point it will reset and
> > > reboot 3 or 4 times and then make it past the failure and fully boot.
> > 
> > Hm, some of that is just artifacts of timing, I'm sure. But now I'm
> > staring at the way that early_setup_idt() can run in parallel on all
> > CPUs, rewriting bringup_idt_descr and loading it.
> > 
> > To start with, let's try unlocking the trampoline_lock much later,
> > after cpu_init_exception_handling() has loaded the real IDT.
> > 
> > I think we can probably make secondaries load the real IDT early and
> > never use bringup_idt_descr at all, can't we? But let's see if this
> > makes it go away, to start with...
> > 
> 
> This still fails. I ran with -d cpu_reset on the command line and will
> forward the full log to you. I ran "grep "[ER]IP=" stderr.log | uniq -c"
> and got:
> 
>      128 EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=0 SMM=0 HLT=0
>      128 EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> These are before running any of the vCPUs.
>
>        1 RIP=ffffffff810705c6 RFL=00000206 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> This is where vCPU0 is at the time of the reset. This address tends to
> be different all the time and so I think it is just where it happens to
> be when the reset occurs and isn't contributing to the reset.

I note that one is in native_write_msr() though. I wonder what it's writing?

Do you have console output (perhaps with earlyprintk=ttyS0) to go with this?

>        5 RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
>        1 RIP=ffffffff8104af06 RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
>       15 RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> These are some of the APs and all are in wait_for_master_cpu().

As is right and proper. They should be coming up to that point and
waiting for the... erm... controlling CPU to tell them to go any
further.


>        1 EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0
> This seems ok because: CS =9900 00099000 0000ffff 00009b00
> So likely in the trampoline code.

Yeah, that'll be in the bitlock waiting for its turn through the real
mode stack.

    1010:       66 0f ba 26 18          btw    $0x18,(%esi)
    1015:       40                      inc    %eax
    1016:       00 73 04                add    %dh,0x4(%ebx)
    1019:       f3 90                   pause  
    101b:       eb f3                   jmp    1010 <trampoline_start+0x10>
>        1 EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> This one seems odd... could it be the one causing the reset?
> CS =f000 ffff0000 0000ffff 00009a00


Yeah. I'm finding it slightly easier without the 'uniq'...

> CPU Reset (CPU 0)
> RIP=ffffffff810705c6 RFL=00000206 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 1)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 2)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 3)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 4)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 5)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 6)
> RIP=ffffffff8104af06 RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 7)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 8)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 9)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 10)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 11)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 12)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 13)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 14)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 15)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 16)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 17)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 18)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 19)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 20)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 21)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0

All those came up and are waiting in wait_for_master_cpu() as they
should.


> CPU Reset (CPU 22)
> EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0

That one's in the bitlock, also waiting.

> CPU Reset (CPU 23)
> EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0

This one we suspect. Is this what a triple-fault would look like? Not
if it's *already* at f000:fff0, surely? 

CPU Reset (CPU 23)
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00800f12
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009a00
SS =0000 00000000 0000ffff 00009200
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008300
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=00000000 CCD=00000000 CCO=DYNAMIC 
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=0000000000000000 0000000000000000 XMM01=0000000000000000 0000000000000000
XMM02=0000000000000000 0000000000000000 XMM03=0000000000000000 0000000000000000
XMM04=0000000000000000 0000000000000000 XMM05=0000000000000000 0000000000000000
XMM06=0000000000000000 0000000000000000 XMM07=0000000000000000 0000000000000000


> CPU Reset (CPU 24)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 25)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 26)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0

These ones made it through the real mode first and are also waiting.

> CPU Reset (CPU 27)
> EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 28)
> EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 29)

Still in the real mode bitlock. And after this point they are still
halted in presumably 32-bit BIOS code because the BSP hasn't even
touched them yet.

> EIP=3f36e11b EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1
> CPU Reset (CPU 30)
> EIP=3f36e11b EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1
> CPU Reset (CPU 31)
> EIP=3f36e11b EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1
> CPU Reset (CPU 32)
> ...
> CPU Reset (CPU 127)
> EIP=3f36e11b EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1



[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]

  reply	other threads:[~2021-12-17 19:11 UTC|newest]

Thread overview: 57+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-12-15 14:56 [PATCH v3 0/9] Parallel CPU bringup for x86_64 David Woodhouse
2021-12-15 14:56 ` [PATCH v3 1/9] x86/apic/x2apic: Fix parallel handling of cluster_mask David Woodhouse
2021-12-15 14:56 ` [PATCH v3 2/9] cpu/hotplug: Move idle_thread_get() to <linux/smpboot.h> David Woodhouse
2021-12-15 14:56 ` [PATCH v3 3/9] cpu/hotplug: Add dynamic parallel bringup states before CPUHP_BRINGUP_CPU David Woodhouse
2021-12-15 14:56 ` [PATCH v3 4/9] x86/smpboot: Reference count on smpboot_setup_warm_reset_vector() David Woodhouse
2021-12-15 14:56 ` [PATCH v3 5/9] x86/smpboot: Split up native_cpu_up into separate phases and document them David Woodhouse
2021-12-15 14:56 ` [PATCH v3 6/9] x86/smpboot: Support parallel startup of secondary CPUs David Woodhouse
2021-12-16 14:24   ` Tom Lendacky
2021-12-16 18:24     ` David Woodhouse
2021-12-16 19:00       ` Tom Lendacky
2021-12-16 19:20         ` David Woodhouse
2022-01-29 12:04           ` David Woodhouse
2022-01-31 13:59             ` Borislav Petkov
2022-02-01 10:25               ` David Woodhouse
2022-02-01 10:56                 ` Borislav Petkov
2022-02-01 12:39                   ` David Woodhouse
2022-02-01 12:56                     ` Borislav Petkov
2022-02-01 13:02                       ` David Woodhouse
2021-12-15 14:56 ` [PATCH v3 7/9] x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel David Woodhouse
2021-12-15 14:56 ` [PATCH v3 8/9] x86/mtrr: Avoid repeated save of MTRRs on boot-time CPU bringup David Woodhouse
2021-12-15 14:56 ` [PATCH v3 9/9] x86/smpboot: Serialize topology updates for secondary bringup David Woodhouse
2021-12-16 16:27 ` [PATCH v3 0/9] Parallel CPU bringup for x86_64 Tom Lendacky
2021-12-16 19:24   ` David Woodhouse
2021-12-16 22:52     ` Tom Lendacky
2021-12-17  0:13       ` David Woodhouse
2021-12-17 10:09         ` Igor Mammedov
2021-12-17 15:40           ` David Woodhouse
2021-12-20 17:10           ` David Woodhouse
2021-12-20 18:54             ` Tom Lendacky
2021-12-20 21:29               ` David Woodhouse
2021-12-20 21:47                 ` Tom Lendacky
2021-12-21 22:25                   ` Tom Lendacky
2021-12-21 22:33                     ` David Woodhouse
2021-12-17 17:48         ` Tom Lendacky
2021-12-17 19:11           ` David Woodhouse [this message]
2021-12-17 19:26             ` David Woodhouse
2021-12-17 20:15               ` Tom Lendacky
2021-12-17 19:46             ` Tom Lendacky
2021-12-17 20:13               ` David Woodhouse
2021-12-17 20:55                 ` Tom Lendacky
2021-12-17 22:48                   ` David Woodhouse
2022-01-28  9:54                   ` David Woodhouse
2022-01-28 21:40                     ` Sean Christopherson
2022-01-28 21:48                       ` David Woodhouse
2022-01-29  9:22                       ` David Woodhouse
2021-12-16 19:52   ` David Woodhouse
2021-12-16 19:55     ` Tom Lendacky
2021-12-16 19:59       ` David Woodhouse
2021-12-27 16:57 ` Paul Menzel
2021-12-28 11:34   ` Paul Menzel
2021-12-28 14:18     ` David Woodhouse
2021-12-29 13:18       ` Paul Menzel
2021-12-29 13:54         ` David Woodhouse
2022-02-14 13:45           ` Paul Menzel
2022-04-21 10:00             ` Mimoja
2022-04-22 21:19               ` Tom Lendacky
2022-06-01  8:30                 ` David Woodhouse

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=2bfb13ed5d565ab09bd794f69a6ef2b1b75e507a.camel@infradead.org \
    --to=dwmw2@infradead.org \
    --cc=bp@alien8.de \
    --cc=dave.hansen@linux.intel.com \
    --cc=hejingxian@huawei.com \
    --cc=hewenliang4@huawei.com \
    --cc=hpa@zytor.com \
    --cc=hushiyuan@huawei.com \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=luolongjun@huawei.com \
    --cc=mimoja@mimoja.de \
    --cc=mingo@redhat.com \
    --cc=paulmck@kernel.org \
    --cc=pbonzini@redhat.com \
    --cc=rcu@vger.kernel.org \
    --cc=tglx@linutronix.de \
    --cc=thomas.lendacky@amd.com \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).