Re: [PATCH v9 0/8] Parallel CPU bringup for x86_64

From: David Woodhouse <dwmw2@infradead.org>
To: Brian Gerst <brgerst@gmail.com>
Cc: Usama Arif <usama.arif@bytedance.com>,
	tglx@linutronix.de, kim.phillips@amd.com, arjan@linux.intel.com,
	mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com,
	hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com,
	paulmck@kernel.org, linux-kernel@vger.kernel.org,
	kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de,
	hewenliang4@huawei.com, thomas.lendacky@amd.com,
	seanjc@google.com, pmenzel@molgen.mpg.de,
	fam.zheng@bytedance.com, punit.agrawal@bytedance.com,
	simon.evans@bytedance.com, liangma@liangbit.com
Subject: Re: [PATCH v9 0/8] Parallel CPU bringup for x86_64
Date: Wed, 22 Feb 2023 12:53:02 +0000	[thread overview]
Message-ID: <EAAA15B0-D402-4EFB-BB65-2E6457698C1C@infradead.org> (raw)
In-Reply-To: <CAMzpN2gEeNY87+ywgizusPVG2fLiRx3v__3Bmqjo_eJLv+5k7Q@mail.gmail.com>

On 22 February 2023 12:08:04 GMT, Brian Gerst <brgerst@gmail.com> wrote:
>On Wed, Feb 22, 2023 at 5:44 AM David Woodhouse <dwmw2@infradead.org> wrote:
>>
>> On Wed, 2023-02-15 at 14:54 +0000, Usama Arif wrote:
>> > The main change over v8 is dropping the patch to avoid repeated saves of MTRR
>> > at boot time. It didn't make a difference to smpboot time and is independent
>> > of parallel CPU bringup, so if needed can be explored in a separate patchset.
>> >
>> > The patches have also been rebased to v6.2-rc8 and retested and the
>> > improvement in boot time is the same as v8.
>>
>> Thanks for picking this up, Usama.
>>
>> So the next thing that might be worth looking at is allowing the APs
>> all to be running their hotplug thread simultaneously, bringing
>> themselves from CPUHP_BRINGUP_CPU to CPUHP_AP_ONLINE. This series eats
>> the initial INIT/SIPI/SIPI latency, but if there's any significant time
>> in the AP hotplug thread, that could be worth parallelising.
>>
>> There may be further wins in the INIT/SIPI/SIPI too. Currently we
>> process each CPU at a time, sending INIT, SIPI, waiting 10µs and
>> sending another SIPI.
>>
>> What if we sent the first INIT+SIPI to all CPUs, then did another pass
>> sending another SIPI only to those which hadn't already started running
>> and set their bit in cpu_initialized_mask ?
>>
>> Might not be worth it, and there's an added complexity that they all
>> have to wait for each other (on the real mode trampoline lock) before
>> they can take their turn and get as far as setting their bit in
>> cpu_initialized_mask. So we'd probably end up sending the second SIPI
>> to most of them *anyway*.
>
>Speaking of next steps, I have a followup patchset ready to go that
>removes the global variables initial_stack, initial_gs, and
>early_gdt_descr.  Should I send that now or wait until this patchset
>lands in -tip?

Happy either way. Want to send it and we can take a look at whether to work it in with this?