[RFC 0/6] S3 SMP support with physcial CPU hotplug

[RFC 0/6] S3 SMP support with physcial CPU hotplug

[Li Shaohua]

Hi,
The following 6 patches try to add suspend-to-ram (or S3) SMP support
for IA32. It's for support HT based system suspend/resume currently and
most of the code are also useful for physical CPU hotplug.

In a SMP system, after S3 resume, the BP is starting to execute the ACPI
wakeup address just like the UP case. And the APs possibly are in a
BIOS's busy loop. This just looks like the boot time case, we must use a
SIPI circle to wakeup the APs.

We uses the CPU hotplug infrastructure. In order to reuse the SMP boot
code, we clean up all CPU states after the CPU is dead, including its
idle thread, runqueue and other CPU states. Since the CPU is in idle
thread before suspend, we don't require to save and restore after resume
most of the CPU states.

Now the sequences of S3 are:
1. hotremove all APs, put them into idle thread.
2. follow UP S3 code path.
3. warm boot all APs.
4. UP all APs.

The patches are against 2.6.11-rc1 with Zwane's CPU hotplug patch in -mm
tree. To test the SMP S3, please don't enable MTRR driver (it's SMP
broken for Suspend/resume). And please kill syslogd, there is a bug in
the sususpend/resume refrigerator mechanism, which can be fixed by
swsusp2 refrigerator.
I'm looking forward to your comments. Thanks in advance!

Thanks,
Shaohua

Re: [RFC 0/6] S3 SMP support with physcial CPU hotplug

[Andrew Morton]

Li Shaohua <shaohua.li@intel.com> wrote:
>
> The patches are against 2.6.11-rc1 with Zwane's CPU hotplug patch in -mm
>  tree.

Should I merge that thing into mainline?  It seems that a few people are
needing it.

Re: [RFC 0/6] S3 SMP support with physcial CPU hotplug

[Li Shaohua]

On Mon, 2005-04-04 at 10:37, Andrew Morton wrote:
> Li Shaohua <shaohua.li@intel.com> wrote:
> >
> > The patches are against 2.6.11-rc1 with Zwane's CPU hotplug patch in -mm
> >  tree.
> 
> Should I merge that thing into mainline?  It seems that a few people are
> needing it.
I'd like to listen to some comments first. There are still some things
I'm not sure, such as the do_exit_idle.

Thanks,
Shaohua

Re: [RFC 0/6] S3 SMP support with physcial CPU hotplug

[Andrew Morton]

Li Shaohua <shaohua.li@intel.com> wrote:
>
> On Mon, 2005-04-04 at 10:37, Andrew Morton wrote:
> > Li Shaohua <shaohua.li@intel.com> wrote:
> > >
> > > The patches are against 2.6.11-rc1 with Zwane's CPU hotplug patch in -mm
> > >  tree.
> > 
> > Should I merge that thing into mainline?  It seems that a few people are
> > needing it.
> I'd like to listen to some comments first. There are still some things
> I'm not sure, such as the do_exit_idle.
> 

I was referring to Zwane's i386-cpu-hotplug-updated-for-mm.patch

Re: [RFC 0/6] S3 SMP support with physcial CPU hotplug

[Li Shaohua]

On Mon, 2005-04-04 at 10:48, Andrew Morton wrote:
> Li Shaohua <shaohua.li@intel.com> wrote:
> >
> > On Mon, 2005-04-04 at 10:37, Andrew Morton wrote:
> > > Li Shaohua <shaohua.li@intel.com> wrote:
> > > >
> > > > The patches are against 2.6.11-rc1 with Zwane's CPU hotplug patch in -mm
> > > >  tree.
> > > 
> > > Should I merge that thing into mainline?  It seems that a few people are
> > > needing it.
> > I'd like to listen to some comments first. There are still some things
> > I'm not sure, such as the do_exit_idle.
> > 
> 
> I was referring to Zwane's i386-cpu-hotplug-updated-for-mm.patch
Yep, great. Pavel's swsusp also need it.

Thanks,
Shaohua

Re: [ACPI] Re: [RFC 0/6] S3 SMP support with physcial CPU hotplug

[Nigel Cunningham]

Hi.

I'm switching suspend2 to use hotplug too. Li, I'll try adding your
patches as well as Zwane's if you like (suspend2 can enter S3, S4 or S5
after writing the image). I'd love to try it on my HT desktop, and
hotplug will get more testing too :>

Nigel

On Mon, 2005-04-04 at 12:49, Li Shaohua wrote:
> On Mon, 2005-04-04 at 10:48, Andrew Morton wrote:
> > Li Shaohua <shaohua.li@intel.com> wrote:
> > >
> > > On Mon, 2005-04-04 at 10:37, Andrew Morton wrote:
> > > > Li Shaohua <shaohua.li@intel.com> wrote:
> > > > >
> > > > > The patches are against 2.6.11-rc1 with Zwane's CPU hotplug patch in -mm
> > > > >  tree.
> > > > 
> > > > Should I merge that thing into mainline?  It seems that a few people are
> > > > needing it.
> > > I'd like to listen to some comments first. There are still some things
> > > I'm not sure, such as the do_exit_idle.
> > > 
> > 
> > I was referring to Zwane's i386-cpu-hotplug-updated-for-mm.patch
> Yep, great. Pavel's swsusp also need it.
> 
> Thanks,
> Shaohua
> 
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Acpi-devel mailing list
> Acpi-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/acpi-devel
-- 
Nigel Cunningham
Software Engineer, Canberra, Australia
http://www.cyclades.com
Bus: +61 (2) 6291 9554; Hme: +61 (2) 6292 8028;  Mob: +61 (417) 100 574

Maintainer of Suspend2 Kernel Patches http://suspend2.net

Re: [ACPI] Re: [RFC 0/6] S3 SMP support with physcial CPU hotplug

[Li Shaohua]

On Mon, 2005-04-04 at 16:01, Nigel Cunningham wrote:
> Hi.
> 
> I'm switching suspend2 to use hotplug too. Li, I'll try adding your
> patches as well as Zwane's if you like 
Great!

> (suspend2 can enter S3, S4 or S5
> after writing the image). I'd love to try it on my HT desktop, and
> hotplug will get more testing too :>
Unfortunately, my patches break Pavel's swsusp SMP, as my patches break
current 'cpu_up' mechanism. S4 doesn't require to boot AP CPUs from real
mode.

Thanks,
Shaohua

Re: [RFC 0/6] S3 SMP support with physcial CPU hotplug

[Pavel Machek]

Hi!

> >
> > The patches are against 2.6.11-rc1 with Zwane's CPU hotplug patch in -mm
> >  tree.
> 
> Should I merge that thing into mainline?  It seems that a few people are
> needing it.

Yes, it would be great. I have patch that cleans up smp/swsusp to
depend on Zwane's patch, too. Its ready AFAIK, but I'd prefer to wait
after 2.6.12 with its merge.
								Pavel

-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: [ACPI] Re: [RFC 0/6] S3 SMP support with physcial CPU hotplug

[Pavel Machek]

Hi!

> > I'm switching suspend2 to use hotplug too. Li, I'll try adding your
> > patches as well as Zwane's if you like 
> Great!
> 
> > (suspend2 can enter S3, S4 or S5
> > after writing the image). I'd love to try it on my HT desktop, and
> > hotplug will get more testing too :>
> Unfortunately, my patches break Pavel's swsusp SMP, as my patches break
> current 'cpu_up' mechanism. S4 doesn't require to boot AP CPUs from real
> mode.

Uh, I don't like that one. Is it possible to put secondary CPUs back
to the real mode so that cpu_up mechanism can handle them?

								Pavel
-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: [ACPI] Re: [RFC 0/6] S3 SMP support with physcial CPU hotplug

[Li Shaohua]

On Mon, 2005-04-04 at 17:10, Pavel Machek wrote:
> Hi!
> 
> > > I'm switching suspend2 to use hotplug too. Li, I'll try adding your
> > > patches as well as Zwane's if you like 
> > Great!
> > 
> > > (suspend2 can enter S3, S4 or S5
> > > after writing the image). I'd love to try it on my HT desktop, and
> > > hotplug will get more testing too :>
> > Unfortunately, my patches break Pavel's swsusp SMP, as my patches break
> > current 'cpu_up' mechanism. S4 doesn't require to boot AP CPUs from real
> > mode.
> 
> Uh, I don't like that one. Is it possible to put secondary CPUs back
> to the real mode 
Possibly doesn't need the trouble. Send a SIPI also can wakeup the a CPU
in protected mode.

> so that cpu_up mechanism can handle them?
If S4 also calls a smp_prepare_cpu, then the patches don't break S4. If
people don't complain warm boot a CPU is slow, I'd like S4 also use
smp_prepare_cpu.

Thanks,
Shaohua

Re: [ACPI] Re: [RFC 0/6] S3 SMP support with physcial CPU hotplug

[Pavel Machek]

Hi!

> > > > I'm switching suspend2 to use hotplug too. Li, I'll try adding your
> > > > patches as well as Zwane's if you like 
> > > Great!
> > > 
> > > > (suspend2 can enter S3, S4 or S5
> > > > after writing the image). I'd love to try it on my HT desktop, and
> > > > hotplug will get more testing too :>
> > > Unfortunately, my patches break Pavel's swsusp SMP, as my patches break
> > > current 'cpu_up' mechanism. S4 doesn't require to boot AP CPUs from real
> > > mode.
> > 
> > Uh, I don't like that one. Is it possible to put secondary CPUs back
> > to the real mode 
> Possibly doesn't need the trouble. Send a SIPI also can wakeup the a CPU
> in protected mode.
> 
> > so that cpu_up mechanism can handle them?
> If S4 also calls a smp_prepare_cpu, then the patches don't break S4. If
> people don't complain warm boot a CPU is slow, I'd like S4 also use
> smp_prepare_cpu.

Okay, that seems like a solution. Slowness should not really be a
problem (unless it takes > few seconds)...
								Pavel

-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: [ACPI] Re: [RFC 0/6] S3 SMP support with physcial CPU hotplug

[Nigel Cunningham]

Hi.

On Mon, 2005-04-04 at 12:37, Andrew Morton wrote:
> Li Shaohua <shaohua.li@intel.com> wrote:
> >
> > The patches are against 2.6.11-rc1 with Zwane's CPU hotplug patch in -mm
> >  tree.
> 
> Should I merge that thing into mainline?  It seems that a few people are
> needing it.

Perhaps we should address the MTRR issue first.

I've had code in Suspend2 for quite a while (6 months+) that removes the
sysdev support for MTRRs and saves and restores them with CPU context,
thereby avoiding the smp_call-while-interrupts-disabled issue. Perhaps
it would be helpful here?

Nigel
-- 
Nigel Cunningham
Software Engineer, Canberra, Australia
http://www.cyclades.com
Bus: +61 (2) 6291 9554; Hme: +61 (2) 6292 8028;  Mob: +61 (417) 100 574

Maintainer of Suspend2 Kernel Patches http://suspend2.net

Re: [ACPI] Re: [RFC 0/6] S3 SMP support with physcial CPU hotplug

[Nigel Cunningham]

Hi Li.

On Mon, 2005-04-04 at 19:24, Li Shaohua wrote:
> > so that cpu_up mechanism can handle them?
> If S4 also calls a smp_prepare_cpu, then the patches don't break S4. If
> people don't complain warm boot a CPU is slow, I'd like S4 also use
> smp_prepare_cpu.

So you have some more changes? Can I get a monolithic patch to try, when
you're ready?

Thanks!

Nigel
-- 
Nigel Cunningham
Software Engineer, Canberra, Australia
http://www.cyclades.com
Bus: +61 (2) 6291 9554; Hme: +61 (2) 6292 8028;  Mob: +61 (417) 100 574

Maintainer of Suspend2 Kernel Patches http://suspend2.net

Re: [ACPI] Re: [RFC 0/6] S3 SMP support with physcial CPU hotplug

[Pavel Machek]

Hi!

> > > The patches are against 2.6.11-rc1 with Zwane's CPU hotplug patch in -mm
> > >  tree.
> > 
> > Should I merge that thing into mainline?  It seems that a few people are
> > needing it.
> 
> Perhaps we should address the MTRR issue first.
> 
> I've had code in Suspend2 for quite a while (6 months+) that removes the
> sysdev support for MTRRs and saves and restores them with CPU context,
> thereby avoiding the smp_call-while-interrupts-disabled issue. Perhaps
> it would be helpful here?

This seems like separate issue... I'd prefer not to block this patch.

MTRRs should be probably handled by some kind of "cpu is going down"
and "cpu is going up" callbacks... Zwane, do you have any ideas?
Linklist of handlers should be enough...

								Pavel
-- 
Boycott Kodak -- for their patent abuse against Java.

Re: [ACPI] Re: [RFC 0/6] S3 SMP support with physcial CPU hotplug

[Nigel Cunningham]

Hi again.

On Mon, 2005-04-04 at 21:31, Pavel Machek wrote:
> Hi!
> 
> > > > The patches are against 2.6.11-rc1 with Zwane's CPU hotplug patch in -mm
> > > >  tree.
> > > 
> > > Should I merge that thing into mainline?  It seems that a few people are
> > > needing it.
> > 
> > Perhaps we should address the MTRR issue first.
> > 
> > I've had code in Suspend2 for quite a while (6 months+) that removes the
> > sysdev support for MTRRs and saves and restores them with CPU context,
> > thereby avoiding the smp_call-while-interrupts-disabled issue. Perhaps
> > it would be helpful here?
> 
> This seems like separate issue... I'd prefer not to block this patch.
> 
> MTRRs should be probably handled by some kind of "cpu is going down"
> and "cpu is going up" callbacks... Zwane, do you have any ideas?
> Linklist of handlers should be enough...

What else is it needed for? CpuFreq?

Nigel
-- 
Nigel Cunningham
Software Engineer, Canberra, Australia
http://www.cyclades.com
Bus: +61 (2) 6291 9554; Hme: +61 (2) 6292 8028;  Mob: +61 (417) 100 574

Maintainer of Suspend2 Kernel Patches http://suspend2.net

Re: [ACPI] Re: [RFC 0/6] S3 SMP support with physcial CPU hotplug

[Zwane Mwaikambo]

Hi Pavel!

On Mon, 4 Apr 2005, Pavel Machek wrote:

> > > > The patches are against 2.6.11-rc1 with Zwane's CPU hotplug patch in -mm
> > > >  tree.
> > > 
> > > Should I merge that thing into mainline?  It seems that a few people are
> > > needing it.
> > 
> > Perhaps we should address the MTRR issue first.
> > 
> > I've had code in Suspend2 for quite a while (6 months+) that removes the
> > sysdev support for MTRRs and saves and restores them with CPU context,
> > thereby avoiding the smp_call-while-interrupts-disabled issue. Perhaps
> > it would be helpful here?
> 
> This seems like separate issue... I'd prefer not to block this patch.
> 
> MTRRs should be probably handled by some kind of "cpu is going down"
> and "cpu is going up" callbacks... Zwane, do you have any ideas?
> Linklist of handlers should be enough...

Hmm, it seems to me that we if we're bringing up additional processors we 
only need to update the processor coming online (using mtrrs from an 
online processor) and hold a lock stopping any other updates. So for that, 
we definitely need online and offline callbacks.

Thanks,
	Zwane