Difference between logical and physical cpu hotplug

Difference between logical and physical cpu hotplug

[Vaibhav Jain]

Hi,

I need to know the difference between making a cpu logically offline
and physically offline. I have read that the following command

$ echo 0  > /sys/device/system/cpu/<cpu number>/online

makes a cpu logically offline. It frees the cpu from interrupts and migrates
running processes.
But then what does it mean to make a cpu physically offline ?
I referred to following article :

http://www.kernel.org/doc/Documentation/cpu-hotplug.txt

Here is the relevant excerpt :

Q: Does hot-add/hot-remove refer to physical add/remove of cpus?
A: The usage of hot-add/remove may not be very consistently used in the code.
CONFIG_HOTPLUG_CPU enables logical online/offline capability in the kernel.
To support physical addition/removal, one would need some BIOS hooks and
the platform should have something like an attention button in PCI hotplug.
CONFIG_ACPI_HOTPLUG_CPU enables ACPI support for physical add/remove of CPUs.



Please help!

Thanks
Vaibhav Jain
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110816/829cf907/attachment.html 

Difference between logical and physical cpu hotplug

[Vladimir Murzin]

On Wed, Aug 17, 2011 at 9:14 AM, Vaibhav Jain <vjoss197@gmail.com> wrote:
> Hi,
>
> I need to know the difference between making a cpu logically offline
> and physically offline. I have read that the following command
>
> $ echo 0? > /sys/device/system/cpu/<cpu number>/online
>
> makes a cpu logically offline. It frees the cpu from interrupts and migrates
> running processes.
> But then what does it mean to make a cpu physically offline ?
> I referred to following article :
>
> http://www.kernel.org/doc/Documentation/cpu-hotplug.txt
>
> Here is the relevant excerpt :
>
> Q: Does hot-add/hot-remove refer to physical add/remove of cpus?
> A: The usage of hot-add/remove may not be very consistently used in the
> code.
> CONFIG_HOTPLUG_CPU enables logical online/offline capability in the kernel.
> To support physical addition/removal, one would need some BIOS hooks and
> the platform should have something like an attention button in PCI hotplug.
> CONFIG_ACPI_HOTPLUG_CPU enables ACPI support for physical add/remove of
> CPUs.
>
> Please help!
>
> Thanks
> Vaibhav Jain
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>
Hi Vaibhav!

I think physical CPU hot-plug is a capability to plug in and out
processing device (CPU) on running machine. Logical CPU hot-plug is a
capability to add on and remove out already available processing units
(CPU s or cores) under SMP domain of OS.

P.S. Sorry for my weak English.

Difference between logical and physical cpu hotplug

[Vaibhav Jain]

Thanks for the reply!
I am looking for a little more detail.

Thanks


On Tue, Aug 16, 2011 at 10:26 PM, Vladimir Murzin <murzin.v@gmail.com>wrote:

> On Wed, Aug 17, 2011 at 9:14 AM, Vaibhav Jain <vjoss197@gmail.com> wrote:
> > Hi,
> >
> > I need to know the difference between making a cpu logically offline
> > and physically offline. I have read that the following command
> >
> > $ echo 0  > /sys/device/system/cpu/<cpu number>/online
> >
> > makes a cpu logically offline. It frees the cpu from interrupts and
> migrates
> > running processes.
> > But then what does it mean to make a cpu physically offline ?
> > I referred to following article :
> >
> > http://www.kernel.org/doc/Documentation/cpu-hotplug.txt
> >
> > Here is the relevant excerpt :
> >
> > Q: Does hot-add/hot-remove refer to physical add/remove of cpus?
> > A: The usage of hot-add/remove may not be very consistently used in the
> > code.
> > CONFIG_HOTPLUG_CPU enables logical online/offline capability in the
> kernel.
> > To support physical addition/removal, one would need some BIOS hooks and
> > the platform should have something like an attention button in PCI
> hotplug.
> > CONFIG_ACPI_HOTPLUG_CPU enables ACPI support for physical add/remove of
> > CPUs.
> >
> > Please help!
> >
> > Thanks
> > Vaibhav Jain
> >
> > _______________________________________________
> > Kernelnewbies mailing list
> > Kernelnewbies at kernelnewbies.org
> > http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> >
> >
> Hi Vaibhav!
>
> I think physical CPU hot-plug is a capability to plug in and out
> processing device (CPU) on running machine. Logical CPU hot-plug is a
> capability to add on and remove out already available processing units
> (CPU s or cores) under SMP domain of OS.
>
> P.S. Sorry for my weak English.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110816/aefd4eeb/attachment.html 

Difference between logical and physical cpu hotplug

[Mulyadi Santosa]

Hi....

On Wed, Aug 17, 2011 at 12:14, Vaibhav Jain <vjoss197@gmail.com> wrote:
> Hi,
>
> I need to know the difference between making a cpu logically offline
> and physically offline. I have read that the following command
>
> $ echo 0? > /sys/device/system/cpu/<cpu number>/online
>
> makes a cpu logically offline. It frees the cpu from interrupts and migrates
> running processes.
> But then what does it mean to make a cpu physically offline ?
> I referred to following article :
>
> http://www.kernel.org/doc/Documentation/cpu-hotplug.txt
>
> Here is the relevant excerpt :
>
> Q: Does hot-add/hot-remove refer to physical add/remove of cpus?
> A: The usage of hot-add/remove may not be very consistently used in the
> code.
> CONFIG_HOTPLUG_CPU enables logical online/offline capability in the kernel.
> To support physical addition/removal, one would need some BIOS hooks and
> the platform should have something like an attention button in PCI hotplug.
> CONFIG_ACPI_HOTPLUG_CPU enables ACPI support for physical add/remove of
> CPUs.

IMHO, logical cpu online capability is a foundation or baseline for
going into physical cpu offline/online. It's like disk hot swap in
RAID software. Since we can "remove" a RAID member, then removing them
is not a problem.

Spesificly about CPU, i think the clue here is "BIOS hook". Meaning?
We do cpu unplug by ... maybe pressing a button or engaging a
menu...and then, motherboard sends notification to the Linux kernel
and Linux kernel does whatever necessary (process migration, turning
off LAPIC etc) before CPU is safe to be removed.

What do you think?
-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com

Difference between logical and physical cpu hotplug

[Vaibhav Jain]

Hi,

I talked to a friend of mine and he suggested that
in a logical offline state the cpu is powered on and ready to execute
instructions
just that the kernel is not aware of it. But in case of physical offline
state the cpu
is powered off and cannot run.
Are you saying something similar ?


On Wed, Aug 17, 2011 at 9:32 AM, Mulyadi Santosa
<mulyadi.santosa@gmail.com>wrote:

> Hi....
>
> On Wed, Aug 17, 2011 at 12:14, Vaibhav Jain <vjoss197@gmail.com> wrote:
> > Hi,
> >
> > I need to know the difference between making a cpu logically offline
> > and physically offline. I have read that the following command
> >
> > $ echo 0  > /sys/device/system/cpu/<cpu number>/online
> >
> > makes a cpu logically offline. It frees the cpu from interrupts and
> migrates
> > running processes.
> > But then what does it mean to make a cpu physically offline ?
> > I referred to following article :
> >
> > http://www.kernel.org/doc/Documentation/cpu-hotplug.txt
> >
> > Here is the relevant excerpt :
> >
> > Q: Does hot-add/hot-remove refer to physical add/remove of cpus?
> > A: The usage of hot-add/remove may not be very consistently used in the
> > code.
> > CONFIG_HOTPLUG_CPU enables logical online/offline capability in the
> kernel.
> > To support physical addition/removal, one would need some BIOS hooks and
> > the platform should have something like an attention button in PCI
> hotplug.
> > CONFIG_ACPI_HOTPLUG_CPU enables ACPI support for physical add/remove of
> > CPUs.
>
> IMHO, logical cpu online capability is a foundation or baseline for
> going into physical cpu offline/online. It's like disk hot swap in
> RAID software. Since we can "remove" a RAID member, then removing them
> is not a problem.
>
> Spesificly about CPU, i think the clue here is "BIOS hook". Meaning?
> We do cpu unplug by ... maybe pressing a button or engaging a
> menu...and then, motherboard sends notification to the Linux kernel
> and Linux kernel does whatever necessary (process migration, turning
> off LAPIC etc) before CPU is safe to be removed.
>
> What do you think?
> --
> regards,
>
> Mulyadi Santosa
> Freelance Linux trainer and consultant
>
> blog: the-hydra.blogspot.com
> training: mulyaditraining.blogspot.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110818/fa949012/attachment-0001.html 

Difference between logical and physical cpu hotplug

[srivatsa bhat]

Hi Vaibhav,

On Thu, Aug 18, 2011 at 8:24 PM, Vaibhav Jain <vjoss197@gmail.com> wrote:

> Hi,
>
> I talked to a friend of mine and he suggested that
> in a logical offline state the cpu is powered on and ready to execute
> instructions
> just that the kernel is not aware of it. But in case of physical offline
> state the cpu
> is powered off and cannot run.
> Are you saying something similar ?
>
> Yes, you are right, mostly.
When you try to logically offline a CPU, the kernel will do task migration
(i.e., move out all the tasks running on that CPU to other CPUs in the
system) and it ensures that it doesn't need that CPU anymore. This also
means that, from now on, the context of that CPU need not be saved (because
the kernel has moved that CPU's tasks elsewhere). At this point, it is as if
the kernel is purposefully using only a subset of the available CPUs. This
step is a necessary prerequisite to do physical CPU offline later on.

But I don't think CPU power ON or OFF is the differentiating factor between
logical and physical offlining. In logical offline, you still have the CPUs
in the system but you just tell the kernel not to use them. At this stage,
you can power off your CPU, to save power for example.
But in physical offline, from a software perspective, you do additional work
at the firmware level (apart from logical offlining at the OS level), to
ensure that physically plugging out the CPUs will not affect the running
system in any way.

Please note that you can logically online and offline the same CPUs over and
over again without rebooting the system. Here, while onlining a CPU which
was offlined previously, the kernel follows almost the same sequence which
it normally follows while booting the CPUs during full system booting.

Also one more thing to be noted is that, to be able to physically hot-plug
CPUs, apart from OS and firmware support, you also need the hardware to
support this feature. That is, the electrical wiring to the individual CPUs
should be such that plugging them in and out does not interfere with the
functioning of the rest of the system. As of today, there are only a few
systems that support physical CPU-hotplug. But you can do logical CPU
hotplug easily, by configuring the kernel appropriately during compilation,
as you have noted in one of your previous mails.

Regards,
Srivatsa S. Bhat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110818/7f84a2af/attachment.html 

Difference between logical and physical cpu hotplug

[Mulyadi Santosa]

Hi

On Thu, Aug 18, 2011 at 21:54, Vaibhav Jain <vjoss197@gmail.com> wrote:
> Hi,
>
> I talked to a friend of mine and he suggested that
> in a logical offline state the cpu is powered on and ready to execute
> instructions
> just that the kernel is not aware of it. But in case of physical offline
> state the cpu
> is powered off and cannot run.
> Are you saying something similar ?


Please don't do top posting :)

More or less, yes....but I'll let others to better explain it. So far
I never did offline/online CPU unplug, so I have no first hand
experience.

-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com

Difference between logical and physical cpu hotplug

[Vaibhav Jain]

On Thu, Aug 18, 2011 at 9:02 AM, srivatsa bhat <bhat.srivatsa@gmail.com>wrote:

> Hi Vaibhav,
>
> On Thu, Aug 18, 2011 at 8:24 PM, Vaibhav Jain <vjoss197@gmail.com> wrote:
>
>> Hi,
>>
>> I talked to a friend of mine and he suggested that
>> in a logical offline state the cpu is powered on and ready to execute
>> instructions
>> just that the kernel is not aware of it. But in case of physical offline
>> state the cpu
>> is powered off and cannot run.
>> Are you saying something similar ?
>>
>> Yes, you are right, mostly.
> When you try to logically offline a CPU, the kernel will do task migration
> (i.e., move out all the tasks running on that CPU to other CPUs in the
> system) and it ensures that it doesn't need that CPU anymore. This also
> means that, from now on, the context of that CPU need not be saved (because
> the kernel has moved that CPU's tasks elsewhere). At this point, it is as if
> the kernel is purposefully using only a subset of the available CPUs. This
> step is a necessary prerequisite to do physical CPU offline later on.
>
> But I don't think CPU power ON or OFF is the differentiating factor between
> logical and physical offlining. In logical offline, you still have the CPUs
> in the system but you just tell the kernel not to use them. At this stage,
> you can power off your CPU, to save power for example.
> But in physical offline, from a software perspective, you do additional
> work at the firmware level (apart from logical offlining at the OS level),
> to ensure that physically plugging out the CPUs will not affect the running
> system in any way.
>
> Please note that you can logically online and offline the same CPUs over
> and over again without rebooting the system. Here, while onlining a CPU
> which was offlined previously, the kernel follows almost the same sequence
> which it normally follows while booting the CPUs during full system booting.
>
> Also one more thing to be noted is that, to be able to physically hot-plug
> CPUs, apart from OS and firmware support, you also need the hardware to
> support this feature. That is, the electrical wiring to the individual CPUs
> should be such that plugging them in and out does not interfere with the
> functioning of the rest of the system. As of today, there are only a few
> systems that support physical CPU-hotplug. But you can do logical CPU
> hotplug easily, by configuring the kernel appropriately during compilation,
> as you have noted in one of your previous mails.
>
> Regards,
> Srivatsa S. Bhat
>


Hi Srivatsa,

That was great explanation! Thanks!
I have just one more query. You mentioned above that " the kernel follows
almost the same *sequence *which it normally follows while booting the CPUs
during full system booting."

Can you please explain this sequence a little ?

Thanks
Vaibhav Jain
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110818/4dd1c1e3/attachment-0001.html 

Difference between logical and physical cpu hotplug

[Srivatsa Bhat]

On Thu, Aug 18, 2011 at 10:44 PM, Vaibhav Jain <vjoss197@gmail.com> wrote:

>
>
> On Thu, Aug 18, 2011 at 9:02 AM, srivatsa bhat <bhat.srivatsa@gmail.com>wrote:
>
>> Hi Vaibhav,
>>
>> On Thu, Aug 18, 2011 at 8:24 PM, Vaibhav Jain <vjoss197@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I talked to a friend of mine and he suggested that
>>> in a logical offline state the cpu is powered on and ready to execute
>>> instructions
>>> just that the kernel is not aware of it. But in case of physical offline
>>> state the cpu
>>> is powered off and cannot run.
>>> Are you saying something similar ?
>>>
>>> Yes, you are right, mostly.
>> When you try to logically offline a CPU, the kernel will do task migration
>> (i.e., move out all the tasks running on that CPU to other CPUs in the
>> system) and it ensures that it doesn't need that CPU anymore. This also
>> means that, from now on, the context of that CPU need not be saved (because
>> the kernel has moved that CPU's tasks elsewhere). At this point, it is as if
>> the kernel is purposefully using only a subset of the available CPUs. This
>> step is a necessary prerequisite to do physical CPU offline later on.
>>
>> But I don't think CPU power ON or OFF is the differentiating factor
>> between logical and physical offlining. In logical offline, you still have
>> the CPUs in the system but you just tell the kernel not to use them. At this
>> stage, you can power off your CPU, to save power for example.
>> But in physical offline, from a software perspective, you do additional
>> work at the firmware level (apart from logical offlining at the OS level),
>> to ensure that physically plugging out the CPUs will not affect the running
>> system in any way.
>>
>> Please note that you can logically online and offline the same CPUs over
>> and over again without rebooting the system. Here, while onlining a CPU
>> which was offlined previously, the kernel follows almost the same sequence
>> which it normally follows while booting the CPUs during full system booting.
>>
>> Also one more thing to be noted is that, to be able to physically hot-plug
>> CPUs, apart from OS and firmware support, you also need the hardware to
>> support this feature. That is, the electrical wiring to the individual CPUs
>> should be such that plugging them in and out does not interfere with the
>> functioning of the rest of the system. As of today, there are only a few
>> systems that support physical CPU-hotplug. But you can do logical CPU
>> hotplug easily, by configuring the kernel appropriately during compilation,
>> as you have noted in one of your previous mails.
>>
>> Regards,
>> Srivatsa S. Bhat
>>
>
>
> Hi Srivatsa,
>
> That was great explanation! Thanks!
> I have just one more query. You mentioned above that " the kernel follows
> almost the same *sequence *which it normally follows while booting the
> CPUs during full system booting."
>
> Can you please explain this sequence a little ?
>
>
Hi Vaibhav,

I'll try to outline a very high level view of what happens while booting an
SMP (Symmetric Multi-Processor) system. Instead of going through the entire
boot sequence, let me just highlight only the part that is of interest in
this discussion: booting multiple CPUs.

The "boot processor" is the one which is booted first while booting a
system. On x86 architecture, CPU 0 is always the boot processor. Hence, if
you have observed, you cannot offline CPU0 using CPU hot-plugging on an x86
machine. (On an Intel box, the file /sys/devices/system/cpu/cpu0/online is
purposefully absent, for this reason!). But in other architectures, this
might not be the case. For example on POWER architecture, any processor in
the system can act as the boot processor.

Once the boot processor does its initialization, the other processors, known
as "secondary processors or application processors (APs)" are
booted/initialized. Here, obviously some synchronization mechanism is
necessary to ensure that this order is followed. So in Linux, we use 2
bitmasks called "cpu_callout_mask" and "cpu_callin_mask". These bitmasks are
used to indicate the processors available in the system.

Once the boot processor initializes itself, it updates cpu_callout_mask to
indicate which secondary processor (or application processor AP) can
initialize itself next (for example, the boot processor sets a particular
bit as 1 in the cpu_callout_mask). On the other hand, the secondary
processor would have done some very basic initialization till then and will
be testing the value of 'cpu_callout_mask' in a while loop to see if its
number has been "called out" by the boot processor. Only after the boot
processor "calls out" this AP, this AP will continue the rest of its
initialization and completes it.

Once the AP completes its initialization, it reports back to the boot
processor by setting its number in the cpu_callin_mask. As expected, the
boot processor would have been waiting in a while loop on cpu_callin_mask to
see if this AP booted OK or not. Once it finds that the cpu_callin_mask for
this AP has been set, the boot processor follows the same procedure to boot
other APs: i.e., it updates cpu_callout_mask and waits for the corresponding
entry to be set in cpu_callin_mask by that AP and so on. This process
continues until all the APs are booted up.

Of course, each of these "waiting" times (of both boot processor and APs)
are capped by some preset value, say for example 5 seconds. If some AP takes
more than that time to boot, the boot processor declares that the AP could
not boot and takes appropriate action (like clearing its bit in
cpu_callout_mask and logically removing that AP from its tables etc,
effectively forgetting about that processor). Similarly while the APs wait
for the boot processor to call them out, if the boot processor does not call
them within a given time period, they declare kernel panic.

Here are some references, if you are interested in more details:

Linux kernel source code:
1. linux/arch/x86/kernel/smpboot.c : start_secondary() and smp_callin()
    These are the functions executed by the APs (secondary or application
processors). Actually smp_callin() is called within start_secondary() which
is the primary function executed by APs.

2. linux/arch/x86/kernel/smpboot.c :


Regards,
Srivatsa S. Bhat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110818/0446e650/attachment.html 

Difference between logical and physical cpu hotplug

[Srivatsa Bhat]

On Thu, Aug 18, 2011 at 11:40 PM, Srivatsa Bhat <bhat.srivatsa@gmail.com>wrote:

>
>
> On Thu, Aug 18, 2011 at 10:44 PM, Vaibhav Jain <vjoss197@gmail.com> wrote:
>
>>
>>
>> On Thu, Aug 18, 2011 at 9:02 AM, srivatsa bhat <bhat.srivatsa@gmail.com>wrote:
>>
>>> Hi Vaibhav,
>>>
>>> On Thu, Aug 18, 2011 at 8:24 PM, Vaibhav Jain <vjoss197@gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> I talked to a friend of mine and he suggested that
>>>> in a logical offline state the cpu is powered on and ready to execute
>>>> instructions
>>>> just that the kernel is not aware of it. But in case of physical offline
>>>> state the cpu
>>>> is powered off and cannot run.
>>>> Are you saying something similar ?
>>>>
>>>> Yes, you are right, mostly.
>>> When you try to logically offline a CPU, the kernel will do task
>>> migration (i.e., move out all the tasks running on that CPU to other CPUs in
>>> the system) and it ensures that it doesn't need that CPU anymore. This also
>>> means that, from now on, the context of that CPU need not be saved (because
>>> the kernel has moved that CPU's tasks elsewhere). At this point, it is as if
>>> the kernel is purposefully using only a subset of the available CPUs. This
>>> step is a necessary prerequisite to do physical CPU offline later on.
>>>
>>> But I don't think CPU power ON or OFF is the differentiating factor
>>> between logical and physical offlining. In logical offline, you still have
>>> the CPUs in the system but you just tell the kernel not to use them. At this
>>> stage, you can power off your CPU, to save power for example.
>>> But in physical offline, from a software perspective, you do additional
>>> work at the firmware level (apart from logical offlining at the OS level),
>>> to ensure that physically plugging out the CPUs will not affect the running
>>> system in any way.
>>>
>>> Please note that you can logically online and offline the same CPUs over
>>> and over again without rebooting the system. Here, while onlining a CPU
>>> which was offlined previously, the kernel follows almost the same sequence
>>> which it normally follows while booting the CPUs during full system booting.
>>>
>>> Also one more thing to be noted is that, to be able to physically
>>> hot-plug CPUs, apart from OS and firmware support, you also need the
>>> hardware to support this feature. That is, the electrical wiring to the
>>> individual CPUs should be such that plugging them in and out does not
>>> interfere with the functioning of the rest of the system. As of today, there
>>> are only a few systems that support physical CPU-hotplug. But you can do
>>> logical CPU hotplug easily, by configuring the kernel appropriately during
>>> compilation, as you have noted in one of your previous mails.
>>>
>>> Regards,
>>> Srivatsa S. Bhat
>>>
>>
>>
>> Hi Srivatsa,
>>
>> That was great explanation! Thanks!
>> I have just one more query. You mentioned above that " the kernel follows
>> almost the same *sequence *which it normally follows while booting the
>> CPUs during full system booting."
>>
>> Can you please explain this sequence a little ?
>>
>>
> Hi Vaibhav,
>
> I'll try to outline a very high level view of what happens while booting an
> SMP (Symmetric Multi-Processor) system. Instead of going through the entire
> boot sequence, let me just highlight only the part that is of interest in
> this discussion: booting multiple CPUs.
>
> The "boot processor" is the one which is booted first while booting a
> system. On x86 architecture, CPU 0 is always the boot processor. Hence, if
> you have observed, you cannot offline CPU0 using CPU hot-plugging on an x86
> machine. (On an Intel box, the file /sys/devices/system/cpu/cpu0/online is
> purposefully absent, for this reason!). But in other architectures, this
> might not be the case. For example on POWER architecture, any processor in
> the system can act as the boot processor.
>
> Once the boot processor does its initialization, the other processors,
> known as "secondary processors or application processors (APs)" are
> booted/initialized. Here, obviously some synchronization mechanism is
> necessary to ensure that this order is followed. So in Linux, we use 2
> bitmasks called "cpu_callout_mask" and "cpu_callin_mask". These bitmasks are
> used to indicate the processors available in the system.
>
> Once the boot processor initializes itself, it updates cpu_callout_mask to
> indicate which secondary processor (or application processor AP) can
> initialize itself next (for example, the boot processor sets a particular
> bit as 1 in the cpu_callout_mask). On the other hand, the secondary
> processor would have done some very basic initialization till then and will
> be testing the value of 'cpu_callout_mask' in a while loop to see if its
> number has been "called out" by the boot processor. Only after the boot
> processor "calls out" this AP, this AP will continue the rest of its
> initialization and completes it.
>
> Once the AP completes its initialization, it reports back to the boot
> processor by setting its number in the cpu_callin_mask. As expected, the
> boot processor would have been waiting in a while loop on cpu_callin_mask to
> see if this AP booted OK or not. Once it finds that the cpu_callin_mask for
> this AP has been set, the boot processor follows the same procedure to boot
> other APs: i.e., it updates cpu_callout_mask and waits for the corresponding
> entry to be set in cpu_callin_mask by that AP and so on. This process
> continues until all the APs are booted up.
>
> Of course, each of these "waiting" times (of both boot processor and APs)
> are capped by some preset value, say for example 5 seconds. If some AP takes
> more than that time to boot, the boot processor declares that the AP could
> not boot and takes appropriate action (like clearing its bit in
> cpu_callout_mask and logically removing that AP from its tables etc,
> effectively forgetting about that processor). Similarly while the APs wait
> for the boot processor to call them out, if the boot processor does not call
> them within a given time period, they declare kernel panic.
>
> Here are some references, if you are interested in more details:
>
> Linux kernel source code:
> 1. linux/arch/x86/kernel/smpboot.c : start_secondary() and smp_callin()
>     These are the functions executed by the APs (secondary or application
> processors). Actually smp_callin() is called within start_secondary() which
> is the primary function executed by APs.
>
> 2. linux/arch/x86/kernel/smpboot.c :  do_boot_cpu()
>
       This is executed by the boot processor.  You can look up other
important functions such as native_cpu_up().

    General SMP booting info:
    1. http://www.cheesecake.org/sac/smp.html

[ Sorry, I accidentally sent the earlier mail before composing the text
fully. ]

Regards,
Srivatsa S. Bhat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110818/b9b60e3f/attachment-0001.html 

Difference between logical and physical cpu hotplug

[Vaibhav Jain]

On Thu, Aug 18, 2011 at 11:14 AM, Srivatsa Bhat <bhat.srivatsa@gmail.com>wrote:

>
>
>   On Thu, Aug 18, 2011 at 11:40 PM, Srivatsa Bhat <bhat.srivatsa@gmail.com
> > wrote:
>
>>
>>
>>    On Thu, Aug 18, 2011 at 10:44 PM, Vaibhav Jain <vjoss197@gmail.com>wrote:
>>
>>>
>>>
>>> On Thu, Aug 18, 2011 at 9:02 AM, srivatsa bhat <bhat.srivatsa@gmail.com>wrote:
>>>
>>>> Hi Vaibhav,
>>>>
>>>>  On Thu, Aug 18, 2011 at 8:24 PM, Vaibhav Jain <vjoss197@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I talked to a friend of mine and he suggested that
>>>>> in a logical offline state the cpu is powered on and ready to execute
>>>>> instructions
>>>>> just that the kernel is not aware of it. But in case of physical
>>>>> offline state the cpu
>>>>> is powered off and cannot run.
>>>>> Are you saying something similar ?
>>>>>
>>>>> Yes, you are right, mostly.
>>>> When you try to logically offline a CPU, the kernel will do task
>>>> migration (i.e., move out all the tasks running on that CPU to other CPUs in
>>>> the system) and it ensures that it doesn't need that CPU anymore. This also
>>>> means that, from now on, the context of that CPU need not be saved (because
>>>> the kernel has moved that CPU's tasks elsewhere). At this point, it is as if
>>>> the kernel is purposefully using only a subset of the available CPUs. This
>>>> step is a necessary prerequisite to do physical CPU offline later on.
>>>>
>>>> But I don't think CPU power ON or OFF is the differentiating factor
>>>> between logical and physical offlining. In logical offline, you still have
>>>> the CPUs in the system but you just tell the kernel not to use them. At this
>>>> stage, you can power off your CPU, to save power for example.
>>>> But in physical offline, from a software perspective, you do additional
>>>> work at the firmware level (apart from logical offlining at the OS level),
>>>> to ensure that physically plugging out the CPUs will not affect the running
>>>> system in any way.
>>>>
>>>> Please note that you can logically online and offline the same CPUs over
>>>> and over again without rebooting the system. Here, while onlining a CPU
>>>> which was offlined previously, the kernel follows almost the same sequence
>>>> which it normally follows while booting the CPUs during full system booting.
>>>>
>>>> Also one more thing to be noted is that, to be able to physically
>>>> hot-plug CPUs, apart from OS and firmware support, you also need the
>>>> hardware to support this feature. That is, the electrical wiring to the
>>>> individual CPUs should be such that plugging them in and out does not
>>>> interfere with the functioning of the rest of the system. As of today, there
>>>> are only a few systems that support physical CPU-hotplug. But you can do
>>>> logical CPU hotplug easily, by configuring the kernel appropriately during
>>>> compilation, as you have noted in one of your previous mails.
>>>>
>>>> Regards,
>>>> Srivatsa S. Bhat
>>>>
>>>
>>>
>>> Hi Srivatsa,
>>>
>>> That was great explanation! Thanks!
>>> I have just one more query. You mentioned above that " the kernel
>>> follows almost the same *sequence *which it normally follows while
>>> booting the CPUs during full system booting."
>>>
>>> Can you please explain this sequence a little ?
>>>
>>>
>> Hi Vaibhav,
>>
>> I'll try to outline a very high level view of what happens while booting
>> an SMP (Symmetric Multi-Processor) system. Instead of going through the
>> entire boot sequence, let me just highlight only the part that is of
>> interest in this discussion: booting multiple CPUs.
>>
>> The "boot processor" is the one which is booted first while booting a
>> system. On x86 architecture, CPU 0 is always the boot processor. Hence, if
>> you have observed, you cannot offline CPU0 using CPU hot-plugging on an x86
>> machine. (On an Intel box, the file /sys/devices/system/cpu/cpu0/online is
>> purposefully absent, for this reason!). But in other architectures, this
>> might not be the case. For example on POWER architecture, any processor in
>> the system can act as the boot processor.
>>
>> Once the boot processor does its initialization, the other processors,
>> known as "secondary processors or application processors (APs)" are
>> booted/initialized. Here, obviously some synchronization mechanism is
>> necessary to ensure that this order is followed. So in Linux, we use 2
>> bitmasks called "cpu_callout_mask" and "cpu_callin_mask". These bitmasks are
>> used to indicate the processors available in the system.
>>
>> Once the boot processor initializes itself, it updates cpu_callout_mask to
>> indicate which secondary processor (or application processor AP) can
>> initialize itself next (for example, the boot processor sets a particular
>> bit as 1 in the cpu_callout_mask). On the other hand, the secondary
>> processor would have done some very basic initialization till then and will
>> be testing the value of 'cpu_callout_mask' in a while loop to see if its
>> number has been "called out" by the boot processor. Only after the boot
>> processor "calls out" this AP, this AP will continue the rest of its
>> initialization and completes it.
>>
>> Once the AP completes its initialization, it reports back to the boot
>> processor by setting its number in the cpu_callin_mask. As expected, the
>> boot processor would have been waiting in a while loop on cpu_callin_mask to
>> see if this AP booted OK or not. Once it finds that the cpu_callin_mask for
>> this AP has been set, the boot processor follows the same procedure to boot
>> other APs: i.e., it updates cpu_callout_mask and waits for the corresponding
>> entry to be set in cpu_callin_mask by that AP and so on. This process
>> continues until all the APs are booted up.
>>
>> Of course, each of these "waiting" times (of both boot processor and APs)
>> are capped by some preset value, say for example 5 seconds. If some AP takes
>> more than that time to boot, the boot processor declares that the AP could
>> not boot and takes appropriate action (like clearing its bit in
>> cpu_callout_mask and logically removing that AP from its tables etc,
>> effectively forgetting about that processor). Similarly while the APs wait
>> for the boot processor to call them out, if the boot processor does not call
>> them within a given time period, they declare kernel panic.
>>
>> Here are some references, if you are interested in more details:
>>
>> Linux kernel source code:
>> 1. linux/arch/x86/kernel/smpboot.c : start_secondary() and smp_callin()
>>     These are the functions executed by the APs (secondary or application
>> processors). Actually smp_callin() is called within start_secondary() which
>> is the primary function executed by APs.
>>
>> 2. linux/arch/x86/kernel/smpboot.c :  do_boot_cpu()
>>
>        This is executed by the boot processor.  You can look up other
> important functions such as native_cpu_up().
>
>     General SMP booting info:
>     1. http://www.cheesecake.org/sac/smp.html
>
> [ Sorry, I accidentally sent the earlier mail before composing the text
> fully. ]
>
> Regards,
> Srivatsa S. Bhat
>



 Awesome explanation Srivatsa!! Thanks a lot!!
Just had one more doubt. I am a little unclear about how the APs get
initialized in the beginning. In the case of Boot Processor
its just like a uniprocessor system. But how do the APs start executing code
?
Could you please explain a little ?

Thanx!!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110819/77e7bd6b/attachment.html 

Difference between logical and physical cpu hotplug

[Srivatsa Bhat]

On Sat, Aug 20, 2011 at 4:05 AM, Vaibhav Jain <vjoss197@gmail.com> wrote:

>
> On Thu, Aug 18, 2011 at 11:14 AM, Srivatsa Bhat <bhat.srivatsa@gmail.com>wrote:
>
>>
>>
>>   On Thu, Aug 18, 2011 at 11:40 PM, Srivatsa Bhat <
>> bhat.srivatsa at gmail.com> wrote:
>>
>>>
>>>
>>>    On Thu, Aug 18, 2011 at 10:44 PM, Vaibhav Jain <vjoss197@gmail.com>wrote:
>>>
>>>>
>>>>
>>>> On Thu, Aug 18, 2011 at 9:02 AM, srivatsa bhat <bhat.srivatsa@gmail.com
>>>> > wrote:
>>>>
>>>>> Hi Vaibhav,
>>>>>
>>>>>  On Thu, Aug 18, 2011 at 8:24 PM, Vaibhav Jain <vjoss197@gmail.com>wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I talked to a friend of mine and he suggested that
>>>>>> in a logical offline state the cpu is powered on and ready to execute
>>>>>> instructions
>>>>>> just that the kernel is not aware of it. But in case of physical
>>>>>> offline state the cpu
>>>>>> is powered off and cannot run.
>>>>>> Are you saying something similar ?
>>>>>>
>>>>>> Yes, you are right, mostly.
>>>>> When you try to logically offline a CPU, the kernel will do task
>>>>> migration (i.e., move out all the tasks running on that CPU to other CPUs in
>>>>> the system) and it ensures that it doesn't need that CPU anymore. This also
>>>>> means that, from now on, the context of that CPU need not be saved (because
>>>>> the kernel has moved that CPU's tasks elsewhere). At this point, it is as if
>>>>> the kernel is purposefully using only a subset of the available CPUs. This
>>>>> step is a necessary prerequisite to do physical CPU offline later on.
>>>>>
>>>>> But I don't think CPU power ON or OFF is the differentiating factor
>>>>> between logical and physical offlining. In logical offline, you still have
>>>>> the CPUs in the system but you just tell the kernel not to use them. At this
>>>>> stage, you can power off your CPU, to save power for example.
>>>>> But in physical offline, from a software perspective, you do additional
>>>>> work at the firmware level (apart from logical offlining at the OS level),
>>>>> to ensure that physically plugging out the CPUs will not affect the running
>>>>> system in any way.
>>>>>
>>>>> Please note that you can logically online and offline the same CPUs
>>>>> over and over again without rebooting the system. Here, while onlining a CPU
>>>>> which was offlined previously, the kernel follows almost the same sequence
>>>>> which it normally follows while booting the CPUs during full system booting.
>>>>>
>>>>> Also one more thing to be noted is that, to be able to physically
>>>>> hot-plug CPUs, apart from OS and firmware support, you also need the
>>>>> hardware to support this feature. That is, the electrical wiring to the
>>>>> individual CPUs should be such that plugging them in and out does not
>>>>> interfere with the functioning of the rest of the system. As of today, there
>>>>> are only a few systems that support physical CPU-hotplug. But you can do
>>>>> logical CPU hotplug easily, by configuring the kernel appropriately during
>>>>> compilation, as you have noted in one of your previous mails.
>>>>>
>>>>> Regards,
>>>>> Srivatsa S. Bhat
>>>>>
>>>>
>>>>
>>>> Hi Srivatsa,
>>>>
>>>> That was great explanation! Thanks!
>>>> I have just one more query. You mentioned above that " the kernel
>>>> follows almost the same *sequence *which it normally follows while
>>>> booting the CPUs during full system booting."
>>>>
>>>> Can you please explain this sequence a little ?
>>>>
>>>>
>>> Hi Vaibhav,
>>>
>>> I'll try to outline a very high level view of what happens while booting
>>> an SMP (Symmetric Multi-Processor) system. Instead of going through the
>>> entire boot sequence, let me just highlight only the part that is of
>>> interest in this discussion: booting multiple CPUs.
>>>
>>> The "boot processor" is the one which is booted first while booting a
>>> system. On x86 architecture, CPU 0 is always the boot processor. Hence, if
>>> you have observed, you cannot offline CPU0 using CPU hot-plugging on an x86
>>> machine. (On an Intel box, the file /sys/devices/system/cpu/cpu0/online is
>>> purposefully absent, for this reason!). But in other architectures, this
>>> might not be the case. For example on POWER architecture, any processor in
>>> the system can act as the boot processor.
>>>
>>> Once the boot processor does its initialization, the other processors,
>>> known as "secondary processors or application processors (APs)" are
>>> booted/initialized. Here, obviously some synchronization mechanism is
>>> necessary to ensure that this order is followed. So in Linux, we use 2
>>> bitmasks called "cpu_callout_mask" and "cpu_callin_mask". These bitmasks are
>>> used to indicate the processors available in the system.
>>>
>>> Once the boot processor initializes itself, it updates cpu_callout_mask
>>> to indicate which secondary processor (or application processor AP) can
>>> initialize itself next (for example, the boot processor sets a particular
>>> bit as 1 in the cpu_callout_mask). On the other hand, the secondary
>>> processor would have done some very basic initialization till then and will
>>> be testing the value of 'cpu_callout_mask' in a while loop to see if its
>>> number has been "called out" by the boot processor. Only after the boot
>>> processor "calls out" this AP, this AP will continue the rest of its
>>> initialization and completes it.
>>>
>>> Once the AP completes its initialization, it reports back to the boot
>>> processor by setting its number in the cpu_callin_mask. As expected, the
>>> boot processor would have been waiting in a while loop on cpu_callin_mask to
>>> see if this AP booted OK or not. Once it finds that the cpu_callin_mask for
>>> this AP has been set, the boot processor follows the same procedure to boot
>>> other APs: i.e., it updates cpu_callout_mask and waits for the corresponding
>>> entry to be set in cpu_callin_mask by that AP and so on. This process
>>> continues until all the APs are booted up.
>>>
>>> Of course, each of these "waiting" times (of both boot processor and APs)
>>> are capped by some preset value, say for example 5 seconds. If some AP takes
>>> more than that time to boot, the boot processor declares that the AP could
>>> not boot and takes appropriate action (like clearing its bit in
>>> cpu_callout_mask and logically removing that AP from its tables etc,
>>> effectively forgetting about that processor). Similarly while the APs wait
>>> for the boot processor to call them out, if the boot processor does not call
>>> them within a given time period, they declare kernel panic.
>>>
>>> Here are some references, if you are interested in more details:
>>>
>>> Linux kernel source code:
>>> 1. linux/arch/x86/kernel/smpboot.c : start_secondary() and smp_callin()
>>>     These are the functions executed by the APs (secondary or application
>>> processors). Actually smp_callin() is called within start_secondary() which
>>> is the primary function executed by APs.
>>>
>>> 2. linux/arch/x86/kernel/smpboot.c :  do_boot_cpu()
>>>
>>        This is executed by the boot processor.  You can look up other
>> important functions such as native_cpu_up().
>>
>>     General SMP booting info:
>>     1. http://www.cheesecake.org/sac/smp.html
>>
>> [ Sorry, I accidentally sent the earlier mail before composing the text
>> fully. ]
>>
>> Regards,
>> Srivatsa S. Bhat
>>
>
>
>
>  Awesome explanation Srivatsa!! Thanks a lot!!
> Just had one more doubt. I am a little unclear about how the APs get
> initialized in the beginning. In the case of Boot Processor
> its just like a uniprocessor system. But how do the APs start executing
> code ?
> Could you please explain a little ?
>
>
Sure. But please note that I will stick to Intel architecture while
explaining the details.

The Boot CPU or the Boot-Strap Processor (BSP) is the one which boots the
Operating System. Then it wakes up the APs (Application Processors) when it
is the right time.

Let us now explore some background details to understand how all this works.

On uniprocessor systems we use PIC (Programmable Interrupt Controller) like
the 8259A Interrupt Controller chip to deliver interrupts to the processor.
On Multi-Processor (MP) systems, we use something known as APICs (Advanced
Programmable Interrupt Controllers). Every processor has a local APIC.
And there are one or more I/O APICs in the system that are shared by all the
processors. As the name suggests, I/O APICs are used to deliver interrupts
from I/O devices to the processors, via the local APICs.

All local APICs have unique IDs that are assigned either by the hardware or
the BIOS during the initialization phase. Using the local APIC ID we can
identify the processors in the system.

Using these local APICs, we can send something known as "Inter-Processor
Interrupts" or IPIs. As the name suggests, this is a mechanism for one
processor to interrupt another processor in the system. Note that this
mechanism can be used by any processor to talk to any other processor in
the system (no distinction between BSP and APs here).

To kick-start the APs, the BSP sends INIT IPI to each AP in turn, waits for
some time for the IPI to be delivered to the AP and then checks if that AP
booted up. Based on the version of the APIC used, the BSP might have to send
2 STARTUP IPIs to the APs with some time delay after each of the IPIs.
[ If you have discrete APICs (i.e., 82489DX APIC) then INIT IPI will do. If
you have integrated APIC, you need to send two STARTUP IPIs. ]
All this is in accordance with the "Universal Start-up Algorithm" to start
APs, as specified by Intel architecture. These IPIs cause an INIT at the AP
to which it was delivered.

Now the question is, how do you make the APs to execute a particular piece
of code (i.e., jump to a specified location) on start-up?
We know that whenever a processor starts after a RESET or INIT, it starts
executing code from the reset vector (a predefined location).
However if you want a processor to immediately jump to an address that you
have specified, you must use the INIT IPI as part of a "warm-reset".
Warm-reset allows you to send INIT signal to a processor without causing the
processor to go through the entire BIOS initialization (POST -- see below
for details) and then start the processor's execution at the
warm-reset-vector.

By putting the appropriate pointer (i.e., pointer to the AP start-up code)
in the warm-reset-vector (system RAM location 40:67h), setting the BIOS
shutdown code to 0Ah (which tells the BIOS that this INIT is part of a
warm-reset) and then causing an INIT at the processor (via the IPIs), the
Operating System can cause the processor to jump immediately to any location
and start executing that code. This is how the BSP can boot the APs and make
them execute some particular piece of code (in this case, the AP start-up
code as designed in the OS).

It would be worthwhile to understand what would be the state of the system
(and the APs) before the Operating System gets control from the BIOS after
switching ON the machine. The BIOS, upon system start, performs a procedure
known as POST (Power-On Self Test). This is to check the status of all the
components/circuitry of the system, including the processors, to ensure that
they are all functioning properly. During this phase the BIOS initializes
all the circuitry (including all the APICs and the processors) to some known
configuration and then puts all the APs to the HALT state with interrupts
disabled. This is to ensure that the APs don't execute Operating System code
(we want only the BSP to execute the OS code initially). Then the BSP starts
executing OS code.

To boot APs, the BSP sends IPIs to them. But IPIs are non-maskable (note
that the APs were in HALT state, with interrupts disabled). Hence the BSP
will be able to kick-start AP execution and by using the warm-reset
mechanism, it can direct the APs to execute some particular piece of code at
startup. The BSP would have put a pointer to that AP start-up code in the
warm-reset-vector address before sending the INIT or STARTUP IPIs to the
APs.

You might be wondering how does the BSP specify to its local APIC as to
which AP it must send an IPI to..
The answer is simple. During BIOS POST, an MP (Multi-Processor)
Configuration Table will be set up (in conjunction with BSP and APs) in a
well-known region of memory, which will be read by the OS during boot up.
This table contains the local APIC IDs of all the processors.
So, while sending the targeted IPIs using its local APIC, the BSP specifies
the local APIC ID of the target AP which it wants to interrupt (and boot in
this case). This ensures the delivery of the IPI to the correct AP.

In short, this is how a Multi-Processor system gets rolling ... :-)

For more details you can refer:

1. Intel Multi-Processor Specification, especially Appendix A and B.
    http://www.intel.com/design/pentium/datashts/242016.htm

2. linux/arch/x86/kernel/smpboot.c :
    do_boot_cpu(), wakeup_secondary_cpu_via_init(), native_cpu_up(),
start_secondary()

3. linux/arch/x86/kernel/head_32.S:
    startup_32_smp()

4. linux/arch/x86/kernel/trampoline_32.S

5. http://tldp.org/HOWTO/Linux-i386-Boot-Code-HOWTO/smpboot.html

Regards,
Srivatsa S. Bhat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110822/f85cf19d/attachment-0001.html 

Difference between logical and physical cpu hotplug

[Vaibhav Jain]

On Sun, Aug 21, 2011 at 1:09 PM, Srivatsa Bhat <bhat.srivatsa@gmail.com>wrote:

>
>
> On Sat, Aug 20, 2011 at 4:05 AM, Vaibhav Jain <vjoss197@gmail.com> wrote:
>
>>
>> On Thu, Aug 18, 2011 at 11:14 AM, Srivatsa Bhat <bhat.srivatsa@gmail.com>wrote:
>>
>>>
>>>
>>>   On Thu, Aug 18, 2011 at 11:40 PM, Srivatsa Bhat <
>>> bhat.srivatsa at gmail.com> wrote:
>>>
>>>>
>>>>
>>>>    On Thu, Aug 18, 2011 at 10:44 PM, Vaibhav Jain <vjoss197@gmail.com>wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, Aug 18, 2011 at 9:02 AM, srivatsa bhat <
>>>>> bhat.srivatsa at gmail.com> wrote:
>>>>>
>>>>>> Hi Vaibhav,
>>>>>>
>>>>>>  On Thu, Aug 18, 2011 at 8:24 PM, Vaibhav Jain <vjoss197@gmail.com>wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I talked to a friend of mine and he suggested that
>>>>>>> in a logical offline state the cpu is powered on and ready to execute
>>>>>>> instructions
>>>>>>> just that the kernel is not aware of it. But in case of physical
>>>>>>> offline state the cpu
>>>>>>> is powered off and cannot run.
>>>>>>> Are you saying something similar ?
>>>>>>>
>>>>>>> Yes, you are right, mostly.
>>>>>> When you try to logically offline a CPU, the kernel will do task
>>>>>> migration (i.e., move out all the tasks running on that CPU to other CPUs in
>>>>>> the system) and it ensures that it doesn't need that CPU anymore. This also
>>>>>> means that, from now on, the context of that CPU need not be saved (because
>>>>>> the kernel has moved that CPU's tasks elsewhere). At this point, it is as if
>>>>>> the kernel is purposefully using only a subset of the available CPUs. This
>>>>>> step is a necessary prerequisite to do physical CPU offline later on.
>>>>>>
>>>>>> But I don't think CPU power ON or OFF is the differentiating factor
>>>>>> between logical and physical offlining. In logical offline, you still have
>>>>>> the CPUs in the system but you just tell the kernel not to use them. At this
>>>>>> stage, you can power off your CPU, to save power for example.
>>>>>> But in physical offline, from a software perspective, you do
>>>>>> additional work at the firmware level (apart from logical offlining at the
>>>>>> OS level), to ensure that physically plugging out the CPUs will not affect
>>>>>> the running system in any way.
>>>>>>
>>>>>> Please note that you can logically online and offline the same CPUs
>>>>>> over and over again without rebooting the system. Here, while onlining a CPU
>>>>>> which was offlined previously, the kernel follows almost the same sequence
>>>>>> which it normally follows while booting the CPUs during full system booting.
>>>>>>
>>>>>> Also one more thing to be noted is that, to be able to physically
>>>>>> hot-plug CPUs, apart from OS and firmware support, you also need the
>>>>>> hardware to support this feature. That is, the electrical wiring to the
>>>>>> individual CPUs should be such that plugging them in and out does not
>>>>>> interfere with the functioning of the rest of the system. As of today, there
>>>>>> are only a few systems that support physical CPU-hotplug. But you can do
>>>>>> logical CPU hotplug easily, by configuring the kernel appropriately during
>>>>>> compilation, as you have noted in one of your previous mails.
>>>>>>
>>>>>> Regards,
>>>>>> Srivatsa S. Bhat
>>>>>>
>>>>>
>>>>>
>>>>> Hi Srivatsa,
>>>>>
>>>>> That was great explanation! Thanks!
>>>>> I have just one more query. You mentioned above that " the kernel
>>>>> follows almost the same *sequence *which it normally follows while
>>>>> booting the CPUs during full system booting."
>>>>>
>>>>> Can you please explain this sequence a little ?
>>>>>
>>>>>
>>>> Hi Vaibhav,
>>>>
>>>> I'll try to outline a very high level view of what happens while booting
>>>> an SMP (Symmetric Multi-Processor) system. Instead of going through the
>>>> entire boot sequence, let me just highlight only the part that is of
>>>> interest in this discussion: booting multiple CPUs.
>>>>
>>>> The "boot processor" is the one which is booted first while booting a
>>>> system. On x86 architecture, CPU 0 is always the boot processor. Hence, if
>>>> you have observed, you cannot offline CPU0 using CPU hot-plugging on an x86
>>>> machine. (On an Intel box, the file /sys/devices/system/cpu/cpu0/online is
>>>> purposefully absent, for this reason!). But in other architectures, this
>>>> might not be the case. For example on POWER architecture, any processor in
>>>> the system can act as the boot processor.
>>>>
>>>> Once the boot processor does its initialization, the other processors,
>>>> known as "secondary processors or application processors (APs)" are
>>>> booted/initialized. Here, obviously some synchronization mechanism is
>>>> necessary to ensure that this order is followed. So in Linux, we use 2
>>>> bitmasks called "cpu_callout_mask" and "cpu_callin_mask". These bitmasks are
>>>> used to indicate the processors available in the system.
>>>>
>>>> Once the boot processor initializes itself, it updates cpu_callout_mask
>>>> to indicate which secondary processor (or application processor AP) can
>>>> initialize itself next (for example, the boot processor sets a particular
>>>> bit as 1 in the cpu_callout_mask). On the other hand, the secondary
>>>> processor would have done some very basic initialization till then and will
>>>> be testing the value of 'cpu_callout_mask' in a while loop to see if its
>>>> number has been "called out" by the boot processor. Only after the boot
>>>> processor "calls out" this AP, this AP will continue the rest of its
>>>> initialization and completes it.
>>>>
>>>> Once the AP completes its initialization, it reports back to the boot
>>>> processor by setting its number in the cpu_callin_mask. As expected, the
>>>> boot processor would have been waiting in a while loop on cpu_callin_mask to
>>>> see if this AP booted OK or not. Once it finds that the cpu_callin_mask for
>>>> this AP has been set, the boot processor follows the same procedure to boot
>>>> other APs: i.e., it updates cpu_callout_mask and waits for the corresponding
>>>> entry to be set in cpu_callin_mask by that AP and so on. This process
>>>> continues until all the APs are booted up.
>>>>
>>>> Of course, each of these "waiting" times (of both boot processor and
>>>> APs) are capped by some preset value, say for example 5 seconds. If some AP
>>>> takes more than that time to boot, the boot processor declares that the AP
>>>> could not boot and takes appropriate action (like clearing its bit in
>>>> cpu_callout_mask and logically removing that AP from its tables etc,
>>>> effectively forgetting about that processor). Similarly while the APs wait
>>>> for the boot processor to call them out, if the boot processor does not call
>>>> them within a given time period, they declare kernel panic.
>>>>
>>>> Here are some references, if you are interested in more details:
>>>>
>>>> Linux kernel source code:
>>>> 1. linux/arch/x86/kernel/smpboot.c : start_secondary() and smp_callin()
>>>>     These are the functions executed by the APs (secondary or
>>>> application processors). Actually smp_callin() is called within
>>>> start_secondary() which is the primary function executed by APs.
>>>>
>>>> 2. linux/arch/x86/kernel/smpboot.c :  do_boot_cpu()
>>>>
>>>        This is executed by the boot processor.  You can look up other
>>> important functions such as native_cpu_up().
>>>
>>>     General SMP booting info:
>>>     1. http://www.cheesecake.org/sac/smp.html
>>>
>>> [ Sorry, I accidentally sent the earlier mail before composing the text
>>> fully. ]
>>>
>>> Regards,
>>> Srivatsa S. Bhat
>>>
>>
>>
>>
>>  Awesome explanation Srivatsa!! Thanks a lot!!
>> Just had one more doubt. I am a little unclear about how the APs get
>> initialized in the beginning. In the case of Boot Processor
>> its just like a uniprocessor system. But how do the APs start executing
>> code ?
>> Could you please explain a little ?
>>
>>
> Sure. But please note that I will stick to Intel architecture while
> explaining the details.
>
> The Boot CPU or the Boot-Strap Processor (BSP) is the one which boots the
> Operating System. Then it wakes up the APs (Application Processors) when it
> is the right time.
>
> Let us now explore some background details to understand how all this
> works.
> On uniprocessor systems we use PIC (Programmable Interrupt Controller) like
> the 8259A Interrupt Controller chip to deliver interrupts to the processor.
> On Multi-Processor (MP) systems, we use something known as APICs (Advanced
> Programmable Interrupt Controllers). Every processor has a local APIC.
> And there are one or more I/O APICs in the system that are shared by all
> the processors. As the name suggests, I/O APICs are used to deliver
> interrupts from I/O devices to the processors, via the local APICs.
>
> All local APICs have unique IDs that are assigned either by the hardware or
> the BIOS during the initialization phase. Using the local APIC ID we can
> identify the processors in the system.
>
> Using these local APICs, we can send something known as "Inter-Processor
> Interrupts" or IPIs. As the name suggests, this is a mechanism for one
> processor to interrupt another processor in the system. Note that this
> mechanism can be used by any processor to talk to any other processor in
> the system (no distinction between BSP and APs here).
>
> To kick-start the APs, the BSP sends INIT IPI to each AP in turn, waits for
> some time for the IPI to be delivered to the AP and then checks if that AP
> booted up. Based on the version of the APIC used, the BSP might have to send
> 2 STARTUP IPIs to the APs with some time delay after each of the IPIs.
> [ If you have discrete APICs (i.e., 82489DX APIC) then INIT IPI will do. If
> you have integrated APIC, you need to send two STARTUP IPIs. ]
> All this is in accordance with the "Universal Start-up Algorithm" to start
> APs, as specified by Intel architecture. These IPIs cause an INIT at the AP
> to which it was delivered.
>
> Now the question is, how do you make the APs to execute a particular piece
> of code (i.e., jump to a specified location) on start-up?
> We know that whenever a processor starts after a RESET or INIT, it starts
> executing code from the reset vector (a predefined location).
> However if you want a processor to immediately jump to an address that you
> have specified, you must use the INIT IPI as part of a "warm-reset".
> Warm-reset allows you to send INIT signal to a processor without causing
> the processor to go through the entire BIOS initialization (POST -- see
> below for details) and then start the processor's execution at the
> warm-reset-vector.
>
> By putting the appropriate pointer (i.e., pointer to the AP start-up code)
> in the warm-reset-vector (system RAM location 40:67h), setting the BIOS
> shutdown code to 0Ah (which tells the BIOS that this INIT is part of a
> warm-reset) and then causing an INIT at the processor (via the IPIs), the
> Operating System can cause the processor to jump immediately to any location
> and start executing that code. This is how the BSP can boot the APs and make
> them execute some particular piece of code (in this case, the AP start-up
> code as designed in the OS).
>
> It would be worthwhile to understand what would be the state of the system
> (and the APs) before the Operating System gets control from the BIOS after
> switching ON the machine. The BIOS, upon system start, performs a procedure
> known as POST (Power-On Self Test). This is to check the status of all the
> components/circuitry of the system, including the processors, to ensure that
> they are all functioning properly. During this phase the BIOS initializes
> all the circuitry (including all the APICs and the processors) to some known
> configuration and then puts all the APs to the HALT state with interrupts
> disabled. This is to ensure that the APs don't execute Operating System code
> (we want only the BSP to execute the OS code initially). Then the BSP starts
> executing OS code.
>
> To boot APs, the BSP sends IPIs to them. But IPIs are non-maskable (note
> that the APs were in HALT state, with interrupts disabled). Hence the BSP
> will be able to kick-start AP execution and by using the warm-reset
> mechanism, it can direct the APs to execute some particular piece of code at
> startup. The BSP would have put a pointer to that AP start-up code in the
> warm-reset-vector address before sending the INIT or STARTUP IPIs to the
> APs.
>
> You might be wondering how does the BSP specify to its local APIC as to
> which AP it must send an IPI to..
> The answer is simple. During BIOS POST, an MP (Multi-Processor)
> Configuration Table will be set up (in conjunction with BSP and APs) in a
> well-known region of memory, which will be read by the OS during boot up.
> This table contains the local APIC IDs of all the processors.
> So, while sending the targeted IPIs using its local APIC, the BSP specifies
> the local APIC ID of the target AP which it wants to interrupt (and boot in
> this case). This ensures the delivery of the IPI to the correct AP.
>
> In short, this is how a Multi-Processor system gets rolling ... :-)
>
> For more details you can refer:
>
> 1. Intel Multi-Processor Specification, especially Appendix A and B.
>     http://www.intel.com/design/pentium/datashts/242016.htm
>
>
> 2. linux/arch/x86/kernel/smpboot.c :
>     do_boot_cpu(), wakeup_secondary_cpu_via_init(), native_cpu_up(),
> start_secondary()
>
> 3. linux/arch/x86/kernel/head_32.S:
>     startup_32_smp()
>
> 4. linux/arch/x86/kernel/trampoline_32.S
>
> 5. http://tldp.org/HOWTO/Linux-i386-Boot-Code-HOWTO/smpboot.html
>
> Regards,
> Srivatsa S. Bhat
>
>

 Srivatsa, you are awsome!  Thanks a lot!!
I am just wondering what all is required to gain this depth of knowledge :)

Thanks
Vaibhav Jain
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110822/6a192568/attachment-0001.html 

Difference between logical and physical cpu hotplug

[Srivatsa Bhat]

Hi Vaibhav,

On Tue, Aug 23, 2011 at 11:40 AM, Vaibhav Jain <vjoss197@gmail.com> wrote:

> On Sun, Aug 21, 2011 at 1:09 PM, Srivatsa Bhat <bhat.srivatsa@gmail.com>wrote:
>
>>   On Sat, Aug 20, 2011 at 4:05 AM, Vaibhav Jain <vjoss197@gmail.com>wrote:
>>
>>>  Awesome explanation Srivatsa!! Thanks a lot!!
>>> Just had one more doubt. I am a little unclear about how the APs get
>>> initialized in the beginning. In the case of Boot Processor
>>> its just like a uniprocessor system. But how do the APs start executing
>>> code ?
>>> Could you please explain a little ?
>>>
>>>
>> Sure. But please note that I will stick to Intel architecture while
>> explaining the details.
>>
>> The Boot CPU or the Boot-Strap Processor (BSP) is the one which boots the
>> Operating System. Then it wakes up the APs (Application Processors) when it
>> is the right time.
>>
>> Let us now explore some background details to understand how all this
>> works.
>> On uniprocessor systems we use PIC (Programmable Interrupt Controller)
>> like the 8259A Interrupt Controller chip to deliver interrupts to the
>> processor.
>> On Multi-Processor (MP) systems, we use something known as APICs (Advanced
>> Programmable Interrupt Controllers). Every processor has a local APIC.
>> And there are one or more I/O APICs in the system that are shared by all
>> the processors. As the name suggests, I/O APICs are used to deliver
>> interrupts from I/O devices to the processors, via the local APICs.
>>
>> All local APICs have unique IDs that are assigned either by the hardware
>> or the BIOS during the initialization phase. Using the local APIC ID we can
>> identify the processors in the system.
>>
>> Using these local APICs, we can send something known as "Inter-Processor
>> Interrupts" or IPIs. As the name suggests, this is a mechanism for one
>> processor to interrupt another processor in the system. Note that this
>> mechanism can be used by any processor to talk to any other processor in
>> the system (no distinction between BSP and APs here).
>>
>> To kick-start the APs, the BSP sends INIT IPI to each AP in turn, waits
>> for some time for the IPI to be delivered to the AP and then checks if that
>> AP booted up. Based on the version of the APIC used, the BSP might have to
>> send 2 STARTUP IPIs to the APs with some time delay after each of the IPIs.
>> [ If you have discrete APICs (i.e., 82489DX APIC) then INIT IPI will do.
>> If you have integrated APIC, you need to send two STARTUP IPIs. ]
>> All this is in accordance with the "Universal Start-up Algorithm" to start
>> APs, as specified by Intel architecture. These IPIs cause an INIT at the AP
>> to which it was delivered.
>>
>> Now the question is, how do you make the APs to execute a particular piece
>> of code (i.e., jump to a specified location) on start-up?
>> We know that whenever a processor starts after a RESET or INIT, it starts
>> executing code from the reset vector (a predefined location).
>> However if you want a processor to immediately jump to an address that you
>> have specified, you must use the INIT IPI as part of a "warm-reset".
>> Warm-reset allows you to send INIT signal to a processor without causing
>> the processor to go through the entire BIOS initialization (POST -- see
>> below for details) and then start the processor's execution at the
>> warm-reset-vector.
>>
>> By putting the appropriate pointer (i.e., pointer to the AP start-up code)
>> in the warm-reset-vector (system RAM location 40:67h), setting the BIOS
>> shutdown code to 0Ah (which tells the BIOS that this INIT is part of a
>> warm-reset) and then causing an INIT at the processor (via the IPIs), the
>> Operating System can cause the processor to jump immediately to any location
>> and start executing that code. This is how the BSP can boot the APs and make
>> them execute some particular piece of code (in this case, the AP start-up
>> code as designed in the OS).
>>
>> It would be worthwhile to understand what would be the state of the system
>> (and the APs) before the Operating System gets control from the BIOS after
>> switching ON the machine. The BIOS, upon system start, performs a procedure
>> known as POST (Power-On Self Test). This is to check the status of all the
>> components/circuitry of the system, including the processors, to ensure that
>> they are all functioning properly. During this phase the BIOS initializes
>> all the circuitry (including all the APICs and the processors) to some known
>> configuration and then puts all the APs to the HALT state with interrupts
>> disabled. This is to ensure that the APs don't execute Operating System code
>> (we want only the BSP to execute the OS code initially). Then the BSP starts
>> executing OS code.
>>
>> To boot APs, the BSP sends IPIs to them. But IPIs are non-maskable (note
>> that the APs were in HALT state, with interrupts disabled). Hence the BSP
>> will be able to kick-start AP execution and by using the warm-reset
>> mechanism, it can direct the APs to execute some particular piece of code at
>> startup. The BSP would have put a pointer to that AP start-up code in the
>> warm-reset-vector address before sending the INIT or STARTUP IPIs to the
>> APs.
>>
>> You might be wondering how does the BSP specify to its local APIC as to
>> which AP it must send an IPI to..
>> The answer is simple. During BIOS POST, an MP (Multi-Processor)
>> Configuration Table will be set up (in conjunction with BSP and APs) in a
>> well-known region of memory, which will be read by the OS during boot up.
>> This table contains the local APIC IDs of all the processors.
>> So, while sending the targeted IPIs using its local APIC, the BSP
>> specifies the local APIC ID of the target AP which it wants to interrupt
>> (and boot in this case). This ensures the delivery of the IPI to the correct
>> AP.
>>
>> In short, this is how a Multi-Processor system gets rolling ... :-)
>>
>> For more details you can refer:
>>
>> 1. Intel Multi-Processor Specification, especially Appendix A and B.
>>     http://www.intel.com/design/pentium/datashts/242016.htm
>>
>>
>> 2. linux/arch/x86/kernel/smpboot.c :
>>     do_boot_cpu(), wakeup_secondary_cpu_via_init(), native_cpu_up(),
>> start_secondary()
>>
>> 3. linux/arch/x86/kernel/head_32.S:
>>     startup_32_smp()
>>
>> 4. linux/arch/x86/kernel/trampoline_32.S
>>
>> 5. http://tldp.org/HOWTO/Linux-i386-Boot-Code-HOWTO/smpboot.html
>>
>> Regards,
>> Srivatsa S. Bhat
>>
>>
>  Srivatsa, you are awsome!  Thanks a lot!!
>

Ha ha :) Thank you!
You are welcome :)


> I am just wondering what all is required to gain this depth of knowledge :)
>
>

I feel humbled by your comment!
By the way, you can get information about almost anything and everything
about Linux nowadays,
given the amount of interest it has generated and the uncountable resources
available on the Internet,
most important among them being the open source code itself..
You just need to look in the right place I guess.. :)

Regards,
Srivatsa S. Bhat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110823/2069f104/attachment.html