Max CPUs on x86_64 under 2.6.x

Max CPUs on x86_64 under 2.6.x

[Colin Coe]

Hi all

Why is the number of CPUs on the x86_64 architecture only 8 but under i386
it is 255?

I've searched the list archives and Google but can't find an answer.

Thanks

CC

Re: Max CPUs on x86_64 under 2.6.x

[William Lee Irwin III]

On Mon, Jan 03, 2005 at 10:00:41PM +0800, Colin Coe wrote:
> Why is the number of CPUs on the x86_64 architecture only 8 but under i386
> it is 255?
> I've searched the list archives and Google but can't find an answer.

i386 machines have had interrupt controllers and "large scale" systems
(to the extent that 32-bit machines can be so) developed for some time.
x86-64 machines are newer, and it is the maintainer's preference to
start with a fresh codebase for the APIC.

So what you see is not a reflection of x86-64's capabilities, but
rather, of the newness of the architecture and the codebase's desire
to be "legacy-free" in manners that don't pose the threat of causing
immediate problems.

It is not now limiting the capabilities of x86-64 machines because
x86-64 machines of 64 cpus or larger have yet to be produced. For the
record, I'm unaware of SSI i386 machines larger than 64 processors.
255 represents nothing more than a theoretical limit of hardware
capabilities, and no i386 machine larger than 64 processors has ever
been constructed to the best of my knowledge.


-- wli

Re: Max CPUs on x86_64 under 2.6.x

[Colin Coe]

> On Mon, Jan 03, 2005 at 10:00:41PM +0800, Colin Coe wrote:
>> Why is the number of CPUs on the x86_64 architecture only 8 but under
>> i386
>> it is 255?
>> I've searched the list archives and Google but can't find an answer.
>
> i386 machines have had interrupt controllers and "large scale" systems
> (to the extent that 32-bit machines can be so) developed for some time.
> x86-64 machines are newer, and it is the maintainer's preference to
> start with a fresh codebase for the APIC.
>
> So what you see is not a reflection of x86-64's capabilities, but
> rather, of the newness of the architecture and the codebase's desire
> to be "legacy-free" in manners that don't pose the threat of causing
> immediate problems.
>
> It is not now limiting the capabilities of x86-64 machines because
> x86-64 machines of 64 cpus or larger have yet to be produced. For the
> record, I'm unaware of SSI i386 machines larger than 64 processors.
> 255 represents nothing more than a theoretical limit of hardware
> capabilities, and no i386 machine larger than 64 processors has ever
> been constructed to the best of my knowledge.
>
>
> -- wli
>

Hi and thanks for the response.

Just one more question, what is the '64 cpus' that you are referring to? 
The ./arch/x86_64/Kconfig file states:
---
# actually 64 maximum, but you need to fix the APIC code first
# to use clustered mode or whatever your big iron needs
config NR_CPUS
        int "Maximum number of CPUs (2-8)"
        range 2 8
        depends on SMP
        default "8"
        help
          This allows you to specify the maximum number of CPUs which this
          kernel will support.  The maximum supported value is 32 and the
          minimum value which makes sense is 2.

          This is purely to save memory - each supported CPU requires
          memory in the static kernel configuration.
---
Can you clarify this as:
- the comments says 64 CPUs,
- the code says 8 CPUs, and
- the help text says 32 CPUs.

The company I work for is looking at IBM x445 servers which support up to
32 processors (although this is via NUMA-Q).  Are NUMA-Q boxes also
subject to the 8 CPU limit?

Thanks

CC

Re: Max CPUs on x86_64 under 2.6.x

[William Lee Irwin III]

On Tue, Jan 04, 2005 at 08:17:09AM +0800, Colin Coe wrote:
> Just one more question, what is the '64 cpus' that you are referring to? 
> The ./arch/x86_64/Kconfig file states:

The "64 cpus" referred to a comparison between i386 and x86-64. i386
SMP machines having 64 cpus were the largest ever constructed with
those particular cpus. No assertion regarding x86-64 systems was
intended to be made by that particular statement.


-- wli

Re: Max CPUs on x86_64 under 2.6.x

[Andi Kleen]

"Colin Coe" <colin@coesta.com> writes:

> Hi all
>
> Why is the number of CPUs on the x86_64 architecture only 8 but under i386
> it is 255?
>
> I've searched the list archives and Google but can't find an answer.

Post 2.6.10 x86-64 will support more CPUs. 2.6.10 actually does too,
but the Kconfig hadn't been changed then. Previously there was an
8 CPU APIC driver limit, however it turned out later that it doesn't
apply to some Opteron machines.

-Andi

Re: Max CPUs on x86_64 under 2.6.x

[William Lee Irwin III]

"Colin Coe" <colin@coesta.com> writes:
>> Why is the number of CPUs on the x86_64 architecture only 8 but under i386
>> it is 255?
>> I've searched the list archives and Google but can't find an answer.

On Tue, Jan 04, 2005 at 01:34:50AM +0100, Andi Kleen wrote:
> Post 2.6.10 x86-64 will support more CPUs. 2.6.10 actually does too,
> but the Kconfig hadn't been changed then. Previously there was an
> 8 CPU APIC driver limit, however it turned out later that it doesn't
> apply to some Opteron machines.

Barring cpus with a different onboard interrupt controller from the
xAPIC or the use of external interrupt controllers to assist with cpu
addressing, 255 serves as an architectural limit for Opteron as well.


-- wli

Re: Max CPUs on x86_64 under 2.6.x

[Andi Kleen]

On Mon, Jan 03, 2005 at 06:20:34PM -0800, William Lee Irwin III wrote:
> "Colin Coe" <colin@coesta.com> writes:
> >> Why is the number of CPUs on the x86_64 architecture only 8 but under i386
> >> it is 255?
> >> I've searched the list archives and Google but can't find an answer.
> 
> On Tue, Jan 04, 2005 at 01:34:50AM +0100, Andi Kleen wrote:
> > Post 2.6.10 x86-64 will support more CPUs. 2.6.10 actually does too,
> > but the Kconfig hadn't been changed then. Previously there was an
> > 8 CPU APIC driver limit, however it turned out later that it doesn't
> > apply to some Opteron machines.
> 
> Barring cpus with a different onboard interrupt controller from the
> xAPIC or the use of external interrupt controllers to assist with cpu
> addressing, 255 serves as an architectural limit for Opteron as well.

Yes, 255 is the limit, but not 8. Opteron can enable a special flat
mode that allows flat APIC addressing upto 255.   I assume the BIOS
will set that bit on machines with that many CPUs. 

I recently audited the flat APIC code and I think it should work without 
changes to 255 CPUs, but I wasn't able to test it so far.

However 2.6.10 supports clustered mode now anyways, so you could
probably use more CPUs given the right x86-64 IBM machine. I don't
know if that has been tested so far.

The change to extend NR_CPUs and the max number of nodes just went into 
Linus' tree, you would need a recent bk snapshot.

http://linux.bkbits.net:8080/linux-2.6/cset@41da1ff62QYI89HDgrcKwnBAz6wgQg?nav=index.html|ChangeSet@-1d

-Andi