RE: [PATCH][2.4] generic support for systems with more than 8 CP Us (2/2)

RE: [PATCH][2.4] generic support for systems with more than 8 CP Us (2/2)

[Protasevich, Natalie]

>What is "flat clustered"?  Has Intel cooked up yet another APIC operating 
>mode?   8^)   As far as I knew, the flat and clustered modes were mutually 
>exclusive, based on the value in the DFR.

Yes, I noticed I was using a different "jargon" here - I meant physical
cluster (versus logical cluster).
Somehow, this expression was adopted in our place...

--Natalie

Re: [PATCH][2.4] generic support for systems with more than 8 CP Us (2/2)

[James Cleverdon]

On Sunday 22 December 2002 01:36 pm, Protasevich, Natalie wrote:
[ Snip! ]

>
> >In the last patch from Venkatesh there was a > 8CPUs option ... that
> >seems like a direct correlation to clustered apic support to me ...
> >maybe we could just switch on CONFIG_X86_CLUSTERED_APIC directly and
> >bypass CONFIG_X86_MANY_CPU? The menu text could stay the same (less
> >confusing for users than asking them about apic modes) ...
>
> Maybe, for other systems MANY_CPU criteria would make sense, but it won't
> work for us: on ES7000s with Fosters/Gallatins, we can run 1 to 32 CPUs and
> have to be in flat clustered mode in any case - whether we run 2 do 32 of

What is "flat clustered"?  Has Intel cooked up yet another APIC operating 
mode?   8^)   As far as I knew, the flat and clustered modes were mutually 
exclusive, based on the value in the DFR.

> them... This is also true for Cascades running on hierarchical cluster
> (logical). Our APIC ID's are hard-coded topologically in the BIOS, so we
> could run 2 processors on the high end of topology, with high APIC IDs. We
> couldn't get around using just ID's (not the EID's), because hardware needs
> the full CPU ID address to deliver IPIs.
>
> >M.

-- 
James Cleverdon
IBM xSeries Linux Solutions
{jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com

RE: [PATCH][2.4] generic support for systems with more than 8 CP Us (2/2)

[Pallipadi, Venkatesh]

> From: Protasevich, Natalie [mailto:Natalie.Protasevich@UNISYS.com]
> +	/*
> +	 * Switch to Physical destination mode in case of generic
> +	 * more than 8 CPU system, which has xAPIC support
> +	 */
> +#define FLAT_APIC_CPU_MAX	8
> +	if ((clustered_apic_mode == CLUSTERED_APIC_NONE) &&
> +	    (xapic_support) &&
> +	    (num_processors > FLAT_APIC_CPU_MAX)) {
> +		clustered_apic_mode = CLUSTERED_APIC_XAPIC;
> +		apic_broadcast_id = APIC_BROADCAST_ID_XAPIC;
> +		int_dest_addr_mode = APIC_DEST_PHYSICAL;
> +		int_delivery_mode = dest_Fixed;
> +		esr_disable = 1;
> +	}
> +#endif
> +
> 
> Venkatesh,
> I could not find the definition for CLUSTERED_APIC_NONE. Is 
> this an attempt
> to set up the clustered mode right in case if it was not compiled with
> CLUSTERED_..._XAPIC on the system with > 8 cpu?

CLUSTERED_APIC_NONE is the normal default mode of flat logical (defined in asm/smpboot.h)
NUMAQ, Summit has a OEM field check and they change it to Cluster or Physical mode
before this point. What I am trying here is to check whether we are still in the default
flat (with max 8 cpu) mode, and we have more than 8 xAPIC supporting CPUs, then
we switch to physical mode rather than flat mode.

> >There is already some code in base that does this. For NUMAQ 
> specifically
> >this check happens and clustered mode is selected for APIC. 
> My patch has
> this 
> >additinal check (after the initial IBM OEM check), for 
> non-NUMAQ systems
> with 
> >more than 8 CPUs and xAPIC support. These systems can not 
> work with the 
> >default flat addressing mode. So, this patch sets up such systems in
> physical 
> >mode.
> 
> In the systems like ES7000, all CPU (up to 32) and IO-APICs have 8-bit
> hard-wired ID's according to their topological position. 
> Therefore, the
> apic_broadcast_id be has to be 0xFF (like summit's, I guess), 
> and modes
> should be clustered logical for Cascades, clustered physical 
> for xAPIC. The
> IO-APIC lines are programmed with the broadcast ID/lowest priority for
> Cascades, and boot CPU ID/XTPR flag/Lowest priority for 
> Fosters/Gallatins.
> This is pretty much the only choices we have. If this is not 
> what the patch
> allows to do, we need a last chance architectural hook to be able to
> accommodate our stuff.

Unfortunately, this patch doesn't do that. My initial attempt (first version of
the patch that I had sent) was trying to set up clustered physical mode for all non-NUMAQ
systems, with more than 8 CPUs with xAPIC support. But due to the amount of code changes
that it involved, we had to fall back to setting up such systems in physical mode, 
with IO-APIC programmed for fixed Priority interrupt.

Thanks,
-Venkatesh

RE: [PATCH][2.4] generic support for systems with more than 8 CP Us (2/2)

[Protasevich, Natalie]

+	/*
+	 * Switch to Physical destination mode in case of generic
+	 * more than 8 CPU system, which has xAPIC support
+	 */
+#define FLAT_APIC_CPU_MAX	8
+	if ((clustered_apic_mode == CLUSTERED_APIC_NONE) &&
+	    (xapic_support) &&
+	    (num_processors > FLAT_APIC_CPU_MAX)) {
+		clustered_apic_mode = CLUSTERED_APIC_XAPIC;
+		apic_broadcast_id = APIC_BROADCAST_ID_XAPIC;
+		int_dest_addr_mode = APIC_DEST_PHYSICAL;
+		int_delivery_mode = dest_Fixed;
+		esr_disable = 1;
+	}
+#endif
+

Venkatesh,
I could not find the definition for CLUSTERED_APIC_NONE. Is this an attempt
to set up the clustered mode right in case if it was not compiled with
CLUSTERED_..._XAPIC on the system with > 8 cpu?

>There is already some code in base that does this. For NUMAQ specifically
>this check happens and clustered mode is selected for APIC. My patch has
this 
>additinal check (after the initial IBM OEM check), for non-NUMAQ systems
with 
>more than 8 CPUs and xAPIC support. These systems can not work with the 
>default flat addressing mode. So, this patch sets up such systems in
physical 
>mode.

In the systems like ES7000, all CPU (up to 32) and IO-APICs have 8-bit
hard-wired ID's according to their topological position. Therefore, the
apic_broadcast_id be has to be 0xFF (like summit's, I guess), and modes
should be clustered logical for Cascades, clustered physical for xAPIC. The
IO-APIC lines are programmed with the broadcast ID/lowest priority for
Cascades, and boot CPU ID/XTPR flag/Lowest priority for Fosters/Gallatins.
This is pretty much the only choices we have. If this is not what the patch
allows to do, we need a last chance architectural hook to be able to
accommodate our stuff.

Thanks,
--Natalie

>Thanks,
>-Venkatesh

Re: [PATCH][2.4] generic support for systems with more than 8 CP Us (2/2)

[William Lee Irwin III]

At some point in the past, I wrote:
> >>IIRC NUMA-Q can be dynamically detected at boot by means of an MP OEM
> >>table's presence, in particular if there's a matching string in the 8B
> >>OEM record in the OEM table, with a value of "IBM NUMA" IIRC. This is
> >>probably a line or two's worth of change to mpparse.c and declaring a
> >>variable for clustered_apic_mode. If it were difficult to detect, I
> >>wouldn't have suggested implementing it (though do so at your leisure).
> >>=)

On Sun, Dec 22, 2002 at 01:01:54PM -0800, Martin J. Bligh wrote:
> I don't think you need the OEM table to detect this, current patches do:
> 
> +static inline void mps_oem_check(struct mp_config_table *mpc, char *oem,
> +               char *productid)
> +{
> +       if (strncmp(oem, "IBM NUMA", 8))
> +               printk("Warning!  May not be a NUMA-Q system!\n");

Well, this is precisely what I meant by checking the OEM ID. It appears
to already be implemented in at least one set of patches, then. =)

Cheers,
Bill

RE: [PATCH][2.4] generic support for systems with more than 8 CP Us (2/2)

[Protasevich, Natalie]

> The format for the OEM table is pretty much freelance. ES7000 uses it in
> the most informal way. However, we need information from this table to
> switch to APIC mode, start CPUs, etc. To have the hook for OEM tables in
> MP parsing (and in ACPI, for that matter) seems pretty natural. Or if
> reading of the OEM table could also be done in more generic way so other
> platforms had chance to read their tables seamlessly...

>Support for parsing the mps oem tables is already there - I use it for
>NUMA-Q ... see if smp_read_mpc_oem will do what you want ... if not,
>maybe we can generalise it out.

Never mind: I just looked closer into the patch - magic - mpc_oem_check()
does just this! 
I apologize for missing this on the first read. Was in a hurry to get to
Christmas shopping :-)

I have to say that the mechanism developed in the patch looks perfect for
ES7000 case. 
I just need to catch up and digest it ASAP so I won't go again down the road
you already went...

>In the last patch from Venkatesh there was a > 8CPUs option ... that
>seems like a direct correlation to clustered apic support to me ...
>maybe we could just switch on CONFIG_X86_CLUSTERED_APIC directly and
>bypass CONFIG_X86_MANY_CPU? The menu text could stay the same (less
>confusing for users than asking them about apic modes) ...

Maybe, for other systems MANY_CPU criteria would make sense, but it won't
work for us: on ES7000s with Fosters/Gallatins, we can run 1 to 32 CPUs and
have to be in flat clustered mode in any case - whether we run 2 do 32 of
them... This is also true for Cascades running on hierarchical cluster
(logical). Our APIC ID's are hard-coded topologically in the BIOS, so we
could run 2 processors on the high end of topology, with high APIC IDs. We
couldn't get around using just ID's (not the EID's), because hardware needs
the full CPU ID address to deliver IPIs.

>M.

RE: [PATCH][2.4] generic support for systems with more than 8 CP Us (2/2)

[Martin J. Bligh]

>> IIRC NUMA-Q can be dynamically detected at boot by means of an MP OEM
>> table's presence, in particular if there's a matching string in the 8B
>> OEM record in the OEM table, with a value of "IBM NUMA" IIRC. This is
>> probably a line or two's worth of change to mpparse.c and declaring a
>> variable for clustered_apic_mode. If it were difficult to detect, I
>> wouldn't have suggested implementing it (though do so at your leisure).
>> =)

I don't think you need the OEM table to detect this, current patches do:

+static inline void mps_oem_check(struct mp_config_table *mpc, char *oem,
+               char *productid)
+{
+       if (strncmp(oem, "IBM NUMA", 8))
+               printk("Warning!  May not be a NUMA-Q system!\n");
+       if (mpc->mpc_oemptr)
+               smp_read_mpc_oem((struct mp_config_oemtable *) 
mpc->mpc_oemptr,
+                               mpc->mpc_oemsize);
+}


 @@ -376,7 +368,7 @@ static int __init smp_read_mpc(struct mp
        str[12]=0;
        printk("Product ID: %s ",str);

-       summit_check(oem, str);
+       mps_oem_check(mpc, oem, str);

        printk("APIC at: 0x%lX\n",mpc->mpc_lapic);

> The format for the OEM table is pretty much freelance. ES7000 uses it in
> the most informal way. However, we need information from this table to
> switch to APIC mode, start CPUs, etc. To have the hook for OEM tables in
> MP parsing (and in ACPI, for that matter) seems pretty natural. Or if
> reading of the OEM table could also be done in more generic way so other
> platforms had chance to read their tables seamlessly...

Support for parsing the mps oem tables is already there - I use it for
NUMA-Q ... see if smp_read_mpc_oem will do what you want ... if not,
maybe we can generalise it out.

> Also, could the clustered_logical and clustered_physical modes be
> implemented as a primary item, with NUMA being a secondary with the set of
> its unique features on top of the certain APIC mode? This way, a clustered
> mode for the APIC could be selectable without NUMA. CLUSTERED_APIC_NUMAQ
> could be still an option with an appropriate clustered mode defined.

Yup - it was only set up that way because we had a 1-1 correlation between
NUMA and clustered_apic_mode at the time. Switching the ordering as you
suggest seems sensible now.

In the last patch from Venkatesh there was a > 8CPUs option ... that
seems like a direct correlation to clustered apic support to me ...
maybe we could just switch on CONFIG_X86_CLUSTERED_APIC directly and
bypass CONFIG_X86_MANY_CPU? The menu text could stay the same (less
confusing for users than asking them about apic modes) ...

M.

RE: [PATCH][2.4] generic support for systems with more than 8 CP Us (2/2)

[Protasevich, Natalie]

>> 2/2 : switching to physical mode APIC setup in case of more than 8 CPU
system
[...]
>> -	printk("Enabling APIC mode: ");
>> -	if(clustered_apic_mode == CLUSTERED_APIC_NUMAQ)
>> -		printk("Clustered Logical.	");
>> -	else if(clustered_apic_mode == CLUSTERED_APIC_XAPIC)
>> -		printk("Physical.	");
>> -	else
>> -		printk("Flat.	");
>> -	printk("Using %d I/O APICs\n",nr_ioapics);

>IIRC NUMA-Q can be dynamically detected at boot by means of an MP OEM
>table's presence, in particular if there's a matching string in the 8B
>OEM record in the OEM table, with a value of "IBM NUMA" IIRC. This is
>probably a line or two's worth of change to mpparse.c and declaring a
>variable for clustered_apic_mode. If it were difficult to detect, I
>wouldn't have suggested implementing it (though do so at your leisure). =)

The format for the OEM table is pretty much freelance. ES7000 uses it in the
most informal way. However, we need information from this table to switch to
APIC mode, start CPUs, etc. To have the hook for OEM tables in MP parsing
(and in ACPI, for that matter) seems pretty natural. Or if reading of the
OEM table could also be done in more generic way so other platforms had
chance to read their tables seamlessly...

Also, could the clustered_logical and clustered_physical modes be
implemented as a primary item, with NUMA being a secondary with the set of
its unique features on top of the certain APIC mode? This way, a clustered
mode for the APIC could be selectable without NUMA. CLUSTERED_APIC_NUMAQ
could be still an option with an appropriate clustered mode defined. 

--Natalie

>I still think this is 2.5 material + backport once it gets testing there.


>Bill