linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* RE: 256 apic id for amd64
@ 2005-01-07  2:53 YhLu
  2005-01-07 12:24 ` Andi Kleen
  0 siblings, 1 reply; 52+ messages in thread
From: YhLu @ 2005-01-07  2:53 UTC (permalink / raw)
  To: Andi Kleen, Matt Domsch; +Cc: linux-kernel, discuss

static unsigned int phys_pkg_id(int index_msb)
{
        return hard_smp_processor_id() >> index_msb;
}

In arch/x86_64/kernel/genapic_cluster.c

Should be changed to 

static unsigned int phys_pkg_id(int index_msb)
{
        /* physical apicid, so we need to substract offset */
        return (hard_smp_processor_id() - boot_cpu_id) >> index_msb;
}

-----Original Message-----
From: YhLu 
Sent: Thursday, January 06, 2005 5:07 PM
To: Andi Kleen; Matt Domsch
Cc: linux-kernel@vger.kernel.org; discuss@x86-64.org
Subject: RE: 256 apic id for amd64

Andi,

I made the Opteron using apic id from 16 later in LinuxBIOS.

So leave 0-15 for ioapic.

The problem is that the kernel (caliberate_delay) doesn't like that.

If using lpj=2170880 as the command line for that, it works well.

What's the jiffies? The TSC is changing but it doesn't.

YH

Without offset:
Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
Memory: 1019456k/1048576k available (3011k kernel code, 0k reserved, 1310k
data, 544k init)
LYH calibrating 0 jiffies = 4294667563, now=2922366256
LYH calibrating 1 jiffies = 4294667568, now=2934372713
LYH calibrating 3 jiffies = 4294667600, now=3003581909
4341.76 BogoMIPS (lpj=2170880)
Mount-cache hash table entries: 256 (order: 0, 4096 bytes)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 0 -> Node 0
CPU: Physical Processor ID: 0

With apic id offset:
Console: colour dummy device 80x25
Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
Memory: 1019456k/1048576k available (3011k kernel code, 0k reserved, 1310k
data, 544k init)
LYH calibrating 0 jiffies = 4294667296, now=1383947209
LYH calibrating 1 jiffies = 4294667296, now=1395952717


Please refer the print in init/main.c

void __devinit calibrate_delay(void)
{
        unsigned long ticks, loopbit;
        int lps_precision = LPS_PREC;
 +       unsigned long now;
 +
 +       rdtscl(now);
 +
 +        printk("LYH calibrating 0 jiffies = %lu, now=%lu\r\n", jiffies,
now);


        if (preset_lpj) {
                loops_per_jiffy = preset_lpj;
                printk("Calibrating delay loop (skipped)... "
                        "%lu.%02lu BogoMIPS preset\n",
                        loops_per_jiffy/(500000/HZ),
                        (loops_per_jiffy/(5000/HZ)) % 100);
        } else {
                loops_per_jiffy = (1<<12);

 +               rdtscl(now);
 +               printk("LYH calibrating 1 jiffies = %lu, now=%lu\r\n",
jiffies, now);
                printk(KERN_DEBUG "Calibrating delay loop... ");
                while ((loops_per_jiffy <<= 1) != 0) {
                        /* wait for "start of" clock tick */
                        ticks = jiffies;
                        while (ticks == jiffies)
                                /* nothing */;
                        /* Go .. */
                        ticks = jiffies;
                        __delay(loops_per_jiffy);
                        ticks = jiffies - ticks;
                        if (ticks)
                                break;
                }

                /*
                 * Do a binary approximation to get loops_per_jiffy set to
                 * equal one clock (up to lps_precision bits)
                 */
  +              rdtscl(now);
  +              printk("LYH calibrating 2 jiffies = %lu, now=%lu\r\n",
jiffies, now);

                loops_per_jiffy >>= 1;
                loopbit = loops_per_jiffy;
                while (lps_precision-- && (loopbit >>= 1)) {
                        loops_per_jiffy |= loopbit;
                        ticks = jiffies;
                       while (ticks == jiffies)
                                /* nothing */;
                        ticks = jiffies;
                        __delay(loops_per_jiffy);
                        if (jiffies != ticks)   /* longer than 1 tick */
                                loops_per_jiffy &= ~loopbit;
                }

  +              rdtscl(now);
  +              printk("LYH calibrating 3 jiffies = %lu, now=%lu\r\n",
jiffies, now);

                /* Round the value and print it */
                printk("%lu.%02lu BogoMIPS (lpj=%lu)\n",
                        loops_per_jiffy/(500000/HZ),
                        (loops_per_jiffy/(5000/HZ)) % 100,
                        loops_per_jiffy);
        }

}
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2005-01-11 19:11 YhLu
  0 siblings, 0 replies; 52+ messages in thread
From: YhLu @ 2005-01-11 19:11 UTC (permalink / raw)
  To: 'Andi Kleen'
  Cc: 'Mikael Pettersson', 'jamesclv@us.ibm.com',
	'Matt_Domsch@dell.com', 'discuss@x86-64.org',
	'linux-kernel@vger.kernel.org',
	'suresh.b.siddha@intel.com'

On 2.4.28 84 bit
~ # cat /proc/interrupts 
           CPU0       CPU1       CPU2       CPU3       
  0:       5302         19          0          0    IO-APIC-edge  timer
  1:          0          0          0          0    IO-APIC-edge  keyboard
  2:          0          0          0          0          XT-PIC  cascade
  4:        175          0          0          0    IO-APIC-edge  serial
 19:          3          0          0          0   IO-APIC-level  ohci1394
 22:          0          0          0          0   IO-APIC-level  usb-ohci
 23:          0          0          0          0   IO-APIC-level  ehci_hcd
NMI:        147        139        110         77 
LOC:       5063       5060       5061       5062 
ERR:        406
MIS:          0
~ # cat /proc/interrupts 
           CPU0       CPU1       CPU2       CPU3       
  0:       5612         19          0          0    IO-APIC-edge  timer
  1:          0          0          0          0    IO-APIC-edge  keyboard
  2:          0          0          0          0          XT-PIC  cascade
  4:        226          0          0          0    IO-APIC-edge  serial
 19:          3          0          0          0   IO-APIC-level  ohci1394
 22:          0          0          0          0   IO-APIC-level  usb-ohci
 23:          0          0          0          0   IO-APIC-level  ehci_hcd
NMI:        147        139        110         77 
LOC:       5373       5370       5371       5372 
ERR:        406
MIS:          0

-----Original Message-----
From: YhLu 
Sent: Tuesday, January 11, 2005 11:05 AM
To: YhLu; Andi Kleen
Cc: 'Mikael Pettersson'; jamesclv@us.ibm.com; Matt_Domsch@dell.com;
discuss@x86-64.org; linux-kernel@vger.kernel.org; suresh.b.siddha@intel.com
Subject: RE: 256 apic id for amd64

On 2.6.10 32 bit, it only crash core1/node0, still can go on but lost one
cpu.

~ # cat /proc/interrupts 
           CPU0       CPU1       CPU2       
  0:      47612          5       7643    IO-APIC-edge  timer
  2:          0          0          0          XT-PIC  cascade
  4:        365          0          1    IO-APIC-edge  serial
  8:          0          0          0    IO-APIC-edge  rtc
 17:          0          0          0   IO-APIC-level  libata
 18:          0          0          0   IO-APIC-level  libata
 19:          2          0          1   IO-APIC-level  ohci1394
 22:          0          0          0   IO-APIC-level  ohci_hcd
 23:          0          0          0   IO-APIC-level  ehci_hcd
NMI:          0          0          0 
LOC:      49259      49254      49253 
ERR:       5729
MIS:          0
~ #

-----Original Message-----
From: Yinghai Lu [mailto:yhlu@tyan.com] 
Sent: Tuesday, January 11, 2005 10:50 AM
To: 'Andi Kleen'
Cc: 'Mikael Pettersson'; 'jamesclv@us.ibm.com'; 'Matt_Domsch@dell.com';
'discuss@x86-64.org'; 'linux-kernel@vger.kernel.org';
'suresh.b.siddha@intel.com'
Subject: RE: 256 apic id for amd64

> It looks like the timer interrupt goes to the second CPU.
Even I don't lift the apic id for bsp, the MB with Nvidia chipset still can
not start core1/node0, from core0/node0 in kernel 2.6. But the 2.4.28 (32
bit, 64 bit) works well. 

How come?

YH

^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2005-01-11 19:04 YhLu
  0 siblings, 0 replies; 52+ messages in thread
From: YhLu @ 2005-01-11 19:04 UTC (permalink / raw)
  To: YhLu, Andi Kleen
  Cc: 'Mikael Pettersson',
	jamesclv, Matt_Domsch, discuss, linux-kernel, suresh.b.siddha

On 2.6.10 32 bit, it only crash core1/node0, still can go on but lost one
cpu.

~ # cat /proc/interrupts 
           CPU0       CPU1       CPU2       
  0:      47612          5       7643    IO-APIC-edge  timer
  2:          0          0          0          XT-PIC  cascade
  4:        365          0          1    IO-APIC-edge  serial
  8:          0          0          0    IO-APIC-edge  rtc
 17:          0          0          0   IO-APIC-level  libata
 18:          0          0          0   IO-APIC-level  libata
 19:          2          0          1   IO-APIC-level  ohci1394
 22:          0          0          0   IO-APIC-level  ohci_hcd
 23:          0          0          0   IO-APIC-level  ehci_hcd
NMI:          0          0          0 
LOC:      49259      49254      49253 
ERR:       5729
MIS:          0
~ #

-----Original Message-----
From: Yinghai Lu [mailto:yhlu@tyan.com] 
Sent: Tuesday, January 11, 2005 10:50 AM
To: 'Andi Kleen'
Cc: 'Mikael Pettersson'; 'jamesclv@us.ibm.com'; 'Matt_Domsch@dell.com';
'discuss@x86-64.org'; 'linux-kernel@vger.kernel.org';
'suresh.b.siddha@intel.com'
Subject: RE: 256 apic id for amd64

> It looks like the timer interrupt goes to the second CPU.
Even I don't lift the apic id for bsp, the MB with Nvidia chipset still can
not start core1/node0, from core0/node0 in kernel 2.6. But the 2.4.28 (32
bit, 64 bit) works well. 

How come?

YH

^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2005-01-10 20:48 YhLu
  0 siblings, 0 replies; 52+ messages in thread
From: YhLu @ 2005-01-10 20:48 UTC (permalink / raw)
  To: Andi Kleen
  Cc: 'Mikael Pettersson',
	jamesclv, Matt_Domsch, discuss, linux-kernel, suresh.b.siddha

Not lifting bsp to 0x10, it fails on TIMER too.

YH

Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
Memory: 1019432k/1048576k available (3006k kernel code, 0k reserved, 1317k
data, 548k init)
LYH calibrating 0 jiffies = 4294667566, now=2500693458
LYH calibrating 1 jiffies = 4294667571, now=2511802784
LYH calibrating 3 jiffies = 4294667601, now=2579969051
4341.76 BogoMIPS (lpj=2170880)
Mount-cache hash table entries: 256 (order: 0, 4096 bytes)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 0(2) -> Node 0
CPU: Physical Processor ID: 0
Using local APIC NMI watchdog using perfctr0
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 0(2) -> Node 0
CPU: Physical Processor ID: 0
CPU0:  stepping 00
per-CPU timeslice cutoff: 1023.91 usecs.
task migration cache decay timeout: 2 msecs.
Booting processor 1/1 rip 6000 rsp ffff810002217f58
Initializing CPU#1
LYH calibrating 0 jiffies = 4294667672, now=2736795282
LYH calibrating 1 jiffies = 4294667672, now=2736800180
Calibrating delay loop... <7>Calibrating delay loop... ----------- [cut here
] --------- [please bite here ] ---------
Kernel BUG at timer:416
invalid operand: 0000 [1] SMP 
CPU 1 
Modules linked in:
Pid: 0, comm: swapper Not tainted 2.6.10-bk13
RIP: 0010:[<ffffffff8013958d>] <ffffffff8013958d>{cascade+45}
RSP: 0018:ffff81000221fed8  EFLAGS: 00010007
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff810001e15820
RBP: ffff810001e16838 R08: 00000000fffffff2 R09: 0000000000000009
R10: 00000000ffffffff R11: 0000000000000000 R12: fff

^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2005-01-10 20:46 YhLu
  2005-01-11  5:36 ` Andi Kleen
  0 siblings, 1 reply; 52+ messages in thread
From: YhLu @ 2005-01-10 20:46 UTC (permalink / raw)
  To: Andi Kleen
  Cc: 'Mikael Pettersson',
	jamesclv, Matt_Domsch, discuss, linux-kernel, suresh.b.siddha

With lifting cpu0 apic id to 0x10, on Nvidia chip set, it could pass the
calibrate_delay in bsp. But it can not start core1/node0.


Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
Memory: 1019432k/1048576k available (3006k kernel code, 0k reserved, 1317k
data, 548k init)
LYH calibrating 0 jiffies = 4294667567, now=1622307524
LYH calibrating 1 jiffies = 4294667572, now=1633421978
LYH calibrating 3 jiffies = 4294667601, now=1699336699
4374.52 BogoMIPS (lpj=2187264)
Mount-cache hash table entries: 256 (order: 0, 4096 bytes)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 0(2) -> Node 0
CPU: Physical Processor ID: 0
Using local APIC NMI watchdog using perfctr0
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 0(2) -> Node 0
CPU: Physical Processor ID: 0
CPU0:  stepping 00
per-CPU timeslice cutoff: 1023.91 usecs.
task migration cache decay timeout: 2 msecs.
Booting processor 1/17 rip 6000 rsp ffff810002217f58
Initializing CPU#1
LYH calibrating 0 jiffies = 4294667674, now=1860098546
LYH calibrating 1 jiffies = 4294667674, now=1860104172
Calibrating delay loop... <7>Calibrating delay loop... <7>spurious 8259A
interrupt: IRQ7.
APIC error on CPU0: 00(08)
APIC error on CPU1: 00(08)
APIC error on CPU1: 08(08)
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at timer:416
invalid operand: 0000 [1] SMP 
CPU 1 
Modules linked in:
Pid: 0, comm: swapper Not tainted 2.6.10-bk13
RIP: 0010:[<ffffffff8013958d>] <ffffffff8013958d>{cascade+45}
RSP: 0018:ffff81000221fed8  EFLAGS: 00010007
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff810001e15820
RBP: ffff810001e16838 R08: 00000000fffffff2 R09: 0000000000000009
R10: 00000000ffffffff R11: 0000000000000000 R12: ffff810001e15820
R13: 0000000000000000 R14: ffff81000221ff08 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffffffff80579080(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000000101000 CR4: 00000000000006a0
Process swapper (pid: 0, threadinfo ffff810002216000, task ffff810002210dd0)
Stack: 0000000000000000 0000000000000000 ffffffff8057a510 ffff810001e15820 
       ffffffff80609900 ffffffff8013970e ffff81000221ff08 ffff81000221ff08 
       ffffffff8057a764 fffffffffffffffe 
Call Trace:<IRQ> <ffffffff8013970e>{run_timer_softirq+126}
<ffffffff801361b1>{__do_softirq+113} 
       <ffffffff80136265>{do_softirq+53} <ffffffff8010ff3f>{do_IRQ+63} 
       <ffffffff8010d865>{ret_from_intr+0}  <EOI>
<ffffffff805880e1>{calibrate_delay+289} 
       <ffffffff8058d83b>{smp_callin+171}
<ffffffff8058d92e>{start_secondary+14} 
       


^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2005-01-10 20:37 YhLu
  0 siblings, 0 replies; 52+ messages in thread
From: YhLu @ 2005-01-10 20:37 UTC (permalink / raw)
  To: Andi Kleen
  Cc: 'Mikael Pettersson',
	jamesclv, Matt_Domsch, discuss, linux-kernel, suresh.b.siddha

You are right, it should be separate that to init_amd and init_intel.

I guess for intel dual core initial apic id different to AMD.
It could be 0,7 for node0. 1, 6 for node1......

Amd would be (0, 1) for node 0, (2,3) for node1 ....

YH

-----Original Message-----
From: Andi Kleen [mailto:ak@muc.de] 
Sent: Monday, January 10, 2005 12:19 PM
To: YhLu
Cc: 'Mikael Pettersson'; jamesclv@us.ibm.com; Matt_Domsch@dell.com;
discuss@x86-64.org; linux-kernel@vger.kernel.org; suresh.b.siddha@intel.com
Subject: Re: 256 apic id for amd64

On Mon, Jan 10, 2005 at 12:09:48PM -0800, YhLu wrote:
> Try this one.

I don't think it will work at all on Intel HT systems, since
nobody initializes c->x86_num_cores there. phys_proc_id[] 
is supposed to be the same on two HT siblings.

You'll either need to initialize c->x86_num_cores on Intel too.
But since Intel seems to have dual cores upcomming and you'll
want HyperThreading support too it would need to be a new field.

Alternatively you can split the function for AMD and Intel and
use the new algorithm on AMD only.  Perhaps that's better.
In the later case I would only use it when CMP_LEGACY is set.  

-Andi

^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2005-01-10 20:09 YhLu
  2005-01-10 20:18 ` Andi Kleen
  0 siblings, 1 reply; 52+ messages in thread
From: YhLu @ 2005-01-10 20:09 UTC (permalink / raw)
  To: Andi Kleen
  Cc: 'Mikael Pettersson',
	jamesclv, Matt_Domsch, discuss, linux-kernel, suresh.b.siddha

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

Try this one.

-----Original Message-----
From: Andi Kleen [mailto:ak@muc.de] 
Sent: Monday, January 10, 2005 11:44 AM
To: YhLu
Cc: 'Mikael Pettersson'; jamesclv@us.ibm.com; Matt_Domsch@dell.com;
discuss@x86-64.org; linux-kernel@vger.kernel.org; suresh.b.siddha@intel.com
Subject: Re: 256 apic id for amd64

On Mon, Jan 10, 2005 at 11:41:02AM -0800, YhLu wrote:
> Please refer the patch.

That's unreadable. Can you generate against a recent BK snapshot
(2.6.10 + latest from ftp://ftp.kernel.org/pub/linux/kernel/v2.6/snapshots/)

and use diff -u  ? 

-Andi


[-- Attachment #2: x86_64_phy_proc_id.patch --]
[-- Type: application/octet-stream, Size: 1175 bytes --]

--- linux-2.6.10/arch/x86_64/kernel/setup.c	2005-01-10 12:26:21.414190368 -0800
+++ linux-2.6.10.new.x86_64/arch/x86_64/kernel/setup.c	2005-01-10 12:30:53.656803200 -0800
@@ -738,7 +738,6 @@
 {
 #ifdef CONFIG_SMP
 	u32 	eax, ebx, ecx, edx;
-	int 	index_lsb, index_msb, tmp;
 	int 	cpu = smp_processor_id();
 	
 	if (!cpu_has(c, X86_FEATURE_HT))
@@ -750,8 +749,6 @@
 	if (smp_num_siblings == 1) {
 		printk(KERN_INFO  "CPU: Hyper-Threading is disabled\n");
 	} else if (smp_num_siblings > 1) {
-		index_lsb = 0;
-		index_msb = 31;
 		/*
 		 * At this point we only support two siblings per
 		 * processor package.
@@ -761,19 +758,8 @@
 			smp_num_siblings = 1;
 			return;
 		}
-		tmp = smp_num_siblings;
-		while ((tmp & 1) == 0) {
-			tmp >>=1 ;
-			index_lsb++;
-		}
-		tmp = smp_num_siblings;
-		while ((tmp & 0x80000000 ) == 0) {
-			tmp <<=1 ;
-			index_msb--;
-		}
-		if (index_lsb != index_msb )
-			index_msb++;
-		phys_proc_id[cpu] = phys_pkg_id(index_msb);
+
+		phys_proc_id[cpu] = c->x86_apicid >> hweight32(c->x86_num_cores - 1);
 		
 		printk(KERN_INFO  "CPU: Physical Processor ID: %d\n",
 		       phys_proc_id[cpu]);

^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2005-01-10 19:41 YhLu
  2005-01-10 19:43 ` Andi Kleen
  2005-01-11  0:42 ` James Cleverdon
  0 siblings, 2 replies; 52+ messages in thread
From: YhLu @ 2005-01-10 19:41 UTC (permalink / raw)
  To: Andi Kleen
  Cc: 'Mikael Pettersson',
	jamesclv, Matt_Domsch, discuss, linux-kernel, suresh.b.siddha

[-- Attachment #1: Type: text/plain, Size: 38 bytes --]

Please refer the patch.

Regards

YH


[-- Attachment #2: x86_64_phy_proc_id.patch --]
[-- Type: application/octet-stream, Size: 1134 bytes --]

664a665
> 	unsigned int n;
687c688,690
< 	if (c->cpuid_level >= 0x80000008) {
---
> 	n = cpuid_eax(0x80000000);
> 
> 	if (n >= 0x80000008) {
698c701
< 			int cpu = c->x86_apicid;
---
> 			int cpu = c->x86_apicid; // that is initial apicid
725,749c728,730
< 		index_lsb = 0;
< 		index_msb = 31;
< 		/*
< 		 * At this point we only support two siblings per
< 		 * processor package.
< 		 */
< 		if (smp_num_siblings > NR_CPUS) {
< 			printk(KERN_WARNING "CPU: Unsupported number of the siblings %d", smp_num_siblings);
< 			smp_num_siblings = 1;
< 			return;
< 		}
< 		tmp = smp_num_siblings;
< 		while ((tmp & 1) == 0) {
< 			tmp >>=1 ;
< 			index_lsb++;
< 		}
< 		tmp = smp_num_siblings;
< 		while ((tmp & 0x80000000 ) == 0) {
< 			tmp <<=1 ;
< 			index_msb--;
< 		}
< 		if (index_lsb != index_msb )
< 			index_msb++;
< 		phys_proc_id[cpu] = phys_pkg_id(index_msb);
< 		
---
> 
> 		phys_proc_id[cpu] =  c->x86_apicid >> hweight32(c->x86_num_cores - 1);
> 
1067c1048
< 		seq_printf(m, "cpu cores\t: %d\n", c->x86_num_cores);
---
> 		seq_printf(m, "\ncpu cores\t: %d\n", c->x86_num_cores);

^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2005-01-10 18:48 YhLu
  2005-01-10 18:45 ` Andi Kleen
  0 siblings, 1 reply; 52+ messages in thread
From: YhLu @ 2005-01-10 18:48 UTC (permalink / raw)
  To: jamesclv
  Cc: Mikael Pettersson, ak, Matt_Domsch, discuss, linux-kernel,
	suresh.b.siddha

James,

I'm working on add amd dual core LinuxBIOS support to our MB. So I can
change the apic id as I want.

When I lift the apic id for CPUS, if the bsp apicid is changed to 0x10, the
jiffies is not changing, So I have to leave to set BSP using apic id 0 in
LinuxBIOS. And lifting others to use 0x11.....
According to Andi, that would be one bug in kernel .....


Regards

YH

^ permalink raw reply	[flat|nested] 52+ messages in thread
* Re: 256 apic id for amd64
@ 2005-01-10 18:44 Andi Kleen
  2005-01-11  4:04 ` Siddha, Suresh B
  0 siblings, 1 reply; 52+ messages in thread
From: Andi Kleen @ 2005-01-10 18:44 UTC (permalink / raw)
  To: Yinghai Lu
  Cc: 'Mikael Pettersson',
	jamesclv, Matt_Domsch, discuss, linux-kernel, suresh.b.siddha

On Mon, Jan 10, 2005 at 10:32:42AM -0800, Yinghai Lu wrote:
> Resend
> 
> Andi, 
> 
> I want to repeat the request:
> Please consider to compute the phy_proc_id with x86_apicid (or the initial
> apic id).

Not sure what you want me to do exactly. 

The phys_proc_id[] array? First that isn't really used for anything and then 
it is filled from CPUID 1 anyways. Or phys_pkg_id()? That uses cpuid too.

If you think something is wrong best you send a patch that makes it 
clear what you're talking about.
> 
> The reason is 
> Phy_proc_id will be used in smpboot.c to produce cpu_siblings_map.
> 
> The two systems I mentioned really exist.
> System 1. 8 ways dual core opteron system and with 3 io apic.
> System 2. 4 ways dual core opteron system and with 15 io apic (7 amd 8131
> and 1 amd 8111).
> 
> We have to lift the apicid of cpus to make io apic to use 0-15.

I don't understand why you want to do that. The IO-APIC and the local APIC ID 
space should separate as far as I know, so you could use 0 for both a CPU 
and an IO-APIC. But in theory it should work anyways even with an BSP 
CPU ID != 0.

> Case 1: lift core0/node0 to use 16, and core1/node0 to use 17.....
> Case 2: core0/node0-->0, core1/node0-->1, core0/node1-->18,
> core1/node1-->19...
> Case 3: core0/node0-->0, core1/node0-->17, core0/node1-->18,
> core1/node1-->19...
> 
> Case 1 will make jiffies not happy and it will hang on calibrate_dalay. (
> jiffies is not changing).

That's because it needs physical APIC mode to handle CPU IDs >7, and that is
 not implemented in the flat case. I added it now, but you likely don't need 
it anyways.

-Andi


^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2005-01-08  2:53 YhLu
  2005-01-09 23:56 ` James Cleverdon
  0 siblings, 1 reply; 52+ messages in thread
From: YhLu @ 2005-01-08  2:53 UTC (permalink / raw)
  To: Mikael Pettersson, ak
  Cc: Matt_Domsch, discuss, jamesclv, linux-kernel, suresh.b.siddha

I mean keep the bsp physical apic id using 0.

YH

-----Original Message-----
From: Mikael Pettersson [mailto:mikpe@csd.uu.se] 
Sent: Friday, January 07, 2005 6:38 PM
To: YhLu; ak@muc.de
Cc: Matt_Domsch@dell.com; discuss@x86-64.org; jamesclv@us.ibm.com;
linux-kernel@vger.kernel.org; suresh.b.siddha@intel.com
Subject: Re: 256 apic id for amd64

On Fri, 7 Jan 2005 22:12:00 +0100, Andi Kleen wrote:
>On Fri, Jan 07, 2005 at 01:14:24PM -0800, YhLu wrote:
>> After keep the bsp using 0, the jiffies works well. Werid?
>
>Probably a bug somewhere.  But since BSP should be always 
>0 I'm not sure it is worth tracking down.

I hope by "0" you're referring to a Linux kernel defined
software value and _not_ what the HW or BIOS conjured up!

Case in point: I was involved a while ago in tracking down
and fixing a local APIC enumeration bug in the x86-32 (i386)
kernel code, where the kernel failed miserably on some
dual K7 boxes because (a) only one CPU socket was populated,
(b) the BIOS assigned that CPU a non-zero ID, and (c) the
kernel (apic.c) had a bug which triggered when BSP ID != 0.

Never trust a BIOS to DTRT.

/Mikael

^ permalink raw reply	[flat|nested] 52+ messages in thread
* Re: 256 apic id for amd64
@ 2005-01-08  2:37 Mikael Pettersson
  2005-01-08 15:46 ` Andi Kleen
  0 siblings, 1 reply; 52+ messages in thread
From: Mikael Pettersson @ 2005-01-08  2:37 UTC (permalink / raw)
  To: YhLu, ak; +Cc: Matt_Domsch, discuss, jamesclv, linux-kernel, suresh.b.siddha

On Fri, 7 Jan 2005 22:12:00 +0100, Andi Kleen wrote:
>On Fri, Jan 07, 2005 at 01:14:24PM -0800, YhLu wrote:
>> After keep the bsp using 0, the jiffies works well. Werid?
>
>Probably a bug somewhere.  But since BSP should be always 
>0 I'm not sure it is worth tracking down.

I hope by "0" you're referring to a Linux kernel defined
software value and _not_ what the HW or BIOS conjured up!

Case in point: I was involved a while ago in tracking down
and fixing a local APIC enumeration bug in the x86-32 (i386)
kernel code, where the kernel failed miserably on some
dual K7 boxes because (a) only one CPU socket was populated,
(b) the BIOS assigned that CPU a non-zero ID, and (c) the
kernel (apic.c) had a bug which triggered when BSP ID != 0.

Never trust a BIOS to DTRT.

/Mikael

^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2005-01-08  1:50 YhLu
  0 siblings, 0 replies; 52+ messages in thread
From: YhLu @ 2005-01-08  1:50 UTC (permalink / raw)
  To: jamesclv, Andi Kleen; +Cc: Matt Domsch, linux-kernel, discuss, suresh.b.siddha

You already get boot_cpu_physical_apic_id and it is a MARCO = boot_cpu_id.
And it always is physical apic id.

Also you already got cpu_to_node array.

Also you already got x86_cpu_to_apicid array. ( physical apic id are stored
according to cpu num).

Here cpu are [0, NR_CPUS-1), and for 8 ways dual core system it should be
16.
And can get in context by smp_processor_id or safe_smp_processor_id ( it
need x86_cpu_to_apicid to be initialized at first)

Cpu_to_node is used for NUMA.

What is the usage for phys_pkg_id?

YH

-----Original Message-----
From: James Cleverdon [mailto:jamesclv@us.ibm.com] 
Sent: Friday, January 07, 2005 5:30 PM
To: Andi Kleen
Cc: YhLu; Matt Domsch; linux-kernel@vger.kernel.org; discuss@x86-64.org;
suresh.b.siddha@intel.com
Subject: Re: 256 apic id for amd64

Andi has already dealt with some of the coding style issues elsewhere in 
the thread.

My comment:  This is playing with fire.  We've gone to considerable 
trouble to make the boot_cpu_id independent of the physical APIC ID 
(which is what hard_smp_processor_id() returns).  Different BIOSes and 
different CPU revisions can cause the boot processor to shift.

Examples:  We have a box where the boot CPU has an APIC ID of 3.  
Another system starts with the BSP == 3, but the BIOS renumbers it to 
zero after first reassigning the original 0 CPU.  So, the APIC IDs end 
up 0, 1, 2, 4.  Yet another system assigns the IDs:  0, 1, 6, 7.

We can expect even stranger numbering schemes in the future, given that 
dual core hyperthreaded CPUs are in the pipeline.  Creating any 
dependency between CPU number and APIC ID is a _bad_ idea.


On Friday 07 January 2005 04:24 am, Andi Kleen wrote:
> On Thu, Jan 06, 2005 at 06:53:11PM -0800, YhLu wrote:
> > static unsigned int phys_pkg_id(int index_msb)
> > {
> >         return hard_smp_processor_id() >> index_msb;
> > }
> >
> > In arch/x86_64/kernel/genapic_cluster.c
> >
> > Should be changed to
> >
> > static unsigned int phys_pkg_id(int index_msb)
> > {
> >         /* physical apicid, so we need to substract offset */
> >         return (hard_smp_processor_id() - boot_cpu_id) >>
> > index_msb; }
>
> Why?
>
> If you want a patch merged you need to supply some more explanation
> please.
>
> Also cc Suresh & James for comment.
>
> -Andi

-- 
James Cleverdon
IBM LTC (xSeries Linux Solutions)
{jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot comm

^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2005-01-08  0:50 YhLu
  2005-01-08  0:42 ` Andi Kleen
  0 siblings, 1 reply; 52+ messages in thread
From: YhLu @ 2005-01-08  0:50 UTC (permalink / raw)
  To: Andi Kleen, James Cleverdon
  Cc: Matt Domsch, linux-kernel, discuss, suresh.b.siddha

But the result looks ugly

I keep core0 and core1 of node0 to use 0/1 got

4407.29 BogoMIPS (lpj=2203648)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 7 -> Node 3
phy_proc_id[0] = 0
phy_proc_id[1] = 0
phy_proc_id[2] = 9
phy_proc_id[3] = 9
phy_proc_id[4] = 10
phy_proc_id[5] = 10
phy_proc_id[6] = 11
phy_proc_id[7] = 11
CPU: Physical Processor ID: 11
 stepping 00
Total of 8 processors activated (35209.21 BogoMIPS).
If only keep core0/node0 to use 0.

Will get
phy_proc_id[0] = 0
phy_proc_id[1] = 8
phy_proc_id[2] = 9
phy_proc_id[3] = 9
phy_proc_id[4] = 10
phy_proc_id[5] = 10
phy_proc_id[6] = 11
phy_proc_id[7] = 11

it separate core0 and core1 of node 1

YH

-----Original Message-----
From: Andi Kleen [mailto:ak@muc.de] 
Sent: Friday, January 07, 2005 4:34 PM
To: James Cleverdon
Cc: YhLu; Matt Domsch; linux-kernel@vger.kernel.org; discuss@x86-64.org;
suresh.b.siddha@intel.com
Subject: Re: 256 apic id for amd64

On Fri, Jan 07, 2005 at 04:26:57PM -0800, James Cleverdon wrote:
> Already done, although not dividing along AMD vs. Intel lines.  
> phys_pkg_id() indirects through the subarch table.  See 
> genapic_cluster.c and genapic_flat.c for details.
> 
> We may need a third subarch for AMD's Extended APIC mode boxes.

I'm not convinced we do. Things seem to work with BSP APIC-ID = 0.
Is there any real reason to not just require that?

> 
> Can you suggest some heuristics for detecting such a system and 
> discerning it from a clustered APIC box?  (Hopefully, without using MPS 
> or ACPI table ID string lookups.)

Early PCI scan would work in the worst case. All Opterons have a builtin
northbridge with a specific ID.  There is already other code that 
checks for these.

-Andi

^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2005-01-08  0:28 YhLu
  2005-01-08  0:26 ` James Cleverdon
  0 siblings, 1 reply; 52+ messages in thread
From: YhLu @ 2005-01-08  0:28 UTC (permalink / raw)
  To: jamesclv; +Cc: Andi Kleen, Matt Domsch, linux-kernel, discuss, suresh.b.siddha

Then sepertate that into init_amd and init_intel.

YH

-----Original Message-----
From: James Cleverdon [mailto:jamesclv@us.ibm.com] 
Sent: Friday, January 07, 2005 4:13 PM
To: YhLu
Cc: Andi Kleen; Matt Domsch; linux-kernel@vger.kernel.org;
discuss@x86-64.org; suresh.b.siddha@intel.com
Subject: Re: 256 apic id for amd64

Clustered APIC systems need the phys_proc_id to come from the APIC ID 
register.  Intel prefers that non-clustered systems get their 
phys_proc_id from the cpuid opcode.

So, using c->x86_apicid is unlikely to satisfy both parties.


On Friday 07 January 2005 04:04 pm, YhLu wrote:
> arch/x86_64/kernel/setup.c
>
> static void __init detect_ht(struct cpuinfo_x86 *c)
>
>                 phys_proc_id[cpu] = phys_pkg_id(index_msb);
>
> --->
> 			  Phy_proc_id[cpu] = cpu_to_node[cpu];
> Or
>    	      // that is initial apicid
>           phys_proc_id[cpu] = c->x86_apicid >>
> hweight32(c->x86_num_cores - 1);
>
> YH
>
> -----Original Message-----
> From: Andi Kleen [mailto:ak@muc.de]
> Sent: Friday, January 07, 2005 2:18 PM
> To: YhLu
> Cc: Matt Domsch; linux-kernel@vger.kernel.org; discuss@x86-64.org;
> jamesclv@us.ibm.com; suresh.b.siddha@intel.com
> Subject: Re: 256 apic id for amd64
>
> On Fri, Jan 07, 2005 at 01:44:19PM -0800, YhLu wrote:
> > Can you consider to use c->x86_apicid to get phy_proc_id, that is
> > initial apicid.?
>
> Where?
>
> -Andi

-- 
James Cleverdon
IBM LTC (xSeries Linux Solutions)
{jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot comm

^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2005-01-08  0:04 YhLu
  2005-01-08  0:12 ` James Cleverdon
  0 siblings, 1 reply; 52+ messages in thread
From: YhLu @ 2005-01-08  0:04 UTC (permalink / raw)
  To: Andi Kleen; +Cc: Matt Domsch, linux-kernel, discuss, jamesclv, suresh.b.siddha

arch/x86_64/kernel/setup.c

static void __init detect_ht(struct cpuinfo_x86 *c)

                phys_proc_id[cpu] = phys_pkg_id(index_msb);

--->
			  Phy_proc_id[cpu] = cpu_to_node[cpu];
Or
   	      // that is initial apicid
          phys_proc_id[cpu] = c->x86_apicid >> hweight32(c->x86_num_cores -
1);

YH

-----Original Message-----
From: Andi Kleen [mailto:ak@muc.de] 
Sent: Friday, January 07, 2005 2:18 PM
To: YhLu
Cc: Matt Domsch; linux-kernel@vger.kernel.org; discuss@x86-64.org;
jamesclv@us.ibm.com; suresh.b.siddha@intel.com
Subject: Re: 256 apic id for amd64

On Fri, Jan 07, 2005 at 01:44:19PM -0800, YhLu wrote:
> Can you consider to use c->x86_apicid to get phy_proc_id, that is initial
> apicid.?

Where? 

-Andi

^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2005-01-07 21:44 YhLu
  2005-01-07 22:18 ` Andi Kleen
  0 siblings, 1 reply; 52+ messages in thread
From: YhLu @ 2005-01-07 21:44 UTC (permalink / raw)
  To: Andi Kleen; +Cc: Matt Domsch, linux-kernel, discuss, jamesclv, suresh.b.siddha

Can you consider to use c->x86_apicid to get phy_proc_id, that is initial
apicid.?

YH


4407.29 BogoMIPS (lpj=2203648)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 7 -> Node 3
phy_proc_id[0] = 0
phy_proc_id[1] = 0
phy_proc_id[2] = 9
phy_proc_id[3] = 9
phy_proc_id[4] = 10
phy_proc_id[5] = 10
phy_proc_id[6] = 11
phy_proc_id[7] = 11
CPU: Physical Processor ID: 11
 stepping 00
Total of 8 processors activated (35209.21 BogoMIPS).

-----Original Message-----
From: Andi Kleen [mailto:ak@muc.de] 
Sent: Friday, January 07, 2005 1:12 PM
To: YhLu
Cc: Matt Domsch; linux-kernel@vger.kernel.org; discuss@x86-64.org;
jamesclv@us.ibm.com; suresh.b.siddha@intel.com
Subject: Re: 256 apic id for amd64

On Fri, Jan 07, 2005 at 01:14:24PM -0800, YhLu wrote:
> After keep the bsp using 0, the jiffies works well. Werid?

Probably a bug somewhere.  But since BSP should be always 
0 I'm not sure it is worth tracking down.

-Andi

^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2005-01-07 21:14 YhLu
  2005-01-07 21:12 ` Andi Kleen
  0 siblings, 1 reply; 52+ messages in thread
From: YhLu @ 2005-01-07 21:14 UTC (permalink / raw)
  To: Andi Kleen; +Cc: Matt Domsch, linux-kernel, discuss, jamesclv, suresh.b.siddha

After keep the bsp using 0, the jiffies works well. Werid?

YH

-----Original Message-----
From: Andi Kleen [mailto:ak@muc.de] 
Sent: Friday, January 07, 2005 11:41 AM
To: YhLu
Cc: Matt Domsch; linux-kernel@vger.kernel.org; discuss@x86-64.org;
jamesclv@us.ibm.com; suresh.b.siddha@intel.com
Subject: Re: 256 apic id for amd64

On Fri, Jan 07, 2005 at 11:43:52AM -0800, YhLu wrote:
> Amd 8111 and 8131 only have 4 bit for apcid. So it only can use 0-15.

How broken. Ok. But I still don't like your patch. You should
give the BSP ID 0 and for the others it shouldn't matter anyways
if they use high APICIDs. 

-Andi

^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2005-01-07 19:43 YhLu
  2005-01-07 19:40 ` Andi Kleen
  0 siblings, 1 reply; 52+ messages in thread
From: YhLu @ 2005-01-07 19:43 UTC (permalink / raw)
  To: Andi Kleen; +Cc: Matt Domsch, linux-kernel, discuss, jamesclv, suresh.b.siddha

Amd 8111 and 8131 only have 4 bit for apcid. So it only can use 0-15.

YH

-----Original Message-----
From: Andi Kleen [mailto:ak@muc.de] 
Sent: Friday, January 07, 2005 11:29 AM
To: YhLu
Cc: Matt Domsch; linux-kernel@vger.kernel.org; discuss@x86-64.org;
jamesclv@us.ibm.com; suresh.b.siddha@intel.com
Subject: Re: 256 apic id for amd64

> Without subtract boot_cpu_id, phys_pkg_id will return 8.
> With that, It will return 0.

Normally this is set up that the CPUs come first and then the IO-APICs.
Why is this not possible with 8111 and 8131?

-Andi

^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2005-01-07 18:27 YhLu
  2005-01-07 19:29 ` Andi Kleen
  0 siblings, 1 reply; 52+ messages in thread
From: YhLu @ 2005-01-07 18:27 UTC (permalink / raw)
  To: YhLu, Andi Kleen
  Cc: Matt Domsch, linux-kernel, discuss, jamesclv, suresh.b.siddha

Also in arch/x86_64/kernel/setup.c  init_amd


        if (c->cpuid_level >= 0x80000008) {

---->

        n = cpuid_eax(0x80000000);

        if (n >= 0x80000008) {

for c->cupid_level is get cupid_eax(0) and it always =1

YH

-----Original Message-----
From: YhLu 
Sent: Friday, January 07, 2005 10:08 AM
To: 'Andi Kleen'
Cc: Matt Domsch; linux-kernel@vger.kernel.org; discuss@x86-64.org;
jamesclv@us.ibm.com; suresh.b.siddha@intel.com
Subject: RE: 256 apic id for amd64

Hard_smp_processor_id is CPU physical apicid.
Boot_cpu_id is boot_cpu_physical_apicid.

There is two configuration that we need to enable APIC_EXT_ID.
1. 8 way + dual core --- 8*2 + 2 +1 = 19, the cpu will use 0-15, and ioapic
need to use 16 above.
2. 4 way + 7 amd 8131 + 1 8111 --- 4+7*2+1=19

After enabling APIC_EXT_ID, the K8 can use 256 apicid. But the io apic
device (amd 8131 and 8111) still need to use 0-15. So We need to use 16
above for cpu apic id.

The BIOS or LinuxBIOS will set the apic id of CPU to 16 later. Or that's to
say
Apicid = initial apic id + apicid_offset.

When dual core is used and nb_cfg_54 is set, node 0 will use initial apicid
0/1 for core0 and core1. after setting apicid_offset. Apicid will be 16/17.

Without subtract boot_cpu_id, phys_pkg_id will return 8.
With that, It will return 0.

The c->x86_apicid is initial apic id and it is by cupid(0x1).

I guess for one core old cpu
nb_cfg_54 can not be set, and node 0 will use initial apidid 0. After
setting apicid_offset. Apicid will be 16.

Without subtract boot_cpu_id, phys_pkg_id will return 16.
With that, It will return 0.

YH

-----Original Message-----
From: Andi Kleen [mailto:ak@muc.de] 
Sent: Friday, January 07, 2005 4:25 AM
To: YhLu
Cc: Matt Domsch; linux-kernel@vger.kernel.org; discuss@x86-64.org;
jamesclv@us.ibm.com; suresh.b.siddha@intel.com
Subject: Re: 256 apic id for amd64

On Thu, Jan 06, 2005 at 06:53:11PM -0800, YhLu wrote:
> static unsigned int phys_pkg_id(int index_msb)
> {
>         return hard_smp_processor_id() >> index_msb;
> }
> 
> In arch/x86_64/kernel/genapic_cluster.c
> 
> Should be changed to 
> 
> static unsigned int phys_pkg_id(int index_msb)
> {
>         /* physical apicid, so we need to substract offset */
>         return (hard_smp_processor_id() - boot_cpu_id) >> index_msb;
> }

Why? 

If you want a patch merged you need to supply some more explanation
please.

Also cc Suresh & James for comment.

-Andi

^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2005-01-07 18:19 YhLu
  2005-01-07 19:29 ` Andi Kleen
  0 siblings, 1 reply; 52+ messages in thread
From: YhLu @ 2005-01-07 18:19 UTC (permalink / raw)
  To: Andi Kleen; +Cc: Matt Domsch, linux-kernel, discuss, jamesclv, suresh.b.siddha

Hard_smp_processor_id is CPU physical apicid.
Boot_cpu_id is boot_cpu_physical_apicid.

There is two configuration that we need to enable APIC_EXT_ID.
1. 8 way + dual core --- 8*2 + 2 +1 = 19, the cpu will use 0-15, and ioapic
need to use 16 above.
2. 4 way + 7 amd 8131 + 1 8111 --- 4+7*2+1=19

After enabling APIC_EXT_ID, the K8 can use 256 apicid. But the io apic
device (amd 8131 and 8111) still need to use 0-15. So We need to use 16
above for cpu apic id.

The BIOS or LinuxBIOS will set the apic id of CPU to 16 later. Or that's to
say
Apicid = initial apic id + apicid_offset.

When dual core is used and nb_cfg_54 is set, node 0 will use initial apicid
0/1 for core0 and core1. after setting apicid_offset. Apicid will be 16/17.

Without subtract boot_cpu_id, phys_pkg_id will return 8.
With that, It will return 0.

The c->x86_apicid is initial apic id and it is by cupid(0x1).

I guess for one core old cpu
nb_cfg_54 can not be set, and node 0 will use initial apidid 0. After
setting apicid_offset. Apicid will be 16.

Without subtract boot_cpu_id, phys_pkg_id will return 16.
With that, It will return 0.

YH

-----Original Message-----
From: Andi Kleen [mailto:ak@muc.de] 
Sent: Friday, January 07, 2005 4:25 AM
To: YhLu
Cc: Matt Domsch; linux-kernel@vger.kernel.org; discuss@x86-64.org;
jamesclv@us.ibm.com; suresh.b.siddha@intel.com
Subject: Re: 256 apic id for amd64

On Thu, Jan 06, 2005 at 06:53:11PM -0800, YhLu wrote:
> static unsigned int phys_pkg_id(int index_msb)
> {
>         return hard_smp_processor_id() >> index_msb;
> }
> 
> In arch/x86_64/kernel/genapic_cluster.c
> 
> Should be changed to 
> 
> static unsigned int phys_pkg_id(int index_msb)
> {
>         /* physical apicid, so we need to substract offset */
>         return (hard_smp_processor_id() - boot_cpu_id) >> index_msb;
> }

Why? 

If you want a patch merged you need to supply some more explanation
please.

Also cc Suresh & James for comment.

-Andi

^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2005-01-07  1:06 YhLu
  2005-01-07 12:44 ` Andi Kleen
  0 siblings, 1 reply; 52+ messages in thread
From: YhLu @ 2005-01-07  1:06 UTC (permalink / raw)
  To: Andi Kleen, Matt Domsch; +Cc: linux-kernel, discuss

Andi,

I made the Opteron using apic id from 16 later in LinuxBIOS.

So leave 0-15 for ioapic.

The problem is that the kernel (caliberate_delay) doesn't like that.

If using lpj=2170880 as the command line for that, it works well.

What's the jiffies? The TSC is changing but it doesn't.

YH

Without offset:
Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
Memory: 1019456k/1048576k available (3011k kernel code, 0k reserved, 1310k
data, 544k init)
LYH calibrating 0 jiffies = 4294667563, now=2922366256
LYH calibrating 1 jiffies = 4294667568, now=2934372713
LYH calibrating 3 jiffies = 4294667600, now=3003581909
4341.76 BogoMIPS (lpj=2170880)
Mount-cache hash table entries: 256 (order: 0, 4096 bytes)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 0 -> Node 0
CPU: Physical Processor ID: 0

With apic id offset:
Console: colour dummy device 80x25
Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
Memory: 1019456k/1048576k available (3011k kernel code, 0k reserved, 1310k
data, 544k init)
LYH calibrating 0 jiffies = 4294667296, now=1383947209
LYH calibrating 1 jiffies = 4294667296, now=1395952717


Please refer the print in init/main.c

void __devinit calibrate_delay(void)
{
        unsigned long ticks, loopbit;
        int lps_precision = LPS_PREC;
 +       unsigned long now;
 +
 +       rdtscl(now);
 +
 +        printk("LYH calibrating 0 jiffies = %lu, now=%lu\r\n", jiffies,
now);


        if (preset_lpj) {
                loops_per_jiffy = preset_lpj;
                printk("Calibrating delay loop (skipped)... "
                        "%lu.%02lu BogoMIPS preset\n",
                        loops_per_jiffy/(500000/HZ),
                        (loops_per_jiffy/(5000/HZ)) % 100);
        } else {
                loops_per_jiffy = (1<<12);

 +               rdtscl(now);
 +               printk("LYH calibrating 1 jiffies = %lu, now=%lu\r\n",
jiffies, now);
                printk(KERN_DEBUG "Calibrating delay loop... ");
                while ((loops_per_jiffy <<= 1) != 0) {
                        /* wait for "start of" clock tick */
                        ticks = jiffies;
                        while (ticks == jiffies)
                                /* nothing */;
                        /* Go .. */
                        ticks = jiffies;
                        __delay(loops_per_jiffy);
                        ticks = jiffies - ticks;
                        if (ticks)
                                break;
                }

                /*
                 * Do a binary approximation to get loops_per_jiffy set to
                 * equal one clock (up to lps_precision bits)
                 */
  +              rdtscl(now);
  +              printk("LYH calibrating 2 jiffies = %lu, now=%lu\r\n",
jiffies, now);

                loops_per_jiffy >>= 1;
                loopbit = loops_per_jiffy;
                while (lps_precision-- && (loopbit >>= 1)) {
                        loops_per_jiffy |= loopbit;
                        ticks = jiffies;
                       while (ticks == jiffies)
                                /* nothing */;
                        ticks = jiffies;
                        __delay(loops_per_jiffy);
                        if (jiffies != ticks)   /* longer than 1 tick */
                                loops_per_jiffy &= ~loopbit;
                }

  +              rdtscl(now);
  +              printk("LYH calibrating 3 jiffies = %lu, now=%lu\r\n",
jiffies, now);

                /* Round the value and print it */
                printk("%lu.%02lu BogoMIPS (lpj=%lu)\n",
                        loops_per_jiffy/(500000/HZ),
                        (loops_per_jiffy/(5000/HZ)) % 100,
                        loops_per_jiffy);
        }

}

^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2004-12-30 23:19 YhLu
  0 siblings, 0 replies; 52+ messages in thread
From: YhLu @ 2004-12-30 23:19 UTC (permalink / raw)
  To: Andi Kleen; +Cc: linux-kernel, discuss, Matt_Domsch

2.6.9 has a good feature and you remove that in 2.6.10.

That is if the io apic is set to above 0xf, io_apic.c will find one below
0xf unused and update the table.

YH

-----Original Message-----
From: Andi Kleen [mailto:ak@muc.de] 
Sent: Thursday, December 30, 2004 10:45 AM
To: YhLu
Cc: linux-kernel@vger.kernel.org; discuss@x86-64.org; Matt_Domsch@dell.com
Subject: Re: 256 apic id for amd64

YhLu <YhLu@tyan.com> writes:

> Can someone who maintains the x86-64 io_apic.c look at my patch about 256
> apic id for amd64?

First in general if you want patches submitted look up the maintainer
in the MAINTAINERS file in the source tree and send it directly
to the appropiate person and mailing list.

Just curious - how many IO-APICs does your system have?

Then I don't like your patch very much, since it doesnt handle 
Intel systems. The best fix is probably to 

i386 also has a different (but Intel specific fix) - uses either
0xf or 0xff based on the APIC version. Just dropping it seems
better to me though. I suppose Matt (cc'ed) who apparently
wrote this code originally used it to work around some BIOS
bug, and at least we can hope for now that there are no 
EM64T boxes with that particular BIOS bug.

I will add this patch.

-Andi

Remove check that limited max number of IO-APIC to 8.

The original check was apparently to work around some old BIOS
bugs and we just assume x86-64 machines don't have this class of
problems.

Signed-off-by: Andi Kleen <ak@suse.de>

diff -u linux-2.6.10/arch/x86_64/kernel/io_apic.c-o
linux-2.6.10/arch/x86_64/kernel/io_apic.c
--- linux-2.6.10/arch/x86_64/kernel/io_apic.c-o	2004-12-24
22:34:45.000000000 +0100
+++ linux-2.6.10/arch/x86_64/kernel/io_apic.c	2004-12-30
19:41:08.000000000 +0100
@@ -1160,13 +1160,6 @@
 		
 		old_id = mp_ioapics[apic].mpc_apicid;
 
-		if (mp_ioapics[apic].mpc_apicid >= 0xf) {
-			apic_printk(APIC_QUIET,KERN_ERR "BIOS bug,
IO-APIC#%d ID is %d in the MPC table!...\n",
-				apic, mp_ioapics[apic].mpc_apicid);
-			apic_printk(APIC_QUIET,KERN_ERR "... fixing up to
%d. (tell your hw vendor)\n",
-				reg_00.bits.ID);
-			mp_ioapics[apic].mpc_apicid = reg_00.bits.ID;
-		}
 
 		printk(KERN_INFO "Using IO-APIC %d\n",
mp_ioapics[apic].mpc_apicid);
 

^ permalink raw reply	[flat|nested] 52+ messages in thread
* RE: 256 apic id for amd64
@ 2004-12-30 23:16 YhLu
  0 siblings, 0 replies; 52+ messages in thread
From: YhLu @ 2004-12-30 23:16 UTC (permalink / raw)
  To: Andi Kleen; +Cc: linux-kernel, discuss, Matt_Domsch, 'Stefan Reinauer'

For the 8 way dual core Opteron system.
The CPUs will use 16 APIC ID, so BIOS or LinuxBIOS will enable APIC_EXT_ID
to use 256 apic id. So the ioapic will begin from 16.....

Another case : one system has two much 8131 will need to enable it.
For example four way system with single core Opteron, will have 7 8131s and
1 8111. So apic num will be 4+7*2+1=19.

Thanks.

YH

-----Original Message-----
From: Andi Kleen [mailto:ak@muc.de] 
Sent: Thursday, December 30, 2004 10:45 AM
To: YhLu
Cc: linux-kernel@vger.kernel.org; discuss@x86-64.org; Matt_Domsch@dell.com
Subject: Re: 256 apic id for amd64

YhLu <YhLu@tyan.com> writes:

> Can someone who maintains the x86-64 io_apic.c look at my patch about 256
> apic id for amd64?

First in general if you want patches submitted look up the maintainer
in the MAINTAINERS file in the source tree and send it directly
to the appropiate person and mailing list.

Just curious - how many IO-APICs does your system have?

Then I don't like your patch very much, since it doesnt handle 
Intel systems. The best fix is probably to 

i386 also has a different (but Intel specific fix) - uses either
0xf or 0xff based on the APIC version. Just dropping it seems
better to me though. I suppose Matt (cc'ed) who apparently
wrote this code originally used it to work around some BIOS
bug, and at least we can hope for now that there are no 
EM64T boxes with that particular BIOS bug.

I will add this patch.

-Andi

Remove check that limited max number of IO-APIC to 8.

The original check was apparently to work around some old BIOS
bugs and we just assume x86-64 machines don't have this class of
problems.

Signed-off-by: Andi Kleen <ak@suse.de>

diff -u linux-2.6.10/arch/x86_64/kernel/io_apic.c-o
linux-2.6.10/arch/x86_64/kernel/io_apic.c
--- linux-2.6.10/arch/x86_64/kernel/io_apic.c-o	2004-12-24
22:34:45.000000000 +0100
+++ linux-2.6.10/arch/x86_64/kernel/io_apic.c	2004-12-30
19:41:08.000000000 +0100
@@ -1160,13 +1160,6 @@
 		
 		old_id = mp_ioapics[apic].mpc_apicid;
 
-		if (mp_ioapics[apic].mpc_apicid >= 0xf) {
-			apic_printk(APIC_QUIET,KERN_ERR "BIOS bug,
IO-APIC#%d ID is %d in the MPC table!...\n",
-				apic, mp_ioapics[apic].mpc_apicid);
-			apic_printk(APIC_QUIET,KERN_ERR "... fixing up to
%d. (tell your hw vendor)\n",
-				reg_00.bits.ID);
-			mp_ioapics[apic].mpc_apicid = reg_00.bits.ID;
-		}
 
 		printk(KERN_INFO "Using IO-APIC %d\n",
mp_ioapics[apic].mpc_apicid);
 

^ permalink raw reply	[flat|nested] 52+ messages in thread
* 256 apic id for amd64
@ 2004-12-29  4:43 YhLu
  2004-12-30 18:45 ` Andi Kleen
  0 siblings, 1 reply; 52+ messages in thread
From: YhLu @ 2004-12-29  4:43 UTC (permalink / raw)
  To: Linux Kernel Mailing List

[-- Attachment #1: Type: text/plain, Size: 98 bytes --]

Can someone who maintains the x86-64 io_apic.c look at my patch about 256
apic id for amd64?

YH


[-- Attachment #2: x86_64_ioapic.patch --]
[-- Type: application/octet-stream, Size: 1300 bytes --]

diff -uNr linux-2.6.10/arch/x86_64/kernel/io_apic.c linux-2.6.10.new.x86_64/arch/x86_64/kernel/io_apic.c
--- linux-2.6.10/arch/x86_64/kernel/io_apic.c	2004-12-24 13:34:45.000000000 -0800
+++ linux-2.6.10.new.x86_64/arch/x86_64/kernel/io_apic.c	2004-12-28 15:46:35.828076192 -0800
@@ -1148,6 +1148,19 @@
 	unsigned char old_id;
 	unsigned long flags;
 
+        unsigned int max_apic;
+        u32 vendor;
+
+        /* get the max apic */
+        vendor = read_pci_config(0, 0x18, 0, PCI_VENDOR_ID);
+        vendor &= 0xffff;
+        if(vendor == PCI_VENDOR_ID_AMD) { /* AMD */
+                max_apic = (((read_pci_config(0, 0x18, 0, 0x68)>>17) & 3) == 3) ? 0xff : 0xf;
+        }
+        else { /* intel:  how to find out if intel em64t support 256 apic id? */
+                max_apic = 0xf;
+        }
+
 	/*
 	 * Set the IOAPIC ID to the value stored in the MPC table.
 	 */
@@ -1160,7 +1173,7 @@
 		
 		old_id = mp_ioapics[apic].mpc_apicid;
 
-		if (mp_ioapics[apic].mpc_apicid >= 0xf) {
+		if (mp_ioapics[apic].mpc_apicid >= max_apic) {
 			apic_printk(APIC_QUIET,KERN_ERR "BIOS bug, IO-APIC#%d ID is %d in the MPC table!...\n",
 				apic, mp_ioapics[apic].mpc_apicid);
 			apic_printk(APIC_QUIET,KERN_ERR "... fixing up to %d. (tell your hw vendor)\n",

^ permalink raw reply	[flat|nested] 52+ messages in thread

end of thread, other threads:[~2005-01-11 19:00 UTC | newest]

Thread overview: 52+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2005-01-07  2:53 256 apic id for amd64 YhLu
2005-01-07 12:24 ` Andi Kleen
2005-01-08  1:30   ` James Cleverdon
  -- strict thread matches above, loose matches on Subject: below --
2005-01-11 19:11 YhLu
2005-01-11 19:04 YhLu
2005-01-10 20:48 YhLu
2005-01-10 20:46 YhLu
2005-01-11  5:36 ` Andi Kleen
2005-01-10 20:37 YhLu
2005-01-10 20:09 YhLu
2005-01-10 20:18 ` Andi Kleen
2005-01-10 19:41 YhLu
2005-01-10 19:43 ` Andi Kleen
2005-01-11  0:42 ` James Cleverdon
2005-01-11  3:28   ` Siddha, Suresh B
2005-01-11  4:42     ` Andi Kleen
2005-01-10 18:48 YhLu
2005-01-10 18:45 ` Andi Kleen
2005-01-10 18:44 Andi Kleen
2005-01-11  4:04 ` Siddha, Suresh B
2005-01-11  4:39   ` Andi Kleen
2005-01-11 17:50   ` Andi Kleen
2005-01-08  2:53 YhLu
2005-01-09 23:56 ` James Cleverdon
2005-01-08  2:37 Mikael Pettersson
2005-01-08 15:46 ` Andi Kleen
2005-01-08  1:50 YhLu
2005-01-08  0:50 YhLu
2005-01-08  0:42 ` Andi Kleen
2005-01-08  0:28 YhLu
2005-01-08  0:26 ` James Cleverdon
2005-01-08  0:34   ` Andi Kleen
2005-01-08  0:04 YhLu
2005-01-08  0:12 ` James Cleverdon
2005-01-07 21:44 YhLu
2005-01-07 22:18 ` Andi Kleen
2005-01-07 21:14 YhLu
2005-01-07 21:12 ` Andi Kleen
2005-01-07 19:43 YhLu
2005-01-07 19:40 ` Andi Kleen
2005-01-07 18:27 YhLu
2005-01-07 19:29 ` Andi Kleen
2005-01-07 18:19 YhLu
2005-01-07 19:29 ` Andi Kleen
2005-01-07  1:06 YhLu
2005-01-07 12:44 ` Andi Kleen
2004-12-30 23:19 YhLu
2004-12-30 23:16 YhLu
2004-12-29  4:43 YhLu
2004-12-30 18:45 ` Andi Kleen
2004-12-30 22:56   ` Matt Domsch
2004-12-30 23:26     ` Andi Kleen

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).