isolcpus weirdness

isolcpus weirdness

[Emmanuel Pacaud]

Hi,

When specifying isolcpus kernel parameters, isolated cpu is always the
same, not the one I asked for.

I'm working with a dual xeon machine, with linux kernel 2.6.15
(unpatched, from kernel.org), hyperthreading desactivated (number of cpu
= 2). I've tried to isolate cpu1, using isolcpus, but I always end with
cpu0 isolated, even with isolcpus=1. With invalid isolcpus argument
(cpuid > 1), no cpu is isolated, which is ok.

You'll find below content of /proc/version, /proc/cpuinfo, the output of
dmesg, and a snapshot of top when running 3 cpuburn processes.

What's wrong ?

Regards,

	Emmanuel.

----

cat /proc/version
Linux version 2.6.15 (pacaud@xxxx.fr) (gcc version 3.4.4 20050721 (Red
Hat 3.4.4-2)) #3 SMP Wed Feb 22 11:04:15 CET 2006

cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 4
model name      : Intel(R) Xeon(TM) CPU 3.20GHz
stepping        : 3
cpu MHz         : 3192.629
cache size      : 2048 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm
pni monitor ds_cpl cid cx16 xtpr
bogomips        : 6439.83

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 15
model           : 4
model name      : Intel(R) Xeon(TM) CPU 3.20GHz
stepping        : 3
cpu MHz         : 3192.629
cache size      : 2048 KB
physical id     : 3
siblings        : 1
core id         : 0
cpu cores       : 1
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm
pni monitor ds_cpl cid cx16 xtpr
bogomips        : 6384.67

----

7:(14:15) ~> dmesg
Linux version 2.6.15 (pacaud@lappc-r093.in2p3.fr) (gcc version 3.4.4
20050721 (Red Hat 3.4.4-2)) #3 SMP Wed Feb 22 11:04:15 CET 2006
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
 BIOS-e820: 0000000000100000 - 000000003ffc0000 (usable)
 BIOS-e820: 000000003ffc0000 - 000000003ffcfc00 (ACPI data)
 BIOS-e820: 000000003ffcfc00 - 000000003ffff000 (reserved)
 BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec90000 (reserved)
 BIOS-e820: 00000000fed00000 - 00000000fed00400 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
 BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved)
Warning only 896MB will be used.
Use a HIGHMEM enabled kernel.
896MB LOWMEM available.
found SMP MP-table at 000fe710
On node 0 totalpages: 229376
  DMA zone: 4096 pages, LIFO batch:0
  DMA32 zone: 0 pages, LIFO batch:0
  Normal zone: 225280 pages, LIFO batch:31
  HighMem zone: 0 pages, LIFO batch:0
DMI 2.3 present.
Intel MultiProcessor Specification v1.4
    Virtual Wire compatibility mode.
OEM ID: DELL     Product ID: PE 0183      APIC at: 0xFEE00000
Processor #0 15:4 APIC version 20
Processor #6 15:4 APIC version 20
I/O APIC #8 Version 32 at 0xFEC00000.
I/O APIC #9 Version 32 at 0xFEC80000.
I/O APIC #10 Version 32 at 0xFEC80800.
Enabling APIC mode:  Flat.  Using 3 I/O APICs
Processors: 2
Allocating PCI resources starting at 40000000 (gap: 3ffff000:a0001000)
Built 1 zonelists
Kernel command line: ro root=LABEL=/ rhgb noirqbalance isolcpus=1
mapped APIC to ffffd000 (fee00000)
mapped IOAPIC to ffffc000 (fec00000)
mapped IOAPIC to ffffb000 (fec80000)
mapped IOAPIC to ffffa000 (fec80800)
Initializing CPU#0
CPU 0 irqstacks, hard=c046b000 soft=c044b000
PID hash table entries: 4096 (order: 12, 65536 bytes)
Detected 3192.629 MHz processor.
Using tsc for high-res timesource
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 900868k/917504k available (2282k kernel code, 16044k reserved,
888k data, 176k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode...
Ok.
Calibrating delay using timer specific routine.. 6439.83 BogoMIPS
(lpj=12879668)Mount-cache hash table entries: 512
CPU: After generic identify, caps: bfebfbff 20100000 00000000 00000000
0000641d 00000000 00000000
CPU: After vendor identify, caps: bfebfbff 20100000 00000000 00000000
0000641d 00000000 00000000
monitor/mwait feature present.
using mwait in idle threads.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 0
CPU: After all inits, caps: bfebfbff 20100000 00000000 00000080 0000641d
00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU0: Intel P4/Xeon Extended MCE MSRs (24) available
CPU0: Thermal monitoring enabled
mtrr: v2.0 (20020519)
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... OK.
CPU0: Intel(R) Xeon(TM) CPU 3.20GHz stepping 03
Booting processor 1/6 eip 2000
CPU 1 irqstacks, hard=c046c000 soft=c044c000
Initializing CPU#1
Calibrating delay using timer specific routine.. 6384.67 BogoMIPS
(lpj=12769340)CPU: After generic identify, caps: bfebfbff 20100000
00000000 00000000 0000641d 00000000 00000000
CPU: After vendor identify, caps: bfebfbff 20100000 00000000 00000000
0000641d 00000000 00000000
monitor/mwait feature present.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 3
CPU: After all inits, caps: bfebfbff 20100000 00000000 00000080 0000641d
00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#1.
CPU1: Intel P4/Xeon Extended MCE MSRs (24) available
CPU1: Thermal monitoring enabled
CPU1: Intel(R) Xeon(TM) CPU 3.20GHz stepping 03
Total of 2 processors activated (12824.50 BogoMIPS).
ExtINT not setup in hardware but reported by MP table
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 apic1=0 pin1=2 apic2=0 pin2=0
checking TSC synchronization across 2 CPUs: passed.
Brought up 2 CPUs
checking if image is initramfs... it is
Freeing initrd memory: 329k freed
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xfc0be, last bus=6
PCI: Using configuration type 1
SCSI subsystem initialized
usbcore: registered new driver usbfs
usbcore: registered new driver hub
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
PCI quirk: region 0800-087f claimed by ICH4 ACPI/GPIO/TCO
PCI quirk: region 0880-08bf claimed by ICH4 GPIO
PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1
PCI: PXH quirk detected, disabling MSI for SHPC device
PCI: PXH quirk detected, disabling MSI for SHPC device
Boot video device is 0000:06:05.0
PCI: Transparent bridge - 0000:00:1e.0
PCI: Using IRQ router PIIX/ICH [8086/24d0] at 0000:00:1f.0
PCI->APIC IRQ transform: 0000:00:02.0[A] -> IRQ 129
PCI->APIC IRQ transform: 0000:00:04.0[A] -> IRQ 129
PCI->APIC IRQ transform: 0000:00:06.0[A] -> IRQ 129
PCI->APIC IRQ transform: 0000:00:1d.0[A] -> IRQ 129
PCI->APIC IRQ transform: 0000:00:1d.1[B] -> IRQ 153
PCI->APIC IRQ transform: 0000:00:1d.2[C] -> IRQ 145
PCI->APIC IRQ transform: 0000:00:1d.7[D] -> IRQ 185
PCI->APIC IRQ transform: 0000:00:1f.1[A] -> IRQ 145
PCI->APIC IRQ transform: 0000:00:1f.2[A] -> IRQ 145
PCI->APIC IRQ transform: 0000:03:07.0[A] -> IRQ 106
PCI->APIC IRQ transform: 0000:06:05.0[A] -> IRQ 137
PCI: Bridge: 0000:01:00.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:01:00.2
  IO window: e000-efff
  MEM window: fe900000-feafffff
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:02.0
  IO window: e000-efff
  MEM window: fe700000-feafffff
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:04.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:06.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:1e.0
  IO window: d000-dfff
  MEM window: fe500000-fe6fffff
  PREFETCH window: f0000000-f7ffffff
PCI: Setting latency timer of device 0000:00:02.0 to 64
PCI: Setting latency timer of device 0000:01:00.0 to 64
PCI: Setting latency timer of device 0000:01:00.2 to 64
PCI: Setting latency timer of device 0000:00:04.0 to 64
PCI: Setting latency timer of device 0000:00:06.0 to 64
PCI: Setting latency timer of device 0000:00:1e.0 to 64
IA-32 Microcode Update Driver: v1.14 <tigran@veritas.com>
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
Intel E7520/7320/7525 detected.<6>lp: driver loaded but no devices found
Real Time Clock Driver v1.12
Non-volatile memory driver v1.2
hw_random hardware driver 1.0.0 loaded
Linux agpgart interface v0.101 (c) Dave Jones
[drm] Initialized drm 1.0.0 20040925
[drm] Initialized radeon 1.19.0 20050911 on minor 0:
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
parport0: PC-style at 0x378 (0x778) [PCSPP,TRISTATE]
parport0: irq 7 detected
lp0: using parport0 (polling).
lp0: console ready
Floppy drive(s): fd0 is 1.44M
FDC 0 is a National Semiconductor PC87306
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
loop: loaded (max 8 devices)
Intel(R) PRO/1000 Network Driver - version 6.1.16-k2-NAPI
Copyright (c) 1999-2005 Intel Corporation.
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
Linux video capture interface: v1.00
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with
idebus=xx
Probing IDE interface ide0...
hda: GCR-8483B, ATAPI CD/DVD-ROM drive
Probing IDE interface ide1...
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
hda: ATAPI 48X CD-ROM drive, 128kB Cache
Uniform CD-ROM driver Revision: 3.20
PCI: Setting latency timer of device 0000:00:1d.7 to 64
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: debug port 1
PCI: cache line size of 128 is not supported by device 0000:00:1d.7
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:1d.7: irq 185, io mem 0xfeb00000
ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 6 ports detected
USB Universal Host Controller Interface driver v2.3
PCI: Setting latency timer of device 0000:00:1d.0 to 64
uhci_hcd 0000:00:1d.0: UHCI Host Controller
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:1d.0: irq 129, io base 0x0000cce0
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
PCI: Setting latency timer of device 0000:00:1d.1 to 64
uhci_hcd 0000:00:1d.1: UHCI Host Controller
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:1d.1: irq 153, io base 0x0000ccc0
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
PCI: Setting latency timer of device 0000:00:1d.2 to 64
uhci_hcd 0000:00:1d.2: UHCI Host Controller
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4
uhci_hcd 0000:00:1d.2: irq 145, io base 0x0000cca0
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
Initializing USB Mass Storage driver...
usbcore: registered new driver usb-storage
USB Mass Storage support registered.
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
mice: PS/2 mouse device common for all mice
oprofile: using NMI interrupt.
NET: Registered protocol family 2
input: AT Translated Set 2 keyboard as /class/input/input0
IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
TCP established hash table entries: 131072 (order: 9, 2621440 bytes)
TCP bind hash table entries: 65536 (order: 8, 1310720 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
TCP bic registered
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
NET: Registered protocol family 15
Using IPI Shortcut mode
Freeing unused kernel memory: 176k freed
ICH5: IDE controller at PCI slot 0000:00:1f.1
PCI: Enabling device 0000:00:1f.1 (0005 -> 0007)
ICH5: chipset revision 2
ICH5: not 100% native mode: will probe irqs later
ICH5: port 0x01f0 already claimed by ide0
ICH5: neither IDE port enabled (BIOS)
libata version 1.20 loaded.
ata_piix 0000:00:1f.2: version 1.05
PCI: Setting latency timer of device 0000:00:1f.2 to 64
ata1: SATA max UDMA/133 cmd 0xCC98 ctl 0xCC92 bmdma 0xCC60 irq 145
ata2: SATA max UDMA/133 cmd 0xCC80 ctl 0xCC7A bmdma 0xCC68 irq 145
logips2pp: Detected unknown logitech mouse model 1
ata1: dev 0 cfg 49:2f00 82:7c6b 83:7f09 84:4673 85:7c69 86:3c21 87:4663
88:207f
ata1: dev 0 ATA-7, max UDMA/133, 312500000 sectors: LBA48
ata1: dev 0 configured for UDMA/133
scsi0 : ata_piix
ata2: SATA port has no device.
scsi1 : ata_piix
  Vendor: ATA       Model: Maxtor 6L160M0    Rev: BACE
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sda: 312500000 512-byte hdwr sectors (160000 MB)
SCSI device sda: drive cache: write back
SCSI device sda: 312500000 512-byte hdwr sectors (160000 MB)
SCSI device sda: drive cache: write back
 sda: sda1 sda2 sda3
sd 0:0:0:0: Attached scsi disk sda
sd 0:0:0:0: Attached scsi generic sg0 type 0
input: PS/2 Logitech Mouse as /class/input/input1
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
EXT3 FS on sda2, internal journal
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Adding 2096472k swap on /dev/sda3.  Priority:-1 extents:1
across:2096472k
e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex



1:(14:15) ~> top

top - 14:16:09 up  2:49,  2 users,  load average: 1.54, 0.41, 0.13
Tasks:  71 total,   4 running,  67 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.0% us,  0.0% sy,  0.0% ni, 99.8% id,  0.0% wa,  0.2% hi,  0.0% si
Cpu1  : 97.7% us,  1.9% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.4% si
Mem:    901584k total,   169848k used,   731736k free,    14764k buffers
Swap:  2096472k total,        0k used,  2096472k free,   108476k cached
 Unknown command - try 'h' for help
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2826 pacaud    25   0    92   12    4 R 31.5  0.0   0:11.60 burnP6
 2827 pacaud    25   0    92    8    4 R 31.5  0.0   0:12.30 burnP6
 2828 pacaud    25   0    96   12    4 R 30.9  0.0   0:12.30 burnP6
 2742 pacaud    15   0  7268 1824 1240 S  3.4  0.2   0:00.48 sshd
 2769 pacaud    15   0 34068  12m 8148 S  2.7  1.4   0:01.00 gnome-terminal
    1 root      16   0  1688  576  492 S  0.0  0.1   0:00.23 init
    2 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0

Re: isolcpus weirdness

[Frederik Deweerdt]

On Wed, Feb 22, 2006 at 02:21:27PM +0100, Emmanuel Pacaud wrote:
> What's wrong ?
> 
Are you able to reproduce the same behaviour after disabling HT in
the kernel config? Also, could you post your .config?
Regards,
Frederik

Re: isolcpus weirdness

[Emmanuel Pacaud]

[-- Attachment #1: Type: text/plain, Size: 682 bytes --]

Le mercredi 22 février 2006 à 22:18 +0100, Frederik Deweerdt a écrit : 
> On Wed, Feb 22, 2006 at 02:21:27PM +0100, Emmanuel Pacaud wrote:
> > What's wrong ?
> > 
> Are you able to reproduce the same behaviour after disabling HT in
> the kernel config? 

I think HT is disabled in kernel config, since I only see 2 cpus. 

In fact, I've tried to enable HT, but did'nt succeed. HT is enabled in
BIOS, but I'm not sure about exact things I must set at kernel config
level for hyperthreading. I've tried to set/unset CONFIG_SCHED_SMT, but
that changes nothing (no hyperthreading, isolated cpu is always cpu0).

Here's attached my config file.

Thanks for your interrest,

Emmanuel.

[-- Attachment #2: config.gz --]
[-- Type: application/x-gzip, Size: 7447 bytes --]

Re: isolcpus weirdness

[Con Kolivas]

On Thursday 23 February 2006 20:55, Emmanuel Pacaud wrote:
> Le mercredi 22 février 2006 à 22:18 +0100, Frederik Deweerdt a écrit :
> > On Wed, Feb 22, 2006 at 02:21:27PM +0100, Emmanuel Pacaud wrote:
> > > What's wrong ?
> >
> > Are you able to reproduce the same behaviour after disabling HT in
> > the kernel config?
>
> I think HT is disabled in kernel config, since I only see 2 cpus.
>
> In fact, I've tried to enable HT, but did'nt succeed. HT is enabled in
> BIOS, but I'm not sure about exact things I must set at kernel config
> level for hyperthreading. I've tried to set/unset CONFIG_SCHED_SMT, but
> that changes nothing (no hyperthreading, isolated cpu is always cpu0).
>
> Here's attached my config file.

CONFIG_ACPI is not set

You need ACPI to enumerate hyperthread siblings. That's why HT never gets 
enabled for you.

Cheers,
Con

Re: isolcpus weirdness

[Emmanuel Pacaud]

[-- Attachment #1: Type: text/plain, Size: 1624 bytes --]

Le jeudi 23 février 2006 à 21:00 +1100, Con Kolivas a écrit :
> On Thursday 23 February 2006 20:55, Emmanuel Pacaud wrote:
> > Le mercredi 22 février 2006 à 22:18 +0100, Frederik Deweerdt a écrit :
> > > On Wed, Feb 22, 2006 at 02:21:27PM +0100, Emmanuel Pacaud wrote:
> > > > What's wrong ?
> > >
> > > Are you able to reproduce the same behaviour after disabling HT in
> > > the kernel config?
> >
> > I think HT is disabled in kernel config, since I only see 2 cpus.
> >
> > In fact, I've tried to enable HT, but did'nt succeed. HT is enabled in
> > BIOS, but I'm not sure about exact things I must set at kernel config
> > level for hyperthreading. I've tried to set/unset CONFIG_SCHED_SMT, but
> > that changes nothing (no hyperthreading, isolated cpu is always cpu0).
> >
> > Here's attached my config file.
> 
> CONFIG_ACPI is not set
> 
> You need ACPI to enumerate hyperthread siblings. That's why HT never gets 
> enabled for you.

Following your advice, I've set CONFIG_ACPI, and now, isolcpus works
fine, when hyperthreading is activated.

But, I'm trying to build a real time system where hyperthreading is not
desired. So, I've tried to disable HT in BIOS, and the result is the
same as when ACPI is not enabled in kernel config: isolated cpu is
always cpu0.

On the same machine, which is originally a Scientific Linux 4, I've
tried with distribution supplied kernel (a customized 2.6.9 with smp/HT
enabled): isolcpus works fine, even with HT disabled at BIOS level.

It seems there's a regression between 2.6.9 and 2.6.15. 

I've attached config file of scientific linux 4.

Regards,

	Emmanuel.


[-- Attachment #2: config-2.6.9-5.0.5.ELsmp.gz --]
[-- Type: application/x-gzip, Size: 12640 bytes --]

Re: isolcpus weirdness

[Con Kolivas]

On Thursday 23 February 2006 21:51, Emmanuel Pacaud wrote:
> Le jeudi 23 février 2006 à 21:00 +1100, Con Kolivas a écrit :
> > On Thursday 23 February 2006 20:55, Emmanuel Pacaud wrote:
> > > Le mercredi 22 février 2006 à 22:18 +0100, Frederik Deweerdt a 
écrit :
> > > > On Wed, Feb 22, 2006 at 02:21:27PM +0100, Emmanuel Pacaud wrote:
> > > > > What's wrong ?
> > > >
> > > > Are you able to reproduce the same behaviour after disabling HT in
> > > > the kernel config?
> > >
> > > I think HT is disabled in kernel config, since I only see 2 cpus.
> > >
> > > In fact, I've tried to enable HT, but did'nt succeed. HT is enabled in
> > > BIOS, but I'm not sure about exact things I must set at kernel config
> > > level for hyperthreading. I've tried to set/unset CONFIG_SCHED_SMT, but
> > > that changes nothing (no hyperthreading, isolated cpu is always cpu0).
> > >
> > > Here's attached my config file.
> >
> > CONFIG_ACPI is not set
> >
> > You need ACPI to enumerate hyperthread siblings. That's why HT never gets
> > enabled for you.
>
> Following your advice, I've set CONFIG_ACPI, and now, isolcpus works
> fine, when hyperthreading is activated.
>
> But, I'm trying to build a real time system where hyperthreading is not
> desired. So, I've tried to disable HT in BIOS, and the result is the
> same as when ACPI is not enabled in kernel config: isolated cpu is
> always cpu0.
>
> On the same machine, which is originally a Scientific Linux 4, I've
> tried with distribution supplied kernel (a customized 2.6.9 with smp/HT
> enabled): isolcpus works fine, even with HT disabled at BIOS level.

Sorry I was only explaining why hyperthreading wasn't working with your 
config. I don't know anything about your problem specifically.

Cheers,
Con

Re: isolcpus weirdness

[Nick Piggin]

Emmanuel Pacaud wrote:
> Hi,
> 
> When specifying isolcpus kernel parameters, isolated cpu is always the
> same, not the one I asked for.
> 
> I'm working with a dual xeon machine, with linux kernel 2.6.15
> (unpatched, from kernel.org), hyperthreading desactivated (number of cpu
> = 2). I've tried to isolate cpu1, using isolcpus, but I always end with
> cpu0 isolated, even with isolcpus=1. With invalid isolcpus argument
> (cpuid > 1), no cpu is isolated, which is ok.
> 
> You'll find below content of /proc/version, /proc/cpuinfo, the output of
> dmesg, and a snapshot of top when running 3 cpuburn processes.
> 
> What's wrong ?
> 

If you have 2 CPUs, and "isolate" one of them, the other is isolated
from it. Ie. there is no difference between isolating one or the other,
the net result is that they are isolated from each other.

This means that the scheduler will never automatically move a task
running on an isolated CPU to another CPU. You have to move all the
tasks manually, for example by using cpu affinity.

-- 
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 

Re: isolcpus weirdness

[Emmanuel Pacaud]

Le jeudi 23 février 2006 à 23:21 +1100, Nick Piggin a écrit :
> Emmanuel Pacaud wrote:
> > Hi,
> > 
> > When specifying isolcpus kernel parameters, isolated cpu is always the
> > same, not the one I asked for.
..
> > 
> > What's wrong ?
> > 
> 
> If you have 2 CPUs, and "isolate" one of them, the other is isolated
> from it. Ie. there is no difference between isolating one or the other,
> the net result is that they are isolated from each other.
> 

>From kernel-parameters.txt:

+   isolcpus=   [KNL,SMP] Isolate CPUs from the general scheduler.
+         Format: <cpu number>, ..., <cpu number>
+         This option can be used to specify one or more CPUs
+         to isolate from the general SMP balancing and scheduling
+         algorithms. The only way to move a process onto or off
+         an "isolated" CPU is via the CPU affinity syscalls.
+
+         This option is the preferred way to isolate CPUs. The
+         alternative - manually setting the CPU mask of all tasks
+         in the system can cause problems and suboptimal load
+         balancer performance.

There's a difference between isolated cpus and other cpus: by default,
there's almost no activity on isolated ones. That's what I want to be
able to do.

	Emmanuel.

Re: isolcpus weirdness

[Nick Piggin]

Emmanuel Pacaud wrote:
> Le jeudi 23 février 2006 à 23:21 +1100, Nick Piggin a écrit :
> 
>>Emmanuel Pacaud wrote:
>>
>>>Hi,
>>>
>>>When specifying isolcpus kernel parameters, isolated cpu is always the
>>>same, not the one I asked for.
> 
> ..
> 
>>>What's wrong ?
>>>
>>
>>If you have 2 CPUs, and "isolate" one of them, the other is isolated
>>from it. Ie. there is no difference between isolating one or the other,
>>the net result is that they are isolated from each other.
>>
> 
> 
>>From kernel-parameters.txt:
> 
> +   isolcpus=   [KNL,SMP] Isolate CPUs from the general scheduler.
> +         Format: <cpu number>, ..., <cpu number>
> +         This option can be used to specify one or more CPUs
> +         to isolate from the general SMP balancing and scheduling
> +         algorithms. The only way to move a process onto or off
> +         an "isolated" CPU is via the CPU affinity syscalls.
> +
> +         This option is the preferred way to isolate CPUs. The
> +         alternative - manually setting the CPU mask of all tasks
> +         in the system can cause problems and suboptimal load
> +         balancer performance.
> 
> There's a difference between isolated cpus and other cpus: by default,
> there's almost no activity on isolated ones. That's what I want to be
> able to do.
> 

Nothing in kernel-parameters.txt says there will be almost no activity
on them. What you see is what you get, AFAIKS. It would seem strange
to me if isolating one CPU in a 2 CPU machine behaved differently from
isolating the other.

Isolating is really in the context of isolating it from scheduler
balancing, rather than suggesting no tasks will run on there. One way
you could make this do what you want is to bind the init process to
your non-isolated-CPU at boot. This is a policy decision that can be
done in userspace though.

-- 
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 

Re: isolcpus weirdness

[Emmanuel Pacaud]

Le vendredi 24 février 2006 à 00:30 +1100, Nick Piggin a écrit :
> Emmanuel Pacaud wrote: 
> > There's a difference between isolated cpus and other cpus: by default,
> > there's almost no activity on isolated ones. That's what I want to be
> > able to do.
> > 
> 
> Nothing in kernel-parameters.txt says there will be almost no activity
> on them. 

isolcpus=   [KNL,SMP] Isolate CPUs from the general scheduler.
+         Format: <cpu number>, ..., <cpu number>
+         This option can be used to specify one or more CPUs
+         to isolate from the general SMP balancing and scheduling
+         algorithms.

If a cpu is isolated from general SMP balancing and scheduling algorithm
(I know this isolation is not complete), there will be no activity on
it, no ? Unless one explicitely move one process on this cpu.

At least, that's what I've seen with my 2.6.15 vanilla kernel with
hyperthreading activated, or with distribution supplied kernel
(Scientific linux 4).  With these kernels, with isolcpus=n, there's
almost no activity on cpun.

My problem is I'm not able to obtain this behaviour with a kernel.org
2.6.15 kernel, when HT is disabled, either in BIOS, kernel config or
acpi=off parameter.


	Emmanuel.

(FWIW, I'm working on a RTAI setup. Use of isolcpus in the context of a
realtime setup is explained in their ISOLCPUS document:

http://cvs.gna.org/cvsweb/vulcano/README.ISOLCPUS?rev=1.6;cvsroot=rtai
)

Re: isolcpus weirdness

[Nick Piggin]

Emmanuel Pacaud wrote:
> Le vendredi 24 février 2006 à 00:30 +1100, Nick Piggin a écrit :
> 
>>Emmanuel Pacaud wrote: 
>>
>>>There's a difference between isolated cpus and other cpus: by default,
>>>there's almost no activity on isolated ones. That's what I want to be
>>>able to do.
>>>
>>
>>Nothing in kernel-parameters.txt says there will be almost no activity
>>on them. 
> 
> 
> isolcpus=   [KNL,SMP] Isolate CPUs from the general scheduler.
> +         Format: <cpu number>, ..., <cpu number>
> +         This option can be used to specify one or more CPUs
> +         to isolate from the general SMP balancing and scheduling
> +         algorithms.
> 
> If a cpu is isolated from general SMP balancing and scheduling algorithm
> (I know this isolation is not complete), there will be no activity on
> it, no ? Unless one explicitely move one process on this cpu.
> 

Well to me it says that SMP balancing and scheduling algorithms
will not consider tasks on this CPU. Ie. tasks on this CPU will
not be moved to another, or vice versa.

So I don't consider it a bug, but I'm looking at things from a
very scheduler-centric point of view. Perhaps it wouldn't be
unreasonable to exclude init from isolated CPUs at bootup... I
wouldn't be against such a patch.

> At least, that's what I've seen with my 2.6.15 vanilla kernel with
> hyperthreading activated, or with distribution supplied kernel
> (Scientific linux 4).  With these kernels, with isolcpus=n, there's
> almost no activity on cpun.
> 
> My problem is I'm not able to obtain this behaviour with a kernel.org
> 2.6.15 kernel, when HT is disabled, either in BIOS, kernel config or
> acpi=off parameter.
> 

I'm not sure how that would happen, if anything I think it would be
an implementation quirk that you should not rely on.

> 
> 	Emmanuel.
> 
> (FWIW, I'm working on a RTAI setup. Use of isolcpus in the context of a
> realtime setup is explained in their ISOLCPUS document:
> 
> http://cvs.gna.org/cvsweb/vulcano/README.ISOLCPUS?rev=1.6;cvsroot=rtai
> )
> 
> 

Very cool!

-- 
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 

Re: isolcpus weirdness

[Emmanuel Pacaud]

Le vendredi 24 février 2006 à 17:02 +1100, Nick Piggin a écrit :
> So I don't consider it a bug, but I'm looking at things from a
> very scheduler-centric point of view. Perhaps it wouldn't be
> unreasonable to exclude init from isolated CPUs at bootup... I
> wouldn't be against such a patch.

Ok. As you may have read in RTAI document, the point of cpu isolation in
a real time system is to make sure nothing apart real time tasks run on
isolated cpus. So if on a RTAI patched kernel I find the linux task
running on one of them, I guess it's a RTAI issue.

Thanks for your answers.

	Regards,

		Emmanuel.