Crash with paravirt-ops 2.6.31.6 kernel

[William Pitcock <nenolod@dereferenced.org>]

I am seeing the following crash with a pvops domU kernel, only when used with a 32bit userland (the kernel is 64bit):

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Linux version 2.6.31.6 (root@tachikoma.dereferenced.org) (gcc version 4.3.4 (Debian 4.3.4-5) ) #2 SMP Mon Nov 16 20:55:31 PST 2009
[    0.000000] Command line: root=/dev/xvda1 ro 
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Centaur CentaurHauls
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 0000000018000000 (usable)
[    0.000000] DMI not present or invalid.
[    0.000000] last_pfn = 0x18000 max_arch_pfn = 0x400000000
[    0.000000] init_memory_mapping: 0000000000000000-0000000018000000
[    0.000000] RAMDISK: 015b7000 - 029d3000
[    0.000000] (7 early reservations) ==> bootmem [0000000000 - 0018000000]
[    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
[    0.000000]   #1 [0002a96000 - 0002aaf000]   XEN PAGETABLES ==> [0002a96000 - 0002aaf000]
[    0.000000]   #2 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
[    0.000000]   #3 [0001000000 - 0001596cbc]    TEXT DATA BSS ==> [0001000000 - 0001596cbc]
[    0.000000]   #4 [00015b7000 - 00029d3000]          RAMDISK ==> [00015b7000 - 00029d3000]
[    0.000000]   #5 [00029d3000 - 0002a96000]   XEN START INFO ==> [00029d3000 - 0002a96000]
[    0.000000]   #6 [0000100000 - 00001a6000]          PGTABLE ==> [0000100000 - 00001a6000]
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000000 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   0x00100000 -> 0x00100000
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[2] active PFN ranges
[    0.000000]     0: 0x00000000 -> 0x000000a0
[    0.000000]     0: 0x00000100 -> 0x00018000
[    0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs
[    0.000000] No local APIC present
[    0.000000] APIC: disable apic facility
[    0.000000] Allocating PCI resources starting at 18000000 (gap: 18000000:e8000000)
[    0.000000] NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:1 nr_node_ids:1
[    0.000000] PERCPU: Allocated 20 4k pages, static data 79904 bytes
[    0.000000] Xen: using vcpu_info placement
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 96695
[    0.000000] Kernel command line: root=/dev/xvda1 ro 
[    0.000000] PID hash table entries: 2048 (order: 11, 16384 bytes)
[    0.000000] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.000000] Initializing CPU#0
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Memory: 357960k/393216k available (2599k kernel code, 384k absent, 34208k reserved, 1703k data, 412k init)
[    0.000000] NR_IRQS:1280
[    0.000000] Detected 2399.998 MHz processor.
[    0.004000] Console: colour dummy device 80x25
[    0.004000] console [tty0] enabled
[    0.004000] console [hvc0] enabled
[    0.004000] installing Xen timer for CPU 0
[    0.004000] Calibrating delay loop (skipped), value calculated using timer frequency.. 4799.99 BogoMIPS (lpj=9599992)
[    0.004000] Security Framework initialized
[    0.004000] SELinux:  Disabled at boot.
[    0.004000] Mount-cache hash table entries: 256
[    0.004000] Initializing cgroup subsys ns
[    0.004000] Initializing cgroup subsys devices
[    0.004000] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[    0.004000] CPU: L2 Cache: 1024K (64 bytes/line)
[    0.004000] CPU: Physical Processor ID: 1
[    0.004000] CPU: Processor Core ID: 0
[    0.004000] Performance Counters: AMD PMU driver.
[    0.004000] ------------[ cut here ]------------
[    0.004000] WARNING: at arch/x86/kernel/apic/apic.c:247 native_apic_write_dummy+0x30/0x3d()
[    0.004000] Modules linked in:
[    0.004000] Pid: 0, comm: swapper Not tainted 2.6.31.6 #2
[    0.004000] Call Trace:
[    0.004000]  [<ffffffff81020137>] ? native_apic_write_dummy+0x30/0x3d
[    0.004000]  [<ffffffff81020137>] ? native_apic_write_dummy+0x30/0x3d
[    0.004000]  [<ffffffff8103801e>] ? warn_slowpath_common+0x77/0xa3
[    0.004000]  [<ffffffff81020137>] ? native_apic_write_dummy+0x30/0x3d
[    0.004000]  [<ffffffff81450e86>] ? init_hw_perf_counters+0x2fe/0x39e
[    0.004000]  [<ffffffff81281551>] ? identify_cpu+0x2ea/0x2f3
[    0.004000]  [<ffffffff81450b31>] ? identify_boot_cpu+0x15/0x3e
[    0.004000]  [<ffffffff81450b63>] ? check_bugs+0x9/0x2e
[    0.004000]  [<ffffffff81449487>] ? start_kernel+0x35d/0x373
[    0.004000] ---[ end trace 4eaa2a86a8e2da22 ]---
[    0.004000] ... version:                 0
[    0.004000] ... bit width:               48
[    0.004000] ... generic counters:        4
[    0.004000] ... value mask:              0000ffffffffffff
[    0.004000] ... max period:              00007fffffffffff
[    0.004000] ... fixed-purpose counters:  0
[    0.004000] ... counter mask:            000000000000000f
[    0.004000] SMP alternatives: switching to UP code
[    0.008736] Freeing SMP alternatives: 26k freed
[    0.008973] Brought up 1 CPUs
[    0.009378] Booting paravirtualized kernel on Xen
[    0.009387] Xen version: 3.2-1
[    0.009559] Grant table initialized
[    0.009633] NET: Registered protocol family 16
[    0.011252] PCI: Fatal: No config space access function found
[    0.013543] bio: create slab <bio-0> at 0
[    0.013667] ACPI: Interpreter disabled.
[    0.013979] xen_balloon: Initialising balloon driver.
[    0.020467] usbcore: registered new interface driver usbfs
[    0.020573] usbcore: registered new interface driver hub
[    0.020643] usbcore: registered new device driver usb
[    0.020874] PCI: System does not support PCI
[    0.020882] PCI: System does not support PCI
[    0.022819] pnp: PnP ACPI: disabled
[    0.023445] NET: Registered protocol family 2
[    0.023556] IP route cache hash table entries: 4096 (order: 3, 32768 bytes)
[    0.023846] TCP established hash table entries: 16384 (order: 6, 262144 bytes)
[    0.024000] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
[    0.024000] TCP: Hash tables configured (established 16384 bind 16384)
[    0.024000] TCP reno registered
[    0.024096] NET: Registered protocol family 1
[    0.024167] Trying to unpack rootfs image as initramfs...
[    0.057841] Freeing initrd memory: 20592k freed
[    0.074576] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.075124] audit: initializing netlink socket (disabled)
[    0.075151] type=2000 audit(1258484309.528:1): initialized
[    0.075443] VFS: Disk quotas dquot_6.5.2
[    0.075479] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.075541] msgmni has been set to 808
[    0.075715] alg: No test for stdrng (krng)
[    0.075789] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    0.075801] io scheduler noop registered
[    0.075808] io scheduler deadline registered (default)
[    0.076605] Event-channel device installed.
[    0.081749] Linux agpgart interface v0.103
[    0.081764] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.084567] brd: module loaded
[    0.135409] input: Macintosh mouse button emulation as /class/input/input0
[    0.135454] Initialising Xen virtual ethernet driver.
[    0.137324] PNP: No PS/2 controller found. Probing ports directly.
[    0.138149] i8042.c: No controller found.
[    0.138309] mice: PS/2 mouse device common for all mice
[    0.138470] rtc_cmos: probe of rtc_cmos failed with error -16
[    0.138600] cpuidle: using governor ladder
[    0.138613] No iBFT detected.
[    0.138945] TCP cubic registered
[    0.138954] NET: Registered protocol family 17
[    0.139121] registered taskstats version 1
[    0.313469] blkfront: xvda2: barriers enabled
[    0.323795] blkfront: xvda1: barriers enabled
[    0.336056] XENBUS: Device with no driver: device/console/0
[    0.336093] drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[    0.336184] Freeing unused kernel memory: 412k freed
[    0.336474] BFS CPU scheduler v0.310 by Con Kolivas.
Loading, please wait...
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... FATAL: Error inserting fan (/lib/modules/2.6.31.6/kernel/drivers/acpi/fan.ko): No such device
FATAL: Error inserting thermal (/lib/modules/2.6.31.6/kernel/drivers/acpi/thermal.ko): No such device
done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... [    0.929769] device-mapper: uevent: version 1.0.3
[    0.931930] device-mapper: ioctl: 4.15.0-ioctl (2009-04-01) initialised: dm-devel@redhat.com
done.
Begin: Running /scripts/local-premount ... done.
[    1.085990] kjournald starting.  Commit interval 5 seconds
[    1.086022] EXT3-fs: mounted filesystem with writeback data mode.
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
[    1.254927] init[1] general protection ip:f779042f sp:ff9b0340 error:0
[    1.255189] Kernel panic - not syncing: Attempted to kill init!
[    1.255202] Pid: 1, comm: init Tainted: G        W  2.6.31.6 #2
[    1.255210] Call Trace:
[    1.255226]  [<ffffffff8128526b>] ? panic+0x86/0x133
[    1.255237]  [<ffffffff810cc7bf>] ? mntput_no_expire+0x23/0xed
[    1.255247]  [<ffffffff81117a83>] ? cap_file_free_security+0x0/0x1
[    1.255258]  [<ffffffff8100d155>] ? xen_force_evtchn_callback+0x9/0xa
[    1.255267]  [<ffffffff8100d7c2>] ? check_events+0x12/0x20
[    1.255275]  [<ffffffff81117a83>] ? cap_file_free_security+0x0/0x1
[    1.255285]  [<ffffffff81287365>] ? _write_lock_irq+0x7/0x16
[    1.255295]  [<ffffffff81040e59>] ? exit_ptrace+0xa7/0x126
[    1.255304]  [<ffffffff8103af48>] ? do_exit+0x72/0x66b
[    1.255312]  [<ffffffff8103b5ad>] ? do_group_exit+0x6c/0x99
[    1.255321]  [<ffffffff8104513c>] ? get_signal_to_deliver+0x303/0x327
[    1.255332]  [<ffffffff8116a0e0>] ? dummycon_dummy+0x0/0x3
[    1.255341]  [<ffffffff812873b3>] ? _spin_unlock_irqrestore+0xc/0xd
[    1.255351]  [<ffffffff8100ee2a>] ? do_notify_resume+0x79/0x6d1
[    1.255360]  [<ffffffff812873b3>] ? _spin_unlock_irqrestore+0xc/0xd
[    1.255369]  [<ffffffff8100d7c2>] ? check_events+0x12/0x20
[    1.255378]  [<ffffffff8100d155>] ? xen_force_evtchn_callback+0x9/0xa
[    1.255387]  [<ffffffff8100d7c2>] ? check_events+0x12/0x20
[    1.255396]  [<ffffffff8116a0e0>] ? dummycon_dummy+0x0/0x3
[    1.255404]  [<ffffffff812873b3>] ? _spin_unlock_irqrestore+0xc/0xd
[    1.255414]  [<ffffffff8100d7af>] ? xen_restore_fl_direct_end+0x0/0x1
[    1.255423]  [<ffffffff812873b3>] ? _spin_unlock_irqrestore+0xc/0xd
[    1.255432]  [<ffffffff81044a57>] ? force_sig_info+0xd3/0xe1
[    1.255441]  [<ffffffff810103dc>] ? retint_signal+0x48/0x8c

I don't know why this is happening.  It does not happen on XenSource 2.6.18 kernel, or the Debian 2.6.26 kernel.

William