Panic on boot on Sun Blade 6270

Panic on boot on Sun Blade 6270

[Dan Gora]

[-- Attachment #1: Type: text/plain, Size: 274 bytes --]

Hi All,

I'm getting the attached panic in the hypervisor on a Sun 6270 running
xen-testing.hg changeset 19913:6063c16aeeaa.  I can run xen-3.3.1 from
the standard SLES 11 distribution (which says that it's changeset
18546 but it has many patches).  Any ideas?

thanks,
dan

[-- Attachment #2: xenpanic.txt --]
[-- Type: text/plain, Size: 14026 bytes --]

  Booting 'Xen-3.4.3-rc1.dan -- SUSE Linux Enterprise Server 11 - 2.6.27.19-5 (
xen).dan'

                                                                                root (hd0,0)
 Filesystem type is ext2fs, partition type 0x83
kernel /xen-3.4.3-rc4-pre.gz crashkernel=128M@16M iommu=1 loglvl=all guest_logl
vl=all console=vga,com1 com1=auto
                                                                                   [Multiboot-elf, <0x100000:0x13e790:0x68870>                                                                                , shtab=0x2a7078, entry=0x100000]
module /vmlinuz-2.6.27.19-5-xen.dan root=/dev/dm-5 resume=/dev/dm-2 splash=sile
nt showopts console=ttyS0,115200 quiet
                                                                                                                                                                                                                                                                                                                                                                                                                   [Multiboot-module @ 0x2a8000, 0x98ce98 bytes]
module /initrd-2.6.27.19-5-xen.dan
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                GRUB Loading stage1.5.                                                                                                                                          GRUB loading, please wait...                                                       [Multiboot-module @ 0xc35000, 0x1e90800 bytes]

                                                                                                                                                                                                                                                 __  __            _____ _  _    _____             _  _                     
 \ \/ /___ _ __   |___ /| || |  |___ /    _ __ ___| || |     _ __  _ __ ___ 
  \  // _ \ '_ \    |_ \| || |_   |_ \ __| '__/ __| || |_ __| '_ \| '__/ _ \
  /  \  __/ | | |  ___) |__   _| ___) |__| | | (__|__   _|__| |_) | | |  __/
 /_/\_\___|_| |_| |____(_) |_|(_)____/   |_|  \___|  |_|    | .__/|_|  \___|
                                                            |_|             
(XEN) Xen version 3.4.3-rc4-pre (dg@local.lan) (gcc version 4.3.2 [gcc-4_3-branch revision 141291] (SUSE Linux) ) Thu Mar 11 14:04:35 BRT 2010
(XEN) Latest ChangeSet: Tue Mar 02 19:22:22 2010 +0000 19913:6063c16aeeaa
(XEN) Command line: crashkernel=128M@16M iommu=1 loglvl=all guest_loglvl=all console=vga,com1 com1=auto
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 1 seconds
(XEN)  EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN)  Found 2 MBR signatures
(XEN)  Found 2 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009a800 (usable)
(XEN)  000000000009a800 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 000000007f780000 (usable)
(XEN)  000000007f78e000 - 000000007f790000 type 9
(XEN)  000000007f790000 - 000000007f79e000 (ACPI data)
(XEN)  000000007f79e000 - 000000007f7d0000 (ACPI NVS)
(XEN)  000000007f7d0000 - 000000007f7e0000 (reserved)
(XEN)  000000007f7ec000 - 0000000080000000 (reserved)
(XEN)  00000000e0000000 - 00000000f0000000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ffb00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000480000000 (usable)
(XEN) Kdump: 128MB (131072kB) at 0x1000000
(XEN) System RAM: 16375MB (16768104kB)
(XEN) ACPI: RSDP 000FA050, 0024 (r2 SUN   )
(XEN) ACPI: XSDT 7F790100, 0094 (r1 SUN    Xxx70    20090519 MSFT       97)
(XEN) ACPI: FACP 7F790290, 00F4 (r4 SUN    Xxx70    20090519 MSFT       97)
(XEN) ACPI: DSDT 7F790580, 8128 (r2 SUN    Xxx70           0 INTL 20051117)
(XEN) ACPI: FACS 7F79E000, 0040
(XEN) ACPI: APIC 7F790390, 00D8 (r2 SUN    Xxx70    20090519 MSFT       97)
(XEN) ACPI: MCFG 7F7904C0, 003C (r1 SUN    Xxx70    20090519 MSFT       97)
(XEN) ACPI: SLIT 7F790500, 0030 (r1 SUN    Xxx70    20090519 MSFT       97)
(XEN) ACPI: SPMI 7F790530, 0041 (r5 SUN    Xxx70    20090519 MSFT       97)
(XEN) ACPI: OEMB 7F79E040, 007A (r1 SUN    Xxx70    20090519 MSFT       97)
(XEN) ACPI: HPET 7F79A580, 0038 (r1 SUN    Xxx70    20090519 MSFT       97)
(XEN) ACPI: DMAR 7F79E0C0, 0128 (r1 SUN    Xxx70           1 MSFT       97)
(XEN) ACPI: SRAT 7F79A5C0, 01D0 (r1 SUN    Xxx70           1 INTC        1)
(XEN) ACPI: SSDT 7F7A3740, 249F (r1  SUN   Xxx70          12 INTL 20051117)
(XEN) ACPI: EINJ 7F79A790, 0130 (r1 SUN    Xxx70    20090519 MSFT       97)
(XEN) ACPI: BERT 7F79A920, 0030 (r1 SUN    Xxx70    20090519 MSFT       97)
(XEN) ACPI: ERST 7F79A950, 01B0 (r1 SUN    Xxx70    20090519 MSFT       97)
(XEN) ACPI: HEST 7F79AB00, 00A8 (r1 SUN    Xxx70    20090519 MSFT       97)
(XEN) NUMA turned off
(XEN) Faking a node at 0000000000000000-0000000480000000
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000ff780
(XEN) DMI present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x808
(XEN) ACPI: ACPI SLEEP INFO: pm1x_cnt[804,0], pm1x_evt[800,0]
(XEN) ACPI:                  wakeup_vec[7f79e00c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) Processor #0 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) Processor #2 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) Processor #4 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
(XEN) Processor #6 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x10] enabled)
(XEN) Processor #16 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x12] enabled)
(XEN) Processor #18 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x14] enabled)
(XEN) Processor #20 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x16] enabled)
(XEN) Processor #22 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x09] lapic_id[0x01] enabled)
(XEN) Processor #1 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x03] enabled)
(XEN) Processor #3 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x05] enabled)
(XEN) Processor #5 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x07] enabled)
(XEN) Processor #7 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x11] enabled)
(XEN) Processor #17 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x13] enabled)
(XEN) Processor #19 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x15] enabled)
(XEN) Processor #21 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x10] lapic_id[0x17] enabled)
(XEN) Processor #23 7:10 APIC version 21
(XEN) Overriding APIC driver with bigsmp
(XEN) ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: IOAPIC (id[0x09] address[0xfec8a000] gsi_base[24])
(XEN) IOAPIC[1]: apic_id 9, version 32, address 0xfec8a000, GSI 24-47
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Phys.  Using 2 I/O APICs
(XEN) ACPI: HPET id: 0xffffffff base: 0xfed00000
(XEN) [VT-D]dmar.c:491: Host address width 40
(XEN) [VT-D]dmar.c:500: found ACPI_DMAR_DRHD
(XEN) [VT-D]dmar.c:349: dmaru->address = fbffe000
(XEN) [VT-D]dmar.c:306: found IOAPIC: bdf = f0:1f.7
(XEN) [VT-D]dmar.c:306: found IOAPIC: bdf = 0:13.0
(XEN) [VT-D]dmar.c:358: found INCLUDE_ALL
(XEN) [VT-D]dmar.c:504: found ACPI_DMAR_RMRR
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.0
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.1
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.2
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.7
(XEN) [VT-D]dmar.c:504: found ACPI_DMAR_RMRR
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.0
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.1
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.2
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.7
(XEN) [VT-D]dmar.c:508: found ACPI_DMAR_ATSR
(XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:3.0  sec = 7  sub = c
(XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:5.0  sec = d  sub = 12
(XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:7.0  sec = 13  sub = 18
(XEN) [VT-D]dmar.c:287: found bridge: bdf = 0:9.0  sec = 19  sub = 1e
(XEN) Intel VT-d DMAR tables have been parsed.
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Initializing CPU#0
(XEN) Detected 2926.110 MHz processor.
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 8192K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 0
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN) VMX: EPT is available.
(XEN) VMX: VPID is available.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging detected.
(XEN) Intel machine check reporting enabled on CPU#0.
(XEN) CPU0: Thermal monitoring enabled (TM1)
(XEN) CMCI: find owner on CPU0
(XEN) CMCI: CPU0 owner_map[6c], no_cmci_map[93]
(XEN) CPU0: Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz stepping 05
(XEN) ----[ Xen-3.4.3-rc4-pre  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff828c80183e31>] init_apic_ldr_phys+0x35/0x53
(XEN) RFLAGS: 0000000000010246   CONTEXT: hypervisor
(XEN) rax: 00000000ffffffff   rbx: 0000000000000020   rcx: 0000000000000400
(XEN) rdx: 0000000000000006   rsi: 0000000000000000   rdi: ffff828c8021ba80
(XEN) rbp: ffff828c80267d88   rsp: ffff828c80267d88   r8:  0000000000000001
(XEN) r9:  ffff828c8021bb80   r10: ffff828c8021ba80   r11: ffff828c8021b8e0
(XEN) r12: 0000000000000000   r13: 0000000000000015   r14: ffff828c8021b860
(XEN) r15: 0000000000000020   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 000000007f46c000   cr2: ffff828bfffff0e0
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff828c80267d88:
(XEN)    ffff828c80267dc8 ffff828c80139da1 0000000000010000 0000000000000020
(XEN)    0000000000000000 0000000000000000 ffff828c8021b860 0000000000000020
(XEN)    ffff828c80267e68 ffff828c80231d65 0000000000010000 ffff83047ffea000
(XEN)    ffff828c80267f18 0000002080267f28 0000000000000001 0000000000000004
(XEN)    ffff83047fff2fd0 ffff83047fff2f60 00000000000026f0 ffff828c8023e338
(XEN)    0000000000000000 ffff828c802a3ee0 ffff828c802a3e88 ffff828c80267f28
(XEN)    ffff83047ffea000 0000000000010000 0000000000000013 0000000000100000
(XEN)    ffff828c80267f18 ffff828c802313e4 0000000000000000 0000000000000000
(XEN)    0000000000087f30 0000000000000000 0000000000000000 ffff830000087fc0
(XEN)    ffff830000087f30 000000000281d698 0000000000000000 ffffffff00000000
(XEN)    0000000800000000 000000010000006e 0000000000000003 00000000000002f8
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000067e3c ffff828c801000b5
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 ffff83007c9e0000
(XEN) Xen call trace:
(XEN)    [<ffff828c80183e31>] init_apic_ldr_phys+0x35/0x53
(XEN)    [<ffff828c80139da1>] setup_local_APIC+0x65/0x3a4
(XEN)    [<ffff828c80231d65>] smp_prepare_cpus+0x408/0x853
(XEN)    [<ffff828c802313e4>] __start_xen+0x22fe/0x274a
(XEN)    
(XEN) Pagetable walk from ffff828bfffff0e0:
(XEN)  L4[0x105] = 000000007f472027 5555555555555555
(XEN)  L3[0x02f] = 000000047fffa063 5555555555555555
(XEN)  L2[0x1ff] = 000000047fff9063 5555555555555555 
(XEN)  L1[0x1ff] = 0000000000000000 ffffffffffffffff
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) FATAL PAGE FAULT
(XEN) [error_code=0002]
(XEN) Faulting linear address: ffff828bfffff0e0
(XEN) ****************************************
(XEN) 
(XEN) Reboot in five seconds...
(XEN) Resetting with ACPI MEMORY or I/O RESET_REG.

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Panic on boot on Sun Blade 6270

[Dan Gora]

Also setting noapic and acpi=off does not change the panic...

thanks,
dan

On Thu, Mar 11, 2010 at 6:40 PM, Dan Gora <dan.gora@gmail.com> wrote:
> Hi All,
>
> I'm getting the attached panic in the hypervisor on a Sun 6270 running
> xen-testing.hg changeset 19913:6063c16aeeaa.  I can run xen-3.3.1 from
> the standard SLES 11 distribution (which says that it's changeset
> 18546 but it has many patches).  Any ideas?
>
> thanks,
> dan

Re: Panic on boot on Sun Blade 6270

[Keir Fraser]

On 11/03/2010 21:40, "Dan Gora" <dan.gora@gmail.com> wrote:

> I'm getting the attached panic in the hypervisor on a Sun 6270 running
> xen-testing.hg changeset 19913:6063c16aeeaa.  I can run xen-3.3.1 from
> the standard SLES 11 distribution (which says that it's changeset
> 18546 but it has many patches).  Any ideas?

The system has quiet a few CPUs which puts Xen into 'bigsmp' mode, and maybe
there is a problem initialising that. Will have to try to reproduce this.

 -- Keir

RE: Re: Panic on boot on Sun Blade 6270

[Jiang, Yunhong]

Why it will fail on 0xffff828bfffff0e0 ? I try to build 19913 and seems it should be 0xffff828bffffe0e0.
If you can paste your objdump output for init_apic_ldr_phys, maybe it will be helpful.
BTW, if SLES 11 works for you, why not continue that?

--jyh


ffff828c801879d0 <init_apic_ldr_phys>:
...........
ffff828c801879f8:   b8 ff ff ff ff          mov    $0xffffffff,%eax
ffff828c801879fd:   a3 e0 e0 ff ff 8b 82    mov    %eax,0xffff828bffffe0e0
ffff828c80187a04:   ff ff
ffff828c80187a06:   a1 d0 e0 ff ff 8b 82    mov    0xffff828bffffe0d0,%eax
ffff828c80187a0d:   ff ff
ffff828c80187a0f:   25 ff ff ff 00          and    $0xffffff,%eax
ffff828c80187a14:   a3 d0 e0 ff ff 8b 82    mov    %eax,0xffff828bffffe0d0
...............

>-----Original Message-----
>From: xen-devel-bounces@lists.xensource.com
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Dan Gora
>Sent: Friday, March 12, 2010 5:55 AM
>To: xen-devel@lists.xensource.com
>Subject: [Xen-devel] Re: Panic on boot on Sun Blade 6270
>
>Also setting noapic and acpi=off does not change the panic...
>
>thanks,
>dan
>
>On Thu, Mar 11, 2010 at 6:40 PM, Dan Gora <dan.gora@gmail.com> wrote:
>> Hi All,
>>
>> I'm getting the attached panic in the hypervisor on a Sun 6270 running
>> xen-testing.hg changeset 19913:6063c16aeeaa.  I can run xen-3.3.1 from
>> the standard SLES 11 distribution (which says that it's changeset
>> 18546 but it has many patches).  Any ideas?
>>
>> thanks,
>> dan
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel

Re: Re: Panic on boot on Sun Blade 6270

[Dan Gora]

On Thu, Mar 11, 2010 at 10:35 PM, Jiang, Yunhong
<yunhong.jiang@intel.com> wrote:
> Why it will fail on 0xffff828bfffff0e0 ? I try to build 19913 and seems it should be 0xffff828bffffe0e0.
> If you can paste your objdump output for init_apic_ldr_phys, maybe it will be helpful.

Here is init_apic_ldr_phys from 'objdump -d xen-syms-3.4.3-rc4-pre':

ffff828c80183d48 <init_apic_ldr_flat>:
ffff828c80183d48:       55                      push   %rbp
ffff828c80183d49:       48 89 e5                mov    %rsp,%rbp
ffff828c80183d4c:       83 3d cd 73 09 00 00    cmpl
$0x0,0x973cd(%rip)        # ffff828c8021b120 <x2apic_enabled>
ffff828c80183d53:       74 4d                   je
ffff828c80183da2 <init_apic_ldr_flat+0x5a>
ffff828c80183d55:       ba 00 00 00 00          mov    $0x0,%edx
ffff828c80183d5a:       b8 ff ff ff ff          mov    $0xffffffff,%eax
ffff828c80183d5f:       b9 0e 08 00 00          mov    $0x80e,%ecx
ffff828c80183d64:       0f 30                   wrmsr
ffff828c80183d66:       b1 0d                   mov    $0xd,%cl
ffff828c80183d68:       0f 32                   rdmsr
ffff828c80183d6a:       48 89 c2                mov    %rax,%rdx
ffff828c80183d6d:       81 e2 ff ff ff 00       and    $0xffffff,%edx
ffff828c80183d73:       48 c7 c0 00 80 ff ff    mov    $0xffffffffffff8000,%rax
ffff828c80183d7a:       48 21 e0                and    %rsp,%rax
ffff828c80183d7d:       48 0d 28 7f 00 00       or     $0x7f28,%rax
ffff828c80183d83:       8b 88 c8 00 00 00       mov    0xc8(%rax),%ecx
ffff828c80183d89:       b8 00 00 00 01          mov    $0x1000000,%eax
ffff828c80183d8e:       48 d3 e0                shl    %cl,%rax
ffff828c80183d91:       48 09 d0                or     %rdx,%rax
ffff828c80183d94:       ba 00 00 00 00          mov    $0x0,%edx
ffff828c80183d99:       b9 0d 08 00 00          mov    $0x80d,%ecx
ffff828c80183d9e:       0f 30                   wrmsr
ffff828c80183da0:       eb 43                   jmp
ffff828c80183de5 <init_apic_ldr_flat+0x9d>
ffff828c80183da2:       b8 ff ff ff ff          mov    $0xffffffff,%eax
ffff828c80183da7:       a3 e0 f0 ff ff 8b 82    mov    %eax,0xffff828bfffff0e0
ffff828c80183dae:       ff ff
ffff828c80183db0:       48 be d0 f0 ff ff 8b    mov    $0xffff828bfffff0d0,%rsi
ffff828c80183db7:       82 ff ff
ffff828c80183dba:       8b 16                   mov    (%rsi),%edx
ffff828c80183dbc:       81 e2 ff ff ff 00       and    $0xffffff,%edx
ffff828c80183dc2:       48 c7 c0 00 80 ff ff    mov    $0xffffffffffff8000,%rax
ffff828c80183dc9:       48 21 e0                and    %rsp,%rax
ffff828c80183dcc:       48 0d 28 7f 00 00       or     $0x7f28,%rax
ffff828c80183dd2:       8b 88 c8 00 00 00       mov    0xc8(%rax),%ecx
ffff828c80183dd8:       b8 00 00 00 01          mov    $0x1000000,%eax
ffff828c80183ddd:       48 d3 e0                shl    %cl,%rax
ffff828c80183de0:       48 09 d0                or     %rdx,%rax
ffff828c80183de3:       89 06                   mov    %eax,(%rsi)
ffff828c80183de5:       c9                      leaveq
ffff828c80183de6:       c3                      retq

I don't quite understand why we have such different addresses.  I'm
thinking maybe I built it wrong or something.  I built it with debug
set to 'y' in Config.mk, then just did 'make dist', copied the
resultant 'dist' directory to the test machine and ran 'sh
./install.sh' on the target machine, and rebooted with:

title Xen-3.4.3-rc1.dan -- SUSE Linux Enterprise Server 11 -
2.6.27.19-5 (xen).dan
    root (hd0,0)
    kernel /xen-3.4.3-rc4-pre.gz crashkernel=128M@16M iommu=1
loglvl=all guest_loglvl=all console=vga,com1 com1=auto
    module /vmlinuz-2.6.27.19-5-xen.dan root=/dev/dm-5
resume=/dev/dm-2 splash=silent showopts console=ttyS0,115200 quiet
    module /initrd-2.6.27.19-5-xen.dan


> BTW, if SLES 11 works for you, why not continue that?

ugh.. long story.  It works in the sense that it boots and can run
VMs.  My problem is that I'm trying to get the board that our company
makes to work in an HVM guest properly..  *something* is messing with
my board's PCI bar registers and I cannot figure out what it is.  I
added printks to all the kernel functions which touch the BAR
registers and to the pciback driver as well, but neither of these is
occurring, so I figured the problem is in qemu, but to test that I
need to be able to compile it.  I thought that the easiest path there
would simply to compile the latest stuff, but it panicked early in the
hypervisor so I thought that I'd report it...

thanks,
dan

RE: Re: Panic on boot on Sun Blade 6270

[Jiang, Yunhong]

The only possibility I can see is changeset 18722 is missed from your build.
So you are really clone a CLEAN xen tree and do the build from scratch?

--jyh

>-----Original Message-----
>From: Dan Gora [mailto:dan.gora@gmail.com]
>Sent: Friday, March 12, 2010 12:01 PM
>To: Jiang, Yunhong
>Cc: xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel] Re: Panic on boot on Sun Blade 6270
>
>On Thu, Mar 11, 2010 at 10:35 PM, Jiang, Yunhong
><yunhong.jiang@intel.com> wrote:
>> Why it will fail on 0xffff828bfffff0e0 ? I try to build 19913 and seems it should be
>0xffff828bffffe0e0.
>> If you can paste your objdump output for init_apic_ldr_phys, maybe it will be
>helpful.
>
>Here is init_apic_ldr_phys from 'objdump -d xen-syms-3.4.3-rc4-pre':
>
>ffff828c80183d48 <init_apic_ldr_flat>:
>ffff828c80183d48:       55                      push   %rbp
>ffff828c80183d49:       48 89 e5                mov    %rsp,%rbp
>ffff828c80183d4c:       83 3d cd 73 09 00 00    cmpl
>$0x0,0x973cd(%rip)        # ffff828c8021b120 <x2apic_enabled>
>ffff828c80183d53:       74 4d                   je
>ffff828c80183da2 <init_apic_ldr_flat+0x5a>
>ffff828c80183d55:       ba 00 00 00 00          mov    $0x0,%edx
>ffff828c80183d5a:       b8 ff ff ff ff          mov    $0xffffffff,%eax
>ffff828c80183d5f:       b9 0e 08 00 00          mov    $0x80e,%ecx
>ffff828c80183d64:       0f 30                   wrmsr
>ffff828c80183d66:       b1 0d                   mov    $0xd,%cl
>ffff828c80183d68:       0f 32                   rdmsr
>ffff828c80183d6a:       48 89 c2                mov    %rax,%rdx
>ffff828c80183d6d:       81 e2 ff ff ff 00       and    $0xffffff,%edx
>ffff828c80183d73:       48 c7 c0 00 80 ff ff    mov    $0xffffffffffff8000,%rax
>ffff828c80183d7a:       48 21 e0                and    %rsp,%rax
>ffff828c80183d7d:       48 0d 28 7f 00 00       or     $0x7f28,%rax
>ffff828c80183d83:       8b 88 c8 00 00 00       mov    0xc8(%rax),%ecx
>ffff828c80183d89:       b8 00 00 00 01          mov    $0x1000000,%eax
>ffff828c80183d8e:       48 d3 e0                shl    %cl,%rax
>ffff828c80183d91:       48 09 d0                or     %rdx,%rax
>ffff828c80183d94:       ba 00 00 00 00          mov    $0x0,%edx
>ffff828c80183d99:       b9 0d 08 00 00          mov    $0x80d,%ecx
>ffff828c80183d9e:       0f 30                   wrmsr
>ffff828c80183da0:       eb 43                   jmp
>ffff828c80183de5 <init_apic_ldr_flat+0x9d>
>ffff828c80183da2:       b8 ff ff ff ff          mov    $0xffffffff,%eax
>ffff828c80183da7:       a3 e0 f0 ff ff 8b 82    mov    %eax,0xffff828bfffff0e0
>ffff828c80183dae:       ff ff
>ffff828c80183db0:       48 be d0 f0 ff ff 8b    mov    $0xffff828bfffff0d0,%rsi
>ffff828c80183db7:       82 ff ff
>ffff828c80183dba:       8b 16                   mov    (%rsi),%edx
>ffff828c80183dbc:       81 e2 ff ff ff 00       and    $0xffffff,%edx
>ffff828c80183dc2:       48 c7 c0 00 80 ff ff    mov    $0xffffffffffff8000,%rax
>ffff828c80183dc9:       48 21 e0                and    %rsp,%rax
>ffff828c80183dcc:       48 0d 28 7f 00 00       or     $0x7f28,%rax
>ffff828c80183dd2:       8b 88 c8 00 00 00       mov    0xc8(%rax),%ecx
>ffff828c80183dd8:       b8 00 00 00 01          mov    $0x1000000,%eax
>ffff828c80183ddd:       48 d3 e0                shl    %cl,%rax
>ffff828c80183de0:       48 09 d0                or     %rdx,%rax
>ffff828c80183de3:       89 06                   mov    %eax,(%rsi)
>ffff828c80183de5:       c9                      leaveq
>ffff828c80183de6:       c3                      retq
>
>I don't quite understand why we have such different addresses.  I'm
>thinking maybe I built it wrong or something.  I built it with debug
>set to 'y' in Config.mk, then just did 'make dist', copied the
>resultant 'dist' directory to the test machine and ran 'sh
>./install.sh' on the target machine, and rebooted with:
>
>title Xen-3.4.3-rc1.dan -- SUSE Linux Enterprise Server 11 -
>2.6.27.19-5 (xen).dan
>    root (hd0,0)
>    kernel /xen-3.4.3-rc4-pre.gz crashkernel=128M@16M iommu=1
>loglvl=all guest_loglvl=all console=vga,com1 com1=auto
>    module /vmlinuz-2.6.27.19-5-xen.dan root=/dev/dm-5
>resume=/dev/dm-2 splash=silent showopts console=ttyS0,115200 quiet
>    module /initrd-2.6.27.19-5-xen.dan
>
>
>> BTW, if SLES 11 works for you, why not continue that?
>
>ugh.. long story.  It works in the sense that it boots and can run
>VMs.  My problem is that I'm trying to get the board that our company
>makes to work in an HVM guest properly..  *something* is messing with
>my board's PCI bar registers and I cannot figure out what it is.  I
>added printks to all the kernel functions which touch the BAR
>registers and to the pciback driver as well, but neither of these is
>occurring, so I figured the problem is in qemu, but to test that I
>need to be able to compile it.  I thought that the easiest path there
>would simply to compile the latest stuff, but it panicked early in the
>hypervisor so I thought that I'd report it...
>
>thanks,
>dan

Re: Re: Panic on boot on Sun Blade 6270

[Keir Fraser]

On 12/03/2010 04:01, "Dan Gora" <dan.gora@gmail.com> wrote:

> On Thu, Mar 11, 2010 at 10:35 PM, Jiang, Yunhong
> <yunhong.jiang@intel.com> wrote:
>> Why it will fail on 0xffff828bfffff0e0 ? I try to build 19913 and seems it
>> should be 0xffff828bffffe0e0.
>> If you can paste your objdump output for init_apic_ldr_phys, maybe it will be
>> helpful.
> 
> Here is init_apic_ldr_phys from 'objdump -d xen-syms-3.4.3-rc4-pre':

Er..

> ffff828c80183d48 <init_apic_ldr_flat>:

...no it's not.

 -- Keir

Re: Re: Panic on boot on Sun Blade 6270

[Dan Gora]

On Fri, Mar 12, 2010 at 3:20 AM, Jiang, Yunhong <yunhong.jiang@intel.com> wrote:
> The only possibility I can see is changeset 18722 is missed from your build.
> So you are really clone a CLEAN xen tree and do the build from scratch?
>

yes... I cloned xen-testing-3.4.hg.  I'm not proficient enough with hg
to remove a single commit.

d

Re: Re: Panic on boot on Sun Blade 6270

[Dan Gora]

On Fri, Mar 12, 2010 at 4:13 AM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
>> Here is init_apic_ldr_phys from 'objdump -d xen-syms-3.4.3-rc4-pre':
>
> Er..
>
>> ffff828c80183d48 <init_apic_ldr_flat>:
>
> ...no it's not.

ok.. what is it then?  I'm pretty darn sure that's what I typed.

d

Re: Re: Panic on boot on Sun Blade 6270

[Dan Gora]

I guess that I can try and redo everything again.  I've been working
with xen a grand total of two weeks, so I'm a little confused about
the building process.  Do you have to somehow build xen against the
kernel that you are going to use or can you just use the default
2.6.18 kernel which gets downloaded automatically?  Like I said, I
just cloned the repo (xen-testing-3.4.hg) and ran 'make dist'.  Since
it's panicking so early in the hypervisor I didn't think that it could
possibly be due to some kernel difference.  Also, is it expected that
the default .hg tree xen will *just work* on a SLES distro by running
install.sh or do you need a bunch of SuSE specific patches?

thanks
dan


On Fri, Mar 12, 2010 at 10:37 AM, Dan Gora <dan.gora@gmail.com> wrote:
> On Fri, Mar 12, 2010 at 4:13 AM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
>>> Here is init_apic_ldr_phys from 'objdump -d xen-syms-3.4.3-rc4-pre':
>>
>> Er..
>>
>>> ffff828c80183d48 <init_apic_ldr_flat>:
>>
>> ...no it's not.
>
> ok.. what is it then?  I'm pretty darn sure that's what I typed.
>
> d
>



-- 
Dan Gora
Software Engineer

Adax, Inc.
Av Dona Maria Alves, 793-LJ 04
Ubatuba, SP
CEP 11680-000
Brasil

Tel: +55 (12) 3833-1021  (Brazil and outside of US)
    : +1 (510) 859-4801  (Inside of US) <= Note new number!
    : dan_gora (Skype)

email: dg@adax.com

Re: Re: Panic on boot on Sun Blade 6270

[Pasi Kärkkäinen]

On Fri, Mar 12, 2010 at 10:43:08AM -0300, Dan Gora wrote:
> I guess that I can try and redo everything again.  I've been working
> with xen a grand total of two weeks, so I'm a little confused about
> the building process.  Do you have to somehow build xen against the
> kernel that you are going to use or can you just use the default
> 2.6.18 kernel which gets downloaded automatically?
>

Xen hypervisor and tools can be built separately from the dom0 kernel.


> Like I said, I
> just cloned the repo (xen-testing-3.4.hg) and ran 'make dist'.  Since
> it's panicking so early in the hypervisor I didn't think that it could
> possibly be due to some kernel difference.
>

You could just grab the latest xen-3.4-testing.hg and do:

make xen
make tools
make stubdom

Now the compiled xen+tools+stubdom is in dist/ directory.

make install-xen
make install-tools
make install-stubdom

After that you could grab whatever dom0 kernel, compile and install it.

latest 2.6.18-xen.hg:
hg clone http://xenbits.xen.org/linux-2.6.18-xen.hg

Or instructions for 2.6.31 / 2.6.32 pv_ops dom0 here:
http://wiki.xensource.com/xenwiki/XenParavirtOps

> Also, is it expected that
> the default .hg tree xen will *just work* on a SLES distro by running
> install.sh or do you need a bunch of SuSE specific patches?
> 

The default hg tree should work.

-- Pasi

> thanks
> dan
> 
> 
> On Fri, Mar 12, 2010 at 10:37 AM, Dan Gora <dan.gora@gmail.com> wrote:
> > On Fri, Mar 12, 2010 at 4:13 AM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> >>> Here is init_apic_ldr_phys from 'objdump -d xen-syms-3.4.3-rc4-pre':
> >>
> >> Er..
> >>
> >>> ffff828c80183d48 <init_apic_ldr_flat>:
> >>
> >> ...no it's not.
> >
> > ok.. what is it then?  I'm pretty darn sure that's what I typed.
> >
> > d
> >
> 
> 
> 
> -- 
> Dan Gora
> Software Engineer
> 
> Adax, Inc.
> Av Dona Maria Alves, 793-LJ 04
> Ubatuba, SP
> CEP 11680-000
> Brasil
> 
> Tel: +55 (12) 3833-1021  (Brazil and outside of US)
>     : +1 (510) 859-4801  (Inside of US) <= Note new number!
>     : dan_gora (Skype)
> 
> email: dg@adax.com
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Re: Panic on boot on Sun Blade 6270

[Dan Gora]

So I tried it all over again from scratch with the xen-testing.hg tree
at commit  19924 and it doesn't panic any more.  It gets to booting
the kernel, then locks up when trying to bring the networking up, but
at least it doesn't have the same panic anymore.

Sorry about the false alarm. I probably gaffed something with hg on
the previous go round..

thanks
dan


On Fri, Mar 12, 2010 at 10:43 AM, Dan Gora <dan.gora@gmail.com> wrote:
> I guess that I can try and redo everything again.  I've been working
> with xen a grand total of two weeks, so I'm a little confused about
> the building process.  Do you have to somehow build xen against the
> kernel that you are going to use or can you just use the default
> 2.6.18 kernel which gets downloaded automatically?  Like I said, I
> just cloned the repo (xen-testing-3.4.hg) and ran 'make dist'.  Since
> it's panicking so early in the hypervisor I didn't think that it could
> possibly be due to some kernel difference.  Also, is it expected that
> the default .hg tree xen will *just work* on a SLES distro by running
> install.sh or do you need a bunch of SuSE specific patches?
>
> thanks
> dan
>
>
> On Fri, Mar 12, 2010 at 10:37 AM, Dan Gora <dan.gora@gmail.com> wrote:
>> On Fri, Mar 12, 2010 at 4:13 AM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
>>>> Here is init_apic_ldr_phys from 'objdump -d xen-syms-3.4.3-rc4-pre':
>>>
>>> Er..
>>>
>>>> ffff828c80183d48 <init_apic_ldr_flat>:
>>>
>>> ...no it's not.
>>
>> ok.. what is it then?  I'm pretty darn sure that's what I typed.
>>
>> d

Re: Re: Panic on boot on Sun Blade 6270

[Dan Gora]

On Fri, Mar 12, 2010 at 1:44 PM, Pasi Kärkkäinen <pasik@iki.fi> wrote:

>> Also, is it expected that
>> the default .hg tree xen will *just work* on a SLES distro by running
>> install.sh or do you need a bunch of SuSE specific patches?
>>
>
> The default hg tree should work.

Thanks for the tips Pasi.. I guess I did pretty much that, but I just
did a 'make dist' and built everything including the 2.6.18 kernel, I
moved everything to the test machine and ran 'sh ./install.sh' and
added a grub line to boot the new xen:

title Xen-3.4.3-rc1.dan -- SUSE Linux Enterprise Server 11 -
2.6.27.19-5 (xen).dan
   root (hd0,0)
   kernel /xen-3.4.3-rc4-pre.gz crashkernel=128M@16M iommu=1
loglvl=all guest_loglvl=all console=vga,com1 com1=auto
   module /vmlinuz-2.6.27.19-5-xen.dan root=/dev/dm-5
resume=/dev/dm-2 splash=silent showopts console=ttyS0,115200 quiet
   module /initrd-2.6.27.19-5-xen.dan

so it should use the new hypervisor (xen-3.4.3-rc4-pre.gz) but the
original SLES 11 kernel /vmlinuz-2.6.27.19-5-xen.dan ( the stock
kernel with a couple of printks thrown in).

This boots to the kernel, but then it locks up the machine when it
tries to bring the networking up:

Mounting local file systems...
/proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
debugfs on /sys/kernel/debug type debugfs (rw)
udev on /dev type tmpfs (rw)
devpts on /dev/pts type devpts (rw,mode=0620,gid=5)
/dev/mapper/ddf1_disk1_part1 on /boot type ext2 (rw,acl,user_xattr)
/dev/mapper/ddf1_disk1_part3 on /temp type ext3 (rw,acl,user_xattr)  done
Setting current sysctl status from /etc/sysctl.conf                  done
Enabling syn flood protection                                        done
Disabling IP forwarding                                              done
                                                                     done
Loading fuse module                                                  done
Mounting fuse control filesystem                                     done
Activating remaining swap-devices in /etc/fstab...                   done
Creating /var/log/boot.msg                                           done
Setting up linker cache (/etc/ld.so.cache) using ldconfig            done
ATTENTION: You have modified /etc/resolv.conf.  Leaving it untouched...
You can find my version in /etc/resolv.conf.netconfig ...
ATTENTION: You have modified /etc/yp.conf.  Leaving it untouched...
You can find my version in /etc/yp.conf.netconfig ...
Setting up hostname 'sunblade1'                                      done
Setting up NIS domainname 'adax.co.uk'                               done
Setting up loopback interface     lo
    lo        IP address: 127.0.0.1/8
              IP address: 127.0.0.2/8
Bridge firewalling registered


After about 5 minutes of sitting there it crashed:

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff8020e9cb>] profile_pc+0x30/0x3d
PGD 0
Oops: 0000 [1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:20:05.0/class
CPU 0
Modules linked in: bridge(N) stp(N) fuse(N) ext2(N) loop(N) rtc_cmos(N) i2c_i80)
Supported: No
Pid: 92, comm: kswapd0 Tainted: G          2.6.27.19-5-xen.dan #5
RIP: e030:[<ffffffff8020e9cb>]  [<ffffffff8020e9cb>] profile_pc+0x30/0x3d
RSP: e02b:ffffffff80783db8  EFLAGS: 00010002
RAX: 0000000000000000 RBX: ffffffff8049077c RCX: ffffffff8067e320
RDX: ffff880081fa5000 RSI: 0000000000000400 RDI: ffffffff8049077c
RBP: ffffffff80783dc8 R08: 0000000000000c31 R09: ffff8803d8608878
R10: ffffffff80783d38 R11: 000000000000000c R12: ffff8803d860b838
R13: ffffffff80774020 R14: 00000000006187a6 R15: ffff880002719000
FS:  00007fde48ab66f0(0000) GS:ffffffff80785080(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kswapd0 (pid: 92, threadinfo ffff8803d860a000, task ffff8803d8608840)
Stack:  ffff8803d860b838 0000000000000001 ffffffff80783de8 ffffffff8024da1a
 0000000000000000 0000000000000020 ffffffff80783e98 ffffffff8020e987
 ffffffff80490942 ffffffff806b7108 ffffffff807e9700 ffffffff80774000
Call Trace:
 [<ffffffff8024da1a>] profile_tick+0x5f/0x7d
 [<ffffffff8020e987>] timer_interrupt+0x436/0x44a
 [<ffffffff80270594>] handle_IRQ_event+0x50/0x9c
 [<ffffffff80271757>] handle_percpu_irq+0x45/0x73
 [<ffffffff8020d8e3>] do_IRQ+0x4a/0x92
 [<ffffffff803dbce8>] evtchn_do_upcall+0x1a9/0x275
 [<ffffffff8020bcae>] do_hypervisor_callback+0x1e/0x30
 [<ffffffff8049077c>] _read_unlock_irqrestore+0x58/0x59
DWARF2 unwinder stuck at _read_unlock_irqrestore+0x58/0x59

Leftover inexact backtrace:

 [<ffffffffa012c8a7>] ? dm_get_table+0x35/0x3d [dm_mod]
 [<ffffffffa012ca10>] ? __split_bio+0x1e/0x47f [dm_mod]
 [<ffffffff802837ac>] ? __set_page_dirty_nobuffers+0x12/0x18d
 [<ffffffff802a691d>] ? __mem_cgroup_uncharge_common+0x10/0x163
 [<ffffffff802896ae>] ? __dec_zone_state+0x9/0x77
 [<ffffffff80490942>] ? _spin_lock_irq+0xc/0x71
 [<ffffffff804902f4>] ? __down_read+0x1a/0x120
 [<ffffffff8027f2ae>] ? mempool_alloc_slab+0x16/0x18
 [<ffffffffa012d056>] ? dm_request+0x13d/0x15f [dm_mod]
 [<ffffffff8033d1c0>] ? generic_make_request+0x39c/0x3df
 [<ffffffff802d282a>] ? bvec_alloc_bs+0x86/0xad
 [<ffffffff802d2780>] ? bio_init+0xd/0x31
 [<ffffffff802d28a1>] ? bio_alloc_bioset+0x50/0x97
 [<ffffffff804909b9>] ? _spin_lock_irqsave+0x12/0x96
 [<ffffffff8033d2ce>] ? submit_bio+0xcb/0xd4
 [<ffffffff8029c3ba>] ? swap_writepage+0xf6/0x103
 [<ffffffff80287748>] ? shrink_page_list+0x37b/0x5f3
 [<ffffffff8021d359>] ? ptep_clear_flush_young+0x7c/0x8b
 [<ffffffff80286829>] ? __isolate_lru_page+0x9/0x62
 [<ffffffff802869e8>] ? isolate_lru_pages+0x166/0x210
 [<ffffffff80287b90>] ? shrink_inactive_list+0x1d0/0x51d
 [<ffffffff80287fe8>] ? shrink_zone+0x10b/0x12e
 [<ffffffff802885f1>] ? kswapd+0x3f3/0x583
 [<ffffffff80286a92>] ? isolate_pages_global+0x0/0x39
 [<ffffffff8024860a>] ? autoremove_wake_function+0x0/0x3d
 [<ffffffff802881fe>] ? kswapd+0x0/0x583
 [<ffffffff802480bf>] ? kthread+0x4e/0x7b
 [<ffffffff8020bef9>] ? child_rip+0xa/0x11
 [<ffffffff80248071>] ? kthread+0x0/0x7b
 [<ffffffff8020beef>] ? child_rip+0x0/0x11


Code: 54 53 e8 89 ca ff ff f6 87 88 00 00 00 03 48 8b 9f 80 00 00 00 49 89 fc 7
RIP  [<ffffffff8020e9cb>] profile_pc+0x30/0x3d
 RSP <ffffffff80783db8>
CR2: 0000000000000008
---[ end trace a2b3d37002a3cae5 ]---
Kernel panic - not syncing: Aiee, killing interrupt handler!
------------[ cut here ]------------
WARNING: at kernel/smp.c:331 smp_call_function_mask+0x5a/0x1e1()
Modules linked in: bridge(N) stp(N) fuse(N) ext2(N) loop(N) rtc_cmos(N) i2c_i80)


Is there some difference in the network configuration or something
that could cause this?

thanks
dan

Re: Re: Panic on boot on Sun Blade 6270

[Jan Beulich]

>>> Dan Gora <dan.gora@gmail.com> 12.03.10 18:37 >>>
>so it should use the new hypervisor (xen-3.4.3-rc4-pre.gz) but the
>original SLES 11 kernel /vmlinuz-2.6.27.19-5-xen.dan ( the stock
>kernel with a couple of printks thrown in).
>
>This boots to the kernel, but then it locks up the machine when it
>tries to bring the networking up:
>...
>After about 5 minutes of sitting there it crashed:
>
>BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
>IP: [<ffffffff8020e9cb>] profile_pc+0x30/0x3d

I would have been tempted to analyze this (given that it may point
out a general problem with the SLE11 kernel) if this wasn't a kernel
that you built on your own and if this wasn't a really old one. I
would guess that it's not only a couple of added printk()-s, but also
a different .config that you're using. Namely (not having seen the
disassembly of your kernel's profile_pc()) I could easily imagine
this oops to be a result of you building with CONFIG_FRAME_POINTER=y.

Jan

Re: Re: Panic on boot on Sun Blade 6270

[Dan Gora]

On Mon, Mar 15, 2010 at 6:21 AM, Jan Beulich <JBeulich@novell.com> wrote:
>>After about 5 minutes of sitting there it crashed:
>>
>>BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
>>IP: [<ffffffff8020e9cb>] profile_pc+0x30/0x3d
>
> I would have been tempted to analyze this (given that it may point
> out a general problem with the SLE11 kernel) if this wasn't a kernel
> that you built on your own and if this wasn't a really old one. I
> would guess that it's not only a couple of added printk()-s, but also
> a different .config that you're using. Namely (not having seen the
> disassembly of your kernel's profile_pc()) I could easily imagine
> this oops to be a result of you building with CONFIG_FRAME_POINTER=y.

:)... Yes I did build with CONFIG_FRAME_POINTER=y.  I guess I
considered the panic more incidental... Really the problem was that
the machine locked up on boot.  I didn't realize that
CONFIG_FRAME_POINTER could cause a panic.. I thought that it just
turned on using the frame pointer register so that backtraces would
work.

I tried to install the xen 3.4.1 from the SuSE rpms after this and
also could not get that to work.  That too would panic on boot up.  I
tried to install the xen 4.0.0 rpms, but quickly ran into prerequisite
problems and just gave up and went back to using xen 3.3.1, building
it from the SRPMS (I still cannot get anything to work just building
it from the xen hg repo).

I realize that this Sun box is probably pretty funky (it's a PCI
express module based machine) so it might have some peculiar issues
that would be of interest, but I've got to figure this other issue out
first because I have customers on my back.  If you are interested,
Jan, I can go back and get the backtrace from the 3.4.1 panic, but if
SLES 11 is too old to try and debug I'll just forget about it..

If there's time later and there is more recent stuff that we can try
and test, let me know what it is and I can give it a go.

thanks,
dan

Re: Re: Panic on boot on Sun Blade 6270

[Jan Beulich]

>>> Dan Gora <dan.gora@gmail.com> 16.03.10 03:39 >>>
>:)... Yes I did build with CONFIG_FRAME_POINTER=y.  I guess I
>considered the panic more incidental... Really the problem was that
>the machine locked up on boot.  I didn't realize that
>CONFIG_FRAME_POINTER could cause a panic.. I thought that it just
>turned on using the frame pointer register so that backtraces would
>work.

Our kernels have working backtraces without CONFIG_FRAME_POINTER,
which is why we are (happily) disabling this option.

>I tried to install the xen 3.4.1 from the SuSE rpms after this and
>also could not get that to work.  That too would panic on boot up.  I

It definitely shouldn't.

>tried to install the xen 4.0.0 rpms, but quickly ran into prerequisite
>problems and just gave up and went back to using xen 3.3.1, building
>it from the SRPMS (I still cannot get anything to work just building
>it from the xen hg repo).

So are you saying even plain SLE11 doesn't work on that box? If so,
why didn't you report this to our support folks yet?

>I realize that this Sun box is probably pretty funky (it's a PCI
>express module based machine) so it might have some peculiar issues
>that would be of interest, but I've got to figure this other issue out
>first because I have customers on my back.  If you are interested,
>Jan, I can go back and get the backtrace from the 3.4.1 panic, but if
>SLES 11 is too old to try and debug I'll just forget about it..

As said above - it's supposed to work (both with the 3.3.1 that SLE11
ships with [or really, with the most recent maintenance updates in
place for both kernel and Xen] and upstream versions down to 3.2.0).
The only limitation I know about for certain (Sun?) systems is that
not all interrupts may be usable by their devices/drivers. That
limitation goes away only with 4.0 (i.e. earlier upstream versions
won't - afaict - get you anywhere either if this is the case).

Jan

Re: Re: Panic on boot on Sun Blade 6270

[Dan Gora]

On Tue, Mar 16, 2010 at 7:17 AM, Jan Beulich <JBeulich@novell.com> wrote:

> So are you saying even plain SLE11 doesn't work on that box?

No, I'm not saying that.. SLES 11 + Xen 3.3.1 "works" (yes I have a
problem with it, but I'm still trying to figure it out).  In trying to
figure out my other problem I wanted to upgrade Xen to something later
but was not having any luck building and installing Xen from the hg
repo.  So I went to download.suse.com and downloaded the Xen 3.4.1
SRPMS, used rpmbuild to build the RPMS, installed those and that
panics on boot.  I could not install the 4.0.0 RPMs because there were
conflicts with the required prerequisites and what I already
installed, so I gave up.

> If so, why didn't you report this to our support folks yet?

I haven't reported the 3.4.1 crash just because:
1) I just found it a couple of days ago.
2) I don't have a support contract.
3) I'm really busy trying to solve the issue with my board.

>
> As said above - it's supposed to work (both with the 3.3.1 that SLE11
> ships with [or really, with the most recent maintenance updates in
> place for both kernel and Xen] and upstream versions down to 3.2.0).
> The only limitation I know about for certain (Sun?) systems is that
> not all interrupts may be usable by their devices/drivers. That
> limitation goes away only with 4.0 (i.e. earlier upstream versions
> won't - afaict - get you anywhere either if this is the case).

So we should be able use 4.0 on SLES 11 somehow?  How can I install a
newer Xen on this system.  The only way that I can get anything to
work that I compile from source is using the Xen 3.3.1 SRPM and using
rpmbuild to build the RPMs from that.  With that I can also modify the
3.3.1 Xen a bit to add printfs and the like to actually debug the
problem with my board (although that process is a total pain).

Is it possible to build something from the hg Xen repo and install it
on the SLES 11 system and have it work?  There seem to be a lot of
suse specific patches in the SRPMs, but I haven't gone through them to
see what they are all about...

thanks
dan

Re: Re: Panic on boot on Sun Blade 6270

[Jan Beulich]

>>> Dan Gora <dan.gora@gmail.com> 16.03.10 12:48 >>>
>On Tue, Mar 16, 2010 at 7:17 AM, Jan Beulich <JBeulich@novell.com> wrote:
>> If so, why didn't you report this to our support folks yet?
>
>I haven't reported the 3.4.1 crash just because:
>1) I just found it a couple of days ago.
>2) I don't have a support contract.
>3) I'm really busy trying to solve the issue with my board.

Reporting 3.4.1 problems there wouldn't be accepted anyway. This
was just under the (now known wrong) assumption that the shipped
hypervisor also wouldn't work.

>So we should be able use 4.0 on SLES 11 somehow?  How can I install a
>newer Xen on this system.  The only way that I can get anything to
>work that I compile from source is using the Xen 3.3.1 SRPM and using
>rpmbuild to build the RPMs from that.  With that I can also modify the
>3.3.1 Xen a bit to add printfs and the like to actually debug the
>problem with my board (although that process is a total pain).

I think you better forget rpm here. Extract the Xen tarball and run
"make xen tools" followed by (as root) "make install-xen" and possibly
(if you don't care overwriting files coming from the Xen RPMs, and if
you're prepared to sort out eventual inconsistencies)
"make install-tools" (I generally avoid doing the latter).

>Is it possible to build something from the hg Xen repo and install it
>on the SLES 11 system and have it work?  There seem to be a lot of
>suse specific patches in the SRPMs, but I haven't gone through them to
>see what they are all about...

Quite a number of them are backports of upstream changesets. But
of course there are also (too) many that don't have upstream
equivalents. Nevertheless, running newer upstream Xen underneath
SLE11 (or other SuSE) kernels is supposed to work (I occasionally
find myself needing to do this).

Jan