Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Christophe Saout]

Hi everyone,

I am very excited to see that dom0 pvops is finally coming close to
working, so I wanted to give it a try.

>From the description it was not clear to me which kernel to chose as
base for the patches.hg, so I took the latest (that was ~ 2 weeks ago)
kernel on git.kernel.org I could find (post-2.6.28 git tip at that
point).

I managed to more or less apply all of the patches in the patches.hg
series file (some where already applied upstream).  It also needed some
minor fixes (for instance arch/x86/pci/pci.h got moved to
arch/x86/include/asm/pci_x86.h, so rather trivial things).

However, I got none of the kernels to boot.  Every time it dies in what
looks like the first context switch (I guess), every time somewhere in
the scheduler shortly after doing the low-level switching stuff.  This
happens for both SMP and UP kernels (however, in different places).  For
the SMP kernel I saw a crash in "task_tick_fair", on the UP kernel in
the arch code __switch_to somewhere around the unlazy_fpu() call (a bit
confusing due to the heavy inlining).  In yet another version I have
"preempt_schedule" directly on the stack, it's not clear where it
crashes, since I don't even get a kernel trace.  Sometimes the crash
would be due to a memory access into Nirvana land (while trying to get a
per_cpu variable) and another time jumping to the null pointer.

I have to say that except for looking a bit at it I have no clue what
might be going wrong.  It seems like the context switch is messed up and
therefore things go wonky quickly after.  The same kernel works fine
with KVM paravirt or on bare metal.

It always crashes directly after "ACPI: Core revision
20080926" (sometimes with a Bug beforehand, sometimes with a few Kernel
oopses before XEN declares the dom0 as dead).  It doesn't seem to have
anything to do with ACPI however.

I have my kernel in a local git area (all patches.hg patches as
individual git commits, as well as my personal fixes), in case someone
is interested.

So my question: Am I missing something here? Or is this known to now
work or has not yet been looked at? Any idea what is happening?

Thank you,
	Christophe


kvm -hda /data/store/tmp -no-kvm -nographic
(yes, I am playing with Qemu, but it gives the same effect when booting
 it natively)

 __  __            _____ _  _                      _        _     _      
 \ \/ /___ _ __   |___ /| || |     _   _ _ __  ___| |_ __ _| |__ | | ___ 
  \  // _ \ '_ \    |_ \| || |_ __| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
  /  \  __/ | | |  ___) |__   _|__| |_| | | | \__ \ || (_| | |_) | |  __/
 /_/\_\___|_| |_| |____(_) |_|     \__,_|_| |_|___/\__\__,_|_.__/|_|\___|
                                                                         
(XEN) Xen version 3.4-unstable (chtephan@intern.saout.de) (gcc-Treiberversion 4.3.2 (Gentoo 4.3.2 p0.2, pie-8.7.9)  führt GCC-Version 4.4.0-alpha20090109 aus) Sat Jan 10 18:05:59 CET 2009
(XEN) Latest ChangeSet: Fri Jan 09 16:56:54 2009 +0000 19024:b999142bca8c
(XEN) Command line: console=com1,vga com1=9600,8n1 
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN) Disc information:
(XEN)  Found 1 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009fc00 (usable)
(XEN)  000000000009fc00 - 00000000000a0000 (reserved)
(XEN)  00000000000e8000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000007ff0000 (usable)
(XEN)  0000000007ff0000 - 0000000008000000 (ACPI data)
(XEN)  00000000fffbd000 - 0000000100000000 (reserved)
(XEN) System RAM: 127MB (130620kB)
(XEN) ACPI: RSDP 000FBC80, 0014 (r0 QEMU  )
(XEN) ACPI: RSDT 07FF0000, 002C (r1 QEMU   QEMURSDT        1 QEMU        1)
(XEN) ACPI: FACP 07FF002C, 0074 (r1 QEMU   QEMUFACP        1 QEMU        1)
(XEN) ACPI: DSDT 07FF0100, 24A4 (r1   BXPC   BXDSDT        1 INTL 20061109)
(XEN) ACPI: FACS 07FF00C0, 0040
(XEN) ACPI: APIC 07FF25A8, 00E0 (r1 QEMU   QEMUAPIC        1 QEMU        1)
(XEN) Xen heap: 14MB (14676kB)
(XEN) Domain heap initialised
(XEN) Processor #0 6:2 APIC version 17
(XEN) IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 1996.258 MHz processor.
(XEN) AMD SVM: ASIDs disabled. 
(XEN) HVM: SVM enabled
(XEN) CPU0: AMD QEMU Virtual CPU version 0.9.1 stepping 03
(XEN) Total of 1 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) Platform timer is 3.579MHz ACPI PM Timer
(XEN) Brought up 1 CPUs
(XEN) I/O virtualisation disabled
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x200000 -> 0xadb3d4
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000004000000->0000000005000000 (21254 pages to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff80200000->ffffffff80adb3d4
(XEN)  Init. ramdisk: ffffffff80adc000->ffffffff80adc000
(XEN)  Phys-Mach map: ffffffff80adc000->ffffffff80b0d830
(XEN)  Start info:    ffffffff80b0e000->ffffffff80b0e4b4
(XEN)  Page tables:   ffffffff80b0f000->ffffffff80b18000
(XEN)  Boot stack:    ffffffff80b18000->ffffffff80b19000
(XEN)  TOTAL:         ffffffff80000000->ffffffff80c00000
(XEN)  ENTRY ADDRESS: ffffffff8086f200
(XEN) Dom0 has maximum 1 VCPUs
(XEN) Scrubbing Free RAM: done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 116kB init memory.
mapping kernel into physical memory
Xen: setup ISA identity maps
about to get started...
PAT disabled on Xen
Linux version 2.6.29-rc0-xen-cs1-dirty (chtephan@leto) (gcc driver version 4.3.2 (Gentoo 4.3.2 p0.2, pie-8.7.9) executing gcc version 4.4.0-alpha20090109) #1 PREEMPT Sat Jan 10 17:44:29 CET 2009
Command line: console=tty0 xencons=ttyS0,115200 console=hvc0 earlyprintk=xen root=/dev/vg/root ro nopat noacpi 
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 000000000009fc00 (usable)
 Xen: 000000000009fc00 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000000adc000 (usable)
 Xen: 0000000000adc000 - 0000000000b0f000 (reserved)
 Xen: 0000000000b0f000 - 0000000006306000 (usable)
 Xen: 0000000007ff0000 - 0000000008000000 (ACPI data)
 Xen: 00000000fffbd000 - 0000000100000000 (reserved)
console [xenboot0] enabled
PAT support disabled.
DMI 2.4 present.
last_pfn = 0x6306 max_arch_pfn = 0x100000000
init_memory_mapping: 0000000000000000-0000000006306000
last_map_addr: 6306000 end: 6306000
ACPI: RSDP 000FBC80, 0014 (r0 QEMU  )
ACPI: RSDT 07FF0000, 002C (r1 QEMU   QEMURSDT        1 QEMU        1)
ACPI: FACP 07FF002C, 0074 (r1 QEMU   QEMUFACP        1 QEMU        1)
ACPI: DSDT 07FF0100, 24A4 (r1   BXPC   BXDSDT        1 INTL 20061109)
ACPI: FACS 07FF00C0, 0040
ACPI: APIC 07FF25A8, 00E0 (r1 QEMU   QEMUAPIC        1 QEMU        1)
(5 early reservations) ==> bootmem [0000000000 - 0006306000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [0000b0f000 - 0000b18000]   XEN PAGETABLES ==> [0000b0f000 - 0000b18000]
  #2 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
  #3 [0000200000 - 0000adb3d4]    TEXT DATA BSS ==> [0000200000 - 0000adb3d4]
  #4 [0000008000 - 0000030000]          PGTABLE ==> [0000008000 - 0000030000]
found SMP MP-table at [ffff8800000fbb60] 000fbb60
Zone PFN ranges:
  DMA      0x00000000 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x00100000
Movable zone start PFN for each node
early_node_map[3] active PFN ranges
    0: 0x00000000 -> 0x0000009f
    0: 0x00000100 -> 0x00000adc
    0: 0x00000b0f -> 0x00006306
ACPI: PM-Timer IO Port: 0xb008
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] disabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] disabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] disabled)
ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled)
ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] disabled)
ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] disabled)
ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] disabled)
ACPI: LAPIC (acpi_id[0x09] lapic_id[0x09] disabled)
ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0a] disabled)
ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0b] disabled)
ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x0c] disabled)
ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x0d] disabled)
ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x0e] disabled)
ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x0f] disabled)
ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 1, version 0, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
Using ACPI (MADT) for SMP configuration information
(XEN) ioapic_guest_write: apic=0, pin=0, old_irq=0, new_irq=-1
(XEN) ioapic_guest_write: old_entry=000009f0, new_entry=00010900
(XEN) ioapic_guest_write: Attempt to remove IO-APIC pin of in-use IRQ!
(XEN) ioapic_guest_write: apic=0, pin=4, old_irq=4, new_irq=-1
(XEN) ioapic_guest_write: old_entry=000009f1, new_entry=00010900
(XEN) ioapic_guest_write: Attempt to remove IO-APIC pin of in-use IRQ!
PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
PM: Registered nosave memory: 0000000000adc000 - 0000000000b0f000
Allocating PCI resources starting at 10000000 (gap: 8000000:f7fbd000)
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 22534
Kernel command line: console=tty0 xencons=ttyS0,115200 console=hvc0 earlyprintk=xen root=/dev/vg/root ro nopat noacpi 
Initializing CPU#0
xen: allocated irq 9 for acpi 9
PID hash table entries: 512 (order: 9, 4096 bytes)
Detected 1996.258 MHz processor.
Console: colour VGA+ 80x25
console [tty0] enabled
PAT disabled on Xen
Linux version 2.6.29-rc0-xen-cs1-dirty (chtephan@leto) (gcc driver version 4.3.2 (Gentoo 4.3.2 p0.2, pie-8.7.9) executing gcc version 4.4.0-alpha20090109) #1 PREEMPT Sat Jan 10 17:44:29 CET 2009
Command line: console=tty0 xencons=ttyS0,115200 console=hvc0 earlyprintk=xen root=/dev/vg/root ro nopat noacpi 
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 000000000009fc00 (usable)
 Xen: 000000000009fc00 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000000adc000 (usable)
 Xen: 0000000000adc000 - 0000000000b0f000 (reserved)
 Xen: 0000000000b0f000 - 0000000006306000 (usable)
 Xen: 0000000007ff0000 - 0000000008000000 (ACPI data)
 Xen: 00000000fffbd000 - 0000000100000000 (reserved)
console [xenboot0] enabled
PAT support disabled.
DMI 2.4 present.
last_pfn = 0x6306 max_arch_pfn = 0x100000000
init_memory_mapping: 0000000000000000-0000000006306000
last_map_addr: 6306000 end: 6306000
ACPI: RSDP 000FBC80, 0014 (r0 QEMU  )
ACPI: RSDT 07FF0000, 002C (r1 QEMU   QEMURSDT        1 QEMU        1)
ACPI: FACP 07FF002C, 0074 (r1 QEMU   QEMUFACP        1 QEMU        1)
ACPI: DSDT 07FF0100, 24A4 (r1   BXPC   BXDSDT        1 INTL 20061109)
ACPI: FACS 07FF00C0, 0040
ACPI: APIC 07FF25A8, 00E0 (r1 QEMU   QEMUAPIC        1 QEMU        1)
(5 early reservations) ==> bootmem [0000000000 - 0006306000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [0000b0f000 - 0000b18000]   XEN PAGETABLES ==> [0000b0f000 - 0000b18000]
  #2 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
  #3 [0000200000 - 0000adb3d4]    TEXT DATA BSS ==> [0000200000 - 0000adb3d4]
  #4 [0000008000 - 0000030000]          PGTABLE ==> [0000008000 - 0000030000]
found SMP MP-table at [ffff8800000fbb60] 000fbb60
Zone PFN ranges:
  DMA      0x00000000 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x00100000
Movable zone start PFN for each node
early_node_map[3] active PFN ranges
    0: 0x00000000 -> 0x0000009f
    0: 0x00000100 -> 0x00000adc
    0: 0x00000b0f -> 0x00006306
ACPI: PM-Timer IO Port: 0xb008
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] disabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] disabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] disabled)
ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled)
ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] disabled)
ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] disabled)
ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] disabled)
ACPI: LAPIC (acpi_id[0x09] lapic_id[0x09] disabled)
ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0a] disabled)
ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0b] disabled)
ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x0c] disabled)
ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x0d] disabled)
ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x0e] disabled)
ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x0f] disabled)
ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 1, version 0, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
Using ACPI (MADT) for SMP configuration information
PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
PM: Registered nosave memory: 0000000000adc000 - 0000000000b0f000
Allocating PCI resources starting at 10000000 (gap: 8000000:f7fbd000)
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 22534
Kernel command line: console=tty0 xencons=ttyS0,115200 console=hvc0 earlyprintk=xen root=/dev/vg/root ro nopat noacpi 
Initializing CPU#0
xen: allocated irq 9 for acpi 9
PID hash table entries: 512 (order: 9, 4096 bytes)
Detected 1996.258 MHz processor.
Console: colour VGA+ 80x25
console [tty0] enabled
console handover: boot [xenboot0] -> real [hvc0]
Dentry cache hash table entries: 16384 (order: 5, 131072 bytes)
Inode-cache hash table entries: 8192 (order: 4, 65536 bytes)
Checking aperture...
No AGP bridge found
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing 64MB software IO TLB between ffff8800012c5000 - ffff8800052c5000
software IO TLB at phys 0x12c5000 - 0x52c5000
Memory: 23464k/101400k available (4673k kernel code, 592k absent, 77280k reserved, 1715k data, 1592k init)
SLUB: Genslabs=12, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
installing Xen timer for CPU 0
Calibrating delay loop (skipped), value calculated using timer frequency.. 3994.87 BogoMIPS (lpj=6654193)
Mount-cache hash table entries: 256
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
CPU: AMD QEMU Virtual CPU version 0.9.1 stepping 03
(XEN) domain.c:493:d0 Attempt to change CR4 flags 00000660 -> 00000620
ACPI: Core revision 20080926
BUG: scheduling while atomic: kthreadd/2/0x00000000
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 0 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-3.4-unstable  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e033:[<ffffffff806868bd>]
(XEN) RFLAGS: 0000000000000246   EM: 0   CONTEXT: pv guest
(XEN) rax: ffff880005c39fd8   rbx: ffffffffff517000   rcx: 00000000ffffffff
(XEN) rdx: 00000000ffffffff   rsi: 00000000000017dd   rdi: 0000000000000000
(XEN) rbp: 0000000000000000   rsp: ffff880005c23ee0   r8:  0000000000000000
(XEN) r9:  0000000000000001   r10: 0000000000000000   r11: ffffffff8041f5b0
(XEN) r12: ffffffff80a8aa77   r13: 0000000000000037   r14: 0000000000000800
(XEN) r15: ffff880005c37e90   cr0: 000000008005003b   cr4: 00000000000006f0
(XEN) cr3: 0000000004201000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffff880005c23ee0:
(XEN)    0000000000000000 0000000000000000 ffffffff8020fb30 ffffffffff517000
(XEN)    ffffffff806868c7 0000000000000000 0000000000000000 ffffffff8020fb30
(XEN)    ffffffffff517000 ffffffff806868c7 0000000000000000 0000000000000000
(XEN)    ffffffff8020fb30 ffffffffff517000 ffffffff806868c7 0000000000000000
(XEN)    0000000000000000 ffffffff8020fb30 ffffffffff517000 ffffffff806868c7
(XEN)    0000000000000000 0000000000000000 ffffffff8020fb30 ffffffffff517000
(XEN)    ffffffff806868c7 0000000000000000 0000000000000000 ffffffff8020fb30
(XEN)    ffffffffff517000 ffffffff806868c7 0000000000000000 0000000000000000
(XEN)    ffffffff8020fb30 ffffffffff517000 ffffffff806868c7 0000000000000000
(XEN)    ffff880005c24030 ffffffff8020fb30 ffffffffff517000 ffffffff806868c3
(XEN)    0000000000000000 0000000000000000 ffffffff8020fb30 ffffffffff517000
(XEN)    ffffffff806868c3 0000000000000000 0000000000000001 ffffffff8020fb30
(XEN)    ffffffffff517000 ffffffff806868c3 0000000000000000 0000000000000000
(XEN)    ffffffff8020fb30 ffffffffff517000 ffffffff806868c3 0000000000000000
(XEN)    0000000000000000 ffffffff8020fb30 ffffffffff517000 ffffffff806868c3
(XEN)    0000000000000000 6c754e0068737570 ffffffff8020fb30 ffffffffff517000
(XEN)    ffffffff806868c3 0000000000000000 ffff880005c24120 ffffffff8020fb30
(XEN)    ffffffffff517000 ffffffff806868c3 0000000000000000 20726f6620657461
(XEN)    ffffffff8020fb30 ffffffffff517000 ffffffff806868c3 0000000000000000
(XEN)    7461745320217974 ffffffff8020fb30 ffffffffff517000 ffffffff806868c3
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Marc - A. Dahlhaus]

Hello Christophe,

Christophe Saout schrieb:
> Hi everyone,
>
> I am very excited to see that dom0 pvops is finally coming close to
> working, so I wanted to give it a try.
>
> >From the description it was not clear to me which kernel to chose as
> base for the patches.hg, so I took the latest (that was ~ 2 weeks ago)
> kernel on git.kernel.org I could find (post-2.6.28 git tip at that
> point).
>   
Just follow the instructions on this wiki page:

http://wiki.xensource.com/xenwiki/XenParavirtOps

You could also search the Archives of xen-devel beginning around 
November 2008 for more
informations...

Marc

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Christophe Saout]

Hi Marc,

> > I am very excited to see that dom0 pvops is finally coming close to
> > working, so I wanted to give it a try.
> >
> > >From the description it was not clear to me which kernel to chose as
> > base for the patches.hg, so I took the latest (that was ~ 2 weeks ago)
> > kernel on git.kernel.org I could find (post-2.6.28 git tip at that
> > point).
> >   
> Just follow the instructions on this wiki page:
> 
> http://wiki.xensource.com/xenwiki/XenParavirtOps
> 
> You could also search the Archives of xen-devel beginning around 
> November 2008 for more
> informations...

Yes, I followed those instruction (or at least I believe I did).  It is
however not specific as to which kernel version from the "tip.git" to
use as base for applying the patches.  I mean, that is not really my
problem, I got everything applied and have a compiling kernel. It just
doesn't boot (or at least doesn't get as far as for instance Pasi and
his experiments).

Actually, I am not really interested in getting a working kernel (I know
that there are still some pieces missing), so this is purely academical
and for testing.  Since the patches are supposed to be merged soon (at
least I got the impression that was the plan) I though I was going to
join the testing effort.  And at this point they are supposed to sort of
work on any bleeding edge kernel, right?  So I took one.

I could have gone back in the history and take something around
2.6.28-rc8 (which seemed to have worked for others), but then they were
using x86_32 and I am testing x86_64, if I see correctly.  So my
question was mostly if this had seen some testing at all, is supposed to
work, and if it is, did I miss something.  In any case, this is the
result of my testing. :-)

I also forward-ported a few things to the latest version a few hours
ago, including some changes in xen-iommu.c to follow the dma_ops merging
thing in the tip head.

I managed to get the native XEN port for 2.6.27 from the other hg
booting on my notebook, I'll try to see if my pvops kernel works as
DomU.

Cheers,
	Christophe

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Pasi Kärkkäinen]

On Sat, Jan 10, 2009 at 09:28:59PM +0100, Christophe Saout wrote:
> Hi Marc,
> 
> > > I am very excited to see that dom0 pvops is finally coming close to
> > > working, so I wanted to give it a try.
> > >
> > > >From the description it was not clear to me which kernel to chose as
> > > base for the patches.hg, so I took the latest (that was ~ 2 weeks ago)
> > > kernel on git.kernel.org I could find (post-2.6.28 git tip at that
> > > point).
> > >   
> > Just follow the instructions on this wiki page:
> > 
> > http://wiki.xensource.com/xenwiki/XenParavirtOps
> > 
> > You could also search the Archives of xen-devel beginning around 
> > November 2008 for more
> > informations...
> 
> Yes, I followed those instruction (or at least I believe I did).  It is
> however not specific as to which kernel version from the "tip.git" to
> use as base for applying the patches.  I mean, that is not really my
> problem, I got everything applied and have a compiling kernel. It just
> doesn't boot (or at least doesn't get as far as for instance Pasi and
> his experiments).
> 

pv_ops dom0 patches are currently for 2.6.28-rc8. 

I think and hope Jeremy will update them when he gets back from his vacation
next week. 

> Actually, I am not really interested in getting a working kernel (I know
> that there are still some pieces missing), so this is purely academical
> and for testing.  Since the patches are supposed to be merged soon (at
> least I got the impression that was the plan) I though I was going to
> join the testing effort.  And at this point they are supposed to sort of
> work on any bleeding edge kernel, right?  So I took one.
> 
> I could have gone back in the history and take something around
> 2.6.28-rc8 (which seemed to have worked for others), but then they were
> using x86_32 and I am testing x86_64, if I see correctly.  So my
> question was mostly if this had seen some testing at all, is supposed to
> work, and if it is, did I miss something.  In any case, this is the
> result of my testing. :-)
> 

I think some people have been trying x86-64 too, with similar results than
me and others on x86-32.

> I also forward-ported a few things to the latest version a few hours
> ago, including some changes in xen-iommu.c to follow the dma_ops merging
> thing in the tip head.
>

Nice.

I also just noticed there are some new patches in
http://xenbits.xen.org/paravirt_ops/patches.hg/

"Fixes for PAE swiotlb with Becky's patches."

Maybe I should try and see if those make any difference.
 
> I managed to get the native XEN port for 2.6.27 from the other hg
> booting on my notebook, I'll try to see if my pvops kernel works as
> DomU.
> 

It should :) 

-- Pasi

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Marc - A. Dahlhaus]

Christophe Saout schrieb:
> Hi Marc,
>
>   
>>> I am very excited to see that dom0 pvops is finally coming close to
>>> working, so I wanted to give it a try.
>>>
>>> >From the description it was not clear to me which kernel to chose as
>>> base for the patches.hg, so I took the latest (that was ~ 2 weeks ago)
>>> kernel on git.kernel.org I could find (post-2.6.28 git tip at that
>>> point).
>>>   
>>>       
>> Just follow the instructions on this wiki page:
>>
>> http://wiki.xensource.com/xenwiki/XenParavirtOps
>>
>> You could also search the Archives of xen-devel beginning around 
>> November 2008 for more
>> informations...
>>     
>
> Yes, I followed those instruction (or at least I believe I did).  It is
> however not specific as to which kernel version from the "tip.git" to
> use as base for applying the patches.  I mean, that is not really my
> problem, I got everything applied and have a compiling kernel. It just
> doesn't boot (or at least doesn't get as far as for instance Pasi and
> his experiments).
>   
hg update `cat patches/KERNEL_VERSION`

this line selects the right version to apply the patches...
> Actually, I am not really interested in getting a working kernel (I know
> that there are still some pieces missing), so this is purely academical
> and for testing.  Since the patches are supposed to be merged soon (at
> least I got the impression that was the plan) I though I was going to
> join the testing effort.  And at this point they are supposed to sort of
> work on any bleeding edge kernel, right?  So I took one.
>
> I could have gone back in the history and take something around
> 2.6.28-rc8 (which seemed to have worked for others), but then they were
> using x86_32 and I am testing x86_64, if I see correctly.  So my
> question was mostly if this had seen some testing at all, is supposed to
> work, and if it is, did I miss something.  In any case, this is the
> result of my testing. :-)
>
> I also forward-ported a few things to the latest version a few hours
> ago, including some changes in xen-iommu.c to follow the dma_ops merging
> thing in the tip head.
>
> I managed to get the native XEN port for 2.6.27 from the other hg
> booting on my notebook, I'll try to see if my pvops kernel works as
> DomU.
>
> Cheers,
> 	Christophe
>   

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Marc - A. Dahlhaus]

Pasi Kärkkäinen schrieb:
> On Sat, Jan 10, 2009 at 09:28:59PM +0100, Christophe Saout wrote:
>   
>> Hi Marc,
>>
>>     
>>>> I am very excited to see that dom0 pvops is finally coming close to
>>>> working, so I wanted to give it a try.
>>>>
>>>> >From the description it was not clear to me which kernel to chose as
>>>> base for the patches.hg, so I took the latest (that was ~ 2 weeks ago)
>>>> kernel on git.kernel.org I could find (post-2.6.28 git tip at that
>>>> point).
>>>>   
>>>>         
>>> Just follow the instructions on this wiki page:
>>>
>>> http://wiki.xensource.com/xenwiki/XenParavirtOps
>>>
>>> You could also search the Archives of xen-devel beginning around 
>>> November 2008 for more
>>> informations...
>>>       
>> Yes, I followed those instruction (or at least I believe I did).  It is
>> however not specific as to which kernel version from the "tip.git" to
>> use as base for applying the patches.  I mean, that is not really my
>> problem, I got everything applied and have a compiling kernel. It just
>> doesn't boot (or at least doesn't get as far as for instance Pasi and
>> his experiments).
>>
>>     
>
> pv_ops dom0 patches are currently for 2.6.28-rc8. 
>
> I think and hope Jeremy will update them when he gets back from his vacation
> next week. 
>
>   
>> Actually, I am not really interested in getting a working kernel (I know
>> that there are still some pieces missing), so this is purely academical
>> and for testing.  Since the patches are supposed to be merged soon (at
>> least I got the impression that was the plan) I though I was going to
>> join the testing effort.  And at this point they are supposed to sort of
>> work on any bleeding edge kernel, right?  So I took one.
>>
>> I could have gone back in the history and take something around
>> 2.6.28-rc8 (which seemed to have worked for others), but then they were
>> using x86_32 and I am testing x86_64, if I see correctly.  So my
>> question was mostly if this had seen some testing at all, is supposed to
>> work, and if it is, did I miss something.  In any case, this is the
>> result of my testing. :-)
>>
>>     
>
> I think some people have been trying x86-64 too, with similar results than
> me and others on x86-32.
>   
>> I also forward-ported a few things to the latest version a few hours
>> ago, including some changes in xen-iommu.c to follow the dma_ops merging
>> thing in the tip head.
>>
>>     
>
> Nice.
>
> I also just noticed there are some new patches in
> http://xenbits.xen.org/paravirt_ops/patches.hg/
>
> "Fixes for PAE swiotlb with Becky's patches."
>
> Maybe I should try and see if those make any difference.
>   
My testing included this changeset already. But it changed nothing on 
the symptoms here, hope you get better results.


Marc

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Christophe Saout]

Hi Pasi,

> > Yes, I followed those instruction (or at least I believe I did).  It is
> > however not specific as to which kernel version from the "tip.git" to
> > use as base for applying the patches.  I mean, that is not really my
> > problem, I got everything applied and have a compiling kernel. It just
> > doesn't boot (or at least doesn't get as far as for instance Pasi and
> > his experiments).
> > 
> 
> pv_ops dom0 patches are currently for 2.6.28-rc8. 

Yes, but it also includes the x86.patch which again applies all sorts of
random newer things from tip.git.  There is always a shitload of patches
coming in, so maybe something broke in there.  Maybe I'll find out what.

There is this file "KERNEL_VERSION" in patches.hg which curresponds to
the git commit id of the "auto-latest" branch from the "tip.git" tree,
so I assumed this was the one the x86.patch is taken from and which
Jeremy was actually using.

> I think and hope Jeremy will update them when he gets back from his vacation
> next week. 

Ah, he's on vacation.  He's probably the guy who would know what is
crashing my kernel.

> I think some people have been trying x86-64 too, with similar results than
> me and others on x86-32.

Ok, good to know.

> > I also forward-ported a few things to the latest version a few hours
> > ago, including some changes in xen-iommu.c to follow the dma_ops merging
> > thing in the tip head.
>
> Nice.
> 
> I also just noticed there are some new patches in
> http://xenbits.xen.org/paravirt_ops/patches.hg/
> 
> "Fixes for PAE swiotlb with Becky's patches."
> 
> Maybe I should try and see if those make any difference.

Yes, already did and it doesn't make a different.  I think something
much more profound is broken in my kernel, something with memory
management during context switches.  As said, I'll try to figure out if
this also happens under DomU, because if it should, I can bisect where
exactly it broke (I am guessing somewhere between the 2.6.28-rc8 +
x86.patch and my version).

> > I managed to get the native XEN port for 2.6.27 from the other hg
> > booting on my notebook, I'll try to see if my pvops kernel works as
> > DomU.
>
> It should :) 

That remains to be proven. :-)

Cheers,
	Christophe

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Christophe Saout]

Hi again,

> > I managed to get the native XEN port for 2.6.27 from the other hg
> > booting on my notebook, I'll try to see if my pvops kernel works as
> > DomU.
>
> It should :) 

Ok, this gives a similar crash.  Seems like xen pvops in the kernel I
was using is busted, not the fault of the dom0 patches.

Sorry for the fuzz.

I'm going to bisect that now. :-)

Cheers,
	Christophe

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Christophe Saout]

Hi again (and a nice Sunday),

SUCCESS!!! (at least partially)

Ok, turns out that Xen pv_ops generelly seems currently broken with
PREEMPT enabled, that was my problem here, also in vanilla 2.6.28 and
this was the cause for my weird scheduler crashes.

So I just went to PREEMPT_VOLUNTARY and that is out of the way.

Now I actually got my kernel with dom0 patches to boot, pass through the
initramfs (dm-crypt and LVM setup), boot the rootfs and start to boot.

However, after one page of output it then Oopses somewhere during
fsck.reiserfs in "grap_swap_tokens".  Again, I am not sure if this is
dom0 related or could also happen in a domU (looks like I will need to
find a bootable rootfs for domU).

The interesting thing, however is, that all the hardware initialisation
seems to be just fine so far.  Most notably, my harddisk and CDROM (an
AHCI controller for the HD and an old-style IDE controller for the
CDROM).

The AHCI has problems being detected if I use "nosmp", then it
apparently fails to route the IRQ, but with smp it works.

I have put the kernel git here for anyone interested:

http://cvs.saout.de/gitweb/?p=linux-dom0-pvops.git;a=summary

(note that the commits with the filenames are the patches
 from patches.hg, eventually with modifications and there
 are also a few commits with fixes afterwards which I have
 not all merged into the original patches)

Cheers,
	Christophe

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Christophe Saout]

Hi again with the Sunday spam,

> However, after one page of output it then Oopses somewhere during
> fsck.reiserfs in "grap_swap_tokens".

Ok, this is somewhere around process exit and it crashes with
current->mm being unexpectedly NULL.  While mergeing commits yesterday I
saw some suspicios fiddling with remote CPU mm freeing (from the tip
tree in xen pvops mm code), maybe it has something to do with that.

I also just rebased what I had against the tip.git HEAD again, and there
were some more merges that needed to be fixed up by hand.

Cheers,
	Christophe

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Christophe Saout]

Hi again,

> > However, after one page of output it then Oopses somewhere during
> > fsck.reiserfs in "grab_swap_tokens".

Hmm, I found this patch in the SuSE kernel where they just return from
grab_swap_tokens if current->mm is NULL and claim that it can happen
(?).

Applying this makes the system happily continue past that point and
almost fully come up.  Hard disk working, wireless working, not looking
bad at all.

The X server however didn't want to start and crashed somewhere in
"vgahw" initialisation. (*)

Also, the console locked up after ~2 minutes, but still reacted to
Alt-SysRq-B.

I could produce some Alt-SysRq-T traces to see what's going on, maybe
using netconsole, but some other time.

Cheers,
	Christophe


Backtrace:
0: /usr/X11R6/bin/X(xorg_backtrace+0x26) [0x4e0d96]
1: /usr/X11R6/bin/X(xf86SigHandler+0x6a) [0x47bbb4]
2: /lib64/libc.so.6 [0x3644232080]
3: /usr/lib64/xorg/modules//libvgahw.so [0x7fc6c3c89b02]
4: /usr/lib64/xorg/modules//libvgahw.so(vgaHWGetIOBase+0xa) [0x7fc6c3c8adad]
5: /usr/lib64/xorg/modules/drivers//radeon_drv.so(RADEONPreInit+0x12a7) [0x7fc6c
6: /usr/X11R6/bin/X(InitOutput+0x832) [0x465248]
7: /usr/X11R6/bin/X(main+0x297) [0x43166e]
8: /lib64/libc.so.6(__libc_start_main+0xfd) [0x364421e60d]
9: /usr/X11R6/bin/X [0x430c29]

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Christophe Saout]

And hello once more,

> > > However, after one page of output it then Oopses somewhere during
> > > fsck.reiserfs in "grab_swap_tokens".
> 
> Hmm, I found this patch in the SuSE kernel where they just return from
> grab_swap_tokens if current->mm is NULL and claim that it can happen
> (?).

Urks, ok, that workaround is probably total BS.  It is supposed to help
when the current process is a kernel thread, and not reiserfsck.  So the
point really is how come that current->mm is NULL at that point.

My workaround leaves me with the attached ugly messages in the log,
which is possibly the cause for the lockup later. (*)

So, my latest one is here:

http://git.saout.de/gitweb/?p=linux-dom0-pvops.git;a=summary

(and forget that very last patch)

Cheers,
	Christophe


(*)

Jan 11 17:05:30 leto swap_dup: Bad swap file entry 80000000003ab1d0
Jan 11 17:05:30 leto swap_dup: Bad swap file entry 80000000003a19c0
Jan 11 17:05:30 leto swap_free: Bad swap file entry 80000000003ab1d0
Jan 11 17:05:30 leto BUG: Bad page map in process fsck.reiserfs  pte:7563a020 pmd:7196f067
Jan 11 17:05:30 leto addr:00000030e3c04000 vm_flags:08000070 anon_vma:(null) mapping:ffff880073513950 index:4
Jan 11 17:05:30 leto vma->vm_ops->fault: filemap_fault+0x0/0x440
Jan 11 17:05:30 leto vma->vm_file->f_op->mmap: reiserfs_file_mmap+0x0/0x70
Jan 11 17:05:30 leto Pid: 3012, comm: fsck.reiserfs Not tainted 2.6.29-rc0-xen-cs1 #1
Jan 11 17:05:30 leto Call Trace:
Jan 11 17:05:30 leto [<ffffffff802ab120>] print_bad_pte+0x1e0/0x2c0
Jan 11 17:05:30 leto [<ffffffff802ac6cd>] unmap_vmas+0x86d/0xaa0
Jan 11 17:05:30 leto [<ffffffff8029f801>] ____pagevec_lru_add+0x151/0x170
Jan 11 17:05:30 leto [<ffffffff802b1f82>] exit_mmap+0xa2/0x170
Jan 11 17:05:30 leto [<ffffffff80247010>] mmput+0x30/0xd0
Jan 11 17:05:30 leto [<ffffffff8024b376>] exit_mm+0x106/0x140
Jan 11 17:05:30 leto [<ffffffff8024d0f4>] do_exit+0x114/0x8e0
Jan 11 17:05:30 leto [<ffffffff802ed15d>] vfs_fsync+0xbd/0x110
Jan 11 17:05:30 leto [<ffffffff8024d8f7>] do_group_exit+0x37/0xb0
Jan 11 17:05:30 leto [<ffffffff802135aa>] system_call_fastpath+0x16/0x1b
Jan 11 17:05:30 leto swap_free: Bad swap file entry 80000000003a19c0
Jan 11 17:05:30 leto BUG: Bad page map in process fsck.reiserfs  pte:74338020 pmd:731b6067
Jan 11 17:05:30 leto addr:0000003644348000 vm_flags:08000070 anon_vma:(null) mapping:ffff8800734f17f0 index:148
Jan 11 17:05:30 leto vma->vm_ops->fault: filemap_fault+0x0/0x440
Jan 11 17:05:30 leto vma->vm_file->f_op->mmap: reiserfs_file_mmap+0x0/0x70
Jan 11 17:05:30 leto Pid: 3012, comm: fsck.reiserfs Tainted: G    B      2.6.29-rc0-xen-cs1 #1
Jan 11 17:05:30 leto Call Trace:
Jan 11 17:05:30 leto [<ffffffff802ab120>] print_bad_pte+0x1e0/0x2c0
Jan 11 17:05:30 leto [<ffffffff802ac6cd>] unmap_vmas+0x86d/0xaa0
Jan 11 17:05:30 leto [<ffffffff8029f801>] ____pagevec_lru_add+0x151/0x170
Jan 11 17:05:30 leto [<ffffffff802b1f82>] exit_mmap+0xa2/0x170
Jan 11 17:05:30 leto [<ffffffff80247010>] mmput+0x30/0xd0
Jan 11 17:05:30 leto [<ffffffff8024b376>] exit_mm+0x106/0x140
Jan 11 17:05:30 leto [<ffffffff8024d0f4>] do_exit+0x114/0x8e0
Jan 11 17:05:30 leto [<ffffffff802ed15d>] vfs_fsync+0xbd/0x110
Jan 11 17:05:30 leto [<ffffffff8024d8f7>] do_group_exit+0x37/0xb0
Jan 11 17:05:30 leto [<ffffffff802135aa>] system_call_fastpath+0x16/0x1b

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Christophe Saout]

Hello,

> > > > However, after one page of output it then Oopses somewhere during
> > > > fsck.reiserfs in "grab_swap_tokens".
> > 
> > Hmm, I found this patch in the SuSE kernel where they just return from
> > grab_swap_tokens if current->mm is NULL and claim that it can happen
> > (?).
> 
> Urks, ok, that workaround is probably total BS.  It is supposed to help
> when the current process is a kernel thread, and not reiserfsck.  So the
> point really is how come that current->mm is NULL at that point.

Ok, a bit further in my analysis:

>From googling around, it seems current->mm == NULL can be okay, but if
this is the case, the kernel should never ever try to access user pages
and hence trigger a page fault.  However, this seems to be the case
here.

It happens during process exit of reiserfsck, in exit_mmap when
unlocking the pages (reiserfsck indeed "mlockall"s itself into the
kernel).

So how does he got so far as to try to allocate swap tokens?!
(at this points the memory usage isn't even close to requiring swapping
anyway)

The path where it goes wrong is something like:

exit_mmap -> munlock_vma_pages_all -> munlock_vma_pages_range
-> __mlock_vma_pages_range -> __get_user_pages -> handle_mm_fault
-> handle_pte_fault -> do_swap_page -> grab_swap_token

Cheers,
	Christophe

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Christophe Saout]

Hi,

ok, I can work around the issue with reiserfsck reported earlier by
disabling CONFIG_UNEVICTABLE_LRU.  His code is using __get_user_pages
during mm teardown and this seems to have problems when current->mm is
already gone, which seems can happen with Xen and, from what I gathered
from some of Linus' older emails, should be allowed to.  (so this might
be a bug in Nick's UNEVICTABLE_LRU version of the mlock code?)

Anyway,

> Also, the console locked up after ~2 minutes, but still reacted to
> Alt-SysRq-B.

this still occurs.  So far I havent been able to get the full crash
since I can't scroll, but from what I have on the screen it is hitting
the BUG() in arch/x86/include/asm/mmu_context_64.h in switch_mm:

#ifdef CONFIG_SMP
        else {   
                write_pda(mmu_state, TLBSTATE_OK);
                if (read_pda(active_mm) != next)  
-->                     BUG();
                if (!cpu_test_and_set(cpu, next->cpu_vm_mask)) {

Might be a secondary oops or something, not sure yet (still need to get
netconsole working).

Cheers,
	Christophe

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Christophe Saout]

Hi,

> this still occurs.  So far I havent been able to get the full crash
> since I can't scroll, but from what I have on the screen it is hitting
> the BUG() in arch/x86/include/asm/mmu_context_64.h in switch_mm:

I caught one with netconsole, it always seems to be the same (happening
in different places though):

------------[ cut here ]------------
kernel BUG at /home/chtephan/linux/arch/x86/include/asm/mmu_context_64.h:35!
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/devices/LNXSYSTM:00/device:00/PNP0A08:00/device:01/PNP0C09:00/PNP0C0A:00/power_supply/BAT0/energy_full
CPU 0 
Modules linked in: netconsole configfs iptable_mangle iptable_nat nf_nat ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_state xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc snd_pcm_oss snd_mixer_oss snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device ext3 jbd reiser4 lzo_decompress lzo_compress nf_conntrack_irc nf_conntrack_ftp nf_conntrack hdaps input_polldev tun arc4 ath5k snd_hda_codec_analog snd_hda_intel snd_hda_codec mac80211 snd_pcm snd_timer yenta_socket cfg80211 rsrc_nonstatic snd e1000e pcmcia_core ata_piix video uhci_hcd snd_page_alloc ehci_hcd output
Pid: 132, comm: kblockd/0 Not tainted 2.6.29-rc1-xen-cs1 #1
RIP: e030:[<ffffffff806778c9>]  [<ffffffff806778c9>] thread_return+0x521/0x6d8
RSP: e02b:ffff8800752a1de0  EFLAGS: 00010087
RAX: ffff880073c60680 RBX: ffff880001245280 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880075298590
RBP: ffff8800750dc2c0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff880072888680
R13: ffff880072888680 R14: 0000000000000000 R15: ffff8800750e9f00
FS:  00002ba52a2ce630(0000) GS:ffffffff80a31000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000024aefb8 CR3: 0000000050920000 CR4: 0000000000002620
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kblockd/0 (pid: 132, threadinfo ffff8800752a0000, task ffff880075298590)
Stack:
 ffffffff8021079b 0000000000000000 ffffffff8022eda9 0000000000000000
 ffffffff8022eda9 ffff880073da4e38 ffffffff8022eda9 ffffffff809fc280
 000000000000004f ffff880075129418 ffff880075298590 ffff8800752987d8
Call Trace:
 [<ffffffff8021079b>] ? xen_spin_lock+0xab/0xe0
 [<ffffffff8022eda9>] ? pvclock_clocksource_read+0x49/0x80
 [<ffffffff8022eda9>] ? pvclock_clocksource_read+0x49/0x80
 [<ffffffff8022eda9>] ? pvclock_clocksource_read+0x49/0x80
 [<ffffffff803db520>] ? blk_unplug_work+0x0/0x70
 [<ffffffff806797de>] ? _spin_lock_irqsave+0x2e/0x40
 [<ffffffff8025c42d>] ? worker_thread+0xdd/0x110
 [<ffffffff80260760>] ? autoremove_wake_function+0x0/0x30
 [<ffffffff8025c350>] ? worker_thread+0x0/0x110
 [<ffffffff8025c350>] ? worker_thread+0x0/0x110
 [<ffffffff80260337>] ? kthread+0x47/0x90
 [<ffffffff8021483a>] ? child_rip+0xa/0x20
 [<ffffffff8021412d>] ? retint_restore_args+0x5/0x20
 [<ffffffff80214830>] ? child_rip+0x0/0x20
Code: 40 48 89 74 24 48 e9 db fd ff ff 48 89 df e8 ff 89 b9 ff 90 57 ff 15 c7 ef 15 00 5f e9 52 fb ff ff e8 0c 22 00 00 e9 70 f8 f
RIP  [<ffffffff806778c9>] thread_return+0x521/0x6d8
 RSP <ffff8800752a1de0>
---[ end trace c482fecdeb931bd7 ]---

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Christophe Saout]

Hi again,

> > this still occurs.  So far I havent been able to get the full crash
> > since I can't scroll, but from what I have on the screen it is hitting
> > the BUG() in arch/x86/include/asm/mmu_context_64.h in switch_mm:
> 
> I caught one with netconsole, it always seems to be the same (happening
> in different places though):
> 
> ------------[ cut here ]------------
> kernel BUG at /home/chtephan/linux/arch/x86/include/asm/mmu_context_64.h:35!

Bryan Donlan bisected 2.6.29-rc1 and found that reverting commit
e97a630eb0f5b8b380fd67504de6cedebb489003 solves the problem (for me
too).

So, to summarize:

On my notebook (Core 2 Duo, SMP):
- 2.6.29-rc1 tip (with a lot of patch fiddling)
- dom0 working stable (AFAICT)
- CONFIG_EVICATBLE_LRU causes problems in exit_mmap
- CONFIG_PREEMPT seems generally broken with Xen
- AHCI works on SMP, has IRQ problem on UP
- all other hardware works
- X server won't come up (problem in "vgahw" init.), no BUG or Oops

Cheers,
	Christophe

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Pasi Kärkkäinen]

On Tue, Jan 13, 2009 at 10:28:08AM +0100, Christophe Saout wrote:
> Hi again,
> 
> > > this still occurs.  So far I havent been able to get the full crash
> > > since I can't scroll, but from what I have on the screen it is hitting
> > > the BUG() in arch/x86/include/asm/mmu_context_64.h in switch_mm:
> > 
> > I caught one with netconsole, it always seems to be the same (happening
> > in different places though):
> > 
> > ------------[ cut here ]------------
> > kernel BUG at /home/chtephan/linux/arch/x86/include/asm/mmu_context_64.h:35!
> 
> Bryan Donlan bisected 2.6.29-rc1 and found that reverting commit
> e97a630eb0f5b8b380fd67504de6cedebb489003 solves the problem (for me
> too).
> 
> So, to summarize:
> 
> On my notebook (Core 2 Duo, SMP):
> - 2.6.29-rc1 tip (with a lot of patch fiddling)
> - dom0 working stable (AFAICT)
> - CONFIG_EVICATBLE_LRU causes problems in exit_mmap
> - CONFIG_PREEMPT seems generally broken with Xen
> - AHCI works on SMP, has IRQ problem on UP
> - all other hardware works
> - X server won't come up (problem in "vgahw" init.), no BUG or Oops
> 

Nice work with finding the bugs!

Does VGA console work for you in the pv_ops dom0 kernel? 

I haven't been able to get it work.. serial console works OK for me. 

If it works for you, could you please post your grub.conf and kernel config?
What Xen version are you using? 

-- Pasi

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Christophe Saout]

Hi Pasi,

> > On my notebook (Core 2 Duo, SMP):
> > - 2.6.29-rc1 tip (with a lot of patch fiddling)
> > - dom0 working stable (AFAICT)
> > - CONFIG_EVICATBLE_LRU causes problems in exit_mmap
> > - CONFIG_PREEMPT seems generally broken with Xen
> > - AHCI works on SMP, has IRQ problem on UP
> > - all other hardware works
> > - X server won't come up (problem in "vgahw" init.), no BUG or Oops
>
> Nice work with finding the bugs!

Thanks.

> Does VGA console work for you in the pv_ops dom0 kernel? 

Yes, it does, without any kernel parameters by the way, otherwise I get
serial working but VGA stays blank.

The X server crashes in some I/O port "inb()" command.  Seems like the
io permissions aren't correctly passed through or something. (iopl or
ioperm or so)  I'll try to debug a bit more later.

> If it works for you, could you please post your grub.conf and kernel config?
> What Xen version are you using? 

Since 3.3.0 was working, I just sticked with that for now.
I am actually using extlinux, but that shouldn't really make a
difference:

LABEL linux-2.6.29-rc1-xen-cs1
MENU LABEL Gentoo + XEN-3.3.0 + 2.6.29-rc1-xen-cs1
KERNEL com32/mboot.c32
APPEND xen-3.3.0.gz --- vmlinux-2.6.29-rc1-xen-cs1.gz root=/dev/vg/root ro nopat

for serial console this (which I only tested under qemu):

LABEL linux-2.6.29-rc1-xen-cs1
MENU LABEL Gentoo + XEN-3.3.0 + 2.6.29-rc1-xen-cs1
KERNEL com32/mboot.c32
APPEND xen-3.3.0.gz console=com1,vga com1=9600,8n1 --- vmlinux-2.6.29-rc1-xen-cs1.gz earlyprintk=xen console=tty0 console=hvc0 root=/dev/vg/root ro nopat

Cheers,
	Christophe

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Christophe Saout]

Hello,

> The X server crashes in some I/O port "inb()" command.  Seems like the
> io permissions aren't correctly passed through or something. (iopl or
> ioperm or so)  I'll try to debug a bit more later.

The native Xen kernel has an arch/x86/kernel/ioport-xen.c which calls:

	HYPERVISOR_physdev_op(PHYSDEVOP_set_iobitmap, &set_iobitmap);

Is this a feature mabye still missing from the paravirtualized Xen?

If yes, what would it take?  A new paravirt call?

Thanks,
	Christophe

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Pasi Kärkkäinen]

On Tue, Jan 13, 2009 at 05:41:58PM +0100, Christophe Saout wrote:
> Hi Pasi,
> 
> > > On my notebook (Core 2 Duo, SMP):
> > > - 2.6.29-rc1 tip (with a lot of patch fiddling)
> > > - dom0 working stable (AFAICT)
> > > - CONFIG_EVICATBLE_LRU causes problems in exit_mmap
> > > - CONFIG_PREEMPT seems generally broken with Xen
> > > - AHCI works on SMP, has IRQ problem on UP
> > > - all other hardware works
> > > - X server won't come up (problem in "vgahw" init.), no BUG or Oops
> >
> > Nice work with finding the bugs!
> 
> Thanks.
> 
> > Does VGA console work for you in the pv_ops dom0 kernel? 
> 
> Yes, it does, without any kernel parameters by the way, otherwise I get
> serial working but VGA stays blank.
> 

Hmm.. I just tried my 2.6.28-rc8 kernel without console= and earlyprintk=
parameters (just like your config below), but it didn't work. 

I see Xen boot messages on the VGA console, and when the dom0 kernel should
start printing boot messages VGA console just goes blank.. Xen messages
disappear too. 

Serial console works just fine. 

> The X server crashes in some I/O port "inb()" command.  Seems like the
> io permissions aren't correctly passed through or something. (iopl or
> ioperm or so)  I'll try to debug a bit more later.
> 
> > If it works for you, could you please post your grub.conf and kernel config?
> > What Xen version are you using? 
> 
> Since 3.3.0 was working, I just sticked with that for now.
> I am actually using extlinux, but that shouldn't really make a
> difference:
> 
> LABEL linux-2.6.29-rc1-xen-cs1
> MENU LABEL Gentoo + XEN-3.3.0 + 2.6.29-rc1-xen-cs1
> KERNEL com32/mboot.c32
> APPEND xen-3.3.0.gz --- vmlinux-2.6.29-rc1-xen-cs1.gz root=/dev/vg/root ro nopat
> 
> for serial console this (which I only tested under qemu):
> 
> LABEL linux-2.6.29-rc1-xen-cs1
> MENU LABEL Gentoo + XEN-3.3.0 + 2.6.29-rc1-xen-cs1
> KERNEL com32/mboot.c32
> APPEND xen-3.3.0.gz console=com1,vga com1=9600,8n1 --- vmlinux-2.6.29-rc1-xen-cs1.gz earlyprintk=xen console=tty0 console=hvc0 root=/dev/vg/root ro nopat
> 

Yep. Thanks.

I've tried both Xen 3.3.0 and 3.3.1, both behave the same. I don't think the
(VGA) console problem is in Xen hypervisor.

-- Pasi

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Jeremy Fitzhardinge]

Christophe Saout wrote:
> Hello,
>
>   
>> The X server crashes in some I/O port "inb()" command.  Seems like the
>> io permissions aren't correctly passed through or something. (iopl or
>> ioperm or so)  I'll try to debug a bit more later.
>>     
>
> The native Xen kernel has an arch/x86/kernel/ioport-xen.c which calls:
>
> 	HYPERVISOR_physdev_op(PHYSDEVOP_set_iobitmap, &set_iobitmap);
>
> Is this a feature mabye still missing from the paravirtualized Xen?
>
> If yes, what would it take?  A new paravirt call?
>   

Hi Christophe,

Firstly, thanks a lot for trying out the pvops dom0 and doing so much 
debugging on it.  I'm still catching up on my rather daunting email 
backlog - including your torrent of messages - so feel free to remind me 
about/resent anything important which still needs looking at.

I haven't been brave enough to even attempt starting X; there's a lot of 
work to be done if you want to do anything beyond dumb framebuffer.

Anyway, in this case, I definitely haven't added this call, so it will 
need to be done somewhere.  Does it need to be called once at startup, 
or on every context switch?  There may well be a suitable place to hook 
this into already, but we can add a new pvop if its really needed.

    J

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Jeremy Fitzhardinge]

Pasi Kärkkäinen wrote:
> Yep. Thanks.
>
> I've tried both Xen 3.3.0 and 3.3.1, both behave the same. I don't think the
> (VGA) console problem is in Xen hypervisor.
>   

Does VGA console work if you boot that kernel native?  I suspect its 
something to do with your kernel config, since its quite easy to get 
wrong.  I would not, for example, expect an fbdev console to work.

    J

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Christophe Saout]

Hi Jeremy,

> > The native Xen kernel has an arch/x86/kernel/ioport-xen.c which calls:
> >
> > 	HYPERVISOR_physdev_op(PHYSDEVOP_set_iobitmap, &set_iobitmap);
> >
> > Is this a feature mabye still missing from the paravirtualized Xen?
> >
> > If yes, what would it take?  A new paravirt call?
>
> Firstly, thanks a lot for trying out the pvops dom0 and doing so much 
> debugging on it.

You're welcome.  It's not like I really need it, but it would simply
things a bit and I just love to play with bleeding edge stuff.

>   I'm still catching up on my rather daunting email 
> backlog - including your torrent of messages - so feel free to remind me 
> about/resent anything important which still needs looking at.

Don't worry.  As you might have noticed, the patches in patches.hg have
sort of fallen behind the tip tree.  I have forward-ported them to the
version from last weekend (no guarantee that I did not put a bug in
there, but at least it compiles).  It's in my git tree I was
referencing, but I didn't bother putting everything back into the patch
it belongs, that would need some fixing up.  In case you are planning on
having some of the patches merged upstream soon, you can take it from
there.  It's not really exciting stuff, mostly some cleanups which were
moving stuff around and such (and some iotlb dma things which were not
entirely trivial but pretty straight-forward).

> I haven't been brave enough to even attempt starting X; there's a lot of 
> work to be done if you want to do anything beyond dumb framebuffer.
> 
> Anyway, in this case, I definitely haven't added this call, so it will 
> need to be done somewhere.  Does it need to be called once at startup, 
> or on every context switch?  There may well be a suitable place to hook 
> this into already, but we can add a new pvop if its really needed.

Looking at the original Xen kernel the call has to be made on every
context switch if the io permissions change between the two threads.
In the past days I have sort of started trying to put some new paravirt
call together in the various places.  This unfortunately also meant
changing a bit of the native stuff to pass through a unified
"set_io_bitmap" call.  I haven't checked if that works, but at least the
Xen versions of the call doesn't get X up and running.  Now I get a
black screen and dead machine.  I guess I will have to do some
debugging. ;-)

If you're interested, I can send you what I have so far, but I guess you
have more pressing things to do first. :-)

Cheers,
	Christophe

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Jeremy Fitzhardinge]

Christophe Saout wrote:
> It's not really exciting stuff, mostly some cleanups which were
> moving stuff around and such (and some iotlb dma things which were not
> entirely trivial but pretty straight-forward).
>   

Yeah the iotlb stuff was in a bit of a flux, but I think I understand 
how it ended up.

>> I haven't been brave enough to even attempt starting X; there's a lot of 
>> work to be done if you want to do anything beyond dumb framebuffer.
>>
>> Anyway, in this case, I definitely haven't added this call, so it will 
>> need to be done somewhere.  Does it need to be called once at startup, 
>> or on every context switch?  There may well be a suitable place to hook 
>> this into already, but we can add a new pvop if its really needed.
>>     
>
> Looking at the original Xen kernel the call has to be made on every
> context switch if the io permissions change between the two threads.
> In the past days I have sort of started trying to put some new paravirt
> call together in the various places.  This unfortunately also meant
> changing a bit of the native stuff to pass through a unified
> "set_io_bitmap" call.  I haven't checked if that works, but at least the
> Xen versions of the call doesn't get X up and running.  Now I get a
> black screen and dead machine.  I guess I will have to do some
> debugging. ;-)
>   

There are several pvops calls made already during context switch, so I 
wouldn't be surprised if you can sneak it into one of those (load_sp0 
looks like a candidate, since it's already passed the tss).  Just make 
sure the hypercall gets batched with the rest of the context-switch 
multicall so that the latency doesn't get worse.

> If you're interested, I can send you what I have so far, but I guess you
> have more pressing things to do first. :-)
>   

Yeah, I'm working on rebasing the patch queue at the moment, then I'll 
try to catch up on the various bugs/observations that people have reported.

    J

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Pasi Kärkkäinen]

On Fri, Jan 16, 2009 at 09:26:04AM -0800, Jeremy Fitzhardinge wrote:
> Pasi Kärkkäinen wrote:
> >Yep. Thanks.
> >
> >I've tried both Xen 3.3.0 and 3.3.1, both behave the same. I don't think 
> >the
> >(VGA) console problem is in Xen hypervisor.
> >  
> 
> Does VGA console work if you boot that kernel native?  I suspect its 
> something to do with your kernel config, since its quite easy to get 
> wrong.  I would not, for example, expect an fbdev console to work.
> 

Booting the same kernel on baremetal without Xen works just fine, ie.
VGA console works on baremetal.

When I boot that same kernel as Xen dom0, I first see Xen bootup messages on
the console, and when dom0 kernel messages should show up on the console, 
the console gets cleared and gets blank. Nothing will show up on the console.

-- Pasi

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1020 bytes --]

Hi Jeremy,

> I haven't been brave enough to even attempt starting X; there's a lot of 
> work to be done if you want to do anything beyond dumb framebuffer.
> 
> Anyway, in this case, I definitely haven't added this call, so it will 
> need to be done somewhere.  Does it need to be called once at startup, 
> or on every context switch?  There may well be a suitable place to hook 
> this into already, but we can add a new pvop if its really needed.

Just to let you know.  I have this very preliminary, incomplete (x86_32
parts are missing) and barely tested patch (barely tested under Xen, not
checked if the native stuff works).  It seems to work on my test case,
with a program where I ioperm some I/O range and try to inb() some bytes
from it (segfaults when it should and/or returns values when it should),
and it also survives context switches.

However, the X server is now giving me a "Corrupted page table", which I
caught on netconsole and I have no clue how I should debug that one.

Cheers,
	Christophe


[-- Attachment #2: Type: text/x-patch, Size: 7482 bytes --]

diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h
index 8a976ea..40795f4 100644
--- a/arch/x86/include/asm/paravirt.h
+++ b/arch/x86/include/asm/paravirt.h
@@ -130,6 +130,8 @@ struct pv_cpu_ops {
 	void (*load_sp0)(struct tss_struct *tss, struct thread_struct *t);
 
 	void (*set_iopl_mask)(unsigned mask);
+	void (*set_io_bitmap)(struct thread_struct *thread,
+	                      int changed, unsigned long bytes_updated);
 
 	void (*wbinvd)(void);
 	void (*io_delay)(void);
@@ -908,6 +910,11 @@ static inline void set_iopl_mask(unsigned mask)
 {
 	PVOP_VCALL1(pv_cpu_ops.set_iopl_mask, mask);
 }
+static inline void set_io_bitmap(struct thread_struct *thread,
+                                 int changed, unsigned long bytes_updated)
+{
+	PVOP_VCALL3(pv_cpu_ops.set_io_bitmap, thread, changed, bytes_updated);
+}
 
 /* The paravirtualized I/O functions */
 static inline void slow_down_io(void)
diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index 091cd88..7ad072e 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -519,6 +519,9 @@ static inline void native_set_iopl_mask(unsigned mask)
 #endif
 }
 
+extern void native_set_io_bitmap(struct thread_struct *thread,
+                                 int changed, unsigned long updated_bytes);
+
 static inline void
 native_load_sp0(struct tss_struct *tss, struct thread_struct *thread)
 {
@@ -560,6 +563,7 @@ static inline void load_sp0(struct tss_struct *tss,
 }
 
 #define set_iopl_mask native_set_iopl_mask
+#define set_io_bitmap native_set_io_bitmap
 #endif /* CONFIG_PARAVIRT */
 
 /*
diff --git a/arch/x86/kernel/ioport.c b/arch/x86/kernel/ioport.c
index b12208f..5778936 100644
--- a/arch/x86/kernel/ioport.c
+++ b/arch/x86/kernel/ioport.c
@@ -30,14 +30,43 @@ static void set_bitmap(unsigned long *bitmap, unsigned int base,
 	}
 }
 
+void native_set_io_bitmap(struct thread_struct *t,
+                          int changed, unsigned long bytes_updated)
+{
+	unsigned long copy = bytes_updated;
+	struct tss_struct *tss;
+
+	if (!bytes_updated)
+		return;
+
+	tss = &__get_cpu_var(init_tss);
+
+#ifdef CONFIG_X86_32
+	/*
+	 * Sets the lazy trigger so that the next I/O operation will
+	 * reload the correct bitmap.
+	 * Reset the owner so that a process switch will not set
+	 * tss->io_bitmap_base to IO_BITMAP_OFFSET.
+	 */
+	tss->x86_tss.io_bitmap_base = INVALID_IO_BITMAP_OFFSET_LAZY;
+	tss->io_bitmap_owner = NULL;
+#else
+	/* Update the TSS: */
+	if (t->io_bitmap_ptr)
+		memcpy(tss->io_bitmap, t->io_bitmap_ptr, bytes_updated);
+	else
+		memset(tss->io_bitmap, 0xff, bytes_updated);
+#endif
+}
+
 /*
  * this changes the io permissions bitmap in the current task.
  */
 asmlinkage long sys_ioperm(unsigned long from, unsigned long num, int turn_on)
 {
 	struct thread_struct *t = &current->thread;
-	struct tss_struct *tss;
 	unsigned int i, max_long, bytes, bytes_updated;
+	int changed;
 
 	if ((from + num <= from) || (from + num > IO_BITMAP_BITS))
 		return -EINVAL;
@@ -58,16 +87,17 @@ asmlinkage long sys_ioperm(unsigned long from, unsigned long num, int turn_on)
 		memset(bitmap, 0xff, IO_BITMAP_BYTES);
 		t->io_bitmap_ptr = bitmap;
 		set_thread_flag(TIF_IO_BITMAP);
-	}
+		changed = 1;
+	} else
+		changed = 0;
 
 	/*
-	 * do it in the per-thread copy and in the TSS ...
-	 *
-	 * Disable preemption via get_cpu() - we must not switch away
+	 * do it in the per-thread copy
+	 *	 * Disable preemption - we must not switch away
 	 * because the ->io_bitmap_max value must match the bitmap
 	 * contents:
 	 */
-	tss = &per_cpu(init_tss, get_cpu());
+	preempt_disable();
 
 	set_bitmap(t->io_bitmap_ptr, from, num, !turn_on);
 
@@ -85,21 +115,9 @@ asmlinkage long sys_ioperm(unsigned long from, unsigned long num, int turn_on)
 
 	t->io_bitmap_max = bytes;
 
-#ifdef CONFIG_X86_32
-	/*
-	 * Sets the lazy trigger so that the next I/O operation will
-	 * reload the correct bitmap.
-	 * Reset the owner so that a process switch will not set
-	 * tss->io_bitmap_base to IO_BITMAP_OFFSET.
-	 */
-	tss->x86_tss.io_bitmap_base = INVALID_IO_BITMAP_OFFSET_LAZY;
-	tss->io_bitmap_owner = NULL;
-#else
-	/* Update the TSS: */
-	memcpy(tss->io_bitmap, t->io_bitmap_ptr, bytes_updated);
-#endif
+	set_io_bitmap(t, changed, bytes_updated);
 
-	put_cpu();
+	preempt_enable();
 
 	return 0;
 }
diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c
index 006cec4..602edc0 100644
--- a/arch/x86/kernel/paravirt.c
+++ b/arch/x86/kernel/paravirt.c
@@ -357,6 +357,7 @@ struct pv_cpu_ops pv_cpu_ops = {
 	.swapgs = native_swapgs,
 
 	.set_iopl_mask = native_set_iopl_mask,
+	.set_io_bitmap = native_set_io_bitmap,
 	.io_delay = native_io_delay,
 
 	.lazy_mode = {
diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c
index efb0396..a75d058 100644
--- a/arch/x86/kernel/process_64.c
+++ b/arch/x86/kernel/process_64.c
@@ -237,17 +237,13 @@ void exit_thread(void)
 	struct thread_struct *t = &me->thread;
 
 	if (me->thread.io_bitmap_ptr) {
-		struct tss_struct *tss = &per_cpu(init_tss, get_cpu());
-
+		preempt_disable();
 		kfree(t->io_bitmap_ptr);
 		t->io_bitmap_ptr = NULL;
 		clear_thread_flag(TIF_IO_BITMAP);
-		/*
-		 * Careful, clear this in the TSS too:
-		 */
-		memset(tss->io_bitmap, 0xff, t->io_bitmap_max);
+		set_io_bitmap(t, 1, t->io_bitmap_max);
 		t->io_bitmap_max = 0;
-		put_cpu();
+		preempt_enable();
 	}
 
 	ds_exit_thread(current);
@@ -513,6 +509,12 @@ static inline void __switch_to_xtra(struct task_struct *prev_p,
 			hard_enable_TSC();
 	}
 
+#if 1
+	if (test_tsk_thread_flag(next_p, TIF_IO_BITMAP) ||
+	    test_tsk_thread_flag(prev_p, TIF_IO_BITMAP))
+		set_io_bitmap(next_p, 1,
+		              max(prev->io_bitmap_max, next->io_bitmap_max));
+#else
 	if (test_tsk_thread_flag(next_p, TIF_IO_BITMAP)) {
 		/*
 		 * Copy the relevant range of the IO bitmap.
@@ -526,6 +528,7 @@ static inline void __switch_to_xtra(struct task_struct *prev_p,
 		 */
 		memset(tss->io_bitmap, 0xff, prev->io_bitmap_max);
 	}
+#endif
 }
 
 /*
@@ -556,6 +559,9 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)
 	 */
 	load_sp0(tss, next);
 
+	if (unlikely(prev->io_bitmap_ptr || next->io_bitmap_ptr))
+		set_io_bitmap(next, 1, prev->io_bitmap_max);
+
 	/*
 	 * Switch DS and ES.
 	 * This won't pick up thread selector changes, but I guess that is ok.
diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index e8a1e0a..a6cd15a 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -563,6 +563,21 @@ static void xen_set_iopl_mask(unsigned mask)
 	HYPERVISOR_physdev_op(PHYSDEVOP_set_iopl, &set_iopl);
 }
 
+static void xen_set_io_bitmap(struct thread_struct *thread,
+                              int changed, unsigned long bytes_updated)
+{
+	struct physdev_set_iobitmap set_iobitmap;
+
+	if (!changed)
+		return;
+
+	set_xen_guest_handle(set_iobitmap.bitmap,
+	                     (char *)current->thread.io_bitmap_ptr);
+	set_iobitmap.nr_ports = thread->io_bitmap_ptr ? IO_BITMAP_BITS : 0;
+	WARN_ON(HYPERVISOR_physdev_op(PHYSDEVOP_set_iobitmap,
+	                              &set_iobitmap));
+}
+
 static void xen_io_delay(void)
 {
 }
@@ -1286,6 +1301,7 @@ static const struct pv_cpu_ops xen_cpu_ops __initdata = {
 	.load_sp0 = xen_load_sp0,
 
 	.set_iopl_mask = xen_set_iopl_mask,
+	.set_io_bitmap = xen_set_io_bitmap,
 	.io_delay = xen_io_delay,
 
 	/* Xen takes care of %gs when switching to usermode for us */

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Christophe Saout]

Hi,

> > I haven't been brave enough to even attempt starting X; there's a lot of 
> > work to be done if you want to do anything beyond dumb framebuffer.
> > 
> > Anyway, in this case, I definitely haven't added this call, so it will 
> > need to be done somewhere.  Does it need to be called once at startup, 
> > or on every context switch?  There may well be a suitable place to hook 
> > this into already, but we can add a new pvop if its really needed.
> 
> Just to let you know.  I have this very preliminary, incomplete (x86_32
> parts are missing) and barely tested patch (barely tested under Xen, not
> checked if the native stuff works).  It seems to work on my test case,
> with a program where I ioperm some I/O range and try to inb() some bytes
> from it (segfaults when it should and/or returns values when it should),
> and it also survives context switches.

Eheh, while trying with gdb I found out that context switches do not
restore the bitmap correctly.  Guess what, two obvious and stupid
bugs. :-)  in enlighten.c:xen_set_io_bitmap it has to say "thread->"
instead of "current->thread." and in process_64.c the call has to pass
the "next" variable and not the "next_p" to set_io_bitmap (the thread,
not the task).  Then context switches actually do seem to work.

> However, the X server is now giving me a "Corrupted page table", which I
> caught on netconsole and I have no clue how I should debug that one.

Just checked, the machine isn't dead, just screen and keyboard, ssh
continues to work.  So, gdb'd it from ssh, and also removed the call to
"pgtable_bad" in arch/x86/mm/fault.c (which is triggered by the RSVD bit
being set on the page table, hmmmmm), so I get a segfault which gdb can
catch.

Ok, the X server first spits a strange message:
 (EE) RADEON(0): V_BIOS address 0x22c0 out of range
instead of
 (II) RADEON(0): Primary V_BIOS segment is: 0xc000
(in xf86int10GetBiosSegment)

and the later dies in the radeon driver with the bad memory / pagetable
access in radeon_card_posted() in what seems to be the fast attempt to
do an access to the card's MMIO area.

That base address of that one seems to be obtained by libpciaccess,
which, as far as I understand, digs around in /sys to get the relevant
PCI data and then maps it into memory, so I guess it is not related to
the warning message about the V_BIOS address.

I guess that X server is one of the few processes that try to map PCI
hardware MMIO ranges into memory, so perhaps there is also something
missing (especially since it triggers acceses to pages with the RSVD bit
set, something which should never happen under normal circumstances, if
I understand correctly).

Cheers,
	Christophe

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Christophe Saout]

Hi again,

> and the later dies in the radeon driver with the bad memory / pagetable
> access in radeon_card_posted() in what seems to be the fast attempt to
> do an access to the card's MMIO area.

Uhoh,

diff -u ioremap.c ioremap-xen.c

suggests there is some more work to do. :-(

The native Xen kernel defines io_remap_pfn_range to
direct_remap_pfn_range, whereas !XEN defines it to remap_pfn_range
(which is also the case for the pv_ops kernel at the moment)

I guess that makes io_remap_pfn_range another candidate for a new
paravirt op.

However, in direct_remap_pfn_range() the first two lines are:

       if (xen_feature(XENFEAT_auto_translated_physmap))
               return remap_pfn_range(vma, address, mfn, size, prot);

I guess this is not the case on my system / hypervisor?

If this really has to be implemented, what about this in asm/mmu.h:

typedef struct {
...
#ifdef CONFIG_XEN
        unsigned has_foreign_mappings:1;
#endif
...
} mm_context_t;

There is some logic in the native Xen kernel which prevents a call to
mm_unpin if there are "foreign mappings", i.e. set by
direct_remap_pfn_range.  What's up with that?


Note that I noticed a completely unrelated issue:  After some minutes it
can happen that the AHCI goes dead on me.  Hard disk accesses hang and
in the log there are several tries to revive the controller (resets and
errors) until the kernel at some point decides to panic.

Cheers,
	Christophe

Re: Xen with dom0 pvops on ultra-recent "git tip"kernel on x86_64

[Keir Fraser]

On 18/01/2009 21:28, "Christophe Saout" <christophe@saout.de> wrote:

> However, in direct_remap_pfn_range() the first two lines are:
> 
>        if (xen_feature(XENFEAT_auto_translated_physmap))
>                return remap_pfn_range(vma, address, mfn, size, prot);
> 
> I guess this is not the case on my system / hypervisor?

You can ignore auto_translated_physmap. It's not a supported mode any more.

> There is some logic in the native Xen kernel which prevents a call to
> mm_unpin if there are "foreign mappings", i.e. set by
> direct_remap_pfn_range.  What's up with that?

It's subtle. Whether it is needed depends on whether the pv_ops kernel
detects mappings of non-reference-counted memory by putting a flag in the
relevant PTEs, or by the more subtle means used in our 2.6.18 kernel (see
include/asm-i386/mach-xen/asm/maddr.h:mfn_to_local_pfn() and the comment
above it). In the former case the flag would not be needed, and I think that
is what Jeremy implemented.

The flag would possibly still be needed for grant mappings in user space
though (i.e., mappings made via the gntdev device driver, or by the blktap
device driver).

 -- Keir

Re: Xen with dom0 pvops on ultra-recent "git tip"kernel on x86_64

[Christophe Saout]

Hi Keir,

> > There is some logic in the native Xen kernel which prevents a call to
> > mm_unpin if there are "foreign mappings", i.e. set by
> > direct_remap_pfn_range.  What's up with that?
> 
> It's subtle. Whether it is needed depends on whether the pv_ops kernel
> detects mappings of non-reference-counted memory by putting a flag in the
> relevant PTEs, or by the more subtle means used in our 2.6.18 kernel (see
> include/asm-i386/mach-xen/asm/maddr.h:mfn_to_local_pfn() and the comment
> above it). In the former case the flag would not be needed, and I think that
> is what Jeremy implemented.

Ok, I think I get it.  I do however not have the big picture, so I'm
trying the "try and fail and learn" approach right now.  There are some
subtle differences between the native and the pvops kernel regarding
definition of some pte macros/calls which are causing errors when I just
copy & paste some of the code over.  Let's see if I can get it to work.

There are some parts of the code doing early io remappings in the kernel
which are very helpful, also a bunch of checks in the necessary places
are already there which is definitely helpful.

> The flag would possibly still be needed for grant mappings in user space
> though (i.e., mappings made via the gntdev device driver, or by the blktap
> device driver).

I think this part has been deliberately left out, I'll leave that to
Jeremy. :-)

Thanks,
	Christophe

Re: Xen with dom0 pvops on ultra-recent "git tip"kernel on x86_64

[Christophe Saout]

Hi again,

ok, to my astonishment this seem to be actually working.

My X server is coming up and seems usable (until my hard disk stops
working after some minutes again :-)).

Here's the patch that's doing the trick for me (the Xen implementation
is mostly copied over from the 2.6.27 SuSE Xen kernel):


Subject: [PATCH] Make io_remap_pfn_range a paravirt operation and have a special
 implementation for it in Xen.

---
 arch/x86/include/asm/paravirt.h   |   12 ++++
 arch/x86/include/asm/pgtable_32.h |    4 ++
 arch/x86/include/asm/pgtable_64.h |    4 ++
 arch/x86/kernel/paravirt.c        |    1 +
 arch/x86/xen/enlighten.c          |    1 +
 arch/x86/xen/mmu.c                |  103 +++++++++++++++++++++++++++++++++++++
 arch/x86/xen/mmu.h                |    4 ++
 7 files changed, 129 insertions(+), 0 deletions(-)

diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h
index 40795f4..d2a7298 100644
--- a/arch/x86/include/asm/paravirt.h
+++ b/arch/x86/include/asm/paravirt.h
@@ -39,6 +39,7 @@ struct desc_ptr;
 struct tss_struct;
 struct mm_struct;
 struct desc_struct;
+struct vm_area_struct;
 
 /* general info */
 struct pv_info {
@@ -326,6 +327,10 @@ struct pv_mmu_ops {
 			   unsigned long phys, pgprot_t flags);
 
 	int (*page_is_ram)(unsigned long pfn);
+
+	int (*io_remap_pfn_range)(struct vm_area_struct *vma,
+	                          unsigned long address, unsigned long mfn,
+	                          unsigned long size, pgprot_t prot);
 };
 
 struct raw_spinlock;
@@ -1402,6 +1407,13 @@ static inline int page_is_ram(unsigned long pfn)
 	return PVOP_CALL1(int, pv_mmu_ops.page_is_ram, pfn);
 }
 
+static inline int io_remap_pfn_range(struct vm_area_struct *vma,
+                                     unsigned long vaddr, unsigned long pfn,
+                                     unsigned long size, pgprot_t prot)
+{
+	return pv_mmu_ops.io_remap_pfn_range(vma, vaddr, pfn, size, prot);
+}
+
 void _paravirt_nop(void);
 #define paravirt_nop	((void *)_paravirt_nop)
 
diff --git a/arch/x86/include/asm/pgtable_32.h b/arch/x86/include/asm/pgtable_32.h
index 56e1560..ab2419b 100644
--- a/arch/x86/include/asm/pgtable_32.h
+++ b/arch/x86/include/asm/pgtable_32.h
@@ -182,7 +182,11 @@ do {						\
 #define kern_addr_valid(kaddr)	(0)
 #endif
 
+#ifdef CONFIG_PARAVIRT
+#include <asm/paravirt.h>
+#else
 #define io_remap_pfn_range(vma, vaddr, pfn, size, prot)	\
 	remap_pfn_range(vma, vaddr, pfn, size, prot)
+#endif
 
 #endif /* _ASM_X86_PGTABLE_32_H */
diff --git a/arch/x86/include/asm/pgtable_64.h b/arch/x86/include/asm/pgtable_64.h
index 537081e..293570f 100644
--- a/arch/x86/include/asm/pgtable_64.h
+++ b/arch/x86/include/asm/pgtable_64.h
@@ -272,8 +272,12 @@ extern int direct_gbpages;
 extern int kern_addr_valid(unsigned long addr);
 extern void cleanup_highmap(void);
 
+#ifdef CONFIG_PARAVIRT
+#include <asm/paravirt.h>
+#else
 #define io_remap_pfn_range(vma, vaddr, pfn, size, prot)	\
 	remap_pfn_range(vma, vaddr, pfn, size, prot)
+#endif
 
 #define HAVE_ARCH_UNMAPPED_AREA
 #define HAVE_ARCH_UNMAPPED_AREA_TOPDOWN
diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c
index 602edc0..81c2e15 100644
--- a/arch/x86/kernel/paravirt.c
+++ b/arch/x86/kernel/paravirt.c
@@ -453,6 +453,7 @@ struct pv_mmu_ops pv_mmu_ops = {
 
 	.set_fixmap = native_set_fixmap,
 	.page_is_ram = native_page_is_ram,
+	.io_remap_pfn_range = remap_pfn_range
 };
 
 EXPORT_SYMBOL_GPL(pv_time_ops);
diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 360f04b..643d00f 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -1401,6 +1401,7 @@ static const struct pv_mmu_ops xen_mmu_ops __initdata = {
 
 	.set_fixmap = xen_set_fixmap,
 	.page_is_ram = xen_page_is_ram,
+	.io_remap_pfn_range = xen_io_remap_pfn_range
 };
 
 static void xen_reboot(int reason)
diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index 72069b1..d0a9348 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -1441,6 +1441,109 @@ void xen_destroy_contiguous_region(unsigned long vstart, unsigned int order)
 }
 EXPORT_SYMBOL_GPL(xen_destroy_contiguous_region);
 
+static inline pte_t pfn_pte_ma(unsigned long page_nr, pgprot_t pgprot)
+{
+	pgprotval_t prot = pgprot_val(pgprot);
+
+	if (prot & _PAGE_PRESENT)
+		prot &= __supported_pte_mask;
+	return __pte_ma(((phys_addr_t)page_nr << PAGE_SHIFT) | prot);
+}
+                                        
+static int direct_remap_area_pte_fn(pte_t *pte,
+				    struct page *pmd_page,
+				    unsigned long address,
+				    void *data)
+{
+	struct mmu_update **v = (struct mmu_update **)data;
+
+	BUG_ON(!pte_none(*pte));
+
+	(*v)->ptr = ((u64)pfn_to_mfn(page_to_pfn(pmd_page)) <<
+		     PAGE_SHIFT) | ((unsigned long)pte & ~PAGE_MASK);
+	(*v)++;
+
+	return 0;
+}
+
+static int __direct_remap_pfn_range(struct mm_struct *mm,
+				    unsigned long address,
+				    unsigned long mfn,
+				    unsigned long size,
+				    pgprot_t prot,
+				    domid_t  domid)
+{
+	int rc;
+	unsigned long i, start_address;
+	struct mmu_update *u, *v, *w;
+
+	u = v = w = (struct mmu_update *)__get_free_page(GFP_KERNEL|__GFP_REPEAT);
+	if (u == NULL)
+		return -ENOMEM;
+
+	start_address = address;
+
+	flush_cache_all();
+
+	for (i = 0; i < size; i += PAGE_SIZE) {
+		if ((v - u) == (PAGE_SIZE / sizeof(struct mmu_update))) {
+			/* Flush a full batch after filling in the PTE ptrs. */
+			rc = apply_to_page_range(mm, start_address,
+						 address - start_address,
+						 direct_remap_area_pte_fn, &w);
+			if (rc)
+				goto out;
+			rc = -EFAULT;
+			if (HYPERVISOR_mmu_update(u, v - u, NULL, domid) < 0)
+				goto out;
+			v = w = u;
+			start_address = address;
+		}
+
+		/*
+		 * Fill in the machine address: PTE ptr is done later by
+		 * apply_to_page_range().
+		 */
+                v->val = pfn_pte_ma(mfn, prot).pte
+                         | _PAGE_SPECIAL | _PAGE_IOMAP;
+
+		mfn++;
+		address += PAGE_SIZE;
+		v++;
+	}
+
+	if (v != u) {
+		/* Final batch. */
+		rc = apply_to_page_range(mm, start_address,
+					 address - start_address,
+					 direct_remap_area_pte_fn, &w);
+		if (rc)
+			goto out;
+		rc = -EFAULT;
+		if (unlikely(HYPERVISOR_mmu_update(u, v - u, NULL, domid) < 0))
+			goto out;
+	}
+
+	rc = 0;
+
+ out:
+	flush_tlb_all();
+
+	free_page((unsigned long)u);
+
+	return rc;
+}
+
+int xen_io_remap_pfn_range(struct vm_area_struct *vma,
+                           unsigned long address, unsigned long mfn,
+                           unsigned long size, pgprot_t prot)
+{
+	vma->vm_flags |= VM_IO | VM_RESERVED | VM_PFNMAP;
+
+	return __direct_remap_pfn_range(
+		vma->vm_mm, address, mfn, size, prot, DOMID_IO);
+}
+
 #ifdef CONFIG_XEN_DEBUG_FS
 
 static struct dentry *d_mmu_debug;
diff --git a/arch/x86/xen/mmu.h b/arch/x86/xen/mmu.h
index 98d7165..e1c8321 100644
--- a/arch/x86/xen/mmu.h
+++ b/arch/x86/xen/mmu.h
@@ -54,4 +54,8 @@ pte_t xen_ptep_modify_prot_start(struct mm_struct *mm, unsigned long addr, pte_t
 void  xen_ptep_modify_prot_commit(struct mm_struct *mm, unsigned long addr,
 				  pte_t *ptep, pte_t pte);
 
+int xen_io_remap_pfn_range(struct vm_area_struct *vma,
+                           unsigned long address, unsigned long mfn,
+                           unsigned long size, pgprot_t prot);
+
 #endif	/* _XEN_MMU_H */
-- 
1.6.1

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Jeremy Fitzhardinge]

Christophe Saout wrote:
> Hi again,
>
>   
>> and the later dies in the radeon driver with the bad memory / pagetable
>> access in radeon_card_posted() in what seems to be the fast attempt to
>> do an access to the card's MMIO area.
>>     
>
> Uhoh,
>
> diff -u ioremap.c ioremap-xen.c
>
> suggests there is some more work to do. :-(
>
> The native Xen kernel defines io_remap_pfn_range to
> direct_remap_pfn_range, whereas !XEN defines it to remap_pfn_range
> (which is also the case for the pv_ops kernel at the moment)
>
> I guess that makes io_remap_pfn_range another candidate for a new
> paravirt op.
>   

Well, I'm taking a different approach for this.  The problem is that 
there's two distinct address spaces: the dom0 pseudo-physical address 
space, and the machine's real physical address space.  When you're 
mapping devices, you need to map the real machine addresses rather than 
the pseudo-phys ones.  When you map with PAGE_IOMAP then the existing 
Xen pte-setup pvops will do the appropriate things to set up a hardware 
mapping.  Therefore, to map devices from userspace, you just need to 
make sure that PAGE_IOMAP gets set appropriately - which is the tricky 
question.

> If this really has to be implemented, what about this in asm/mmu.h:
>
> typedef struct {
> ...
> #ifdef CONFIG_XEN
>         unsigned has_foreign_mappings:1;
> #endif
> ...
> } mm_context_t;
>
> There is some logic in the native Xen kernel which prevents a call to
> mm_unpin if there are "foreign mappings", i.e. set by
> direct_remap_pfn_range.  What's up with that?
>   

You have to remove any foreign mappings (ie, established with a grant 
table operation) before unpinning.  My plan is to do something a bit 
different from this, by using a shadow pagetable to keep track of the 
grant references, and use a page flag in the pgd when there's a grant 
reference present in a pagetable (which is uncommon).

> Note that I noticed a completely unrelated issue:  After some minutes it
> can happen that the AHCI goes dead on me.  Hard disk accesses hang and
> in the log there are several tries to revive the controller (resets and
> errors) until the kernel at some point decides to panic.
>   

Yes, that's an open mystery.  For me AHCI fails immediately (fails to 
probe drives).  Others have varying degrees of success.  Several minutes 
uptime is very good.

    J

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Pasi Kärkkäinen]

On Tue, Jan 20, 2009 at 12:08:44PM -0800, Jeremy Fitzhardinge wrote:
> 
> >Note that I noticed a completely unrelated issue:  After some minutes it
> >can happen that the AHCI goes dead on me.  Hard disk accesses hang and
> >in the log there are several tries to revive the controller (resets and
> >errors) until the kernel at some point decides to panic.
> >  
> 
> Yes, that's an open mystery.  For me AHCI fails immediately (fails to 
> probe drives).  Others have varying degrees of success.  Several minutes 
> uptime is very good.
> 

I can see the same problem with ata_piix driver too.. it fails to probe
drives. 

If you have any tips about how to debug that feel free to post them to the
other thread about ata_piix problems.. I'm happy to (try to) help. 

-- Pasi

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Andrew Lyon]

On Tue, Jan 20, 2009 at 9:27 PM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> On Tue, Jan 20, 2009 at 12:08:44PM -0800, Jeremy Fitzhardinge wrote:
>>
>> >Note that I noticed a completely unrelated issue:  After some minutes it
>> >can happen that the AHCI goes dead on me.  Hard disk accesses hang and
>> >in the log there are several tries to revive the controller (resets and
>> >errors) until the kernel at some point decides to panic.
>> >
>>
>> Yes, that's an open mystery.  For me AHCI fails immediately (fails to
>> probe drives).  Others have varying degrees of success.  Several minutes
>> uptime is very good.
>>
>
> I can see the same problem with ata_piix driver too.. it fails to probe
> drives.
>
> If you have any tips about how to debug that feel free to post them to the
> other thread about ata_piix problems.. I'm happy to (try to) help.
>
> -- Pasi
>

I tried pv_ops dom0 a while ago and had the same problems with ahci
and ata_piix, see thread
http://www.nabble.com/pv_ops-dom0-USB-fixed-td20943120.html , I
updated my tree today and tried again on the same system and it fails
in the same way.

I tried using a usb stick for the root filesystem but that too is
broken, I think interrupts are not getting through at all but I don't
know of a way to check that without a working userspace.

Is there anything more I can do to assist in finding the cause of this
problem? I can do more testing this week, and even give remote access
to the test system with serial console if a developer would like to
have a look.

Andy

Re: Xen with dom0 pvops on ultra-recent "git tip" kernel on x86_64

[Adam Wendt]

[-- Attachment #1.1: Type: text/plain, Size: 3806 bytes --]

Here is kernel crash running pvops on x86-64 on 2.6.29-rc1:

[ 1396.821933] ------------[ cut here ]------------
[ 1396.822062] kernel BUG at arch/x86/xen/enlighten.c:659!
[ 1396.822179] invalid opcode: 0000 [#1] SMP
[ 1396.822255] last sysfs file:
/sys/devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0/scsi_level
[ 1396.822255] CPU 1
[ 1396.822255] Modules linked in: netconsole [last unloaded: netconsole]
[ 1396.822255] Pid: 372, comm: kswapd0 Not tainted 2.6.29-rc1-tip #9
[ 1396.822255] RIP: e030:[<ffffffff8102375d>]  [<ffffffff8102375d>]
xen_flush_tlb_others+0x1a/0xbb
[ 1396.822255] RSP: e02b:ffff88001f72fb50  EFLAGS: 00010202
[ 1396.822255] RAX: 0000000000000001 RBX: ffff88001e4363c0 RCX:
ffffe200005fb1e0
[ 1396.822255] RDX: 00002ab00db41000 RSI: ffff88001e4363c0 RDI:
ffff88001e436668
[ 1396.822255] RBP: ffff88001e436668 R08: ffffffff814b045f R09:
ffff88001e436668
[ 1396.822255] R10: ffffffff8104fb85 R11: ffffffff8112280c R12:
00002ab00db41000
[ 1396.822255] R13: ffff88001e4363c0 R14: ffff88001f72fc1c R15:
0000000000000000
[ 1396.822255] FS:  00002b065606d6f0(0000) GS:ffff88001fce3580(0000)
knlGS:0000000000000000
[ 1396.822255] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1396.822255] CR2: 00000000028ef000 CR3: 0000000001001000 CR4:
0000000000002660
[ 1396.822255] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 1396.822255] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 1396.822255] Process kswapd0 (pid: 372, threadinfo ffff88001f72e000, task
ffff88001f47c410)
[ 1396.822255] Stack:
[ 1396.822255]  0000000000000001 ffffffff811d9d4b ffffe200005fb1e0
ffffffff811d9dbc
[ 1396.822255]  ffff88001e436668 00002ab00db41000 ffff88001e4363c0
ffffffff8103aa3a
[ 1396.822255]  00000000ffffffff 00002ab00db41000 ffff88001b5f98f0
ffffffff8104aa4f
[ 1396.822255] Call Trace:
[ 1396.822255]  [<ffffffff811d9d4b>] ? cpumask_next+0x19/0x1b
[ 1396.822255]  [<ffffffff811d9dbc>] ? cpumask_any_but+0x1e/0x2b
[ 1396.822255]  [<ffffffff8103aa3a>] ? flush_tlb_page+0x6c/0x72
[ 1396.822255]  [<ffffffff8104aa4f>] ? ptep_clear_flush_young+0x20/0x27
[ 1396.822255]  [<ffffffff810b9810>] ? page_referenced_one+0x66/0xca
[ 1396.822255]  [<ffffffff810ba3e3>] ? page_referenced+0x72/0xe9
[ 1396.822255]  [<ffffffff810a90b3>] ? shrink_list+0x454/0x476
[ 1396.822255]  [<ffffffff810a84d9>] ? shrink_active_list+0x17f/0x35d
[ 1396.822255]  [<ffffffff810a6dfa>] ? __pagevec_release+0x19/0x22
[ 1396.822255]  [<ffffffff810a86a5>] ? shrink_active_list+0x34b/0x35d
[ 1396.822255]  [<ffffffff810a93cc>] ? shrink_zone+0x2f7/0x30e
[ 1396.822255]  [<ffffffff814b0563>] ? _spin_unlock_irqrestore+0x14/0x17
[ 1396.822255]  [<ffffffff810aa1ae>] ? kswapd+0x34f/0x48f
[ 1396.822255]  [<ffffffff810a7bcb>] ? isolate_pages_global+0x0/0x1a7
[ 1396.822255]  [<ffffffff8106c92f>] ? autoremove_wake_function+0x0/0x2e
[ 1396.822255]  [<ffffffff8104eab7>] ? __wake_up_common+0x3f/0x72
[ 1396.822255]  [<ffffffff810a9e5f>] ? kswapd+0x0/0x48f
[ 1396.822255]  [<ffffffff810a9e5f>] ? kswapd+0x0/0x48f
[ 1396.822255]  [<ffffffff8106c7de>] ? kthread+0x47/0x75
[ 1396.822255]  [<ffffffff8102b2ba>] ? child_rip+0xa/0x20
[ 1396.822255]  [<ffffffff8102abad>] ? retint_restore_args+0x5/0x20
[ 1396.822255]  [<ffffffff8102b2b0>] ? child_rip+0x0/0x20
[ 1396.822255] Code: 00 00 00 48 c7 40 10 00 00 00 00 48 83 c4 28 eb 99 41
54 49 89 d4 55 48 89 fd 53 48 89 f3 48 83 ec 20 e8 2f ff ff ff 84 c0 74 04
<0f> 0b eb fe 48 85 db 75 04 0f 0b eb fe bf 20 00 00 00 e8 82 fe
[ 1396.822255] RIP  [<ffffffff8102375d>] xen_flush_tlb_others+0x1a/0xbb
[ 1396.822255]  RSP <ffff88001f72fb50>
[ 1396.833108] ---[ end trace b1dc69f7f3b7ebee ]---
[ 1396.833316] kswapd0 used greatest stack depth: 4520 bytes left

Let me know if there's anything else I can do to help test this.

Adam
Adam Wendt Consulting

[-- Attachment #1.2: Type: text/html, Size: 4342 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Xen with dom0 pvops on ultra-recent "git tip"kernel on x86_64

[Jeremy Fitzhardinge]

Christophe Saout wrote:
> Hi again,
>
> ok, to my astonishment this seem to be actually working.
>
> My X server is coming up and seems usable (until my hard disk stops
> working after some minutes again :-)).
>
> Here's the patch that's doing the trick for me (the Xen implementation
> is mostly copied over from the 2.6.27 SuSE Xen kernel):
>   

Thanks for your work on this.  In the last couple of days I've taken 
these patches and filled them out a bit.  I'm now getting full 
accelerated 3D with the Intel chipset on both my laptop and desktop 
machines.  Look at the xen/dom0/drm branch (its all merged into hackery 
as well).

    J

Re: Xen with dom0 pvops on ultra-recent "git tip"kernel on x86_64

[Christophe Saout]

Hi Jeremy,

> > ok, to my astonishment this seem to be actually working.
> >
> > My X server is coming up and seems usable (until my hard disk stops
> > working after some minutes again :-)).
> >
> > Here's the patch that's doing the trick for me (the Xen implementation
> > is mostly copied over from the 2.6.27 SuSE Xen kernel):
>
> Thanks for your work on this.  In the last couple of days I've taken 
> these patches and filled them out a bit.  I'm now getting full 
> accelerated 3D with the Intel chipset on both my laptop and desktop 
> machines.  Look at the xen/dom0/drm branch (its all merged into hackery 
> as well).

You're welcome.  Thanks for cleaning it up.  Unfortunately work has
caught up with me after a few calm days, so I haven't been able to
follow up on it much.  I'll give your tree a try though.


	Christophe

Re: Xen with dom0 pvops on ultra-recent "git tip"kernel on x86_64

[Christophe Saout]

Hi again,

> > > ok, to my astonishment this seem to be actually working.
> > >
> > > My X server is coming up and seems usable (until my hard disk stops
> > > working after some minutes again :-)).
> > >
> > > Here's the patch that's doing the trick for me (the Xen implementation
> > > is mostly copied over from the 2.6.27 SuSE Xen kernel):
> >
> > Thanks for your work on this.  In the last couple of days I've taken 
> > these patches and filled them out a bit.  I'm now getting full 
> > accelerated 3D with the Intel chipset on both my laptop and desktop 
> > machines.  Look at the xen/dom0/drm branch (its all merged into hackery 
> > as well).
> 
> You're welcome.  Thanks for cleaning it up.  Unfortunately work has
> caught up with me after a few calm days, so I haven't been able to
> follow up on it much.  I'll give your tree a try though.

Wow.  Booted, everything has come up (including compiz on my notebook
radeon) and looks fine so far.

Very nice, congratulations and thank you.


	Christophe

Re: Xen with dom0 pvops on ultra-recent "git tip"kernel on x86_64

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 1680 bytes --]

X-Server fails to start on GeForce 8500 GT
Attempting "startx" i get a trace back.
Does it mean , that a proper NVIDIA driver is just not available for the most recent
2.6.29-rc5 ?

--- On Tue, 2/24/09, Christophe Saout <christophe@saout.de> wrote:
From: Christophe Saout <christophe@saout.de>
Subject: Re: [Xen-devel] Xen with dom0 pvops on ultra-recent "git tip"kernel on x86_64
To: "Jeremy Fitzhardinge" <jeremy@goop.org>
Cc: "xen-devel" <xen-devel@lists.xensource.com>
Date: Tuesday, February 24, 2009, 5:41 AM

Hi again,

> > > ok, to my astonishment this seem to be actually working.
> > >
> > > My X server is coming up and seems usable (until my hard disk
stops
> > > working after some minutes again :-)).
> > >
> > > Here's the patch that's doing the trick for me (the Xen
implementation
> > > is mostly copied over from the 2.6.27 SuSE Xen kernel):
> >
> > Thanks for your work on this.  In the last couple of days I've
taken 
> > these patches and filled them out a bit.  I'm now getting full 
> > accelerated 3D with the Intel chipset on both my laptop and desktop 
> > machines.  Look at the xen/dom0/drm branch (its all merged into
hackery 
> > as well).
> 
> You're welcome.  Thanks for cleaning it up.  Unfortunately work has
> caught up with me after a few calm days, so I haven't been able to
> follow up on it much.  I'll give your tree a try though.

Wow.  Booted, everything has come up (including compiz on my notebook
radeon) and looks fine so far.

Very nice, congratulations and thank you.


	Christophe



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      

[-- Attachment #1.2: Type: text/html, Size: 2223 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Xen with dom0 pvops on ultra-recent "git tip"kernel on x86_64

[Jeremy Fitzhardinge]

Boris Derzhavets wrote:
> X-Server fails to start on GeForce 8500 GT
> Attempting "startx" i get a trace back.
> Does it mean , that a proper NVIDIA driver is just not available for 
> the most recent
> 2.6.29-rc5 ?
>

Are you using the nvidia binary driver?  I would be surprised if it 
worked out of the box; if changes need to be made to the binary parts, 
it may not be fixable at all.

    J

Re: Xen with dom0 pvops on ultra-recent "git tip"kernel on x86_64

[Andrew Lyon]

On Tue, Feb 24, 2009 at 9:22 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
> Boris Derzhavets wrote:
>>
>> X-Server fails to start on GeForce 8500 GT
>> Attempting "startx" i get a trace back.
>> Does it mean , that a proper NVIDIA driver is just not available for the
>> most recent
>> 2.6.29-rc5 ?
>>
>
> Are you using the nvidia binary driver?  I would be surprised if it worked
> out of the box; if changes need to be made to the binary parts, it may not
> be fixable at all.
>
>   J

I believe the current driver works with Xen because somebody outside
of nvidia contributed a patch, I use the nvidia binary driver with Xen
3.2.1 and my custom dom0 kernel which is 2.6.27.10 patched with
openSUSE Xen patches rebased to apply to vanilla, it works very well
but when pv_ops is merged into mainline the nvidia driver will
probably work when it is running on the bare metal but not under Xen
:(.

Having said that the patch for Xen support only touched the open
source parts of the driver, as soon as the current problems with hvm
on pv_ops are fixed I will be testing pv_ops and nvidia driver and I
will attempt to fix any problems, my C skills are improving so I might
be able to figure it out.

Andy

>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Xen with dom0 pvops on ultra-recent "git tip"kernel on x86_64

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 949 bytes --]

I've tried to install 180.22 Nvidia Driver. No luck

--- On Tue, 2/24/09, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
From: Jeremy Fitzhardinge <jeremy@goop.org>
Subject: Re: [Xen-devel] Xen with dom0 pvops on ultra-recent "git tip"kernel on x86_64
To: bderzhavets@yahoo.com
Cc: "xen-devel" <xen-devel@lists.xensource.com>, "Christophe Saout" <christophe@saout.de>
Date: Tuesday, February 24, 2009, 4:22 PM

Boris Derzhavets wrote:
> X-Server fails to start on GeForce 8500 GT
> Attempting "startx" i get a trace back.
> Does it mean , that a proper NVIDIA driver is just not available for the
most recent
> 2.6.29-rc5 ?
> 

Are you using the nvidia binary driver?  I would be surprised if it worked out
of the box; if changes need to be made to the binary parts, it may not be
fixable at all.

   J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      

[-- Attachment #1.2: Type: text/html, Size: 1326 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Xen with dom0 pvops on ultra-recent "git tip"kernel on x86_64

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 2299 bytes --]

>I use the nvidia binary driver with Xen
>3.2.1 and my custom dom0 kernel which is 2.6.27.10 patched with
>openSUSE Xen patches rebased to apply to vanilla

Suse's 2.6.27 kernel under Xen-Unstable (and 3.3.X) seems to be self sufficient for X-server startup. It's my current environment . No nvidia drivers have been installed at all.

--- On Tue, 2/24/09, Andrew Lyon <andrew.lyon@gmail.com> wrote:
From: Andrew Lyon <andrew.lyon@gmail.com>
Subject: Re: [Xen-devel] Xen with dom0 pvops on ultra-recent "git tip"kernel  on x86_64
To: "Jeremy Fitzhardinge" <jeremy@goop.org>
Cc: bderzhavets@yahoo.com, "xen-devel" <xen-devel@lists.xensource.com>, "Christophe Saout" <christophe@saout.de>
Date: Tuesday, February 24, 2009, 4:51 PM

On Tue, Feb 24, 2009 at 9:22 PM, Jeremy Fitzhardinge <jeremy@goop.org>
wrote:
> Boris Derzhavets wrote:
>>
>> X-Server fails to start on GeForce 8500 GT
>> Attempting "startx" i get a trace back.
>> Does it mean , that a proper NVIDIA driver is just not available for
the
>> most recent
>> 2.6.29-rc5 ?
>>
>
> Are you using the nvidia binary driver?  I would be surprised if it
worked
> out of the box; if changes need to be made to the binary parts, it may not
> be fixable at all.
>
>   J

I believe the current driver works with Xen because somebody outside
of nvidia contributed a patch, I use the nvidia binary driver with Xen
3.2.1 and my custom dom0 kernel which is 2.6.27.10 patched with
openSUSE Xen patches rebased to apply to vanilla, it works very well
but when pv_ops is merged into mainline the nvidia driver will
probably work when it is running on the bare metal but not under Xen
:(.

Having said that the patch for Xen support only touched the open
source parts of the driver, as soon as the current problems with hvm
on pv_ops are fixed I will be testing pv_ops and nvidia driver and I
will attempt to fix any problems, my C skills are improving so I might
be able to figure it out.

Andy

>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      

[-- Attachment #1.2: Type: text/html, Size: 2808 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Nvidia drivers on Xen with dom0 pvops on ultra-recent "git tip"kernel on x86_64

[Nigel Gamble]

On Feb 24, 2009, at 1:51 PM, Andrew Lyon wrote:

> On Tue, Feb 24, 2009 at 9:22 PM, Jeremy Fitzhardinge  
> <jeremy@goop.org> wrote:
>> Boris Derzhavets wrote:
>>>
>>> X-Server fails to start on GeForce 8500 GT
>>> Attempting "startx" i get a trace back.
>>> Does it mean , that a proper NVIDIA driver is just not available  
>>> for the
>>> most recent
>>> 2.6.29-rc5 ?
>>>
>>
>> Are you using the nvidia binary driver?  I would be surprised if it  
>> worked
>> out of the box; if changes need to be made to the binary parts, it  
>> may not
>> be fixable at all.
>>
>>   J
>
> I believe the current driver works with Xen because somebody outside
> of nvidia contributed a patch, I use the nvidia binary driver with Xen
> 3.2.1 and my custom dom0 kernel which is 2.6.27.10 patched with
> openSUSE Xen patches rebased to apply to vanilla, it works very well
> but when pv_ops is merged into mainline the nvidia driver will
> probably work when it is running on the bare metal but not under Xen
> :(.
>
> Having said that the patch for Xen support only touched the open
> source parts of the driver, as soon as the current problems with hvm
> on pv_ops are fixed I will be testing pv_ops and nvidia driver and I
> will attempt to fix any problems, my C skills are improving so I might
> be able to figure it out.


Did you ever figure this out?  I've just hit the same problem.  It  
looks like the source code part of the Nvidia kernel module has the  
code necessary to work on an old Xen Linux, but if it detects  
CONFIG_PARAVIRT set for a pv_ops kernel, it gives up an doesn't  
attempt to work with Xen at all.

Nigel

Re: Nvidia drivers on Xen with dom0 pvops on ultra-recent "git tip"kernel on x86_64

[Andrew Lyon]

On Thu, Apr 23, 2009 at 10:49 PM, Nigel Gamble <nigel@nrg.org> wrote:
> On Feb 24, 2009, at 1:51 PM, Andrew Lyon wrote:
>
>> On Tue, Feb 24, 2009 at 9:22 PM, Jeremy Fitzhardinge <jeremy@goop.org>
>> wrote:
>>>
>>> Boris Derzhavets wrote:
>>>>
>>>> X-Server fails to start on GeForce 8500 GT
>>>> Attempting "startx" i get a trace back.
>>>> Does it mean , that a proper NVIDIA driver is just not available for the
>>>> most recent
>>>> 2.6.29-rc5 ?
>>>>
>>>
>>> Are you using the nvidia binary driver?  I would be surprised if it
>>> worked
>>> out of the box; if changes need to be made to the binary parts, it may
>>> not
>>> be fixable at all.
>>>
>>>  J
>>
>> I believe the current driver works with Xen because somebody outside
>> of nvidia contributed a patch, I use the nvidia binary driver with Xen
>> 3.2.1 and my custom dom0 kernel which is 2.6.27.10 patched with
>> openSUSE Xen patches rebased to apply to vanilla, it works very well
>> but when pv_ops is merged into mainline the nvidia driver will
>> probably work when it is running on the bare metal but not under Xen
>> :(.
>>
>> Having said that the patch for Xen support only touched the open
>> source parts of the driver, as soon as the current problems with hvm
>> on pv_ops are fixed I will be testing pv_ops and nvidia driver and I
>> will attempt to fix any problems, my C skills are improving so I might
>> be able to figure it out.
>
>
> Did you ever figure this out?  I've just hit the same problem.  It looks
> like the source code part of the Nvidia kernel module has the code necessary
> to work on an old Xen Linux, but if it detects CONFIG_PARAVIRT set for a
> pv_ops kernel, it gives up an doesn't attempt to work with Xen at all.
>
> Nigel
>

hvm is still not working with pv_ops dom0 so I've not looked at it.

With a traditional patched Xen dom0 kernel you can use
IGNORE_XEN_PRESENCE=y to bypass the CONFIG_PARAVIRT check.

Andy

Andy

Re: Nvidia drivers on Xen with dom0 pvops on ultra-recent "git tip"kernel on x86_64

[Jeremy Fitzhardinge]

William Pitcock wrote:
> Jeremy, I need get_phys_to_machine() and set_phys_to_machine() to be
> changed to EXPORT_SYMBOL() for this. AFAIK, the equivilant symbols are
> EXPORT_SYMBOL() in XenLinux 2.6.18.
>   

get_phys_to_machine is no problem, but I'm curious about why it needs 
set_phys_to_machine.

> Can you make it happen? Also, if we had a way of knowing whether or not
> we are running under the hypervisor, I could ultimately make it work
> with the right codepaths under both configurations.
>   

In theory the driver shouldn't need to special case any code; the normal 
driver API functions should do the right things in both cases.  Could 
you go into a bit more detail about what your needs are?

    J

Re: Nvidia drivers on Xen with dom0 pvops on ultra-recent "git tip"kernel on x86_64

[William Pitcock]

On Mon, 2009-04-27 at 17:07 -0700, Jeremy Fitzhardinge wrote:
> William Pitcock wrote:
> > Jeremy, I need get_phys_to_machine() and set_phys_to_machine() to be
> > changed to EXPORT_SYMBOL() for this. AFAIK, the equivilant symbols are
> > EXPORT_SYMBOL() in XenLinux 2.6.18.
> >   
> 
> get_phys_to_machine is no problem, but I'm curious about why it needs 
> set_phys_to_machine.

It actually doesn't appear to.

> 
> > Can you make it happen? Also, if we had a way of knowing whether or not
> > we are running under the hypervisor, I could ultimately make it work
> > with the right codepaths under both configurations.
> >   
> 
> In theory the driver shouldn't need to special case any code; the normal 
> driver API functions should do the right things in both cases.  Could 
> you go into a bit more detail about what your needs are?

Actually, something is more broken. While I got video, the X server is
very crashhappy. Right now I've given up on NVidia cards and put in a
Radeon x800 xt I have laying around... which just works.

If you want to look at the nvidia interface code, let me know, and I'll
send you my observations/patches.

William

Attempt to load the most recent 2.6.30-rc-tip under Xen-3.3.1 on F11-preview

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 393 bytes --]

Booting hangs at xend startup :-

mount : block device xen is write-protected mounting read-only
mount : cannot mount block device xen read only
grep: /proc/xen/capabilities
No such file or directory

I believe the last message is from /etc/init.d/xend attempting to start
and exiting.
if ! grep -q "control_d" /proc/xen/capabilities ; then
        exit 0
fi

Boris.



      

[-- Attachment #1.2: Type: text/html, Size: 571 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Attempt to load the most recent 2.6.30-rc-tip under Xen-3.3.1 on F11-preview

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 1209 bytes --]

Actually,  the screen freeze after the next message

libvirtd : /proc/xen/capabilities
No such file or directory

Boris.
P.S.  Same kernel been built on Ubuntu Intrepid Server, loads fine under
Xen 3.4-rc3-pre been built from source. Something seems to be 
wrong with  /proc/xen entry into /etc/fstab on F11.

--- On Wed, 4/29/09, Boris Derzhavets <bderzhavets@yahoo.com> wrote:
From: Boris Derzhavets <bderzhavets@yahoo.com>
Subject: [Xen-devel] Attempt to load the most recent 2.6.30-rc-tip under Xen-3.3.1 on F11-preview
To: "Jeremy Fitzhardinge" <jeremy@goop.org>
Cc: xen-devel@lists.xensource.com
Date: Wednesday, April 29, 2009, 11:13 AM

Booting hangs at xend startup :-

mount : block device xen is write-protected mounting read-only
mount : cannot mount block device xen read only
grep: /proc/xen/capabilities
No such file or directory

I believe the last message is from /etc/init.d/xend attempting to start
and exiting.
if ! grep -q "control_d" /proc/xen/capabilities ; then
        exit 0
fi

Boris.



      _______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      

[-- Attachment #1.2: Type: text/html, Size: 1950 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Attempt to load the most recent 2.6.30-rc-tip under Xen-3.3.1 on F11-preview

[Jeremy Fitzhardinge]

Boris Derzhavets wrote:
> Actually,  the screen freeze after the next message
>
> libvirtd : /proc/xen/capabilities
> No such file or directory
>
> Boris.
> P.S.  Same kernel been built on Ubuntu Intrepid Server, loads fine under
> Xen 3.4-rc3-pre been built from source. Something seems to be
> wrong with  /proc/xen entry into /etc/fstab on F11.
>

Where is /proc/xen being mounted?  Do you have it in fstab?

Do you have selinux enabled?  Sometimes it prevents /proc/xen from being 
mounted.

    J

Re: Attempt to load the most recent 2.6.30-rc-tip under Xen-3.3.1 on F11-preview

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 1869 bytes --]

Thank you, Jeremy.
    That was SELINUX.  I also have to press CTRL+F2 to login to Xen Host.
 Login prompt doesn't appear on booting screen output. It's also flashing
 negotiating xen bridge for 1-2 min.

Grub Entry:-

title Xen 3.3.1 / Fedora 11, kernel 2.6.30-rc3-tip
  root (hd0,6)
  kernel /xen-3.3.gz
  module /vmlinuz-2.6.30-rc3-tip ro root=/dev/mapper/vg_serverfdr11-LogVol00     console=tty0
  module /initrd-2.6.30-rc3-tip.img



dmesg report still has :-

Kernel failure message 1:
------------[ cut here ]------------
WARNING: at kernel/smp.c:289 smp_call_function_single+0x65/0x108()
Hardware name: P5K Premium
Modules linked in:
Pid: 26, comm: khubd Tainted: G        W  2.6.30-rc3-tip #1
Call Trace:


Kernel failure message 2:
------------[ cut here ]------------
WARNING: at kernel/smp.c:369 smp_call_function_many+0x59/0x1bb()
Hardware name: P5K Premium
Modules linked in:
Pid: 26, comm: khubd Not tainted 2.6.30-rc3-tip #1
Call Trace:

It's attached.

Boris

--- On Wed, 4/29/09, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
From: Jeremy Fitzhardinge <jeremy@goop.org>
Subject: Re: [Xen-devel] Attempt to load the most recent 2.6.30-rc-tip under Xen-3.3.1 on F11-preview
To: bderzhavets@yahoo.com
Cc: xen-devel@lists.xensource.com
Date: Wednesday, April 29, 2009, 12:35 PM

Boris Derzhavets wrote:
> Actually,  the screen freeze after the next message
>
> libvirtd : /proc/xen/capabilities
> No such file or directory
>
> Boris.
> P.S.  Same kernel been built on Ubuntu Intrepid Server, loads fine under
> Xen 3.4-rc3-pre been built from source. Something seems to be
> wrong with  /proc/xen entry into /etc/fstab on F11.
>

Where is /proc/xen being mounted?  Do you have it in fstab?

Do you have selinux enabled?  Sometimes it prevents /proc/xen from being 
mounted.

    J



      

[-- Attachment #1.2: Type: text/html, Size: 2402 bytes --]

[-- Attachment #2: dmesg.f11.gz --]
[-- Type: application/x-gzip, Size: 14468 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel