Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[Konrad Rzeszutek Wilk]

> speculation of what may be the case isn't helping _me_ find the
> problem, so i'll try to debug this a step at a time.  any suggestions
> as to *specific* steps to take in debugging this would be great.
> here's all the info i can muster atm ...

You are missing this one:

[2010-01-26 10:09:10 4764] WARNING (image:561) domain test: device                                                                  
model failure: pid 5379: malfunctioning (closed sentinel), killed; see                                                              
/var/log/xen/qemu-dm-test.log        

Can you see what it has?

trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[0bo0]

in another thread @xen-devel, a crash @ DomU init under Xen 4.0.x, i
was advised,

On Mon, Jan 25, 2010 at 11:34 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> If qemu-dm died without logging anything, it probably crashed again. If it
> produced a core file you could load that in gdb and get a symbolic
> backtrace.

trying to turn on core dumps in xend-config, unfortunately, Xen seems
to fail @ core dump ... in addition to failing to launch a DomU which
works fine under Xen3,

  ...
  [2010-01-26 09:44:36 8552] ERROR (XendDomainInfo:2327) core dump
failed: id = 6 name = test: (1, 'Internal error', 'p2m_size < nr_pages
-1 (0 < fffff')
  ...

> My guess, since qemu-dm does seem to be working in most people's tests, is
> that either you have managed to install and not replace a mismatching
> version, somehow, or the set of options getting passed to it from you domain
> configuration have not been tested and do not work.

speculation of what may be the case isn't helping _me_ find the
problem, so i'll try to debug this a step at a time.  any suggestions
as to *specific* steps to take in debugging this would be great.
here's all the info i can muster atm ...

starting with a clean opensuse 11.2 system install
uname -a
  Linux server 2.6.31.8-0.1-default #1 SMP 2009-12-15 23:55:40 +0100
x86_64 x86_64 x86_64 GNU/Linux
zypper in xen xen-devel kernel-xen kernel-xen-devel xen-tools

rpm -qa | grep -i xen
  xen-libs-4.0.0_20809_01-28.2.x86_64
  xen-tools-4.0.0_20809_01-28.2.x86_64
  kernel-xen-2.6.31.8-0.1.1.x86_64
  xen-devel-4.0.0_20809_01-28.2.x86_64
  xen-4.0.0_20809_01-28.2.x86_64
  kernel-xen-devel-2.6.31.8-0.1.1.x86_64

updatedb
ls -al `locate qemu-dm | grep lib`
  lrwxrwxrwx 1 root root       7 2010-01-26 09:28
/usr/lib64/xen/bin/qemu-dm -> qemu-dm
  -rwxr-xr-x 1 root root 1287042 2010-01-19 17:13 /usr/lib/xen/bin/qemu-dm
/usr/lib/xen/bin/qemu-dm --help | grep "emulator version"
  QEMU PC emulator version 0.10.2, Copyright (c) 2003-2008 Fabrice Bellard

cat /boot/grub/menu.lst
---------------------------------
...
title Xen (symlink)
    root (hd0,0)
    kernel /xen.gz loglvl=all loglvl_guest=all apic_verbosity=debug
dom0_mem=1024M vga=gfx-1280x1024x32 console=vga,com1 com1=57600,8n1
iommu=1 dom0_max_vcpus=4 dom0_vcpus_pin=true sched=credit cpuidle=1
cpufreq=xen clocksource=acpi numa=on
    module /vmlinuz-xen root=/dev/VG0/ROOT rootfstype=ext4
rootflags=journal_checksum resume=/dev/VG0/SWAP showopts splash=silent
selinux=0 SELINUX_INIT=NO elevator=cfq vga=0x31a console=tty0
console=xvc0,57600 guestdev=0000:04:07.0,0000:02:00.0,0000:04:06.0
reassign_resources iommu=1 clocksource=xen mce=off
    module /initrd-xen
...
---------------------------------

cat /etc/xen/xend-config.sxp
---------------------------------
(logfile /var/log/xen/xend.log)
(loglevel DEBUG)
(console-limit 1024)
(keymap 'en-us')
(qemu-dm-logrotate-count 10)
(dom0-min-mem 1024)
(enable-dom0-ballooning no)
(total_available_memory 0)
(dom0-cpus 1)
(enable-dump yes)
(xend-unix-server yes)
(xend-unix-path /var/lib/xend/xend-socket)
(xend-http-server no)
(xend-relocation-server no)
(external-migration-tool '')
(xen-api-server ((unix)))
(xend-unix-xmlrpc-server yes)
(xend-tcp-xmlrpc-server no)
(vnc-listen '172.30.11.2')
(vncpasswd '')
(network-script       )
(vif-script vif-bridge)
(pci-passthrough-strict-check no)
---------------------------------


cat /etc/xen/vm/test.cfg
---------------------------------
name         = 'test'
builder      = 'linux'
bootloader   = '/usr/lib/xen/boot/domUloader.py'
bootargs     = '--entry=xvda1:vmlinuz-xen,/boot/initrd-xen'
disk         = [
'phy:/dev/VG0/boot,xvda,w','phy:/dev/VG0/swap,xvdb,w','phy:/dev/VG0/root,xvdc,w',]
boot         = 'c'   #d is cdrom boot, c is disk boot
root         = '/dev/xvdc1'
vif          = ['']
vfb          = [ 'type=vnc, vncdisplay=102, vnclisten=127.0.0.1' ]
extra        = 'textmode=1 xencons=xvc0 elevator=noop'
maxmem       = 4096
memory       = 4096
vcpus        = 4
cpu_cap      = 100
cpu_weight   = 256
localtime    = 0
netif        = 'yes'
on_shutdown  = 'destroy'
on_reboot    = 'restart'
on_crash     = 'destroy'
---------------------------------

reboot


...

uname -a
  Linux server 2.6.31.8-0.1-xen #1 SMP 2009-12-15 23:55:40 +0100
x86_64 x86_64 x86_64 GNU/Linux
xm dmesg | grep "Xen version" -A2
  (XEN) Xen version 4.0.0_20809_01-28.2 (abuild@) (gcc version 4.4.1
[gcc-4_4-branch revision 150839] (SUSE Linux) ) Wed Jan 20 01:09:45
UTC 2010
  (XEN) Latest ChangeSet: 20809
  (XEN) Command line: loglvl=all loglvl_guest=all apic_verbosity=debug
dom0_mem=1024M vga=gfx-1280x1024x32 console=vga,com1 com1=57600,8n1
iommu=1 dom0_max_vcpus=4 dom0_vcpus_pin=true sched=credit cpuidle=0
cpufreq=xen clocksource=acpi numa=on

xm create -c /etc/xen/vm/test.cfg
 Using config file "/etc/xen/vm/test.cfg".
 Error: Domain 'test' does not exist.

tail -f /var/log/xen/*
------------------------------------
==> /var/log/xen/xend.log <==
[2010-01-26 10:09:07 4764] WARNING (XendConfig:876) Unconverted key: Description
[2010-01-26 10:09:07 4764] DEBUG (XendDomainInfo:102)
XendDomainInfo.create(['vm', ['name', 'test'], ['memory', 4096],
['maxmem', 4096], ['on_reboot', 'restart'], ['on_crash', 'destroy'],
['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['vcpus', 4],
['cpu_cap', 100], ['cpu_weight', 256], ['backend', ['netif']], ['oos',
1], ['bootloader', '/usr/lib/xen/boot/domUloader.py'],
['bootloader_args', '--entry=xvda1:vmlinuz-xen,/boot/initrd-xen'],
['image', ['linux', ['root', '/dev/xvdc1'], ['videoram', 4], ['args',
'textmode=1 xencons=xvc0'], ['tsc_mode', 0], ['nomigrate', 0]]],
['s3_integrity', 1], ['device', ['vbd', ['uname',
'phy:/dev/VG0/boot'], ['dev', 'xvda'], ['mode', 'w']]], ['device',
['vbd', ['uname', 'phy:/dev/VG0/swap'], ['dev', 'xvdb'], ['mode',
'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VG0/root'], ['dev',
'xvdc'], ['mode', 'w']]], ['device', ['vif']], ['device', ['vkbd']],
['device', ['vfb', ['vnc', '1'], ['vnclisten', '127.0.0.1'],
['vncdisplay', '102'], ['display', 'localhost:11.0'], ['xauthority',
'/root/.Xauthority']]]])
[2010-01-26 10:09:07 4764] DEBUG (XendDomainInfo:2513)
XendDomainInfo.constructDomain
[2010-01-26 10:09:07 4764] DEBUG (balloon:239) Balloon: 6946324 KiB
free; need 4096; done.
[2010-01-26 10:09:07 4764] DEBUG (XendDomain:465) Adding Domain: 1
[2010-01-26 10:09:07 4764] DEBUG (XendDomainInfo:2821)
XendDomainInfo.initDomain: 1 256
[2010-01-26 10:09:07 5315] DEBUG (XendBootloader:130) Launching
bootloader as ['/usr/lib/xen/boot/domUloader.py',
'--args=root=/dev/xvdc1 textmode=1 xencons=xvc0',
'--output=/var/run/xend/boot/xenbl.24317',
'--entry=xvda1:vmlinuz-xen,/boot/initrd-xen', '/dev/VG0/boot'].

==> /var/log/xen/domain-builder-ng.log <==
### ----- xc domain builder logfile opened -----
xc_dom_allocate: cmdline="root=/dev/xvdc1 textmode=1 xencons=xvc0", features=""
xc_dom_kernel_file: filename="/var/lib/xen/tmp/kernel.GF9ro1"
xc_dom_malloc_filemap    : 3297 kB
xc_dom_malloc            : 12278 kB
xc_dom_do_gunzip: unzip ok, 0x338703 -> 0xbfd9c8
xc_dom_ramdisk_file: filename="/var/lib/xen/tmp/ramdisk.usyS9V"
xc_dom_malloc_filemap    : 8150 kB
xc_dom_boot_xen_init: ver 4.0, caps xen-3.0-x86_64 xen-3.0-x86_32p
hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xc_dom_parse_image: called
xc_dom_find_loader: trying ELF-generic loader ... OK
elf_parse_binary: phdr: paddr=0x2000 memsz=0x733000
elf_parse_binary: phdr: paddr=0x735000 memsz=0x51dd0
elf_parse_binary: phdr: paddr=0x787000 memsz=0x8c8
elf_parse_binary: phdr: paddr=0x788000 memsz=0xaca0
elf_parse_binary: phdr: paddr=0x792ca0 memsz=0x14e360
elf_parse_binary: memory: 0x2000 -> 0x8e1000
elf_xen_parse_note: GUEST_OS = "linux"
elf_xen_parse_note: GUEST_VERSION = "2.6"
elf_xen_parse_note: XEN_VERSION = "xen-3.0"
elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
elf_xen_parse_note: PADDR_OFFSET = 0x0
elf_xen_parse_note: ENTRY = 0xffffffff80002000
elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff80008000
elf_xen_parse_note: unknown xen elf note (0xd)
elf_xen_parse_note: INIT_P2M = 0xffffea0000000000
elf_xen_parse_note: FEATURES =
"writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel"
elf_xen_parse_note: LOADER = "generic"
elf_xen_parse_note: SUSPEND_CANCEL = 0x1
elf_xen_addr_calc_check: addresses:
    virt_base        = 0xffffffff80000000
    elf_paddr_offset = 0x0
    virt_offset      = 0xffffffff80000000
    virt_kstart      = 0xffffffff80002000
    virt_kend        = 0xffffffff808e1000
    virt_entry       = 0xffffffff80002000
    p2m_base         = 0xffffea0000000000
xc_dom_parse_elf_kernel: xen-3.0-x86_64: 0xffffffff80002000 ->
0xffffffff808e1000
xc_dom_mem_init: mem 4096 MB, pages 0x100000 pages, 4k each
xc_dom_mem_init: 0x100000 pages
xc_dom_boot_mem_init: called
x86_compat: guest xen-3.0-x86_64, address size 64
xc_dom_malloc            : 8192 kB
xc_dom_build_image: called
xc_dom_alloc_segment:   kernel       : 0xffffffff80002000 ->
0xffffffff808e1000  (pfn 0x2 + 0x8df pages)
xc_dom_pfn_to_ptr: domU mapping: pfn 0x2+0x8df at 0x7fc2e4726000
elf_load_binary: phdr 0 at 0x0x7fc2e4726000 -> 0x0x7fc2e4e59000
elf_load_binary: phdr 1 at 0x0x7fc2e4e59000 -> 0x0x7fc2e4eaadd0
elf_load_binary: phdr 2 at 0x0x7fc2e4eab000 -> 0x0x7fc2e4eab8c8
elf_load_binary: phdr 3 at 0x0x7fc2e4eac000 -> 0x0x7fc2e4eb6ca0
elf_load_binary: phdr 4 at 0x0x7fc2e4eb6ca0 -> 0x0x7fc2e4f11000
xc_dom_alloc_segment:   ramdisk      : 0xffffffff808e1000 ->
0xffffffff81cde000  (pfn 0x8e1 + 0x13fd pages)
xc_dom_malloc            : 119 kB
xc_dom_pfn_to_ptr: domU mapping: pfn 0x8e1+0x13fd at 0x7fc2e3329000
xc_dom_do_gunzip: unzip ok, 0x7f5ba8 -> 0x13fcc10
xc_dom_alloc_segment:   phys2mach    : 0xffffffff81cde000 ->
0xffffffff824de000  (pfn 0x1cde + 0x800 pages)
xc_dom_pfn_to_ptr: domU mapping: pfn 0x1cde+0x800 at 0x7fc2e2b29000
xc_dom_alloc_page   :   start info   : 0xffffffff824de000 (pfn 0x24de)
xc_dom_alloc_page   :   xenstore     : 0xffffffff824df000 (pfn 0x24df)
xc_dom_alloc_page   :   console      : 0xffffffff824e0000 (pfn 0x24e0)
nr_page_tables: 0x0000ffffffffffff/48: 0xffff000000000000 ->
0xffffffffffffffff, 1 table(s)
nr_page_tables: 0x0000007fffffffff/39: 0xffffff8000000000 ->
0xffffffffffffffff, 1 table(s)
nr_page_tables: 0x000000003fffffff/30: 0xffffffff80000000 ->
0xffffffffbfffffff, 1 table(s)
nr_page_tables: 0x00000000001fffff/21: 0xffffffff80000000 ->
0xffffffff827fffff, 20 table(s)
xc_dom_alloc_segment:   page tables  : 0xffffffff824e1000 ->
0xffffffff824f8000  (pfn 0x24e1 + 0x17 pages)
xc_dom_pfn_to_ptr: domU mapping: pfn 0x24e1+0x17 at 0x7fc2f5336000
xc_dom_alloc_page   :   boot stack   : 0xffffffff824f8000 (pfn 0x24f8)
xc_dom_build_image  : virt_alloc_end : 0xffffffff824f9000
xc_dom_build_image  : virt_pgtab_end : 0xffffffff82800000
xc_dom_boot_image: called
arch_setup_bootearly: doing nothing
xc_dom_compat_check: supported guest type: xen-3.0-x86_64 <= matches
xc_dom_compat_check: supported guest type: xen-3.0-x86_32p
xc_dom_compat_check: supported guest type: hvm-3.0-x86_32
xc_dom_compat_check: supported guest type: hvm-3.0-x86_32p
xc_dom_compat_check: supported guest type: hvm-3.0-x86_64
xc_dom_update_guest_p2m: dst 64bit, pages 0x100000
clear_page: pfn 0x24e0, mfn 0x215140
clear_page: pfn 0x24df, mfn 0x215141
xc_dom_pfn_to_ptr: domU mapping: pfn 0x24de+0x1 at 0x7fc2f5335000
start_info_x86_64: called
setup_hypercall_page: vaddr=0xffffffff80008000 pfn=0x8
domain builder memory footprint
   allocated
      malloc             : 20694 kB
      anon mmap          : 0 bytes
   mapped
      file mmap          : 11448 kB
      domU mmap          : 36 MB
arch_setup_bootlate: shared_info: pfn 0x0, mfn 0xcfa87
shared_info_x86_64: called
vcpu_x86_64: called
vcpu_x86_64: cr3: pfn 0x24e1 mfn 0x21513f
launch_vm: called, ctxt=0x7fc2e67f73e0
xc_dom_release: called

==> /var/log/xen/xend.log <==
[2010-01-26 10:09:08 4764] DEBUG (image:339) No VNC passwd configured
for vfb access
[2010-01-26 10:09:08 4764] DEBUG (XendDomainInfo:2848)
_initDomain:shadow_memory=0x0, memory_static_max=0x100000000,
memory_static_min=0x0.
[2010-01-26 10:09:08 4764] INFO (image:182) buildDomain os=linux dom=1
vcpus=4
[2010-01-26 10:09:08 4764] DEBUG (image:734) domid          = 1
[2010-01-26 10:09:08 4764] DEBUG (image:735) memsize        = 4096
[2010-01-26 10:09:08 4764] DEBUG (image:736) image          =
/var/lib/xen/tmp/kernel.GF9ro1
[2010-01-26 10:09:08 4764] DEBUG (image:737) store_evtchn   = 1
[2010-01-26 10:09:08 4764] DEBUG (image:738) console_evtchn = 2
[2010-01-26 10:09:08 4764] DEBUG (image:739) cmdline        =
root=/dev/xvdc1 textmode=1 xencons=xvc0
[2010-01-26 10:09:08 4764] DEBUG (image:740) ramdisk        =
/var/lib/xen/tmp/ramdisk.usyS9V
[2010-01-26 10:09:08 4764] DEBUG (image:741) vcpus          = 4
[2010-01-26 10:09:08 4764] DEBUG (image:742) features       =
[2010-01-26 10:09:08 4764] DEBUG (image:743) flags          = 0
[2010-01-26 10:09:08 4764] DEBUG (image:744) superpages     = 0
[2010-01-26 10:09:08 4764] INFO (XendDomainInfo:2372) createDevice:
vkbd : {'uuid': '3e629ba2-c496-f073-e797-f6317e3bdcd4'}
[2010-01-26 10:09:08 4764] DEBUG (DevController:95) DevController:
writing {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0',
'backend': '/local/domain/0/backend/vkbd/1/0'} to
/local/domain/1/device/vkbd/0.
[2010-01-26 10:09:08 4764] DEBUG (DevController:97) DevController:
writing {'frontend-id': '1', 'domain': 'test', 'frontend':
'/local/domain/1/device/vkbd/0', 'state': '1', 'online': '1'} to
/local/domain/0/backend/vkbd/1/0.
[2010-01-26 10:09:08 4764] INFO (XendDomainInfo:2372) createDevice:
vfb : {'other_config': {'display': 'localhost:11.0', 'vnclisten':
'127.0.0.1', 'vnc': '1', 'xauthority': '/root/.Xauthority',
'vncdisplay': '102'}, 'vnc': '1', 'xauthority': '/root/.Xauthority',
'vnclisten': '127.0.0.1', 'vncdisplay': '102', 'keymap': 'en-us',
'display': 'localhost:11.0', 'uuid':
'219090ab-f3d2-f998-46f3-efbfa6221f7f'}
[2010-01-26 10:09:08 4764] DEBUG (DevController:95) DevController:
writing {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0',
'backend': '/local/domain/0/backend/vfb/1/0'} to
/local/domain/1/device/vfb/0.
[2010-01-26 10:09:08 4764] DEBUG (DevController:97) DevController:
writing {'domain': 'test', 'frontend': '/local/domain/1/device/vfb/0',
'xauthority': '/root/.Xauthority', 'frontend-id': '1', 'vnclisten':
'127.0.0.1', 'vncdisplay': '102', 'state': '1', 'keymap': 'en-us',
'online': '1', 'vnc': '1', 'display': 'localhost:11.0', 'uuid':
'219090ab-f3d2-f998-46f3-efbfa6221f7f'} to
/local/domain/0/backend/vfb/1/0.
[2010-01-26 10:09:08 4764] INFO (XendDomainInfo:2372) createDevice:
vbd : {'uuid': '41bdfd95-b845-e04e-0643-7803fcb5645b', 'bootable': 1,
'driver': 'paravirtualised', 'dev': 'xvda', 'uname':
'phy:/dev/VG0/boot', 'mode': 'w'}
[2010-01-26 10:09:08 4764] DEBUG (DevController:95) DevController:
writing {'virtual-device': '51712', 'device-type': 'disk', 'protocol':
'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend':
'/local/domain/0/backend/vbd/1/51712'} to
/local/domain/1/device/vbd/51712.
[2010-01-26 10:09:08 4764] DEBUG (DevController:97) DevController:
writing {'domain': 'test', 'frontend':
'/local/domain/1/device/vbd/51712', 'uuid':
'41bdfd95-b845-e04e-0643-7803fcb5645b', 'bootable': '1', 'dev':
'xvda', 'state': '1', 'params': '/dev/VG0/boot', 'mode': 'w',
'online': '1', 'frontend-id': '1', 'type': 'phy'} to
/local/domain/0/backend/vbd/1/51712.
[2010-01-26 10:09:08 4764] INFO (XendDomainInfo:2372) createDevice:
vbd : {'uuid': '6ee5a108-2bef-fa26-5d3d-a6be7d088ea3', 'bootable': 0,
'driver': 'paravirtualised', 'dev': 'xvdb', 'uname':
'phy:/dev/VG0/swap', 'mode': 'w'}
[2010-01-26 10:09:08 4764] DEBUG (DevController:95) DevController:
writing {'virtual-device': '51728', 'device-type': 'disk', 'protocol':
'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend':
'/local/domain/0/backend/vbd/1/51728'} to
/local/domain/1/device/vbd/51728.
[2010-01-26 10:09:08 4764] DEBUG (DevController:97) DevController:
writing {'domain': 'test', 'frontend':
'/local/domain/1/device/vbd/51728', 'uuid':
'6ee5a108-2bef-fa26-5d3d-a6be7d088ea3', 'bootable': '0', 'dev':
'xvdb', 'state': '1', 'params': '/dev/VG0/swap', 'mode': 'w',
'online': '1', 'frontend-id': '1', 'type': 'phy'} to
/local/domain/0/backend/vbd/1/51728.
[2010-01-26 10:09:08 4764] INFO (XendDomainInfo:2372) createDevice:
vbd : {'uuid': 'e7a397a0-903b-eed9-e1c4-a9639b2416e7', 'bootable': 0,
'driver': 'paravirtualised', 'dev': 'xvdc', 'uname':
'phy:/dev/VG0/root', 'mode': 'w'}
[2010-01-26 10:09:08 4764] DEBUG (DevController:95) DevController:
writing {'virtual-device': '51744', 'device-type': 'disk', 'protocol':
'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend':
'/local/domain/0/backend/vbd/1/51744'} to
/local/domain/1/device/vbd/51744.
[2010-01-26 10:09:08 4764] DEBUG (DevController:97) DevController:
writing {'domain': 'test', 'frontend':
'/local/domain/1/device/vbd/51744', 'uuid':
'e7a397a0-903b-eed9-e1c4-a9639b2416e7', 'bootable': '0', 'dev':
'xvdc', 'state': '1', 'params': '/dev/VG0/root', 'mode': 'w',
'online': '1', 'frontend-id': '1', 'type': 'phy'} to
/local/domain/0/backend/vbd/1/51744.
[2010-01-26 10:09:08 4764] INFO (XendDomainInfo:2372) createDevice:
vif : {'mac': '00:16:3e:4f:32:d7', 'uuid':
'e54d312a-774d-0d65-e250-0b3c1f42e6bb'}
[2010-01-26 10:09:08 4764] DEBUG (DevController:95) DevController:
writing {'mac': '00:16:3e:4f:32:d7', 'handle': '0', 'protocol':
'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend':
'/local/domain/0/backend/vif/1/0'} to /local/domain/1/device/vif/0.
[2010-01-26 10:09:08 4764] DEBUG (DevController:97) DevController:
writing {'domain': 'test', 'frontend': '/local/domain/1/device/vif/0',
'uuid': 'e54d312a-774d-0d65-e250-0b3c1f42e6bb', 'script':
'/etc/xen/scripts/vif-bridge', 'mac': '00:16:3e:4f:32:d7',
'frontend-id': '1', 'state': '1', 'online': '1', 'handle': '0'} to
/local/domain/0/backend/vif/1/0.
[2010-01-26 10:09:09 4764] INFO (image:420) spawning device models:
/usr/lib/xen/bin/qemu-dm ['/usr/lib/xen/bin/qemu-dm', '-d', '1',
'-serial', 'pty', '-domain-name', 'test', '-videoram', '4', '-k',
'en-us', '-vnc', '127.0.0.1:102', '-vncunused', '-monitor', 'null',
'-M', 'xenpv']
[2010-01-26 10:09:09 4764] INFO (image:469) device model pid: 5379
[2010-01-26 10:09:09 4764] INFO (image:603) waiting for sentinel_fifo
[2010-01-26 10:09:09 4764] DEBUG (XendDomainInfo:3420) Storing VM
details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid':
'730a466e-80a3-4aa3-79f8-7eb6a6f3abc8', 'on_reboot': 'restart',
'start_time': '1264529349.19', 'on_poweroff': 'destroy',
'bootloader_args': '--entry=xvda1:vmlinuz-xen,/boot/initrd-xen',
'on_xend_start': 'ignore', 'on_crash': 'destroy',
'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15',
'bootloader': '/usr/lib/xen/boot/domUloader.py', 'image': "(linux
(kernel ) (args 'root=/dev/xvdc1 textmode=1 xencons=xvc0') (superpages
0) (videoram 4) (pci ()) (nomigrate 0) (tsc_mode 0) (device_model
/usr/lib/xen/bin/qemu-dm) (keymap en-us) (notes (FEATURES
'writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel')
(VIRT_BASE 18446744071562067968) (GUEST_VERSION 2.6) (PADDR_OFFSET 0)
(GUEST_OS linux) (HYPERCALL_PAGE 18446744071562100736) (LOADER
generic) (INIT_P2M 18446719884453740544) (SUSPEND_CANCEL 1) (ENTRY
18446744071562076160) (XEN_VERSION xen-3.0)))", 'name': 'test'}
[2010-01-26 10:09:09 4764] DEBUG (XendDomainInfo:1805) Storing domain
details: {'console/ring-ref': '2183488', 'image/entry':
'18446744071562076160', 'console/port': '2', 'cpu/3/availability':
'online', 'store/ring-ref': '2183489', 'image/loader': 'generic',
'vm': '/vm/730a466e-80a3-4aa3-79f8-7eb6a6f3abc8',
'control/platform-feature-multiprocessor-suspend': '1', 'description':
'', 'cpu/2/availability': 'online', 'cpu/1/availability': 'online',
'image/features/writable-descriptor-tables': '1', 'image/virt-base':
'18446744071562067968', 'memory/target': '4194304',
'image/guest-version': '2.6', 'image/features/supervisor-mode-kernel':
'1', 'image/guest-os': 'linux', 'console/limit': '1048576',
'image/paddr-offset': '0', 'image/hypercall-page':
'18446744071562100736', 'image/suspend-cancel': '1',
'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb':
'1', 'image/features/writable-page-tables': '1', 'console/type':
'ioemu', 'image/features/auto-translated-physmap': '1', 'name':
'test', 'domid': '1', 'image/xen-version': 'xen-3.0',
'image/init-p2m': '18446719884453740544', 'store/port': '1'}
[2010-01-26 10:09:10 4764] WARNING (image:561) domain test: device
model failure: pid 5379: malfunctioning (closed sentinel), killed; see
/var/log/xen/qemu-dm-test.log
[2010-01-26 10:09:10 4764] DEBUG (DevController:95) DevController:
writing {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0',
'backend': '/local/domain/0/backend/console/1/0'} to
/local/domain/1/device/console/0.
[2010-01-26 10:09:10 4764] DEBUG (DevController:97) DevController:
writing {'domain': 'test', 'frontend':
'/local/domain/1/device/console/0', 'uuid':
'164d06b3-1ce7-ccfc-4258-72184fd3757a', 'frontend-id': '1', 'state':
'1', 'location': '2', 'online': '1', 'protocol': 'vt100'} to
/local/domain/0/backend/console/1/0.
[2010-01-26 10:09:10 4764] WARNING (XendDomainInfo:2072) Domain has
crashed: name=test id=1.

==> /var/log/xen/xend-debug.log <==
ERROR Internal error: p2m_size < nr_pages -1 (0 < fffff

==> /var/log/xen/xend.log <==
[2010-01-26 10:09:10 4764] DEBUG (DevController:139) Waiting for devices tap2.
[2010-01-26 10:09:10 4764] DEBUG (DevController:139) Waiting for devices vif.
[2010-01-26 10:09:10 4764] DEBUG (DevController:144) Waiting for 0.
[2010-01-26 10:09:11 4764] ERROR (XendDomainInfo:2327) core dump
failed: id = 1 name = test: (1, 'Internal error', 'p2m_size < nr_pages
-1 (0 < fffff')
[2010-01-26 10:09:11 4764] DEBUG (XendDomainInfo:3061)
XendDomainInfo.destroy: domid=1
[2010-01-26 10:09:12 4764] DEBUG (XendDomainInfo:2416) Destroying device model
[2010-01-26 10:09:12 4764] DEBUG (DevController:653)
hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status.
[2010-01-26 10:09:12 4764] DEBUG (DevController:667) hotplugStatusCallback 1.
[2010-01-26 10:09:12 4764] DEBUG (DevController:139) Waiting for devices vkbd.
[2010-01-26 10:09:13 4764] DEBUG (DevController:139) Waiting for
devices ioports.
[2010-01-26 10:09:13 4764] DEBUG (DevController:139) Waiting for devices tap.
[2010-01-26 10:09:13 4764] DEBUG (DevController:139) Waiting for devices vif2.
[2010-01-26 10:09:13 4764] DEBUG (DevController:139) Waiting for
devices console.
[2010-01-26 10:09:13 4764] DEBUG (DevController:144) Waiting for 0.
[2010-01-26 10:09:13 4764] DEBUG (DevController:139) Waiting for devices vscsi.
[2010-01-26 10:09:13 4764] DEBUG (XendDomainInfo:2423) Releasing devices
[2010-01-26 10:09:13 4764] DEBUG (XendDomainInfo:2429) Removing vif/0
[2010-01-26 10:09:13 4764] DEBUG (XendDomainInfo:1271)
XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0
[2010-01-26 10:09:13 4764] DEBUG (XendDomainInfo:2429) Removing vkbd/0
[2010-01-26 10:09:13 4764] DEBUG (XendDomainInfo:1271)
XendDomainInfo.destroyDevice: deviceClass = vkbd, device = vkbd/0
[2010-01-26 10:09:13 4764] DEBUG (XendDomainInfo:2429) Removing console/0
[2010-01-26 10:09:13 4764] DEBUG (XendDomainInfo:1271)
XendDomainInfo.destroyDevice: deviceClass = console, device =
console/0
[2010-01-26 10:09:13 4764] DEBUG (XendDomainInfo:2429) Removing vbd/51712
[2010-01-26 10:09:13 4764] DEBUG (XendDomainInfo:1271)
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51712
[2010-01-26 10:09:14 4764] DEBUG (XendDomainInfo:2429) Removing vbd/51728
[2010-01-26 10:09:14 4764] DEBUG (XendDomainInfo:1271)
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51728
[2010-01-26 10:09:14 4764] DEBUG (XendDomainInfo:2429) Removing vbd/51744
[2010-01-26 10:09:14 4764] DEBUG (XendDomainInfo:1271)
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51744
[2010-01-26 10:09:14 4764] DEBUG (DevController:139) Waiting for devices vbd.
[2010-01-26 10:09:14 4764] DEBUG (DevController:144) Waiting for 51712.
[2010-01-26 10:09:14 4764] DEBUG (XendDomainInfo:2429) Removing vfb/0
[2010-01-26 10:09:14 4764] DEBUG (XendDomainInfo:1271)
XendDomainInfo.destroyDevice: deviceClass = vfb, device = vfb/0
[2010-01-26 10:09:15 4764] DEBUG (DevController:653)
hotplugStatusCallback
/local/domain/0/backend/vbd/1/51712/hotplug-status.
[2010-01-26 10:09:15 4764] DEBUG (DevController:667) hotplugStatusCallback 1.
[2010-01-26 10:09:15 4764] DEBUG (DevController:144) Waiting for 51728.
------------------------------------

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[0bo0]

hi,

On Tue, Jan 26, 2010 at 10:36 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> You are missing this one:
>
> [2010-01-26 10:09:10 4764] WARNING (image:561) domain test: device
> model failure: pid 5379: malfunctioning (closed sentinel), killed; see
> /var/log/xen/qemu-dm-test.log
>
> Can you see what it has?

yep, it was created after I launched the tail ... checking, not much there:

cat qemu-dm-test.log
  domid: 1
  Warning: vlan 0 is not connected to host network
  Using xvda for guest's hda

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[Keir Fraser]

On 26/01/2010 18:36, "0bo0" <0.bugs.only.0@gmail.com> wrote:

>> If qemu-dm died without logging anything, it probably crashed again. If it
>> produced a core file you could load that in gdb and get a symbolic
>> backtrace.
> 
> trying to turn on core dumps in xend-config, unfortunately, Xen seems
> to fail @ core dump ... in addition to failing to launch a DomU which
> works fine under Xen3,

Ah, sorry, I meant a core file produced when qemu-dm crashes (that is,
assuming that qemu-dm does crash, which appears likely). All the log files
tell us is that xend thinks that qemu-dm process simply disappeared without
warning. The question is obviously: why did it disappear, and only for you
in your setup.

 -- Keir

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[0bo0]

On Tue, Jan 26, 2010 at 1:10 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> On 26/01/2010 18:36, "0bo0" <0.bugs.only.0@gmail.com> wrote:
>
>>> If qemu-dm died without logging anything, it probably crashed again. If it
>>> produced a core file you could load that in gdb and get a symbolic
>>> backtrace.
>>
>> trying to turn on core dumps in xend-config, unfortunately, Xen seems
>> to fail @ core dump ... in addition to failing to launch a DomU which
>> works fine under Xen3,
>
> Ah, sorry, I meant a core file produced when qemu-dm crashes (that is,
> assuming that qemu-dm does crash, which appears likely). All the log files
> tell us is that xend thinks that qemu-dm process simply disappeared without
> warning. The question is obviously: why did it disappear, and only for you
> in your setup.

np. i'm happy to provide the info.

 so since i'm not launching qemu-dm directly, how -- specifically --
do I grab that core dump for you?

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[Keir Fraser]

On 26/01/2010 21:25, "0bo0" <0.bugs.only.0@gmail.com> wrote:

>> Ah, sorry, I meant a core file produced when qemu-dm crashes (that is,
>> assuming that qemu-dm does crash, which appears likely). All the log files
>> tell us is that xend thinks that qemu-dm process simply disappeared without
>> warning. The question is obviously: why did it disappear, and only for you
>> in your setup.
> 
> np. i'm happy to provide the info.
> 
>  so since i'm not launching qemu-dm directly, how -- specifically --
> do I grab that core dump for you?

As root:
 ulimit -c unlimited
 echo '/var/core/core.%e.%u.%p' > /proc/sys/kernel/core_pattern

Now try starting your guest. It fails, and hopefully you have a core file in
/var/core/. You can load it into gdb as 'gdb /usr/lib/xen/qemu-dm
/var/core/name-of-core-file' and then execute bt command at the gdb prompt.
That hopefully tells us where qemu-dm crashed.

 -- Keir

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[0bo0]

On Tue, Jan 26, 2010 at 1:38 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> As root:
>    echo '/var/core/core.%e.%u.%p' > /proc/sys/kernel/core_pattern
>
> Now try starting your guest. It fails, and hopefully you have a core file in
> /var/core/. You can load it into gdb as 'gdb /usr/lib/xen/qemu-dm
> /var/core/name-of-core-file' and then execute bt command at the gdb prompt.
> That hopefully tells us where qemu-dm crashed.


bash
ulimit -c unlimited
echo '/var/core/core.%e.%u.%p' > /proc/sys/kernel/core_pattern
mkdir -p /var/core
cat /proc/sys/kernel/core_pattern
/var/core/core.%e.%u.%p

xm create /etc/xen/vm/test.cfg
  Using config file "/etc/xen/vm/test.cfg".
  Error: Domain 'test' does not exist.

ls -al /var/core
  total 8
  drwxr-xr-x  2 root root 4096 2010-01-26 13:48 ./
  drwxr-xr-x 18 root root 4096 2010-01-26 13:48 ../

@ syslog, though, i do see ....

Jan 26 14:09:17 server kernel: [  547.023464] kjournald starting.
Commit interval 15 seconds
Jan 26 14:09:17 server kernel: [  547.118565] EXT3-fs: mounted
filesystem with ordered data mode.
Jan 26 14:09:18 server logger: /etc/xen/scripts/block: add
XENBUS_PATH=backend/vbd/3/51712
Jan 26 14:09:18 server logger: /etc/xen/scripts/block: add
XENBUS_PATH=backend/vbd/3/51744
Jan 26 14:09:18 server logger: /etc/xen/scripts/block: add
XENBUS_PATH=backend/vbd/3/51728
Jan 26 14:09:18 server kernel: [  548.456142] qemu-dm[6791]: segfault
at 7fff7ee4169a ip 000000000047b974 sp 00007fff7ee40210 error 4 in
qemu-dm[400000+10b000]
Jan 26 14:09:18 server logger: /etc/xen/scripts/vif-bridge: online
XENBUS_PATH=backend/vif/3/0
Jan 26 14:09:18 server kernel: [  548.697863] device vif3.0 entered
promiscuous mode
Jan 26 14:09:19 server kernel: [  548.828028] br02: port 1(vif3.0)
entering forwarding state
Jan 26 14:09:19 server kernel: [  548.984448] physdev match: using
--physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for
non-bridged traffic is not supported anymore.
Jan 26 14:09:19 server logger: /etc/xen/scripts/vif-bridge: Successful
vif-bridge online for vif3.0, bridge br02.
Jan 26 14:09:19 server logger: /etc/xen/scripts/vif-bridge: Writing
backend/vif/3/0/hotplug-status connected to xenstore.
Jan 26 14:09:19 server logger: /etc/xen/scripts/block: Writing
backend/vbd/3/51712/physical-device fd:d to xenstore.
Jan 26 14:09:19 server kernel: [  549.267962] (cdrom_add_media_watch()
file=/usr/src/packages/BUILD/kernel-xen-2.6.31.8/linux-2.6.31/drivers/xen/blkback/cdrom.c,
line=108) nodename:backend/vbd/3/51712
Jan 26 14:09:19 server kernel: [  549.474577] (cdrom_is_type()
file=/usr/src/packages/BUILD/kernel-xen-2.6.31.8/linux-2.6.31/drivers/xen/blkback/cdrom.c,
line=95) type:0
Jan 26 14:09:19 server logger: /etc/xen/scripts/block: Writing
backend/vbd/3/51712/hotplug-status connected to xenstore.
Jan 26 14:09:20 server logger: /etc/xen/scripts/block: Writing
backend/vbd/3/51728/physical-device fd:e to xenstore.
Jan 26 14:09:20 server kernel: [  550.443509] (cdrom_add_media_watch()
file=/usr/src/packages/BUILD/kernel-xen-2.6.31.8/linux-2.6.31/drivers/xen/blkback/cdrom.c,
line=108) nodename:backend/vbd/3/51728
Jan 26 14:09:20 server kernel: [  550.657671] (cdrom_is_type()
file=/usr/src/packages/BUILD/kernel-xen-2.6.31.8/linux-2.6.31/drivers/xen/blkback/cdrom.c,
line=95) type:0
Jan 26 14:09:20 server logger: /etc/xen/scripts/block: Writing
backend/vbd/3/51728/hotplug-status connected to xenstore.
Jan 26 14:09:21 server logger: /etc/xen/scripts/xen-hotplug-cleanup:
XENBUS_PATH=backend/vkbd/3/0
Jan 26 14:09:21 server logger: /etc/xen/scripts/xen-hotplug-cleanup:
XENBUS_PATH=backend/console/3/0
Jan 26 14:09:22 server kernel: [  552.224799] br02: port 1(vif3.0)
entering disabled state
Jan 26 14:09:22 server kernel: [  552.335801] br02: port 1(vif3.0)
entering disabled state
Jan 26 14:09:22 server logger: /etc/xen/scripts/xen-hotplug-cleanup:
XENBUS_PATH=backend/vfb/3/0
Jan 26 14:09:22 server logger: /etc/xen/scripts/block: remove
XENBUS_PATH=backend/vbd/3/51728
Jan 26 14:09:22 server logger: /etc/xen/scripts/block: remove
XENBUS_PATH=backend/vbd/3/51712
Jan 26 14:09:22 server logger: /etc/xen/scripts/vif-bridge: offline
XENBUS_PATH=backend/vif/3/0
Jan 26 14:09:22 server logger: /etc/xen/scripts/xen-hotplug-cleanup:
XENBUS_PATH=backend/vbd/3/51728
Jan 26 14:09:22 server logger: /etc/xen/scripts/xen-hotplug-cleanup:
XENBUS_PATH=backend/vbd/3/51712
Jan 26 14:09:22 server logger: /etc/xen/scripts/vif-bridge: brctl
delif br02 vif3.0 failed
Jan 26 14:09:22 server logger: /etc/xen/scripts/vif-bridge: ifconfig
vif3.0 down failed
Jan 26 14:09:22 server kernel: [  552.662465] physdev match: using
--physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for
non-bridged traffic is not supported anymore.
Jan 26 14:09:22 server logger: /etc/xen/scripts/vif-bridge: Successful
vif-bridge offline for vif3.0, bridge br02.
Jan 26 14:09:22 server logger: /etc/xen/scripts/xen-hotplug-cleanup:
XENBUS_PATH=backend/vif/3/0
Jan 26 14:09:23 server logger: /etc/xen/scripts/block: Writing
backend/vbd/3/51744/physical-device fd:f to xenstore.
Jan 26 14:09:23 server logger: /etc/xen/scripts/block: Writing
backend/vbd/3/51744/hotplug-status connected to xenstore.
Jan 26 14:09:23 server logger: /etc/xen/scripts/block: remove
XENBUS_PATH=backend/vbd/3/51744
Jan 26 14:09:23 server logger: /etc/xen/scripts/xen-hotplug-cleanup:
XENBUS_PATH=backend/vbd/3/51744

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[Keir Fraser]

On 26/01/2010 22:12, "0bo0" <0.bugs.only.0@gmail.com> wrote:

> On Tue, Jan 26, 2010 at 1:38 PM, Keir Fraser <keir.fraser@eu.citrix.com>
> wrote:
>> As root:
>>    echo '/var/core/core.%e.%u.%p' > /proc/sys/kernel/core_pattern
>> 
>> Now try starting your guest. It fails, and hopefully you have a core file in
>> /var/core/. You can load it into gdb as 'gdb /usr/lib/xen/qemu-dm
>> /var/core/name-of-core-file' and then execute bt command at the gdb prompt.
>> That hopefully tells us where qemu-dm crashed.
> 
> bash
> ulimit -c unlimited

Actually that probably only enable core dumps for the shell you are running.
Qemu-dm will inherit its ulimits from xend. So you need to enable core dumps
by running 'ulimit -c unlimited' before you start xend, or put it in the
startup script that starts xend. You could also script the echo to
/proc/sys/kernel/core_pattern, but it's definitely safe to do that later
from a shall, as it's a global value, rather than per-process.

 -- Keir

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[0bo0]

On Tue, Jan 26, 2010 at 2:30 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
>>> Now try starting your guest. It fails, and hopefully you have a core file in
>>> /var/core/. You can load it into gdb as 'gdb /usr/lib/xen/qemu-dm
>>> /var/core/name-of-core-file' and then execute bt command at the gdb prompt.
>>> That hopefully tells us where qemu-dm crashed.
>>
>> bash
>> ulimit -c unlimited
>
> Actually that probably only enable core dumps for the shell you are running.
> Qemu-dm will inherit its ulimits from xend. So you need to enable core dumps
> by running 'ulimit -c unlimited' before you start xend, or put it in the
> startup script that starts xend. You could also script the echo to
> /proc/sys/kernel/core_pattern, but it's definitely safe to do that later
> from a shall, as it's a global value, rather than per-process.

bash
ulimit -c unlimited
echo '/var/core/core.%e.%u.%p' > /proc/sys/kernel/core_pattern
mkdir -p /var/core
cat /proc/sys/kernel/core_pattern
/var/core/core.%e.%u.%p

service xend restart
xm create /etc/xen/vm/test.cfg
ls -al /var/core/ | grep core
  -rw-------  1 root root 4542464 2010-01-26 14:36 core.qemu-dm.0.7772

gdb /usr/lib64/xen/bin/qemu-dm /var/core/core.qemu-dm.0.7772
  ...
  /usr/lib64/xen/bin/qemu-dm: Too many levels of symbolic links.
  Missing separate debuginfo for the main executable file
  Try: zypper install -C
"debuginfo(build-id)=9326caa0db4accd719f10b02a9a41a40c10b662a"
  Core was generated by `/usr/lib/xen/bin/qemu-dm -d 4 -serial pty
-domain-name test -videoram 4 -k en-u'.
  Program terminated with signal 11, Segmentation fault.
  #0  0x000000000047b974 in ?? ()
  (gdb) quit

zypper install -C "debuginfo(build-id)=9326caa0db4accd719f10b02a9a41a40c10b662a"
  Loading repository data...
  Reading installed packages...
  'debuginfo(build-id)=9326caa0db4accd719f10b02a9a41a40c10b662a' not found.
  Resolving package dependencies...

so where/how do I get the debug symbols?

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[0bo0]

apparently, the install debuginfo by buildid has -- challenges.

installing as much as I can find by hand,

xm create -c /etc/xen/vm/test.cfg
gdb /usr/lib/xen/bin/qemu-dm /var/core/core.qemu-dm.0.17726

Using config file "/etc/xen/vm/test.cfg".
xenconsole: Could not read tty from store: No such file or directory
GNU gdb (GDB) SUSE (6.8.91.20090930-2.4)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/lib/xen/bin/qemu-dm...(no debugging symbols
found)...done.

warning: Can't read pathname for load map: Input/output error.
Missing separate debuginfo for /lib64/libz.so.1
Try: zypper install -C
"debuginfo(build-id)=4a569edbadd351f13aab9ec81e31a7f31d76a80f"
Missing separate debuginfo for /usr/lib64/libxenctrl.so.4.0
Try: zypper install -C
"debuginfo(build-id)=219d2e00aba1b2bfd538cb1ad88d3eddd6db682b"
Missing separate debuginfo for /usr/lib64/libxenstore.so.3.0
Try: zypper install -C
"debuginfo(build-id)=fcf7d0904b10edcaf468e44ba4e9964478083f29"
Reading symbols from /lib64/libz.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libpthread-2.10.1.so...Reading symbols
from /usr/lib/debug/lib64/libpthread-2.10.1.so.debug...done.
done.
Loaded symbols for /lib64/libpthread-2.10.1.so
Reading symbols from /lib64/librt-2.10.1.so...Reading symbols from
/usr/lib/debug/lib64/librt-2.10.1.so.debug...done.
done.
Loaded symbols for /lib64/librt-2.10.1.so
Reading symbols from /lib64/libutil-2.10.1.so...Reading symbols from
/usr/lib/debug/lib64/libutil-2.10.1.so.debug...done.
done.
Loaded symbols for /lib64/libutil-2.10.1.so
Reading symbols from /usr/lib64/libxenctrl.so.4.0...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libxenctrl.so.4.0
Reading symbols from /usr/lib64/libxenstore.so.3.0...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libxenstore.so.3.0
Reading symbols from /lib64/libpci.so.3.1.3...Reading symbols from
/usr/lib/debug/lib64/libpci.so.3.1.3.debug...done.
done.
Loaded symbols for /lib64/libpci.so.3.1.3
Reading symbols from /usr/lib64/libSDL-1.2.so.0.11.2...Reading symbols
from /usr/lib/debug/usr/lib64/libSDL-1.2.so.0.11.2.debug...done.
done.
Loaded symbols for /usr/lib64/libSDL-1.2.so.0.11.2
Reading symbols from /usr/lib64/libX11.so.6.3.0...Reading symbols from
/usr/lib/debug/usr/lib64/libX11.so.6.3.0.debug...done.
done.
Loaded symbols for /usr/lib64/libX11.so.6.3.0
Reading symbols from /usr/lib64/libGL.so.1.2...Reading symbols from
/usr/lib/debug/usr/lib64/libGL.so.1.2.debug...done.
done.
Loaded symbols for /usr/lib64/libGL.so.1.2
Reading symbols from /lib64/libc-2.10.1.so...Reading symbols from
/usr/lib/debug/lib64/libc-2.10.1.so.debug...done.
done.
Loaded symbols for /lib64/libc-2.10.1.so
Reading symbols from /lib64/ld-2.10.1.so...Reading symbols from
/usr/lib/debug/lib64/ld-2.10.1.so.debug...done.
done.
Loaded symbols for /lib64/ld-2.10.1.so
Reading symbols from /lib64/libresolv-2.10.1.so...Reading symbols from
/usr/lib/debug/lib64/libresolv-2.10.1.so.debug...done.
done.
Loaded symbols for /lib64/libresolv-2.10.1.so
Reading symbols from /lib64/libm-2.10.1.so...Reading symbols from
/usr/lib/debug/lib64/libm-2.10.1.so.debug...done.
done.
Loaded symbols for /lib64/libm-2.10.1.so
Reading symbols from /lib64/libdl-2.10.1.so...Reading symbols from
/usr/lib/debug/lib64/libdl-2.10.1.so.debug...done.
done.
Loaded symbols for /lib64/libdl-2.10.1.so
Reading symbols from /usr/lib64/libXext.so.6.4.0...Reading symbols
from /usr/lib/debug/usr/lib64/libXext.so.6.4.0.debug...done.
done.
Loaded symbols for /usr/lib64/libXext.so.6.4.0
Reading symbols from /usr/lib64/libXrandr.so.2.2.0...Reading symbols
from /usr/lib/debug/usr/lib64/libXrandr.so.2.2.0.debug...done.
done.
Loaded symbols for /usr/lib64/libXrandr.so.2.2.0
Reading symbols from /usr/lib64/libxcb.so.1.1.0...Reading symbols from
/usr/lib/debug/usr/lib64/libxcb.so.1.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libxcb.so.1.1.0
Reading symbols from /usr/lib64/libXxf86vm.so.1.0.0...Reading symbols
from /usr/lib/debug/usr/lib64/libXxf86vm.so.1.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libXxf86vm.so.1.0.0
Reading symbols from /usr/lib64/libXdamage.so.1.1.0...Reading symbols
from /usr/lib/debug/usr/lib64/libXdamage.so.1.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libXdamage.so.1.1.0
Reading symbols from /usr/lib64/libXfixes.so.3.1.0...Reading symbols
from /usr/lib/debug/usr/lib64/libXfixes.so.3.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libXfixes.so.3.1.0
Reading symbols from /usr/lib64/libdrm.so.2.4.0...Reading symbols from
/usr/lib/debug/usr/lib64/libdrm.so.2.4.0.debug...done.
done.
Loaded symbols for /usr/lib64/libdrm.so.2.4.0
Reading symbols from /usr/lib64/libXrender.so.1.3.0...Reading symbols
from /usr/lib/debug/usr/lib64/libXrender.so.1.3.0.debug...done.
done.
Loaded symbols for /usr/lib64/libXrender.so.1.3.0
Reading symbols from /usr/lib64/libXau.so.6.0.0...Reading symbols from
/usr/lib/debug/usr/lib64/libXau.so.6.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libXau.so.6.0.0
Core was generated by `/usr/lib/xen/bin/qemu-dm -d 11 -serial pty
-domain-name test -videoram 4 -k en-'.
Program terminated with signal 11, Segmentation fault.
#0  0x000000000047b974 in bdrv_get_type_hint ()
(gdb) Error: Domain 'test' does not exist.\
(gbd) bt
#0  0x000000000047b974 in bdrv_get_type_hint ()
#1  0x0000000000464082 in xenstore_parse_domain_config ()
#2  0x000000000040c099 in main ()
(gbd) quit

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[0bo0]

On Tue, Jan 26, 2010 at 4:25 PM, 0bo0 <0.bugs.only.0@gmail.com> wrote:
> #1  0x0000000000464082 in xenstore_parse_domain_config ()

making note of the 'domain_config' ...

if in test.cfg i change,

 -   vfb          = [ 'type=vnc, vncdisplay=102, vnclisten=127.0.0.1' ]
 +   #vfb          = [ 'type=vnc, vncdisplay=102, vnclisten=127.0.0.1' ]

then the DomU launches OK,

xm list
  Name                                        ID   Mem VCPUs
State   Time(s)
  Domain-0                                     0  1022     1     r-----    119.5
  test                                          8  4096     4
-b----     17.1

@ DomU ...
uname -a
  Linux test 2.6.31.8-0.1-xen #1 SMP 2009-12-15 23:55:40 +0100 x86_64
x86_64 x86_64 GNU/Linux

so, is the parser broken, or has something changed in allowed vfb specification?

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[Keir Fraser]

On 27/01/2010 00:25, "0bo0" <0.bugs.only.0@gmail.com> wrote:

> (gbd) bt
> #0  0x000000000047b974 in bdrv_get_type_hint ()
> #1  0x0000000000464082 in xenstore_parse_domain_config ()
> #2  0x000000000040c099 in main ()

This is a weird backtrace since bdrv_get_type_hint() is not called from
xenstore_parse_domain_config(). Further, I wouldn't expect
bdrv_get_type_hint() (a function specific to block devices) to be called for
parsing of a vfb configuration, which is apparently the domain-config line
causing the problems.

 -- Keir

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[Keir Fraser]

On 27/01/2010 00:59, "0bo0" <0.bugs.only.0@gmail.com> wrote:

> On Tue, Jan 26, 2010 at 4:25 PM, 0bo0 <0.bugs.only.0@gmail.com> wrote:
>> #1  0x0000000000464082 in xenstore_parse_domain_config ()
> 
> making note of the 'domain_config' ...
> 
> if in test.cfg i change,
> 
>  -   vfb          = [ 'type=vnc, vncdisplay=102, vnclisten=127.0.0.1' ]
>  +   #vfb          = [ 'type=vnc, vncdisplay=102, vnclisten=127.0.0.1' ]

Maybe I can repro the crash.

 -- Keir

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[Keir Fraser]

On 27/01/2010 08:27, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:

>> On Tue, Jan 26, 2010 at 4:25 PM, 0bo0 <0.bugs.only.0@gmail.com> wrote:
>>> #1  0x0000000000464082 in xenstore_parse_domain_config ()
>> 
>> making note of the 'domain_config' ...
>> 
>> if in test.cfg i change,
>> 
>>  -   vfb          = [ 'type=vnc, vncdisplay=102, vnclisten=127.0.0.1' ]
>>  +   #vfb          = [ 'type=vnc, vncdisplay=102, vnclisten=127.0.0.1' ]
> 
> Maybe I can repro the crash.

I had a go but couldn't reproduce the qemu crash. I did note however that
your PV guest only gets a qemu-dm process when you include the vfb config
line above. Hence the crash is not really related to vfb directly, but
simply the fact that qemu starts up and then crashes for some other bizarre
reason. I tried reproducing a vbd config similar to yours (since your crash
backtrace indicated crashing in blockdevice-parsing code), but that caused
no problems for me.

I don't know whether it could be related to the domUloader.py bootloader
stuff. I've never used that and I think it's suse specific?

 -- Keir

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[0bo0]

hi

On Wed, Jan 27, 2010 at 1:10 AM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> I don't know whether it could be related to the domUloader.py bootloader
> stuff. I've never used that and I think it's suse specific?

yes, it's suse-specific (it's already been requested that they support
pvgrub ...)

but, doesn't seem like the issue :-/

removing the use of the domUloader.py bootloader

     builder      = 'linux'
-    bootloader   = '/usr/lib/xen/boot/domUloader.py'
-    bootargs     = '--entry=xvda1:vmlinuz-xen,/boot/initrd-xen'
+   kernel       = '/home/xen_store/distros/opensuse112/xenboot/vmlinuz-xen'
+   ramdisk      = '/home/xen_store/distros/opensuse112/xenboot/initrd-xen'

still results on the same problem/crash -- as lang as

    vfb = ['']

exists.  again, simply removing the vfb spec'n allows the DomU to sanely boot.

thanks.

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[Keir Fraser]

On 27/01/2010 15:41, "0bo0" <0.bugs.only.0@gmail.com> wrote:

> -    bootloader   = '/usr/lib/xen/boot/domUloader.py'
> -    bootargs     = '--entry=xvda1:vmlinuz-xen,/boot/initrd-xen'
> +   kernel       = '/home/xen_store/distros/opensuse112/xenboot/vmlinuz-xen'
> +   ramdisk      = '/home/xen_store/distros/opensuse112/xenboot/initrd-xen'
> 
> still results on the same problem/crash -- as lang as
> 
>     vfb = ['']
> 
> exists.  again, simply removing the vfb spec'n allows the DomU to sanely boot.

Hm, well, can you try stripping your config file right back (but still
specifying vfb=['']) -- like removing all vbds for example? Obviously your
guest will not successfully boot, but qemu-dm crashing is no doubt
preventing your guest from even starting in the first place. So you can
strip back and see whether at any point behaviour changes from 'qemu-dm
crashes' to 'guest partially boots but then fails to find a rootfs'.

 -- Keir

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[Sander Eikelenboom]

Hmm I could suspect parsing errors for the last comma in the disk option, or the empty vif ?


cat /etc/xen/vm/test.cfg
---------------------------------
name         = 'test'
builder      = 'linux'
bootloader   = '/usr/lib/xen/boot/domUloader.py'
bootargs     = '--entry=xvda1:vmlinuz-xen,/boot/initrd-xen'
disk         = [
'phy:/dev/VG0/boot,xvda,w','phy:/dev/VG0/swap,xvdb,w','phy:/dev/VG0/root,xvdc,w',]
boot         = 'c'   #d is cdrom boot, c is disk boot
root         = '/dev/xvdc1'
vif          = ['']
vfb          = [ 'type=vnc, vncdisplay=102, vnclisten=127.0.0.1' ]
extra        = 'textmode=1 xencons=xvc0 elevator=noop'
maxmem       = 4096
memory       = 4096
vcpus        = 4
cpu_cap      = 100
cpu_weight   = 256
localtime    = 0
netif        = 'yes'
on_shutdown  = 'destroy'
on_reboot    = 'restart'
on_crash     = 'destroy'
---------------------------------




Wednesday, January 27, 2010, 4:46:40 PM, you wrote:

> On 27/01/2010 15:41, "0bo0" <0.bugs.only.0@gmail.com> wrote:

>> -    bootloader   = '/usr/lib/xen/boot/domUloader.py'
>> -    bootargs     = '--entry=xvda1:vmlinuz-xen,/boot/initrd-xen'
>> +   kernel       = '/home/xen_store/distros/opensuse112/xenboot/vmlinuz-xen'
>> +   ramdisk      = '/home/xen_store/distros/opensuse112/xenboot/initrd-xen'
>> 
>> still results on the same problem/crash -- as lang as
>> 
>>     vfb = ['']
>> 
>> exists.  again, simply removing the vfb spec'n allows the DomU to sanely boot.

> Hm, well, can you try stripping your config file right back (but still
> specifying vfb=['']) -- like removing all vbds for example? Obviously your
> guest will not successfully boot, but qemu-dm crashing is no doubt
> preventing your guest from even starting in the first place. So you can
> strip back and see whether at any point behaviour changes from 'qemu-dm
> crashes' to 'guest partially boots but then fails to find a rootfs'.

>  -- Keir







-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[0bo0]

On Wed, Jan 27, 2010 at 7:46 AM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> Hm, well, can you try stripping your config file right back (but still
> specifying vfb=['']) -- like removing all vbds for example? Obviously your
> guest will not successfully boot, but qemu-dm crashing is no doubt
> preventing your guest from even starting in the first place. So you can
> strip back and see whether at any point behaviour changes from 'qemu-dm
> crashes' to 'guest partially boots but then fails to find a rootfs'.

i've been stripping back as suggested, and trying different config
option combos.  nothing seems to make any difference to ability to
launch, so far, except removal of 'vfb=['']' :-/

i've switched to booting from a different local vmlinuz & initrd,
instead of either the domUbootloader.py or the distro release -xen
imgs, and have noticed one change in logs:

...
[2010-01-27 08:16:51 2217] WARNING (XendDomainInfo:2072) Domain has
crashed: name=nas id=30.
[2010-01-27 08:16:51 2217] DEBUG (XendDomainInfo:3061)
XendDomainInfo.destroy: domid=30
[2010-01-27 08:16:52 2217] DEBUG (pciif:460)
XendDomainInfo.handleAerStateWatch
[2010-01-27 08:16:52 2217] DEBUG (XendDomainInfo:1892)
XendDomainInfo.handleShutdownWatch
[2010-01-27 08:16:52 2217] DEBUG (XendDomainInfo:2416) Destroying
device model
[2010-01-27 08:16:53 2217] DEBUG (XendDomainInfo:2423) Releasing
devices
[2010-01-27 08:16:53 2217] DEBUG (XendDomainInfo:2429) Removing vif/0
[2010-01-27 08:16:53 2217] DEBUG (XendDomainInfo:1271)
XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0
[2010-01-27 08:16:54 2217] DEBUG (XendDomainInfo:2429) Removing vif/1
[2010-01-27 08:16:54 2217] DEBUG (XendDomainInfo:1271)
XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/1
[2010-01-27 08:16:54 2217] DEBUG (XendDomainInfo:2429) Removing vif/2
[2010-01-27 08:16:54 2217] DEBUG (XendDomainInfo:1271)
XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/2
[2010-01-27 08:16:54 2217] DEBUG (XendDomainInfo:2429) Removing vif/3
[2010-01-27 08:16:54 2217] DEBUG (XendDomainInfo:1271)
XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/3
[2010-01-27 08:16:54 2217] DEBUG (XendDomainInfo:2429) Removing vkbd/0
[2010-01-27 08:16:54 2217] DEBUG (XendDomainInfo:1271)
XendDomainInfo.destroyDevice: deviceClass = vkbd, device = vkbd/0
[2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:2429) Removing
vbd/51712
[2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:1271)
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51712
[2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:2429) Removing
vbd/51728
[2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:1271)
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51728
[2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:2429) Removing
vbd/51744
[2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:1271)
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51744
[2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:2429) Removing vfb/0
[2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:1271)
XendDomainInfo.destroyDevice: deviceClass = vfb, device = vfb/0
[2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:2429) Removing pci/0
[2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:1271)
XendDomainInfo.destroyDevice: deviceClass = pci, device = pci/0
[2010-01-27 08:16:55 2217] DEBUG (pciif:578) pci: unregister aer watch
[2010-01-27 08:16:55 2217] ERROR (XendConfig:1175) dumping sxp from
device controllers
Traceback (most recent call last):
  File "/usr/lib64/python2.6/site-packages/xen/xend/XendConfig.py",
line 1161, in to_sxp
    configs = controller.configurations(txn)
  File "/usr/lib64/python2.6/site-packages/xen/xend/server/DevController.py",
line 242, in configurations
    return map(lambda x: self.configuration(x, transaction),
self.deviceIDs(transaction))
  File "/usr/lib64/python2.6/site-packages/xen/xend/server/DevController.py",
line 242, in <lambda>
    return map(lambda x: self.configuration(x, transaction),
self.deviceIDs(transaction))
  File "/usr/lib64/python2.6/site-packages/xen/xend/server/DevController.py",
line 249, in configuration
    configDict = self.getDeviceConfiguration(devid, transaction)
  File "/usr/lib64/python2.6/site-packages/xen/xend/server/netif.py",
line 194, in getDeviceConfiguration
    y = self.readBackendTxn(transaction, devid, x)
  File "/usr/lib64/python2.6/site-packages/xen/xend/server/DevController.py",
line 447, in readBackendTxn
    raise VmError("Device %s not connected" % devid)
VmError: Device 2 not connected

the

[2010-01-27 08:16:55 2217] ERROR (XendConfig:1175) dumping sxp from
device controllers
Traceback (most recent call last):
...

are not there in other circumstances ...

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[0bo0]

On Wed, Jan 27, 2010 at 8:19 AM, Sander Eikelenboom
<linux@eikelenboom.it> wrote:
> Hmm I could suspect parsing errors for the last comma in the disk option, or the empty vif ?

although neither was a problem under Xen3x, never hurts to try.

i removed the trailing comma, and replaced the populated vif.

no luck -- same inability to boot, until/unless 'vfb=['']' is removed.

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[0bo0]

fyi,


BOOTS OK
 name         = 'test'
 builder      = 'linux'
 kernel       = '/boot/vmlinuz-2.6.31.12-2-xen'
 ramdisk      = '/boot/initrd-2.6.31.12-2-xen'
 disk         = [
'phy:/dev/VG0/boot,xvda,w','phy:/dev/VG0/swap,xvdb,w','phy:/dev/VG0/root,xvdc,w',]
 root         = '/dev/xvdc1'
 #vfb          = ['']

CRASHES @ qemu-dm, AS REPORTED
 name         = 'test'
 builder      = 'linux'
 kernel       = '/boot/vmlinuz-2.6.31.12-2-xen'
 ramdisk      = '/boot/initrd-2.6.31.12-2-xen'
 disk         = [
'phy:/dev/VG0/boot,xvda,w','phy:/dev/VG0/swap,xvdb,w','phy:/dev/VG0/root,xvdc,w',]
 root         = '/dev/xvdc1'
 vfb          = ['']

NO CRASH, JUST WAITS FOR NON-EXSITENT DISK
 name         = 'test'
 builder      = 'linux'
 kernel       = '/boot/vmlinuz-2.6.31.12-2-xen'
 ramdisk      = '/boot/initrd-2.6.31.12-2-xen'
 #disk         = [
'phy:/dev/VG0/boot,xvda,w','phy:/dev/VG0/swap,xvdb,w','phy:/dev/VG0/root,xvdc,w',]
 root         = '/dev/xvdc1'
 vfb          = ['']

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[0bo0]

just fyi, i did a quick check & reproduced this behavior on a
colleague's box.  not configured/installed by me, but admittedly still
opensuse.

afaict, with vfb=[''] removed from the config, there are no suspicious
effects i see at the DomU -- yet.

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[Justin T. Gibbs]

[-- Attachment #1.1: Type: text/plain, Size: 19985 bytes --]

On 1/26/2010 11:36 AM, 0bo0 wrote:
>  in another thread @xen-devel, a crash @ DomU init under Xen 4.0.x, i
>  was advised,
>
>  On Mon, Jan 25, 2010 at 11:34 PM, Keir Fraser 
<keir.fraser@eu.citrix.com> wrote:
> > If qemu-dm died without logging anything, it probably crashed again. 
If it
> > produced a core file you could load that in gdb and get a symbolic
> > backtrace.
>
>  trying to turn on core dumps in xend-config, unfortunately, Xen seems
>  to fail @ core dump ... in addition to failing to launch a DomU which
>  works fine under Xen3,

...

I encountered this same problem.  I debugged it by pulling the OpenSuSE
source package for xen-4.0.0 and then manually rebuilding qemu with
symbols.  The problem is in a supplemental patch that OpenSuSE is adding
for CDROM media change support: cdrom-removable.patch.  This patched code
accesses an uninitialized variable, causing the segfault.

If you want to rebuild the package from source, I've attached a corrected
version of the cdrom-removable.patch file.  Just install the source rpm
(zypper si xen), replace the /usr/src/packages/SOURCES/cdrom-removable.patch
file with what is below, rebuild your packages (rpmbuild -ba
/usr/src/packages/SPECS/xen.spec), and install the packages found in
/usr/src/packages/RPMS/<arch>/.  You only need to reinstall xen-tools
to get the fix.

I've also posted information about this problem to the OpenSuSE-virtual
list.  Hopefully they will quickly incorporate a fix.

--
Justin

Index: xen-4.0.0-testing/tools/python/xen/xend/server/HalDaemon.py
===================================================================
--- /dev/null
+++ xen-4.0.0-testing/tools/python/xen/xend/server/HalDaemon.py
@@ -0,0 +1,243 @@
+#!/usr/bin/env python
+#  -*- mode: python; -*-
+#============================================================================
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of version 2.1 of the GNU Lesser General Public
+# License as published by the Free Software Foundation.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+#============================================================================
+# Copyright (C) 2007 Pat Campbell <plc@novell.com>
+# Copyright (C) 2007 Novell Inc.
+#============================================================================
+
+"""hald (Hardware Abstraction Layer Daemon) watcher for Xen management
+   of removable block device media.
+
+"""
+
+import gobject
+import dbus
+import dbus.glib
+import os
+import types
+import sys
+import signal
+import traceback
+from xen.xend.xenstore.xstransact import xstransact, complete
+from xen.xend.xenstore.xsutil import xshandle
+from xen.xend import PrettyPrint
+from xen.xend import XendLogging
+from xen.xend.XendLogging import log
+
+DEVICE_TYPES = ['vbd', 'tap']
+
+class HalDaemon:
+    """The Hald block device watcher for XEN
+    """
+
+    """Default path to the log file. """
+    logfile_default = "/var/log/xen/hald.log"
+
+    """Default level of information to be logged."""
+    loglevel_default = 'INFO'
+
+
+    def __init__(self):
+
+        XendLogging.init(self.logfile_default, self.loglevel_default)
+        log.debug( "%s", "__init__")
+
+        self.udi_dict = {}
+        self.debug = 0
+        self.dbpath = "/local/domain/0/backend"
+        self.bus = dbus.SystemBus()
+        self.hal_manager_obj = 
self.bus.get_object('org.freedesktop.Hal', '/org/freedesktop/Hal/Manager')
+        self.hal_manager = dbus.Interface( self.hal_manager_obj, 
'org.freedesktop.Hal.Manager')
+        self.gatherBlockDevices()
+        self.registerDeviceCallbacks()
+
+    def run(self):
+        log.debug( "%s", "In new run" );
+        try:
+            self.mainloop = gobject.MainLoop()
+            self.mainloop.run()
+        except KeyboardInterrupt, ex:
+            log.debug('Keyboard exception handler: %s', ex )
+            self.mainloop.quit()
+        except Exception, ex:
+            log.debug('Generic exception handler: %s', ex )
+            self.mainloop.quit()
+
+    def __del__(self):
+        log.debug( "%s", "In del " );
+        self.unRegisterDeviceCallbacks()
+        self.mainloop.quit()
+
+    def shutdown(self):
+        log.debug( "%s", "In shutdown now " );
+        self.unRegisterDeviceCallbacks()
+        self.mainloop.quit()
+
+    def stop(self):
+        log.debug( "%s", "In stop now " );
+        self.unRegisterDeviceCallbacks()
+        self.mainloop.quit()
+
+    def gatherBlockDevices(self):
+
+        # Get all the current devices from hal and save in a dictionary
+        try:
+            device_names = self.hal_manager.GetAllDevices()
+            i = 0;
+            for name in device_names:
+                #log.debug("device name, device=%s",name)
+               dev_obj = self.bus.get_object ('org.freedesktop.Hal', name)
+               dev = dbus.Interface (dev_obj, 'org.freedesktop.Hal.Device')
+               dev_properties = 
dev_obj.GetAllProperties(dbus_interface="org.freedesktop.Hal.Device")
+               if dev_properties.has_key('block.device'):
+                   dev_str = dev_properties['block.device']
+                   dev_major = dev_properties['block.major']
+                   dev_minor = dev_properties['block.minor']
+                   udi_info = {}
+                   udi_info['device'] = dev_str
+                   udi_info['major'] = dev_major
+                   udi_info['minor'] = dev_minor
+                   udi_info['udi'] = name
+                   self.udi_dict[i] = udi_info
+                   i = i + 1
+        except Exception, ex:
+            print >>sys.stderr, 'Exception gathering block devices:', ex
+            log.warn("Exception gathering block devices (%s)",ex)
+
+    #
+    def registerDeviceCallbacks(self):
+        # setup the callbacks for when the gdl changes
+        self.hal_manager.connect_to_signal('DeviceAdded', 
self.device_added_callback)
+        self.hal_manager.connect_to_signal('DeviceRemoved', 
self.device_removed_callback)
+
+    #
+    def unRegisterDeviceCallbacks(self):
+        # setup the callbacks for when the gdl changes
+        
self.hal_manager.remove_signal_receiver(self.device_added_callback,'DeviceAdded')
+        
self.hal_manager.remove_signal_receiver(self.device_removed_callback,'DeviceRemoved')
+
+    #
+    def device_removed_callback(self,udi):
+        log.debug('UDI %s was removed',udi)
+        self.show_dict(self.udi_dict)
+        for key in self.udi_dict:
+            udi_info = self.udi_dict[key]
+            if udi_info['udi'] == udi:
+                device = udi_info['device']
+                major = udi_info['major']
+                minor = udi_info['minor']
+                self.change_xenstore( "remove", device, major, minor)
+
+    # Adds device to dictionary if not already there
+    def device_added_callback(self,udi):
+        log.debug('UDI %s was added', udi)
+        self.show_dict(self.udi_dict)
+        dev_obj = self.bus.get_object ('org.freedesktop.Hal', udi)
+        dev = dbus.Interface (dev_obj, 'org.freedesktop.Hal.Device')
+        device = dev.GetProperty ('block.device')
+        major = dev.GetProperty ('block.major')
+        minor = dev.GetProperty ('block.minor')
+        udi_info = {}
+        udi_info['device'] = device
+        udi_info['major'] = major
+        udi_info['minor'] = minor
+        udi_info['udi'] = udi
+        already = 0
+        cnt = 0;
+        for key in self.udi_dict:
+            info = self.udi_dict[key]
+            if info['udi'] == udi:
+                already = 1
+                break
+            cnt = cnt + 1
+        if already == 0:
+           self.udi_dict[cnt] = udi_info;
+           log.debug('UDI %s was added, device:%s major:%s minor:%s 
index:%d\n', udi, device, major, minor, cnt)
+        self.change_xenstore( "add", device, major, minor)
+
+    # Debug helper, shows dictionary contents
+    def show_dict(self,dict=None):
+        if self.debug == 0 :
+            return
+        if dict == None :
+            dict = self.udi_dict
+        for key in dict:
+            log.debug('udi_info %s udi_info:%s',key,dict[key])
+
+    # Set or clear xenstore media-present depending on the action argument
+    #  for every vbd that has this block device
+    def change_xenstore(self,action, device, major, minor):
+        for type in DEVICE_TYPES:
+            path = self.dbpath + '/' + type
+            domains = xstransact.List(path)
+            log.debug('domains: %s', domains)
+            for domain in domains:   # for each domain
+                devices = xstransact.List( path + '/' + domain)
+                log.debug('devices: %s',devices)
+                for device in devices:  # for each vbd device
+                   str = device.split('/')
+                   vbd_type = None;
+                   vbd_physical_device = None
+                   vbd_media = None
+                   vbd_device_path = path + '/' + domain + '/' + device
+                   listing = xstransact.List(vbd_device_path)
+                   for entry in listing: # for each entry
+                       item = path + '/' + entry
+                       value = xstransact.Read( vbd_device_path + '/' + 
entry)
+                       log.debug('%s=%s',item,value)
+                       if item.find('media-present') != -1:
+                           vbd_media = item;
+                           vbd_media_path = item
+                       if item.find('physical-device') != -1:
+                           vbd_physical_device = value;
+                       if item.find('type') != -1:
+                           vbd_type = value;
+                   if vbd_type is not None and vbd_physical_device is 
not None and vbd_media is not None :
+                       inode = vbd_physical_device.split(':')
+                       imajor = parse_hex(inode[0])
+                       iminor = parse_hex(inode[1])
+                       log.debug("action:%s major:%s- minor:%s- 
imajor:%s- iminor:%s- inode: %s",
+                               action,major,minor, imajor, iminor, inode)
+                       if int(imajor) == int(major) and int(iminor) == 
int(minor):
+                           if action == "add":
+                               xs_dict = {'media': "1"}
+                               xstransact.Write(vbd_device_path, 
'media-present', "1" )
+                               log.debug("wrote xenstore media-present 
1 path:%s",vbd_media_path)
+                           else:
+                               xstransact.Write(vbd_device_path, 
'media-present', "0" )
+                               log.debug("wrote xenstore media 0 
path:%s",vbd_media_path)
+
+def mylog( fmt, *args):
+    f = open('/tmp/haldaemon.log', 'a')
+    print >>f, "HalDaemon ", fmt % args
+    f.close()
+
+
+def parse_hex(val):
+    try:
+        if isinstance(val, types.StringTypes):
+            return int(val, 16)
+        else:
+            return val
+    except ValueError:
+        return None
+
+if __name__ == "__main__":
+    watcher = HalDaemon()
+    watcher.run()
+    print 'Falling off end'
+
+
Index: xen-4.0.0-testing/tools/python/xen/xend/server/Hald.py
===================================================================
--- /dev/null
+++ xen-4.0.0-testing/tools/python/xen/xend/server/Hald.py
@@ -0,0 +1,125 @@
+#============================================================================
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of version 2.1 of the GNU Lesser General Public
+# License as published by the Free Software Foundation.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+#============================================================================
+# Copyright (C) 2007 Pat Campbell <plc@novell.com>
+# Copyright (C) 2007 Novell Inc.
+#============================================================================
+
+import errno
+import types
+import os
+import sys
+import time
+import signal
+from traceback import print_exc
+
+from xen.xend.XendLogging import log
+
+class Hald:
+    def __init__(self):
+        self.ready = False
+        self.running = True
+
+    def run(self):
+        """Starts the HalDaemon process
+        """
+        self.ready = True
+        try:
+            myfile =  self.find("xen/xend/server/HalDaemon.py")
+            args = (["python", myfile ])
+            self.pid = self.daemonize("python", args )
+            #log.debug( "%s %s pid:%d", "Hald.py starting ", args, 
self.pid )
+        except:
+            self.pid = -1
+            log.debug("Unable to start HalDaemon process")
+
+    def shutdown(self):
+        """Shutdown the HalDaemon process
+        """
+        log.debug("%s  pid:%d", "Hald.shutdown()", self.pid)
+        self.running = False
+        self.ready = False
+        if self.pid != -1:
+            try:
+                os.kill(self.pid, signal.SIGINT)
+            except:
+                print_exc()
+
+    def daemonize(self,prog, args):
+        """Runs a program as a daemon with the list of arguments.  
Returns the PID
+        of the daemonized program, or returns 0 on error.
+        Copied from xm/create.py instead of importing to reduce coupling
+        """
+        r, w = os.pipe()
+        pid = os.fork()
+
+        if pid == 0:
+            os.close(r)
+            w = os.fdopen(w, 'w')
+            os.setsid()
+            try:
+                pid2 = os.fork()
+            except:
+                pid2 = None
+            if pid2 == 0:
+                os.chdir("/")
+                env = os.environ.copy()
+                env['PYTHONPATH'] = self.getpythonpath()
+                for fd in range(0, 256):
+                    try:
+                        os.close(fd)
+                    except:
+                        pass
+                os.open("/dev/null", os.O_RDWR)
+                os.dup2(0, 1)
+                os.dup2(0, 2)
+                os.execvpe(prog, args, env)
+                os._exit(1)
+            else:
+                w.write(str(pid2 or 0))
+                w.close()
+                os._exit(0)
+        os.close(w)
+        r = os.fdopen(r)
+        daemon_pid = int(r.read())
+        r.close()
+        os.waitpid(pid, 0)
+        #log.debug( "daemon_pid: %d", daemon_pid )
+        return daemon_pid
+
+    def getpythonpath(self):
+        str = " "
+        for p in sys.path:
+            if str != " ":
+                str = str + ":" + p
+            else:
+                if str != "":
+                   str = p
+        return str
+
+    def find(self,path, matchFunc=os.path.isfile):
+        """Find a module in the sys.path
+        From web page: 
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52224
+        """
+        for dirname in sys.path:
+            candidate = os.path.join(dirname, path)
+            if matchFunc(candidate):
+                return candidate
+        raise Error("Can't find file %s" % path)
+
+if __name__ == "__main__":
+    watcher = Hald()
+    watcher.run()
+    time.sleep(10)
+    watcher.shutdown()
Index: xen-4.0.0-testing/tools/python/xen/xend/server/SrvServer.py
===================================================================
--- xen-4.0.0-testing.orig/tools/python/xen/xend/server/SrvServer.py
+++ xen-4.0.0-testing/tools/python/xen/xend/server/SrvServer.py
@@ -56,6 +56,7 @@ from xen.web.SrvDir import SrvDir

  from SrvRoot import SrvRoot
  from XMLRPCServer import XMLRPCServer
+from xen.xend.server.Hald import Hald

  xoptions = XendOptions.instance()

@@ -245,6 +246,8 @@ def _loadConfig(servers, root, reload):
      if xoptions.get_xend_unix_xmlrpc_server():
          servers.add(XMLRPCServer(XendAPI.AUTH_PAM, False))

+    servers.add(Hald())
+

  def create():
      root = SrvDir()
Index: xen-4.0.0-testing/tools/ioemu-remote/xenstore.c
===================================================================
--- xen-4.0.0-testing.orig/tools/ioemu-remote/xenstore.c
+++ xen-4.0.0-testing/tools/ioemu-remote/xenstore.c
@@ -594,6 +594,20 @@
  #endif

          bs = bdrv_new(dev);
+        /* if cdrom pyhsical put a watch on media-present */
+        if (bdrv_get_type_hint(bs) ==  BDRV_TYPE_CDROM) {
+            if (drv && !strcmp(drv, "phy")) {
+                if (pasprintf(&buf, "%s/media-present", bpath) != -1) {
+                    if (bdrv_is_inserted(bs))
+                        xs_write(xsh, XBT_NULL, buf, "1", strlen("1"));
+                    else {
+                        xs_write(xsh, XBT_NULL, buf, "0", strlen("0"));
+                    }
+                    xs_watch(xsh, buf, "media-present");
+                }
+            }
+        }
+
          /* check if it is a cdrom */
          if (danger_type && !strcmp(danger_type, "cdrom")) {
              bdrv_set_type_hint(bs, BDRV_TYPE_CDROM);
@@ -938,6 +951,50 @@ void xenstore_record_dm_state(const char
      xenstore_record_dm("state", state);
  }

+void xenstore_process_media_change_event(char **vec)
+{
+    char *media_present = NULL;
+    unsigned int len;
+
+    media_present = xs_read(xsh, XBT_NULL, vec[XS_WATCH_PATH], &len);
+
+    if (media_present) {
+        BlockDriverState *bs;
+        char *buf = NULL, *cp = NULL, *path = NULL, *dev = NULL;
+
+        path = strdup(vec[XS_WATCH_PATH]);
+        cp = strstr(path, "media-present");
+        if (cp){
+            *(cp-1) = '\0';
+            pasprintf(&buf, "%s/dev", path);
+            dev = xs_read(xsh, XBT_NULL, buf, &len);
+            if (dev) {
+                if ( !strncmp(dev, "xvd", 3)) {
+                    memmove(dev, dev+1, strlen(dev));
+                    dev[0] = 'h';
+                    dev[1] = 'd';
+                }
+                bs = bdrv_find(dev);
+                if (!bs) {
+                    term_printf("device not found\n");
+                    return;
+                }
+                if (strcmp(media_present, "0") == 0 && bs) {
+                    bdrv_close(bs);
+                }
+                else if (strcmp(media_present, "1") == 0 &&
+                        bs != NULL && bs->drv == NULL) {
+                    if (bdrv_open(bs, bs->filename, 0 /* snapshot */) < 
0) {
+                        fprintf(logfile, "%s() qemu: could not open 
cdrom disk '%s'\n",
+                                __func__, bs->filename);
+                    }
+                    bs->media_changed = 1;
+                }
+            }
+        }
+    }
+}
+
  void xenstore_process_event(void *opaque)
  {
      char **vec, *offset, *bpath = NULL, *buf = NULL, *drv = NULL, 
*image = NULL;
@@ -968,6 +1025,11 @@ void xenstore_process_event(void *opaque
              xenstore_watch_callbacks[i].cb(vec[XS_WATCH_TOKEN],
                                             
xenstore_watch_callbacks[i].opaque);

+    if (!strcmp(vec[XS_WATCH_TOKEN], "media-present")) {
+        xenstore_process_media_change_event(vec);
+        goto out;
+    }
+
      if (strncmp(vec[XS_WATCH_TOKEN], "hd", 2) ||
          strlen(vec[XS_WATCH_TOKEN]) != 3)
          goto out;


[-- Attachment #1.2: Type: text/html, Size: 41115 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[0bo0]

hi

On Thu, Jan 28, 2010 at 4:21 PM, Justin T. Gibbs <gibbs@scsiguy.com> wrote:
> I encountered this same problem.

_which_ problem did you encounter? getting hard to keep track of them all :-)

does your patch address

(1) the crash on DomU launch in the presence of 'vfb=['']'?

or

(2) the failure to core dump with dumps enabled in xend-config.sxp

?

thanks.

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[Justin T. Gibbs]

On 1/28/2010 5:42 PM, 0bo0 wrote:
> hi
>
> On Thu, Jan 28, 2010 at 4:21 PM, Justin T. Gibbs<gibbs@scsiguy.com>  wrote:
>    
>> I encountered this same problem.
>>      
> _which_ problem did you encounter? getting hard to keep track of them all :-)
>
> does your patch address
>
> (1) the crash on DomU launch in the presence of 'vfb=['']'?
>    

Just 1.

--
Justin

Re: trying to debug xen4 & qemu-dm, Xen reports "core dump failed"

[0bo0]

On Thu, Jan 28, 2010 at 4:44 PM, Justin T. Gibbs <gibbs@scsiguy.com> wrote:
>> (1) the crash on DomU launch in the presence of 'vfb=['']'?
>>
>
> Just 1.

thanks. in that case, your patch is relevant here
https://bugzilla.novell.com/show_bug.cgi?id=574046#c4, despite the
protestations that Xen4 on OpenSuse 11.2 is something they neither
support nor test :-S

it's already referenced there, added by @novell ... now, perhaps,
they'll include the fix.

thanks again.