2.6.37 PV on HVM issues

2.6.37 PV on HVM issues

[Alex Bligh]

I'm running Ubuntu's current kernel from Natty which I believe
to be 2.6.37-rc1, and hoping to get PV drivers on HVM working.
I think I must be missing something as despite a helpful dmesg
I can't actually see any PV devices (or think I can't).

I see this:

# uname -a ; dmesg | egrep -i '(pv|xen|hvm)'
Linux 109-231-66-11 2.6.37-7-virtual #19-Ubuntu SMP Wed Dec 1 02:15:20 UTC 
2010 x86_64 GNU/Linux
[    0.000000] DMI: /HVM domU, BIOS 3.3.1 02/02/2009
[    0.000000] Hypervisor detected: Xen HVM       <-----********************
[    0.000000] Xen version 3.3.
[    0.000000] Xen Platform PCI: unrecognised magic value  <--- ++++++++++++
[    0.000000] HVMOP_pagetable_dying not supported
[    0.000000] ACPI: RSDP 00000000000ea010 00024 (v02    Xen)
[    0.000000] ACPI: XSDT 000000001fff8b70 00034 (v01    Xen      HVM 
00000000 HVML 00000000)
[    0.000000] ACPI: FACP 000000001fff8a00 000F4 (v04    Xen      HVM 
00000000 HVML 00000000)
[    0.000000] ACPI: DSDT 000000001fff7840 01132 (v02    Xen      HVM 
00000000 INTL 20060707)
[    0.000000] ACPI: APIC 000000001fff8b00 00068 (v02    Xen      HVM 
00000000 HVML 00000000)
[    0.000000] Booting paravirtualized kernel on Xen <---*******************
[    0.800582] Initialising Xen virtual ethernet driver.
[   13.980075] eth0: no IPv6 routers present

The lines marked with asterisks are confusing to me as they appear to be
in contradiction.

Should I be worried about the line marked +++++?

Moreover, despite the line re "initialising Xen virtual ethernet driver"
I see one ethernet device (eth0), and one block device (sda). I was
expecting to see eth1 and either vbda or xvdba.

$ ls -lad /sys/block/sda
lrwxrwxrwx 1 root root 0 2010-12-02 14:18 /sys/block/sda -> 
../devices/pci0000:00/0000:00:01.1/host0/target0:0:0/0:0:0:0/block/sda
$ ls -lad /sys/class/net/eth0
lrwxrwxrwx 1 root root 0 2010-12-02 14:03 /sys/class/net/eth0 -> 
../../devices/pci0000:00/0000:00:04.0/net/eth0

This suggests they are not paravirtualised.

# egrep _XEN_ /boot/config-2.6.37-7-virtual
CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=128
CONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not set
CONFIG_XEN_PCIDEV_FRONTEND=m
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_XEN_KBDDEV_FRONTEND=m
CONFIG_XEN_FBDEV_FRONTEND=m
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=m
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_PLATFORM_PCI=m

That would seem to be right to me.

-- 
Alex Bligh

Re: 2.6.37 PV on HVM issues

[Alex Bligh]

--On 2 December 2010 15:23:16 +0000 Alex Bligh <alex@alex.org.uk> wrote:

> I'm running Ubuntu's current kernel from Natty which I believe
> to be 2.6.37-rc1, and hoping to get PV drivers on HVM working.
> I think I must be missing something as despite a helpful dmesg
> I can't actually see any PV devices (or think I can't).

I should have added that PV drivers on HVM using either XenSource
2.6.18 or the patches on my blog, or any other patches I have
tried work just fine.

-- 
Alex Bligh

Re: 2.6.37 PV on HVM issues

[Stefano Stabellini]

On Thu, 2 Dec 2010, Alex Bligh wrote:
> I'm running Ubuntu's current kernel from Natty which I believe
> to be 2.6.37-rc1, and hoping to get PV drivers on HVM working.
> I think I must be missing something as despite a helpful dmesg
> I can't actually see any PV devices (or think I can't).
> 
> I see this:
> 
> # uname -a ; dmesg | egrep -i '(pv|xen|hvm)'
> Linux 109-231-66-11 2.6.37-7-virtual #19-Ubuntu SMP Wed Dec 1 02:15:20 UTC 
> 2010 x86_64 GNU/Linux
> [    0.000000] DMI: /HVM domU, BIOS 3.3.1 02/02/2009
> [    0.000000] Hypervisor detected: Xen HVM       <-----********************
> [    0.000000] Xen version 3.3.
> [    0.000000] Xen Platform PCI: unrecognised magic value  <--- ++++++++++++
> [    0.000000] HVMOP_pagetable_dying not supported
> [    0.000000] ACPI: RSDP 00000000000ea010 00024 (v02    Xen)
> [    0.000000] ACPI: XSDT 000000001fff8b70 00034 (v01    Xen      HVM 
> 00000000 HVML 00000000)
> [    0.000000] ACPI: FACP 000000001fff8a00 000F4 (v04    Xen      HVM 
> 00000000 HVML 00000000)
> [    0.000000] ACPI: DSDT 000000001fff7840 01132 (v02    Xen      HVM 
> 00000000 INTL 20060707)
> [    0.000000] ACPI: APIC 000000001fff8b00 00068 (v02    Xen      HVM 
> 00000000 HVML 00000000)
> [    0.000000] Booting paravirtualized kernel on Xen <---*******************
> [    0.800582] Initialising Xen virtual ethernet driver.
> [   13.980075] eth0: no IPv6 routers present
> 
> The lines marked with asterisks are confusing to me as they appear to be
> in contradiction.
> 

They mean that Linux recognized that it is running on Xen but the
platform pci device is too old to support the device unplug protocol,
therefore pv drivers have been disabled.
You can try to enable them anyway at your own risk passing
xen_emul_unplug=unnecessary to the kernel command line options.
Be careful with enabling both the pv disk and the IDE disk interfaces
at the same time because it might cause disk data corruptions.

Re: 2.6.37 PV on HVM issues

[Alex Bligh]

Stefano,

--On 2 December 2010 15:41:08 +0000 Stefano Stabellini 
<stefano.stabellini@eu.citrix.com> wrote:
>> [    0.000000] Hypervisor detected: Xen HVM
>> [    0.000000] Xen Platform PCI: unrecognised magic value
>> [    0.000000] Booting paravirtualized kernel on Xen
>> [    0.800582] Initialising Xen virtual ethernet
>> driver.
>> [   13.980075] eth0: no IPv6 routers present
>>
>> The lines marked with asterisks are confusing to me as they appear to be
>> in contradiction.
>>
>
> They mean that Linux recognized that it is running on Xen but the
> platform pci device is too old to support the device unplug protocol,
> therefore pv drivers have been disabled.

OK. How modern a version of Xen am I meant to need to run 2.6.37? This
version of Xen supports non-inline pvdrivers fine.

> You can try to enable them anyway at your own risk passing
> xen_emul_unplug=unnecessary to the kernel command line options.

OK, so I get the following, which looks superficially nasty, but
in fact suggests I need to unplug the disks. But the same happens
with "xen_emul_unplug=unnecessary,all".

[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 2.6.37-7-virtual (buildd@yellow) (gcc version 
4.5.2 (Ubuntu/Linaro 4.5.1-10ubuntu3) ) #19-Ubuntu SMP Wed Dec 1 02:15:20 
UTC 2010 (Ubuntu 2.6.37-7.19-virtual 2.6.37-rc3)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-2.6.37-7-virtual 
root=UUID=f7f04f87-04f3-488b-a6ae-014ce5d63850 ro 
xen_emul_unplug=unnecessary console=ttyS0
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
--
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI 2.4 present.
[    0.000000] DMI: /HVM domU, BIOS 3.3.1 02/02/2009
[    0.000000] Hypervisor detected: Xen HVM
[    0.000000] Xen version 3.3.
[    0.000000] Xen Platform PCI: unrecognised magic value
[    0.000000] HVMOP_pagetable_dying not supported
[    0.000000] e820 update range: 0000000000000000 - 0000000000010000 
(usable) ==> (reserved)
[    0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 
(usable)
--
[    0.000000] kernel direct mapping tables up to 1fff7000 @ 
1fff4000-1fff7000
[    0.000000] RAMDISK: 1f348000 - 1f799000
[    0.000000] ACPI: RSDP 00000000000ea010 00024 (v02    Xen)
[    0.000000] ACPI: XSDT 000000001fff8b70 00034 (v01    Xen      HVM 
00000000 HVML 00000000)
[    0.000000] ACPI: FACP 000000001fff8a00 000F4 (v04    Xen      HVM 
00000000 HVML 00000000)
[    0.000000] ACPI: DSDT 000000001fff7840 01132 (v02    Xen      HVM 
00000000 INTL 20060707)
[    0.000000] ACPI: FACS 000000001fff7800 00040
[    0.000000] ACPI: APIC 000000001fff8b00 00068 (v02    Xen      HVM 
00000000 HVML 00000000)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] No NUMA configuration found
--
[    0.000000] PM: Registered nosave memory: 00000000000e0000 - 
0000000000100000
[    0.000000] Allocating PCI resources starting at 20000000 (gap: 
20000000:e0000000)
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:1 
nr_node_ids:1
[    0.000000] PERCPU: Embedded 28 pages/cpu @ffff88001fc00000 s83712 r8192 
d22784 u2097152
--
[    0.000000] Built 1 zonelists in Node order, mobility grouping on. 
Total pages: 129152
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: 
BOOT_IMAGE=/boot/vmlinuz-2.6.37-7-virtual 
root=UUID=f7f04f87-04f3-488b-a6ae-014ce5d63850 ro 
xen_emul_unplug=unnecessary console=ttyS0
[    0.000000] PID hash table entries: 2048 (order: 2, 16384 bytes)
[    0.000000] Checking aperture...
--
[    0.821839] Fixed MDIO Bus: probed
[    0.823360] PPP generic driver version 2.4.2
[    0.825496] Initialising Xen virtual ethernet driver.
[    0.827998] tun: Universal TUN/TAP device driver, 1.6
[    0.830247] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
--
[    2.791086] xen-platform-pci 0000:00:03.0: PCI INT A -> GSI 28 (level, 
low) -> IRQ 28
[    2.798695] Grant table initialized
[    2.889459] blkfront device/vbd/51712 num-ring-pages 1 nr_ents 32.
[    2.897163] blkfront: sda: barriers disabled
[    2.900483] ------------[ cut here ]------------
[    2.900608] WARNING: at /build/buildd/linux-2.6.37/fs/sysfs/dir.c:451 
sysfs_add_one+0xd8/0x150()
[    2.900616] Hardware name: HVM domU
[    2.900618] sysfs: cannot create duplicate filename '/class/block/sda'
[    2.900619] Modules linked in: platform_pci acpiphp floppy 8139too 8139cp
[    2.900656] Pid: 10, comm: xenwatch Not tainted 2.6.37-7-virtual 
#19-Ubuntu
[    2.900659] Call Trace:
[    2.900668]  [<ffffffff8106623f>] warn_slowpath_common+0x7f/0xc0
[    2.900671]  [<ffffffff81066336>] warn_slowpath_fmt+0x46/0x50
[    2.900674]  [<ffffffff811d4608>] sysfs_add_one+0xd8/0x150
[    2.900677]  [<ffffffff811d4fbb>] sysfs_do_create_link+0x12b/0x210
[    2.900680]  [<ffffffff811d50b3>] sysfs_create_link+0x13/0x20
[    2.900717]  [<ffffffff813ab923>] device_add_class_symlinks+0xb3/0xd0
[    2.900721]  [<ffffffff813acd98>] device_add+0x248/0x410
[    2.900725]  [<ffffffff811cb2b1>] register_disk+0x41/0x170
[    2.900748]  [<ffffffff812cbf26>] add_disk+0xa6/0x160
[    2.900753]  [<ffffffff813c9a60>] blkfront_connect+0x120/0x1e0
[    2.900756]  [<ffffffff813c9b60>] blkback_changed+0x40/0x50
[    2.900761]  [<ffffffff8136d420>] otherend_changed+0xb0/0xc0
[    2.900763]  [<ffffffff8136b9b6>] xenwatch_thread+0xb6/0x170
[    2.900768]  [<ffffffff81087da0>] ? autoremove_wake_function+0x0/0x40
[    2.900771]  [<ffffffff8136b900>] ? xenwatch_thread+0x0/0x170
[    2.900773]  [<ffffffff81087676>] kthread+0x96/0xa0
[    2.900777]  [<ffffffff8100cee4>] kernel_thread_helper+0x4/0x10
[    2.900780]  [<ffffffff810875e0>] ? kthread+0x0/0xa0
[    2.900783]  [<ffffffff8100cee0>] ? kernel_thread_helper+0x0/0x10
[    2.900785] ---[ end trace d4f9aa3628762311 ]---
[    2.900811] ------------[ cut here ]------------
[    2.902999] kernel BUG at /build/buildd/linux-2.6.37/fs/sysfs/group.c:65!
[    2.906099] invalid opcode: 0000 [#1] SMP
[    2.908061] last sysfs file: /sys/devices/pci0000:00/0000:00:03.0/uevent
[    2.910028] CPU 0
[    2.910028] Modules linked in: platform_pci acpiphp floppy 8139too 8139cp
[    2.910028]
[    2.910028] Pid: 10, comm: xenwatch Tainted: G        W 
2.6.37-7-virtual #19-Ubuntu /HVM domU
[    2.910028] RIP: 0010:[<ffffffff811d64b7>]  [<ffffffff811d64b7>] 
internal_create_group+0x187/0x1a0
[    2.910028] RSP: 0018:ffff88001f9b3ce0  EFLAGS: 00010246
[    2.910028] RAX: 00000000ffffffef RBX: ffff88001c4f3800 RCX: 
ffff88001cf70900
[    2.910028] RDX: ffffffff81a1faa0 RSI: 0000000000000000 RDI: 
ffff88001c4f3870
[    2.910028] RBP: ffff88001f9b3d30 R08: 000000000000000d R09: 
000000000000000b
[    2.910028] R10: 0000000000000000 R11: 0000000000000000 R12: 
ffff88001c6368e0
[    2.910028] R13: ffff88001c4f3860 R14: ffffffff81a1faa0 R15: 
0000000000000000
[    2.910028] FS:  00007f1c2c9e4700(0000) GS:ffff88001fc00000(0000) 
knlGS:0000000000000000
[    2.910028] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[    2.910028] CR2: 00007fe87eb2c1d8 CR3: 000000001cd33000 CR4: 
00000000000006f0
[    2.910028] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[    2.910028] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[    2.910028] Process xenwatch (pid: 10, threadinfo ffff88001f9b2000, task 
ffff88001f9a5b00)
[    2.910028] Stack:
[    2.910028]  ffff88001f9b3d40 ffff88001c4f3870 000000303a323032 
ffff88001f9b3d50
[    2.910028]  ffff88001f9b3d10 ffff88001c4f3800 ffff88001c6368e0 
ffff88001c4f3860
[    2.910028]  ffff88001dfc1de0 ffff88001c4f3800 ffff88001f9b3d40 
ffffffff811d6503
[    2.910028] Call Trace:
[    2.910028]  [<ffffffff811d6503>] sysfs_create_group+0x13/0x20
[    2.910028]  [<ffffffff810f7174>] blk_trace_init_sysfs+0x14/0x20
[    2.910028]  [<ffffffff812c5fa0>] blk_register_queue+0x40/0x110
[    2.910028]  [<ffffffff812cbf2e>] add_disk+0xae/0x160
[    2.910028]  [<ffffffff813c9a60>] blkfront_connect+0x120/0x1e0
[    2.910028]  [<ffffffff813c9b60>] blkback_changed+0x40/0x50
[    2.910028]  [<ffffffff8136d420>] otherend_changed+0xb0/0xc0
[    2.910028]  [<ffffffff8136b9b6>] xenwatch_thread+0xb6/0x170
[    2.910028]  [<ffffffff81087da0>] ? autoremove_wake_function+0x0/0x40
[    2.910028]  [<ffffffff8136b900>] ? xenwatch_thread+0x0/0x170
[    2.910028]  [<ffffffff81087676>] kthread+0x96/0xa0
[    2.910028]  [<ffffffff8100cee4>] kernel_thread_helper+0x4/0x10
[    2.910028]  [<ffffffff810875e0>] ? kthread+0x0/0xa0
[    2.910028]  [<ffffffff8100cee0>] ? kernel_thread_helper+0x0/0x10
[    2.910028] Code: 8b 45 b0 74 15 48 8b 7d c8 89 45 b0 e8 93 e3 ff ff 4c 
8b 6d c8 8b 45 b0 eb 90 4c 8b 6d c8 eb 8a 48 83 7f 30 00 0f 85 be fe ff ff 
<0f> 0b be b7 00 00 00 48 c7 c7 58 79 7d 81 e8 b6 fd e8 ff e9 dc
[    2.910028] RIP  [<ffffffff811d64b7>] internal_create_group+0x187/0x1a0
[    2.910028]  RSP <ffff88001f9b3ce0>
[    2.928247] ---[ end trace d4f9aa3628762312 ]---
[    3.836521] eth0: link up, 100Mbps, full-duplex, lpa 0x05E1
[    4.142405] parport_pc 00:0a: reported by Plug and Play ACPI


-- 
Alex Bligh

Re: 2.6.37 PV on HVM issues

[Stefano Stabellini]

On Thu, 2 Dec 2010, Alex Bligh wrote:
> Stefano,
> 
> --On 2 December 2010 15:41:08 +0000 Stefano Stabellini 
> <stefano.stabellini@eu.citrix.com> wrote:
> >> [    0.000000] Hypervisor detected: Xen HVM
> >> [    0.000000] Xen Platform PCI: unrecognised magic value
> >> [    0.000000] Booting paravirtualized kernel on Xen
> >> [    0.800582] Initialising Xen virtual ethernet
> >> driver.
> >> [   13.980075] eth0: no IPv6 routers present
> >>
> >> The lines marked with asterisks are confusing to me as they appear to be
> >> in contradiction.
> >>
> >
> > They mean that Linux recognized that it is running on Xen but the
> > platform pci device is too old to support the device unplug protocol,
> > therefore pv drivers have been disabled.
> 
> OK. How modern a version of Xen am I meant to need to run 2.6.37?

You need xen 3.4 or later.


> This
> version of Xen supports non-inline pvdrivers fine.

They use hacks to disable the emulated interfaces that couldn't possibly
be upstreamed.


> 
> > You can try to enable them anyway at your own risk passing
> > xen_emul_unplug=unnecessary to the kernel command line options.
> 
> OK, so I get the following, which looks superficially nasty, but
> in fact suggests I need to unplug the disks. But the same happens
> with "xen_emul_unplug=unnecessary,all".
> 

You could try specifying "xvda" instead of hda in the VM config file,
and make sure the root= command line option points to /dev/xvda1 so that
there is no confusion with possible emulated paths.
Or you could always disable the IDE driver or blkfront in the kernel.

Re: 2.6.37 PV on HVM issues

[Alex Bligh]

Stefano,

>> OK. How modern a version of Xen am I meant to need to run 2.6.37?
>
> You need xen 3.4 or later.

Thanks, that's useful.

>> This
>> version of Xen supports non-inline pvdrivers fine.
>
> They use hacks to disable the emulated interfaces that couldn't possibly
> be upstreamed.

I don't need the emulated interfaces to be disabled. Both the emulated
interfaces and the PV-Ops ones appear and we use only one of them.
So, for instance, we see /dev/xvda and /dev/sda. We mount by UUID or label
and have technology that causes the emulated ones to be preferred (if
present).

>> > You can try to enable them anyway at your own risk passing
>> > xen_emul_unplug=unnecessary to the kernel command line options.
>>
>> OK, so I get the following, which looks superficially nasty, but
>> in fact suggests I need to unplug the disks. But the same happens
>> with "xen_emul_unplug=unnecessary,all".
>
> You could try specifying "xvda" instead of hda in the VM config file,

We're in HVM mode on 3.3.1. I'm not quite sure what you mean here.
The name "xvda" comes from blkfront.c (or at least it used to last
time I was playing around with modified_drivers and produced a standalone
patch into the new kernel). Looking at the current source, it seems
to still think it's called /dev/xvda. However, something in the
xen_watch thread is causing it to try and register as /dev/sda
(even though that exists). Where exactly is it picking this up from?
IE how is that name being passed through? Is there some new technology
that causes the device name in the guest to be called something in
the config file? We're using the same config file (well, system)
as with the unmodified_drivers based machines, and they come up
fine with /dev/xvdX.

> and make sure the root= command line option points to /dev/xvda1 so that
> there is no confusion with possible emulated paths.

That should make any difference as it just controls which root option
is mounted. We pass a label or UUID.

> Or you could always disable the IDE driver or blkfront in the kernel.

That would sort of defeat the object. I already have kernels that run
just fine (and support /dev/xvda and /dev/sda). What I am trying to
do is to get a stock kernel to run (specifically a stock Ubuntu
kernel).

-- 
Alex Bligh

Re: 2.6.37 PV on HVM issues

[Stefano Stabellini]

On Thu, 2 Dec 2010, Alex Bligh wrote:
> >> This
> >> version of Xen supports non-inline pvdrivers fine.
> >
> > They use hacks to disable the emulated interfaces that couldn't possibly
> > be upstreamed.
> 
> I don't need the emulated interfaces to be disabled. Both the emulated
> interfaces and the PV-Ops ones appear and we use only one of them.
> So, for instance, we see /dev/xvda and /dev/sda. We mount by UUID or label
> and have technology that causes the emulated ones to be preferred (if
> present).
> 

I guess you mean the paravirtualized ones to be preferred.


> >> > You can try to enable them anyway at your own risk passing
> >> > xen_emul_unplug=unnecessary to the kernel command line options.
> >>
> >> OK, so I get the following, which looks superficially nasty, but
> >> in fact suggests I need to unplug the disks. But the same happens
> >> with "xen_emul_unplug=unnecessary,all".
> >
> > You could try specifying "xvda" instead of hda in the VM config file,
> 
> We're in HVM mode on 3.3.1. I'm not quite sure what you mean here.
> The name "xvda" comes from blkfront.c (or at least it used to last
> time I was playing around with modified_drivers and produced a standalone
> patch into the new kernel). Looking at the current source, it seems
> to still think it's called /dev/xvda. However, something in the
> xen_watch thread is causing it to try and register as /dev/sda
> (even though that exists). Where exactly is it picking this up from?
> IE how is that name being passed through? Is there some new technology
> that causes the device name in the guest to be called something in
> the config file? We're using the same config file (well, system)
> as with the unmodified_drivers based machines, and they come up
> fine with /dev/xvdX.
> 

I was referring to the device name in the disk config file option,
usually is "hda" in HVM config files. Some blkfront development versions
don't always create an xvd device, so if you manually specify xvda in
the disk config file you would rule that problem out (even though it
also depends on the behaviour toolstack and I don't remember what xend 3.3
would do).

But in any case upstream blkfront should always create xvd devices so I am
not sure how you could end up with a /dev/sda there.


> > and make sure the root= command line option points to /dev/xvda1 so that
> > there is no confusion with possible emulated paths.
> 
> That should make any difference as it just controls which root option
> is mounted. We pass a label or UUID.
> 

If you pass a label or a UUID you cannot be sure if you end up using the
emulated or the paravitualized interface, unless xen supports the unplug
protocol.
Maybe the technology you mentioned before solves also this problem for
you.


> > Or you could always disable the IDE driver or blkfront in the kernel.
> 
> That would sort of defeat the object. I already have kernels that run
> just fine (and support /dev/xvda and /dev/sda). What I am trying to
> do is to get a stock kernel to run (specifically a stock Ubuntu
> kernel).
> 
 
If you'd like a stock kernel, the best thing to do is upgrade xen, then
you don't need any changes in the guest.
Keep in mind that xen 3.4 was released almost 2 years ago.
If you know what you are doing you can manually remove the check for the
unplug protocol in the kernel, just hack
arch/x86/xen/platform-pci-unplug.c:check_platform_magic.

Re: 2.6.37 PV on HVM issues

[Ian Campbell]

On Fri, 2010-12-03 at 11:18 +0000, Stefano Stabellini wrote:
> 
> If you know what you are doing you can manually remove the check for
> the unplug protocol in the kernel, just hack
> arch/x86/xen/platform-pci-unplug.c:check_platform_magic. 

This is exactly equivalent to adding "xen_emul_unplug=unnecessary" to
the kernel command line, isn't it?

Conversely if you want to always force emulated devices (i.e. disable
any attempt to unplug) then "xen_emul_unplug=never" will do what you
want.

Ian.

Re: 2.6.37 PV on HVM issues

[Stefano Stabellini]

On Fri, 3 Dec 2010, Ian Campbell wrote:
> On Fri, 2010-12-03 at 11:18 +0000, Stefano Stabellini wrote:
> > 
> > If you know what you are doing you can manually remove the check for
> > the unplug protocol in the kernel, just hack
> > arch/x86/xen/platform-pci-unplug.c:check_platform_magic. 
> 
> This is exactly equivalent to adding "xen_emul_unplug=unnecessary" to
> the kernel command line, isn't it?
> 

yeah, easier your way :) 

Re: 2.6.37 PV on HVM issues

[Alex Bligh]

Stefano,

> I guess you mean the paravirtualized ones to be preferred.

Ooops, yes :-)

> I was referring to the device name in the disk config file option,
> usually is "hda" in HVM config files. Some blkfront development versions
> don't always create an xvd device, so if you manually specify xvda in
> the disk config file you would rule that problem out (even though it
> also depends on the behaviour toolstack and I don't remember what xend 3.3
> would do).

Would this not break booting a non-PV driver equipped kernel? In the
general case, we don't know what guests will be booted (up to the
customer).

> But in any case upstream blkfront should always create xvd devices so I am
> not sure how you could end up with a /dev/sda there.

The /dev/sda that is there is the emulated devices. xen_watch (judging
by the call trace) is trying to create another /dev/sda, presumably
for the paravirtualised device (given the call path through blkfront).

> If you pass a label or a UUID you cannot be sure if you end up using the
> emulated or the paravitualized interface, unless xen supports the unplug
> protocol.
> Maybe the technology you mentioned before solves also this problem for
> you.

The technology is simply ensuring the modules in the kernel initialise
in the right order. This makes UUID and label mounting prefer PV drivers.

It's described here:
 http://blog.alex.org.uk/2010/05/09/linux-pv-drivers-for-xen-hvm-building-nor
mally-within-an-arbitrary-kernel-tree/
with patches.

I think our current method is to use a modular sd_mod and a builtin
blkfront etc., but it's possible to do it without that.

> If you'd like a stock kernel, the best thing to do is upgrade xen, then
> you don't need any changes in the guest.
> Keep in mind that xen 3.4 was released almost 2 years ago.
> If you know what you are doing you can manually remove the check for the
> unplug protocol in the kernel, just hack
> arch/x86/xen/platform-pci-unplug.c:check_platform_magic.

That's easier said than done on a running platform with large numbers
of nodes, a huge variety of operating systems etc.; we have Xen 4 working
in the lab, but the code change our end is very substantial,

My first priority is to ensure 2.6.37 doesn't break anything (that's
fine, because it comes up without PV drivers which is no worse than
2.6.36). My second priority is to try and allow end users to get
2.6.37+ working with PV drivers easily (so adding grub command line
options is doable).

-- 
Alex Bligh

Re: 2.6.37 PV on HVM issues

[Alex Bligh]

--On 3 December 2010 11:24:43 +0000 Ian Campbell <Ian.Campbell@citrix.com> 
wrote:

> On Fri, 2010-12-03 at 11:18 +0000, Stefano Stabellini wrote:
>>
>> If you know what you are doing you can manually remove the check for
>> the unplug protocol in the kernel, just hack
>> arch/x86/xen/platform-pci-unplug.c:check_platform_magic.
>
> This is exactly equivalent to adding "xen_emul_unplug=unnecessary" to
> the kernel command line, isn't it?
>
> Conversely if you want to always force emulated devices (i.e. disable
> any attempt to unplug) then "xen_emul_unplug=never" will do what you
> want.

OK, I am confused. What I want do is is bring the PV devices up as
/dev/xvda not /dev/sda. I don't much care whether or not it attempts to
unplug the emulated devices, as I can cope with or without the additional
presence of the emulated devices.

IE I want it to work just like it used to work with unmodified_drivers.

Is there a command line option for that? Or if not, I am happy
to try and work on a patch.

-- 
Alex Bligh

Re: 2.6.37 PV on HVM issues

[Ian Campbell]

On Fri, 2010-12-03 at 16:29 +0000, Alex Bligh wrote:
> 
> --On 3 December 2010 11:24:43 +0000 Ian Campbell <Ian.Campbell@citrix.com> 
> wrote:
> 
> > On Fri, 2010-12-03 at 11:18 +0000, Stefano Stabellini wrote:
> >>
> >> If you know what you are doing you can manually remove the check for
> >> the unplug protocol in the kernel, just hack
> >> arch/x86/xen/platform-pci-unplug.c:check_platform_magic.
> >
> > This is exactly equivalent to adding "xen_emul_unplug=unnecessary" to
> > the kernel command line, isn't it?
> >
> > Conversely if you want to always force emulated devices (i.e. disable
> > any attempt to unplug) then "xen_emul_unplug=never" will do what you
> > want.
> 
> OK, I am confused. What I want do is is bring the PV devices up as
> /dev/xvda not /dev/sda. I don't much care whether or not it attempts to
> unplug the emulated devices, as I can cope with or without the additional
> presence of the emulated devices.
> 
> IE I want it to work just like it used to work with unmodified_drivers.
> 
> Is there a command line option for that?

Yes, it is "xen_emul_unplug=unnecessary" which tells the driver that you
know what you are doing and that it is not necessary to unplug the
emulated devices before enabling the pv path because you as the admin
have ensured that the pv path is safe to use.

Without this option present the default is to err on the side of caution
and not enable the pv path unless the emulated path can be unplugged.

Ian.

> Or if not, I am happy
> to try and work on a patch.

Re: 2.6.37 PV on HVM issues

[Pasi Kärkkäinen]

On Fri, Dec 03, 2010 at 04:27:20PM +0000, Alex Bligh wrote:
> Stefano,
>
>> I guess you mean the paravirtualized ones to be preferred.
>
> Ooops, yes :-)
>
>> I was referring to the device name in the disk config file option,
>> usually is "hda" in HVM config files. Some blkfront development versions
>> don't always create an xvd device, so if you manually specify xvda in
>> the disk config file you would rule that problem out (even though it
>> also depends on the behaviour toolstack and I don't remember what xend 3.3
>> would do).
>
> Would this not break booting a non-PV driver equipped kernel? In the
> general case, we don't know what guests will be booted (up to the
> customer).
>
>> But in any case upstream blkfront should always create xvd devices so I am
>> not sure how you could end up with a /dev/sda there.
>
> The /dev/sda that is there is the emulated devices. xen_watch (judging
> by the call trace) is trying to create another /dev/sda, presumably
> for the paravirtualised device (given the call path through blkfront).
>
>> If you pass a label or a UUID you cannot be sure if you end up using the
>> emulated or the paravitualized interface, unless xen supports the unplug
>> protocol.
>> Maybe the technology you mentioned before solves also this problem for
>> you.
>
> The technology is simply ensuring the modules in the kernel initialise
> in the right order. This makes UUID and label mounting prefer PV drivers.
>
> It's described here:
> http://blog.alex.org.uk/2010/05/09/linux-pv-drivers-for-xen-hvm-building-nor
> mally-within-an-arbitrary-kernel-tree/
> with patches.
>

Btw I added a link to your blog to http://wiki.xen.org/xenwiki/XenLinuxPVonHVMdrivers

-- Pasi

Re: 2.6.37 PV on HVM issues

[Stefano Stabellini]

On Fri, 3 Dec 2010, Alex Bligh wrote:
> > I was referring to the device name in the disk config file option,
> > usually is "hda" in HVM config files. Some blkfront development versions
> > don't always create an xvd device, so if you manually specify xvda in
> > the disk config file you would rule that problem out (even though it
> > also depends on the behaviour toolstack and I don't remember what xend 3.3
> > would do).
> 
> Would this not break booting a non-PV driver equipped kernel? In the
> general case, we don't know what guests will be booted (up to the
> customer).
 
It wouldn't break anything because qemu emulates the IDE disks anyway,
but I am not recommending this in production, just for debugging this
particular issue.


> > But in any case upstream blkfront should always create xvd devices so I am
> > not sure how you could end up with a /dev/sda there.
> 
> The /dev/sda that is there is the emulated devices. xen_watch (judging
> by the call trace) is trying to create another /dev/sda, presumably
> for the paravirtualised device (given the call path through blkfront).
> 

This shouldn't happen. What kernel are you using? Can I see the sources?


> > If you'd like a stock kernel, the best thing to do is upgrade xen, then
> > you don't need any changes in the guest.
> > Keep in mind that xen 3.4 was released almost 2 years ago.
> > If you know what you are doing you can manually remove the check for the
> > unplug protocol in the kernel, just hack
> > arch/x86/xen/platform-pci-unplug.c:check_platform_magic.
> 
> That's easier said than done on a running platform with large numbers
> of nodes, a huge variety of operating systems etc.; we have Xen 4 working
> in the lab, but the code change our end is very substantial,
> 
> My first priority is to ensure 2.6.37 doesn't break anything (that's
> fine, because it comes up without PV drivers which is no worse than
> 2.6.36). My second priority is to try and allow end users to get
> 2.6.37+ working with PV drivers easily (so adding grub command line
> options is doable).
> 

Another option would be to backport the appended patch that implements
the unplug protocol in qemu to your qemu-xen tree:


commit e7911109f4321e9ba0cc56a253b653600aa46bea
Author: Ian Jackson <ian.jackson@eu.citrix.com>
Date:   Wed Dec 31 16:00:20 2008 +0000

    disable qemu PCI devices in HVM domains
    
    Magic ioport (0x10) protocol for negotating with guest PV drivers
    during startup, and allowing PV drivers to disable hardware emulations
    thus preventing guest from seeing the same device through two paths.
    
    Protocol and implementation from the Citrix Xenserver product line.
    Documentation (protocol spec) will follow in a moment.
    
    Contributed-By: Steven Smith <steven.smith@eu.citrix.com>
    Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

diff --git a/block-raw-posix.c b/block-raw-posix.c
index 044d1f4..f705c51 100644
--- a/block-raw-posix.c
+++ b/block-raw-posix.c
@@ -51,6 +51,7 @@
 #include <sys/ioctl.h>
 #include <linux/cdrom.h>
 #include <linux/fd.h>
+#include <sys/mount.h>
 #endif
 #ifdef __FreeBSD__
 #include <signal.h>
@@ -792,6 +793,10 @@ static void raw_close(BlockDriverState *bs)
 {
     BDRVRawState *s = bs->opaque;
     if (s->fd >= 0) {
+#ifndef CONFIG_STUBDOM
+        /* Invalidate buffer cache for this device. */
+        ioctl(s->fd, BLKFLSBUF, 0);
+#endif
         close(s->fd);
         s->fd = -1;
         if (s->aligned_buf != NULL)
diff --git a/hw/ide.c b/hw/ide.c
index bf97d69..da7057c 100644
--- a/hw/ide.c
+++ b/hw/ide.c
@@ -501,6 +501,7 @@ typedef struct PCIIDEState {
     int type; /* see IDE_TYPE_xxx */
 } PCIIDEState;
 
+static PCIIDEState *principal_ide_controller;
 
 #if defined(__ia64__)
 #include <xen/hvm/ioreq.h>
@@ -2906,6 +2907,47 @@ static void ide_reset(IDEState *s)
     s->media_changed = 0;
 }
 
+/* Unplug all of the IDE hard disks, starting at index @start in the
+   table. */
+static void _ide_unplug_harddisks(int start)
+{
+    IDEState *s;
+    int i, j;
+
+    if (!principal_ide_controller) {
+        fprintf(stderr, "No principal controller?\n");
+        return;
+    }
+    for (i = start; i < 4; i++) {
+        s = principal_ide_controller->ide_if + i;
+        if (!s->bs)
+            continue; /* drive not present */
+        if (s->is_cdrom)
+            continue; /* cdrom */
+        /* Is a hard disk, unplug it. */
+        for (j = 0; j < nb_drives; j++)
+            if (drives_table[j].bdrv == s->bs)
+                drives_table[j].bdrv = NULL;
+        bdrv_close(s->bs);
+        s->bs = NULL;
+        ide_reset(s);
+    }
+}
+
+/* Unplug all hard disks except for the primary master (which will
+   almost always be the boot device). */
+void ide_unplug_aux_harddisks(void)
+{
+    _ide_unplug_harddisks(1);
+}
+
+/* Unplug all hard disks, including the boot device. */
+void ide_unplug_harddisks(void)
+{
+    _ide_unplug_harddisks(0);
+}
+
+
 struct partition {
 	uint8_t boot_ind;		/* 0x80 - active */
 	uint8_t head;		/* starting head */
@@ -3423,6 +3465,9 @@ void pci_cmd646_ide_init(PCIBus *bus, BlockDriverState **hd_table,
                                            sizeof(PCIIDEState),
                                            -1,
                                            NULL, NULL);
+    if (principal_ide_controller)
+	abort();
+    principal_ide_controller = d;
     d->type = IDE_TYPE_CMD646;
     pci_conf = d->dev.config;
     pci_conf[0x00] = 0x95; // CMD646
@@ -3557,6 +3602,9 @@ void pci_piix3_ide_init(PCIBus *bus, BlockDriverState **hd_table, int devfn,
                                            sizeof(PCIIDEState),
                                            devfn,
                                            NULL, NULL);
+    if (principal_ide_controller)
+	abort();
+    principal_ide_controller = d;
     d->type = IDE_TYPE_PIIX3;
 
     pci_conf = d->dev.config;
diff --git a/hw/pc.h b/hw/pc.h
index 02b7018..455306d 100644
--- a/hw/pc.h
+++ b/hw/pc.h
@@ -150,6 +150,8 @@ void pci_piix3_ide_init(PCIBus *bus, BlockDriverState **hd_table, int devfn,
                         qemu_irq *pic);
 void pci_piix4_ide_init(PCIBus *bus, BlockDriverState **hd_table, int devfn,
                         qemu_irq *pic);
+void ide_unplug_harddisks(void);
+void ide_unplug_aux_harddisks(void);
 
 /* ne2000.c */
 
diff --git a/hw/pci.c b/hw/pci.c
index 2d4099b..e72c669 100644
--- a/hw/pci.c
+++ b/hw/pci.c
@@ -26,6 +26,9 @@
 #include "console.h"
 #include "net.h"
 
+#include "exec-all.h"
+#include "qemu-xen.h"
+
 //#define DEBUG_PCI
 
 struct PCIBus {
@@ -648,6 +651,46 @@ void pci_nic_init(PCIBus *bus, NICInfo *nd, int devfn)
     }
 }
 
+void pci_unplug_netifs(void)
+{
+    PCIBus *bus;
+    PCIDevice *dev;
+    PCIIORegion *region;
+    int x;
+    int i;
+
+    /* We only support one PCI bus */
+    for (bus = first_bus; bus; bus = NULL) {
+       for (x = 0; x < 256; x++) {
+           dev = bus->devices[x];
+           if (dev &&
+               dev->config[0xa] == 0 &&
+               dev->config[0xb] == 2) {
+               /* Found a netif.  Remove it from the bus.  Note that
+                  we don't free it here, since there could still be
+                  references to it floating around.  There are only
+                  ever one or two structures leaked, and it's not
+                  worth finding them all. */
+               bus->devices[x] = NULL;
+               for (i = 0; i < PCI_NUM_REGIONS; i++) {
+                   region = &dev->io_regions[i];
+                   if (region->addr == (uint32_t)-1 ||
+                       region->size == 0)
+                       continue;
+                   fprintf(logfile, "region type %d at [%x,%x).\n",
+                           region->type, region->addr,
+                           region->addr+region->size);
+                   if (region->type == PCI_ADDRESS_SPACE_IO) {
+                       isa_unassign_ioport(region->addr, region->size);
+                   } else if (region->type == PCI_ADDRESS_SPACE_MEM) {
+                       unregister_iomem(region->addr);
+                   }
+               }
+           }
+       }
+    }
+}
+
 typedef struct {
     PCIDevice dev;
     PCIBus *bus;
diff --git a/hw/xen_platform.c b/hw/xen_platform.c
index 937b802..84a2b19 100644
--- a/hw/xen_platform.c
+++ b/hw/xen_platform.c
@@ -24,13 +24,20 @@
  */
 
 #include "hw.h"
+#include "pc.h"
 #include "pci.h"
 #include "irq.h"
 #include "qemu-xen.h"
 
+#include <assert.h>
 #include <xenguest.h>
 
+static int drivers_blacklisted;
+static uint16_t driver_product_version;
+static int throttling_disabled;
 extern FILE *logfile;
+static char log_buffer[4096];
+static int log_buffer_off;
 
 #define PFFLAG_ROM_LOCK 1 /* Sets whether ROM memory area is RW or RO */
 
@@ -41,6 +48,88 @@ typedef struct PCIXenPlatformState
   uint64_t   vga_stolen_ram;
 } PCIXenPlatformState;
 
+/* We throttle access to dom0 syslog, to avoid DOS attacks.  This is
+   modelled as a token bucket, with one token for every byte of log.
+   The bucket size is 128KB (->1024 lines of 128 bytes each) and
+   refills at 256B/s.  It starts full.  The guest is blocked if no
+   tokens are available when it tries to generate a log message. */
+#define BUCKET_MAX_SIZE (128*1024)
+#define BUCKET_FILL_RATE 256
+
+static void throttle(unsigned count)
+{
+    static unsigned available;
+    static struct timespec last_refil;
+    static int started;
+    static int warned;
+
+    struct timespec waiting_for, now;
+    double delay;
+    struct timespec ts;
+
+    if (throttling_disabled)
+        return;
+
+    if (!started) {
+        clock_gettime(CLOCK_MONOTONIC, &last_refil);
+        available = BUCKET_MAX_SIZE;
+        started = 1;
+    }
+
+    if (count > BUCKET_MAX_SIZE) {
+        fprintf(logfile, "tried to get %d tokens, but bucket size is %d\n",
+                BUCKET_MAX_SIZE, count);
+        exit(1);
+    }
+
+    if (available < count) {
+        /* The bucket is empty.  Refil it */
+
+        /* When will it be full enough to handle this request? */
+        delay = (double)(count - available) / BUCKET_FILL_RATE;
+        waiting_for = last_refil;
+        waiting_for.tv_sec += delay;
+        waiting_for.tv_nsec += (delay - (int)delay) * 1e9;
+        if (waiting_for.tv_nsec >= 1000000000) {
+            waiting_for.tv_nsec -= 1000000000;
+            waiting_for.tv_sec++;
+        }
+
+        /* How long do we have to wait? (might be negative) */
+        clock_gettime(CLOCK_MONOTONIC, &now);
+        ts.tv_sec = waiting_for.tv_sec - now.tv_sec;
+        ts.tv_nsec = waiting_for.tv_nsec - now.tv_nsec;
+        if (ts.tv_nsec < 0) {
+            ts.tv_sec--;
+            ts.tv_nsec += 1000000000;
+        }
+
+        /* Wait for it. */
+        if (ts.tv_sec > 0 ||
+            (ts.tv_sec == 0 && ts.tv_nsec > 0)) {
+            if (!warned) {
+                fprintf(logfile, "throttling guest access to syslog");
+                warned = 1;
+            }
+            while (nanosleep(&ts, &ts) < 0 && errno == EINTR)
+                ;
+        }
+
+        /* Refil */
+        clock_gettime(CLOCK_MONOTONIC, &now);
+        delay = (now.tv_sec - last_refil.tv_sec) +
+            (now.tv_nsec - last_refil.tv_nsec) * 1.0e-9;
+        available += BUCKET_FILL_RATE * delay;
+        if (available > BUCKET_MAX_SIZE)
+            available = BUCKET_MAX_SIZE;
+        last_refil = now;
+    }
+
+    assert(available >= count);
+
+    available -= count;
+}
+
 static uint32_t xen_platform_ioport_readb(void *opaque, uint32_t addr)
 {
     PCIXenPlatformState *s = opaque;
@@ -68,6 +157,19 @@ static void xen_platform_ioport_writeb(void *opaque, uint32_t addr, uint32_t val
             d->platform_flags = val & PFFLAG_ROM_LOCK;
         break;
     }
+    case 8:
+        {
+            if (val == '\n' || log_buffer_off == sizeof(log_buffer) - 1) {
+                /* Flush buffer */
+                log_buffer[log_buffer_off] = 0;
+                throttle(log_buffer_off);
+                fprintf(logfile, "%s\n", log_buffer);
+                log_buffer_off = 0;
+                break;
+            }
+            log_buffer[log_buffer_off++] = val;
+        }
+        break;
     default:
         break;
     }
@@ -163,6 +265,113 @@ static void platform_mmio_map(PCIDevice *d, int region_num,
     cpu_register_physical_memory(addr, 0x1000000, mmio_io_addr);
 }
 
+#define UNPLUG_ALL_IDE_DISKS 1
+#define UNPLUG_ALL_NICS 2
+#define UNPLUG_AUX_IDE_DISKS 4
+
+static void platform_fixed_ioport_write2(void *opaque, uint32_t addr, uint32_t val)
+{
+    switch (addr - 0x10) {
+    case 0:
+        /* Unplug devices.  Value is a bitmask of which devices to
+           unplug, with bit 0 the IDE devices, bit 1 the network
+           devices, and bit 2 the non-primary-master IDE devices. */
+        if (val & UNPLUG_ALL_IDE_DISKS)
+            ide_unplug_harddisks();
+        if (val & UNPLUG_ALL_NICS) {
+            pci_unplug_netifs();
+            net_tap_shutdown_all();
+        }
+        if (val & UNPLUG_AUX_IDE_DISKS) {
+            ide_unplug_aux_harddisks();
+        }
+        break;
+    case 2:
+        switch (val) {
+        case 1:
+            fprintf(logfile, "Citrix Windows PV drivers loaded in guest\n");
+            break;
+        case 0:
+            fprintf(logfile, "Guest claimed to be running PV product 0?\n");
+            break;
+        default:
+            fprintf(logfile, "Unknown PV product %d loaded in guest\n", val);
+            break;
+        }
+        driver_product_version = val;
+        break;
+    }
+}
+
+static void platform_fixed_ioport_write4(void *opaque, uint32_t addr,
+                                         uint32_t val)
+{
+    switch (addr - 0x10) {
+    case 0:
+        /* PV driver version */
+        if (driver_product_version == 0) {
+            fprintf(logfile,
+                    "Drivers tried to set their version number (%d) before setting the product number?\n",
+                    val);
+            return;
+        }
+        fprintf(logfile, "PV driver build %d\n", val);
+        if (xenstore_pv_driver_build_blacklisted(driver_product_version,
+                                                 val)) {
+            fprintf(logfile, "Drivers are blacklisted!\n");
+            drivers_blacklisted = 1;
+        }
+        break;
+    }
+}
+
+
+static void platform_fixed_ioport_write1(void *opaque, uint32_t addr, uint32_t val)
+{
+    switch (addr - 0x10) {
+    case 2:
+        /* Send bytes to syslog */
+        if (val == '\n' || log_buffer_off == sizeof(log_buffer) - 1) {
+            /* Flush buffer */
+            log_buffer[log_buffer_off] = 0;
+            throttle(log_buffer_off);
+            fprintf(logfile, "%s\n", log_buffer);
+            log_buffer_off = 0;
+            break;
+        }
+        log_buffer[log_buffer_off++] = val;
+        break;
+    }
+}
+
+static uint32_t platform_fixed_ioport_read2(void *opaque, uint32_t addr)
+{
+    switch (addr - 0x10) {
+    case 0:
+        if (drivers_blacklisted) {
+            /* The drivers will recognise this magic number and refuse
+             * to do anything. */
+            return 0xd249;
+        } else {
+            /* Magic value so that you can identify the interface. */
+            return 0x49d2;
+        }
+    default:
+        return 0xffff;
+    }
+}
+
+static uint32_t platform_fixed_ioport_read1(void *opaque, uint32_t addr)
+{
+    switch (addr - 0x10) {
+    case 2:
+        /* Version number */
+        return 1;
+    default:
+        return 0xff;
+    }
+}
+
 struct pci_config_header {
     uint16_t vendor_id;
     uint16_t device_id;
@@ -224,6 +433,7 @@ void pci_xen_platform_init(PCIBus *bus)
 {
     PCIXenPlatformState *d;
     struct pci_config_header *pch;
+    struct stat stbuf;
 
     printf("Register xen platform.\n");
     d = (PCIXenPlatformState *)pci_register_device(
@@ -255,4 +465,13 @@ void pci_xen_platform_init(PCIBus *bus)
 
     register_savevm("platform", 0, 2, xen_pci_save, xen_pci_load, d);
     printf("Done register platform.\n");
+    register_ioport_write(0x10, 16, 4, platform_fixed_ioport_write4, NULL);
+    register_ioport_write(0x10, 16, 2, platform_fixed_ioport_write2, NULL);
+    register_ioport_write(0x10, 16, 1, platform_fixed_ioport_write1, NULL);
+    register_ioport_read(0x10, 16, 2, platform_fixed_ioport_read2, NULL);
+    register_ioport_read(0x10, 16, 1, platform_fixed_ioport_read1, NULL);
+
+    if (stat("/etc/disable-guest-log-throttle", &stbuf) == 0)
+        throttling_disabled = 1;
+
 }
diff --git a/i386-dm/exec-dm.c b/i386-dm/exec-dm.c
index 6dbd460..20ac93c 100644
--- a/i386-dm/exec-dm.c
+++ b/i386-dm/exec-dm.c
@@ -272,7 +272,7 @@ void cpu_abort(CPUState *env, const char *fmt, ...)
 
 /* XXX: Simple implementation. Fix later */
 #define MAX_MMIO 32
-struct mmio_space {
+static struct mmio_space {
         target_phys_addr_t start;
         unsigned long size;
         unsigned long io_index;
@@ -418,6 +418,17 @@ int iomem_index(target_phys_addr_t addr)
         return 0;
 }
 
+void unregister_iomem(target_phys_addr_t start)
+{
+    int index = iomem_index(start);
+    if (index) {
+        fprintf(logfile, "squash iomem [%lx, %lx).\n", mmio[index].start,
+                mmio[index].start + mmio[index].size);
+        mmio[index].start = mmio[index].size = 0;
+    }
+}
+
+
 #if defined(__i386__) || defined(__x86_64__)
 #define phys_ram_addr(x) (qemu_map_cache(x))
 #elif defined(__ia64__)
diff --git a/qemu-xen.h b/qemu-xen.h
index 8375b79..44467a3 100644
--- a/qemu-xen.h
+++ b/qemu-xen.h
@@ -26,8 +26,11 @@ void xen_vga_populate_vram(uint64_t vram_addr);
 void xen_vga_vram_map(uint64_t vram_addr, int copy);
 #endif
 
-
+void ide_unplug_harddisks(void);
+void net_tap_shutdown_all(void);
+void pci_unplug_netifs(void);
 void destroy_hvm_domain(void);
+void unregister_iomem(target_phys_addr_t start);
 
 #ifdef __ia64__
 static inline void xc_domain_shutdown_hook(int xc_handle, uint32_t domid)
@@ -88,6 +91,8 @@ char *xenstore_vm_read(int domid, const char *key, unsigned int *len);
 char *xenstore_device_model_read(int domid, char *key, unsigned int *len);
 char *xenstore_read_battery_data(int battery_status);
 int xenstore_refresh_battery_status(void);
+int xenstore_pv_driver_build_blacklisted(uint16_t product_number,
+                                         uint32_t build_nr);
 
 /* xenfbfront.c */
 int xenfb_pv_display_init(DisplayState *ds);
diff --git a/vl.c b/vl.c
index ac38c2a..ae9170d 100644
--- a/vl.c
+++ b/vl.c
@@ -275,6 +275,20 @@ uint8_t qemu_uuid[16];
 
 #include "xen-vl-extra.c"
 
+typedef struct IOHandlerRecord {
+    int fd;
+    IOCanRWHandler *fd_read_poll;
+    IOHandler *fd_read;
+    IOHandler *fd_write;
+    int deleted;
+    void *opaque;
+    /* temporary data */
+    struct pollfd *ufd;
+    struct IOHandlerRecord *next;
+} IOHandlerRecord;
+
+static IOHandlerRecord *first_io_handler;
+
 /***********************************************************/
 /* x86 ISA bus support */
 
@@ -4355,6 +4369,7 @@ void do_info_slirp(void)
 typedef struct TAPState {
     VLANClientState *vc;
     int fd;
+    struct TAPState *next;
     char down_script[1024];
     char script_arg[1024];
 } TAPState;
@@ -4392,6 +4407,34 @@ static void tap_send(void *opaque)
     }
 }
 
+static TAPState *head_net_tap;
+
+void net_tap_shutdown_all(void)
+{
+    struct IOHandlerRecord **pioh, *ioh;
+
+    while (head_net_tap) {
+       pioh = &first_io_handler;
+       for (;;) {
+           ioh = *pioh;
+           if (ioh == NULL)
+               break;
+           if (ioh->fd == head_net_tap->fd) {
+               *pioh = ioh->next;
+               qemu_free(ioh);
+               break;
+           }
+           pioh = &ioh->next;
+       }
+       if (!ioh)
+           fprintf(stderr,
+                   "warning: can't find iohandler for %d to close it properly.\n",
+                   head_net_tap->fd);
+       close(head_net_tap->fd);
+       head_net_tap = head_net_tap->next;
+    }
+}
+
 /* fd support */
 
 static TAPState *net_tap_fd_init(VLANState *vlan, int fd)
@@ -4403,6 +4446,8 @@ static TAPState *net_tap_fd_init(VLANState *vlan, int fd)
         return NULL;
     s->fd = fd;
     s->vc = qemu_new_vlan_client(vlan, tap_receive, NULL, s);
+    s->next = head_net_tap;
+    head_net_tap = s;
     qemu_set_fd_handler(s->fd, tap_send, NULL, s);
     snprintf(s->vc->info_str, sizeof(s->vc->info_str), "tap: fd=%d", fd);
     return s;
@@ -6133,20 +6178,6 @@ static void dumb_display_init(DisplayState *ds)
 
 #define MAX_IO_HANDLERS 64
 
-typedef struct IOHandlerRecord {
-    int fd;
-    IOCanRWHandler *fd_read_poll;
-    IOHandler *fd_read;
-    IOHandler *fd_write;
-    int deleted;
-    void *opaque;
-    /* temporary data */
-    struct pollfd *ufd;
-    struct IOHandlerRecord *next;
-} IOHandlerRecord;
-
-static IOHandlerRecord *first_io_handler;
-
 /* XXX: fd_read_poll should be suppressed, but an API change is
    necessary in the character devices to suppress fd_can_read(). */
 int qemu_set_fd_handler2(int fd,
diff --git a/xenstore.c b/xenstore.c
index 86e8b63..e57fd66 100644
--- a/xenstore.c
+++ b/xenstore.c
@@ -778,6 +778,34 @@ void xenstore_record_dm(char *subpath, char *state)
     free(path);
 }
 
+int
+xenstore_pv_driver_build_blacklisted(uint16_t product_nr,
+                                     uint32_t build_nr)
+{
+    char *buf = NULL;
+    char *tmp;
+    const char *product;
+
+    switch (product_nr) {
+    case 1:
+        product = "xensource-windows";
+        break;
+    default:
+        /* Don't know what product this is -> we can't blacklist
+         * it. */
+        return 0;
+    }
+    if (asprintf(&buf, "/mh/driver-blacklist/%s/%d", product, build_nr) < 0)
+        return 0;
+    tmp = xs_read(xsh, XBT_NULL, buf, NULL);
+    free(tmp);
+    free(buf);
+    if (tmp == NULL)
+        return 0;
+    else
+        return 1;
+}
+
 void xenstore_record_dm_state(char *state)
 {
     xenstore_record_dm("state", state);

Re: 2.6.37 PV on HVM issues

[Stefano Stabellini]

On Fri, 3 Dec 2010, Alex Bligh wrote:
> --On 3 December 2010 11:24:43 +0000 Ian Campbell <Ian.Campbell@citrix.com> 
> wrote:
> 
> > On Fri, 2010-12-03 at 11:18 +0000, Stefano Stabellini wrote:
> >>
> >> If you know what you are doing you can manually remove the check for
> >> the unplug protocol in the kernel, just hack
> >> arch/x86/xen/platform-pci-unplug.c:check_platform_magic.
> >
> > This is exactly equivalent to adding "xen_emul_unplug=unnecessary" to
> > the kernel command line, isn't it?
> >
> > Conversely if you want to always force emulated devices (i.e. disable
> > any attempt to unplug) then "xen_emul_unplug=never" will do what you
> > want.
> 
> OK, I am confused. What I want do is is bring the PV devices up as
> /dev/xvda not /dev/sda. I don't much care whether or not it attempts to
> unplug the emulated devices, as I can cope with or without the additional
> presence of the emulated devices.
> 

This is what it should happen on a vanilla upstream kernel if you pass
xen_emul_unplug=unnecessary to the kernel command line options.
Blkfront should never create devices other than xvd.

Re: 2.6.37 PV on HVM issues

[Alex Bligh]

Ian,

--On 3 December 2010 16:44:42 +0000 Ian Campbell 
<Ian.Campbell@eu.citrix.com> wrote:

> Yes, it is "xen_emul_unplug=unnecessary" which tells the driver that you
> know what you are doing and that it is not necessary to unplug the
> emulated devices before enabling the pv path because you as the admin
> have ensured that the pv path is safe to use.

Yes, that's what I thought.

However, that is what produces 'kernel bug' I posted earlier.

See:
  Kernel command line: BOOT_IMAGE=/boot/vmlinuz-2.6.37-7-virtual 
root=UUID=f7f04f87-04f3-488b-a6ae-014ce5d63850 ro 
xen_emul_unplug=unnecessary console=ttyS0

For some reason, there is a call path through xenwatch_thread, then
blkfront that causes it to try and register something (I presume
the PV driver) as /dev/sda (not /dev/xvda).

-- 
Alex Bligh



[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 2.6.37-7-virtual (buildd@yellow) (gcc version 
4.5.2 (Ubuntu/Linaro 4.5.1-10ubuntu3) ) #19-Ubuntu SMP Wed Dec 1 02:15:20 
UTC 2010 (Ubuntu 2.6.37-7.19-virtual 2.6.37-rc3)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-2.6.37-7-virtual 
root=UUID=f7f04f87-04f3-488b-a6ae-014ce5d63850 ro 
xen_emul_unplug=unnecessary console=ttyS0
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
--
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI 2.4 present.
[    0.000000] DMI: /HVM domU, BIOS 3.3.1 02/02/2009
[    0.000000] Hypervisor detected: Xen HVM
[    0.000000] Xen version 3.3.
[    0.000000] Xen Platform PCI: unrecognised magic value
[    0.000000] HVMOP_pagetable_dying not supported
[    0.000000] e820 update range: 0000000000000000 - 0000000000010000 
(usable) ==> (reserved)
[    0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 
(usable)
--
[    0.000000] kernel direct mapping tables up to 1fff7000 @ 
1fff4000-1fff7000
[    0.000000] RAMDISK: 1f348000 - 1f799000
[    0.000000] ACPI: RSDP 00000000000ea010 00024 (v02    Xen)
[    0.000000] ACPI: XSDT 000000001fff8b70 00034 (v01    Xen      HVM 
00000000 HVML 00000000)
[    0.000000] ACPI: FACP 000000001fff8a00 000F4 (v04    Xen      HVM 
00000000 HVML 00000000)
[    0.000000] ACPI: DSDT 000000001fff7840 01132 (v02    Xen      HVM 
00000000 INTL 20060707)
[    0.000000] ACPI: FACS 000000001fff7800 00040
[    0.000000] ACPI: APIC 000000001fff8b00 00068 (v02    Xen      HVM 
00000000 HVML 00000000)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] No NUMA configuration found
--
[    0.000000] PM: Registered nosave memory: 00000000000e0000 - 
0000000000100000
[    0.000000] Allocating PCI resources starting at 20000000 (gap: 
20000000:e0000000)
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:1 
nr_node_ids:1
[    0.000000] PERCPU: Embedded 28 pages/cpu @ffff88001fc00000 s83712 r8192 
d22784 u2097152
--
[    0.000000] Built 1 zonelists in Node order, mobility grouping on. Total 
pages: 129152
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: 
BOOT_IMAGE=/boot/vmlinuz-2.6.37-7-virtual 
root=UUID=f7f04f87-04f3-488b-a6ae-014ce5d63850 ro 
xen_emul_unplug=unnecessary console=ttyS0
[    0.000000] PID hash table entries: 2048 (order: 2, 16384 bytes)
[    0.000000] Checking aperture...
--
[    0.821839] Fixed MDIO Bus: probed
[    0.823360] PPP generic driver version 2.4.2
[    0.825496] Initialising Xen virtual ethernet driver.
[    0.827998] tun: Universal TUN/TAP device driver, 1.6
[    0.830247] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
--
[    2.791086] xen-platform-pci 0000:00:03.0: PCI INT A -> GSI 28 (level, 
low) -> IRQ 28
[    2.798695] Grant table initialized
[    2.889459] blkfront device/vbd/51712 num-ring-pages 1 nr_ents 32.
[    2.897163] blkfront: sda: barriers disabled
[    2.900483] ------------[ cut here ]------------
[    2.900608] WARNING: at /build/buildd/linux-2.6.37/fs/sysfs/dir.c:451 
sysfs_add_one+0xd8/0x150()
[    2.900616] Hardware name: HVM domU
[    2.900618] sysfs: cannot create duplicate filename '/class/block/sda'
[    2.900619] Modules linked in: platform_pci acpiphp floppy 8139too 8139cp
[    2.900656] Pid: 10, comm: xenwatch Not tainted 2.6.37-7-virtual 
#19-Ubuntu
[    2.900659] Call Trace:
[    2.900668]  [<ffffffff8106623f>] warn_slowpath_common+0x7f/0xc0
[    2.900671]  [<ffffffff81066336>] warn_slowpath_fmt+0x46/0x50
[    2.900674]  [<ffffffff811d4608>] sysfs_add_one+0xd8/0x150
[    2.900677]  [<ffffffff811d4fbb>] sysfs_do_create_link+0x12b/0x210
[    2.900680]  [<ffffffff811d50b3>] sysfs_create_link+0x13/0x20
[    2.900717]  [<ffffffff813ab923>] device_add_class_symlinks+0xb3/0xd0
[    2.900721]  [<ffffffff813acd98>] device_add+0x248/0x410
[    2.900725]  [<ffffffff811cb2b1>] register_disk+0x41/0x170
[    2.900748]  [<ffffffff812cbf26>] add_disk+0xa6/0x160
[    2.900753]  [<ffffffff813c9a60>] blkfront_connect+0x120/0x1e0
[    2.900756]  [<ffffffff813c9b60>] blkback_changed+0x40/0x50
[    2.900761]  [<ffffffff8136d420>] otherend_changed+0xb0/0xc0
[    2.900763]  [<ffffffff8136b9b6>] xenwatch_thread+0xb6/0x170
[    2.900768]  [<ffffffff81087da0>] ? autoremove_wake_function+0x0/0x40
[    2.900771]  [<ffffffff8136b900>] ? xenwatch_thread+0x0/0x170
[    2.900773]  [<ffffffff81087676>] kthread+0x96/0xa0
[    2.900777]  [<ffffffff8100cee4>] kernel_thread_helper+0x4/0x10
[    2.900780]  [<ffffffff810875e0>] ? kthread+0x0/0xa0
[    2.900783]  [<ffffffff8100cee0>] ? kernel_thread_helper+0x0/0x10
[    2.900785] ---[ end trace d4f9aa3628762311 ]---
[    2.900811] ------------[ cut here ]------------
[    2.902999] kernel BUG at /build/buildd/linux-2.6.37/fs/sysfs/group.c:65!
[    2.906099] invalid opcode: 0000 [#1] SMP
[    2.908061] last sysfs file: /sys/devices/pci0000:00/0000:00:03.0/uevent
[    2.910028] CPU 0
[    2.910028] Modules linked in: platform_pci acpiphp floppy 8139too 8139cp
[    2.910028]
[    2.910028] Pid: 10, comm: xenwatch Tainted: G        W 2.6.37-7-virtual 
#19-Ubuntu /HVM domU
[    2.910028] RIP: 0010:[<ffffffff811d64b7>]  [<ffffffff811d64b7>] 
internal_create_group+0x187/0x1a0
[    2.910028] RSP: 0018:ffff88001f9b3ce0  EFLAGS: 00010246
[    2.910028] RAX: 00000000ffffffef RBX: ffff88001c4f3800 RCX: 
ffff88001cf70900
[    2.910028] RDX: ffffffff81a1faa0 RSI: 0000000000000000 RDI: 
ffff88001c4f3870
[    2.910028] RBP: ffff88001f9b3d30 R08: 000000000000000d R09: 
000000000000000b
[    2.910028] R10: 0000000000000000 R11: 0000000000000000 R12: 
ffff88001c6368e0
[    2.910028] R13: ffff88001c4f3860 R14: ffffffff81a1faa0 R15: 
0000000000000000
[    2.910028] FS:  00007f1c2c9e4700(0000) GS:ffff88001fc00000(0000) 
knlGS:0000000000000000
[    2.910028] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[    2.910028] CR2: 00007fe87eb2c1d8 CR3: 000000001cd33000 CR4: 
00000000000006f0
[    2.910028] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[    2.910028] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[    2.910028] Process xenwatch (pid: 10, threadinfo ffff88001f9b2000, task 
ffff88001f9a5b00)
[    2.910028] Stack:
[    2.910028]  ffff88001f9b3d40 ffff88001c4f3870 000000303a323032 
ffff88001f9b3d50
[    2.910028]  ffff88001f9b3d10 ffff88001c4f3800 ffff88001c6368e0 
ffff88001c4f3860
[    2.910028]  ffff88001dfc1de0 ffff88001c4f3800 ffff88001f9b3d40 
ffffffff811d6503
[    2.910028] Call Trace:
[    2.910028]  [<ffffffff811d6503>] sysfs_create_group+0x13/0x20
[    2.910028]  [<ffffffff810f7174>] blk_trace_init_sysfs+0x14/0x20
[    2.910028]  [<ffffffff812c5fa0>] blk_register_queue+0x40/0x110
[    2.910028]  [<ffffffff812cbf2e>] add_disk+0xae/0x160
[    2.910028]  [<ffffffff813c9a60>] blkfront_connect+0x120/0x1e0
[    2.910028]  [<ffffffff813c9b60>] blkback_changed+0x40/0x50
[    2.910028]  [<ffffffff8136d420>] otherend_changed+0xb0/0xc0
[    2.910028]  [<ffffffff8136b9b6>] xenwatch_thread+0xb6/0x170
[    2.910028]  [<ffffffff81087da0>] ? autoremove_wake_function+0x0/0x40
[    2.910028]  [<ffffffff8136b900>] ? xenwatch_thread+0x0/0x170
[    2.910028]  [<ffffffff81087676>] kthread+0x96/0xa0
[    2.910028]  [<ffffffff8100cee4>] kernel_thread_helper+0x4/0x10
[    2.910028]  [<ffffffff810875e0>] ? kthread+0x0/0xa0
[    2.910028]  [<ffffffff8100cee0>] ? kernel_thread_helper+0x0/0x10
[    2.910028] Code: 8b 45 b0 74 15 48 8b 7d c8 89 45 b0 e8 93 e3 ff ff 4c 
8b 6d c8 8b 45 b0 eb 90 4c 8b 6d c8 eb 8a 48 83 7f 30 00 0f 85 be fe ff ff 
<0f> 0b be b7 00 00 00 48 c7 c7 58 79 7d 81 e8 b6 fd e8 ff e9 dc
[    2.910028] RIP  [<ffffffff811d64b7>] internal_create_group+0x187/0x1a0
[    2.910028]  RSP <ffff88001f9b3ce0>
[    2.928247] ---[ end trace d4f9aa3628762312 ]---
[    3.836521] eth0: link up, 100Mbps, full-duplex, lpa 0x05E1
[    4.142405] parport_pc 00:0a: reported by Plug and Play ACPI

Re: 2.6.37 PV on HVM issues

[Alex Bligh]

Stefano,

> This shouldn't happen. What kernel are you using? Can I see the sources?

ubuntu natty 2.6.37-7.19-virtual kernel as of yesterday.

Git repo at:
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-natty.git;a=commit;h=1645c7e0f14120357fbeba339abbe2540cd4e083

> Another option would be to backport the appended patch that implements
> the unplug protocol in qemu to your qemu-xen tree:

Oooh, thanks. That looks really doable. I'd quite like to find out
what the sda bizarreness is.

-- 
Alex Bligh

Re: 2.6.37 PV on HVM issues

[Alex Bligh]

--On 3 December 2010 17:21:13 +0000 Stefano Stabellini 
<stefano.stabellini@eu.citrix.com> wrote:

> This is what it should happen on a vanilla upstream kernel if you pass
> xen_emul_unplug=unnecessary to the kernel command line options.
> Blkfront should never create devices other than xvd.

That appears to not be what's happening. See the call trace. It's
complaining it can't create another device called "sda". No, I have
no explanation either.

If you want access to a VM or three, I can of course get you one.

-- 
Alex Bligh

Re: 2.6.37 PV on HVM issues

[Alex Bligh]

--On 3 December 2010 17:32:10 +0000 Alex Bligh <alex@alex.org.uk> wrote:

> That appears to not be what's happening. See the call trace. It's
> complaining it can't create another device called "sda". No, I have
> no explanation either.

Ah, I am guessing the problem is this patch:

http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-natty.git;a=commitdiff;h=38096c28f13d0c2dd08584ff834da6d81306c7b3

I'll go moan at the Ubuntu people.

-- 
Alex Bligh

Re: 2.6.37 PV on HVM issues

[Stefano Stabellini]

On Fri, 3 Dec 2010, Alex Bligh wrote:
> 
> 
> --On 3 December 2010 17:32:10 +0000 Alex Bligh <alex@alex.org.uk> wrote:
> 
> > That appears to not be what's happening. See the call trace. It's
> > complaining it can't create another device called "sda". No, I have
> > no explanation either.
> 
> Ah, I am guessing the problem is this patch:
> 
> http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-natty.git;a=commitdiff;h=38096c28f13d0c2dd08584ff834da6d81306c7b3
> 
> I'll go moan at the Ubuntu people.
 
well spotted!!

Re: 2.6.37 PV on HVM issues

[Alex Bligh]

--On 3 December 2010 17:48:26 +0000 Stefano Stabellini 
<stefano.stabellini@eu.citrix.com> wrote:

>> Ah, I am guessing the problem is this patch:
>>
>> http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-natty.git;a=commitdiff;h=38
>> 096c28f13d0c2dd08584ff834da6d81306c7b3
>>
>> I'll go moan at the Ubuntu people.
>
> well spotted!!

Bug filed:
 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/684875

-- 
Alex Bligh

Re: 2.6.37 PV on HVM issues

[Christian Tramnitz]

Alex Bligh wrote:
>
> Bug filed:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/684875
>

The bug has been marked as fixed and I can confirm that it now (using 
2.6.37-12-server) tries to create an xvd device (instead of sd) but I'm 
still unable to boot Natty PV-on-HVM (as HVM domain with just xvd and 
non-ioemu devices).
According to dmesg everything is fine:

DMI: /HVM domU, BIOS 4.0.1 09/28/2010
Hypervisor detected: Xen HVM
Xen version 4.0.
Xen Platform PCI: I/O protocol version 1
Netfront and the Xen platform PCI driver have been compiled for this 
kernel: unplug emulated NICs.
Blkfront and the Xen platform PCI driver have been compiled for this 
kernel: unplug emulated disks.
Booting paravirtualized kernel on Xen
Xen HVM callback vector for event delivery is enabled


But I cannot load xen-blkfront (manually creating the 202 device nodes 
does not help either):

register_blkdev: cannot get major 202 for xvd
xen_blk: can't get major 202 with name xvd
FATAL: Error inserting xen_blkfront 
(/lib/modules/2.6.37-12-server/kernel/drivers/block/xen-blkfront.ko): No 
such device

Does anyone have a running Natty domU with PV-on-HVM?


Best regards,
    Christian

Re: Re: 2.6.37 PV on HVM issues

[Stefano Stabellini]

On Sun, 9 Jan 2011, Christian Tramnitz wrote:
> Alex Bligh wrote:
> >
> > Bug filed:
> > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/684875
> >
> 
> The bug has been marked as fixed and I can confirm that it now (using 
> 2.6.37-12-server) tries to create an xvd device (instead of sd) but I'm 
> still unable to boot Natty PV-on-HVM (as HVM domain with just xvd and 
> non-ioemu devices).
> According to dmesg everything is fine:
> 
> DMI: /HVM domU, BIOS 4.0.1 09/28/2010
> Hypervisor detected: Xen HVM
> Xen version 4.0.
> Xen Platform PCI: I/O protocol version 1
> Netfront and the Xen platform PCI driver have been compiled for this 
> kernel: unplug emulated NICs.
> Blkfront and the Xen platform PCI driver have been compiled for this 
> kernel: unplug emulated disks.
> Booting paravirtualized kernel on Xen
> Xen HVM callback vector for event delivery is enabled
> 
> 
> But I cannot load xen-blkfront (manually creating the 202 device nodes 
> does not help either):
> 
> register_blkdev: cannot get major 202 for xvd
> xen_blk: can't get major 202 with name xvd
> FATAL: Error inserting xen_blkfront 
> (/lib/modules/2.6.37-12-server/kernel/drivers/block/xen-blkfront.ko): No 
> such device
> 
> Does anyone have a running Natty domU with PV-on-HVM?
> 

It looks like another block driver has already registered the same major
number 202 but unfortunately the error message in register_blkdev
doesn't tell you who is it.

Re: Re: 2.6.37 PV on HVM issues

[Ian Campbell]

On Mon, 2011-01-10 at 12:10 +0000, Stefano Stabellini wrote:
> It looks like another block driver has already registered the same major
> number 202 but unfortunately the error message in register_blkdev
> doesn't tell you who is it.

If you can get to a command prompt somehow (even the rescue shell in the
initrd) then /proc/devices will tell you who has registered it.

Ian.

> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Re: 2.6.37 PV on HVM issues

[Stefano Stabellini]

On Mon, 10 Jan 2011, Ian Campbell wrote:
> On Mon, 2011-01-10 at 12:10 +0000, Stefano Stabellini wrote:
> > It looks like another block driver has already registered the same major
> > number 202 but unfortunately the error message in register_blkdev
> > doesn't tell you who is it.
> 
> If you can get to a command prompt somehow (even the rescue shell in the
> initrd) then /proc/devices will tell you who has registered it.

I looked into this issue and it turns out that the problem is completely
different from what I originally guessed.
I think Ubuntu builds platform-pci as a module by default and
mkinitramfs doesn't include it in the initrd but it is required by
blkfront and netfront.

Could you please rebuild your initramfs making sure that platform-pci is
included (add platform-pci to /etc/initramfs-tools/modules) and try
again?

Re: Re: 2.6.37 PV on HVM issues

[Stefano Stabellini]

On Mon, 10 Jan 2011, Stefano Stabellini wrote:
> On Mon, 10 Jan 2011, Ian Campbell wrote:
> > On Mon, 2011-01-10 at 12:10 +0000, Stefano Stabellini wrote:
> > > It looks like another block driver has already registered the same major
> > > number 202 but unfortunately the error message in register_blkdev
> > > doesn't tell you who is it.
> > 
> > If you can get to a command prompt somehow (even the rescue shell in the
> > initrd) then /proc/devices will tell you who has registered it.
> 
> I looked into this issue and it turns out that the problem is completely
> different from what I originally guessed.
> I think Ubuntu builds platform-pci as a module by default and
> mkinitramfs doesn't include it in the initrd but it is required by
> blkfront and netfront.
> 
> Could you please rebuild your initramfs making sure that platform-pci is
> included (add platform-pci to /etc/initramfs-tools/modules) and try
> again?
> 

I finally got a Natty VM setup and running: it really seems that adding
platform_pci to /etc/initramfs-tools/modules and rebuilding the
initramfs is enough to solve the problem.

Could somebody be kind enough to fill the bug on launchpad?