hotplug virtual disk to a Linux guest virtual machine with udev

hotplug virtual disk to a Linux guest virtual machine with udev

[Shi Jin]

Hi there,

I am running KVM virtual machines and with libvirt, I am able to
attach volumes to the running VM instance.
However, it turned out that different latest Linux distributions have
different results on getting the volume live.

I've tried Fedora Core 12, Ubuntu 9.10 and CentOS 5.4. Only FC12 is
able to see the attached volume right away.
Ubuntu and CentOS need to be rebooted to see the newly attached volumes.

In my understanding, all three distributions are running udev now and
I think udev is what is responsible for hot plug devices.
Am I right?

I suspected this is caused by the udev kernel configurations.
Therefore, I obtained the FC12 kernel source and build them and
installed the newly built kernel onto my Ubuntu 9.10 and CentOS 5.4
VMs and viola, they both work with live attached volumes seen
immediately, no need for reboot.

So it is confirmed it is the kernel that matters here, instead of user
space software. However, I googled around and found that to enable
udev, we need the following kernel options:

21       CONFIG_HOTPLUG=y

22       CONFIG_UEVENT_HELPER_PATH=""
23       CONFIG_NET=y
24       CONFIG_UNIX=y
25       CONFIG_SYSFS=y
26       CONFIG_SYSFS_DEPRECATED*=n
27       CONFIG_PROC_FS=y
28       CONFIG_TMPFS=y

29       CONFIG_INOTIFY_USER=y
30       CONFIG_SIGNALFD=y

Interestingly, all 3 original kernels for the 3 distributions have the
above kernel options enabled.
Apparently, they are not the deciding factors for the volume hot-plug
for work or fail.

I am sending this email to ask what other kernel options I have to pay
attention to.
In short, if I am going to change the  Ubuntu or CentOS kernels to
support volume hot plug, where do I change?

Thank you very much.

Shi

Re: hotplug virtual disk to a Linux guest virtual machine with udev

[Kay Sievers]

On Wed, Apr 14, 2010 at 21:10, Shi Jin <jinzishuai@gmail.com> wrote:
> I am running KVM virtual machines and with libvirt, I am able to
> attach volumes to the running VM instance.
> However, it turned out that different latest Linux distributions have
> different results on getting the volume live.
>
> I've tried Fedora Core 12, Ubuntu 9.10 and CentOS 5.4. Only FC12 is
> able to see the attached volume right away.
> Ubuntu and CentOS need to be rebooted to see the newly attached volumes.
>
> In my understanding, all three distributions are running udev now and
> I think udev is what is responsible for hot plug devices.
> Am I right?
>
> I suspected this is caused by the udev kernel configurations.
> Therefore, I obtained the FC12 kernel source and build them and
> installed the newly built kernel onto my Ubuntu 9.10 and CentOS 5.4
> VMs and viola, they both work with live attached volumes seen
> immediately, no need for reboot.
>
> So it is confirmed it is the kernel that matters here, instead of user
> space software. However, I googled around and found that to enable
> udev, we need the following kernel options:

Check "dmsg" if the kernel sees the new device. If it does, it's
probably a pure userspace issue.

Kay

Re: hotplug virtual disk to a Linux guest virtual machine with udev

[Shi Jin]

>
> Check "dmsg" if the kernel sees the new device. If it does, it's
> probably a pure userspace issue.
>
> Kay
>
Thank Kay. Yes, I did check dmesg. In fact, this is how I know it when
hot plug works.
On ubuntu 9.10 and centos 5.4, the dmesg shows nothing while in FC12,
it shows messages like
[265565.693825] pci 0000:00:06.0: reg 10 io port: [0x00-0x3f]
[265565.694573] pci 0000:00:02.0: BAR 6: bogus alignment [0x0-0x0] flags 0x2
[265565.694640] decode_hpp: Could not get hotplug parameters. Use defaults
[265565.701747] virtio-pci 0000:00:06.0: enabling device (0000 -> 0001)
[265565.702059] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11
[265565.702077] virtio-pci 0000:00:06.0: PCI INT A -> Link[LNKB] ->
GSI 11 (level, high) -> IRQ 11
[265565.702634]  vdb: vdb1 vdb2


Thanks.


-- 
Shi Jin, Ph.D.

Re: hotplug virtual disk to a Linux guest virtual machine with udev

[Kay Sievers]

On Wed, Apr 14, 2010 at 23:55, Shi Jin <jinzishuai@gmail.com> wrote:
> Thank Kay. Yes, I did check dmesg. In fact, this is how I know it when
> hot plug works.
> On ubuntu 9.10 and centos 5.4, the dmesg shows nothing while in FC12,
> it shows messages like
> [265565.693825] pci 0000:00:06.0: reg 10 io port: [0x00-0x3f]
> [265565.694573] pci 0000:00:02.0: BAR 6: bogus alignment [0x0-0x0] flags 0x2
> [265565.694640] decode_hpp: Could not get hotplug parameters. Use defaults
> [265565.701747] virtio-pci 0000:00:06.0: enabling device (0000 -> 0001)
> [265565.702059] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11
> [265565.702077] virtio-pci 0000:00:06.0: PCI INT A -> Link[LNKB] ->
> GSI 11 (level, high) -> IRQ 11
> [265565.702634]  vdb: vdb1 vdb2

You can run:
  udevadm monitor
add a volume to the host and check if you get events for new block
devices. If you do, it's an issue with userspace in the system.

Kay

Re: hotplug virtual disk to a Linux guest virtual machine with udev

[Shi Jin]

> You can run:
>  udevadm monitor
> add a volume to the host and check if you get events for new block
> devices. If you do, it's an issue with userspace in the system.
>
> Kay
>

Thank you Kay.
This is useful tool but it again confirmed my conjecture of this being
an kernel issue.
On FC12 where it is working, I do the a add and remove messages.
But on Ubuntu 9.10 and CentOS 5.4, I don't see any message.

So it is a kernel problem. Anything I can do?
Thanks.

Re: hotplug virtual disk to a Linux guest virtual machine with udev

[Kay Sievers]

On Thu, Apr 15, 2010 at 16:37, Shi Jin <jinzishuai@gmail.com> wrote:
>> You can run:
>>  udevadm monitor
>> add a volume to the host and check if you get events for new block
>> devices. If you do, it's an issue with userspace in the system.

> Thank you Kay.
> This is useful tool but it again confirmed my conjecture of this being
> an kernel issue.
> On FC12 where it is working, I do the a add and remove messages.
> But on Ubuntu 9.10 and CentOS 5.4, I don't see any message.
>
> So it is a kernel problem. Anything I can do?

Yeah, looks like some kernel hotplug driver issue. Probably related to
the virtio driver or something related to that infrastructure. There
is not much on the generic part which plugs in udev which can be
configured in the kernel.

Kay

Re: hotplug virtual disk to a Linux guest virtual machine with udev

[Jim Paris]

Shi Jin wrote:
> > You can run:
> >  udevadm monitor
> > add a volume to the host and check if you get events for new block
> > devices. If you do, it's an issue with userspace in the system.
> >
> > Kay
> >
> 
> Thank you Kay.
> This is useful tool but it again confirmed my conjecture of this being
> an kernel issue.
> On FC12 where it is working, I do the a add and remove messages.
> But on Ubuntu 9.10 and CentOS 5.4, I don't see any message.
> 
> So it is a kernel problem. Anything I can do?

Check that PCI hotplug is enabled in your guest kernels?

-jim

> Thanks.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-hotplug" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: hotplug virtual disk to a Linux guest virtual machine with udev

[Shi Jin]

> Check that PCI hotplug is enabled in your guest kernels?
>
> -jim
>
Thanks Jim.
The things I think import are all checked. Here is what the
config-2.6.28-14-generic for the ubuntu 9.10 kernel
[cloudadmin@node2 ubuntu9.04]$ grep UEVENT config-2.6.28-14-generic
CONFIG_DM_UEVENT=y
CONFIG_UEVENT_HELPER_PATH=""
[cloudadmin@node2 ubuntu9.04]$ grep SYSFS config-2.6.28-14-generic
CONFIG_ACPI_SYSFS_POWER=y
CONFIG_GPIO_SYSFS=y
CONFIG_RTC_INTF_SYSFS=y
CONFIG_SYSFS=y
# CONFIG_SYSFS_DEPRECATED_V2 is not set
CONFIG_VIDEO_PVRUSB2_SYSFS=y
CONFIG_WIRELESS_EXT_SYSFS=y
[cloudadmin@node2 ubuntu9.04]$ grep HOTPLUG_PCI config-2.6.28-14-generic
CONFIG_HOTPLUG_PCI=y
CONFIG_HOTPLUG_PCI_ACPI=m
CONFIG_HOTPLUG_PCI_ACPI_IBM=m
CONFIG_HOTPLUG_PCI_COMPAQ=m
CONFIG_HOTPLUG_PCI_COMPAQ_NVRAM=y
CONFIG_HOTPLUG_PCI_CPCI=y
CONFIG_HOTPLUG_PCI_CPCI_GENERIC=m
CONFIG_HOTPLUG_PCI_CPCI_ZT5550=m
CONFIG_HOTPLUG_PCI_FAKE=m
CONFIG_HOTPLUG_PCI_IBM=m
CONFIG_HOTPLUG_PCI_PCIE=y
CONFIG_HOTPLUG_PCI_SHPC=m
[cloudadmin@node2 ubuntu9.04]$ grep HOTPLUG= config-2.6.28-14-generic
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_HOTPLUG=y


Do you see anything that should have been there in order for the udev to work?

Thank you very much.

Shi

Re: hotplug virtual disk to a Linux guest virtual machine with udev

[Jim Paris]

Shi Jin wrote:
> [cloudadmin@node2 ubuntu9.04]$ grep HOTPLUG_PCI config-2.6.28-14-generic
> CONFIG_HOTPLUG_PCI=y
> CONFIG_HOTPLUG_PCI_ACPI=m
> CONFIG_HOTPLUG_PCI_ACPI_IBM=m
> CONFIG_HOTPLUG_PCI_COMPAQ=m
> CONFIG_HOTPLUG_PCI_COMPAQ_NVRAM=y
> CONFIG_HOTPLUG_PCI_CPCI=y
> CONFIG_HOTPLUG_PCI_CPCI_GENERIC=m
> CONFIG_HOTPLUG_PCI_CPCI_ZT5550=m
> CONFIG_HOTPLUG_PCI_FAKE=m
> CONFIG_HOTPLUG_PCI_IBM=m
> CONFIG_HOTPLUG_PCI_PCIE=y
> CONFIG_HOTPLUG_PCI_SHPC=m
..
> Do you see anything that should have been there in order for the udev to work?

Make sure those things built as modules are actually loaded (probably
acpi_pcihp for the kvm case).  Other than that, you have a working
kernel config and a non-working kernel config, so by changing them one
step at a time to be more in sync (including loaded module lists), you
should eventually be able to track down the difference...

-jim

Re: hotplug virtual disk to a Linux guest virtual machine with udev

[Shi Jin]

> Make sure those things built as modules are actually loaded (probably
> acpi_pcihp for the kvm case).  Other than that, you have a working
> kernel config and a non-working kernel config, so by changing them one
> step at a time to be more in sync (including loaded module lists), you
> should eventually be able to track down the difference...
>
> -jim
>

Thank you very much Jim. You are right on.
If I simply do a "modprobe acpiphp" before plugging the volumes, it works.

Thanks everyone.

-- 
Shi Jin, Ph.D.