KVM git hangs with if=virtio (works under kvm 0.12.3)

KVM git hangs with if=virtio (works under kvm 0.12.3)

[ewheeler]

Hello all,

I'm booting a CentOS kernel under today's KVM git and it hangs after
initializing the serial port when the drive if=virtio, but not when
drive if=ide.  Look close---this is not a "forgot to add virtio_blk"
problem.  If I use 0.12.3 from Ubuntu 10.04 it works properly.

Reproduction:

Using kvm 0.12.3 on ubuntu 10.04 (1:84+dfsg-0ubuntu16+0.12.3+noroms
+0ubuntu9) it will work properly:

  qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
 			-kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus 

As expected, the kernel panics unable to mount root (good-boot.png).
This makes sense, as "dummy-disk-image" is 1MB of 0x00 bytes.
  
---However---if I use today's git (2010-07-01) of kvm:

   /usr/local/kvm-git/bin/qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
   	-kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus    

This hangs just after initializing the Serial device (obtained by adding
-serial stdio -append console=ttyS0):

Note that this only happens with the disk interface set to virtio
(if=virtio).  It works fine for ide (if=ide).


Am I doing something wrong here?  
Is anyone else having this problem?

The pictures, disk and kernel image is available here: 
  http://www.portlandlinuxsupport.com/src/kvm-20100701-regression/

-Eric


More detail:

	==== snip ====   (see also bad-boot.png)
	PCI: PIIX3: Enabling Passive Release on 0000:00:01.0
	Activating ISA DMA hang workarounds.
	pci_hotplug: PCI Hot Plug PCI Core version: 0.5
	Real Time Clock Driver v1.12ac
	Non-volatile memory driver v1.2
	Linux agpgart interface v0.101 (c) Dave Jones
	Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
	  [[ hangs here and spins at >100% cpu ]]
	==== snip ====   

And the qemu-system-x86_64 process spins at >100% cpu:

== snip from top ==
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                              
 8895 root      20   0  287m  28m 3124 S  102  0.5   0:56.82 qemu-system-x86                                                      
== snip from top ==

Re: KVM git hangs with if=virtio (works under kvm 0.12.3)

[Stefan Hajnoczi]

On Fri, Jul 2, 2010 at 4:31 AM, ewheeler <kvm@ew.ewheeler.org> wrote:
> Hello all,
>
> I'm booting a CentOS kernel under today's KVM git and it hangs after
> initializing the serial port when the drive if=virtio, but not when
> drive if=ide.  Look close---this is not a "forgot to add virtio_blk"
> problem.  If I use 0.12.3 from Ubuntu 10.04 it works properly.
>
> Reproduction:
>
> Using kvm 0.12.3 on ubuntu 10.04 (1:84+dfsg-0ubuntu16+0.12.3+noroms
> +0ubuntu9) it will work properly:
>
>  qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
>                        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
>
> As expected, the kernel panics unable to mount root (good-boot.png).
> This makes sense, as "dummy-disk-image" is 1MB of 0x00 bytes.
>
> ---However---if I use today's git (2010-07-01) of kvm:
>
>   /usr/local/kvm-git/bin/qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
>        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
>
> This hangs just after initializing the Serial device (obtained by adding
> -serial stdio -append console=ttyS0):
>
> Note that this only happens with the disk interface set to virtio
> (if=virtio).  It works fine for ide (if=ide).
>
>
> Am I doing something wrong here?
> Is anyone else having this problem?

I have seen this issue with a RHEL 5.5 guest running under
qemu-kvm.git.  It boots a new guest fine but hangs as you described
with the RHEL 5.5 kernel.  I have not investigated.

Stefan

Re: KVM git hangs with if=virtio (works under kvm 0.12.3)

[Stefan Hajnoczi]

On Fri, Jul 2, 2010 at 9:07 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Fri, Jul 2, 2010 at 4:31 AM, ewheeler <kvm@ew.ewheeler.org> wrote:
>> Hello all,
>>
>> I'm booting a CentOS kernel under today's KVM git and it hangs after
>> initializing the serial port when the drive if=virtio, but not when
>> drive if=ide.  Look close---this is not a "forgot to add virtio_blk"
>> problem.  If I use 0.12.3 from Ubuntu 10.04 it works properly.
>>
>> Reproduction:
>>
>> Using kvm 0.12.3 on ubuntu 10.04 (1:84+dfsg-0ubuntu16+0.12.3+noroms
>> +0ubuntu9) it will work properly:
>>
>>  qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
>>                        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
>>
>> As expected, the kernel panics unable to mount root (good-boot.png).
>> This makes sense, as "dummy-disk-image" is 1MB of 0x00 bytes.
>>
>> ---However---if I use today's git (2010-07-01) of kvm:
>>
>>   /usr/local/kvm-git/bin/qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
>>        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
>>
>> This hangs just after initializing the Serial device (obtained by adding
>> -serial stdio -append console=ttyS0):
>>
>> Note that this only happens with the disk interface set to virtio
>> (if=virtio).  It works fine for ide (if=ide).
>>
>>
>> Am I doing something wrong here?
>> Is anyone else having this problem?
>
> I have seen this issue with a RHEL 5.5 guest running under
> qemu-kvm.git.  It boots a new guest fine but hangs as you described
> with the RHEL 5.5 kernel.  I have not investigated.

This issue is affected by extboot, a feature that enables booting from
virtio-blk devices.  I have just sent a patch to the KVM mailing list
to restore extboot functionality which has been broken in
qemu-kvm.git.  That patch can be used to work around this issue by
using "-drive ...,boot=on" but it doesn't explain why the RHEL 5.5
kernel hangs during serial initialization when extboot is not present.

Stefan

Re: KVM git hangs with if=virtio (works under kvm 0.12.3)

[Gleb Natapov]

On Mon, Jul 05, 2010 at 01:11:25PM +0100, Stefan Hajnoczi wrote:
> On Fri, Jul 2, 2010 at 9:07 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > On Fri, Jul 2, 2010 at 4:31 AM, ewheeler <kvm@ew.ewheeler.org> wrote:
> >> Hello all,
> >>
> >> I'm booting a CentOS kernel under today's KVM git and it hangs after
> >> initializing the serial port when the drive if=virtio, but not when
> >> drive if=ide.  Look close---this is not a "forgot to add virtio_blk"
> >> problem.  If I use 0.12.3 from Ubuntu 10.04 it works properly.
> >>
> >> Reproduction:
> >>
> >> Using kvm 0.12.3 on ubuntu 10.04 (1:84+dfsg-0ubuntu16+0.12.3+noroms
> >> +0ubuntu9) it will work properly:
> >>
> >>  qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
> >>                        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
> >>
> >> As expected, the kernel panics unable to mount root (good-boot.png).
> >> This makes sense, as "dummy-disk-image" is 1MB of 0x00 bytes.
> >>
> >> ---However---if I use today's git (2010-07-01) of kvm:
> >>
> >>   /usr/local/kvm-git/bin/qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
> >>        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
> >>
> >> This hangs just after initializing the Serial device (obtained by adding
> >> -serial stdio -append console=ttyS0):
> >>
> >> Note that this only happens with the disk interface set to virtio
> >> (if=virtio).  It works fine for ide (if=ide).
> >>
> >>
> >> Am I doing something wrong here?
> >> Is anyone else having this problem?
> >
> > I have seen this issue with a RHEL 5.5 guest running under
> > qemu-kvm.git.  It boots a new guest fine but hangs as you described
> > with the RHEL 5.5 kernel.  I have not investigated.
> 
> This issue is affected by extboot, a feature that enables booting from
> virtio-blk devices.  I have just sent a patch to the KVM mailing list
> to restore extboot functionality which has been broken in
> qemu-kvm.git.  That patch can be used to work around this issue by
> using "-drive ...,boot=on" but it doesn't explain why the RHEL 5.5
> kernel hangs during serial initialization when extboot is not present.
> 
Hang that happens during guest boot (after bootloader started the
kernel) cannot be worked around by extboot. extboot is also not needed
with latest qemu git to boot from virtio disks since the support for
that is in the bios now.
 
--
			Gleb.

Re: KVM git hangs with if=virtio (works under kvm 0.12.3)

[Stefan Hajnoczi]

2010/7/5 Gleb Natapov <gleb@redhat.com>:
> On Mon, Jul 05, 2010 at 01:11:25PM +0100, Stefan Hajnoczi wrote:
>> On Fri, Jul 2, 2010 at 9:07 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> > On Fri, Jul 2, 2010 at 4:31 AM, ewheeler <kvm@ew.ewheeler.org> wrote:
>> >> Hello all,
>> >>
>> >> I'm booting a CentOS kernel under today's KVM git and it hangs after
>> >> initializing the serial port when the drive if=virtio, but not when
>> >> drive if=ide.  Look close---this is not a "forgot to add virtio_blk"
>> >> problem.  If I use 0.12.3 from Ubuntu 10.04 it works properly.
>> >>
>> >> Reproduction:
>> >>
>> >> Using kvm 0.12.3 on ubuntu 10.04 (1:84+dfsg-0ubuntu16+0.12.3+noroms
>> >> +0ubuntu9) it will work properly:
>> >>
>> >>  qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
>> >>                        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
>> >>
>> >> As expected, the kernel panics unable to mount root (good-boot.png).
>> >> This makes sense, as "dummy-disk-image" is 1MB of 0x00 bytes.
>> >>
>> >> ---However---if I use today's git (2010-07-01) of kvm:
>> >>
>> >>   /usr/local/kvm-git/bin/qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
>> >>        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
>> >>
>> >> This hangs just after initializing the Serial device (obtained by adding
>> >> -serial stdio -append console=ttyS0):
>> >>
>> >> Note that this only happens with the disk interface set to virtio
>> >> (if=virtio).  It works fine for ide (if=ide).
>> >>
>> >>
>> >> Am I doing something wrong here?
>> >> Is anyone else having this problem?
>> >
>> > I have seen this issue with a RHEL 5.5 guest running under
>> > qemu-kvm.git.  It boots a new guest fine but hangs as you described
>> > with the RHEL 5.5 kernel.  I have not investigated.
>>
>> This issue is affected by extboot, a feature that enables booting from
>> virtio-blk devices.  I have just sent a patch to the KVM mailing list
>> to restore extboot functionality which has been broken in
>> qemu-kvm.git.  That patch can be used to work around this issue by
>> using "-drive ...,boot=on" but it doesn't explain why the RHEL 5.5
>> kernel hangs during serial initialization when extboot is not present.
>>
> Hang that happens during guest boot (after bootloader started the
> kernel) cannot be worked around by extboot. extboot is also not needed
> with latest qemu git to boot from virtio disks since the support for
> that is in the bios now.

I agree that something else is going on here and needs to be
investigated, but I do think that extboot can indirectly affect the
guest boot.

With extboot the virtio-blk PCI adapter is not touched by the
firmware/bootloader.  Is it possible that a virtio-blk interrupt is
raised and not acknowledged before entering Linux.  When Linux brings
up the serial port it gets swamped with interrupts?  That's just a
guess.

Newer (non-RHEL 5.5) kernels boot fine without extboot, so another
path to solving this bug is to bisect guest kernels.

Stefan

Re: KVM git hangs with if=virtio (works under kvm 0.12.3)

[Gleb Natapov]

On Mon, Jul 05, 2010 at 01:36:08PM +0100, Stefan Hajnoczi wrote:
> 2010/7/5 Gleb Natapov <gleb@redhat.com>:
> > On Mon, Jul 05, 2010 at 01:11:25PM +0100, Stefan Hajnoczi wrote:
> >> On Fri, Jul 2, 2010 at 9:07 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >> > On Fri, Jul 2, 2010 at 4:31 AM, ewheeler <kvm@ew.ewheeler.org> wrote:
> >> >> Hello all,
> >> >>
> >> >> I'm booting a CentOS kernel under today's KVM git and it hangs after
> >> >> initializing the serial port when the drive if=virtio, but not when
> >> >> drive if=ide.  Look close---this is not a "forgot to add virtio_blk"
> >> >> problem.  If I use 0.12.3 from Ubuntu 10.04 it works properly.
> >> >>
> >> >> Reproduction:
> >> >>
> >> >> Using kvm 0.12.3 on ubuntu 10.04 (1:84+dfsg-0ubuntu16+0.12.3+noroms
> >> >> +0ubuntu9) it will work properly:
> >> >>
> >> >>  qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
> >> >>                        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
> >> >>
> >> >> As expected, the kernel panics unable to mount root (good-boot.png).
> >> >> This makes sense, as "dummy-disk-image" is 1MB of 0x00 bytes.
> >> >>
> >> >> ---However---if I use today's git (2010-07-01) of kvm:
> >> >>
> >> >>   /usr/local/kvm-git/bin/qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
> >> >>        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
> >> >>
> >> >> This hangs just after initializing the Serial device (obtained by adding
> >> >> -serial stdio -append console=ttyS0):
> >> >>
> >> >> Note that this only happens with the disk interface set to virtio
> >> >> (if=virtio).  It works fine for ide (if=ide).
> >> >>
> >> >>
> >> >> Am I doing something wrong here?
> >> >> Is anyone else having this problem?
> >> >
> >> > I have seen this issue with a RHEL 5.5 guest running under
> >> > qemu-kvm.git.  It boots a new guest fine but hangs as you described
> >> > with the RHEL 5.5 kernel.  I have not investigated.
> >>
> >> This issue is affected by extboot, a feature that enables booting from
> >> virtio-blk devices.  I have just sent a patch to the KVM mailing list
> >> to restore extboot functionality which has been broken in
> >> qemu-kvm.git.  That patch can be used to work around this issue by
> >> using "-drive ...,boot=on" but it doesn't explain why the RHEL 5.5
> >> kernel hangs during serial initialization when extboot is not present.
> >>
> > Hang that happens during guest boot (after bootloader started the
> > kernel) cannot be worked around by extboot. extboot is also not needed
> > with latest qemu git to boot from virtio disks since the support for
> > that is in the bios now.
> 
> I agree that something else is going on here and needs to be
> investigated, but I do think that extboot can indirectly affect the
> guest boot.
> 
> With extboot the virtio-blk PCI adapter is not touched by the
> firmware/bootloader.  Is it possible that a virtio-blk interrupt is
> raised and not acknowledged before entering Linux.  When Linux brings
> up the serial port it gets swamped with interrupts?  That's just a
> guess.
> 
That is possible when bios is actually used to boot the guest, but bug
reporter uses -kernel option so no bios boot code should run at all.
Virtio is initialized anyway, but this will happen with boot=on too.

> Newer (non-RHEL 5.5) kernels boot fine without extboot, so another
> path to solving this bug is to bisect guest kernels.
> 
> Stefan

--
			Gleb.

Re: KVM git hangs with if=virtio (works under kvm 0.12.3)

[Stefan Hajnoczi]

2010/7/5 Gleb Natapov <gleb@redhat.com>:
> On Mon, Jul 05, 2010 at 01:36:08PM +0100, Stefan Hajnoczi wrote:
>> 2010/7/5 Gleb Natapov <gleb@redhat.com>:
>> > On Mon, Jul 05, 2010 at 01:11:25PM +0100, Stefan Hajnoczi wrote:
>> >> On Fri, Jul 2, 2010 at 9:07 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> >> > On Fri, Jul 2, 2010 at 4:31 AM, ewheeler <kvm@ew.ewheeler.org> wrote:
>> >> >> Hello all,
>> >> >>
>> >> >> I'm booting a CentOS kernel under today's KVM git and it hangs after
>> >> >> initializing the serial port when the drive if=virtio, but not when
>> >> >> drive if=ide.  Look close---this is not a "forgot to add virtio_blk"
>> >> >> problem.  If I use 0.12.3 from Ubuntu 10.04 it works properly.
>> >> >>
>> >> >> Reproduction:
>> >> >>
>> >> >> Using kvm 0.12.3 on ubuntu 10.04 (1:84+dfsg-0ubuntu16+0.12.3+noroms
>> >> >> +0ubuntu9) it will work properly:
>> >> >>
>> >> >>  qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
>> >> >>                        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
>> >> >>
>> >> >> As expected, the kernel panics unable to mount root (good-boot.png).
>> >> >> This makes sense, as "dummy-disk-image" is 1MB of 0x00 bytes.
>> >> >>
>> >> >> ---However---if I use today's git (2010-07-01) of kvm:
>> >> >>
>> >> >>   /usr/local/kvm-git/bin/qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
>> >> >>        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
>> >> >>
>> >> >> This hangs just after initializing the Serial device (obtained by adding
>> >> >> -serial stdio -append console=ttyS0):
>> >> >>
>> >> >> Note that this only happens with the disk interface set to virtio
>> >> >> (if=virtio).  It works fine for ide (if=ide).
>> >> >>
>> >> >>
>> >> >> Am I doing something wrong here?
>> >> >> Is anyone else having this problem?
>> >> >
>> >> > I have seen this issue with a RHEL 5.5 guest running under
>> >> > qemu-kvm.git.  It boots a new guest fine but hangs as you described
>> >> > with the RHEL 5.5 kernel.  I have not investigated.
>> >>
>> >> This issue is affected by extboot, a feature that enables booting from
>> >> virtio-blk devices.  I have just sent a patch to the KVM mailing list
>> >> to restore extboot functionality which has been broken in
>> >> qemu-kvm.git.  That patch can be used to work around this issue by
>> >> using "-drive ...,boot=on" but it doesn't explain why the RHEL 5.5
>> >> kernel hangs during serial initialization when extboot is not present.
>> >>
>> > Hang that happens during guest boot (after bootloader started the
>> > kernel) cannot be worked around by extboot. extboot is also not needed
>> > with latest qemu git to boot from virtio disks since the support for
>> > that is in the bios now.
>>
>> I agree that something else is going on here and needs to be
>> investigated, but I do think that extboot can indirectly affect the
>> guest boot.
>>
>> With extboot the virtio-blk PCI adapter is not touched by the
>> firmware/bootloader.  Is it possible that a virtio-blk interrupt is
>> raised and not acknowledged before entering Linux.  When Linux brings
>> up the serial port it gets swamped with interrupts?  That's just a
>> guess.
>>
> That is possible when bios is actually used to boot the guest, but bug
> reporter uses -kernel option so no bios boot code should run at all.
> Virtio is initialized anyway, but this will happen with boot=on too.

Okay, I got to the bottom of this.  Here's the story, see bottom of the mail for
the solution and workarounds:

It turns out that -kernel does involve the BIOS.  KVM pulls apart the bzImage
and makes it available via the fw_cfg interface.  The linuxboot.bin option ROM
is executed by the BIOS inside the VM to actually jump into the kernel.  This
means the BIOS does POST and sets itself up; the kernel's real-mode boot
code is going to use BIOS interrupts.

So now the VM has booted, BIOS finished POST and executed linuxboot.bin,
linuxboot.bin transferred control to Linux.  Then, in Linux arch/x86/boot/edd.c
a disk read to sector=0 bytes=512 is made using INT 13h.  Since this disk
read comes from the Linux kernel, it happens regardless of -kernel or not.

The disk read is serviced by the BIOS.  Older versions of SeaBIOS leave the
interrupt raised here, so then the kernel hangs in serial initialization later.

However, there is a simple workaround:

  x86_64-softmmu/qemu-system-x86_64 -m 512 -drive
file=~/rhel5u5.img,if=virtio -kernel /boot/vmlinuz-2.6.32-5-amd64
-append edd=skipmbr

When the edd=skipmbr kernel parameter is used, the kernel will not perform
the disk read and the interrupt will never get stuck.  The VM boots
successfully.

The edd=skipmbr workaround is not enough when booting from disk and not
-kernel.  The BIOS and bootloader will invoke INT 13h and the only way
around that is to use extboot.bin with boot=on as I originally described.

The root cause was fixed in SeaBIOS commit:

  commit 4030db0d2c5a79de2a1b5c31514503e4ff2a3cd1
  Author: Gleb Natapov <gleb@redhat.com>
  Date:   Mon May 17 16:27:27 2010 +0300

      fix two issues with virtio-blk

      1. Check if blk_size is valid in virtio_blk config.
      2. Disable interrupt otherwise interrupt may stuck
         with some guests.

If you build SeaBIOS from source and launch KVM with -bios
path/to/seabios/out/bios.bin, RHEL5.5 will boot without hanging at serial
initialization time.

QEMU and KVM need to grab a later SeaBIOS build that contains 4030db0.

Stefan

PS: If you test this on qemu.git, you may find that the kernel doesn't hang
at serial initialization, despite the bug being present in the BIOS.  I'm not
sure how QEMU gets away with it but I wonder if the interrupt configuration
is different.

Re: KVM git hangs with if=virtio (works under kvm 0.12.3)

[Gleb Natapov]

On Wed, Jul 07, 2010 at 10:03:31AM +0100, Stefan Hajnoczi wrote:
> 2010/7/5 Gleb Natapov <gleb@redhat.com>:
> > On Mon, Jul 05, 2010 at 01:36:08PM +0100, Stefan Hajnoczi wrote:
> >> 2010/7/5 Gleb Natapov <gleb@redhat.com>:
> >> > On Mon, Jul 05, 2010 at 01:11:25PM +0100, Stefan Hajnoczi wrote:
> >> >> On Fri, Jul 2, 2010 at 9:07 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >> >> > On Fri, Jul 2, 2010 at 4:31 AM, ewheeler <kvm@ew.ewheeler.org> wrote:
> >> >> >> Hello all,
> >> >> >>
> >> >> >> I'm booting a CentOS kernel under today's KVM git and it hangs after
> >> >> >> initializing the serial port when the drive if=virtio, but not when
> >> >> >> drive if=ide.  Look close---this is not a "forgot to add virtio_blk"
> >> >> >> problem.  If I use 0.12.3 from Ubuntu 10.04 it works properly.
> >> >> >>
> >> >> >> Reproduction:
> >> >> >>
> >> >> >> Using kvm 0.12.3 on ubuntu 10.04 (1:84+dfsg-0ubuntu16+0.12.3+noroms
> >> >> >> +0ubuntu9) it will work properly:
> >> >> >>
> >> >> >>  qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
> >> >> >>                        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
> >> >> >>
> >> >> >> As expected, the kernel panics unable to mount root (good-boot.png).
> >> >> >> This makes sense, as "dummy-disk-image" is 1MB of 0x00 bytes.
> >> >> >>
> >> >> >> ---However---if I use today's git (2010-07-01) of kvm:
> >> >> >>
> >> >> >>   /usr/local/kvm-git/bin/qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
> >> >> >>        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
> >> >> >>
> >> >> >> This hangs just after initializing the Serial device (obtained by adding
> >> >> >> -serial stdio -append console=ttyS0):
> >> >> >>
> >> >> >> Note that this only happens with the disk interface set to virtio
> >> >> >> (if=virtio).  It works fine for ide (if=ide).
> >> >> >>
> >> >> >>
> >> >> >> Am I doing something wrong here?
> >> >> >> Is anyone else having this problem?
> >> >> >
> >> >> > I have seen this issue with a RHEL 5.5 guest running under
> >> >> > qemu-kvm.git.  It boots a new guest fine but hangs as you described
> >> >> > with the RHEL 5.5 kernel.  I have not investigated.
> >> >>
> >> >> This issue is affected by extboot, a feature that enables booting from
> >> >> virtio-blk devices.  I have just sent a patch to the KVM mailing list
> >> >> to restore extboot functionality which has been broken in
> >> >> qemu-kvm.git.  That patch can be used to work around this issue by
> >> >> using "-drive ...,boot=on" but it doesn't explain why the RHEL 5.5
> >> >> kernel hangs during serial initialization when extboot is not present.
> >> >>
> >> > Hang that happens during guest boot (after bootloader started the
> >> > kernel) cannot be worked around by extboot. extboot is also not needed
> >> > with latest qemu git to boot from virtio disks since the support for
> >> > that is in the bios now.
> >>
> >> I agree that something else is going on here and needs to be
> >> investigated, but I do think that extboot can indirectly affect the
> >> guest boot.
> >>
> >> With extboot the virtio-blk PCI adapter is not touched by the
> >> firmware/bootloader.  Is it possible that a virtio-blk interrupt is
> >> raised and not acknowledged before entering Linux.  When Linux brings
> >> up the serial port it gets swamped with interrupts?  That's just a
> >> guess.
> >>
> > That is possible when bios is actually used to boot the guest, but bug
> > reporter uses -kernel option so no bios boot code should run at all.
> > Virtio is initialized anyway, but this will happen with boot=on too.
> 
> Okay, I got to the bottom of this.  Here's the story, see bottom of the mail for
> the solution and workarounds:
> 
Great. Thanks Stefan. I was sure that fix you found is in qemu.git
already. It is good to know that Linux actually uses int13 once. Didn't
know that. To be absolutely sure no interrupt will stuck we should probably
clear interrupt after each virtio read in bios by reading status register.

> It turns out that -kernel does involve the BIOS.  KVM pulls apart the bzImage
> and makes it available via the fw_cfg interface.  The linuxboot.bin option ROM
> is executed by the BIOS inside the VM to actually jump into the kernel.  This
> means the BIOS does POST and sets itself up; the kernel's real-mode boot
> code is going to use BIOS interrupts.
> 
> So now the VM has booted, BIOS finished POST and executed linuxboot.bin,
> linuxboot.bin transferred control to Linux.  Then, in Linux arch/x86/boot/edd.c
> a disk read to sector=0 bytes=512 is made using INT 13h.  Since this disk
> read comes from the Linux kernel, it happens regardless of -kernel or not.
> 
> The disk read is serviced by the BIOS.  Older versions of SeaBIOS leave the
> interrupt raised here, so then the kernel hangs in serial initialization later.
> 
> However, there is a simple workaround:
> 
>   x86_64-softmmu/qemu-system-x86_64 -m 512 -drive
> file=~/rhel5u5.img,if=virtio -kernel /boot/vmlinuz-2.6.32-5-amd64
> -append edd=skipmbr
> 
> When the edd=skipmbr kernel parameter is used, the kernel will not perform
> the disk read and the interrupt will never get stuck.  The VM boots
> successfully.
> 
> The edd=skipmbr workaround is not enough when booting from disk and not
> -kernel.  The BIOS and bootloader will invoke INT 13h and the only way
> around that is to use extboot.bin with boot=on as I originally described.
> 
> The root cause was fixed in SeaBIOS commit:
> 
>   commit 4030db0d2c5a79de2a1b5c31514503e4ff2a3cd1
>   Author: Gleb Natapov <gleb@redhat.com>
>   Date:   Mon May 17 16:27:27 2010 +0300
> 
>       fix two issues with virtio-blk
> 
>       1. Check if blk_size is valid in virtio_blk config.
>       2. Disable interrupt otherwise interrupt may stuck
>          with some guests.
> 
> If you build SeaBIOS from source and launch KVM with -bios
> path/to/seabios/out/bios.bin, RHEL5.5 will boot without hanging at serial
> initialization time.
> 
> QEMU and KVM need to grab a later SeaBIOS build that contains 4030db0.
> 
> Stefan
> 
> PS: If you test this on qemu.git, you may find that the kernel doesn't hang
> at serial initialization, despite the bug being present in the BIOS.  I'm not
> sure how QEMU gets away with it but I wonder if the interrupt configuration
> is different.

--
			Gleb.

Re: KVM git hangs with if=virtio (works under kvm 0.12.3)

[Stefan Hajnoczi]

2010/7/7 Gleb Natapov <gleb@redhat.com>:
> On Wed, Jul 07, 2010 at 10:03:31AM +0100, Stefan Hajnoczi wrote:
>> 2010/7/5 Gleb Natapov <gleb@redhat.com>:
>> > On Mon, Jul 05, 2010 at 01:36:08PM +0100, Stefan Hajnoczi wrote:
>> >> 2010/7/5 Gleb Natapov <gleb@redhat.com>:
>> >> > On Mon, Jul 05, 2010 at 01:11:25PM +0100, Stefan Hajnoczi wrote:
>> >> >> On Fri, Jul 2, 2010 at 9:07 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> >> >> > On Fri, Jul 2, 2010 at 4:31 AM, ewheeler <kvm@ew.ewheeler.org> wrote:
>> >> >> >> Hello all,
>> >> >> >>
>> >> >> >> I'm booting a CentOS kernel under today's KVM git and it hangs after
>> >> >> >> initializing the serial port when the drive if=virtio, but not when
>> >> >> >> drive if=ide.  Look close---this is not a "forgot to add virtio_blk"
>> >> >> >> problem.  If I use 0.12.3 from Ubuntu 10.04 it works properly.
>> >> >> >>
>> >> >> >> Reproduction:
>> >> >> >>
>> >> >> >> Using kvm 0.12.3 on ubuntu 10.04 (1:84+dfsg-0ubuntu16+0.12.3+noroms
>> >> >> >> +0ubuntu9) it will work properly:
>> >> >> >>
>> >> >> >>  qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
>> >> >> >>                        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
>> >> >> >>
>> >> >> >> As expected, the kernel panics unable to mount root (good-boot.png).
>> >> >> >> This makes sense, as "dummy-disk-image" is 1MB of 0x00 bytes.
>> >> >> >>
>> >> >> >> ---However---if I use today's git (2010-07-01) of kvm:
>> >> >> >>
>> >> >> >>   /usr/local/kvm-git/bin/qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
>> >> >> >>        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
>> >> >> >>
>> >> >> >> This hangs just after initializing the Serial device (obtained by adding
>> >> >> >> -serial stdio -append console=ttyS0):
>> >> >> >>
>> >> >> >> Note that this only happens with the disk interface set to virtio
>> >> >> >> (if=virtio).  It works fine for ide (if=ide).
>> >> >> >>
>> >> >> >>
>> >> >> >> Am I doing something wrong here?
>> >> >> >> Is anyone else having this problem?
>> >> >> >
>> >> >> > I have seen this issue with a RHEL 5.5 guest running under
>> >> >> > qemu-kvm.git.  It boots a new guest fine but hangs as you described
>> >> >> > with the RHEL 5.5 kernel.  I have not investigated.
>> >> >>
>> >> >> This issue is affected by extboot, a feature that enables booting from
>> >> >> virtio-blk devices.  I have just sent a patch to the KVM mailing list
>> >> >> to restore extboot functionality which has been broken in
>> >> >> qemu-kvm.git.  That patch can be used to work around this issue by
>> >> >> using "-drive ...,boot=on" but it doesn't explain why the RHEL 5.5
>> >> >> kernel hangs during serial initialization when extboot is not present.
>> >> >>
>> >> > Hang that happens during guest boot (after bootloader started the
>> >> > kernel) cannot be worked around by extboot. extboot is also not needed
>> >> > with latest qemu git to boot from virtio disks since the support for
>> >> > that is in the bios now.
>> >>
>> >> I agree that something else is going on here and needs to be
>> >> investigated, but I do think that extboot can indirectly affect the
>> >> guest boot.
>> >>
>> >> With extboot the virtio-blk PCI adapter is not touched by the
>> >> firmware/bootloader.  Is it possible that a virtio-blk interrupt is
>> >> raised and not acknowledged before entering Linux.  When Linux brings
>> >> up the serial port it gets swamped with interrupts?  That's just a
>> >> guess.
>> >>
>> > That is possible when bios is actually used to boot the guest, but bug
>> > reporter uses -kernel option so no bios boot code should run at all.
>> > Virtio is initialized anyway, but this will happen with boot=on too.
>>
>> Okay, I got to the bottom of this.  Here's the story, see bottom of the mail for
>> the solution and workarounds:
>>
> Great. Thanks Stefan. I was sure that fix you found is in qemu.git
> already. It is good to know that Linux actually uses int13 once. Didn't
> know that. To be absolutely sure no interrupt will stuck we should probably
> clear interrupt after each virtio read in bios by reading status register.

Yes, this is the approach I took when rewriting gPXE's virtio-net
driver recently:

http://git.etherboot.org/?p=people/stefanha/gpxe.git;a=blob;f=src/drivers/net/virtio-net.c;h=50c580049c3663f0d1332d51e30c8a59b5663b68;hb=d58bf6fc8db4a55075e370c7e1d436a570400f55#l299

Are you spinning a patch or should I submit one?

>> It turns out that -kernel does involve the BIOS.  KVM pulls apart the bzImage
>> and makes it available via the fw_cfg interface.  The linuxboot.bin option ROM
>> is executed by the BIOS inside the VM to actually jump into the kernel.  This
>> means the BIOS does POST and sets itself up; the kernel's real-mode boot
>> code is going to use BIOS interrupts.
>>
>> So now the VM has booted, BIOS finished POST and executed linuxboot.bin,
>> linuxboot.bin transferred control to Linux.  Then, in Linux arch/x86/boot/edd.c
>> a disk read to sector=0 bytes=512 is made using INT 13h.  Since this disk
>> read comes from the Linux kernel, it happens regardless of -kernel or not.
>>
>> The disk read is serviced by the BIOS.  Older versions of SeaBIOS leave the
>> interrupt raised here, so then the kernel hangs in serial initialization later.
>>
>> However, there is a simple workaround:
>>
>>   x86_64-softmmu/qemu-system-x86_64 -m 512 -drive
>> file=~/rhel5u5.img,if=virtio -kernel /boot/vmlinuz-2.6.32-5-amd64
>> -append edd=skipmbr
>>
>> When the edd=skipmbr kernel parameter is used, the kernel will not perform
>> the disk read and the interrupt will never get stuck.  The VM boots
>> successfully.
>>
>> The edd=skipmbr workaround is not enough when booting from disk and not
>> -kernel.  The BIOS and bootloader will invoke INT 13h and the only way
>> around that is to use extboot.bin with boot=on as I originally described.
>>
>> The root cause was fixed in SeaBIOS commit:
>>
>>   commit 4030db0d2c5a79de2a1b5c31514503e4ff2a3cd1
>>   Author: Gleb Natapov <gleb@redhat.com>
>>   Date:   Mon May 17 16:27:27 2010 +0300
>>
>>       fix two issues with virtio-blk
>>
>>       1. Check if blk_size is valid in virtio_blk config.
>>       2. Disable interrupt otherwise interrupt may stuck
>>          with some guests.
>>
>> If you build SeaBIOS from source and launch KVM with -bios
>> path/to/seabios/out/bios.bin, RHEL5.5 will boot without hanging at serial
>> initialization time.
>>
>> QEMU and KVM need to grab a later SeaBIOS build that contains 4030db0.
>>
>> Stefan
>>
>> PS: If you test this on qemu.git, you may find that the kernel doesn't hang
>> at serial initialization, despite the bug being present in the BIOS.  I'm not
>> sure how QEMU gets away with it but I wonder if the interrupt configuration
>> is different.
>
> --
>                        Gleb.
>

Stefan

Re: KVM git hangs with if=virtio (works under kvm 0.12.3)

[Gleb Natapov]

On Wed, Jul 07, 2010 at 10:36:17AM +0100, Stefan Hajnoczi wrote:
> 2010/7/7 Gleb Natapov <gleb@redhat.com>:
> > On Wed, Jul 07, 2010 at 10:03:31AM +0100, Stefan Hajnoczi wrote:
> >> 2010/7/5 Gleb Natapov <gleb@redhat.com>:
> >> > On Mon, Jul 05, 2010 at 01:36:08PM +0100, Stefan Hajnoczi wrote:
> >> >> 2010/7/5 Gleb Natapov <gleb@redhat.com>:
> >> >> > On Mon, Jul 05, 2010 at 01:11:25PM +0100, Stefan Hajnoczi wrote:
> >> >> >> On Fri, Jul 2, 2010 at 9:07 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >> >> >> > On Fri, Jul 2, 2010 at 4:31 AM, ewheeler <kvm@ew.ewheeler.org> wrote:
> >> >> >> >> Hello all,
> >> >> >> >>
> >> >> >> >> I'm booting a CentOS kernel under today's KVM git and it hangs after
> >> >> >> >> initializing the serial port when the drive if=virtio, but not when
> >> >> >> >> drive if=ide.  Look close---this is not a "forgot to add virtio_blk"
> >> >> >> >> problem.  If I use 0.12.3 from Ubuntu 10.04 it works properly.
> >> >> >> >>
> >> >> >> >> Reproduction:
> >> >> >> >>
> >> >> >> >> Using kvm 0.12.3 on ubuntu 10.04 (1:84+dfsg-0ubuntu16+0.12.3+noroms
> >> >> >> >> +0ubuntu9) it will work properly:
> >> >> >> >>
> >> >> >> >>  qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
> >> >> >> >>                        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
> >> >> >> >>
> >> >> >> >> As expected, the kernel panics unable to mount root (good-boot.png).
> >> >> >> >> This makes sense, as "dummy-disk-image" is 1MB of 0x00 bytes.
> >> >> >> >>
> >> >> >> >> ---However---if I use today's git (2010-07-01) of kvm:
> >> >> >> >>
> >> >> >> >>   /usr/local/kvm-git/bin/qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
> >> >> >> >>        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
> >> >> >> >>
> >> >> >> >> This hangs just after initializing the Serial device (obtained by adding
> >> >> >> >> -serial stdio -append console=ttyS0):
> >> >> >> >>
> >> >> >> >> Note that this only happens with the disk interface set to virtio
> >> >> >> >> (if=virtio).  It works fine for ide (if=ide).
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Am I doing something wrong here?
> >> >> >> >> Is anyone else having this problem?
> >> >> >> >
> >> >> >> > I have seen this issue with a RHEL 5.5 guest running under
> >> >> >> > qemu-kvm.git.  It boots a new guest fine but hangs as you described
> >> >> >> > with the RHEL 5.5 kernel.  I have not investigated.
> >> >> >>
> >> >> >> This issue is affected by extboot, a feature that enables booting from
> >> >> >> virtio-blk devices.  I have just sent a patch to the KVM mailing list
> >> >> >> to restore extboot functionality which has been broken in
> >> >> >> qemu-kvm.git.  That patch can be used to work around this issue by
> >> >> >> using "-drive ...,boot=on" but it doesn't explain why the RHEL 5.5
> >> >> >> kernel hangs during serial initialization when extboot is not present.
> >> >> >>
> >> >> > Hang that happens during guest boot (after bootloader started the
> >> >> > kernel) cannot be worked around by extboot. extboot is also not needed
> >> >> > with latest qemu git to boot from virtio disks since the support for
> >> >> > that is in the bios now.
> >> >>
> >> >> I agree that something else is going on here and needs to be
> >> >> investigated, but I do think that extboot can indirectly affect the
> >> >> guest boot.
> >> >>
> >> >> With extboot the virtio-blk PCI adapter is not touched by the
> >> >> firmware/bootloader.  Is it possible that a virtio-blk interrupt is
> >> >> raised and not acknowledged before entering Linux.  When Linux brings
> >> >> up the serial port it gets swamped with interrupts?  That's just a
> >> >> guess.
> >> >>
> >> > That is possible when bios is actually used to boot the guest, but bug
> >> > reporter uses -kernel option so no bios boot code should run at all.
> >> > Virtio is initialized anyway, but this will happen with boot=on too.
> >>
> >> Okay, I got to the bottom of this.  Here's the story, see bottom of the mail for
> >> the solution and workarounds:
> >>
> > Great. Thanks Stefan. I was sure that fix you found is in qemu.git
> > already. It is good to know that Linux actually uses int13 once. Didn't
> > know that. To be absolutely sure no interrupt will stuck we should probably
> > clear interrupt after each virtio read in bios by reading status register.
> 
> Yes, this is the approach I took when rewriting gPXE's virtio-net
> driver recently:
> 
> http://git.etherboot.org/?p=people/stefanha/gpxe.git;a=blob;f=src/drivers/net/virtio-net.c;h=50c580049c3663f0d1332d51e30c8a59b5663b68;hb=d58bf6fc8db4a55075e370c7e1d436a570400f55#l299
> 
> Are you spinning a patch or should I submit one?
> 
If you have time do it please. I am a little bit busy right now :(

--
			Gleb.

Re: KVM git hangs with if=virtio (works under kvm 0.12.3)

[ewheeler]

On Wed, 2010-07-07 at 10:03 +0100, Stefan Hajnoczi wrote:
> 2010/7/5 Gleb Natapov <gleb@redhat.com>:
> > On Mon, Jul 05, 2010 at 01:36:08PM +0100, Stefan Hajnoczi wrote:
> >> 2010/7/5 Gleb Natapov <gleb@redhat.com>:
> >> > On Mon, Jul 05, 2010 at 01:11:25PM +0100, Stefan Hajnoczi wrote:
> >> >> On Fri, Jul 2, 2010 at 9:07 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >> >> > On Fri, Jul 2, 2010 at 4:31 AM, ewheeler <kvm@ew.ewheeler.org> wrote:
> >> >> >> Hello all,
> >> >> >>
> >> >> >> I'm booting a CentOS kernel under today's KVM git and it hangs after
> >> >> >> initializing the serial port when the drive if=virtio, but not when
> >> >> >> drive if=ide.  Look close---this is not a "forgot to add virtio_blk"
> >> >> >> problem.  If I use 0.12.3 from Ubuntu 10.04 it works properly.
> >> >> >>
> >> >> >> Reproduction:
> >> >> >>
> >> >> >> Using kvm 0.12.3 on ubuntu 10.04 (1:84+dfsg-0ubuntu16+0.12.3+noroms
> >> >> >> +0ubuntu9) it will work properly:
> >> >> >>
> >> >> >>  qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
> >> >> >>                        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
> >> >> >>
> >> >> >> As expected, the kernel panics unable to mount root (good-boot.png).
> >> >> >> This makes sense, as "dummy-disk-image" is 1MB of 0x00 bytes.
> >> >> >>
> >> >> >> ---However---if I use today's git (2010-07-01) of kvm:
> >> >> >>
> >> >> >>   /usr/local/kvm-git/bin/qemu-system-x86_64 -drive file=dummy-disk-image,if=virtio \
> >> >> >>        -kernel vmlinuz-2.6.18-194.3.1.el5.centos.plus
> >> >> >>
> >> >> >> This hangs just after initializing the Serial device (obtained by adding
> >> >> >> -serial stdio -append console=ttyS0):
> >> >> >>
> >> >> >> Note that this only happens with the disk interface set to virtio
> >> >> >> (if=virtio).  It works fine for ide (if=ide).
> >> >> >>
> >> >> >>
> >> >> >> Am I doing something wrong here?
> >> >> >> Is anyone else having this problem?
> >> >> >
> >> >> > I have seen this issue with a RHEL 5.5 guest running under
> >> >> > qemu-kvm.git.  It boots a new guest fine but hangs as you described
> >> >> > with the RHEL 5.5 kernel.  I have not investigated.
> >> >>
> >> >> This issue is affected by extboot, a feature that enables booting from
> >> >> virtio-blk devices.  I have just sent a patch to the KVM mailing list
> >> >> to restore extboot functionality which has been broken in
> >> >> qemu-kvm.git.  That patch can be used to work around this issue by
> >> >> using "-drive ...,boot=on" but it doesn't explain why the RHEL 5.5
> >> >> kernel hangs during serial initialization when extboot is not present.
> >> >>
> >> > Hang that happens during guest boot (after bootloader started the
> >> > kernel) cannot be worked around by extboot. extboot is also not needed
> >> > with latest qemu git to boot from virtio disks since the support for
> >> > that is in the bios now.
> >>
> >> I agree that something else is going on here and needs to be
> >> investigated, but I do think that extboot can indirectly affect the
> >> guest boot.
> >>
> >> With extboot the virtio-blk PCI adapter is not touched by the
> >> firmware/bootloader.  Is it possible that a virtio-blk interrupt is
> >> raised and not acknowledged before entering Linux.  When Linux brings
> >> up the serial port it gets swamped with interrupts?  That's just a
> >> guess.
> >>
> > That is possible when bios is actually used to boot the guest, but bug
> > reporter uses -kernel option so no bios boot code should run at all.
> > Virtio is initialized anyway, but this will happen with boot=on too.
> 
> Okay, I got to the bottom of this.  Here's the story, see bottom of the mail for
> the solution and workarounds:
> 
> It turns out that -kernel does involve the BIOS.  KVM pulls apart the bzImage
> and makes it available via the fw_cfg interface.  The linuxboot.bin option ROM
> is executed by the BIOS inside the VM to actually jump into the kernel.  This
> means the BIOS does POST and sets itself up; the kernel's real-mode boot
> code is going to use BIOS interrupts.
> 
> So now the VM has booted, BIOS finished POST and executed linuxboot.bin,
> linuxboot.bin transferred control to Linux.  Then, in Linux arch/x86/boot/edd.c
> a disk read to sector=0 bytes=512 is made using INT 13h.  Since this disk
> read comes from the Linux kernel, it happens regardless of -kernel or not.
> 
> The disk read is serviced by the BIOS.  Older versions of SeaBIOS leave the
> interrupt raised here, so then the kernel hangs in serial initialization later.
> 
> However, there is a simple workaround:
> 
>   x86_64-softmmu/qemu-system-x86_64 -m 512 -drive
> file=~/rhel5u5.img,if=virtio -kernel /boot/vmlinuz-2.6.32-5-amd64
> -append edd=skipmbr
> 
> When the edd=skipmbr kernel parameter is used, the kernel will not perform
> the disk read and the interrupt will never get stuck.  The VM boots
> successfully.

Wow, brilliant.  Thank you for jumping into this so quickly!

-Eric