[Qemu-devel] About the light VM solution!

[Qemu-devel] About the light VM solution!

[Yang Zhong]

Hello all,

As you know, AWS has decided to switch to KVM in their clouds. This news make almost all
china CSPs(clouds service provider) pay more attention on KVM/Qemu, especially light VM
solution.

Below are intel solution for light VM, qemu-lite.
http://events.linuxfoundation.org/sites/events/files/slides/Light%20weight%20virtualization%20with%20QEMU%26KVM_0.pdf

My question is whether community has some plan to implement light VM or alternative solutions? If no, whether our 
qemu-lite solution is suitable for upstream again? Many thanks!

Regards,

Yang

Re: [Qemu-devel] About the light VM solution!

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1841 bytes --]

On Tue, Dec 05, 2017 at 02:33:13PM +0800, Yang Zhong wrote:
> As you know, AWS has decided to switch to KVM in their clouds. This news make almost all
> china CSPs(clouds service provider) pay more attention on KVM/Qemu, especially light VM
> solution.
> 
> Below are intel solution for light VM, qemu-lite.
> http://events.linuxfoundation.org/sites/events/files/slides/Light%20weight%20virtualization%20with%20QEMU%26KVM_0.pdf
> 
> My question is whether community has some plan to implement light VM or alternative solutions? If no, whether our 
> qemu-lite solution is suitable for upstream again? Many thanks!

Booting VMs faster is appreciated upstream.  I think there is interest
in patches that further this goal and hope you have time to contribute.

What caused a lot of discussion and held back progress was the approach
that was taken.  The basic philosophy seems to be bypassing or
special-casing components in order to avoid slow operations.  This
requires special QEMU, firmware, and/or guest kernel binaries and causes
extra work for the management stack, distributions, and testers.  It is
preferrable to have just one binary and no special configuration
settings.

I think patches are more likely to be merged with the following
approach:
1. Benchmark or profile to find slow operations.
2. Perform an experiment to see if bypassing the operations improves
   performance.  If no, go back to step 1.
3. Investigate the slow operation to see if it can be optimized or
   skipped completely based on run-time knowledge.  This means no
   special '-lite' binaries or new config options.
4. Implement patches to do this, retest, and send them upstream.

My view is that qemu-lite only got to Step 2 in some cases.  Going to
Step 4 is more work but the result will be easier to merge.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] About the light VM solution!

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 2528 bytes --]

On 05/12/2017 13:06, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 02:33:13PM +0800, Yang Zhong wrote:
>> As you know, AWS has decided to switch to KVM in their clouds. This news make almost all
>> china CSPs(clouds service provider) pay more attention on KVM/Qemu, especially light VM
>> solution.
>>
>> Below are intel solution for light VM, qemu-lite.
>> http://events.linuxfoundation.org/sites/events/files/slides/Light%20weight%20virtualization%20with%20QEMU%26KVM_0.pdf
>>
>> My question is whether community has some plan to implement light VM or alternative solutions? If no, whether our 
>> qemu-lite solution is suitable for upstream again? Many thanks!
> 
> What caused a lot of discussion and held back progress was the approach
> that was taken.  The basic philosophy seems to be bypassing or
> special-casing components in order to avoid slow operations.  This
> requires special QEMU, firmware, and/or guest kernel binaries and causes
> extra work for the management stack, distributions, and testers.

I think having a special firmware (be it qboot or a special-purpose
SeaBIOS) is acceptable.

So is having a special QEMU binary with fewer runtime dependencies,
though that should only be a stopgap measure; the real solution is to
modularize e.g. the UI and audio subsystems to remove runtime
dependencies from the QEMU binary itself.

I agree with Stefan however that there should be no special machine
types or kernels.

Referring to your slides, the remaining points for fast boot are:

* parallelize VCPU initialization: do you have patches? :)

* q35-lite: any other machine options that have not been merged yet?

* SeaBIOS+Option ROM: can you take new numbers with DMA-based option
ROM, or with qboot?

* guest kernel: my proposal to make Linux a multiboot kernel has been
nacked upstream, but Oracle is working on supporting Xen PVH binaries in
QEMU.  These are very similar to multiboot and in particular they're
uncompressed.

For memory consumption, 2.11 should have improved things already thanks
to shared FlatViews.  Your malloc_trim patch should also be merged in 2.12.

So are things really that bad?  Almost all "qemu-lite" patches that have
been proposed upstream have been accepted.

Only mmap-ing the kernel into the guest is not going to be accepted, but
maybe you can look into replacing stdio with open+mmap in load_linux and
load_multiboot, for both the kernel and the initrd.  This should save a
few milliseconds too.

Paolo


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] About the light VM solution!

[Stefan Hajnoczi]

On Tue, Dec 5, 2017 at 1:35 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> On 05/12/2017 13:06, Stefan Hajnoczi wrote:
>> On Tue, Dec 05, 2017 at 02:33:13PM +0800, Yang Zhong wrote:
>>> As you know, AWS has decided to switch to KVM in their clouds. This news make almost all
>>> china CSPs(clouds service provider) pay more attention on KVM/Qemu, especially light VM
>>> solution.
>>>
>>> Below are intel solution for light VM, qemu-lite.
>>> http://events.linuxfoundation.org/sites/events/files/slides/Light%20weight%20virtualization%20with%20QEMU%26KVM_0.pdf
>>>
>>> My question is whether community has some plan to implement light VM or alternative solutions? If no, whether our
>>> qemu-lite solution is suitable for upstream again? Many thanks!
>>
>> What caused a lot of discussion and held back progress was the approach
>> that was taken.  The basic philosophy seems to be bypassing or
>> special-casing components in order to avoid slow operations.  This
>> requires special QEMU, firmware, and/or guest kernel binaries and causes
>> extra work for the management stack, distributions, and testers.
>
> I think having a special firmware (be it qboot or a special-purpose
> SeaBIOS) is acceptable.

The work Marc Mari Barcelo did in 2015 showed that SeaBIOS can boot
guests quickly.  The guest kernel was entered in <35 milliseconds
IIRC.  Why is special firmware necessary?

I'm not against additional binaries if there's no other way, but it's
important to demonstrate why special-casing is necessary.

Re: [Qemu-devel] About the light VM solution!

[Paolo Bonzini]

On 05/12/2017 14:47, Stefan Hajnoczi wrote:
> On Tue, Dec 5, 2017 at 1:35 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> On 05/12/2017 13:06, Stefan Hajnoczi wrote:
>>> On Tue, Dec 05, 2017 at 02:33:13PM +0800, Yang Zhong wrote:
>>>> As you know, AWS has decided to switch to KVM in their clouds. This news make almost all
>>>> china CSPs(clouds service provider) pay more attention on KVM/Qemu, especially light VM
>>>> solution.
>>>>
>>>> Below are intel solution for light VM, qemu-lite.
>>>> http://events.linuxfoundation.org/sites/events/files/slides/Light%20weight%20virtualization%20with%20QEMU%26KVM_0.pdf
>>>>
>>>> My question is whether community has some plan to implement light VM or alternative solutions? If no, whether our
>>>> qemu-lite solution is suitable for upstream again? Many thanks!
>>>
>>> What caused a lot of discussion and held back progress was the approach
>>> that was taken.  The basic philosophy seems to be bypassing or
>>> special-casing components in order to avoid slow operations.  This
>>> requires special QEMU, firmware, and/or guest kernel binaries and causes
>>> extra work for the management stack, distributions, and testers.
>>
>> I think having a special firmware (be it qboot or a special-purpose
>> SeaBIOS) is acceptable.
> 
> The work Marc Mari Barcelo did in 2015 showed that SeaBIOS can boot
> guests quickly.  The guest kernel was entered in <35 milliseconds
> IIRC.  Why is special firmware necessary?

I thought that wasn't the "conventional" SeaBIOS, but rather one with
reduced configuration options, but I may be remembering wrong.

Paolo

> I'm not against additional binaries if there's no other way, but it's
> important to demonstrate why special-casing is necessary.
> 

Re: [Qemu-devel] About the light VM solution!

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2043 bytes --]

On Tue, Dec 05, 2017 at 03:00:10PM +0100, Paolo Bonzini wrote:
> On 05/12/2017 14:47, Stefan Hajnoczi wrote:
> > On Tue, Dec 5, 2017 at 1:35 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> >> On 05/12/2017 13:06, Stefan Hajnoczi wrote:
> >>> On Tue, Dec 05, 2017 at 02:33:13PM +0800, Yang Zhong wrote:
> >>>> As you know, AWS has decided to switch to KVM in their clouds. This news make almost all
> >>>> china CSPs(clouds service provider) pay more attention on KVM/Qemu, especially light VM
> >>>> solution.
> >>>>
> >>>> Below are intel solution for light VM, qemu-lite.
> >>>> http://events.linuxfoundation.org/sites/events/files/slides/Light%20weight%20virtualization%20with%20QEMU%26KVM_0.pdf
> >>>>
> >>>> My question is whether community has some plan to implement light VM or alternative solutions? If no, whether our
> >>>> qemu-lite solution is suitable for upstream again? Many thanks!
> >>>
> >>> What caused a lot of discussion and held back progress was the approach
> >>> that was taken.  The basic philosophy seems to be bypassing or
> >>> special-casing components in order to avoid slow operations.  This
> >>> requires special QEMU, firmware, and/or guest kernel binaries and causes
> >>> extra work for the management stack, distributions, and testers.
> >>
> >> I think having a special firmware (be it qboot or a special-purpose
> >> SeaBIOS) is acceptable.
> > 
> > The work Marc Mari Barcelo did in 2015 showed that SeaBIOS can boot
> > guests quickly.  The guest kernel was entered in <35 milliseconds
> > IIRC.  Why is special firmware necessary?
> 
> I thought that wasn't the "conventional" SeaBIOS, but rather one with
> reduced configuration options, but I may be remembering wrong.

Marc didn't spend much time on optimizing SeaBIOS, he used the build
options that were suggested.  An extra flag can be added in
qemu_preinit() to skip slow init that's unnecessary on optimized
machines.  That would allow a single SeaBIOS binary to run both full and
lite systems.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] About the light VM solution!

[Gonglei (Arei)]

> -----Original Message-----
> From: Qemu-devel
> [mailto:qemu-devel-bounces+arei.gonglei=huawei.com@nongnu.org] On
> Behalf Of Stefan Hajnoczi
> Sent: Wednesday, December 06, 2017 12:31 AM
> To: Paolo Bonzini
> Cc: Yang Zhong; Stefan Hajnoczi; qemu-devel
> Subject: Re: [Qemu-devel] About the light VM solution!
> 
> On Tue, Dec 05, 2017 at 03:00:10PM +0100, Paolo Bonzini wrote:
> > On 05/12/2017 14:47, Stefan Hajnoczi wrote:
> > > On Tue, Dec 5, 2017 at 1:35 PM, Paolo Bonzini <pbonzini@redhat.com>
> wrote:
> > >> On 05/12/2017 13:06, Stefan Hajnoczi wrote:
> > >>> On Tue, Dec 05, 2017 at 02:33:13PM +0800, Yang Zhong wrote:
> > >>>> As you know, AWS has decided to switch to KVM in their clouds. This
> news make almost all
> > >>>> china CSPs(clouds service provider) pay more attention on KVM/Qemu,
> especially light VM
> > >>>> solution.
> > >>>>
> > >>>> Below are intel solution for light VM, qemu-lite.
> > >>>>
> http://events.linuxfoundation.org/sites/events/files/slides/Light%20weight%2
> 0virtualization%20with%20QEMU%26KVM_0.pdf
> > >>>>
> > >>>> My question is whether community has some plan to implement light
> VM or alternative solutions? If no, whether our
> > >>>> qemu-lite solution is suitable for upstream again? Many thanks!
> > >>>
> > >>> What caused a lot of discussion and held back progress was the approach
> > >>> that was taken.  The basic philosophy seems to be bypassing or
> > >>> special-casing components in order to avoid slow operations.  This
> > >>> requires special QEMU, firmware, and/or guest kernel binaries and
> causes
> > >>> extra work for the management stack, distributions, and testers.
> > >>
> > >> I think having a special firmware (be it qboot or a special-purpose
> > >> SeaBIOS) is acceptable.
> > >
> > > The work Marc Mari Barcelo did in 2015 showed that SeaBIOS can boot
> > > guests quickly.  The guest kernel was entered in <35 milliseconds
> > > IIRC.  Why is special firmware necessary?
> >
> > I thought that wasn't the "conventional" SeaBIOS, but rather one with
> > reduced configuration options, but I may be remembering wrong.
> 
> Marc didn't spend much time on optimizing SeaBIOS, he used the build
> options that were suggested.  An extra flag can be added in
> qemu_preinit() to skip slow init that's unnecessary on optimized
> machines.  That would allow a single SeaBIOS binary to run both full and
> lite systems.
> 
What's options do you remember? Stefan. Or any links about that
thread? I'm Interesting with this topic.

Thanks,
-Gonglei

Re: [Qemu-devel] About the light VM solution!

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3176 bytes --]

On Wed, Dec 06, 2017 at 09:21:55AM +0000, Gonglei (Arei) wrote:
> 
> > -----Original Message-----
> > From: Qemu-devel
> > [mailto:qemu-devel-bounces+arei.gonglei=huawei.com@nongnu.org] On
> > Behalf Of Stefan Hajnoczi
> > Sent: Wednesday, December 06, 2017 12:31 AM
> > To: Paolo Bonzini
> > Cc: Yang Zhong; Stefan Hajnoczi; qemu-devel
> > Subject: Re: [Qemu-devel] About the light VM solution!
> > 
> > On Tue, Dec 05, 2017 at 03:00:10PM +0100, Paolo Bonzini wrote:
> > > On 05/12/2017 14:47, Stefan Hajnoczi wrote:
> > > > On Tue, Dec 5, 2017 at 1:35 PM, Paolo Bonzini <pbonzini@redhat.com>
> > wrote:
> > > >> On 05/12/2017 13:06, Stefan Hajnoczi wrote:
> > > >>> On Tue, Dec 05, 2017 at 02:33:13PM +0800, Yang Zhong wrote:
> > > >>>> As you know, AWS has decided to switch to KVM in their clouds. This
> > news make almost all
> > > >>>> china CSPs(clouds service provider) pay more attention on KVM/Qemu,
> > especially light VM
> > > >>>> solution.
> > > >>>>
> > > >>>> Below are intel solution for light VM, qemu-lite.
> > > >>>>
> > http://events.linuxfoundation.org/sites/events/files/slides/Light%20weight%2
> > 0virtualization%20with%20QEMU%26KVM_0.pdf
> > > >>>>
> > > >>>> My question is whether community has some plan to implement light
> > VM or alternative solutions? If no, whether our
> > > >>>> qemu-lite solution is suitable for upstream again? Many thanks!
> > > >>>
> > > >>> What caused a lot of discussion and held back progress was the approach
> > > >>> that was taken.  The basic philosophy seems to be bypassing or
> > > >>> special-casing components in order to avoid slow operations.  This
> > > >>> requires special QEMU, firmware, and/or guest kernel binaries and
> > causes
> > > >>> extra work for the management stack, distributions, and testers.
> > > >>
> > > >> I think having a special firmware (be it qboot or a special-purpose
> > > >> SeaBIOS) is acceptable.
> > > >
> > > > The work Marc Mari Barcelo did in 2015 showed that SeaBIOS can boot
> > > > guests quickly.  The guest kernel was entered in <35 milliseconds
> > > > IIRC.  Why is special firmware necessary?
> > >
> > > I thought that wasn't the "conventional" SeaBIOS, but rather one with
> > > reduced configuration options, but I may be remembering wrong.
> > 
> > Marc didn't spend much time on optimizing SeaBIOS, he used the build
> > options that were suggested.  An extra flag can be added in
> > qemu_preinit() to skip slow init that's unnecessary on optimized
> > machines.  That would allow a single SeaBIOS binary to run both full and
> > lite systems.
> > 
> What's options do you remember? Stefan. Or any links about that
> thread? I'm Interesting with this topic.

Here is what I found:

Marc Mari's fastest SeaBIOS build took 8 ms from the first guest CPU
instruction to entering the guest kernel.  CBFS was used instead of a
normal boot device (e.g. virtio-blk).  Most hardware support was
disabled.

https://mail.coreboot.org/pipermail/seabios/2015-July/009554.html

The SeaBIOS configuration file is here:

https://mail.coreboot.org/pipermail/seabios/2015-July/009548.html

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] About the light VM solution!

[Gonglei (Arei)]

> -----Original Message-----
> From: Stefan Hajnoczi [mailto:stefanha@redhat.com]
> Sent: Wednesday, December 06, 2017 11:10 PM
> To: Gonglei (Arei)
> Cc: Paolo Bonzini; Yang Zhong; Stefan Hajnoczi; qemu-devel
> Subject: Re: [Qemu-devel] About the light VM solution!
> 
> On Wed, Dec 06, 2017 at 09:21:55AM +0000, Gonglei (Arei) wrote:
> >
> > > -----Original Message-----
> > > From: Qemu-devel
> > > [mailto:qemu-devel-bounces+arei.gonglei=huawei.com@nongnu.org] On
> > > Behalf Of Stefan Hajnoczi
> > > Sent: Wednesday, December 06, 2017 12:31 AM
> > > To: Paolo Bonzini
> > > Cc: Yang Zhong; Stefan Hajnoczi; qemu-devel
> > > Subject: Re: [Qemu-devel] About the light VM solution!
> > >
> > > On Tue, Dec 05, 2017 at 03:00:10PM +0100, Paolo Bonzini wrote:
> > > > On 05/12/2017 14:47, Stefan Hajnoczi wrote:
> > > > > On Tue, Dec 5, 2017 at 1:35 PM, Paolo Bonzini <pbonzini@redhat.com>
> > > wrote:
> > > > >> On 05/12/2017 13:06, Stefan Hajnoczi wrote:
> > > > >>> On Tue, Dec 05, 2017 at 02:33:13PM +0800, Yang Zhong wrote:
> > > > >>>> As you know, AWS has decided to switch to KVM in their clouds. This
> > > news make almost all
> > > > >>>> china CSPs(clouds service provider) pay more attention on
> KVM/Qemu,
> > > especially light VM
> > > > >>>> solution.
> > > > >>>>
> > > > >>>> Below are intel solution for light VM, qemu-lite.
> > > > >>>>
> > >
> http://events.linuxfoundation.org/sites/events/files/slides/Light%20weight%2
> > > 0virtualization%20with%20QEMU%26KVM_0.pdf
> > > > >>>>
> > > > >>>> My question is whether community has some plan to implement
> light
> > > VM or alternative solutions? If no, whether our
> > > > >>>> qemu-lite solution is suitable for upstream again? Many thanks!
> > > > >>>
> > > > >>> What caused a lot of discussion and held back progress was the
> approach
> > > > >>> that was taken.  The basic philosophy seems to be bypassing or
> > > > >>> special-casing components in order to avoid slow operations.  This
> > > > >>> requires special QEMU, firmware, and/or guest kernel binaries and
> > > causes
> > > > >>> extra work for the management stack, distributions, and testers.
> > > > >>
> > > > >> I think having a special firmware (be it qboot or a special-purpose
> > > > >> SeaBIOS) is acceptable.
> > > > >
> > > > > The work Marc Mari Barcelo did in 2015 showed that SeaBIOS can boot
> > > > > guests quickly.  The guest kernel was entered in <35 milliseconds
> > > > > IIRC.  Why is special firmware necessary?
> > > >
> > > > I thought that wasn't the "conventional" SeaBIOS, but rather one with
> > > > reduced configuration options, but I may be remembering wrong.
> > >
> > > Marc didn't spend much time on optimizing SeaBIOS, he used the build
> > > options that were suggested.  An extra flag can be added in
> > > qemu_preinit() to skip slow init that's unnecessary on optimized
> > > machines.  That would allow a single SeaBIOS binary to run both full and
> > > lite systems.
> > >
> > What's options do you remember? Stefan. Or any links about that
> > thread? I'm Interesting with this topic.
> 
> Here is what I found:
> 
> Marc Mari's fastest SeaBIOS build took 8 ms from the first guest CPU
> instruction to entering the guest kernel.  CBFS was used instead of a
> normal boot device (e.g. virtio-blk).  Most hardware support was
> disabled.
> 
> https://mail.coreboot.org/pipermail/seabios/2015-July/009554.html
> 
> The SeaBIOS configuration file is here:
> 
> https://mail.coreboot.org/pipermail/seabios/2015-July/009548.html
> 
Thanks for your information. :)
 
Thanks,
-Gonglei

Re: [Qemu-devel] About the light VM solution!

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1022 bytes --]

On Tue, Dec 05, 2017 at 02:35:42PM +0100, Paolo Bonzini wrote:
> On 05/12/2017 13:06, Stefan Hajnoczi wrote:
> > On Tue, Dec 05, 2017 at 02:33:13PM +0800, Yang Zhong wrote:
> Referring to your slides, the remaining points for fast boot are:
> 
> * parallelize VCPU initialization: do you have patches? :)
> 
> * q35-lite: any other machine options that have not been merged yet?
> 
> * SeaBIOS+Option ROM: can you take new numbers with DMA-based option
> ROM, or with qboot?
> 
> * guest kernel: my proposal to make Linux a multiboot kernel has been
> nacked upstream, but Oracle is working on supporting Xen PVH binaries in
> QEMU.  These are very similar to multiboot and in particular they're
> uncompressed.

By the way, sending a separate patch series for each optimization is
good.  That way patches requiring more discussion don't hold up patches
that could be applied immediately.  It also makes reviewing and
understanding performance results easier since there is only one change.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] About the light VM solution!

[Yang Zhong]

On Wed, Dec 06, 2017 at 03:11:55PM +0000, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 02:35:42PM +0100, Paolo Bonzini wrote:
> > On 05/12/2017 13:06, Stefan Hajnoczi wrote:
> > > On Tue, Dec 05, 2017 at 02:33:13PM +0800, Yang Zhong wrote:
> > Referring to your slides, the remaining points for fast boot are:
> > 
> > * parallelize VCPU initialization: do you have patches? :)
> > 
> > * q35-lite: any other machine options that have not been merged yet?
> > 
> > * SeaBIOS+Option ROM: can you take new numbers with DMA-based option
> > ROM, or with qboot?
> > 
> > * guest kernel: my proposal to make Linux a multiboot kernel has been
> > nacked upstream, but Oracle is working on supporting Xen PVH binaries in
> > QEMU.  These are very similar to multiboot and in particular they're
> > uncompressed.
> 
> By the way, sending a separate patch series for each optimization is
> good.  That way patches requiring more discussion don't hold up patches
> that could be applied immediately.  It also makes reviewing and
> understanding performance results easier since there is only one change.
> 
> Stefan

  Hello Stefan,Paolo and Richard,

  Many thanks for your detailed comments! Really thanks for your great help!

  We are talking about light VM solution with our customers firstly and want
  to know their concerns. Then we will continue to talk this on this thread.

  Richard, your paper and test script will do great help for our further work.

  I also CC PengChao and Anthony in this thread, they will join this talk.

  Regards,

  Yang 

Re: [Qemu-devel] About the light VM solution!

[Richard W.M. Jones]

I did a bit of work on this back in early 2016 and wrote a paper which
analyzes what Intel were doing with Clear Containers back then, and
how it fitted with the more distribution-centric view of Fedora and
Red Hat, ie that we ideally want a single qemu binary, a single
kernel, a single SeaBIOS, etc.

You can read the paper here:

  http://oirase.annexia.org/tmp/paper.pdf

and the source is here:

  http://git.annexia.org/?p=libguestfs-talks.git;a=tree;f=2016-eng-talk;h=5a0a29ceb1e9db39539669717ec06e4f94eba086;hb=HEAD

To address a few points from this thread:

* As Paolo mentioned one problem is we link qemu to so many libraries,
  and glibc / ELF dynamic loading is very slow.  Modularizing qemu
  could help here.  Reducing symbol interposition
  (-fvisibility=hidden) in more libraries would help a bit.  Linker
  security features enabled in downstream distros don't help.
  Rewriting ELF to be less crazy would help a lot but good luck there :-)

* As Stefan & Paolo mentioned, it would be nice if SeaBIOS was faster
  in the default configuration.  I ended up compiling a special
  minimal SeaBIOS which saved a load of time, mainly not probing PCI
  unnecessarily IIRC.

* Considerable time is taken in booting the kernel, and that's mostly
  in running all the initcall functions.  We wanted to use a Fedora
  distro kernel, but unfortunately many subsystems do loads of
  initcall work which is run even when that subsystem is not used.
  This is why compiling a custom kernel (compiling out these
  subsystems) is enticing.  Parallelizing initialization could help
  here (however at the moment using -smp slows things down), also
  parallelizing was rejected upstream IIRC.

* PCI config space probing is really slow.  Unfortunately accelerating
  it in the kernel doesn't seem either very easy or very acceptable to
  KVM upstream: the use case is rather narrow & the implementation
  seems like it would be very complex.  I also write a parallelizing
  PCI probe for Linux which helped a bit but wasn't upstream material.

* udev is another huge problem.  It's slow, it's monolithic, it
  resists modifications such as modularization or removing parts.

* Debugging over the UART is slow.

libguestfs also ships with benchmarking tools which can be very useful
to actually measure boot time:

  https://github.com/libguestfs/libguestfs/tree/master/utils

HTH,

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html