[Qemu-devel] SeaBIOS booting time optimization

[Qemu-devel] SeaBIOS booting time optimization

[Stefano Garzarella]

Hi,
I'm investigating the SeaBIOS booting time, to understand if we can reduce
the boot time in some cases (e.g. legacy hardware is not needed). I think
this
can be interesting also for NEMU developers.

Following this thread (
https://mail.coreboot.org/pipermail/seabios/2015-July/009497.html),
I'm using qboot (https://github.com/bonzini/qboot) to compare the SeaBIOS
booting time.

As Paolo did in qboot, I manually add small debug port writes in SeaBIOS and
linuxboot_dma.c (QEMU) to trace the events with perf (kvm_pio events)

The goal is to have only one image of SeaBIOS configurable at runtime to
reduce
the boot time, avoiding unnecessary initialization.

Any pointers or suggestions would be helpful.

Following I put some preliminary measurements that I obtained:
I used this QEMU command line:
./qemu-system-x86_64 -bios $BIOS -m 1G -cpu host -M accel=kvm \
-kernel /boot/vmlinuz-4.18.18-300.fc29.x86_64 -append 'console=ttyS0' \
-nographic -serial mon:stdio

For each test, I measured these times (in milliseconds) relative to the
"sched_process_exec" event:
- qemu_init_end: first kvm_entry (i.e. QEMU initialized has finished)
- fw_start: first entry of the BIOS
- fw_do_boot: after the BIOS initialization (e.g. PCI setup, etc.)
- linux_start_boot: before the jump to the Linux kernel

# qboot
BIOS=/home/stefano/repos/qboot/bios.bin
 qemu_init_end: 40.561234
 fw_start: 40.721729 (+0.160495)
 fw_do_boot: 47.025591 (+6.303862)
 linux_start_boot: 48.874112 (+1.848521)

# SeaBIOS with default configuration
BIOS=/home/stefano/repos/seabios/out_default/bios.bin
 qemu_init_end: 40.419451
 fw_start: 40.639967 (+0.220516)
 fw_do_boot: 886.668828 (+846.028861)
 linux_start_boot: 889.723547 (+3.054719)

# SeaBIOS with Kevin's configuration
# (https://mail.coreboot.org/pipermail/seabios/2015-July/009508.html)
# Note: this SeaBIOS setup is so stripped down that it can't actually boot
an OS
BIOS=/home/stefano/repos/seabios/out_kevin/bios.bin
 qemu_init_end: 40.676412
 fw_start: 40.755757 (+0.079345)
 fw_do_boot: 56.427023 (+15.671266)

I did the same tests also with NEMU (without using -M virt) and I have
approximately the same results.

As the next step, I'll start from Kevin's configuration to have a minimal
SeaBIOS image ables to boot a Linux kernel.

Thanks,
Stefano

-- 
Stefano Garzarella
Red Hat

Re: [Qemu-devel] SeaBIOS booting time optimization

[Stefan Hajnoczi]

On Fri, Nov 16, 2018 at 4:21 PM Stefano Garzarella <sgarzare@redhat.com> wrote:
> I'm investigating the SeaBIOS booting time, to understand if we can reduce
> the boot time in some cases (e.g. legacy hardware is not needed). I think
> this
> can be interesting also for NEMU developers.

Questions for NEMU folks:

qboot isn't part of the virt-x86 NEMU tree and is only mentioned in
one or two places.  Is NEMU focussing exclusively on OVMF guest
firmware instead of qboot?

Are there other boot time optimizations in NEMU that are a candidate
for upstreaming?

Stefan

Re: [Qemu-devel] SeaBIOS booting time optimization

[Samuel Ortiz]

Hi Stefan,

On Fri, Nov 16, 2018 at 04:39:20PM +0000, Stefan Hajnoczi wrote:
> On Fri, Nov 16, 2018 at 4:21 PM Stefano Garzarella <sgarzare@redhat.com> wrote:
> > I'm investigating the SeaBIOS booting time, to understand if we can reduce
> > the boot time in some cases (e.g. legacy hardware is not needed). I think
> > this
> > can be interesting also for NEMU developers.
> 
> Questions for NEMU folks:
> 
> qboot isn't part of the virt-x86 NEMU tree and is only mentioned in
> one or two places.  Is NEMU focussing exclusively on OVMF guest
> firmware instead of qboot?
Initally we were only focusing on OVMF, but we added support for qboot
[1] as we were not happy with the OVMF boot latencies we measured,
especially in the Kata containers context. Rob was about to start
looking at seabios support as well.

We are going to upstream a cleaner version of the virt support to qboot
very soon now.

> Are there other boot time optimizations in NEMU that are a candidate
> for upstreaming?

We did not add anything specific, but when comparing PC+qboot with
virt+qboot, we're seeing interesting overall boot time improvements. We
assume the simpler virt machine type explains the difference.

Cheers,
Samuel.

[1] https://github.com/rbradford/qboot/commit/dbcaaa1f6921064cd46f330a3625eb01e44aa4df

Re: [Qemu-devel] SeaBIOS booting time optimization

[Samuel Ortiz]

Hi Stefano,

On Fri, Nov 16, 2018 at 05:13:59PM +0100, Stefano Garzarella wrote:
> Hi,
> I'm investigating the SeaBIOS booting time, to understand if we can reduce
> the boot time in some cases (e.g. legacy hardware is not needed). I think
> this
> can be interesting also for NEMU developers.
Definitely, thanks.


> Following this thread (
> https://mail.coreboot.org/pipermail/seabios/2015-July/009497.html),
> I'm using qboot (https://github.com/bonzini/qboot) to compare the SeaBIOS
> booting time.
> 
> As Paolo did in qboot, I manually add small debug port writes in SeaBIOS and
> linuxboot_dma.c (QEMU) to trace the events with perf (kvm_pio events)
When doing similar measuremnts, I also used the debug-exit qemu device
in order to make imho simpler measuremnts, see
https://github.com/intel/nemu/wiki/Measuring-Boot-Latency

Probably not as precised as yours, but it was enough for what we tried
to characterize.

> The goal is to have only one image of SeaBIOS configurable at runtime to
> reduce
> the boot time, avoiding unnecessary initialization.
> 
> Any pointers or suggestions would be helpful.
> 
> Following I put some preliminary measurements that I obtained:
> I used this QEMU command line:
> ./qemu-system-x86_64 -bios $BIOS -m 1G -cpu host -M accel=kvm \
> -kernel /boot/vmlinuz-4.18.18-300.fc29.x86_64 -append 'console=ttyS0' \
> -nographic -serial mon:stdio
> 
> For each test, I measured these times (in milliseconds) relative to the
> "sched_process_exec" event:
> - qemu_init_end: first kvm_entry (i.e. QEMU initialized has finished)
> - fw_start: first entry of the BIOS
> - fw_do_boot: after the BIOS initialization (e.g. PCI setup, etc.)
> - linux_start_boot: before the jump to the Linux kernel
Are you planning to also measure the total time to userspace as well?
How we set the hardware up from the FW/BIOS can also influence the
overall (up to ring 3) boot latency on our experiments.

> # qboot
> BIOS=/home/stefano/repos/qboot/bios.bin
>  qemu_init_end: 40.561234
>  fw_start: 40.721729 (+0.160495)
>  fw_do_boot: 47.025591 (+6.303862)
>  linux_start_boot: 48.874112 (+1.848521)
> 
> # SeaBIOS with default configuration
> BIOS=/home/stefano/repos/seabios/out_default/bios.bin
>  qemu_init_end: 40.419451
>  fw_start: 40.639967 (+0.220516)
>  fw_do_boot: 886.668828 (+846.028861)
>  linux_start_boot: 889.723547 (+3.054719)
> 
> # SeaBIOS with Kevin's configuration
> # (https://mail.coreboot.org/pipermail/seabios/2015-July/009508.html)
> # Note: this SeaBIOS setup is so stripped down that it can't actually boot
> an OS
> BIOS=/home/stefano/repos/seabios/out_kevin/bios.bin
>  qemu_init_end: 40.676412
>  fw_start: 40.755757 (+0.079345)
>  fw_do_boot: 56.427023 (+15.671266)
That's a slight improvement ;)


> I did the same tests also with NEMU (without using -M virt) and I have
> approximately the same results.
Yes, as expected.


> As the next step, I'll start from Kevin's configuration to have a minimal
> SeaBIOS image ables to boot a Linux kernel.
Please keep us posted.

Cheers,
Samuel.

Re: [Qemu-devel] SeaBIOS booting time optimization

[Stefano Garzarella]

Hi Samuel,

On Mon, Nov 19, 2018 at 12:31 AM Samuel Ortiz <sameo@linux.intel.com> wrote:

> Hi Stefano,
>
> On Fri, Nov 16, 2018 at 05:13:59PM +0100, Stefano Garzarella wrote:
> > Hi,
> > I'm investigating the SeaBIOS booting time, to understand if we can
> reduce
> > the boot time in some cases (e.g. legacy hardware is not needed). I think
> > this
> > can be interesting also for NEMU developers.
> Definitely, thanks.

You are welcome!

>
>
> > Following this thread (
> > https://mail.coreboot.org/pipermail/seabios/2015-July/009497.html),
> > I'm using qboot (https://github.com/bonzini/qboot) to compare the
> SeaBIOS
> > booting time.
> >
> > As Paolo did in qboot, I manually add small debug port writes in SeaBIOS
> and
> > linuxboot_dma.c (QEMU) to trace the events with perf (kvm_pio events)
> When doing similar measuremnts, I also used the debug-exit qemu device
> in order to make imho simpler measuremnts, see
> https://github.com/intel/nemu/wiki/Measuring-Boot-Latency
>
> Probably not as precised as yours, but it was enough for what we tried
> to characterize.
>
> > The goal is to have only one image of SeaBIOS configurable at runtime to
> > reduce
> > the boot time, avoiding unnecessary initialization.
> >
> > Any pointers or suggestions would be helpful.
> >
> > Following I put some preliminary measurements that I obtained:
> > I used this QEMU command line:
> > ./qemu-system-x86_64 -bios $BIOS -m 1G -cpu host -M accel=kvm \
> > -kernel /boot/vmlinuz-4.18.18-300.fc29.x86_64 -append 'console=ttyS0' \
> > -nographic -serial mon:stdio
> >
> > For each test, I measured these times (in milliseconds) relative to the
> > "sched_process_exec" event:
> > - qemu_init_end: first kvm_entry (i.e. QEMU initialized has finished)
> > - fw_start: first entry of the BIOS
> > - fw_do_boot: after the BIOS initialization (e.g. PCI setup, etc.)
> > - linux_start_boot: before the jump to the Linux kernel
> Are you planning to also measure the total time to userspace as well?
> How we set the hardware up from the FW/BIOS can also influence the
> overall (up to ring 3) boot latency on our experiments.
>

I can add another debug port write in the kernel_init (as you mentioned
in https://github.com/intel/nemu/wiki/Measuring-Boot-Latency)


> > # qboot
> > BIOS=/home/stefano/repos/qboot/bios.bin
> >  qemu_init_end: 40.561234
> >  fw_start: 40.721729 (+0.160495)
> >  fw_do_boot: 47.025591 (+6.303862)
> >  linux_start_boot: 48.874112 (+1.848521)
> >
> > # SeaBIOS with default configuration
> > BIOS=/home/stefano/repos/seabios/out_default/bios.bin
> >  qemu_init_end: 40.419451
> >  fw_start: 40.639967 (+0.220516)
> >  fw_do_boot: 886.668828 (+846.028861)
> >  linux_start_boot: 889.723547 (+3.054719)
> >
> > # SeaBIOS with Kevin's configuration
> > # (https://mail.coreboot.org/pipermail/seabios/2015-July/009508.html)
> > # Note: this SeaBIOS setup is so stripped down that it can't actually
> boot
> > an OS
> > BIOS=/home/stefano/repos/seabios/out_kevin/bios.bin
> >  qemu_init_end: 40.676412
> >  fw_start: 40.755757 (+0.079345)
> >  fw_do_boot: 56.427023 (+15.671266)
> That's a slight improvement ;)
>
>
> > I did the same tests also with NEMU (without using -M virt) and I have
> > approximately the same results.
> Yes, as expected.
>
>
> > As the next step, I'll start from Kevin's configuration to have a minimal
> > SeaBIOS image ables to boot a Linux kernel.
> Please keep us posted.
>

Sure, I'll keep you updated with the progress.

Cheers,
Stefano

>
> Cheers,
> Samuel.
>


-- 
Stefano Garzarella
Red Hat

Re: [Qemu-devel] SeaBIOS booting time optimization

[Gerd Hoffmann]

  Hi,

> I'm investigating the SeaBIOS booting time, to understand if we can
> reduce the boot time in some cases (e.g. legacy hardware is not
> needed). I think this can be interesting also for NEMU developers.

> The goal is to have only one image of SeaBIOS configurable at runtime
> to reduce the boot time, avoiding unnecessary initialization.

Why at runtime?  What is bad with two images?  With a runtime option
you probably need some way to enable the "fastboot" hint for seabios,
because some optimizations (like skipping ps/2 initialization) breaks
functionality.  So it must be opt-in so you can enable it if you know
your use case is fine with that.  But "qemu -boot fast=on" isn't much
different from "qemu -bios seabios-fastboot.bin" after all ...

> Any pointers or suggestions would be helpful.

seabios could try to skip initializing hardware without bootindex with
"qemu -boot menu=off,strict=on".

mmconfig support?  I think that would reduce the number of vmexits
needed for pci initialization.

cheers,
  Gerd

Re: [Qemu-devel] SeaBIOS booting time optimization

[Stefano Garzarella]

On Mon, Nov 19, 2018 at 9:49 AM Gerd Hoffmann <kraxel@redhat.com> wrote:

>   Hi,
>
> > I'm investigating the SeaBIOS booting time, to understand if we can
> > reduce the boot time in some cases (e.g. legacy hardware is not
> > needed). I think this can be interesting also for NEMU developers.
>
> > The goal is to have only one image of SeaBIOS configurable at runtime
> > to reduce the boot time, avoiding unnecessary initialization.
>
> Why at runtime?  What is bad with two images?  With a runtime option
> you probably need some way to enable the "fastboot" hint for seabios,
> because some optimizations (like skipping ps/2 initialization) breaks
> functionality.  So it must be opt-in so you can enable it if you know
> your use case is fine with that.  But "qemu -boot fast=on" isn't much
> different from "qemu -bios seabios-fastboot.bin" after all ...
>

You are right, but maybe having a single image can be simpler to distribute,
and we can implement something (eg. using fw_cfg) to selectively enable
features needed.
Anyway, as the first step, I'll try to build a separate image of SeaBIOS.


> > Any pointers or suggestions would be helpful.
>
> seabios could try to skip initializing hardware without bootindex with
> "qemu -boot menu=off,strict=on".
>
> mmconfig support?  I think that would reduce the number of vmexits
> needed for pci initialization.
>

Thank you very much, I'll investigate that points.

Cheers,
Stefano


> cheers,
>   Gerd
>
>

-- 
Stefano Garzarella
Red Hat

Re: [Qemu-devel] SeaBIOS booting time optimization

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1847 bytes --]

On Mon, Nov 19, 2018 at 11:42:28AM +0100, Stefano Garzarella wrote:
> On Mon, Nov 19, 2018 at 9:49 AM Gerd Hoffmann <kraxel@redhat.com> wrote:
> 
> >   Hi,
> >
> > > I'm investigating the SeaBIOS booting time, to understand if we can
> > > reduce the boot time in some cases (e.g. legacy hardware is not
> > > needed). I think this can be interesting also for NEMU developers.
> >
> > > The goal is to have only one image of SeaBIOS configurable at runtime
> > > to reduce the boot time, avoiding unnecessary initialization.
> >
> > Why at runtime?  What is bad with two images?  With a runtime option
> > you probably need some way to enable the "fastboot" hint for seabios,
> > because some optimizations (like skipping ps/2 initialization) breaks
> > functionality.  So it must be opt-in so you can enable it if you know
> > your use case is fine with that.  But "qemu -boot fast=on" isn't much
> > different from "qemu -bios seabios-fastboot.bin" after all ...
> >
> 
> You are right, but maybe having a single image can be simpler to distribute,
> and we can implement something (eg. using fw_cfg) to selectively enable
> features needed.
> Anyway, as the first step, I'll try to build a separate image of SeaBIOS.

Gerd, you may be right that explicit an command-line option like -boot
fast=on is required.  I was hoping SeaBIOS improvements could
automatically detect cases where no functionality is lost and would not
require new command-line options for users or management tools (e.g.
libvirt).

For example, if SeaBIOS sees -kernel, is it necessary to follow the full
BIOS boot process including loading option ROMs?  This way we could
avoid scanning PCI devices.

Admittedly I don't know the answer to how transparent we can make this,
but I hope Stefano will identify how far we can go.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] SeaBIOS booting time optimization

[Gerd Hoffmann]

On Mon, Nov 19, 2018 at 01:07:13PM +0000, Stefan Hajnoczi wrote:
> On Mon, Nov 19, 2018 at 11:42:28AM +0100, Stefano Garzarella wrote:
> > On Mon, Nov 19, 2018 at 9:49 AM Gerd Hoffmann <kraxel@redhat.com> wrote:
> > 
> > > Why at runtime?  What is bad with two images?  With a runtime option
> > > you probably need some way to enable the "fastboot" hint for seabios,
> > > because some optimizations (like skipping ps/2 initialization) breaks
> > > functionality.  So it must be opt-in so you can enable it if you know
> > > your use case is fine with that.  But "qemu -boot fast=on" isn't much
> > > different from "qemu -bios seabios-fastboot.bin" after all ...
> > >
> > 
> > You are right, but maybe having a single image can be simpler to distribute,
> > and we can implement something (eg. using fw_cfg) to selectively enable
> > features needed.
> > Anyway, as the first step, I'll try to build a separate image of SeaBIOS.
> 
> Gerd, you may be right that explicit an command-line option like -boot
> fast=on is required.  I was hoping SeaBIOS improvements could
> automatically detect cases where no functionality is lost and would not
> require new command-line options for users or management tools (e.g.
> libvirt).

Will probably be possible to a certain degree, but I expect you can't
get 100% of the qboot savings without something like -boot fast=on.

> For example, if SeaBIOS sees -kernel, is it necessary to follow the full
> BIOS boot process including loading option ROMs?

Well, right now -kernel loading is actually implemented by an option rom ...

> This way we could avoid scanning PCI devices.

That has consequences for the acpi tables, because qemu checks where the
firmware mapped the pci bars when generating the tables.

Also it is not a given that skipping pci initialization in seabios is a
win in the end.  The linux kernel has to handle pci bar configuration
then so it might be we just shift around boot times.

> Admittedly I don't know the answer to how transparent we can make this,
> but I hope Stefano will identify how far we can go.

Yes, lets see.

cheers,
  Gerd

Re: [Qemu-devel] SeaBIOS booting time optimization

[Stefano Garzarella]

On Mon, Nov 19, 2018 at 3:15 PM Gerd Hoffmann <kraxel@redhat.com> wrote:

> On Mon, Nov 19, 2018 at 01:07:13PM +0000, Stefan Hajnoczi wrote:
> > On Mon, Nov 19, 2018 at 11:42:28AM +0100, Stefano Garzarella wrote:
> > > On Mon, Nov 19, 2018 at 9:49 AM Gerd Hoffmann <kraxel@redhat.com>
> wrote:
> > >
> > > > Why at runtime?  What is bad with two images?  With a runtime option
> > > > you probably need some way to enable the "fastboot" hint for seabios,
> > > > because some optimizations (like skipping ps/2 initialization) breaks
> > > > functionality.  So it must be opt-in so you can enable it if you know
> > > > your use case is fine with that.  But "qemu -boot fast=on" isn't much
> > > > different from "qemu -bios seabios-fastboot.bin" after all ...
> > > >
> > >
> > > You are right, but maybe having a single image can be simpler to
> distribute,
> > > and we can implement something (eg. using fw_cfg) to selectively enable
> > > features needed.
> > > Anyway, as the first step, I'll try to build a separate image of
> SeaBIOS.
> >
> > Gerd, you may be right that explicit an command-line option like -boot
> > fast=on is required.  I was hoping SeaBIOS improvements could
> > automatically detect cases where no functionality is lost and would not
> > require new command-line options for users or management tools (e.g.
> > libvirt).
>
> Will probably be possible to a certain degree, but I expect you can't
> get 100% of the qboot savings without something like -boot fast=on.
>
> > For example, if SeaBIOS sees -kernel, is it necessary to follow the full
> > BIOS boot process including loading option ROMs?
>
> Well, right now -kernel loading is actually implemented by an option rom
> ...
>
> > This way we could avoid scanning PCI devices.
>
> That has consequences for the acpi tables, because qemu checks where the
> firmware mapped the pci bars when generating the tables.
>
> Also it is not a given that skipping pci initialization in seabios is a
> win in the end.  The linux kernel has to handle pci bar configuration
> then so it might be we just shift around boot times.
>
> > Admittedly I don't know the answer to how transparent we can make this,
> > but I hope Stefano will identify how far we can go.
>
> Yes, lets see.
>

Hi guys,
just an update, I enabled the debug prints and I saw two timeouts fired
with a lot
of time lost (~780ms between "init timer" and "Scan for VGA ..."),
putting other prints I discovered that a lot of time is spent in the
tpm_setup(),
during the probe of the 2 TPM devices:

00.548869 init timer
00.549677 ./src/post.c:157 platform_hardware_setup
00.550182 ./src/hw/tpm_drivers.c:579 tpmhw_probe
01.300833 WARNING - Timeout at wait_reg8:81!
01.301388 ./src/hw/tpm_drivers.c:579 tpmhw_probe
01.331843 WARNING - Timeout at wait_reg8:81!
01.332316 ./src/post.c:160 platform_hardware_setup
01.333358 Scan for VGA option rom

Indeed, in the probe of the TPM devices (TIS and CRB)  there are timeouts of
750 ms and 30 ms respectively.

Then, statically disabling TPM and TCG (CONFIG_TCGBIOS) the time spent
in SeaBIOS goes down from ~846ms to ~56ms:
# SeaBIOS disabling CONFIG_TCGBIOS
BIOS=/home/stefano/repos/seabios/out_notcgbios/bios.bin
 qemu_init_end: 40.658371
 fw_start: 40.850395 (+0.192024)
 fw_do_boot: 96.750142 (+55.899747)
 linux_start_boot: 98.880578 (+2.130436)

The tpm_setup() is called after the qemu_cfg_init(), so I think we can
disable
this call using some fw_cfg parameters, if we will decide to implement a
runtime
approach.

Cheers,
Stefano


> cheers,
>   Gerd
>
>

-- 
Stefano Garzarella
Red Hat

Re: [Qemu-devel] SeaBIOS booting time optimization

[Gerd Hoffmann]

  Hi,

> just an update, I enabled the debug prints and I saw two timeouts fired
> with a lot
> of time lost (~780ms between "init timer" and "Scan for VGA ..."),
> putting other prints I discovered that a lot of time is spent in the
> tpm_setup(),
> during the probe of the 2 TPM devices:
> 
> 00.548869 init timer
> 00.549677 ./src/post.c:157 platform_hardware_setup
> 00.550182 ./src/hw/tpm_drivers.c:579 tpmhw_probe
> 01.300833 WARNING - Timeout at wait_reg8:81!
> 01.301388 ./src/hw/tpm_drivers.c:579 tpmhw_probe
> 01.331843 WARNING - Timeout at wait_reg8:81!
> 01.332316 ./src/post.c:160 platform_hardware_setup
> 01.333358 Scan for VGA option rom
> 
> Indeed, in the probe of the TPM devices (TIS and CRB)  there are timeouts of
> 750 ms and 30 ms respectively.
> 
> Then, statically disabling TPM and TCG (CONFIG_TCGBIOS) the time spent
> in SeaBIOS goes down from ~846ms to ~56ms:
> # SeaBIOS disabling CONFIG_TCGBIOS
> BIOS=/home/stefano/repos/seabios/out_notcgbios/bios.bin
>  qemu_init_end: 40.658371
>  fw_start: 40.850395 (+0.192024)
>  fw_do_boot: 96.750142 (+55.899747)
>  linux_start_boot: 98.880578 (+2.130436)
> 
> The tpm_setup() is called after the qemu_cfg_init(), so I think we can
> disable
> this call using some fw_cfg parameters, if we will decide to implement a
> runtime
> approach.

Another possible option is to simply use shorter timeouts on qemu (using
runningOnQEMU()), I'd suggest to check with the TPM maintainers whenever
that is ok.

Background:  The drivers in seabios work on both virtual and physical
hardware (you can run seabios as coreboot payload on physical hardware).
There are quite a few places where are delays and timeouts which are
required to work properly on physical hardware.  But typically virtual
hardware is alot faster.  There is -- for example -- no time needed to
establish a sata link.  Link detection in ahci is instant on qemu.

cheers,
  Gerd

Re: [Qemu-devel] SeaBIOS booting time optimization

[Stefano Garzarella]

Hi Gerd,

On Tue, Nov 20, 2018 at 7:21 AM Gerd Hoffmann <kraxel@redhat.com> wrote:
>
>   Hi,
>
> > just an update, I enabled the debug prints and I saw two timeouts fired
> > with a lot
> > of time lost (~780ms between "init timer" and "Scan for VGA ..."),
> > putting other prints I discovered that a lot of time is spent in the
> > tpm_setup(),
> > during the probe of the 2 TPM devices:
> >
> > 00.548869 init timer
> > 00.549677 ./src/post.c:157 platform_hardware_setup
> > 00.550182 ./src/hw/tpm_drivers.c:579 tpmhw_probe
> > 01.300833 WARNING - Timeout at wait_reg8:81!
> > 01.301388 ./src/hw/tpm_drivers.c:579 tpmhw_probe
> > 01.331843 WARNING - Timeout at wait_reg8:81!
> > 01.332316 ./src/post.c:160 platform_hardware_setup
> > 01.333358 Scan for VGA option rom
> >
> > Indeed, in the probe of the TPM devices (TIS and CRB)  there are timeouts of
> > 750 ms and 30 ms respectively.
> >
> > Then, statically disabling TPM and TCG (CONFIG_TCGBIOS) the time spent
> > in SeaBIOS goes down from ~846ms to ~56ms:
> > # SeaBIOS disabling CONFIG_TCGBIOS
> > BIOS=/home/stefano/repos/seabios/out_notcgbios/bios.bin
> >  qemu_init_end: 40.658371
> >  fw_start: 40.850395 (+0.192024)
> >  fw_do_boot: 96.750142 (+55.899747)
> >  linux_start_boot: 98.880578 (+2.130436)
> >
> > The tpm_setup() is called after the qemu_cfg_init(), so I think we can
> > disable
> > this call using some fw_cfg parameters, if we will decide to implement a
> > runtime
> > approach.
>
> Another possible option is to simply use shorter timeouts on qemu (using
> runningOnQEMU()), I'd suggest to check with the TPM maintainers whenever
> that is ok.

Yes, it is a good idea!

>
> Background:  The drivers in seabios work on both virtual and physical
> hardware (you can run seabios as coreboot payload on physical hardware).
> There are quite a few places where are delays and timeouts which are
> required to work properly on physical hardware.  But typically virtual
> hardware is alot faster.  There is -- for example -- no time needed to
> establish a sata link.  Link detection in ahci is instant on qemu.

I'll investigate also other timeouts.

Thanks,
Stefano

>
> cheers,
>   Gerd
>


--
Stefano Garzarella
Red Hat

Re: [Qemu-devel] SeaBIOS booting time optimization

[Gerd Hoffmann]

  Hi,

> > Background:  The drivers in seabios work on both virtual and physical
> > hardware (you can run seabios as coreboot payload on physical hardware).
> > There are quite a few places where are delays and timeouts which are
> > required to work properly on physical hardware.  But typically virtual
> > hardware is alot faster.  There is -- for example -- no time needed to
> > establish a sata link.  Link detection in ahci is instant on qemu.
> 
> I'll investigate also other timeouts.

I'd focus on legacy lpc/isa hardware here.  PCI-ish hardware which has
such timeouts (sata, usb, ...) typically isn't present in the virtual
machine if you care about performance.  Or, in other words, the best way
to handle the sata link detection delay is to use "qemu -M q35,sata=off".

cheers,
  Gerd

Re: [Qemu-devel] SeaBIOS booting time optimization

[Stefano Garzarella]

Hi,
I continued to investigate how to reduce the boot time with SeaBIOS
and QEMU when it used with linuxboot_dma.bin (-kernel parameter).
I reached ~12ms with a SeaBIOS configuration (attached) where I
disabled debug output, all Hardware support (except SMM & MTRRs) and I
applied a small patch to disable VGA setup and console (attached).

Samuel, I put also the total time to userspace (linux_start_user)
adding a probe in the kernel_init() and when I disabled the VGA in the
SeaBIOS I noticed also a speed up in the kernel boot phase.

Another approach that I tried, obtaining similar results, is to
recognize the linuxboot_dma.bin at runtime, then skip
device_hardware_setup(), enable_vga_console(), and vgarom_setup(). In
this case, I used the default configuration without disabling
anything.

Gerd, Kevin do you think these approaches can be acceptable in SeaBIOS?

Thanks,
Stefano

Booting times with several configurations:
 - SeaBIOS default config + Stephen's TPM patch
   qemu_init_end: 40.342836
   fw_start: 40.528355 (+0.185519)
   fw_do_boot: 108.679648 (+68.151293)
   linux_start_boot: 110.961394 (+2.281746)
   linux_start_user: 647.981995 (+537.020601)
- SeaBIOS config (CONFIG_DEBUG_LEVEL=0) + Stephen's TPM patch
   qemu_init_end: 44.533900
   fw_start: 44.763200 (+0.229300)
   fw_do_boot: 98.881728 (+54.118528)
   linux_start_boot: 100.925424 (+2.043696)
   linux_start_user: 638.000481 (+537.075057)
- SeaBIOS config (CONFIG_DEBUG_LEVEL=0, disable all HW support except
SMM & MTRRs) + Stephen's TPM patch
   qemu_init_end: 42.705257
   fw_start: 42.915464 (+0.210207)
   fw_do_boot: 91.982749 (+49.067285)
   linux_start_boot: 93.970654 (+1.987905)
   linux_start_user: 632.916320 (+538.945666)
- SeaBIOS config (CONFIG_DEBUG_LEVEL=0, disable all HW support except
SMM & MTRRs, CONFIG_DISABLE_VGA=y) + Stephen's TPM patch
   qemu_init_end: 42.968648
   fw_start: 43.154598 (+0.185950)
   fw_do_boot: 55.808438 (+12.653840)
   linux_start_boot: 57.170154 (+1.361716)
   linux_start_user: 529.189856 (+472.019702)


Disable VGA patch:
From: Stefano Garzarella <sgarzare@redhat.com>
Date: Thu, 22 Nov 2018 10:55:23 +0100
Subject: [PATCH] qemu: Add CONFIG_DISABLE_VGA to disable VGA setup and console

Allow users to disable VGA setup and console. Useful to speed
up the boot with QEMU when -kernel parameter is used. Linux
kernel will also initialize the VGA during the boot phase.

Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
---
 src/Kconfig      | 7 +++++++
 src/bootsplash.c | 2 ++
 src/optionroms.c | 2 +-
 3 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/src/Kconfig b/src/Kconfig
index daf8222..b1f0ce6 100644
--- a/src/Kconfig
+++ b/src/Kconfig
@@ -364,6 +364,13 @@ menu "Hardware support"
         help
             Support for using the CPU timestamp counter as an internal
             timing source.
+    config DISABLE_VGA
+        depends on QEMU
+        bool "Disable VGA setup"
+        default n
+        help
+            Skip VGA setup and console (useful to speed up the boot with QEMU
+            when -kernel parameter is used).
 endmenu

 menu "BIOS interfaces"
diff --git a/src/bootsplash.c b/src/bootsplash.c
index 165c98d..dde3314 100644
--- a/src/bootsplash.c
+++ b/src/bootsplash.c
@@ -39,6 +39,8 @@ call16_int10(struct bregs *br)
 void
 enable_vga_console(void)
 {
+    if (CONFIG_DISABLE_VGA)
+        return;
     dprintf(1, "Turning on vga text mode console\n");
     struct bregs br;

diff --git a/src/optionroms.c b/src/optionroms.c
index fc992f6..a3b5a16 100644
--- a/src/optionroms.c
+++ b/src/optionroms.c
@@ -428,7 +428,7 @@ vgarom_setup(void)
 {
     int have_vga = 0;

-    if (! CONFIG_OPTIONROMS)
+    if (!CONFIG_OPTIONROMS || CONFIG_DISABLE_VGA)
         return;

     dprintf(1, "Scan for VGA option rom\n");
--
2.19.1


SeaBIOS final configuration:
#
# Automatically generated file; DO NOT EDIT.
# SeaBIOS Configuration
#

#
# General Features
#
# CONFIG_COREBOOT is not set
CONFIG_QEMU=y
# CONFIG_CSM is not set
CONFIG_QEMU_HARDWARE=y
CONFIG_XEN=y
CONFIG_THREADS=y
CONFIG_RELOCATE_INIT=y
CONFIG_BOOTMENU=y
CONFIG_BOOTSPLASH=y
CONFIG_BOOTORDER=y
CONFIG_ENTRY_EXTRASTACK=y
CONFIG_MALLOC_UPPERMEMORY=y
CONFIG_ROM_SIZE=0

#
# Hardware support
#
# CONFIG_ATA is not set
# CONFIG_AHCI is not set
# CONFIG_SDCARD is not set
# CONFIG_VIRTIO_BLK is not set
# CONFIG_VIRTIO_SCSI is not set
# CONFIG_PVSCSI is not set
# CONFIG_ESP_SCSI is not set
# CONFIG_LSI_SCSI is not set
# CONFIG_MEGASAS is not set
# CONFIG_MPT_SCSI is not set
# CONFIG_FLASH_FLOPPY is not set
# CONFIG_NVME is not set
# CONFIG_PS2PORT is not set
# CONFIG_USB is not set
# CONFIG_SERIAL is not set
# CONFIG_SERCON is not set
# CONFIG_LPT is not set
# CONFIG_HARDWARE_IRQ is not set
CONFIG_USE_SMM=y
CONFIG_CALL32_SMM=y
CONFIG_MTRR_INIT=y
# CONFIG_PMTIMER is not set
# CONFIG_TSC_TIMER is not set
CONFIG_DISABLE_VGA=y

#
# BIOS interfaces
#
CONFIG_DRIVES=y
CONFIG_CDROM_BOOT=y
CONFIG_CDROM_EMU=y
CONFIG_PCIBIOS=y
CONFIG_APMBIOS=y
CONFIG_PNPBIOS=y
CONFIG_OPTIONROMS=y
CONFIG_PMM=y
CONFIG_BOOT=y
CONFIG_KEYBOARD=y
CONFIG_KBD_CALL_INT15_4F=y
CONFIG_MOUSE=y
CONFIG_S3_RESUME=y
CONFIG_VGAHOOKS=y
# CONFIG_DISABLE_A20 is not set
# CONFIG_WRITABLE_UPPERMEMORY is not set
CONFIG_TCGBIOS=y

#
# BIOS Tables
#
CONFIG_PIRTABLE=y
CONFIG_MPTABLE=y
CONFIG_SMBIOS=y
CONFIG_ACPI=y
CONFIG_ACPI_DSDT=y
CONFIG_FW_ROMFILE_LOAD=y

#
# VGA ROM
#
CONFIG_NO_VGABIOS=y
# CONFIG_VGA_STANDARD_VGA is not set
# CONFIG_VGA_CIRRUS is not set
# CONFIG_VGA_BOCHS is not set
# CONFIG_VGA_GEODEGX2 is not set
# CONFIG_VGA_GEODELX is not set
# CONFIG_DISPLAY_BOCHS is not set
# CONFIG_VGA_RAMFB is not set
# CONFIG_BUILD_VGABIOS is not set
CONFIG_VGA_EXTRA_STACK_SIZE=512

#
# Debugging
#
CONFIG_DEBUG_LEVEL=0

On Tue, Nov 20, 2018 at 12:22 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
>
>   Hi,
>
> > > Background:  The drivers in seabios work on both virtual and physical
> > > hardware (you can run seabios as coreboot payload on physical hardware).
> > > There are quite a few places where are delays and timeouts which are
> > > required to work properly on physical hardware.  But typically virtual
> > > hardware is alot faster.  There is -- for example -- no time needed to
> > > establish a sata link.  Link detection in ahci is instant on qemu.
> >
> > I'll investigate also other timeouts.
>
> I'd focus on legacy lpc/isa hardware here.  PCI-ish hardware which has
> such timeouts (sata, usb, ...) typically isn't present in the virtual
> machine if you care about performance.  Or, in other words, the best way
> to handle the sata link detection delay is to use "qemu -M q35,sata=off".
>
> cheers,
>   Gerd
>


--
Stefano Garzarella
Red Hat

Re: [Qemu-devel] SeaBIOS booting time optimization

[Gerd Hoffmann]

On Thu, Nov 22, 2018 at 12:08:55PM +0100, Stefano Garzarella wrote:
> Hi,
> I continued to investigate how to reduce the boot time with SeaBIOS
> and QEMU when it used with linuxboot_dma.bin (-kernel parameter).
> I reached ~12ms with a SeaBIOS configuration (attached) where I
> disabled debug output, all Hardware support (except SMM & MTRRs) and I
> applied a small patch to disable VGA setup and console (attached).

Is there any difference to "qemu -vga none" ?

> Samuel, I put also the total time to userspace (linux_start_user)
> adding a probe in the kernel_init() and when I disabled the VGA in the
> SeaBIOS I noticed also a speed up in the kernel boot phase.

Might come from linux 16bit boot code scanning vgabios for available
video modes (for vga=<mode> cmdline arg).

> Another approach that I tried, obtaining similar results, is to
> recognize the linuxboot_dma.bin at runtime, then skip
> device_hardware_setup(), enable_vga_console(), and vgarom_setup().

Hmm, tricky.

linuxboot_dma.bin just registers a boot entry, so with "qemu -kernel
-boot menu=on" you can still get a boot menu, and the direkt kernel boot
is one menu item there.  So you can't know for sure whenever a direct
kernel boot will happen or not.

So, I think at minimum we should make this depend on both linuxboot_dma
and boot menu disabled.

cheers,
  Gerd

Re: [Qemu-devel] SeaBIOS booting time optimization

[Stefano Garzarella]

On Thu, Nov 22, 2018 at 12:51 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
>
> On Thu, Nov 22, 2018 at 12:08:55PM +0100, Stefano Garzarella wrote:
> > Hi,
> > I continued to investigate how to reduce the boot time with SeaBIOS
> > and QEMU when it used with linuxboot_dma.bin (-kernel parameter).
> > I reached ~12ms with a SeaBIOS configuration (attached) where I
> > disabled debug output, all Hardware support (except SMM & MTRRs) and I
> > applied a small patch to disable VGA setup and console (attached).
>
> Is there any difference to "qemu -vga none" ?

Using both (qemu -vga none, and my patch) we are around 10.8 ms.
Note: using only the patch, Linux is still able to initialize and use the VGA.

- QEMU -vga none + SeaBIOS config (CONFIG_DEBUG_LEVEL=0, disable all
HW support except
SMM & MTRRs) + Stephen's TPM patch
 qemu_init_end: 43.675803
 fw_start: 43.865178 (+0.189375)
 fw_do_boot: 58.093161 (+14.227983)
 linux_start_boot: 59.490308 (+1.397147)
 linux_start_user: 556.782354 (+497.292046)

- QEMU -vga none + SeaBIOS config (CONFIG_DEBUG_LEVEL=0, disable all
HW support except
SMM & MTRRs, CONFIG_DISABLE_VGA=y) + Stephen's TPM patch
 qemu_init_end: 42.387412
 fw_start: 42.579257 (+0.191845)
 fw_do_boot: 53.381517 (+10.802260)
 linux_start_boot: 54.848643 (+1.467126)
 linux_start_user: 498.517050 (+443.668407)

>
> > Samuel, I put also the total time to userspace (linux_start_user)
> > adding a probe in the kernel_init() and when I disabled the VGA in the
> > SeaBIOS I noticed also a speed up in the kernel boot phase.
>
> Might come from linux 16bit boot code scanning vgabios for available
> video modes (for vga=<mode> cmdline arg).
>
> > Another approach that I tried, obtaining similar results, is to
> > recognize the linuxboot_dma.bin at runtime, then skip
> > device_hardware_setup(), enable_vga_console(), and vgarom_setup().
>
> Hmm, tricky.
>"
> linuxboot_dma.bin just registers a boot entry, so with "qemu -kernel
> -boot menu=on" you can still get a boot menu, and the direkt kernel boot
> is one menu item there.  So you can't know for sure whenever a direct
> kernel boot will happen or not.
>
> So, I think at minimum we should make this depend on both linuxboot_dma
> and boot menu disabled.

You are right! I'll prepare a patch adding the check of boot menu.

Thanks,
Stefano

>
> cheers,
>   Gerd
>


--
Stefano Garzarella
Red Hat

Re: [Qemu-devel] SeaBIOS booting time optimization

[Gerd Hoffmann]

On Thu, Nov 22, 2018 at 04:13:38PM +0100, Stefano Garzarella wrote:
> On Thu, Nov 22, 2018 at 12:51 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
> >
> > On Thu, Nov 22, 2018 at 12:08:55PM +0100, Stefano Garzarella wrote:
> > > Hi,
> > > I continued to investigate how to reduce the boot time with SeaBIOS
> > > and QEMU when it used with linuxboot_dma.bin (-kernel parameter).
> > > I reached ~12ms with a SeaBIOS configuration (attached) where I
> > > disabled debug output, all Hardware support (except SMM & MTRRs) and I
> > > applied a small patch to disable VGA setup and console (attached).
> >
> > Is there any difference to "qemu -vga none" ?
> 
> Using both (qemu -vga none, and my patch) we are around 10.8 ms.
> Note: using only the patch, Linux is still able to initialize and use the VGA.

But do you want linux use the vga console if you care about boot times?
I'd expect virtio-console would be fastest in that case.

> - QEMU -vga none + SeaBIOS config (CONFIG_DEBUG_LEVEL=0, disable all
> HW support except
> SMM & MTRRs) + Stephen's TPM patch
>  qemu_init_end: 43.675803
>  fw_start: 43.865178 (+0.189375)
>  fw_do_boot: 58.093161 (+14.227983)
>  linux_start_boot: 59.490308 (+1.397147)
>  linux_start_user: 556.782354 (+497.292046)
> 
> - QEMU -vga none + SeaBIOS config (CONFIG_DEBUG_LEVEL=0, disable all
> HW support except
> SMM & MTRRs, CONFIG_DISABLE_VGA=y) + Stephen's TPM patch
>  qemu_init_end: 42.387412
>  fw_start: 42.579257 (+0.191845)
>  fw_do_boot: 53.381517 (+10.802260)
>  linux_start_boot: 54.848643 (+1.467126)
>  linux_start_user: 498.517050 (+443.668407)

Interesting that CONFIG_DISABLE_VGA=y makes a noticable difference even
without vga hardware being preset.  And not only in seabios but also for
the linux kernel.

Do you know why?

cheers,
  Gerd

Re: [Qemu-devel] SeaBIOS booting time optimization

[Stefano Garzarella]

On Fri, Nov 23, 2018 at 7:21 AM Gerd Hoffmann <kraxel@redhat.com> wrote:
>
> On Thu, Nov 22, 2018 at 04:13:38PM +0100, Stefano Garzarella wrote:
> > On Thu, Nov 22, 2018 at 12:51 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
> > >
> > > On Thu, Nov 22, 2018 at 12:08:55PM +0100, Stefano Garzarella wrote:
> > > > Hi,
> > > > I continued to investigate how to reduce the boot time with SeaBIOS
> > > > and QEMU when it used with linuxboot_dma.bin (-kernel parameter).
> > > > I reached ~12ms with a SeaBIOS configuration (attached) where I
> > > > disabled debug output, all Hardware support (except SMM & MTRRs) and I
> > > > applied a small patch to disable VGA setup and console (attached).
> > >
> > > Is there any difference to "qemu -vga none" ?
> >
> > Using both (qemu -vga none, and my patch) we are around 10.8 ms.
> > Note: using only the patch, Linux is still able to initialize and use the VGA.
>
> But do you want linux use the vga console if you care about boot times?
> I'd expect virtio-console would be fastest in that case.

I agree with you, but if we will implement a QEMU fastboot feature in
SeaBIOS, we are sure that it works with Linux kernel if someone uses
"qemu -kernel" and wants also use a VGA.

>
> > - QEMU -vga none + SeaBIOS config (CONFIG_DEBUG_LEVEL=0, disable all
> > HW support except
> > SMM & MTRRs) + Stephen's TPM patch
> >  qemu_init_end: 43.675803
> >  fw_start: 43.865178 (+0.189375)
> >  fw_do_boot: 58.093161 (+14.227983)
> >  linux_start_boot: 59.490308 (+1.397147)
> >  linux_start_user: 556.782354 (+497.292046)
> >
> > - QEMU -vga none + SeaBIOS config (CONFIG_DEBUG_LEVEL=0, disable all
> > HW support except
> > SMM & MTRRs, CONFIG_DISABLE_VGA=y) + Stephen's TPM patch
> >  qemu_init_end: 42.387412
> >  fw_start: 42.579257 (+0.191845)
> >  fw_do_boot: 53.381517 (+10.802260)
> >  linux_start_boot: 54.848643 (+1.467126)
> >  linux_start_user: 498.517050 (+443.668407)
>
> Interesting that CONFIG_DISABLE_VGA=y makes a noticable difference even
> without vga hardware being preset.  And not only in seabios but also for
> the linux kernel.
>
> Do you know why?

In SeaBIOS I think the reasons are:
- vgarom_setup() scan all PCI devices to find a VGA
- enable_vga_console() invokes an int10 without check if there is a
VGA (maybe here we can add a check)

Cheers,
Stefano

--
Stefano Garzarella
Red Hat

Re: [Qemu-devel] SeaBIOS booting time optimization

[Kevin O'Connor]

On Fri, Nov 23, 2018 at 12:18:13PM +0100, Stefano Garzarella wrote:
> On Fri, Nov 23, 2018 at 7:21 AM Gerd Hoffmann <kraxel@redhat.com> wrote:
> > On Thu, Nov 22, 2018 at 04:13:38PM +0100, Stefano Garzarella wrote:
> > > On Thu, Nov 22, 2018 at 12:51 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
> > > > On Thu, Nov 22, 2018 at 12:08:55PM +0100, Stefano Garzarella wrote:
> > > > > Hi,
> > > > > I continued to investigate how to reduce the boot time with SeaBIOS
> > > > > and QEMU when it used with linuxboot_dma.bin (-kernel parameter).
> > > > > I reached ~12ms with a SeaBIOS configuration (attached) where I
> > > > > disabled debug output, all Hardware support (except SMM & MTRRs) and I
> > > > > applied a small patch to disable VGA setup and console (attached).
> > > >
> > > > Is there any difference to "qemu -vga none" ?
> > >
> > > Using both (qemu -vga none, and my patch) we are around 10.8 ms.
> > > Note: using only the patch, Linux is still able to initialize and use the VGA.
> >
> > But do you want linux use the vga console if you care about boot times?
> > I'd expect virtio-console would be fastest in that case.
> 
> I agree with you, but if we will implement a QEMU fastboot feature in
> SeaBIOS, we are sure that it works with Linux kernel if someone uses
> "qemu -kernel" and wants also use a VGA.

It should be possible to use "-device VGA,romfile=" to enable VGA
without an optionrom.

> > > - QEMU -vga none + SeaBIOS config (CONFIG_DEBUG_LEVEL=0, disable all
> > > HW support except
> > > SMM & MTRRs) + Stephen's TPM patch
> > >  qemu_init_end: 43.675803
> > >  fw_start: 43.865178 (+0.189375)
> > >  fw_do_boot: 58.093161 (+14.227983)
> > >  linux_start_boot: 59.490308 (+1.397147)
> > >  linux_start_user: 556.782354 (+497.292046)
> > >
> > > - QEMU -vga none + SeaBIOS config (CONFIG_DEBUG_LEVEL=0, disable all
> > > HW support except
> > > SMM & MTRRs, CONFIG_DISABLE_VGA=y) + Stephen's TPM patch
> > >  qemu_init_end: 42.387412
> > >  fw_start: 42.579257 (+0.191845)
> > >  fw_do_boot: 53.381517 (+10.802260)
> > >  linux_start_boot: 54.848643 (+1.467126)
> > >  linux_start_user: 498.517050 (+443.668407)
> >
> > Interesting that CONFIG_DISABLE_VGA=y makes a noticable difference even
> > without vga hardware being preset.  And not only in seabios but also for
> > the linux kernel.
> >
> > Do you know why?
> 
> In SeaBIOS I think the reasons are:
> - vgarom_setup() scan all PCI devices to find a VGA
> - enable_vga_console() invokes an int10 without check if there is a
> VGA (maybe here we can add a check)

Where is CONFIG_DISABLE_VGA set - it's not a standard SeaBIOS config
option?

In my experience, the only meaningful delays are the result of
accessing hardware.  Neither of the above two items would result in
additional hardware accesses, so I don't think they would be the cause
of an additional delay.

-Kevin

Re: [Qemu-devel] SeaBIOS booting time optimization

[Stefano Garzarella]

On Fri, Nov 23, 2018 at 4:54 PM Kevin O'Connor <kevin@koconnor.net> wrote:
>
> On Fri, Nov 23, 2018 at 12:18:13PM +0100, Stefano Garzarella wrote:
> > On Fri, Nov 23, 2018 at 7:21 AM Gerd Hoffmann <kraxel@redhat.com> wrote:
> > > On Thu, Nov 22, 2018 at 04:13:38PM +0100, Stefano Garzarella wrote:
> > > > On Thu, Nov 22, 2018 at 12:51 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
> > > > > On Thu, Nov 22, 2018 at 12:08:55PM +0100, Stefano Garzarella wrote:
> > > > > > Hi,
> > > > > > I continued to investigate how to reduce the boot time with SeaBIOS
> > > > > > and QEMU when it used with linuxboot_dma.bin (-kernel parameter).
> > > > > > I reached ~12ms with a SeaBIOS configuration (attached) where I
> > > > > > disabled debug output, all Hardware support (except SMM & MTRRs) and I
> > > > > > applied a small patch to disable VGA setup and console (attached).
> > > > >
> > > > > Is there any difference to "qemu -vga none" ?
> > > >
> > > > Using both (qemu -vga none, and my patch) we are around 10.8 ms.
> > > > Note: using only the patch, Linux is still able to initialize and use the VGA.
> > >
> > > But do you want linux use the vga console if you care about boot times?
> > > I'd expect virtio-console would be fastest in that case.
> >
> > I agree with you, but if we will implement a QEMU fastboot feature in
> > SeaBIOS, we are sure that it works with Linux kernel if someone uses
> > "qemu -kernel" and wants also use a VGA.
>
> It should be possible to use "-device VGA,romfile=" to enable VGA
> without an optionrom.
>
> > > > - QEMU -vga none + SeaBIOS config (CONFIG_DEBUG_LEVEL=0, disable all
> > > > HW support except
> > > > SMM & MTRRs) + Stephen's TPM patch
> > > >  qemu_init_end: 43.675803
> > > >  fw_start: 43.865178 (+0.189375)
> > > >  fw_do_boot: 58.093161 (+14.227983)
> > > >  linux_start_boot: 59.490308 (+1.397147)
> > > >  linux_start_user: 556.782354 (+497.292046)
> > > >
> > > > - QEMU -vga none + SeaBIOS config (CONFIG_DEBUG_LEVEL=0, disable all
> > > > HW support except
> > > > SMM & MTRRs, CONFIG_DISABLE_VGA=y) + Stephen's TPM patch
> > > >  qemu_init_end: 42.387412
> > > >  fw_start: 42.579257 (+0.191845)
> > > >  fw_do_boot: 53.381517 (+10.802260)
> > > >  linux_start_boot: 54.848643 (+1.467126)
> > > >  linux_start_user: 498.517050 (+443.668407)
> > >
> > > Interesting that CONFIG_DISABLE_VGA=y makes a noticable difference even
> > > without vga hardware being preset.  And not only in seabios but also for
> > > the linux kernel.
> > >
> > > Do you know why?
> >
> > In SeaBIOS I think the reasons are:
> > - vgarom_setup() scan all PCI devices to find a VGA
> > - enable_vga_console() invokes an int10 without check if there is a
> > VGA (maybe here we can add a check)
>
> Where is CONFIG_DISABLE_VGA set - it's not a standard SeaBIOS config
> option?

Yes, it is not standard.
I posted a small patch in this thread
(https://mail.coreboot.org/pipermail/seabios/2018-November/012592.html)
to explain my experiments.
CONFIG_DISABLE_VGA=y simply skips the vgarom_setup() and enable_vga_console().

>
> In my experience, the only meaningful delays are the result of
> accessing hardware.  Neither of the above two items would result in
> additional hardware accesses, so I don't think they would be the cause
> of an additional delay.

Thanks, I'll investigate better the delays.

Cheers,
Stefano

>
> -Kevin



-- 
Stefano Garzarella
Red Hat

Re: [Qemu-devel] [SeaBIOS] SeaBIOS booting time optimization

[Kevin O'Connor]

On Mon, Nov 19, 2018 at 07:38:39PM +0100, Stefano Garzarella wrote:
> just an update, I enabled the debug prints and I saw two timeouts fired
> with a lot
> of time lost (~780ms between "init timer" and "Scan for VGA ..."),
> putting other prints I discovered that a lot of time is spent in the
> tpm_setup(),
> during the probe of the 2 TPM devices:
> 
> 00.548869 init timer
> 00.549677 ./src/post.c:157 platform_hardware_setup
> 00.550182 ./src/hw/tpm_drivers.c:579 tpmhw_probe
> 01.300833 WARNING - Timeout at wait_reg8:81!
> 01.301388 ./src/hw/tpm_drivers.c:579 tpmhw_probe
> 01.331843 WARNING - Timeout at wait_reg8:81!
> 01.332316 ./src/post.c:160 platform_hardware_setup
> 01.333358 Scan for VGA option rom

FYI, this was raised a few months ago - see:

https://mail.coreboot.org/pipermail/seabios/2018-March/012186.html

IIRC, it should be possible to verify the TPM device is present before
trying to wait for it.

-Kevin

Re: [Qemu-devel] [SeaBIOS] SeaBIOS booting time optimization

[Steve Douthit]

On 11/20/18 10:55 AM, Kevin O'Connor wrote:
> On Mon, Nov 19, 2018 at 07:38:39PM +0100, Stefano Garzarella wrote:
>> just an update, I enabled the debug prints and I saw two timeouts fired
>> with a lot
>> of time lost (~780ms between "init timer" and "Scan for VGA ..."),
>> putting other prints I discovered that a lot of time is spent in the
>> tpm_setup(),
>> during the probe of the 2 TPM devices:
>>
>> 00.548869 init timer
>> 00.549677 ./src/post.c:157 platform_hardware_setup
>> 00.550182 ./src/hw/tpm_drivers.c:579 tpmhw_probe
>> 01.300833 WARNING - Timeout at wait_reg8:81!
>> 01.301388 ./src/hw/tpm_drivers.c:579 tpmhw_probe
>> 01.331843 WARNING - Timeout at wait_reg8:81!
>> 01.332316 ./src/post.c:160 platform_hardware_setup
>> 01.333358 Scan for VGA option rom
> 
> FYI, this was raised a few months ago - see:
> 
> https://mail.coreboot.org/pipermail/seabios/2018-March/012186.html
> 
> IIRC, it should be possible to verify the TPM device is present before
> trying to wait for it.

We could skip probing entirely if there's no TCPA or TPM2 ACPI tables.
There'd need to be some option to force probing in the case of
missing/broken ACPI configurations.

> -Kevin
> 

Re: [Qemu-devel] [SeaBIOS] SeaBIOS booting time optimization

[Stefano Garzarella]

On Tue, Nov 20, 2018 at 5:13 PM Steve Douthit <stephend@silicom-usa.com> wrote:
>
> On 11/20/18 10:55 AM, Kevin O'Connor wrote:
> > On Mon, Nov 19, 2018 at 07:38:39PM +0100, Stefano Garzarella wrote:
> >> just an update, I enabled the debug prints and I saw two timeouts fired
> >> with a lot
> >> of time lost (~780ms between "init timer" and "Scan for VGA ..."),
> >> putting other prints I discovered that a lot of time is spent in the
> >> tpm_setup(),
> >> during the probe of the 2 TPM devices:
> >>
> >> 00.548869 init timer
> >> 00.549677 ./src/post.c:157 platform_hardware_setup
> >> 00.550182 ./src/hw/tpm_drivers.c:579 tpmhw_probe
> >> 01.300833 WARNING - Timeout at wait_reg8:81!
> >> 01.301388 ./src/hw/tpm_drivers.c:579 tpmhw_probe
> >> 01.331843 WARNING - Timeout at wait_reg8:81!
> >> 01.332316 ./src/post.c:160 platform_hardware_setup
> >> 01.333358 Scan for VGA option rom
> >
> > FYI, this was raised a few months ago - see:
> >
> > https://mail.coreboot.org/pipermail/seabios/2018-March/012186.html
> >
> > IIRC, it should be possible to verify the TPM device is present before
> > trying to wait for it.
>
> We could skip probing entirely if there's no TCPA or TPM2 ACPI tables.
> There'd need to be some option to force probing in the case of
> missing/broken ACPI configurations.

I've just tried the
0001-tpm-Check-for-TPM-related-ACPI-tables-before-attempt.patch
(https://mail.coreboot.org/pipermail/seabios/2018-March/012188.html)
and it solves my issue with the TPM.

Thanks,
Stefano

>
> > -Kevin
> >
>


-- 
Stefano Garzarella
Red Hat

Re: [Qemu-devel] [SeaBIOS] SeaBIOS booting time optimization

[Stefan Berger]

On 11/20/18 11:51 AM, Stefano Garzarella wrote:
> On Tue, Nov 20, 2018 at 5:13 PM Steve Douthit <stephend@silicom-usa.com> wrote:
>> On 11/20/18 10:55 AM, Kevin O'Connor wrote:
>>> On Mon, Nov 19, 2018 at 07:38:39PM +0100, Stefano Garzarella wrote:
>>>> just an update, I enabled the debug prints and I saw two timeouts fired
>>>> with a lot
>>>> of time lost (~780ms between "init timer" and "Scan for VGA ..."),
>>>> putting other prints I discovered that a lot of time is spent in the
>>>> tpm_setup(),
>>>> during the probe of the 2 TPM devices:
>>>>
>>>> 00.548869 init timer
>>>> 00.549677 ./src/post.c:157 platform_hardware_setup
>>>> 00.550182 ./src/hw/tpm_drivers.c:579 tpmhw_probe
>>>> 01.300833 WARNING - Timeout at wait_reg8:81!
>>>> 01.301388 ./src/hw/tpm_drivers.c:579 tpmhw_probe
>>>> 01.331843 WARNING - Timeout at wait_reg8:81!
>>>> 01.332316 ./src/post.c:160 platform_hardware_setup
>>>> 01.333358 Scan for VGA option rom
>>> FYI, this was raised a few months ago - see:
>>>
>>> https://mail.coreboot.org/pipermail/seabios/2018-March/012186.html
>>>
>>> IIRC, it should be possible to verify the TPM device is present before
>>> trying to wait for it.
>> We could skip probing entirely if there's no TCPA or TPM2 ACPI tables.
>> There'd need to be some option to force probing in the case of
>> missing/broken ACPI configurations.
> I've just tried the
> 0001-tpm-Check-for-TPM-related-ACPI-tables-before-attempt.patch
> (https://mail.coreboot.org/pipermail/seabios/2018-March/012188.html)
> and it solves my issue with the TPM.

I have tried it with the attached patch of that mail. We have to apply 
it, also for QEMU. It is this patch here:

From: Stephen Douthit <stephend@silicom-usa.com>
Date: Wed, 7 Mar 2018 13:17:36 -0500
Subject: [PATCH] tpm: Check for TPM related ACPI tables before attempting hw
  probe

Signed-off-by: Stephen Douthit <stephend@silicom-usa.com>
---
  src/tcgbios.c | 14 +++++++-------
  1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/src/tcgbios.c b/src/tcgbios.c
index 40b3028..24846d3 100644
--- a/src/tcgbios.c
+++ b/src/tcgbios.c
@@ -968,6 +968,13 @@ tpm_setup(void)
      if (!CONFIG_TCGBIOS)
          return;

+    int ret = tpm_tpm2_probe();
+    if (ret) {
+        ret = tpm_tcpa_probe();
+        if (ret)
+            return;
+    }
+
      TPM_version = tpmhw_probe();
      if (TPM_version == TPM_VERSION_NONE)
          return;
@@ -976,13 +983,6 @@ tpm_setup(void)
              "TCGBIOS: Detected a TPM %s.\n",
               (TPM_version == TPM_VERSION_1_2) ? "1.2" : "2");

-    int ret = tpm_tpm2_probe();
-    if (ret) {
-        ret = tpm_tcpa_probe();
-        if (ret)
-            return;
-    }
-
      TPM_working = 1;

      if (runningOnXen())
-- 
2.14.3




>
> Thanks,
> Stefano
>
>>> -Kevin
>>>
>

Re: [Qemu-devel] [SeaBIOS] SeaBIOS booting time optimization

[Kevin O'Connor]

On Wed, Nov 21, 2018 at 12:43:34AM -0500, Stefan Berger wrote:
> On 11/20/18 11:51 AM, Stefano Garzarella wrote:
> > On Tue, Nov 20, 2018 at 5:13 PM Steve Douthit <stephend@silicom-usa.com> wrote:
> > > On 11/20/18 10:55 AM, Kevin O'Connor wrote:
> > > > FYI, this was raised a few months ago - see:
> > > > 
> > > > https://mail.coreboot.org/pipermail/seabios/2018-March/012186.html
> > > > 
> > > > IIRC, it should be possible to verify the TPM device is present before
> > > > trying to wait for it.
> > > We could skip probing entirely if there's no TCPA or TPM2 ACPI tables.
> > > There'd need to be some option to force probing in the case of
> > > missing/broken ACPI configurations.
> > I've just tried the
> > 0001-tpm-Check-for-TPM-related-ACPI-tables-before-attempt.patch
> > (https://mail.coreboot.org/pipermail/seabios/2018-March/012188.html)
> > and it solves my issue with the TPM.
> 
> I have tried it with the attached patch of that mail. We have to apply it,
> also for QEMU. It is this patch here:

Okay, thanks.  I applied Steve's patch.

-Kevin