All of lore.kernel.org
 help / color / mirror / Atom feed
* Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
@ 2022-02-17 11:19 Alex Bennée
  2022-02-22 16:17 ` Alex Bennée
  0 siblings, 1 reply; 21+ messages in thread
From: Alex Bennée @ 2022-02-17 11:19 UTC (permalink / raw)
  To: qemu-devel
  Cc: Peter Maydell, Cleber Rosa, Oleg Vasilev, Richard Henderson,
	Emilio Cota, Philippe Mathieu-Daudé,
	qemu-arm, Paolo Bonzini, Idan Horowitz

Hi,

TL;DR:

  - pc-bios/edk2-aarch64-code.fd should be rebuilt without debug
  - -cpu max needs pauth-impdef=on 
  - PageDesc->code_bitmap is wastefully thrown away
  - the kernels memmap_init_range interacts badly with pages containing firmware code
  - kernel features like KFENCE also contribute to TB churn

End of TL;DR

I've spent some time looking at fixing the slow tests in check-avocado.
It was a surprising slowdown because I regularly run a full fat Debian
Bullseye system (with systemd!) and it boots to a login prompt in about
26 seconds. A full-fat EFI boot which my distro built EFI image takes a
bit longer but does get to the login prompt in 56 seconds (with some
time taken for selecting grub menu items).

I initially thought I could just replace the heavyweight Fedora image
with the lightest of Alpine distros:

  https://dl-cdn.alpinelinux.org/alpine/v3.15/releases/aarch64/alpine-standard-3.15.0-aarch64.iso

which has a single busybox userspace task and a very lightweight init.
However this very simple setup still takes 160s to get to the login
prompt. Using my distro EDK2 shaves about nearly 40s off the boot but it
is still slow. Clearly something other than the amount of userspace is
to blame here.

The initial perf traces showed a lot of undue prominence to
do_tb_phys_invalidate() which is the mechanism by which we remove
generated code when it has potentially changed. We do this by marking
softmmu pages with DIRTY_MEMORY_CODE which will force a slow-path cputlb
lookup for every write. I added a bunch of instrumentation (see
https://gitlab.com/stsquad/qemu/-/tree/plugins/next) and it quickly
becomes clear what the problem is:

  tb_invalidate_phys_page_fast addr:0x3f3ee000/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:42 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee008/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:42 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee010/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_invalidate_phys_page_range checked:42 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee018/8 pc:0x7f6d4c94 wc:3/(nil)
  tb_invalidate_phys_page_range checked:42 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee020/8 pc:0x7f6d4c94 wc:4/(nil)
  tb_invalidate_phys_page_range checked:42 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee028/8 pc:0x7f6d4c94 wc:5/(nil)
  tb_invalidate_phys_page_range checked:42 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee030/8 pc:0x7f6d4c94 wc:6/(nil)
  tb_invalidate_phys_page_range checked:42 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee038/8 pc:0x7f6d4c94 wc:7/(nil)
  tb_invalidate_phys_page_range checked:42 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee040/8 pc:0x7f6d4c94 wc:8/(nil)
  tb_invalidate_phys_page_range checked:42 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee048/8 pc:0x7f6d4c94 wc:9/(nil)
  tb_invalidate_phys_page_fast addr:0x3f3ee050/8 pc:0x7f6d4c94 wc:10/0x7fd2400fac10

Here we have written to a page SMC_BITMAP_USE_THRESHOLD so create a
bitmap to make testing for hit pages faster. However...

  tb_invalidate_phys_page_fast addr:0x3f3ee058/8 pc:0x7f6d4c94 wc:10/0x7fd2400fac10
  ...
  tb_invalidate_phys_page_fast addr:0x3f3ee710/8 pc:0x7f6d4c94 wc:10/0x7fd2400fac10
  tb_invalidate_phys_page_fast addr:0x3f3ee718/8 pc:0x7f6d4c94 wc:10/0x7fd2400fac10
  tb_invalidate_phys_page_fast addr:0x3f3ee720/8 pc:0x7f6d4c94 wc:10/0x7fd2400fac10
  tb_phys_invalidate tb:0x7fd2945b36c0, pc:0x7f3ee720
  tb_invalidate_phys_page_range checked:42 removed:1 first_tb_pc:0x7f3eede0

The page filling code finally hits a TB that needs removing and we throw
away the bitmap (invalidate_page_bitmap) rather than clearing the bits
no longer relevant. Meaning we have to regenerate anyway...

  tb_invalidate_phys_page_fast addr:0x3f3ee728/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:41 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee730/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:41 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee738/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_phys_invalidate tb:0x7fd2945b3ac0, pc:0x7f3ee738
  tb_invalidate_phys_page_range checked:41 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee740/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_phys_invalidate tb:0x7fd2945b3c40, pc:0x7f3ee740
  tb_invalidate_phys_page_range checked:40 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee748/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:39 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee750/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:39 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee758/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_phys_invalidate tb:0x7fd2945b3d80, pc:0x7f3ee75c
  tb_invalidate_phys_page_range checked:39 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee760/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:38 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee768/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:38 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee770/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_invalidate_phys_page_range checked:38 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee778/8 pc:0x7f6d4c94 wc:3/(nil)
  tb_phys_invalidate tb:0x7fd2945b3ec0, pc:0x7f3ee778
  tb_invalidate_phys_page_range checked:38 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee780/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:37 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee788/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_phys_invalidate tb:0x7fd2945b2e80, pc:0x7f3ee78c
  tb_invalidate_phys_page_range checked:37 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee790/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:36 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee798/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:36 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee7a0/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_invalidate_phys_page_range checked:36 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee7a8/8 pc:0x7f6d4c94 wc:3/(nil)
  tb_invalidate_phys_page_range checked:36 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee7b0/8 pc:0x7f6d4c94 wc:4/(nil)
  tb_invalidate_phys_page_range checked:36 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee7b8/8 pc:0x7f6d4c94 wc:5/(nil)
  tb_invalidate_phys_page_range checked:36 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee7c0/8 pc:0x7f6d4c94 wc:6/(nil)
  tb_phys_invalidate tb:0x7fd2945b48c0, pc:0x7f3ee7c0
  tb_invalidate_phys_page_range checked:36 removed:1 first_tb_pc:0x7f3eede0

And now we continue to remove the pages one at a time until eventually
the page is clear. There is a XXX message that says:

    /* XXX: see if in some cases it could be faster to invalidate all
       the code */

I experimented with unconditionally flushing all the TBs in a page which
did improve things somewhat but then caused excess TB invalidation for
the normal case. One idea would maybe be to tracking the number of
leading zeros in the bitmap when it exists and if it increases from one
call to the next that could be the used as a heuristic to trigger a more
aggressive flush of the TBs for that page.

  tb_invalidate_phys_page_fast addr:0x3f3ee7c8/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:35 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee7d0/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:35 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee7d8/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_phys_invalidate tb:0x7fd2945b3500, pc:0x7f3ee7d8
  tb_invalidate_phys_page_range checked:35 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee7e0/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:34 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee7e8/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_phys_invalidate tb:0x7fd2945b4600, pc:0x7f3ee7ec
  tb_invalidate_phys_page_range checked:34 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee7f0/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_phys_invalidate tb:0x7fd2945b4ac0, pc:0x7f3ee7f4
  tb_invalidate_phys_page_range checked:33 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee7f8/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:32 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee800/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:32 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee808/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_invalidate_phys_page_range checked:32 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee810/8 pc:0x7f6d4c94 wc:3/(nil)
  tb_phys_invalidate tb:0x7fd2945b4d40, pc:0x7f3ee810
  tb_invalidate_phys_page_range checked:32 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee818/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_phys_invalidate tb:0x7fd2945b4e80, pc:0x7f3ee818
  tb_invalidate_phys_page_range checked:31 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee820/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:30 removed:0 first_tb_pc:0x7f3eede0
  ...
  tb_invalidate_phys_page_fast addr:0x3f3ee858/8 pc:0x7f6d4c94 wc:7/(nil)
  tb_invalidate_phys_page_range checked:30 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee860/8 pc:0x7f6d4c94 wc:8/(nil)
  tb_phys_invalidate tb:0x7fd2945b4fc0, pc:0x7f3ee864
  tb_invalidate_phys_page_range checked:30 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee868/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_phys_invalidate tb:0x7fd2945b7c40, pc:0x7f3ee868
  tb_invalidate_phys_page_range checked:29 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee870/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_phys_invalidate tb:0x7fd2945b7d80, pc:0x7f3ee870
  tb_invalidate_phys_page_range checked:28 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee878/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:27 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee880/8 pc:0x7f6d4c94 wc:1/(nil)
  ...
  tb_invalidate_phys_page_range checked:27 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee8b0/8 pc:0x7f6d4c94 wc:7/(nil)
  tb_invalidate_phys_page_range checked:27 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee8b8/8 pc:0x7f6d4c94 wc:8/(nil)
  tb_phys_invalidate tb:0x7fd2945b7ec0, pc:0x7f3ee8bc
  tb_invalidate_phys_page_range checked:27 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee8c0/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_phys_invalidate tb:0x7fd2945b8140, pc:0x7f3ee8c0
  tb_invalidate_phys_page_range checked:26 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee8c8/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_phys_invalidate tb:0x7fd2945b8280, pc:0x7f3ee8c8
  tb_invalidate_phys_page_range checked:25 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee8d0/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:24 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee8d8/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:24 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee8e0/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_invalidate_phys_page_range checked:24 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee8e8/8 pc:0x7f6d4c94 wc:3/(nil)
  tb_invalidate_phys_page_range checked:24 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee8f0/8 pc:0x7f6d4c94 wc:4/(nil)
  tb_invalidate_phys_page_range checked:24 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee8f8/8 pc:0x7f6d4c94 wc:5/(nil)
  tb_invalidate_phys_page_range checked:24 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee900/8 pc:0x7f6d4c94 wc:6/(nil)
  tb_invalidate_phys_page_range checked:24 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee908/8 pc:0x7f6d4c94 wc:7/(nil)
  tb_invalidate_phys_page_range checked:24 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee910/8 pc:0x7f6d4c94 wc:8/(nil)
  tb_phys_invalidate tb:0x7fd2945b83c0, pc:0x7f3ee914
  tb_invalidate_phys_page_range checked:24 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee918/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:23 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee920/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:23 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee928/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_invalidate_phys_page_range checked:23 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee930/8 pc:0x7f6d4c94 wc:3/(nil)
  tb_invalidate_phys_page_range checked:23 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee938/8 pc:0x7f6d4c94 wc:4/(nil)
  tb_invalidate_phys_page_range checked:23 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee940/8 pc:0x7f6d4c94 wc:5/(nil)
  tb_invalidate_phys_page_range checked:23 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee948/8 pc:0x7f6d4c94 wc:6/(nil)
  tb_invalidate_phys_page_range checked:23 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee950/8 pc:0x7f6d4c94 wc:7/(nil)
  tb_phys_invalidate tb:0x7fd2945b8680, pc:0x7f3ee950
  tb_invalidate_phys_page_range checked:23 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee958/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:22 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee960/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:22 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee968/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_invalidate_phys_page_range checked:22 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee970/8 pc:0x7f6d4c94 wc:3/(nil)
  tb_invalidate_phys_page_range checked:22 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee978/8 pc:0x7f6d4c94 wc:4/(nil)
  tb_invalidate_phys_page_range checked:22 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee980/8 pc:0x7f6d4c94 wc:5/(nil)
  tb_invalidate_phys_page_range checked:22 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee988/8 pc:0x7f6d4c94 wc:6/(nil)
  tb_phys_invalidate tb:0x7fd2945b8980, pc:0x7f3ee98c
  tb_invalidate_phys_page_range checked:22 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee990/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:21 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee998/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:21 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee9a0/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_invalidate_phys_page_range checked:21 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee9a8/8 pc:0x7f6d4c94 wc:3/(nil)
  tb_phys_invalidate tb:0x7fd2945b8bc0, pc:0x7f3ee9ac
  tb_invalidate_phys_page_range checked:21 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee9b0/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_phys_invalidate tb:0x7fd2945b8d00, pc:0x7f3ee9b4
  tb_invalidate_phys_page_range checked:20 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee9b8/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:19 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee9c0/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:19 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee9c8/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_invalidate_phys_page_range checked:19 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee9d0/8 pc:0x7f6d4c94 wc:3/(nil)
  tb_invalidate_phys_page_range checked:19 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee9d8/8 pc:0x7f6d4c94 wc:4/(nil)
  tb_invalidate_phys_page_range checked:19 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee9e0/8 pc:0x7f6d4c94 wc:5/(nil)
  tb_invalidate_phys_page_range checked:19 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee9e8/8 pc:0x7f6d4c94 wc:6/(nil)
  tb_invalidate_phys_page_range checked:19 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee9f0/8 pc:0x7f6d4c94 wc:7/(nil)
  tb_invalidate_phys_page_range checked:19 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3ee9f8/8 pc:0x7f6d4c94 wc:8/(nil)
  tb_invalidate_phys_page_range checked:19 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea00/8 pc:0x7f6d4c94 wc:9/(nil)
  tb_phys_invalidate tb:0x7fd2945b8e40, pc:0x7f3eea00
  tb_invalidate_phys_page_range checked:19 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea08/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:18 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea10/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:18 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea18/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_invalidate_phys_page_range checked:18 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea20/8 pc:0x7f6d4c94 wc:3/(nil)
  tb_phys_invalidate tb:0x7fd2945b9000, pc:0x7f3eea20
  tb_invalidate_phys_page_range checked:18 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea28/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:17 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea30/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:17 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea38/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_invalidate_phys_page_range checked:17 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea40/8 pc:0x7f6d4c94 wc:3/(nil)
  tb_invalidate_phys_page_range checked:17 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea48/8 pc:0x7f6d4c94 wc:4/(nil)
  tb_invalidate_phys_page_range checked:17 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea50/8 pc:0x7f6d4c94 wc:5/(nil)
  tb_invalidate_phys_page_range checked:17 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea58/8 pc:0x7f6d4c94 wc:6/(nil)
  tb_invalidate_phys_page_range checked:17 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea60/8 pc:0x7f6d4c94 wc:7/(nil)
  tb_invalidate_phys_page_range checked:17 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea68/8 pc:0x7f6d4c94 wc:8/(nil)
  tb_phys_invalidate tb:0x7fd2945b9140, pc:0x7f3eea6c
  tb_invalidate_phys_page_range checked:17 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea70/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_phys_invalidate tb:0x7fd2945cd5c0, pc:0x7f3eea70
  tb_invalidate_phys_page_range checked:16 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea78/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:15 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea80/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:15 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea88/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_invalidate_phys_page_range checked:15 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea90/8 pc:0x7f6d4c94 wc:3/(nil)
  tb_invalidate_phys_page_range checked:15 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eea98/8 pc:0x7f6d4c94 wc:4/(nil)
  tb_invalidate_phys_page_range checked:15 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeaa0/8 pc:0x7f6d4c94 wc:5/(nil)
  tb_invalidate_phys_page_range checked:15 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeaa8/8 pc:0x7f6d4c94 wc:6/(nil)
  tb_invalidate_phys_page_range checked:15 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeab0/8 pc:0x7f6d4c94 wc:7/(nil)
  tb_invalidate_phys_page_range checked:15 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeab8/8 pc:0x7f6d4c94 wc:8/(nil)
  tb_invalidate_phys_page_range checked:15 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeac0/8 pc:0x7f6d4c94 wc:9/(nil)
  tb_phys_invalidate tb:0x7fd2945cd840, pc:0x7f3eeac4
  tb_phys_invalidate tb:0x7fd2945cd700, pc:0x7f3eeac0
  tb_invalidate_phys_page_range checked:15 removed:2 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeac8/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:13 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eead0/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:13 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eead8/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_invalidate_phys_page_range checked:13 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeae0/8 pc:0x7f6d4c94 wc:3/(nil)
  tb_invalidate_phys_page_range checked:13 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeae8/8 pc:0x7f6d4c94 wc:4/(nil)
  tb_invalidate_phys_page_range checked:13 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeaf0/8 pc:0x7f6d4c94 wc:5/(nil)
  tb_invalidate_phys_page_range checked:13 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeaf8/8 pc:0x7f6d4c94 wc:6/(nil)
  tb_phys_invalidate tb:0x7fd2945d2b40, pc:0x7f3eeaf8
  tb_invalidate_phys_page_range checked:13 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb00/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_phys_invalidate tb:0x7fd2945b4780, pc:0x7f3eeb00
  tb_invalidate_phys_page_range checked:12 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb08/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_phys_invalidate tb:0x7fd2945cd9c0, pc:0x7f3eeb08
  tb_invalidate_phys_page_range checked:11 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb10/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_phys_invalidate tb:0x7fd2945d2a00, pc:0x7f3eeb10
  tb_invalidate_phys_page_range checked:10 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb18/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:9 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb20/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:9 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb28/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_invalidate_phys_page_range checked:9 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb30/8 pc:0x7f6d4c94 wc:3/(nil)
  tb_invalidate_phys_page_range checked:9 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb38/8 pc:0x7f6d4c94 wc:4/(nil)
  tb_invalidate_phys_page_range checked:9 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb40/8 pc:0x7f6d4c94 wc:5/(nil)
  tb_invalidate_phys_page_range checked:9 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb48/8 pc:0x7f6d4c94 wc:6/(nil)
  tb_invalidate_phys_page_range checked:9 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb50/8 pc:0x7f6d4c94 wc:7/(nil)
  tb_invalidate_phys_page_range checked:9 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb58/8 pc:0x7f6d4c94 wc:8/(nil)
  tb_phys_invalidate tb:0x7fd2945d2c80, pc:0x7f3eeb58
  tb_invalidate_phys_page_range checked:9 removed:1 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb60/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:8 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb68/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:8 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb70/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_invalidate_phys_page_range checked:8 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb78/8 pc:0x7f6d4c94 wc:3/(nil)
  tb_invalidate_phys_page_range checked:8 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb80/8 pc:0x7f6d4c94 wc:4/(nil)
  tb_invalidate_phys_page_range checked:8 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb88/8 pc:0x7f6d4c94 wc:5/(nil)
  tb_invalidate_phys_page_range checked:8 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb90/8 pc:0x7f6d4c94 wc:6/(nil)
  tb_invalidate_phys_page_range checked:8 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeb98/8 pc:0x7f6d4c94 wc:7/(nil)
  tb_invalidate_phys_page_range checked:8 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeba0/8 pc:0x7f6d4c94 wc:8/(nil)
  tb_invalidate_phys_page_range checked:8 removed:0 first_tb_pc:0x7f3eede0
  tb_invalidate_phys_page_fast addr:0x3f3eeba8/8 pc:0x7f6d4c94 wc:9/(nil)
  tb_invalidate_phys_page_fast addr:0x3f3eebb0/8 pc:0x7f6d4c94 wc:10/0x7fd2400fae20
  ...
  tb_invalidate_phys_page_fast addr:0x3f3eede0/8 pc:0x7f6d4c94 wc:10/0x7fd2400fae20
  tb_phys_invalidate tb:0x7fd2945d2dc0, pc:0x7f3eede0
  tb_invalidate_phys_page_range checked:8 removed:1 first_tb_pc:0x7f3eef98
  tb_invalidate_phys_page_fast addr:0x3f3eede8/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:7 removed:0 first_tb_pc:0x7f3eef98
  tb_invalidate_phys_page_fast addr:0x3f3eedf0/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:7 removed:0 first_tb_pc:0x7f3eef98
  tb_invalidate_phys_page_fast addr:0x3f3eedf8/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_invalidate_phys_page_range checked:7 removed:0 first_tb_pc:0x7f3eef98
  tb_invalidate_phys_page_fast addr:0x3f3eee00/8 pc:0x7f6d4c94 wc:3/(nil)
  tb_invalidate_phys_page_range checked:7 removed:0 first_tb_pc:0x7f3eef98
  tb_invalidate_phys_page_fast addr:0x3f3eee08/8 pc:0x7f6d4c94 wc:4/(nil)
  tb_invalidate_phys_page_range checked:7 removed:0 first_tb_pc:0x7f3eef98
  tb_invalidate_phys_page_fast addr:0x3f3eee10/8 pc:0x7f6d4c94 wc:5/(nil)
  tb_invalidate_phys_page_range checked:7 removed:0 first_tb_pc:0x7f3eef98
  tb_invalidate_phys_page_fast addr:0x3f3eee18/8 pc:0x7f6d4c94 wc:6/(nil)
  tb_invalidate_phys_page_range checked:7 removed:0 first_tb_pc:0x7f3eef98
  tb_invalidate_phys_page_fast addr:0x3f3eee20/8 pc:0x7f6d4c94 wc:7/(nil)
  tb_invalidate_phys_page_range checked:7 removed:0 first_tb_pc:0x7f3eef98
  tb_invalidate_phys_page_fast addr:0x3f3eee28/8 pc:0x7f6d4c94 wc:8/(nil)
  tb_invalidate_phys_page_range checked:7 removed:0 first_tb_pc:0x7f3eef98
  tb_invalidate_phys_page_fast addr:0x3f3eee30/8 pc:0x7f6d4c94 wc:9/(nil)
  tb_invalidate_phys_page_fast addr:0x3f3eee38/8 pc:0x7f6d4c94 wc:10/0x7fd2400fae20
  ...
  tb_invalidate_phys_page_fast addr:0x3f3eef90/8 pc:0x7f6d4c94 wc:10/0x7fd2400fae20
  tb_phys_invalidate tb:0x7fd2945c0840, pc:0x7f3eef90
  tb_invalidate_phys_page_range checked:7 removed:1 first_tb_pc:0x7f3eef98
  tb_invalidate_phys_page_fast addr:0x3f3eef98/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_phys_invalidate tb:0x7fd2945c16c0, pc:0x7f3eef98
  tb_invalidate_phys_page_range checked:6 removed:1 first_tb_pc:0x7f3eefb8
  tb_invalidate_phys_page_fast addr:0x3f3eefa0/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_phys_invalidate tb:0x7fd2945c0a40, pc:0x7f3eefa4
  tb_invalidate_phys_page_range checked:5 removed:1 first_tb_pc:0x7f3eefb8
  tb_invalidate_phys_page_fast addr:0x3f3eefa8/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_phys_invalidate tb:0x7fd2945c1380, pc:0x7f3eefa8
  tb_invalidate_phys_page_range checked:4 removed:1 first_tb_pc:0x7f3eefb8
  tb_invalidate_phys_page_fast addr:0x3f3eefb0/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:3 removed:0 first_tb_pc:0x7f3eefb8
  tb_invalidate_phys_page_fast addr:0x3f3eefb8/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_phys_invalidate tb:0x7fd2945c1580, pc:0x7f3eefb8
  tb_invalidate_phys_page_range checked:3 removed:1 first_tb_pc:0x7f3eeffc
  tb_invalidate_phys_page_fast addr:0x3f3eefc0/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_phys_invalidate tb:0x7fd2945bc800, pc:0x7f3eefc4
  tb_invalidate_phys_page_range checked:2 removed:1 first_tb_pc:0x7f3eeffc
  tb_invalidate_phys_page_fast addr:0x3f3eefc8/8 pc:0x7f6d4c94 wc:0/(nil)
  tb_invalidate_phys_page_range checked:1 removed:0 first_tb_pc:0x7f3eeffc
  tb_invalidate_phys_page_fast addr:0x3f3eefd0/8 pc:0x7f6d4c94 wc:1/(nil)
  tb_invalidate_phys_page_range checked:1 removed:0 first_tb_pc:0x7f3eeffc
  tb_invalidate_phys_page_fast addr:0x3f3eefd8/8 pc:0x7f6d4c94 wc:2/(nil)
  tb_invalidate_phys_page_range checked:1 removed:0 first_tb_pc:0x7f3eeffc
  tb_invalidate_phys_page_fast addr:0x3f3eefe0/8 pc:0x7f6d4c94 wc:3/(nil)
  tb_invalidate_phys_page_range checked:1 removed:0 first_tb_pc:0x7f3eeffc
  tb_invalidate_phys_page_fast addr:0x3f3eefe8/8 pc:0x7f6d4c94 wc:4/(nil)
  tb_invalidate_phys_page_range checked:1 removed:0 first_tb_pc:0x7f3eeffc
  tb_invalidate_phys_page_fast addr:0x3f3eeff0/8 pc:0x7f6d4c94 wc:5/(nil)
  tb_invalidate_phys_page_range checked:1 removed:0 first_tb_pc:0x7f3eeffc
  tb_invalidate_phys_page_fast addr:0x3f3eeff8/8 pc:0x7f6d4c94 wc:6/(nil)
  tb_phys_invalidate tb:0x7fd2945bcf00, pc:0x7f3eeffc

Having finally cleared all the TBs from the page we finally unprotect
the RAM region ensuring that future writes to the page can be fast
pathed (until we generate new code for a page).

  tlb_unprotect_code 0x3f3eeff8

So my working theory is:

  - booting with EFI leaves a number stale code pages
  - as the kernel boots it clears these pages triggering lots of
    slow-path writes
  - result very slow boot

I think the only real solution is to improve our TB invalidation code
but perhaps another approach would be to detect a high TB invalidation
churn and just tb_flush the whole thing and start again?

Any thoughts or ideas?

-- 
Alex Bennée


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
  2022-02-17 11:19 Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*) Alex Bennée
@ 2022-02-22 16:17 ` Alex Bennée
  2022-02-22 16:38   ` Daniel P. Berrangé
  0 siblings, 1 reply; 21+ messages in thread
From: Alex Bennée @ 2022-02-22 16:17 UTC (permalink / raw)
  To: qemu-devel, Laszlo Ersek
  Cc: Peter Maydell, Cleber Rosa, Oleg Vasilev, Richard Henderson,
	Emilio Cota, Philippe Mathieu-Daudé,
	qemu-arm, Paolo Bonzini, Idan Horowitz


Alex Bennée <alex.bennee@linaro.org> writes:

> Hi,
>
> TL;DR:
>
>   - pc-bios/edk2-aarch64-code.fd should be rebuilt without debug

Laszlo,

Would it be possible to do a less debug enabled version of EDK2 on the
next update to pc-bios/edk2-*?

>
> So my working theory is:
>
>   - booting with EFI leaves a number stale code pages
>   - as the kernel boots it clears these pages triggering lots of
>     slow-path writes
>   - result very slow boot
>
> I think the only real solution is to improve our TB invalidation code
> but perhaps another approach would be to detect a high TB invalidation
> churn and just tb_flush the whole thing and start again?
>
> Any thoughts or ideas?


-- 
Alex Bennée


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
  2022-02-22 16:17 ` Alex Bennée
@ 2022-02-22 16:38   ` Daniel P. Berrangé
  2022-02-22 17:33     ` Philippe Mathieu-Daudé
  0 siblings, 1 reply; 21+ messages in thread
From: Daniel P. Berrangé @ 2022-02-22 16:38 UTC (permalink / raw)
  To: Alex Bennée
  Cc: Peter Maydell, Oleg Vasilev, Idan Horowitz, Cleber Rosa,
	Richard Henderson, qemu-devel, Philippe Mathieu-Daudé,
	Emilio Cota, Gerd Hoffmann, qemu-arm, Paolo Bonzini,
	Laszlo Ersek

On Tue, Feb 22, 2022 at 04:17:23PM +0000, Alex Bennée wrote:
> 
> Alex Bennée <alex.bennee@linaro.org> writes:
> 
> > Hi,
> >
> > TL;DR:
> >
> >   - pc-bios/edk2-aarch64-code.fd should be rebuilt without debug
> 
> Laszlo,
> 
> Would it be possible to do a less debug enabled version of EDK2 on the
> next update to pc-bios/edk2-*?

NB, Laszlo is no longer  maintaining EDK2 in QEMU, it was handed
over to Philippe.  I'm CC'ing Gerd too since he's a reviewer and
an EDK2 contributor taking over from Lazslo in EDK2 community

> 
> >
> > So my working theory is:
> >
> >   - booting with EFI leaves a number stale code pages
> >   - as the kernel boots it clears these pages triggering lots of
> >     slow-path writes
> >   - result very slow boot
> >
> > I think the only real solution is to improve our TB invalidation code
> > but perhaps another approach would be to detect a high TB invalidation
> > churn and just tb_flush the whole thing and start again?
> >
> > Any thoughts or ideas?
> 
> 
> -- 
> Alex Bennée
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
  2022-02-22 16:38   ` Daniel P. Berrangé
@ 2022-02-22 17:33     ` Philippe Mathieu-Daudé
  2022-02-22 17:37       ` Peter Maydell
                         ` (2 more replies)
  0 siblings, 3 replies; 21+ messages in thread
From: Philippe Mathieu-Daudé @ 2022-02-22 17:33 UTC (permalink / raw)
  To: Daniel P. Berrangé, Alex Bennée, Michael S. Tsirkin
  Cc: Peter Maydell, Andrew Jones, Oleg Vasilev, Idan Horowitz,
	Cleber Rosa, Richard Henderson, qemu-devel,
	Philippe Mathieu-Daudé,
	Emilio Cota, Gerd Hoffmann, qemu-arm, Igor Mammedov,
	Paolo Bonzini, Laszlo Ersek

+Igor/MST for UEFI tests.

On 22/2/22 17:38, Daniel P. Berrangé wrote:
> On Tue, Feb 22, 2022 at 04:17:23PM +0000, Alex Bennée wrote:
>>
>> Alex Bennée <alex.bennee@linaro.org> writes:
>>
>>> Hi,
>>>
>>> TL;DR:
>>>
>>>    - pc-bios/edk2-aarch64-code.fd should be rebuilt without debug
>>
>> Laszlo,
>>
>> Would it be possible to do a less debug enabled version of EDK2 on the
>> next update to pc-bios/edk2-*?
> 
> NB, Laszlo is no longer  maintaining EDK2 in QEMU, it was handed
> over to Philippe.  I'm CC'ing Gerd too since he's a reviewer and
> an EDK2 contributor taking over from Lazslo in EDK2 community

We need the DEBUG profile to ensure the bios-tables-tests work.

If you want to boot a guest using EDK2, you should use the images
build by your distribution (/usr/share/qemu/edk2-aarch64-code.fd),
not these images.

Regards,

Phil.


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
  2022-02-22 17:33     ` Philippe Mathieu-Daudé
@ 2022-02-22 17:37       ` Peter Maydell
  2022-02-22 18:05         ` Philippe Mathieu-Daudé
  2022-02-23  9:19       ` Peter Maydell
  2022-02-23 11:07       ` Daniel P. Berrangé
  2 siblings, 1 reply; 21+ messages in thread
From: Peter Maydell @ 2022-02-22 17:37 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Oleg Vasilev, Andrew Jones, Daniel P. Berrangé,
	Cleber Rosa, Laszlo Ersek, Michael S. Tsirkin, Richard Henderson,
	qemu-devel, Idan Horowitz, Philippe Mathieu-Daudé,
	Emilio Cota, Gerd Hoffmann, qemu-arm, Igor Mammedov,
	Paolo Bonzini, Alex Bennée

On Tue, 22 Feb 2022 at 17:33, Philippe Mathieu-Daudé
<philippe.mathieu.daude@gmail.com> wrote:
>
> +Igor/MST for UEFI tests.
>
> On 22/2/22 17:38, Daniel P. Berrangé wrote:
> > On Tue, Feb 22, 2022 at 04:17:23PM +0000, Alex Bennée wrote:
> >>
> >> Alex Bennée <alex.bennee@linaro.org> writes:
> >>
> >>> Hi,
> >>>
> >>> TL;DR:
> >>>
> >>>    - pc-bios/edk2-aarch64-code.fd should be rebuilt without debug
> >>
> >> Laszlo,
> >>
> >> Would it be possible to do a less debug enabled version of EDK2 on the
> >> next update to pc-bios/edk2-*?

> We need the DEBUG profile to ensure the bios-tables-tests work.
>
> If you want to boot a guest using EDK2, you should use the images
> build by your distribution (/usr/share/qemu/edk2-aarch64-code.fd),
> not these images.

But we're booting guests using EDK2 as part of our avocado tests:
if we say "booting guests with EDK2 should be done with non-debug
images" we need to also be building and providing those in
pc-bios for the avocado tests.

-- PMM


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
  2022-02-22 17:37       ` Peter Maydell
@ 2022-02-22 18:05         ` Philippe Mathieu-Daudé
  2022-02-23  8:48           ` Gerd Hoffmann
  0 siblings, 1 reply; 21+ messages in thread
From: Philippe Mathieu-Daudé @ 2022-02-22 18:05 UTC (permalink / raw)
  To: Peter Maydell, Cleber Rosa, Beraldo Leal
  Cc: Oleg Vasilev, Andrew Jones, Daniel P. Berrangé,
	Michael S. Tsirkin, Laszlo Ersek, Idan Horowitz,
	Richard Henderson, qemu-devel, Philippe Mathieu-Daudé,
	Emilio Cota, Gerd Hoffmann, qemu-arm, Igor Mammedov,
	Paolo Bonzini, Alex Bennée

+Beraldo

On 22/2/22 18:37, Peter Maydell wrote:
> On Tue, 22 Feb 2022 at 17:33, Philippe Mathieu-Daudé
> <philippe.mathieu.daude@gmail.com> wrote:
>>
>> +Igor/MST for UEFI tests.
>>
>> On 22/2/22 17:38, Daniel P. Berrangé wrote:
>>> On Tue, Feb 22, 2022 at 04:17:23PM +0000, Alex Bennée wrote:
>>>>
>>>> Alex Bennée <alex.bennee@linaro.org> writes:
>>>>
>>>>> Hi,
>>>>>
>>>>> TL;DR:
>>>>>
>>>>>     - pc-bios/edk2-aarch64-code.fd should be rebuilt without debug
>>>>
>>>> Laszlo,
>>>>
>>>> Would it be possible to do a less debug enabled version of EDK2 on the
>>>> next update to pc-bios/edk2-*?
> 
>> We need the DEBUG profile to ensure the bios-tables-tests work.
>>
>> If you want to boot a guest using EDK2, you should use the images
>> build by your distribution (/usr/share/qemu/edk2-aarch64-code.fd),
>> not these images.
> 
> But we're booting guests using EDK2 as part of our avocado tests:
> if we say "booting guests with EDK2 should be done with non-debug
> images" we need to also be building and providing those in
> pc-bios for the avocado tests.

Then we should add edk2-aarch64 and edk2-ovmf to lcitool, to have
the distrib images in our generated Docker images.

Cleber, you added this test in commit 6fd52d671d ("Acceptance test:
add "boot_linux" tests"), can you have a look?

Thanks,

Phil.


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
  2022-02-22 18:05         ` Philippe Mathieu-Daudé
@ 2022-02-23  8:48           ` Gerd Hoffmann
       [not found]             ` <87zgmhlwjw.fsf@linaro.org>
  0 siblings, 1 reply; 21+ messages in thread
From: Gerd Hoffmann @ 2022-02-23  8:48 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Peter Maydell, Andrew Jones, Daniel P. Berrangé,
	Beraldo Leal, Michael S. Tsirkin, Laszlo Ersek, Oleg Vasilev,
	Cleber Rosa, Richard Henderson, qemu-devel, Idan Horowitz,
	Philippe Mathieu-Daudé,
	Emilio Cota, qemu-arm, Igor Mammedov, Paolo Bonzini,
	Alex Bennée

  Hi,

> > > If you want to boot a guest using EDK2, you should use the images
> > > build by your distribution (/usr/share/qemu/edk2-aarch64-code.fd),
> > > not these images.

> Then we should add edk2-aarch64 and edk2-ovmf to lcitool, to have
> the distrib images in our generated Docker images.
> 
> Cleber, you added this test in commit 6fd52d671d ("Acceptance test:
> add "boot_linux" tests"), can you have a look?

Well, it's not *that* simple.  Names are not consistent across
distributions.  I think if we want go that route we have to inspect
the *.json files in /usr/share/qemu/firmware to find the correct
distro firmware images.

Also note that at least fedora ships both verbose and non-verbose
images ...

    kraxel@sirius ~# rpm -ql edk2-aarch64 
    [ ... ]
    /usr/share/edk2/aarch64/QEMU_EFI-pflash.raw
    /usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.raw
    /usr/share/edk2/aarch64/QEMU_EFI.fd
    /usr/share/edk2/aarch64/QEMU_EFI.silent.fd
    /usr/share/edk2/aarch64/QEMU_VARS.fd
    [ ... ]
    /usr/share/qemu/firmware/60-edk2-aarch64.json
    /usr/share/qemu/firmware/70-edk2-aarch64-verbose.json

... so maybe it is an option to use the distro images for the bios
tables test cases too.

take care,
  Gerd



^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
  2022-02-22 17:33     ` Philippe Mathieu-Daudé
  2022-02-22 17:37       ` Peter Maydell
@ 2022-02-23  9:19       ` Peter Maydell
  2022-02-23 10:53         ` Philippe Mathieu-Daudé
  2022-02-23 11:07       ` Daniel P. Berrangé
  2 siblings, 1 reply; 21+ messages in thread
From: Peter Maydell @ 2022-02-23  9:19 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Oleg Vasilev, Andrew Jones, Daniel P. Berrangé,
	Cleber Rosa, Laszlo Ersek, Michael S. Tsirkin, Richard Henderson,
	qemu-devel, Idan Horowitz, Philippe Mathieu-Daudé,
	Emilio Cota, Gerd Hoffmann, qemu-arm, Igor Mammedov,
	Paolo Bonzini, Alex Bennée

On Tue, 22 Feb 2022 at 17:33, Philippe Mathieu-Daudé
<philippe.mathieu.daude@gmail.com> wrote:
> We need the DEBUG profile to ensure the bios-tables-tests work.
>
> If you want to boot a guest using EDK2, you should use the images
> build by your distribution (/usr/share/qemu/edk2-aarch64-code.fd),
> not these images.

Do we have any other images in pc-bios that are silently
"do not use these images, they are only for a very specific
test case and nothing else" ?

Also, "make install" installs these EDK2 images, which doesn't
seem like the right thing for "this is only for one test case".

thanks
-- PMM


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
  2022-02-23  9:19       ` Peter Maydell
@ 2022-02-23 10:53         ` Philippe Mathieu-Daudé
  2022-02-23 11:11           ` Daniel P. Berrangé
  2022-02-23 11:50           ` Gerd Hoffmann
  0 siblings, 2 replies; 21+ messages in thread
From: Philippe Mathieu-Daudé @ 2022-02-23 10:53 UTC (permalink / raw)
  To: Peter Maydell
  Cc: Oleg Vasilev, Andrew Jones, Daniel P. Berrangé,
	Cleber Rosa, Laszlo Ersek, Michael S. Tsirkin, Richard Henderson,
	qemu-devel, Idan Horowitz, Philippe Mathieu-Daudé,
	Emilio Cota, Gerd Hoffmann, qemu-arm, Igor Mammedov,
	Paolo Bonzini, Alex Bennée

On 23/2/22 10:19, Peter Maydell wrote:
> On Tue, 22 Feb 2022 at 17:33, Philippe Mathieu-Daudé
> <philippe.mathieu.daude@gmail.com> wrote:
>> We need the DEBUG profile to ensure the bios-tables-tests work.
>>
>> If you want to boot a guest using EDK2, you should use the images
>> build by your distribution (/usr/share/qemu/edk2-aarch64-code.fd),
>> not these images.
> 
> Do we have any other images in pc-bios that are silently
> "do not use these images, they are only for a very specific
> test case and nothing else" ?

I might try to move that to test/. Gerd, are you OK with that?

> Also, "make install" installs these EDK2 images, which doesn't
> seem like the right thing for "this is only for one test case".

Well I'd prefer we never had them installed. Today I don't remember
why it ended that way. Gerd, similarly, are you OK with not installing
these images (after a 2 release deprecation warning)?


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
       [not found]             ` <87zgmhlwjw.fsf@linaro.org>
@ 2022-02-23 10:58               ` Philippe Mathieu-Daudé
  0 siblings, 0 replies; 21+ messages in thread
From: Philippe Mathieu-Daudé @ 2022-02-23 10:58 UTC (permalink / raw)
  To: Alex Bennée, Gerd Hoffmann
  Cc: Peter Maydell, Andrew Jones, Daniel P. Berrangé,
	Beraldo Leal, Michael S. Tsirkin, Idan Horowitz, Oleg Vasilev,
	Cleber Rosa, Richard Henderson, qemu-devel,
	Philippe Mathieu-Daudé,
	Emilio Cota, qemu-arm, Igor Mammedov, Paolo Bonzini,
	Laszlo Ersek

On 23/2/22 11:10, Alex Bennée wrote:
> 
> Gerd Hoffmann <kraxel@redhat.com> writes:
> 
>>    Hi,
>>
>>>>> If you want to boot a guest using EDK2, you should use the images
>>>>> build by your distribution (/usr/share/qemu/edk2-aarch64-code.fd),
>>>>> not these images.
>>
>>> Then we should add edk2-aarch64 and edk2-ovmf to lcitool, to have
>>> the distrib images in our generated Docker images.
>>>
>>> Cleber, you added this test in commit 6fd52d671d ("Acceptance test:
>>> add "boot_linux" tests"), can you have a look?
>>
>> Well, it's not *that* simple.  Names are not consistent across
>> distributions.  I think if we want go that route we have to inspect
>> the *.json files in /usr/share/qemu/firmware to find the correct
>> distro firmware images.
>>
>> Also note that at least fedora ships both verbose and non-verbose
>> images ...
>>
>>      kraxel@sirius ~# rpm -ql edk2-aarch64
>>      [ ... ]
>>      /usr/share/edk2/aarch64/QEMU_EFI-pflash.raw
>>      /usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.raw
>>      /usr/share/edk2/aarch64/QEMU_EFI.fd
>>      /usr/share/edk2/aarch64/QEMU_EFI.silent.fd
>>      /usr/share/edk2/aarch64/QEMU_VARS.fd
>>      [ ... ]
>>      /usr/share/qemu/firmware/60-edk2-aarch64.json
>>      /usr/share/qemu/firmware/70-edk2-aarch64-verbose.json
>>
>> ... so maybe it is an option to use the distro images for the bios
>> tables test cases too.
> 
> Debian isn't quite so free with it's use of disk space although:
> 
>    10:09:19 [root@zen:~] # dpkg -L qemu-efi-aarch64
>    /.
>    /usr
>    /usr/share
>    /usr/share/AAVMF
>    /usr/share/AAVMF/AAVMF_CODE.fd
>    /usr/share/AAVMF/AAVMF_VARS.fd
>    /usr/share/doc
>    /usr/share/doc/qemu-efi-aarch64
>    /usr/share/doc/qemu-efi-aarch64/changelog.Debian.gz
>    /usr/share/doc/qemu-efi-aarch64/copyright
>    /usr/share/qemu
>    /usr/share/qemu/firmware
>    /usr/share/qemu/firmware/60-edk2-aarch64.json
>    /usr/share/qemu-efi-aarch64
>    /usr/share/qemu-efi-aarch64/QEMU_EFI.fd
>    10:09:25 [root@zen:~] # md5sum /usr/share/AAVMF/AAVMF_CODE.fd /usr/share/qemu-efi-aarch64/QEMU_EFI.fd
>    573b65b6e04981abb5b10afc8f30feea  /usr/share/AAVMF/AAVMF_CODE.fd
>    99812e842b6b40add0d8f7766e0aac9e  /usr/share/qemu-efi-aarch64/QEMU_EFI.fd
>    10:09:37 [root@zen:~] # ls -lh /usr/share/AAVMF/AAVMF_CODE.fd /usr/share/qemu-efi-aarch64/QEMU_EFI.fd
>    -rw-r--r-- 1 root root  64M Aug 18  2021 /usr/share/AAVMF/AAVMF_CODE.fd
>    -rw-r--r-- 1 root root 2.0M Aug 18  2021 /usr/share/qemu-efi-aarch64/QEMU_EFI.fd
> 
> I think the QEMU_EFI.fd is the firmware and AAVF_CODE is the same
> firmware but packaged in the "right" size of flash file.
> 
> However if we are to use the distro version (or at least favour it) do
> we need to start encoding searches through common paths?
> 
> I'm also sympathetic to Peter's point that distros might just end up packaging
> what we give them in pc-bios and we'll be back to square one. I'd favour
> pc-bios having both a edk2-aarch64-code.fd and a edk2-aarch64-code-debug.fd.

Distro have security teams with crypto knowledge who track CVE vulns
and integrate fixes, often embedding their own certificates in the
UEFI image. They certainly don't use that old/unsecure version
QEMU provides (edk2-stable202008, current is edk2-stable202111).


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
  2022-02-22 17:33     ` Philippe Mathieu-Daudé
  2022-02-22 17:37       ` Peter Maydell
  2022-02-23  9:19       ` Peter Maydell
@ 2022-02-23 11:07       ` Daniel P. Berrangé
  2022-02-23 13:34         ` Philippe Mathieu-Daudé
  2 siblings, 1 reply; 21+ messages in thread
From: Daniel P. Berrangé @ 2022-02-23 11:07 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Peter Maydell, Andrew Jones, Cleber Rosa, Alex Bennée,
	Oleg Vasilev, Michael S. Tsirkin, Richard Henderson, qemu-devel,
	Idan Horowitz, Philippe Mathieu-Daudé,
	Emilio Cota, Gerd Hoffmann, qemu-arm, Igor Mammedov,
	Paolo Bonzini, Laszlo Ersek

On Tue, Feb 22, 2022 at 06:33:41PM +0100, Philippe Mathieu-Daudé wrote:
> +Igor/MST for UEFI tests.
> 
> On 22/2/22 17:38, Daniel P. Berrangé wrote:
> > On Tue, Feb 22, 2022 at 04:17:23PM +0000, Alex Bennée wrote:
> > > 
> > > Alex Bennée <alex.bennee@linaro.org> writes:
> > > 
> > > > Hi,
> > > > 
> > > > TL;DR:
> > > > 
> > > >    - pc-bios/edk2-aarch64-code.fd should be rebuilt without debug
> > > 
> > > Laszlo,
> > > 
> > > Would it be possible to do a less debug enabled version of EDK2 on the
> > > next update to pc-bios/edk2-*?
> > 
> > NB, Laszlo is no longer  maintaining EDK2 in QEMU, it was handed
> > over to Philippe.  I'm CC'ing Gerd too since he's a reviewer and
> > an EDK2 contributor taking over from Lazslo in EDK2 community
> 
> We need the DEBUG profile to ensure the bios-tables-tests work.

Can you elaborate on what bios-tables-tests needs this for, and
what coverage we would loose by disabling DEBUG.

It may well be a better tradeoff to sacrifice part of bios-tables-tests
in favour of shipping more broadly usable images without DEBUG.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
  2022-02-23 10:53         ` Philippe Mathieu-Daudé
@ 2022-02-23 11:11           ` Daniel P. Berrangé
  2022-02-23 11:50           ` Gerd Hoffmann
  1 sibling, 0 replies; 21+ messages in thread
From: Daniel P. Berrangé @ 2022-02-23 11:11 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Peter Maydell, Andrew Jones, Cleber Rosa, Laszlo Ersek,
	Oleg Vasilev, Michael S. Tsirkin, Richard Henderson, qemu-devel,
	Idan Horowitz, Philippe Mathieu-Daudé,
	Emilio Cota, Gerd Hoffmann, qemu-arm, Igor Mammedov,
	Paolo Bonzini, Alex Bennée

On Wed, Feb 23, 2022 at 11:53:28AM +0100, Philippe Mathieu-Daudé wrote:
> On 23/2/22 10:19, Peter Maydell wrote:
> > On Tue, 22 Feb 2022 at 17:33, Philippe Mathieu-Daudé
> > <philippe.mathieu.daude@gmail.com> wrote:
> > > We need the DEBUG profile to ensure the bios-tables-tests work.
> > > 
> > > If you want to boot a guest using EDK2, you should use the images
> > > build by your distribution (/usr/share/qemu/edk2-aarch64-code.fd),
> > > not these images.
> > 
> > Do we have any other images in pc-bios that are silently
> > "do not use these images, they are only for a very specific
> > test case and nothing else" ?
> 
> I might try to move that to test/. Gerd, are you OK with that?
> 
> > Also, "make install" installs these EDK2 images, which doesn't
> > seem like the right thing for "this is only for one test case".
> 
> Well I'd prefer we never had them installed. Today I don't remember
> why it ended that way. Gerd, similarly, are you OK with not installing
> these images (after a 2 release deprecation warning)?

If we didn't install them, wouldn't that mean people using QEMU upstream
would not have all the pieces needed to use AAch64 virt, without now
building EDK2 themselves. That feels wrong/bad to me, given we ship
firmware needed to use everything else.

I don't disagree with recommendation that most people are better off
using distro built QEMU + EDK, but if someone chooses to use a upstream
QEMU I feel it ought to come with firmware needed to make it work with
all targets we support.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
  2022-02-23 10:53         ` Philippe Mathieu-Daudé
  2022-02-23 11:11           ` Daniel P. Berrangé
@ 2022-02-23 11:50           ` Gerd Hoffmann
  2022-02-24  9:10             ` Igor Mammedov
  1 sibling, 1 reply; 21+ messages in thread
From: Gerd Hoffmann @ 2022-02-23 11:50 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Peter Maydell, Andrew Jones, Daniel P. Berrangé,
	Michael S. Tsirkin, Laszlo Ersek, Oleg Vasilev, Cleber Rosa,
	Richard Henderson, qemu-devel, Idan Horowitz,
	Philippe Mathieu-Daudé,
	Emilio Cota, qemu-arm, Igor Mammedov, Paolo Bonzini,
	Alex Bennée

  Hi,

> > Also, "make install" installs these EDK2 images, which doesn't
> > seem like the right thing for "this is only for one test case".
> 
> Well I'd prefer we never had them installed. Today I don't remember
> why it ended that way.

Probably to behave simliar to other firmware, which makes sense to me.

So maybe do non-debug builds for install and debug builds for the test
cases?  Why do the test cases need debug builds btw?

take care,
  Gerd



^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
  2022-02-23 11:07       ` Daniel P. Berrangé
@ 2022-02-23 13:34         ` Philippe Mathieu-Daudé
  2022-02-23 13:43           ` Daniel P. Berrangé
  2022-02-23 16:37           ` Laszlo Ersek
  0 siblings, 2 replies; 21+ messages in thread
From: Philippe Mathieu-Daudé @ 2022-02-23 13:34 UTC (permalink / raw)
  To: Daniel P. Berrangé
  Cc: Peter Maydell, Andrew Jones, Cleber Rosa, Alex Bennée,
	Oleg Vasilev, Michael S. Tsirkin, Richard Henderson, qemu-devel,
	Idan Horowitz, Philippe Mathieu-Daudé,
	Emilio Cota, Gerd Hoffmann, qemu-arm, Igor Mammedov,
	Paolo Bonzini, Laszlo Ersek

On 23/2/22 12:07, Daniel P. Berrangé wrote:
> On Tue, Feb 22, 2022 at 06:33:41PM +0100, Philippe Mathieu-Daudé wrote:
>> +Igor/MST for UEFI tests.
>>
>> On 22/2/22 17:38, Daniel P. Berrangé wrote:
>>> On Tue, Feb 22, 2022 at 04:17:23PM +0000, Alex Bennée wrote:
>>>>
>>>> Alex Bennée <alex.bennee@linaro.org> writes:
>>>>
>>>>> Hi,
>>>>>
>>>>> TL;DR:
>>>>>
>>>>>     - pc-bios/edk2-aarch64-code.fd should be rebuilt without debug
>>>>
>>>> Laszlo,
>>>>
>>>> Would it be possible to do a less debug enabled version of EDK2 on the
>>>> next update to pc-bios/edk2-*?
>>>
>>> NB, Laszlo is no longer  maintaining EDK2 in QEMU, it was handed
>>> over to Philippe.  I'm CC'ing Gerd too since he's a reviewer and
>>> an EDK2 contributor taking over from Lazslo in EDK2 community
>>
>> We need the DEBUG profile to ensure the bios-tables-tests work.
> 
> Can you elaborate on what bios-tables-tests needs this for, and
> what coverage we would loose by disabling DEBUG.

Maybe it was only required when the tests were developed...
I'll defer that question to Igor.

> It may well be a better tradeoff to sacrifice part of bios-tables-tests
> in favour of shipping more broadly usable images without DEBUG.

Why not, if users are aware/happy to use a unsafe image with various
unfixed CVEs.

Removing the debug profile is as simple as this one-line patch:

-- >8 --
diff --git a/roms/edk2-build.sh b/roms/edk2-build.sh
index d5391c7637..ea79dc27a2 100755
--- a/roms/edk2-build.sh
+++ b/roms/edk2-build.sh
@@ -50,6 +50,6 @@ qemu_edk2_set_cross_env "$emulation_target"
  build \
    --cmd-len=65536 \
    -n "$edk2_thread_count" \
-  --buildtarget=DEBUG \
+  --buildtarget=RELEASE \
    --tagname="$edk2_toolchain" \
    "${args[@]}"
---



^ permalink raw reply related	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
  2022-02-23 13:34         ` Philippe Mathieu-Daudé
@ 2022-02-23 13:43           ` Daniel P. Berrangé
  2022-02-23 14:00             ` Philippe Mathieu-Daudé
  2022-02-23 16:37           ` Laszlo Ersek
  1 sibling, 1 reply; 21+ messages in thread
From: Daniel P. Berrangé @ 2022-02-23 13:43 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Peter Maydell, Andrew Jones, Cleber Rosa, Alex Bennée,
	Oleg Vasilev, Michael S. Tsirkin, Richard Henderson, qemu-devel,
	Idan Horowitz, Philippe Mathieu-Daudé,
	Emilio Cota, Gerd Hoffmann, qemu-arm, Igor Mammedov,
	Paolo Bonzini, Laszlo Ersek

On Wed, Feb 23, 2022 at 02:34:23PM +0100, Philippe Mathieu-Daudé wrote:
> On 23/2/22 12:07, Daniel P. Berrangé wrote:
> > On Tue, Feb 22, 2022 at 06:33:41PM +0100, Philippe Mathieu-Daudé wrote:
> > > +Igor/MST for UEFI tests.
> > > 
> > > On 22/2/22 17:38, Daniel P. Berrangé wrote:
> > > > On Tue, Feb 22, 2022 at 04:17:23PM +0000, Alex Bennée wrote:
> > > > > 
> > > > > Alex Bennée <alex.bennee@linaro.org> writes:
> > > > > 
> > > > > > Hi,
> > > > > > 
> > > > > > TL;DR:
> > > > > > 
> > > > > >     - pc-bios/edk2-aarch64-code.fd should be rebuilt without debug
> > > > > 
> > > > > Laszlo,
> > > > > 
> > > > > Would it be possible to do a less debug enabled version of EDK2 on the
> > > > > next update to pc-bios/edk2-*?
> > > > 
> > > > NB, Laszlo is no longer  maintaining EDK2 in QEMU, it was handed
> > > > over to Philippe.  I'm CC'ing Gerd too since he's a reviewer and
> > > > an EDK2 contributor taking over from Lazslo in EDK2 community
> > > 
> > > We need the DEBUG profile to ensure the bios-tables-tests work.
> > 
> > Can you elaborate on what bios-tables-tests needs this for, and
> > what coverage we would loose by disabling DEBUG.
> 
> Maybe it was only required when the tests were developed...
> I'll defer that question to Igor.
> 
> > It may well be a better tradeoff to sacrifice part of bios-tables-tests
> > in favour of shipping more broadly usable images without DEBUG.
> 
> Why not, if users are aware/happy to use a unsafe image with various
> unfixed CVEs.

Note there's nothing special about EDK2 in regard of CVE fixes (or lack
thereof). The same applies to every other firmware we ship, as well as
QEMU code itself.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
  2022-02-23 13:43           ` Daniel P. Berrangé
@ 2022-02-23 14:00             ` Philippe Mathieu-Daudé
  0 siblings, 0 replies; 21+ messages in thread
From: Philippe Mathieu-Daudé @ 2022-02-23 14:00 UTC (permalink / raw)
  To: Daniel P. Berrangé
  Cc: Peter Maydell, Andrew Jones, Cleber Rosa, Alex Bennée,
	Oleg Vasilev, Michael S. Tsirkin, Richard Henderson, qemu-devel,
	Idan Horowitz, Philippe Mathieu-Daudé,
	Emilio Cota, Gerd Hoffmann, qemu-arm, Igor Mammedov,
	Paolo Bonzini, Laszlo Ersek

On 23/2/22 14:43, Daniel P. Berrangé wrote:
> On Wed, Feb 23, 2022 at 02:34:23PM +0100, Philippe Mathieu-Daudé wrote:
>> On 23/2/22 12:07, Daniel P. Berrangé wrote:
>>> On Tue, Feb 22, 2022 at 06:33:41PM +0100, Philippe Mathieu-Daudé wrote:
>>>> +Igor/MST for UEFI tests.
>>>>
>>>> On 22/2/22 17:38, Daniel P. Berrangé wrote:
>>>>> On Tue, Feb 22, 2022 at 04:17:23PM +0000, Alex Bennée wrote:
>>>>>>
>>>>>> Alex Bennée <alex.bennee@linaro.org> writes:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> TL;DR:
>>>>>>>
>>>>>>>      - pc-bios/edk2-aarch64-code.fd should be rebuilt without debug
>>>>>>
>>>>>> Laszlo,
>>>>>>
>>>>>> Would it be possible to do a less debug enabled version of EDK2 on the
>>>>>> next update to pc-bios/edk2-*?
>>>>>
>>>>> NB, Laszlo is no longer  maintaining EDK2 in QEMU, it was handed
>>>>> over to Philippe.  I'm CC'ing Gerd too since he's a reviewer and
>>>>> an EDK2 contributor taking over from Lazslo in EDK2 community
>>>>
>>>> We need the DEBUG profile to ensure the bios-tables-tests work.
>>>
>>> Can you elaborate on what bios-tables-tests needs this for, and
>>> what coverage we would loose by disabling DEBUG.
>>
>> Maybe it was only required when the tests were developed...
>> I'll defer that question to Igor.
>>
>>> It may well be a better tradeoff to sacrifice part of bios-tables-tests
>>> in favour of shipping more broadly usable images without DEBUG.
>>
>> Why not, if users are aware/happy to use a unsafe image with various
>> unfixed CVEs.
> 
> Note there's nothing special about EDK2 in regard of CVE fixes (or lack
> thereof). The same applies to every other firmware we ship, as well as
> QEMU code itself.

OK, we are good then, thanks.

Phil.


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
  2022-02-23 13:34         ` Philippe Mathieu-Daudé
  2022-02-23 13:43           ` Daniel P. Berrangé
@ 2022-02-23 16:37           ` Laszlo Ersek
  2022-02-23 19:28             ` Peter Maydell
  1 sibling, 1 reply; 21+ messages in thread
From: Laszlo Ersek @ 2022-02-23 16:37 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé, Daniel P. Berrangé
  Cc: Peter Maydell, Andrew Jones, Oleg Vasilev, Idan Horowitz,
	Cleber Rosa, Richard Henderson, qemu-devel,
	Philippe Mathieu-Daudé,
	Emilio Cota, Michael S. Tsirkin, Gerd Hoffmann, qemu-arm,
	Igor Mammedov, Paolo Bonzini, Alex Bennée

On 02/23/22 14:34, Philippe Mathieu-Daudé wrote:
> On 23/2/22 12:07, Daniel P. Berrangé wrote:
>> On Tue, Feb 22, 2022 at 06:33:41PM +0100, Philippe Mathieu-Daudé wrote:
>>> +Igor/MST for UEFI tests.
>>>
>>> On 22/2/22 17:38, Daniel P. Berrangé wrote:
>>>> On Tue, Feb 22, 2022 at 04:17:23PM +0000, Alex Bennée wrote:
>>>>>
>>>>> Alex Bennée <alex.bennee@linaro.org> writes:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> TL;DR:
>>>>>>
>>>>>>     - pc-bios/edk2-aarch64-code.fd should be rebuilt without debug
>>>>>
>>>>> Laszlo,
>>>>>
>>>>> Would it be possible to do a less debug enabled version of EDK2 on the
>>>>> next update to pc-bios/edk2-*?
>>>>
>>>> NB, Laszlo is no longer  maintaining EDK2 in QEMU, it was handed
>>>> over to Philippe.  I'm CC'ing Gerd too since he's a reviewer and
>>>> an EDK2 contributor taking over from Lazslo in EDK2 community
>>>
>>> We need the DEBUG profile to ensure the bios-tables-tests work.
>>
>> Can you elaborate on what bios-tables-tests needs this for, and
>> what coverage we would loose by disabling DEBUG.
> 
> Maybe it was only required when the tests were developed...
> I'll defer that question to Igor.

I've briefly rechecked commits 77db55fc8155 ("tests/uefi-test-tools: add
build scripts", 2019-02-21) and 536d2173b2b3 ("roms: build edk2 firmware
binaries and variable store templates", 2019-04-17). I think my only
reason for picking the DEBUG build target was that other build targets
are generally useless for debugging -- they produce no logs (or fewer logs).

> 
>> It may well be a better tradeoff to sacrifice part of bios-tables-tests
>> in favour of shipping more broadly usable images without DEBUG.
> 
> Why not, if users are aware/happy to use a unsafe image with various
> unfixed CVEs.
> 
> Removing the debug profile is as simple as this one-line patch:
> 
> -- >8 --
> diff --git a/roms/edk2-build.sh b/roms/edk2-build.sh
> index d5391c7637..ea79dc27a2 100755
> --- a/roms/edk2-build.sh
> +++ b/roms/edk2-build.sh
> @@ -50,6 +50,6 @@ qemu_edk2_set_cross_env "$emulation_target"
>  build \
>    --cmd-len=65536 \
>    -n "$edk2_thread_count" \
> -  --buildtarget=DEBUG \
> +  --buildtarget=RELEASE \
>    --tagname="$edk2_toolchain" \
>    "${args[@]}"
> ---
> 

The patch would be larger; the DEBUG build target is included in a bunch
of pathnames (see those original two commits).

BTW I still don't understand the problem with the DEBUG firmware builds;
in the test suite, as many debug messages should be printed as possible,
for helping with the analysis of any new issue that pops up. I've
re-read Alex's message that I got first CC'd on, and I can't connect the
dots, sorry.

Thanks
Laszlo



^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
  2022-02-23 16:37           ` Laszlo Ersek
@ 2022-02-23 19:28             ` Peter Maydell
  0 siblings, 0 replies; 21+ messages in thread
From: Peter Maydell @ 2022-02-23 19:28 UTC (permalink / raw)
  To: Laszlo Ersek
  Cc: Oleg Vasilev, Andrew Jones, Daniel P. Berrangé,
	Cleber Rosa, Idan Horowitz, Michael S. Tsirkin,
	Richard Henderson, qemu-devel, Philippe Mathieu-Daudé,
	Emilio Cota, Philippe Mathieu-Daudé,
	Gerd Hoffmann, qemu-arm, Igor Mammedov, Paolo Bonzini,
	Alex Bennée

On Wed, 23 Feb 2022 at 16:38, Laszlo Ersek <lersek@redhat.com> wrote:
> BTW I still don't understand the problem with the DEBUG firmware builds;
> in the test suite, as many debug messages should be printed as possible,
> for helping with the analysis of any new issue that pops up. I've
> re-read Alex's message that I got first CC'd on, and I can't connect the
> dots, sorry.

As well as the performance question, these images aren't purely
used by the test suite -- we install them for end-users via
'make install'. If we want debug-images for use with the
test suite as well as generally usable ones, we should label
them appropriately and not install them.

-- PMM


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
  2022-02-23 11:50           ` Gerd Hoffmann
@ 2022-02-24  9:10             ` Igor Mammedov
  2022-02-24 11:39               ` Laszlo Ersek
  2022-03-08 12:42               ` Gerd Hoffmann
  0 siblings, 2 replies; 21+ messages in thread
From: Igor Mammedov @ 2022-02-24  9:10 UTC (permalink / raw)
  To: Gerd Hoffmann
  Cc: Peter Maydell, Andrew Jones, Daniel P. Berrangé,
	Michael S. Tsirkin, Laszlo Ersek, Oleg Vasilev, Cleber Rosa,
	Richard Henderson, qemu-devel, Idan Horowitz,
	Philippe Mathieu-Daudé,
	Emilio Cota, Philippe Mathieu-Daudé,
	qemu-arm, Paolo Bonzini, Alex Bennée

On Wed, 23 Feb 2022 12:50:42 +0100
Gerd Hoffmann <kraxel@redhat.com> wrote:

>   Hi,
> 
> > > Also, "make install" installs these EDK2 images, which doesn't
> > > seem like the right thing for "this is only for one test case".  
> > 
> > Well I'd prefer we never had them installed. Today I don't remember
> > why it ended that way.  
> 
> Probably to behave simliar to other firmware, which makes sense to me.
> 
> So maybe do non-debug builds for install and debug builds for the test
> cases?  Why do the test cases need debug builds btw?

wrt bios-tables-test, it doesn't need debug version and should work fine
with non-debug builds.
(if memory serves me right it's this test case that prompted to add
uefi images to qemu)


> 
> take care,
>   Gerd
> 



^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
  2022-02-24  9:10             ` Igor Mammedov
@ 2022-02-24 11:39               ` Laszlo Ersek
  2022-03-08 12:42               ` Gerd Hoffmann
  1 sibling, 0 replies; 21+ messages in thread
From: Laszlo Ersek @ 2022-02-24 11:39 UTC (permalink / raw)
  To: Igor Mammedov, Gerd Hoffmann
  Cc: Peter Maydell, Andrew Jones, Daniel P. Berrangé,
	Michael S. Tsirkin, Idan Horowitz, Oleg Vasilev, Cleber Rosa,
	Richard Henderson, qemu-devel, Philippe Mathieu-Daudé,
	Emilio Cota, Philippe Mathieu-Daudé,
	qemu-arm, Paolo Bonzini, Alex Bennée

On 02/24/22 10:10, Igor Mammedov wrote:
> On Wed, 23 Feb 2022 12:50:42 +0100
> Gerd Hoffmann <kraxel@redhat.com> wrote:
> 
>>   Hi,
>>
>>>> Also, "make install" installs these EDK2 images, which doesn't
>>>> seem like the right thing for "this is only for one test case".  
>>>
>>> Well I'd prefer we never had them installed. Today I don't remember
>>> why it ended that way.  
>>
>> Probably to behave simliar to other firmware, which makes sense to me.
>>
>> So maybe do non-debug builds for install and debug builds for the test
>> cases?  Why do the test cases need debug builds btw?
> 
> wrt bios-tables-test, it doesn't need debug version and should work fine
> with non-debug builds.
> (if memory serves me right it's this test case that prompted to add
> uefi images to qemu)

Yes, commit 77db55fc8155 came first, commit 536d2173b2b3 came later.

Thanks
Laszlo



^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*)
  2022-02-24  9:10             ` Igor Mammedov
  2022-02-24 11:39               ` Laszlo Ersek
@ 2022-03-08 12:42               ` Gerd Hoffmann
  1 sibling, 0 replies; 21+ messages in thread
From: Gerd Hoffmann @ 2022-03-08 12:42 UTC (permalink / raw)
  To: Igor Mammedov
  Cc: Peter Maydell, Andrew Jones, Daniel P. Berrangé,
	Michael S. Tsirkin, Laszlo Ersek, Oleg Vasilev, Cleber Rosa,
	Richard Henderson, qemu-devel, Idan Horowitz,
	Philippe Mathieu-Daudé,
	Emilio Cota, Philippe Mathieu-Daudé,
	qemu-arm, Paolo Bonzini, Alex Bennée

On Thu, Feb 24, 2022 at 10:10:20AM +0100, Igor Mammedov wrote:

> wrt bios-tables-test, it doesn't need debug version and should work fine
> with non-debug builds.

Well, almost.  Using non-debug builds changes the memory layout enough
that we see a change in the memory hotplug ssdt (patch series should
follow later today or tomorrow).

take care,
  Gerd



^ permalink raw reply	[flat|nested] 21+ messages in thread

end of thread, other threads:[~2022-03-08 12:52 UTC | newest]

Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-02-17 11:19 Analysis of slow distro boots in check-avocado (BootLinuxAarch64.test_virt_tcg*) Alex Bennée
2022-02-22 16:17 ` Alex Bennée
2022-02-22 16:38   ` Daniel P. Berrangé
2022-02-22 17:33     ` Philippe Mathieu-Daudé
2022-02-22 17:37       ` Peter Maydell
2022-02-22 18:05         ` Philippe Mathieu-Daudé
2022-02-23  8:48           ` Gerd Hoffmann
     [not found]             ` <87zgmhlwjw.fsf@linaro.org>
2022-02-23 10:58               ` Philippe Mathieu-Daudé
2022-02-23  9:19       ` Peter Maydell
2022-02-23 10:53         ` Philippe Mathieu-Daudé
2022-02-23 11:11           ` Daniel P. Berrangé
2022-02-23 11:50           ` Gerd Hoffmann
2022-02-24  9:10             ` Igor Mammedov
2022-02-24 11:39               ` Laszlo Ersek
2022-03-08 12:42               ` Gerd Hoffmann
2022-02-23 11:07       ` Daniel P. Berrangé
2022-02-23 13:34         ` Philippe Mathieu-Daudé
2022-02-23 13:43           ` Daniel P. Berrangé
2022-02-23 14:00             ` Philippe Mathieu-Daudé
2022-02-23 16:37           ` Laszlo Ersek
2022-02-23 19:28             ` Peter Maydell

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.