Confusing Performance Numbers

Confusing Performance Numbers

[Richard Purdie]

I had some ideas I decided to test out. Specifically:

a) stripping -native binaries before putting them in the sysroot
b) compiling -natives with -march=native
c) using hardlinks for sysroot_stage_dir

I was wondering if any of these could give a speedup to the build and if
so, whether it was a useful one. I ran this through our standard
performance script and the results were:

# Baseline
#./perfscript -c d8baa6c89b2f29e8ccc814c61dfd3b106af4fd65
#dax,(nobranch):d8baa6c89b2f29e8ccc814c61dfd3b106af4fd65,poky-10.0.0.final-391-gd8baa6c,38:41.96,11:01.73,5:02.05,4:34.08,0:11.63,0:08.04,0:01.76,25582020,4853400

# staging: Strip native binaries
#./perfscript -c 3e7cab95423c688d7db7355d86b461c1408381d0
#dax,(nobranch):3e7cab95423c688d7db7355d86b461c1408381d0,poky-10.0.0.final-392-g3e7cab9,39:07.56,11:02.05,4:46.99,4:33.99,0:11.63,0:09.40,0:01.73,25398400,4669464

#bitbake.conf: Add -march=native
#./perfscript -c d30b212c8975ad96a35ee6bace731b9823216570    
#dax,(nobranch):d30b212c8975ad96a35ee6bace731b9823216570,poky-10.0.0.final-393-gd30b212,39:16.88,11:41.67,5:05.63,4:31.88,0:11.61,0:07.94,0:01.70,25401000,4670192

# staging: Experiment with hardlinking for sysroot_stage_dir
#./perfscript -c 2feafbbd6338acd52577074db993da4e6d2de03f
#dax,master2:2feafbbd6338acd52577074db993da4e6d2de03f,poky-10.0.0.final-394-g2feafbb,39:49.88,11:11.63,4:41.16,4:38.85,0:11.40,0:07.90,0:01.73,23438704,4669984

so each time, the overall build time seems to rise slightly (first
number). The time to bitbake the kernel remains roughly the same (second
number), maybe the -march=native makes it take fractionally longer?
Stripping the binaries is worth about 200MB of disk use (with and
without rmwork). The hardlink patch is worth about 2GB disk use
(non-rmwork).

I also totalled the time in each task type, the output is below. The
numbers there are interesting as the strip patch seems to add 100s to
do_populate_sysroot. There doesn't seem to be a big difference in
compile time.

Its very hard to figure out what in these tests might be noise and what
if anything was a real gain (other than the disk usage improvements). I
suspect the hardlinking patch may be worthwhile, the other two are
perhaps not useful.

I often get asked what things we've tried so this should go some way to
documenting that and give people in insight into how we can test
changes.

Cheers,

Richard

baseline (master):
do_compile                 4062.83s
do_compile_kernelmodules   69.14s
do_compile_ptest_base      1.09s
do_configure               5455.18s
do_configure_ptest_base    1.11s
do_deploy                  13.34s
do_fetch                   20.4s
do_generate_toolchain_file 0.09s
do_install                 672.76s
do_install_locale          0.24s
do_install_ptest_base      0.72s
do_kernel_checkout         109.76s
do_kernel_configcheck      4.37s
do_kernel_configme         69.8s
do_kernel_link_vmlinux     0.09s
do_multilib_install        0.0s
do_package                 1136.39s
do_package_write_rpm       964.58s
do_packagedata             132.65s
do_patch                   96.29s
do_populate_lic            160.33s
do_populate_sysroot        490.19s
do_rootfs                  178.94s
do_sizecheck               0.08s
do_strip                   0.08s
do_uboot_mkimage           0.09s
do_unpack                  258.55s
do_validate_branches       18.69s
richard@dax:/media/build1/pokyperf/build-perf-test$ /home/richard/buildstats-tasktotals.py results-3e7cab9-20131112082932/buildstats-test1/core-image-sato-qemux86/201311120837/
do_compile                 4108.84s
do_compile_kernelmodules   69.34s
do_compile_ptest_base      0.99s
do_configure               5419.83s
do_configure_ptest_base    1.02s
do_deploy                  13.88s
do_fetch                   37.07s
do_generate_toolchain_file 0.1s
do_install                 628.89s
do_install_locale          0.53s
do_install_ptest_base      0.85s
do_kernel_checkout         47.62s
do_kernel_configcheck      4.25s
do_kernel_configme         94.91s
do_kernel_link_vmlinux     0.15s
do_multilib_install        0.0s
do_package                 1176.35s
do_package_write_rpm       947.94s
do_packagedata             182.74s
do_patch                   101.38s
do_populate_lic            152.46s
do_populate_sysroot        593.62s
do_rootfs                  180.68s
do_sizecheck               0.09s
do_strip                   0.14s
do_uboot_mkimage           0.08s
do_unpack                  290.88s
do_validate_branches       73.12s
richard@dax:/media/build1/pokyperf/build-perf-test$ /home/richard/buildstats-tasktotals.py results-d30b212-20131112093824/buildstats-test1/core-image-sato-qemux86/201311120946/
do_bundle_initramfs        0.09s
do_compile                 4183.91s
do_compile_kernelmodules   69.55s
do_compile_ptest_base      1.1s
do_configure               5535.51s
do_configure_ptest_base    0.98s
do_deploy                  13.37s
do_evacuate_scripts        4.32s
do_fetch                   47.25s
do_generate_toolchain_file 0.09s
do_install                 667.07s
do_install_locale          0.23s
do_install_ptest_base      0.81s
do_kernel_checkout         39.4s
do_kernel_configcheck      4.39s
do_kernel_configme         11.32s
do_kernel_link_vmlinux     0.09s
do_multilib_install        0.0s
do_package                 1176.05s
do_package_write_rpm       987.45s
do_packagedata             162.89s
do_patch                   97.33s
do_populate_lic            127.9s
do_populate_sysroot        594.76s
do_rootfs                  180.98s
do_sizecheck               0.1s
do_strip                   0.12s
do_uboot_mkimage           0.08s
do_unpack                  319.22s
do_validate_branches       12.08s
:/media/build1/pokyperf/build-perf-test$ /home/richard/buildstats-tasktotals.py results-2feafbb-20131112002904/buildstats-test1/core-image-sato-qemux86/201311120037/
do_bundle_initramfs        0.09s
do_compile                 4195.68s
do_compile_kernelmodules   71.07s
do_compile_ptest_base      1.09s
do_configure               5741.49s
do_configure_ptest_base    1.12s
do_deploy                  13.33s
do_evacuate_scripts        0.18s
do_fetch                   18.1s
do_generate_toolchain_file 0.09s
do_install                 697.21s
do_install_locale          0.38s
do_install_ptest_base      0.84s
do_kernel_checkout         143.08s
do_kernel_configcheck      3.85s
do_kernel_configme         60.15s
do_kernel_link_vmlinux     0.09s
do_multilib_install        0.0s
do_package                 1118.97s
do_package_write_rpm       957.41s
do_packagedata             134.04s
do_patch                   101.13s
do_populate_lic            123.11s
do_populate_sysroot        598.03s
do_rootfs                  182.04s
do_sizecheck               0.11s
do_strip                   0.15s
do_uboot_mkimage           0.1s
do_unpack                  286.99s
do_validate_branches       11.91s

Re: Confusing Performance Numbers

[Phil Blundell]

On Tue, 2013-11-12 at 15:31 +0000, Richard Purdie wrote:
> I also totalled the time in each task type, the output is below. The
> numbers there are interesting as the strip patch seems to add 100s to
> do_populate_sysroot. There doesn't seem to be a big difference in
> compile time.

That's probably to be expected: the amount of disk bandwidth you save by
not copying the symbols into the sysroot (and into sstate) is almost
certainly going to be outweighed by the time it takes to strip the files
in the first place.  So that set of numbers at least does seem
believable.

In theory at least, adding -Wl,-s to BUILD_LDFLAGS could give you the
stripping behaviour at lower cost since the linker could avoid writing
out the symbol table in the first place.  But I'm not entirely sure how
this option is actually implemented within the linker (or whether ld.bfd
and ld.gold do the same thing in that respect) and it's entirely
possible that it might be no better in practice.  

Using -march=native for natives is an interesting idea, though even if
this was a win on a single machine it might obviously be a net loss in
the presence of a shared sstate cache.  Of course, the vast majority of
-native binaries are run so seldom that their impact on the overall
build performance is going to be negligible: the ones that dominate are
going to be the toolchain.  One idea that I've seen suggested from time
to time (but never actually tried out) is that it might be a win under
some circumstances to do a 2-stage bootstrap of gcc-native and then use
that gcc-native to build gcc-cross as well as all other native packages.
If you were building everything from scratch every time then it's hard
to imagine this being a real benefit unless your distribution bundled
compiler was spectacularly rubbish, but if you assume that these gccs
will come out of sstate for most builds then it starts to seem more
plausible that this might be worth doing.

p.

Re: Confusing Performance Numbers

[Enrico Scholz]

Richard Purdie
<richard.purdie-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r@public.gmane.org>
writes:

> I had some ideas I decided to test out. Specifically:
> Its very hard to figure out what in these tests might be noise and what
> if anything was a real gain (other than the disk usage improvements). I
> suspect the hardlinking patch may be worthwhile, the other two are
> perhaps not useful.

fwiw, we record performance data as part of our normal build process and
it should be possible to compare these statistics.  Results are e.g.

             https://www.cvg.de/people/ensc/metrics.txt.gz
             https://www.cvg.de/people/ensc/metrics.html

(there is much room to improve the visualization; a new colleague just
started to work on it).  Relevant changes/classes are

  https://www.cvg.de/people/ensc/metrics.py
  https://www.cvg.de/people/ensc/elito-metrics.bbclass
  https://www.cvg.de/people/ensc/0008-build.py-fire-TaskFailed-with-same-environment-as-Ta.patch




Enrico