Re: Compressed man pages (was: Accessibility of man pages (was: Playground pager lsp(1)))

From: Alejandro Colomar <alx.manpages@gmail.com>
To: linux-man <linux-man@vger.kernel.org>
Cc: Alexis <flexibeast@gmail.com>,
	groff@gnu.org, Ingo Schwarze <schwarze@usta.de>,
	Dirk Gouders <dirk@gouders.net>,
	Colin Watson <cjwatson@debian.org>,
	Ralph Corderoy <ralph@inputplus.co.uk>,
	Mingye Wang <arthur200126@gmail.com>,
	Kerin Millar <kfm@plushkava.net>, Sam James <sam@gentoo.org>
Subject: Re: Compressed man pages (was: Accessibility of man pages (was: Playground pager lsp(1)))
Date: Wed, 12 Apr 2023 16:24:42 +0200	[thread overview]
Message-ID: <44768e26-ed92-0562-2318-68fec781126b@gmail.com> (raw)
In-Reply-To: <20230412140451.f03a6c19983694fe844bbb5a@plushkava.net>

[-- Attachment #1.1: Type: text/plain, Size: 7495 bytes --]

Hi all,

After the suggestion by Ralph of trying .lz, Sam's comment about .xz),
and Kerin's comment about tuning the compression parameters, I decided
to try out everything at once, so we can see the effects of the
alternatives.

TL;DR:  For manual pages, use uncompressed source, or gzip(1).
        Everything else is unreasonably slow.

Here go the numbers.  Below, will be a conclusion I get from them.
The following tests have been produced with man-db's man(1) built
from source, since Colin fixed an relevant bug a few days ago[1].
This improves performance considerably compared to the latest
release.

$ sudo make install-man prefix=/opt/local/man/bz2_1 -j LINK_PAGES=symlink Z=.bz2 BZIP2FLAGS=-1 | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/bz2_9 -j LINK_PAGES=symlink Z=.bz2 BZIP2FLAGS=-9 | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/bz2__ -j LINK_PAGES=symlink Z=.bz2 BZIP2FLAGS=   | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/gz__1 -j LINK_PAGES=symlink Z=.gz  GZIPFLAGS=-1  | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/gz__9 -j LINK_PAGES=symlink Z=.gz  GZIPFLAGS=-9  | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/gz___ -j LINK_PAGES=symlink Z=.gz  GZIPFLAGS=    | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/lz__1 -j LINK_PAGES=symlink Z=.lz  LZIPFLAGS=-1  | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/lz__9 -j LINK_PAGES=symlink Z=.lz  LZIPFLAGS=-9  | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/lz___ -j LINK_PAGES=symlink Z=.lz  LZIPFLAGS=    | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/xz__1 -j LINK_PAGES=symlink Z=.xz  XZFLAGS=-1    | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/xz__9 -j LINK_PAGES=symlink Z=.xz  XZFLAGS=-9    | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/xz___ -j LINK_PAGES=symlink Z=.xz  XZFLAGS=      | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/man__ -j LINK_PAGES=symlink Z=                   | wc -l
2571
$ du -sh /opt/local/man/*
5.4M	/opt/local/man/bz2_1
5.4M	/opt/local/man/bz2_9
5.4M	/opt/local/man/bz2__
5.7M	/opt/local/man/gz__1
5.5M	/opt/local/man/gz__9
5.5M	/opt/local/man/gz___
5.5M	/opt/local/man/lz__1
5.4M	/opt/local/man/lz__9
5.4M	/opt/local/man/lz___
9.4M	/opt/local/man/man__
5.5M	/opt/local/man/xz__1
5.4M	/opt/local/man/xz__9
5.4M	/opt/local/man/xz___

$ export MANPATH=/opt/local/man/bz2_1/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 3.15
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do bzip2 -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '"
17; 1.22

$ export MANPATH=/opt/local/man/bz2_9/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 3.15
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do bzip2 -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '"
17; 1.23

$ export MANPATH=/opt/local/man/bz2__/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 3.19
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do bzip2 -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '"
17; 1.23

$ export MANPATH=/opt/local/man/gz__1/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 0.21
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do gzip -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '"
17; 1.16

$ export MANPATH=/opt/local/man/gz__9/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 0.20
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do gzip -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '"
17; 1.17

$ export MANPATH=/opt/local/man/gz___/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 0.20
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do gzip -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '"
17; 1.15

$ export MANPATH=/opt/local/man/lz__1/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 3.95
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do lzip -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '"
17; 1.40

$ export MANPATH=/opt/local/man/lz__9/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 3.93
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do lzip -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '"
17; 1.40

$ export MANPATH=/opt/local/man/lz___/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 3.94
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do lzip -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '"
17; 1.40

$ export MANPATH=/opt/local/man/xz__1/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 3.43
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do xz -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '"
17; 1.24

$ export MANPATH=/opt/local/man/xz__9/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 4.21
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do xz -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '"
17; 1.55

$ export MANPATH=/opt/local/man/xz___/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 4.17
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do xz -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '"
17; 1.55

$ export MANPATH=/opt/local/man/man__/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 0.55
$ /bin/time -f %e dash -c "find $MANPATH -type f | xargs -P0 grep -l RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 0.01

Conclussions:

Any compression formats other than .gz are unreasonably slow.
I'd say either use .gz, or plain text, or prepare to
contribute code yourself to man-db to optimize for your favourite
compression format.

.bz2, .lz, and .xz have similar times, and tuning the compression
doesn't produce important changes in speed (except slightly for
.xz, but I don't see any advantage of using .xz).

Similarly, tuning the compression of .gz doesn't produce
important changes in speed.

Plain text has the advantage that you can use all the power of
Unix tools to search through the source code of the pages
instantaneously, without being restricted to what man(1) allows.

I hope this was useful.

Cheers,
Alex

[1]:  <https://lists.nongnu.org/archive/html/man-db-devel/2023-04/msg00000.html>

-- 
<http://www.alejandro-colomar.es/>
GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]