Re: Linux man-pages Makefile portability

[Alejandro Colomar <alx.manpages@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 11326 bytes --]

Hi Ingo,

On 6/19/22 23:06, Ingo Schwarze wrote:
> Hi Alejandro,
> 
> Alejandro Colomar wrote on Sun, Jun 19, 2022 at 07:52:33PM +0200:
> 
>> Would you mind checking if the current Makefile works on your OpenBSD?
>> I think it depends on gmake(1),
> 
> After doing
> 
>     $ git clone https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/
>     $ cd man-pages

Thanks for trying!

> 
> i have a hard time finding any targets in the ./Makefile that i could
> possibly run for testing...

Sorry, I need to document the targets.

> 
> I'm certainly not going to run targets like "all" or "install".

The only interesting targets for users are 'dist', 'install', and 
uninstall'.  You could test the install target with DESTDIR=/tmp/somedir

You don't even need to be root to test that in /tmp.

<https://www.gnu.org/prep/standards/html_node/DESTDIR.html>


> I don't think testing the "dist" target makes much sense because
> nobody is going to build your distribution tarballs on OpenBSD.

Makes sense.

> I'm not interested in testing the "src" group of targets;
> i think we discussed in the past that i dislike that whole idea.

I remember.

> 
> The html target could potentially be interesting, but
>    https://savannah.nongnu.org/projects/man2html/
> and the repo linked to from there,
>    https://cvs.savannah.nongnu.org/viewvc/man2html/
> gives me the impression this is abandonware that did not receive
> any maintenance in 19 years.

I used that because the archaic Makefile present in the tree when I 
joined was using it (and I didn't know any other tools):

<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/Makefile?id=b32a6935592118ce4acfddf95e52fca8fbd3aded>

But that Makefile was clearly unused since no-one knows.

If you know a better tool, I could start using it.  Maybe I could use 
groff(1) directly, with grohtml(1).

>  Besides, an OpenBSD port does not
> currently exist and i don't feel like creating one just for testing.
> 
> The only remaining target i can readily see that might be fit for
> testing is "lint", which does seem quite interesting indeed.

Thanks :-)

> 
> But look at this:
> 
>     $ make lint
>     $
> 
> It does absolutely nothing.
> 
> Running "make -d g1 lint" to show the input graph tells me there
> is a rule
> 
>    lint:
>    	@:
> 
> so i'm not surprised it does nothing.
> 
> Next, i tried
> 
>     $ make lint-mandoc
>    make: don't know how to make lint-mandoc
>    Stop in /co/man-pages
> 
> The target simply doesn't exist.

I guess it's due to the use of $(foreach ).  I guess it's a GNU 
extension and make(1POSIX) ignores that creating an empty string.  Since 
the Makefile uses a lot of functions[1], I guess it's not easy to make 
it portable.

The lint part is mostly implemented between lines 432 and 486.

[1]:  addsuffix, wildcard, foreach, filter, patsubst, sort, shell, 
basename, notdir, info.  Not sure how many of those are supported by 
your make(1); maybe none?

> 
> My impression is your Makefile is full of POSIX violations and
> utterly non-portable and gmake(1)-only.  But that's probably OK.
> I mean, the purpose of your project is to document parts of Linux.
> So requiring a tool that almost all Linux users have readily
> available and that users of other systems can install it they
> want to does not seem unreasonable to me.

Yeah, I guess that it's fine, since the extensions are useful, and they 
are virtually everywhere in Linux.

I was wondering if I could make it portable with little changes, but it 
seems not, and not worth even trying.

>  (It's a completely
> different story for a project like GNU troff that is about the
> opposite of Linux only and that basically all UNIX systems want
> to be able to use in production.)

Yes, although they could definitely simplify the build system a little 
bit.  It's one of the most complex builds I've ever seen.

> 
> If i use GNU make instead of our native make, i do get output
> that might be potentially meaningful, even on OpenBSD-current:
> 
>     $ gmake lint-mandoc
>    INSTALL tmp/lint/
>    INSTALL tmp/lint/man0/
>    LINT (mandoc)   tmp/lint/man0/sysexits.h.0.lint.mandoc.touch
>    INSTALL tmp/lint/man1/
>    LINT (mandoc)   tmp/lint/man1/getent.1.lint.mandoc.touch
>    LINT (mandoc)   tmp/lint/man1/iconv.1.lint.mandoc.touch
>    LINT (mandoc)   tmp/lint/man1/intro.1.lint.mandoc.touch
>    LINT (mandoc)   tmp/lint/man1/ldd.1.lint.mandoc.touch
>    LINT (mandoc)   tmp/lint/man1/locale.1.lint.mandoc.touch
>    LINT (mandoc)   tmp/lint/man1/localedef.1.lint.mandoc.touch
>    LINT (mandoc)   tmp/lint/man1/memusage.1.lint.mandoc.touch
>    mandoc: man1/memusage.1:187:2: WARNING: empty block: UR
>    gmake: *** [Makefile:468: tmp/lint/man1/memusage.1.lint.mandoc.touch] Error 2
> 
> That makes sense to me given that i also get
> 
>     $ mandoc -T lint man1/memusage.1
>    mandoc: man1/memusage.1:187:2: WARNING: empty block: UR
> 
> when running the mandoc linter manually.
> 
>> but I'd like to state that in the dependencies
> 
> That sounds reasonable to me.
> 
>> (which I need to write in a new INSTALL file).
> 
> Yes, i think if you want people to test and use that Makefile,
> then documentating in some way what it is supposed to do and
> how it is supposed to be used would make sense.  Sorry if i somehow
> missed the documentation, i looked at README and MAINTAINER_NOTES
> to no avail.

No, you didn't.  I expected autocomplete to help, and targets to be 
obvious in what they mean (as I followed GNU conventions as close as 
possible so that they would be intuitive), but if your make doesn't 
understand my Makefile, autocomplete is obviously not going to help.

Sorry :/

> 
> 
> By the way, this surprised me in the README file:
> 
>    Consider using multiple threads (at least 2) when installing
>    these man pages, as the Makefile is optimized for multiple threads:
>    "make -j install".
> 
> I did not do measurements, but are you sure this is sane advice?
> For an installation procedure, i would have expected execution
> speed to be limited by disk I/O and i would have expected CPU
> consumption to be negligible.  Did you do measurements to confirm
> the relevance of this advice?

Yes, I did:

[
$ sudo make uninstall >/dev/null
$ time sudo make install >/dev/null

real	0m7.722s
user	0m0.004s
sys	0m0.004s
$ sudo make uninstall >/dev/null
$ time sudo make -j2 install >/dev/null

real	0m4.634s
user	0m0.005s
sys	0m0.004s
$ sudo make uninstall >/dev/null
$ time sudo make -j install >/dev/null

real	0m3.170s
user	0m0.000s
sys	0m0.009s
$ touch man2/membarrier.2
$ time sudo make install >/dev/null

real	0m1.766s
user	0m0.005s
sys	0m0.004s
$ touch man2/membarrier.2
$ time sudo make -j2 install >/dev/null

real	0m1.735s
user	0m0.004s
sys	0m0.004s
$ touch man2/membarrier.2
$ time sudo make -j install >/dev/null

real	0m1.737s
user	0m0.001s
sys	0m0.008s
]

You can see that just for the $(shell ...) and all the text functions 
that process the filenames to be installed it is taking almost 1.7 s, 
even if it only needs to update a single page.  That's a lot, but it has 
it's advantages (generating the file list on-the-fly; no ./configure).

Then, the actual installation of the ~2.5k pages (most of them are link 
pages), takes another 1.4 s in multi-process mode, and 6 s in 
single-process mode (so at least 4.6 s that are not I/O).  Maybe it's 
make(1) that has a hard time traversing the tree... I don't know where 
the bottleneck is, but it's clearly there.

> 
> Finally, i don't think it is particularly good style that "make all"
> implies "make install", even if it is clearly documented, which it is.

I completely agree.  That predates me, and I wanted to change it for a 
long time, but I was worried that I might be breaking others' stuff.

But, since you called it, I feel supported to remove that behavior.

> That seems very unusual and not at all what people would expect.
> Usually, "make all" does specifically *not* modify the production
> parts of the system but only *builds* the software such that it can
> later be installed with a separate command.  Arguably, having "make all"
> imply "make uninstall" is even worse.  I don't think anybody would
> expect that running "make all" would attempt to delete stuff from
> their production system!

I guess that the Makefile before I touched is was so basic that you 
could only uninstall everything or install everything, whoever wrote 
that in the Makefile wanted to just type those two with just one command:

<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/Makefile?id=b32a6935592118ce4acfddf95e52fca8fbd3aded>

But that Makefile had one good thing: it was very readable, especially 
compared to mine; but it was no more than a couple of sh(1) scripts 
embedded in it.

> 
>> Maybe, if it's not too hard, we can tweak the Makefile to make it
>> work with make(1POSIX).
> 
> I see lots and lots of syntax in the Makefile where i don't even
> know what it means.  I think if you want to make it POSIX compatible,
> that would likely amount to throwing it all away and starting over
> with a complete rewrite from scratch.  It doesn't look like there is
> much you would be able to keep.

Nah, I think I'll keep it.  Although if I'm shown a simpler POSIX 
Makefile that achieves the same behavior, I'd consider it.

> 
> 
> There is yet another reason why i don't consider it a serious
> issue if the Makefile is for GNU make(1) only.  It is not a rare
> occurence that i use your work - i have repeatedly looked at it
> when working on stuff in OpenBSD to find out how Linux defines
> some features for comparison.

Thanks!  I do check yours too :)

BTW, did you check the changes to queue.3?  I guess you could improve 
yours in a similar manner.

<https://linux-man-pages.blogspot.com/2020/11/man-pages-509-is-released.html>

Also, if you have been following the addition of pages about types, and 
would like to comment, you'll be welcome!

<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=178eaf37e2e971cae88bd4d3f124ede0afbb1015>

>  But i never even noticed there
> is a Makefile in your distribution.

Heh!

>  Commands like
> 
>     $ man -M /co/man-pages open
> 
> work perfectly fine on my system to view the Linux open(2) manual,
> nicely formatted, with no need for installation or a Makefile.
> Even when i put up a copy at
> 
>    https://man.bsd.lv/Linux-5.13/open

How do you generate your HTML pages?  mandoc(1)?  They are nice.

> 
> just copying the eight man* directories and running the makewhatis(8)
> command distributed with mandoc is completely sufficient, no further
> steps are needed for installation, not even on the production web
> server.
> 
> It seems likely to me that other non-Linux users interested in your
> work don't need the Makefile either.

Makes sense.  I'm happy that I didn't break anything by making heavy use 
of GNU extensions.


Cheers,
Alex

-- 
Alejandro Colomar
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]