* Linux man-pages Makefile portability @ 2022-06-19 17:52 Alejandro Colomar 2022-06-19 21:06 ` Ingo Schwarze 0 siblings, 1 reply; 15+ messages in thread From: Alejandro Colomar @ 2022-06-19 17:52 UTC (permalink / raw) To: Ingo Schwarze; +Cc: linux-man [-- Attachment #1.1: Type: text/plain, Size: 455 bytes --] Hi Ingo, Would you mind checking if the current Makefile works on your OpenBSD? I think it depends on gmake(1), but I'd like to state that in the dependencies (which I need to write in a new INSTALL file). Maybe, if it's not too hard, we can tweak the Makefile to make it work with make(1POSIX). <https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/> Thanks, Alex -- Alejandro Colomar <http://www.alejandro-colomar.es/> [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux man-pages Makefile portability 2022-06-19 17:52 Linux man-pages Makefile portability Alejandro Colomar @ 2022-06-19 21:06 ` Ingo Schwarze 2022-06-19 22:23 ` Alejandro Colomar 0 siblings, 1 reply; 15+ messages in thread From: Ingo Schwarze @ 2022-06-19 21:06 UTC (permalink / raw) To: Alejandro Colomar; +Cc: linux-man Hi Alejandro, Alejandro Colomar wrote on Sun, Jun 19, 2022 at 07:52:33PM +0200: > Would you mind checking if the current Makefile works on your OpenBSD? > I think it depends on gmake(1), After doing $ git clone https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/ $ cd man-pages i have a hard time finding any targets in the ./Makefile that i could possibly run for testing... I'm certainly not going to run targets like "all" or "install". I don't think testing the "dist" target makes much sense because nobody is going to build your distribution tarballs on OpenBSD. I'm not interested in testing the "src" group of targets; i think we discussed in the past that i dislike that whole idea. The html target could potentially be interesting, but https://savannah.nongnu.org/projects/man2html/ and the repo linked to from there, https://cvs.savannah.nongnu.org/viewvc/man2html/ gives me the impression this is abandonware that did not receive any maintenance in 19 years. Besides, an OpenBSD port does not currently exist and i don't feel like creating one just for testing. The only remaining target i can readily see that might be fit for testing is "lint", which does seem quite interesting indeed. But look at this: $ make lint $ It does absolutely nothing. Running "make -d g1 lint" to show the input graph tells me there is a rule lint: @: so i'm not surprised it does nothing. Next, i tried $ make lint-mandoc make: don't know how to make lint-mandoc Stop in /co/man-pages The target simply doesn't exist. My impression is your Makefile is full of POSIX violations and utterly non-portable and gmake(1)-only. But that's probably OK. I mean, the purpose of your project is to document parts of Linux. So requiring a tool that almost all Linux users have readily available and that users of other systems can install it they want to does not seem unreasonable to me. (It's a completely different story for a project like GNU troff that is about the opposite of Linux only and that basically all UNIX systems want to be able to use in production.) If i use GNU make instead of our native make, i do get output that might be potentially meaningful, even on OpenBSD-current: $ gmake lint-mandoc INSTALL tmp/lint/ INSTALL tmp/lint/man0/ LINT (mandoc) tmp/lint/man0/sysexits.h.0.lint.mandoc.touch INSTALL tmp/lint/man1/ LINT (mandoc) tmp/lint/man1/getent.1.lint.mandoc.touch LINT (mandoc) tmp/lint/man1/iconv.1.lint.mandoc.touch LINT (mandoc) tmp/lint/man1/intro.1.lint.mandoc.touch LINT (mandoc) tmp/lint/man1/ldd.1.lint.mandoc.touch LINT (mandoc) tmp/lint/man1/locale.1.lint.mandoc.touch LINT (mandoc) tmp/lint/man1/localedef.1.lint.mandoc.touch LINT (mandoc) tmp/lint/man1/memusage.1.lint.mandoc.touch mandoc: man1/memusage.1:187:2: WARNING: empty block: UR gmake: *** [Makefile:468: tmp/lint/man1/memusage.1.lint.mandoc.touch] Error 2 That makes sense to me given that i also get $ mandoc -T lint man1/memusage.1 mandoc: man1/memusage.1:187:2: WARNING: empty block: UR when running the mandoc linter manually. > but I'd like to state that in the dependencies That sounds reasonable to me. > (which I need to write in a new INSTALL file). Yes, i think if you want people to test and use that Makefile, then documentating in some way what it is supposed to do and how it is supposed to be used would make sense. Sorry if i somehow missed the documentation, i looked at README and MAINTAINER_NOTES to no avail. By the way, this surprised me in the README file: Consider using multiple threads (at least 2) when installing these man pages, as the Makefile is optimized for multiple threads: "make -j install". I did not do measurements, but are you sure this is sane advice? For an installation procedure, i would have expected execution speed to be limited by disk I/O and i would have expected CPU consumption to be negligible. Did you do measurements to confirm the relevance of this advice? Finally, i don't think it is particularly good style that "make all" implies "make install", even if it is clearly documented, which it is. That seems very unusual and not at all what people would expect. Usually, "make all" does specifically *not* modify the production parts of the system but only *builds* the software such that it can later be installed with a separate command. Arguably, having "make all" imply "make uninstall" is even worse. I don't think anybody would expect that running "make all" would attempt to delete stuff from their production system! > Maybe, if it's not too hard, we can tweak the Makefile to make it > work with make(1POSIX). I see lots and lots of syntax in the Makefile where i don't even know what it means. I think if you want to make it POSIX compatible, that would likely amount to throwing it all away and starting over with a complete rewrite from scratch. It doesn't look like there is much you would be able to keep. There is yet another reason why i don't consider it a serious issue if the Makefile is for GNU make(1) only. It is not a rare occurence that i use your work - i have repeatedly looked at it when working on stuff in OpenBSD to find out how Linux defines some features for comparison. But i never even noticed there is a Makefile in your distribution. Commands like $ man -M /co/man-pages open work perfectly fine on my system to view the Linux open(2) manual, nicely formatted, with no need for installation or a Makefile. Even when i put up a copy at https://man.bsd.lv/Linux-5.13/open just copying the eight man* directories and running the makewhatis(8) command distributed with mandoc is completely sufficient, no further steps are needed for installation, not even on the production web server. It seems likely to me that other non-Linux users interested in your work don't need the Makefile either. Yours, Ingo ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux man-pages Makefile portability 2022-06-19 21:06 ` Ingo Schwarze @ 2022-06-19 22:23 ` Alejandro Colomar 2022-06-20 13:49 ` Ingo Schwarze 0 siblings, 1 reply; 15+ messages in thread From: Alejandro Colomar @ 2022-06-19 22:23 UTC (permalink / raw) To: Ingo Schwarze; +Cc: linux-man [-- Attachment #1.1: Type: text/plain, Size: 11326 bytes --] Hi Ingo, On 6/19/22 23:06, Ingo Schwarze wrote: > Hi Alejandro, > > Alejandro Colomar wrote on Sun, Jun 19, 2022 at 07:52:33PM +0200: > >> Would you mind checking if the current Makefile works on your OpenBSD? >> I think it depends on gmake(1), > > After doing > > $ git clone https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/ > $ cd man-pages Thanks for trying! > > i have a hard time finding any targets in the ./Makefile that i could > possibly run for testing... Sorry, I need to document the targets. > > I'm certainly not going to run targets like "all" or "install". The only interesting targets for users are 'dist', 'install', and uninstall'. You could test the install target with DESTDIR=/tmp/somedir You don't even need to be root to test that in /tmp. <https://www.gnu.org/prep/standards/html_node/DESTDIR.html> > I don't think testing the "dist" target makes much sense because > nobody is going to build your distribution tarballs on OpenBSD. Makes sense. > I'm not interested in testing the "src" group of targets; > i think we discussed in the past that i dislike that whole idea. I remember. > > The html target could potentially be interesting, but > https://savannah.nongnu.org/projects/man2html/ > and the repo linked to from there, > https://cvs.savannah.nongnu.org/viewvc/man2html/ > gives me the impression this is abandonware that did not receive > any maintenance in 19 years. I used that because the archaic Makefile present in the tree when I joined was using it (and I didn't know any other tools): <https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/Makefile?id=b32a6935592118ce4acfddf95e52fca8fbd3aded> But that Makefile was clearly unused since no-one knows. If you know a better tool, I could start using it. Maybe I could use groff(1) directly, with grohtml(1). > Besides, an OpenBSD port does not > currently exist and i don't feel like creating one just for testing. > > The only remaining target i can readily see that might be fit for > testing is "lint", which does seem quite interesting indeed. Thanks :-) > > But look at this: > > $ make lint > $ > > It does absolutely nothing. > > Running "make -d g1 lint" to show the input graph tells me there > is a rule > > lint: > @: > > so i'm not surprised it does nothing. > > Next, i tried > > $ make lint-mandoc > make: don't know how to make lint-mandoc > Stop in /co/man-pages > > The target simply doesn't exist. I guess it's due to the use of $(foreach ). I guess it's a GNU extension and make(1POSIX) ignores that creating an empty string. Since the Makefile uses a lot of functions[1], I guess it's not easy to make it portable. The lint part is mostly implemented between lines 432 and 486. [1]: addsuffix, wildcard, foreach, filter, patsubst, sort, shell, basename, notdir, info. Not sure how many of those are supported by your make(1); maybe none? > > My impression is your Makefile is full of POSIX violations and > utterly non-portable and gmake(1)-only. But that's probably OK. > I mean, the purpose of your project is to document parts of Linux. > So requiring a tool that almost all Linux users have readily > available and that users of other systems can install it they > want to does not seem unreasonable to me. Yeah, I guess that it's fine, since the extensions are useful, and they are virtually everywhere in Linux. I was wondering if I could make it portable with little changes, but it seems not, and not worth even trying. > (It's a completely > different story for a project like GNU troff that is about the > opposite of Linux only and that basically all UNIX systems want > to be able to use in production.) Yes, although they could definitely simplify the build system a little bit. It's one of the most complex builds I've ever seen. > > If i use GNU make instead of our native make, i do get output > that might be potentially meaningful, even on OpenBSD-current: > > $ gmake lint-mandoc > INSTALL tmp/lint/ > INSTALL tmp/lint/man0/ > LINT (mandoc) tmp/lint/man0/sysexits.h.0.lint.mandoc.touch > INSTALL tmp/lint/man1/ > LINT (mandoc) tmp/lint/man1/getent.1.lint.mandoc.touch > LINT (mandoc) tmp/lint/man1/iconv.1.lint.mandoc.touch > LINT (mandoc) tmp/lint/man1/intro.1.lint.mandoc.touch > LINT (mandoc) tmp/lint/man1/ldd.1.lint.mandoc.touch > LINT (mandoc) tmp/lint/man1/locale.1.lint.mandoc.touch > LINT (mandoc) tmp/lint/man1/localedef.1.lint.mandoc.touch > LINT (mandoc) tmp/lint/man1/memusage.1.lint.mandoc.touch > mandoc: man1/memusage.1:187:2: WARNING: empty block: UR > gmake: *** [Makefile:468: tmp/lint/man1/memusage.1.lint.mandoc.touch] Error 2 > > That makes sense to me given that i also get > > $ mandoc -T lint man1/memusage.1 > mandoc: man1/memusage.1:187:2: WARNING: empty block: UR > > when running the mandoc linter manually. > >> but I'd like to state that in the dependencies > > That sounds reasonable to me. > >> (which I need to write in a new INSTALL file). > > Yes, i think if you want people to test and use that Makefile, > then documentating in some way what it is supposed to do and > how it is supposed to be used would make sense. Sorry if i somehow > missed the documentation, i looked at README and MAINTAINER_NOTES > to no avail. No, you didn't. I expected autocomplete to help, and targets to be obvious in what they mean (as I followed GNU conventions as close as possible so that they would be intuitive), but if your make doesn't understand my Makefile, autocomplete is obviously not going to help. Sorry :/ > > > By the way, this surprised me in the README file: > > Consider using multiple threads (at least 2) when installing > these man pages, as the Makefile is optimized for multiple threads: > "make -j install". > > I did not do measurements, but are you sure this is sane advice? > For an installation procedure, i would have expected execution > speed to be limited by disk I/O and i would have expected CPU > consumption to be negligible. Did you do measurements to confirm > the relevance of this advice? Yes, I did: [ $ sudo make uninstall >/dev/null $ time sudo make install >/dev/null real 0m7.722s user 0m0.004s sys 0m0.004s $ sudo make uninstall >/dev/null $ time sudo make -j2 install >/dev/null real 0m4.634s user 0m0.005s sys 0m0.004s $ sudo make uninstall >/dev/null $ time sudo make -j install >/dev/null real 0m3.170s user 0m0.000s sys 0m0.009s $ touch man2/membarrier.2 $ time sudo make install >/dev/null real 0m1.766s user 0m0.005s sys 0m0.004s $ touch man2/membarrier.2 $ time sudo make -j2 install >/dev/null real 0m1.735s user 0m0.004s sys 0m0.004s $ touch man2/membarrier.2 $ time sudo make -j install >/dev/null real 0m1.737s user 0m0.001s sys 0m0.008s ] You can see that just for the $(shell ...) and all the text functions that process the filenames to be installed it is taking almost 1.7 s, even if it only needs to update a single page. That's a lot, but it has it's advantages (generating the file list on-the-fly; no ./configure). Then, the actual installation of the ~2.5k pages (most of them are link pages), takes another 1.4 s in multi-process mode, and 6 s in single-process mode (so at least 4.6 s that are not I/O). Maybe it's make(1) that has a hard time traversing the tree... I don't know where the bottleneck is, but it's clearly there. > > Finally, i don't think it is particularly good style that "make all" > implies "make install", even if it is clearly documented, which it is. I completely agree. That predates me, and I wanted to change it for a long time, but I was worried that I might be breaking others' stuff. But, since you called it, I feel supported to remove that behavior. > That seems very unusual and not at all what people would expect. > Usually, "make all" does specifically *not* modify the production > parts of the system but only *builds* the software such that it can > later be installed with a separate command. Arguably, having "make all" > imply "make uninstall" is even worse. I don't think anybody would > expect that running "make all" would attempt to delete stuff from > their production system! I guess that the Makefile before I touched is was so basic that you could only uninstall everything or install everything, whoever wrote that in the Makefile wanted to just type those two with just one command: <https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/Makefile?id=b32a6935592118ce4acfddf95e52fca8fbd3aded> But that Makefile had one good thing: it was very readable, especially compared to mine; but it was no more than a couple of sh(1) scripts embedded in it. > >> Maybe, if it's not too hard, we can tweak the Makefile to make it >> work with make(1POSIX). > > I see lots and lots of syntax in the Makefile where i don't even > know what it means. I think if you want to make it POSIX compatible, > that would likely amount to throwing it all away and starting over > with a complete rewrite from scratch. It doesn't look like there is > much you would be able to keep. Nah, I think I'll keep it. Although if I'm shown a simpler POSIX Makefile that achieves the same behavior, I'd consider it. > > > There is yet another reason why i don't consider it a serious > issue if the Makefile is for GNU make(1) only. It is not a rare > occurence that i use your work - i have repeatedly looked at it > when working on stuff in OpenBSD to find out how Linux defines > some features for comparison. Thanks! I do check yours too :) BTW, did you check the changes to queue.3? I guess you could improve yours in a similar manner. <https://linux-man-pages.blogspot.com/2020/11/man-pages-509-is-released.html> Also, if you have been following the addition of pages about types, and would like to comment, you'll be welcome! <https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=178eaf37e2e971cae88bd4d3f124ede0afbb1015> > But i never even noticed there > is a Makefile in your distribution. Heh! > Commands like > > $ man -M /co/man-pages open > > work perfectly fine on my system to view the Linux open(2) manual, > nicely formatted, with no need for installation or a Makefile. > Even when i put up a copy at > > https://man.bsd.lv/Linux-5.13/open How do you generate your HTML pages? mandoc(1)? They are nice. > > just copying the eight man* directories and running the makewhatis(8) > command distributed with mandoc is completely sufficient, no further > steps are needed for installation, not even on the production web > server. > > It seems likely to me that other non-Linux users interested in your > work don't need the Makefile either. Makes sense. I'm happy that I didn't break anything by making heavy use of GNU extensions. Cheers, Alex -- Alejandro Colomar <http://www.alejandro-colomar.es/> [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux man-pages Makefile portability 2022-06-19 22:23 ` Alejandro Colomar @ 2022-06-20 13:49 ` Ingo Schwarze 2022-07-03 21:44 ` Alejandro Colomar 0 siblings, 1 reply; 15+ messages in thread From: Ingo Schwarze @ 2022-06-20 13:49 UTC (permalink / raw) To: Alejandro Colomar; +Cc: linux-man Hi Alejandro, Alejandro Colomar wrote on Mon, Jun 20, 2022 at 12:23:45AM +0200: > On 6/19/22 23:06, Ingo Schwarze wrote: >> The html target could potentially be interesting, but >> https://savannah.nongnu.org/projects/man2html/ >> and the repo linked to from there, >> https://cvs.savannah.nongnu.org/viewvc/man2html/ >> gives me the impression this is abandonware that did not receive >> any maintenance in 19 years. > I used that because the archaic Makefile present in the tree when I > joined was using it (and I didn't know any other tools): > > <https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/Makefile?id=b32a6935592118ce4acfddf95e52fca8fbd3aded> I see, now it makes sense to me. I'm relieved i won't have to dig into man2html. :-) > But that Makefile was clearly unused since no-one knows. I'm not saying "noone knows"; some packaging tools on some Linux distributions might very well rely on the Makefile - i don't know. All i'm saying is i don't readily see a reason why people *not* running Linux might need the Makefile... > If you know a better tool, I could start using it. Maybe I could use > groff(1) directly, with grohtml(1). I don't think using groff to generate HTML output from manual pages is a good idea. It generates low-quality HTML code, and it is impossible to fix because of the basic concept how groff works. You see, both mdoc(7) and modern HTML are semantic markup languages. Groff, however, does the transformation in a two-step process. In the first step, it throws away *all* the semantic information, retaining only *physical* formatting cues. And then in the second step, it tries to generate HTML from those physical formatting instructions. That cannot possibly work. At best it might generate early-1990ies-style HTML 1 or HTML 2 code with no support for CSS whatsoever. That's an unavoidable consequence of the basic design of grohtml(1). For man(7), the problem is not quite as bad as for mdoc(7) because the man(7) language is a 1970ies-style, purely presentational markup language that conveys no semantic information in the first place, so there isn't really much that could get lost during the intermediate step. Then again, i think nowadays, it makes more sense to use a converter that can handle *both* languages well instead of one that is kind of nearly good enough for man(7) but really inadequate for mdoc(7). [...] > I guess it's due to the use of $(foreach ). I guess it's a GNU > extension and make(1POSIX) ignores that creating an empty string. Since > the Makefile uses a lot of functions[1], I guess it's not easy to make > it portable. [...] > [1]: addsuffix, wildcard, foreach, filter, patsubst, sort, shell, > basename, notdir, info. Not sure how many of those are supported by > your make(1); maybe none? The OpenBSD implementation of make(1) is much more powerful than POSIX make, but according to the manual page, you are right that none of these keywords are supported by OpenBSD make, let alone by POSIX make. >> (It's a completely >> different story for a project like GNU troff that is about the >> opposite of Linux only and that basically all UNIX systems want >> to be able to use in production.) > Yes, although they could definitely simplify the build system a little > bit. It's one of the most complex builds I've ever seen. I wouldn't go that far; i guess then you never looked at build systems of large software like X11 or firefox or boost... Even the OpenBSD ports build framework is more complicated than the groff build system. Then again, the groff build system *does* have a lot of abstraction, indirection, and complexity *for such a small piece of software*. :-/ But we're disgressing. [ ... regarding installation instructions ...] > No, you didn't. I expected autocomplete to help, I almost never use autocomplete except for the names of commands that are installed system-wide and for the names of files in the local file system. I didn't even know it is possible to use autocompletion for make targets, and i dislike the idea. But don't worry! Your build system *is* complicated for a package that actually doesn't need to build anything, but not so bad that i didn't find what i looked for. :) >> I did not do measurements, but are you sure this is sane advice? >> For an installation procedure, i would have expected execution >> speed to be limited by disk I/O and i would have expected CPU >> consumption to be negligible. Did you do measurements to confirm >> the relevance of this advice? > Yes, I did: > > [ > $ sudo make uninstall >/dev/null > $ time sudo make install >/dev/null > > real 0m7.722s > user 0m0.004s > sys 0m0.004s > $ sudo make uninstall >/dev/null > $ time sudo make -j2 install >/dev/null > > real 0m4.634s > user 0m0.005s > sys 0m0.004s > $ sudo make uninstall >/dev/null > $ time sudo make -j install >/dev/null > > real 0m3.170s > user 0m0.000s > sys 0m0.009s > $ touch man2/membarrier.2 > $ time sudo make install >/dev/null > > real 0m1.766s > user 0m0.005s > sys 0m0.004s > $ touch man2/membarrier.2 > $ time sudo make -j2 install >/dev/null > > real 0m1.735s > user 0m0.004s > sys 0m0.004s > $ touch man2/membarrier.2 > $ time sudo make -j install >/dev/null > > real 0m1.737s > user 0m0.001s > sys 0m0.008s > ] > > You can see that just for the $(shell ...) and all the text functions > that process the filenames to be installed it is taking almost 1.7 s, > even if it only needs to update a single page. As a side remark, i consider it bad style to use dependencies during installation: dependencies are for the build stage, not for the installation stage. When i say "make install", i just want *all* the files installed unconditionally for two reasons: On the one hand, dependency handling is error-prone and it would be bad if some file does not get installed due to the notorious problem of oversights in dependency handling (and dependency handling in parallel Makefiles is even more fragile than in serial ones). On the other hand, "make install" also has the purpose of repairing an installation that got broken in some way or other, and skipping some files because the build system *thinks* they are probably still installed properly defeats the purpose IMHO. > That's a lot, but it has > it's advantages (generating the file list on-the-fly; no ./configure). > > Then, the actual installation of the ~2.5k pages (most of them are link > pages), As another aside, i consider using .so bad style. It is unnecessarily fragile. Using hard links on the file system level (see ln(1)) is significantly more robust. With mandoc(1), you don't need links at all, but i admit traditional man(1) implementations including man-db still require them for manual pages having more than one name. > takes another 1.4 s in multi-process mode, and 6 s in > single-process mode (so at least 4.6 s that are not I/O). Maybe it's > make(1) that has a hard time traversing the tree... I don't know where > the bottleneck is, but it's clearly there. I see. So you need multiple processors purely for dealing with make(1) overhead... Gee! :-/ [...] > BTW, did you check the changes to queue.3? I guess you could improve > yours in a similar manner. > > <https://linux-man-pages.blogspot.com/2020/11/man-pages-509-is-released.html> I think in OpenBSD, these changes would get vetoed by large numbers of developers because they violate the way OpenBSD manual pages are organized in several ways: 1. Your queue(7) manual page is placed in the wrong section. It is purely about an API provided by a library for the C language. Such information unambiguously belongs in section 3 and certainly not in section 7. It is not even an edge case; it is perfectly clear what the correct section is. 2. Your file names and .TH names violate the OpenBSD convention that section 2 and 3 manual pages must be named after functions or macros. For example, the page name "slist" is not acceptable because no sname() function or macro exists. 3. Splitting the page up into multiple pages is a bad idea for two reasons: it results in significant duplication of information and it splits information about interfaces so closely related to each other that most of their features are identical across multiple pages. You would have no chance of getting anything like that committed to OpenBSD. > Also, if you have been following the addition of pages about types, and > would like to comment, you'll be welcome! > > <https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=178eaf37e2e971cae88bd4d3f124ede0afbb1015> BSD doesn't have manual pages about types, and i don't think there is a significant benefit from having them. Most standard types are trivial and can easily be looked up in the header files with no need for separate documentation. In the unusual case that a type has non-trivial syntax and/or semantics, it can be documented in the manual page of the most closely related API function; for example, "struct pollfd" is documented in our poll(2) page. That makes it easy to find with the usual "man -k Vt=typename" search command. Documenting a non-trivial type separately from the functions using it is counter-productive and makes the documentation hard to read because programmers *never* need to use a type defined by a library unless they also want to use related API functions. Actually, i find it better to *not* add type names as names of manual pages because that way, the classical syntax "man functname" can be used to search for function and macro names and the advanced syntax "man -k Vt=typename" to search for types, with less potential for confusion. So in OpenBSD, your pages about types would get vetoed on the grounds of "pages not named after functions or macros" as well as on the grounds of "these pages do not document any function or macro; instead of creating a new page, put the information where it belongs." That said, other projects are of course free to have such pages if they want to. The mandoc(1) program is also able to handle paths like "man3/id_t.3type". It will consider that page to be *both* in section "3" (as specified by the directory name) and in section "3type" (as specified by the file name and by the .TH macro). I would consider it better style to keep section names consistent, i.e. to use either "man3/id_t.3" .TH id_t 3 or "man3type/id_t.3type" .TH id_t 3type, but it's not a big deal: since many systems (in particular various Linux distros) suffer from such inconsistencies, handling such inconsistencies gracefully is an important feature that certainly won't get removed. >> Commands like >> >> $ man -M /co/man-pages open >> >> work perfectly fine on my system to view the Linux open(2) manual, >> nicely formatted, with no need for installation or a Makefile. >> Even when i put up a copy at >> >> https://man.bsd.lv/Linux-5.13/open > How do you generate your HTML pages? mandoc(1)? They are nice. https://man.openbsd.org/mandoc.1#HTML_Output https://man.openbsd.org/man.cgi.8 I think that's the usual way to generate HTML from manual pages nowadays. The following sites also use mandoc for HTML output: * https://www.freebsd.org/cgi/man.cgi * https://manpages.debian.org/ * https://man.archlinux.org/ * https://man.voidlinux.org/ Some of these have their own CGI handling and/or database code, but they all use the mandoc parser and HTML generator. Yours, Ingo ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux man-pages Makefile portability 2022-06-20 13:49 ` Ingo Schwarze @ 2022-07-03 21:44 ` Alejandro Colomar 2022-07-21 14:17 ` Alejandro Colomar 0 siblings, 1 reply; 15+ messages in thread From: Alejandro Colomar @ 2022-07-03 21:44 UTC (permalink / raw) To: Ingo Schwarze; +Cc: linux-man, G . Branden Robinson [-- Attachment #1.1: Type: text/plain, Size: 17436 bytes --] [added Branden, as he was involved in discussions regarding man3type; Branden, you might want to visit this thread from the begining, as I only copied the minimum to reply; it's in linux-man@] Hi, Ingo! and Branden! On 6/20/22 15:49, Ingo Schwarze wrote: > >> But that Makefile was clearly unused since no-one knows. > > I'm not saying "noone knows"; some packaging tools on some Linux > distributions might very well rely on the Makefile - i don't know. Oh, Michael seemed a bit surprised that I started patching the Makefile, as if the features I was patching hadn't been used in a very long time. Anyway, I fixed `make all`. I never liked it. Now it builds all that it can build, which now is HTML pages, and in the future will probably include PDF pages too. It was that, or make it a no-op, so I thought HTML+PDF was more useful. > All i'm saying is i don't readily see a reason why people *not* > running Linux might need the Makefile... Well, the Makefile is basically meant to install (copy) the files to the system, so since you copy them to somewhere, `gmake install mandir=...` should work, but cp(1) -r tends to be as useful for such a simple package. > >> If you know a better tool, I could start using it. Maybe I could use >> groff(1) directly, with grohtml(1). > > I don't think using groff to generate HTML output from manual pages > is a good idea. It generates low-quality HTML code, and it is > impossible to fix because of the basic concept how groff works. [...] Hmm, I'll try both and see. Thanks. > > [...] >> I guess it's due to the use of $(foreach ). I guess it's a GNU >> extension and make(1POSIX) ignores that creating an empty string. Since >> the Makefile uses a lot of functions[1], I guess it's not easy to make >> it portable. > [...] >> [1]: addsuffix, wildcard, foreach, filter, patsubst, sort, shell, >> basename, notdir, info. Not sure how many of those are supported by >> your make(1); maybe none? > > The OpenBSD implementation of make(1) is much more powerful than > POSIX make, but according to the manual page, you are right that > none of these keywords are supported by OpenBSD make, let alone by > POSIX make. Heh, then to be compatible with BSD make(1) I guess I'd have to hardcode the page names, or use suffix rules, none of which convinces me. Maybe suffix rules could work, but I'd have to stop testing the EXAMPLES, which is the most complex part of the Makefile. I'll consider using them, or at least Substitution References[1] if it adds compatibility, at least for the simplest tasks, such as `make install`. [1]: <https://www.gnu.org/software/make/manual/html_node/Substitution-Refs.html#Substitution-Refs> >> No, you didn't. I expected autocomplete to help, > > I almost never use autocomplete except for the names of commands that are > installed system-wide and for the names of files in the local file system. Oh, I use autocomplete everyday for things like arguments to git commands, and I really feel it when I'm in a system where I don't have such help. > I didn't even know it is possible to use autocompletion for make targets, > and i dislike the idea. But don't worry! Your build system *is* > complicated for a package that actually doesn't need to build anything, > but not so bad that i didn't find what i looked for. :) :) > As a side remark, i consider it bad style to use dependencies during > installation: dependencies are for the build stage, not for the > installation stage. When i say "make install", i just want *all* > the files installed unconditionally for two reasons: On the one hand, > dependency handling is error-prone and it would be bad if some file > does not get installed due to the notorious problem of oversights in > dependency handling (and dependency handling in parallel Makefiles > is even more fragile than in serial ones). On the other hand, "make > install" also has the purpose of repairing an installation that got > broken in some way or other, and skipping some files because the > build system *thinks* they are probably still installed properly > defeats the purpose IMHO. I've had doubts about that, and in the past I tended to do the same as you suggest, not because of fear to broken deps, but for making sure I don't create temporary files owned by root. But in this case, where there are thousands of files to install, there's an important time difference between installing just the diff and installing the whole repo, so I asked the following question[2] just to confirm my doubts, and added the deps. Regarding the possibility of broken deps, I believe the solution is to fix the Makefile, not to assume that it can't be done right and make it dumb; and I try hard to make sure my Makefiles work in multi-process mode. There's always a chance that I got some corner case wrong, but this case it's pretty low (and if someone doesn't trust my Makefiles to behave well with -j, I don't force to use it, but I recommend it very much :)). Regarding `make install` having a secondary purpose of being kind of a reinstall, I disagree. I tend to write an explicit `make reinstall` target for that purpose (implemented as `\t$(MAKE) uninstall\n\t$(MAKE) install`); I didn't write it yet for the man-pages, but I'm going to add it now. [2]: <https://stackoverflow.com/questions/70901364/should-make-install-depend-on-compilation> > >> That's a lot, but it has >> it's advantages (generating the file list on-the-fly; no ./configure). >> >> Then, the actual installation of the ~2.5k pages (most of them are link >> pages), > > As another aside, i consider using .so bad style. It is unnecessarily > fragile. Using hard links on the file system level (see ln(1)) > is significantly more robust. With mandoc(1), you don't need links > at all, but i admit traditional man(1) implementations including > man-db still require them for manual pages having more than one name. I also had that feeling at first. I just leave it there because of "don't fix it if it ain't broke" and it just works. .so has a good side, which is that the Makefile is simpler, as it doesn't need to create links. > >> takes another 1.4 s in multi-process mode, and 6 s in >> single-process mode (so at least 4.6 s that are not I/O). Maybe it's >> make(1) that has a hard time traversing the tree... I don't know where >> the bottleneck is, but it's clearly there. > > I see. So you need multiple processors purely for dealing with make(1) > overhead... Gee! :-/ Yupee! :/ I'll see if I can reduce that overhead without losing features. Maybe I improve compatibility in the way. :) > > [...] >> BTW, did you check the changes to queue.3? I guess you could improve >> yours in a similar manner. >> >> <https://linux-man-pages.blogspot.com/2020/11/man-pages-509-is-released.html> > > I think in OpenBSD, these changes would get vetoed by large numbers > of developers because they violate the way OpenBSD manual pages are > organized in several ways: > > 1. Your queue(7) manual page is placed in the wrong section. > It is purely about an API provided by a library for the C language. > Such information unambiguously belongs in section 3 and certainly > not in section 7. It is not even an edge case; it is perfectly > clear what the correct section is. See 2. > > 2. Your file names and .TH names violate the OpenBSD convention > that section 2 and 3 manual pages must be named after functions > or macros. For example, the page name "slist" is not acceptable > because no sname() function or macro exists. Heh, I agree! I would have put them in section 7, but I was new to the project, and didn't want to change things too much at the time. Since queue(3) was in man3, I kept the tradition, and the child pages were kept in man3. Probably I should have put them in man7, but blame history, not me :) Buuut, is it me, or I see a contradiction with point 1, which claims that queue(3) should be in man3? We don't have a slist() function/macro, but we don't have a queue() one either (maybe historically there was one and I don't know it, but I guess not. My systems say: alx@devuan:/usr/include$ grepc queue alx@devuan:/usr/include$ alx@debian:/usr/include$ grepc queue alx@debian:/usr/include$ Should now queue() be in man3 or man7? > > 3. Splitting the page up into multiple pages is a bad idea for > two reasons: it results in significant duplication of information > and it splits information about interfaces so closely related > to each other that most of their features are identical across > multiple pages. Actually, I didn't duplicate information at all, AFAIR. It was already _very_ duplicated in the same queue(3) page, so I just splitted it at the right points. I only had to cut the page into many little ones, then translate the pages from mdoc(7) to man(7), and then fix minor style issues. See: $ wc -l man3/circleq.3 man3/list.3 man3/slist.3 man3/stailq.3 man3/tailq.3 man7/queue.7 318 man3/circleq.3 306 man3/list.3 317 man3/slist.3 375 man3/stailq.3 395 man3/tailq.3 133 man7/queue.7 1844 total $ git checkout man-pages-5.08 >/dev/null 2>&1 $ wc -l man3/queue.3 1231 man3/queue.3 The difference is just source code overhead; the text is almost the same. Maybe you could still simplify your queue(7) page in a different way, without splitting it; it is very repetitive. > > You would have no chance of getting anything like that committed to > OpenBSD. Heh, I know. The only thing that was well received from my side in that list was a bug report about some exec(3) function (about alloca(3)). > >> Also, if you have been following the addition of pages about types, and >> would like to comment, you'll be welcome! >> >> <https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=178eaf37e2e971cae88bd4d3f124ede0afbb1015> > > BSD doesn't have manual pages about types, and i don't think there is a > significant benefit from having them. Most standard types are trivial > and can easily be looked up in the header files with no need for separate > documentation. Oh, yes, I envy your headers. They are really readable! But glibc source code is not as friendly, and it took me a long long time to get used to search things in those headers (I still can't find some things in their code; function definitions, for example, are very cryptic in some cases). Also, some programmers, especially when starting (but I know of many programmers that are "senior" and still have serious issues with types), would benefit from documentation specific about types. That would help understand their limitations, and what a type is appropriate for or not. > In the unusual case that a type has non-trivial syntax > and/or semantics, it can be documented in the manual page of the most > closely related API function; for example, "struct pollfd" is documented > in our poll(2) page. > That makes it easy to find with the usual > "man -k Vt=typename" search command. Oh, that's where man(7) sucks :) I wouldn't need these: $ sed -n /^man_lsfunc/,/^}/p <scripts/bash_aliases man_lsfunc() { if [ $# -lt 1 ]; then >&2 echo "Usage: ${FUNCNAME[0]} <manpage|manNdir>..."; return $EX_USAGE; fi for arg in "$@"; do man_section "$arg" 'SYNOPSIS'; done \ |sed_rm_ccomments \ |pcregrep -Mn '(?s)^ [\w ]+ \**\w+\([\w\s(,)[\]*]*?(...)?\s*\); *$' \ |grep '^[0-9]' \ |sed -E 's/syscall\(SYS_(\w*),?/\1(/' \ |sed -E 's/^[^(]+ \**(\w+)\(.*/\1/' \ |uniq; } $ sed -n /^man_lsvar/,/^}/p <scripts/bash_aliases man_lsvar() { if [ $# -lt 1 ]; then >&2 echo "Usage: ${FUNCNAME[0]} <manpage|manNdir>..."; return $EX_USAGE; fi for arg in "$@"; do man_section "$arg" 'SYNOPSIS'; done \ |sed_rm_ccomments \ |pcregrep -Mv '(?s)^ [\w ]+ \**\w+\([\w\s(,)[\]*]+?(...)?\s*\); *$' \ |pcregrep -Mn \ -e '(?s)^ +extern [\w ]+ \**\(\*+[\w ]+\)\([\w\s(,)[\]*]+?\s*\); *$' \ -e '^ +extern [\w ]+ \**[\w ]+; *$' \ |grep '^[0-9]' \ |grep -v 'typedef' \ |sed -E 's/^[0-9]+: +extern [^(]+ \**\(\*+(\w* )?(\w+)\)\(.*/\2/' \ |sed 's/^[0-9]\+: \+extern .* \**\(\w\+\); */\1/' \ |uniq; } And they're not perfect... > > Documenting a non-trivial type separately from the functions using it > is counter-productive and makes the documentation hard to read because > programmers *never* need to use a type defined by a library unless they > also want to use related API functions. That's not true. I've needed (or better phrased, wanted) types, even when I wasn't using any APIs that used them. The reason was that I was designing an API, and wanted to use the most appropriate types for my functions. Having had documentation about types would have helped *a lot* at the time. Many programmers don't know all the differences between size_t and ssize_t, and for example that ISO C only provides one of them (the other is added by POSIX), and I know of one *great* programmer that learnt the difference between those types from me a few months ago :). One could search in the standard documents about the types, but I guess we will agree that those documents are not very friendly, especially for beginners. Another case where I've found my type pages very useful was when I contributed[3] to iwyu(1)[4]. Having to read the POSIX or ISO C documents would have been crazy (I had to do it anyway, to write the pages, but I don't want to repeat that process again ;)). [3]: <https://github.com/include-what-you-use/include-what-you-use/pull/930> [4]: <https://include-what-you-use.org/> > Actually, i find it better > to *not* add type names as names of manual pages because that way, > the classical syntax "man functname" can be used to search for > function and macro names and the advanced syntax "man -k Vt=typename" > to search for types, with less potential for confusion. We don't have that feature in man(7), so the closest thing that I do is to grep in the glibc and BSDs source code with grepc(1)[3], and also `grep -rn ...` inside the man-pages repo. [5]: <http://www.alejandro-colomar.es/src/alx/alx/grepc.git/> > > So in OpenBSD, your pages about types would get vetoed on the grounds > of "pages not named after functions or macros" as well as on the > grounds of "these pages do not document any function or macro; > instead of creating a new page, put the information where it belongs." The first argument, I agree, and it's why I didn't use section 3, but subsection 3type. The second, I disagree for the reasons above, but can understand why others might disagree with me. Maybe I can convince you :) > > That said, other projects are of course free to have such pages if > they want to. The mandoc(1) program is also able to handle paths like > "man3/id_t.3type". It will consider that page to be *both* in section > "3" (as specified by the directory name) and in section "3type" (as > specified by the file name and by the .TH macro). I would consider > it better style to keep section names consistent, i.e. to use either > "man3/id_t.3" .TH id_t 3 or "man3type/id_t.3type" .TH id_t 3type, > but it's not a big deal: since many systems (in particular various > Linux distros) suffer from such inconsistencies, handling such > inconsistencies gracefully is an important feature that certainly > won't get removed. I considered[6] using man3type, and used man3 in the end just because when in doubt I opted for the smallest change. Knowing that it breaks mandoc(1), I'll definitely move to <man3type/>. [6]: <https://lore.kernel.org/linux-man/761bb12f-31e0-369d-8315-d2e1545505c7@gmail.com/T/#u> > >>> Commands like >>> >>> $ man -M /co/man-pages open >>> >>> work perfectly fine on my system to view the Linux open(2) manual, >>> nicely formatted, with no need for installation or a Makefile. >>> Even when i put up a copy at >>> >>> https://man.bsd.lv/Linux-5.13/open Yes, since there's no compilation, `make install` is basically a wrapper around `cp -r`. It has nice features, such as reduced install time by checking timestamps, but that's more useful to me as a maintainer (since I install several times a minute in some cases), and not so much for end users, where a few seconds are not important. > >> How do you generate your HTML pages? mandoc(1)? They are nice. > > https://man.openbsd.org/mandoc.1#HTML_Output > https://man.openbsd.org/man.cgi.8 > > I think that's the usual way to generate HTML from manual pages > nowadays. The following sites also use mandoc for HTML output: > > * https://www.freebsd.org/cgi/man.cgi > * https://manpages.debian.org/ > * https://man.archlinux.org/ > * https://man.voidlinux.org/ > > Some of these have their own CGI handling and/or database code, > but they all use the mandoc parser and HTML generator. Interesting. Thanks! Cheers, Alex -- Alejandro Colomar <http://www.alejandro-colomar.es/> [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux man-pages Makefile portability 2022-07-03 21:44 ` Alejandro Colomar @ 2022-07-21 14:17 ` Alejandro Colomar 2022-07-22 16:59 ` Ingo Schwarze 0 siblings, 1 reply; 15+ messages in thread From: Alejandro Colomar @ 2022-07-21 14:17 UTC (permalink / raw) To: Ingo Schwarze; +Cc: linux-man, G . Branden Robinson [-- Attachment #1.1: Type: text/plain, Size: 1827 bytes --] Hi Ingo! On 7/3/22 23:44, Alejandro Colomar wrote: > [added Branden, as he was involved in discussions regarding man3type; > Branden, you might want to visit this thread from the begining, as I > only copied the minimum to reply; it's in linux-man@] > > On 6/20/22 15:49, Ingo Schwarze wrote: [...] >> That said, other projects are of course free to have such pages if >> they want to. The mandoc(1) program is also able to handle paths like >> "man3/id_t.3type". It will consider that page to be *both* in section >> "3" (as specified by the directory name) and in section "3type" (as >> specified by the file name and by the .TH macro). I would consider >> it better style to keep section names consistent, i.e. to use either >> "man3/id_t.3" .TH id_t 3 or "man3type/id_t.3type" .TH id_t 3type, >> but it's not a big deal: since many systems (in particular various >> Linux distros) suffer from such inconsistencies, handling such >> inconsistencies gracefully is an important feature that certainly >> won't get removed. > > I considered[6] using man3type, and used man3 in the end just because > when in doubt I opted for the smallest change. Knowing that it breaks > mandoc(1), I'll definitely move to <man3type/>. > > [6]: > <https://lore.kernel.org/linux-man/761bb12f-31e0-369d-8315-d2e1545505c7@gmail.com/T/#u> > I fixed it: <https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=451a27a78d51973b01bfb5d3b1e0ec081d9161e1> And Debian seems to work fine with man3type/ and man2type/ out of the box, so I prefer it this way. I hope that other projects follow the example; and that packagers/distributions also create subsection directories, and don't undo my work. Cheers, Alex -- Alejandro Colomar <http://www.alejandro-colomar.es/> [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux man-pages Makefile portability 2022-07-21 14:17 ` Alejandro Colomar @ 2022-07-22 16:59 ` Ingo Schwarze 2022-07-22 17:37 ` Alejandro Colomar 0 siblings, 1 reply; 15+ messages in thread From: Ingo Schwarze @ 2022-07-22 16:59 UTC (permalink / raw) To: Alejandro Colomar; +Cc: linux-man, g.branden.robinson Hi Alejandro, Alejandro Colomar wrote on Thu, Jul 21, 2022 at 04:17:18PM +0200: > On 7/3/22 23:44, Alejandro Colomar wrote: >> [added Branden, as he was involved in discussions regarding man3type; >> Branden, you might want to visit this thread from the begining, as I >> only copied the minimum to reply; it's in linux-man@] >> On 6/20/22 15:49, Ingo Schwarze wrote: > [...] >>> That said, other projects are of course free to have such pages if >>> they want to. The mandoc(1) program is also able to handle paths like >>> "man3/id_t.3type". It will consider that page to be *both* in section >>> "3" (as specified by the directory name) and in section "3type" (as >>> specified by the file name and by the .TH macro). I would consider >>> it better style to keep section names consistent, i.e. to use either >>> "man3/id_t.3" .TH id_t 3 or "man3type/id_t.3type" .TH id_t 3type, >>> but it's not a big deal: since many systems (in particular various >>> Linux distros) suffer from such inconsistencies, handling such >>> inconsistencies gracefully is an important feature that certainly >>> won't get removed. >> I considered[6] using man3type, and used man3 in the end just because >> when in doubt I opted for the smallest change. Knowing that it breaks >> mandoc(1), I'll definitely move to <man3type/>. I didn't mean to say man3/id_t.3type "breaks mandoc". Quite to the contrary, the above quotation explains that mandoc copes with it. However, when it comes to robustness with respect to *other* man(1) implementations apart from mandoc and man-db, i suspect the most portable and reliable way is using man1 - man9 only with no suffixes, consistent suffixes like "man3type/id_t.3type" are probably medium portability and medium reliability, and inconsistent suffixes like "man3/id_t.3type" and "man3type/id_t.3" are likely the least portable and the most fragile. So your change is an improvement. The system making the heaviest use of section suffixes i'm aware of is Solaris: > uname -a SunOS unstable11s 5.11 11.3 sun4u sparc SUNW,SPARC-Enterprise > ls /usr/share/man/ entities man3dat man3mvec man3sysevent man4b fr man3dax man3nsl man3tcl man5 fr.ISO8859-1 man3devid man3nvpair man3tecla man5oldap fr.UTF-8 man3devinfo man3oldap man3tiff man5openssl it man3dlpi man3openssl man3tsol man7 it.ISO8859-1 man3dns_sd man3p man3uuid man7d it.UTF-8 man3elf man3pam man3volmgt man7fs ja_JP.UTF-8 man3exacct man3pcap man3x man7i man-index man3ext man3perl man3x11 man7ipp man.cf man3f man3pi man3xau man7m man1 man3fcoe man3picl man3xaw man7openssl man1b man3fm man3picltree man3xcb man7p man1c man3fstyp man3plot man3xcomposite man8 man1m man3gen man3pool man3xcurses man8oldap man1oldap man3gss man3proc man3xcursor man8s man1openssl man3hbaapi man3project man3xevie man9 man1s man3head man3rad man3xext man9e man1t man3iscsit man3reparse man3xi man9f man2 man3kstat man3resolv man3xinerama man9p man3 man3kvm man3rpc man3xmu man9s man3archive man3layout man3sasl man3xnet pl man3c man3ldap man3scf man3xrandr pl.ISO8859-2 man3c_db man3lgrp man3sec man3xss pl.UTF-8 man3cc4 man3lib man3sip man3xt ru.KOI8-R man3cfgadm man3m man3slp man3xtsol ru.UTF-8 man3cmi man3mail man3snmp man3xtst zh_CN.UTF-8 man3commputil man3malloc man3socket man3xv man3contract man3mlib man3srpt man3xxf86vm man3cpc man3mp man3ssh2 man3zonestat man3curses man3mpapi man3stmf man4 Inside these directories, they are *mostly* using the convention "keep both section names consistent", so i do think that is good to follow. Even Solaris isn't perfect in that respect, though, they have, for example, /usr/share/man/man3cc4/cartpol.3 but on first sight, i only found about two dozen such examples. > I fixed it: > <https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=451a27a78d51973b01bfb5d3b1e0ec081d9161e1> I did not scrutinize your change in detail, but had a brief look at it and did not see any obvious problems. > And Debian seems to work fine with man3type/ and man2type/ out of the > box, so I prefer it this way. I hope that other projects follow the > example; and that packagers/distributions also create subsection > directories, and don't undo my work. On Linux, undoing it would make very little sense to me because i expect that all man(1) programs commonly used on Linux can cope with section suffixes, and in particular with consistent use of session suffixes. Besides, undoing it properly is hardly possibly for a packager. If would require changing all these: - directory names - file names - .TH macros - and manual page cross references So for better or worse, the only sane option for a packages is to follow your lead. Yours, Ingo ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux man-pages Makefile portability 2022-07-22 16:59 ` Ingo Schwarze @ 2022-07-22 17:37 ` Alejandro Colomar 2022-07-23 18:16 ` Ingo Schwarze 0 siblings, 1 reply; 15+ messages in thread From: Alejandro Colomar @ 2022-07-22 17:37 UTC (permalink / raw) To: Ingo Schwarze; +Cc: linux-man, g.branden.robinson [-- Attachment #1.1: Type: text/plain, Size: 5984 bytes --] Hi Ingo! On 7/22/22 18:59, Ingo Schwarze wrote: [...] >>> I considered[6] using man3type, and used man3 in the end just because >>> when in doubt I opted for the smallest change. Knowing that it breaks >>> mandoc(1), I'll definitely move to <man3type/>. > > I didn't mean to say man3/id_t.3type "breaks mandoc". Quite to the > contrary, the above quotation explains that mandoc copes with it. Yeah, I didn't mean break as in "mandoc(1) goes nuts, or crashes", but rather as in "it doesn't do what I wanted it to do". > > However, when it comes to robustness with respect to *other* man(1) > implementations apart from mandoc and man-db, i suspect the most > portable and reliable way is using man1 - man9 only with no suffixes, > consistent suffixes like "man3type/id_t.3type" are probably medium > portability and medium reliability, and inconsistent suffixes > like "man3/id_t.3type" and "man3type/id_t.3" are likely the least > portable and the most fragile. > > So your change is an improvement. :) > > The system making the heaviest use of section suffixes i'm aware of > is Solaris: > > > uname -a > SunOS unstable11s 5.11 11.3 sun4u sparc SUNW,SPARC-Enterprise > > ls /usr/share/man/ > entities man3dat man3mvec man3sysevent man4b > fr man3dax man3nsl man3tcl man5 > fr.ISO8859-1 man3devid man3nvpair man3tecla man5oldap > fr.UTF-8 man3devinfo man3oldap man3tiff man5openssl > it man3dlpi man3openssl man3tsol man7 > it.ISO8859-1 man3dns_sd man3p man3uuid man7d > it.UTF-8 man3elf man3pam man3volmgt man7fs > ja_JP.UTF-8 man3exacct man3pcap man3x man7i > man-index man3ext man3perl man3x11 man7ipp > man.cf man3f man3pi man3xau man7m > man1 man3fcoe man3picl man3xaw man7openssl > man1b man3fm man3picltree man3xcb man7p > man1c man3fstyp man3plot man3xcomposite man8 > man1m man3gen man3pool man3xcurses man8oldap > man1oldap man3gss man3proc man3xcursor man8s > man1openssl man3hbaapi man3project man3xevie man9 > man1s man3head man3rad man3xext man9e > man1t man3iscsit man3reparse man3xi man9f > man2 man3kstat man3resolv man3xinerama man9p > man3 man3kvm man3rpc man3xmu man9s > man3archive man3layout man3sasl man3xnet pl > man3c man3ldap man3scf man3xrandr pl.ISO8859-2 > man3c_db man3lgrp man3sec man3xss pl.UTF-8 > man3cc4 man3lib man3sip man3xt ru.KOI8-R > man3cfgadm man3m man3slp man3xtsol ru.UTF-8 > man3cmi man3mail man3snmp man3xtst zh_CN.UTF-8 > man3commputil man3malloc man3socket man3xv > man3contract man3mlib man3srpt man3xxf86vm > man3cpc man3mp man3ssh2 man3zonestat > man3curses man3mpapi man3stmf man4 Wow! Although it's interesting to know that this list exists: I can check it when trying to come up with a section name. I guess Illumos shares this subsectioning scheme. Do you know from the top of your head if any of those is dedicated to constants such as NULL, PATH_MAX, or BUFSIZ? In my git clone of Illumos, I can't find anything relevant: alx@asus5775:~/src/illumos$ find | grep -i null | grep man ./illumos-gate/usr/src/man/man9f/nulldev.9f ./illumos-gate/usr/src/man/man4d/null.4d ./illumos-gate/usr/src/man/man4d/nulldriver.4d > > Inside these directories, they are *mostly* using the convention > "keep both section names consistent", so i do think that is good > to follow. Even Solaris isn't perfect in that respect, though, > they have, for example, > > /usr/share/man/man3cc4/cartpol.3 > > but on first sight, i only found about two dozen such examples. > >> I fixed it: >> <https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=451a27a78d51973b01bfb5d3b1e0ec081d9161e1> > > I did not scrutinize your change in detail, but had a brief look at it > and did not see any obvious problems. > >> And Debian seems to work fine with man3type/ and man2type/ out of the >> box, so I prefer it this way. I hope that other projects follow the >> example; and that packagers/distributions also create subsection >> directories, and don't undo my work. > > On Linux, undoing it would make very little sense to me because i expect > that all man(1) programs commonly used on Linux can cope with section > suffixes, and in particular with consistent use of session suffixes. > Besides, undoing it properly is hardly possibly for a packager. > If would require changing all these: > > - directory names I actually make it as easy as I can to modify directory names in the Makefile through the standard GNU directory variables. > - file names I don't expect them to change filenames, even if they change dirs. It wouldn't make sense at all. > - .TH macros The same as fith file names. I expect them to want the pages to be in subsections, even if they wouldn't like the directory structure. > - and manual page cross references That's a more difficult one. They would need to run a sed script for every release. A good reason to not modify it. If they change dirs and don't change this, they'd break .so links. > > So for better or worse, the only sane option for a packages is to follow > your lead. :) Cheers, Alex > > Yours, > Ingo -- Alejandro Colomar <http://www.alejandro-colomar.es/> [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux man-pages Makefile portability 2022-07-22 17:37 ` Alejandro Colomar @ 2022-07-23 18:16 ` Ingo Schwarze 2022-07-24 11:09 ` Alejandro Colomar (man-pages) 0 siblings, 1 reply; 15+ messages in thread From: Ingo Schwarze @ 2022-07-23 18:16 UTC (permalink / raw) To: Alejandro Colomar; +Cc: linux-man, g.branden.robinson Hi Alejandro, Alejandro Colomar wrote on Fri, Jul 22, 2022 at 07:37:14PM +0200: > On 7/22/22 18:59, Ingo Schwarze wrote: >> Alejandro Colomar wrote: >>> I considered[6] using man3type, and used man3 in the end just because >>> when in doubt I opted for the smallest change. Knowing that it breaks >>> mandoc(1), I'll definitely move to <man3type/>. >> I didn't mean to say man3/id_t.3type "breaks mandoc". Quite to the >> contrary, the above quotation explains that mandoc copes with it. > Yeah, I didn't mean break as in "mandoc(1) goes nuts, or crashes", but > rather as in "it doesn't do what I wanted it to do". Even that is an overstatement. The difference between being in section 3p only and being in both section 3p and 3 is barely user-visible. Here is a manual page that is in section 3p and in section 3p only, on OpenBSD-current: $ man -w warnings /usr/share/man/man3p/warnings.3p $ grep '^\.TH' $(man -w warnings) .TH warnings 3p "2021-03-02" "perl v5.32.1" "Perl Programmers Reference Guide" But lo and behold: $ man -s 3 -w warnings /usr/share/man/man3p/warnings.3p Mandoc still finds the page in section 3 because "3" is a substring of "3p", and that not only makes sense from the string processing perspective but also from a logical perspective because section 3p is a particular corner of section 3 after all. To cause mandoc to *not* find the page in section 3, the user would have to type something like $ man -k 'sec~3$' -a warnings man: nothing appropriate Even i don't normally use such advanced syntax (regex matching on section names using the "sec" search keyword and the explicit regex operator and the -a logical operator for the conjunction of two search criteria). I doubt whether man-db even supports similar features... [...] >> The system making the heaviest use of section suffixes i'm aware of >> is Solaris: >> >> > uname -a >> SunOS unstable11s 5.11 11.3 sun4u sparc SUNW,SPARC-Enterprise >> > ls /usr/share/man/ >> entities man3dat man3mvec man3sysevent man4b >> fr man3dax man3nsl man3tcl man5 >> fr.ISO8859-1 man3devid man3nvpair man3tecla man5oldap >> fr.UTF-8 man3devinfo man3oldap man3tiff man5openssl >> it man3dlpi man3openssl man3tsol man7 >> it.ISO8859-1 man3dns_sd man3p man3uuid man7d >> it.UTF-8 man3elf man3pam man3volmgt man7fs >> ja_JP.UTF-8 man3exacct man3pcap man3x man7i >> man-index man3ext man3perl man3x11 man7ipp >> man.cf man3f man3pi man3xau man7m >> man1 man3fcoe man3picl man3xaw man7openssl >> man1b man3fm man3picltree man3xcb man7p >> man1c man3fstyp man3plot man3xcomposite man8 >> man1m man3gen man3pool man3xcurses man8oldap >> man1oldap man3gss man3proc man3xcursor man8s >> man1openssl man3hbaapi man3project man3xevie man9 >> man1s man3head man3rad man3xext man9e >> man1t man3iscsit man3reparse man3xi man9f >> man2 man3kstat man3resolv man3xinerama man9p >> man3 man3kvm man3rpc man3xmu man9s >> man3archive man3layout man3sasl man3xnet pl >> man3c man3ldap man3scf man3xrandr pl.ISO8859-2 >> man3c_db man3lgrp man3sec man3xss pl.UTF-8 >> man3cc4 man3lib man3sip man3xt ru.KOI8-R >> man3cfgadm man3m man3slp man3xtsol ru.UTF-8 >> man3cmi man3mail man3snmp man3xtst zh_CN.UTF-8 >> man3commputil man3malloc man3socket man3xv >> man3contract man3mlib man3srpt man3xxf86vm >> man3cpc man3mp man3ssh2 man3zonestat >> man3curses man3mpapi man3stmf man4 > Wow! > Although it's interesting to know that this list exists: > I can check it when trying to come up with a section name. I would somewhat advise against that. While i do think that consistency is good *if* you decide to use a section suffix, i'd still recommend to use section suffixes sparingly, at least in operating systems that use them sparingly now. They provide relatively little value, make the top directory of the manpath larger and less readable, and they are in particular *not* well suited to moving stuff out of the non-suffix directories that users may not wish to see by default. For example, suppose you created a directory man3py to document python library functions. As explained above, these pages will *still* show up even when people type "man 3" or "man 3p" rather than "man 3py". > I guess Illumos shares this subsectioning scheme. More or less, according to https://src.illumos.org/source/xref/illumos-gate/usr/src/man/ - it's not identical though. > Do you know from the top of your head if any of those is dedicated to > constants such as NULL, PATH_MAX, or BUFSIZ? I doubt it, for two reasons. On the one hand, my impression is that at least in Solaris, section suffixes are not so much used for logical subdivision of sections but more according to provenance; for example, man1b is for BSD compat features, man1s is for commands specific to SunOS, man3c is for libc, man3ext is for an "extended library", man3p is for the Sun performance library (available for both C and FORTRAN 95), and so on. On the other hand, naming manual pages after symbolic constants or after type names is so unsual that i doubt any scheme exists for that. The most widely used way to look up manual pages by the names of symbolic constants or type names probably is using macro keys as implemented in the mandoc version of apropos(1). That is used by most FreeBSD, OpenBSD, Alpine Linux, and Void Linux. I admit that doesn't qualify as "widely used", but "most widely used" is probably true all the same. ;-) Yours, Ingo ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux man-pages Makefile portability 2022-07-23 18:16 ` Ingo Schwarze @ 2022-07-24 11:09 ` Alejandro Colomar (man-pages) 2022-07-24 15:57 ` Ingo Schwarze 2022-07-25 9:28 ` Linux man-pages Makefile portability Colin Watson 0 siblings, 2 replies; 15+ messages in thread From: Alejandro Colomar (man-pages) @ 2022-07-24 11:09 UTC (permalink / raw) To: Ingo Schwarze, Colin Watson; +Cc: linux-man, g.branden.robinson, man-db-devel [TO += Colin; CC += man-dv-devel@; there's a bug in man(1)'s autocomplete] Hi Ingo (and Colin), On 7/23/22 20:16, Ingo Schwarze wrote: > [...] >>> The system making the heaviest use of section suffixes i'm aware of >>> is Solaris: >>> >>> > uname -a >>> SunOS unstable11s 5.11 11.3 sun4u sparc SUNW,SPARC-Enterprise >>> > ls /usr/share/man/ >>> entities man3dat man3mvec man3sysevent man4b >>> fr man3dax man3nsl man3tcl man5 >>> fr.ISO8859-1 man3devid man3nvpair man3tecla man5oldap >>> fr.UTF-8 man3devinfo man3oldap man3tiff man5openssl >>> it man3dlpi man3openssl man3tsol man7 >>> it.ISO8859-1 man3dns_sd man3p man3uuid man7d >>> it.UTF-8 man3elf man3pam man3volmgt man7fs >>> ja_JP.UTF-8 man3exacct man3pcap man3x man7i >>> man-index man3ext man3perl man3x11 man7ipp >>> man.cf man3f man3pi man3xau man7m >>> man1 man3fcoe man3picl man3xaw man7openssl >>> man1b man3fm man3picltree man3xcb man7p >>> man1c man3fstyp man3plot man3xcomposite man8 >>> man1m man3gen man3pool man3xcurses man8oldap >>> man1oldap man3gss man3proc man3xcursor man8s >>> man1openssl man3hbaapi man3project man3xevie man9 >>> man1s man3head man3rad man3xext man9e >>> man1t man3iscsit man3reparse man3xi man9f >>> man2 man3kstat man3resolv man3xinerama man9p >>> man3 man3kvm man3rpc man3xmu man9s >>> man3archive man3layout man3sasl man3xnet pl >>> man3c man3ldap man3scf man3xrandr pl.ISO8859-2 >>> man3c_db man3lgrp man3sec man3xss pl.UTF-8 >>> man3cc4 man3lib man3sip man3xt ru.KOI8-R >>> man3cfgadm man3m man3slp man3xtsol ru.UTF-8 >>> man3cmi man3mail man3snmp man3xtst zh_CN.UTF-8 >>> man3commputil man3malloc man3socket man3xv >>> man3contract man3mlib man3srpt man3xxf86vm >>> man3cpc man3mp man3ssh2 man3zonestat >>> man3curses man3mpapi man3stmf man4 > >> Wow! >> Although it's interesting to know that this list exists: >> I can check it when trying to come up with a section name. > > I would somewhat advise against that. While i do think that consistency > is good *if* you decide to use a section suffix, i'd still recommend > to use section suffixes sparingly, at least in operating systems > that use them sparingly now. They provide relatively little value, > make the top directory of the manpath larger and less readable, I don't think there's any benefit of keeping $(mandir) contain only the standard man? subdirs. Everybody already knows about them, so nobody needs to read the output of ls(1) in such a system. On the other hand, reading the output of ls(1) in a system with free use of subsections provides useful information. If one needs to filter a bit, standard Unix tools come into play. > and they are in particular *not* well suited to moving stuff out of > the non-suffix directories that users may not wish to see by default. > For example, suppose you created a directory man3py to document > python library functions. As explained above, these pages will *still* > show up even when people type "man 3" or "man 3p" rather than "man 3py". But the good thing is they can be put the last in the list of pages to be shown by man(1), so they won't hide another page. In my Debian system, man(1) shows: 1 n l 8 3 0 2 3posix 3pm 3perl 3am 5 4 9 6 7 3type and 3const could go at the end of that, to avoid hiding pages in section 7. While testing some related stuff to add to my arguments, I just understood of a bug that I knew existed but never thought too much about it, and related to one that Branden reported to me recently: man 3 foo[TAB] autocomplete is smart enough to only search pages in section 3 (I don't know if it finds pages in subsections from 3; I guess it does). man [-s]3type foo[TAB] autocomplete should search in 3type, but it doesn't quite work. However, I noticed there's a difference in behavior between 3posix and 3type (3posix only exists as a suffix, but pages live in man3; 3type is a correct subsection, using man3type and .3type), so maybe my work can help fix that bug in the autocomplete scripts. I know you don't use autocomplete for those things, but it comes in handy to me :) > >> I guess Illumos shares this subsectioning scheme. > > More or less, according to > https://src.illumos.org/source/xref/illumos-gate/usr/src/man/ - > it's not identical though. > >> Do you know from the top of your head if any of those is dedicated to >> constants such as NULL, PATH_MAX, or BUFSIZ? > > I doubt it, for two reasons. On the one hand, my impression is that > at least in Solaris, section suffixes are not so much used for > logical subdivision of sections but more according to provenance; > for example, man1b is for BSD compat features, man1s is for commands > specific to SunOS, man3c is for libc, man3ext is for an "extended > library", man3p is for the Sun performance library (available for > both C and FORTRAN 95), and so on. > > On the other hand, naming manual pages after symbolic constants > or after type names is so unsual that i doubt any scheme exists > for that. Actually, man3type exists in several systems. Solaris has it (I guess Illumos too), and I've seen it in other systems (something from Oracle, IIRC). libbsd(7) also documents types, although they put them in the global namespace, which I think you and I agree that it's not quite right because of "documentation about a non-function in man3". > The most widely used way to look up manual pages > by the names of symbolic constants or type names probably is > using macro keys as implemented in the mandoc version of apropos(1). > That is used by most FreeBSD, OpenBSD, Alpine Linux, and Void Linux. > I admit that doesn't qualify as "widely used", but "most widely used" > is probably true all the same. ;-) That leaves out man(7). And types tend to be not very well documented if they are documented as part of a function page. And they also tend to be documented several times (out of sync, of course). Cheers, Alex -- Alejandro Colomar Linux man-pages comaintainer; http://www.kernel.org/doc/man-pages/ http://www.alejandro-colomar.es/ ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux man-pages Makefile portability 2022-07-24 11:09 ` Alejandro Colomar (man-pages) @ 2022-07-24 15:57 ` Ingo Schwarze 2022-07-24 17:29 ` Semantic man(7) markup (was: Linux man-pages Makefile portability) G. Branden Robinson 2022-07-25 9:28 ` Linux man-pages Makefile portability Colin Watson 1 sibling, 1 reply; 15+ messages in thread From: Ingo Schwarze @ 2022-07-24 15:57 UTC (permalink / raw) To: alx.manpages; +Cc: Colin Watson, linux-man, g.branden.robinson, man-db-devel Hi Alejandro, Alejandro Colomar wrote on Sun, Jul 24, 2022 at 01:09:23PM +0200: > On 7/23/22 20:16, Ingo Schwarze wrote: >> Alejandro Colomar wrote: [...] >>> Although it's interesting to know that this list exists: >>> I can check it when trying to come up with a section name. >> I would somewhat advise against that. While i do think that consistency >> is good *if* you decide to use a section suffix, i'd still recommend >> to use section suffixes sparingly, at least in operating systems >> that use them sparingly now. They provide relatively little value, >> make the top directory of the manpath larger and less readable, > I don't think there's any benefit of keeping $(mandir) contain only the > standard man? subdirs. Everybody already knows about them, so nobody > needs to read the output of ls(1) in such a system. Admitted, it's a weak argument. Having lots of useless directories does not do *significant* harm. Then again, in general, i would say it is good practice to not add needless files or directories, even when they are not particularly harmful. They still make find(1) and ls -R output and Makefiles longer. Not doing needless things is a general aim of design economy. Besides, when there is not much to gain in the first place, even weak counter-arguments wield non-zero weight. > On the other hand, reading the output of ls(1) in a system with free > use of subsections provides useful information. I flatly deny that argument. The Solaris list of suffixes deleted above does not tell me anything of value. It's just a list of random sources of software, often with arbitrary partitions, and many of the suffixes are non-descriptive. [...] >> On the other hand, naming manual pages after symbolic constants >> or after type names is so unsual that i doubt any scheme exists >> for that. > Actually, man3type exists in several systems. Solaris has it Not on the Solaris 11 system i have access to: > uname -a SunOS unstable11s 5.11 11.3 sun4u sparc SUNW,SPARC-Enterprise > man -K e | grep 3type [ no output] > ls -d /usr/share/man/*type* /usr/share/man/*type*: No such file or directory > ls -d /usr/gnu/share/man/*type* /usr/gnu/share/man/*type*: No such file or directory > (I guess Illumos too), Not according to https://src.illumos.org/source/search?path=3type nor according to https://src.illumos.org/source/xref/illumos-gate/usr/src/man/ . Again, Solaris suffixes do *not* indicate logical subdivisions, but only a rather fuzzy, arbitrary, and inconsistent "where we got these files from", at least if i understand correctly. > and I've seen it in other systems (something from Oracle, > IIRC). libbsd(7) also documents types, although they put them in the > global namespace, which I think you and I agree that it's not quite > right because of "documentation about a non-function in man3". That argument has no merit. The pre-eminent rule is "all documentation of function libraries belongs in section 3" and explaining data types (typically defined by the function libraries in the same header files as the functions) are clearly part of that. The rule "all library documentation should be organized by functions" is secondary to that, and the rule "all section 3 names should be function names" is merely a corollary to that. So even if you throw the rule "all library dicumentation should be organized by functions" overboard, that is still no excuse for moving part of the library documentation out of section 3. It also makes sense to keep it all together in the same section because it is needed by exactly the same people (programmers) in exactly the same way (when reading and writing code and wondering what words contained in the code mean). >> The most widely used way to look up manual pages >> by the names of symbolic constants or type names probably is >> using macro keys as implemented in the mandoc version of apropos(1). >> That is used by most FreeBSD, OpenBSD, Alpine Linux, and Void Linux. >> I admit that doesn't qualify as "widely used", but "most widely used" >> is probably true all the same. ;-) > That leaves out man(7). Yes. Searching for preprocessor constants and searching for data type names are essentially semantic search features. So it is your choice to pick a 197x-era markup language that does not provide semantic markup but only physical markup. But than it feels irrational to me to turn around and complain not getting semantic search. Unless you are a Prime Minister, you cannot have your cake and eat it. Trying to work around the lack of semantic markup by moving everything into the manual page names feels like very poor design to me. > And types tend to be not very well documented if they are > documented as part of a function page. We disagree about that, and i won't repeat my full explanation why. I'll repeat only this one aspect: There are few syntax elements where context matters as much as for types; types live by how they are used. Separating their description from the functions using them is a disservice to users, forcing them to jump around various pages instead of having explained together what belongs together. > And they also tend to be documented several times (out of sync, > of course). Any kind of documentation needs discipline and diligence and maintenance, and none of it can ever be perfect. But that's no excuse for artificially tearing apart what users need together. Yours, Ingo ^ permalink raw reply [flat|nested] 15+ messages in thread
* Semantic man(7) markup (was: Linux man-pages Makefile portability) 2022-07-24 15:57 ` Ingo Schwarze @ 2022-07-24 17:29 ` G. Branden Robinson 2022-07-24 21:26 ` Ingo Schwarze 0 siblings, 1 reply; 15+ messages in thread From: G. Branden Robinson @ 2022-07-24 17:29 UTC (permalink / raw) To: Ingo Schwarze; +Cc: alx.manpages, Colin Watson, linux-man, man-db-devel [-- Attachment #1: Type: text/plain, Size: 5353 bytes --] At 2022-07-24T17:57:31+0200, Ingo Schwarze wrote: > Alejandro Colomar wrote on Sun, Jul 24, 2022 at 01:09:23PM +0200: > > On 7/23/22 20:16, Ingo Schwarze wrote: > >> The most widely used way to look up manual pages by the names of > >> symbolic constants or type names probably is using macro keys as > >> implemented in the mandoc version of apropos(1). That is used by > >> most FreeBSD, OpenBSD, Alpine Linux, and Void Linux. I admit that > >> doesn't qualify as "widely used", but "most widely used" is > >> probably true all the same. ;-) > > > That leaves out man(7). Perhaps not for long... > Yes. Searching for preprocessor constants and searching for data > type names are essentially semantic search features. So it is > your choice to pick a 197x-era markup language that does not provide > semantic markup but only physical markup. But than it feels > irrational to me to turn around and complain not getting semantic > search. Unless you are a Prime Minister, you cannot have your cake > and eat it. > > Trying to work around the lack of semantic markup by moving > everything into the manual page names feels like very poor design > to me. It will not surprise, but might horrify, Ingo to learn that I have an idea for how to add semantic markup to man(7). Consider this hypothetical example. $ cat man3/man-pages.man .DC type B .DC field I $ cat man3/tm.3type .so man3/man-pages.man [...] .SH DESCRIPTION .TG type "struct tm" describes time, broken down into distinct components. .PP .TG field tm_isdst describes wether daylight saving time is in effect at the time described. [...] Here, "DC" means "define class", a class of tags. "TG", if one could not guess, declares a tag of the type in its first argument with the remaining arguments being the content thus tagged. Returning to "DC", we see that it takes a second argument naming a macro to call which will then apply any desired presentational markup to style the tagged word. This second argument need not be present. In other words, tagged content need not be visually distinct from its surroundings. Even in that event, it can still be useful; see #1 below. Further, it will be obvious to the experienced *roff user that the macro called by DC to style the applicable arguments given to TG need not even be part of the man(7) language. You could populate "man-pages.man" like this. $ cat man3/man-pages.man .de CW . ie t \&\f[CR]\\$*\f[] . el \&\\$* .. .DC type CW .DC field I This technique breaks the stranglehold of the man(7) font selection macros. (You're still limited by the output device's font repertoire, however.) If rendering to PostScript or PDF, you could decide to style certain tags in Zapf Chancery Medium italic, if you wished. (I cannot warrant that you won't get yelled at.) Here are a few perhaps less obvious things this approach would offer. 1. It enables keyword search by tag. Whatever does the searching need only look for "TG" calls, match the class argument, and return the remainder. A search could be narrowed by limiting both the class _and_ the keyword arguments of course, perhaps to answer questions like "what pages use 'stat' as data type?". 2. Degraded operation for other/older man(7) implementations is straightforward. 'DC' can be completely ignored. 'TG' can be defined as follows. .de TG \&\\$* .. or, for truly bloody-minded portability, thus. .de TG \&\\$1 \\$2 \\$3 \\$4 \\$5 \\$6 \\$7 \\$8 \\$9 .. 3. Everyday man(7) page authors need only learn 'TG' and the available list of keywords for the suite of man pages to which they are contributing. Hammering out the repertoire of available tag classes and the surely monumental bikeshedding of text styling decisions to be associated with each tag class is delegated to the project that chooses to define them. The man(7) macro package itself will impose no policy and may not even define any tag classes to start with. (groff would have some for its own man pages, of course, as I would expect Linux man-pages to do.) 4. Site admins offended at the styling decisions undertaken by various projects could reliably override them by editing the files sourced by the relevant man pages. Maybe those should live in /etc rather than the man page hierarchy proper. 5. Misspelling a tag class or using an unavailable one is an error that would be easily diagnosed and reported. To reiterate, groff man(7) would impose no policy regarding the tag classes or their rendering on anyone. It similarly would escape the ongoing problem that mdoc(7) chose for itself by administering centralized authoritative lists of standards documents, operating system releases, and other lexica. Tagful man(7) pages under my proposal would opt into whatever keyword/class discipline they desire, or not at all. I am not wedded to the nomenclature for the included files, nor the `DC` or `TG` macros, except to note that the macro names are available. (`DT`, putatively for "define tag", is not. It is already taken.) I stand ready for the hail of rotten tomatoes. Regards, Branden [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Semantic man(7) markup (was: Linux man-pages Makefile portability) 2022-07-24 17:29 ` Semantic man(7) markup (was: Linux man-pages Makefile portability) G. Branden Robinson @ 2022-07-24 21:26 ` Ingo Schwarze 0 siblings, 0 replies; 15+ messages in thread From: Ingo Schwarze @ 2022-07-24 21:26 UTC (permalink / raw) To: g.branden.robinson; +Cc: alx.manpages, Colin Watson, linux-man, man-db-devel Hi Branden, > I stand ready for the hail of rotten tomatoes. I hope you are not surprised that i'll try to answer constructively. Even though i'm not sure how relevent such an early design draft is for linux-man@ or man-db-devel@, i'm not trimming the distribution list just yet, but you might perhaps wish to consider choosing one appropriate forum rather than cross-posting a design effort that is likely to be long and difficult to three lists. G. Branden Robinson wrote on Sun, Jul 24, 2022 at 12:29:47PM -0500: > It will not surprise, but might horrify, Ingo to learn that I have an > idea for how to add semantic markup to man(7). It does not surprise me, you have already moved ahead with adding new syntax to man(7), though so far in a piece-meal manner. It would only horrify me if this were done badly. Since this is an early draft, it is not a surpise that i will raise several points of the kind "won't work as you apparently hope" below. That's not intended as denigration. In an early design phase, it is normal that not every idea is viable. The job is to find the working ideas in a larger pool of potential ideas and hammer them into a working whole of minimal complexity that does the job. > Consider this hypothetical example. > > $ cat man3/man-pages.man > .DC type B > .DC field I > $ cat man3/tm.3type > .so man3/man-pages.man There is a danger of collisions here between the helper files of different software projects. That's maybe manageable; page names can already collide right now. Do such helper files really need to be section-specific? mdoc(7) teaches that in practice, there are basically two markup domains: command line utilities and library functions. A possible third domain, kernel configuartion directives, leads a fringe existence at best. In practice, combining both domains in one namespace poses no problems. Besides, you will hardly be surprised that i deem making the styles configurable detrimental in the first place. The whole point of having such styles is enabling a uniform, consistent user experience. > [...] > .SH DESCRIPTION > .TG type "struct tm" This utterly breaks compatibility. On formatters not suuporting .TG, the most important content of the dpocument will simply disappear. Even if you would somehow manage to get a fallback .de TG \&\\$2 .. in place - and so far, i have no idea how you could do that portably on an arbitrary and potentially unknown formatter - that would leave "struct tm" with no markup whatsoever. One obvious idea to trivially improve the design is to use the equivalent syntax .TG type .I "struct tm" with the syntax constraint ".TG needs to be followed by a font macro" and the semantic effect "in that case, the custom style overrides the default style", reminiscent of how CSS can be used to override the default styles of HTML elements. This suggestion for improvement is not yet a definitive recommendation though, it is merely another idea for the pool. Let me also note that the markup overhead for your proposed scheme is so far at least three times as large as for mdoc(7). With modc(7), marking up a work usually requires one two-letter macro as the markup overhead. Your scheme requites the tag (which will usually be at least four letters, sometimes more) in addition. For spacing and punctuation control, mdoc(7) usually requires no furthes overhead, except occasionally two letters for Ns. It's not yet clear how your scheme will fare in that respect because you proposed no mechanism for spacing and punction control yet, but that will likely require some overhead, too. > describes time, broken down into distinct components. > .PP > .TG field tm_isdst > describes wether daylight saving time is in effect at the time > described. > [...] > > Here, "DC" means "define class", a class of tags. "TG", if one could > not guess, declares a tag of the type in its first argument with the > remaining arguments being the content thus tagged. > > Returning to "DC", we see that it takes a second argument naming a macro > to call which will then apply any desired presentational markup to style > the tagged word. This second argument need not be present. In other > words, tagged content need not be visually distinct from its > surroundings. Even in that event, it can still be useful; see #1 below. > > Further, it will be obvious to the experienced *roff user that the macro > called by DC to style the applicable arguments given to TG need not even > be part of the man(7) language. > > You could populate "man-pages.man" like this. > > $ cat man3/man-pages.man > .de CW > . ie t \&\f[CR]\\$*\f[] > . el \&\\$* > .. > .DC type CW > .DC field I > > This technique breaks the stranglehold of the man(7) font selection > macros. (You're still limited by the output device's font repertoire, > however.) I fear you are missing the main aspect that makes using the man(7) font macros complicated: spacing and punctuation. Let us start with a moderately simple, everyday example: .Ft double Fn atan2 "double y" "double x" What would that become with your proposed syntax? .TG type double .TG funcname atan2 ( .TG funcarg "double y" , .TG funcarg "double x" ) What is your plan for that? Recommending \c everywhere? Or mdoc(7)-style punctuation handling like .TG type double .TG funcname atan2 NS .TG funcarg ( "double y" , .TG funcarg "double x" ) Practical work shows these trivial cases of spacing around common punctuation like parentheses and commas are not the only ones that occur. Sometimes, spacing needs to be inserted or suppressed in ways that differ from default rules. mdoc(7) has several mechanisms for that, the most important one being .Ns. Your new scheme might choose a different approach, but i predict how well that mechanism is going to be will be one determining factor of how pleasant or unpleasant the language will become for authors and maintainers. > If rendering to PostScript or PDF, you could decide to style > certain tags in Zapf Chancery Medium italic, if you wished. (I cannot > warrant that you won't get yelled at.) > > Here are a few perhaps less obvious things this approach would offer. > > 1. It enables keyword search by tag. Whatever does the searching need > only look for "TG" calls, match the class argument, and return the > remainder. A search could be narrowed by limiting both the class > _and_ the keyword arguments of course, perhaps to answer questions > like "what pages use 'stat' as data type?". Sure, just like man -k Vt=stat with mandoc. > 2. Degraded operation for other/older man(7) implementations is > straightforward. 'DC' can be completely ignored. 'TG' can be > defined as follows. > > .de TG > \&\\$* > .. No, that is plain wrong. It results in type struct tm describes time, broken down into distinct components. field tm_isdst describes wether daylight saving time is in effect at the time described. You certainly do not want to print out \\$1. > or, for truly bloody-minded portability, thus. > > .de TG > \&\\$1 \\$2 \\$3 \\$4 \\$5 \\$6 \\$7 \\$8 \\$9 > .. That would exclude any kind of mdoc(7)-style punctuation handling, making spacing control much harder to implement. I'm not saying impossible, but harder. One way out might be to use \\$2 only and require quoting of multi-word arguments. > 3. Everyday man(7) page authors need only learn 'TG' and the available > list of keywords for the suite of man pages to which they are > contributing. Hammering out the repertoire of available tag classes > and the surely monumental bikeshedding of text styling decisions to > be associated with each tag class is delegated to the project that > chooses to define them. I fear that is a terrible idea. The DocBook design fiasco teaches us that even a single committee is easily capable of utterly screwing up markup consistency. Leaving the markup lexicon up to individual projects guarantees hopeless global inconsistency. Different projects will mark up different syntax elements, assign different tags for the same purpose, and assign conflicting styles to the same tags. Consequently, semantic searching will *not* work at all. You cannot even search for a function name if one group tags it as "function" and another as "funcname" and another as "sub" and another as "procname". Rendering will be wildly inconsistent because every group of pages, large or small, comes with its own styles - and larger projects are unlikely to remain consistent even within themselves. Due to the undefined nature of tag names, distributions are unable to fix up the mess, too. > The man(7) macro package itself will impose > no policy and may not even define any tag classes to start with. Branden, seriously, standardizing this is your number two most important job during this design. The top one priority, of course, is to choose a syntax that is short, simple, robust, esay to read, and easy to write for both usual and unusual use cases, and - if you want to continue calling the language man(7) - backward compatible. > (groff would have some for its own man pages, of course, as I would > expect Linux man-pages to do.) > > 4. Site admins offended at the styling decisions undertaken by various > projects could reliably override them by editing the files sourced > by the relevant man pages. That's illusionary until you provide an authoritative and long-time stable list of valid tags. Failing to do so is one among the reasons why SGML and XML turned out as hard to use in practice, and one of the reasons why HTML ultimately gave up on being XML and instead defined an exhaustive list of standard elements. Besides, on one of my machines, i currently have almost 600 packages installed even though lots of things that are packages on Linux do *not* count as packages on OpenBSD but are part of the base system that cannot ever be uninstalled, including thing like compilers and other development tools and lots of Internet daemons like web server, mail server, routing daemons, DNS servers and much more. And yet, i still have almost 600 packages. As a rule of thumb, on average one third of packages have manual pages. So you want me to edit 200 config files and resolve conflicts in them whenever updating packages? I'm not yet convinced that's a very good plan. :-o > Maybe those should live in /etc rather > than the man page hierarchy proper. > > 5. Misspelling a tag class or using an unavailable one is an error that > would be easily diagnosed and reported. > > To reiterate, groff man(7) would impose no policy regarding the tag > classes or their rendering on anyone. It similarly would escape the > ongoing problem that mdoc(7) chose for itself by administering > centralized authoritative lists of standards documents, operating system > releases, and other lexica. I freely admit that .St .At .Bx .Nx .Fx .Ox .Dx .Lb are among the least well designed parts of mdoc(7). But that design problem is completely unrelated to the fact that the list of macros (in mdoc(7) parlance) or tags (in your proposed terminology) needs to be fixed. That's *the* central purpuse of a markup language. Personally, i would go as far as calling a non-extensible markup language usually better than an extensible one, but even for an extensible markup language, the quality of the standard lexicon is the second highest priority, right after the quality of the basic syntax. > Tagful man(7) pages under my proposal would > opt into whatever keyword/class discipline they desire, or not at all. > > I am not wedded to the nomenclature for the included files, nor the `DC` > or `TG` macros, except to note that the macro names are available. > (`DT`, putatively for "define tag", is not. It is already taken.) Given how utterly your .TG proposal is breaking compatibility, i wonder whether it would be wiser to give your new macroset a new name and leave the legacy man(7) language alone. That would allow cleaning up some design deficiencies of man(7) at the same time, for example: * properly define display macros, a purpose now served in a manner the is partially quirky, partially redundant, and yet incomplete by .RS and .EX * get rid of the font alternation macros; it is well known that even though those squat on a considerable parcel of namespace, they only solve part of their task - the infamous three-font issue; your proposal needs a solution for that anyway that still needs to be developed * get rid of the redundancy among .TP .TQ .IP .HP * get rid of legacy macros like .AT .DT .LP .P .SM .SB .UC that are useless in practice * get rid of misdesigned macros like .SY Please also consider that having a mix of semantic and presentational markup in a language is usually not the best idea. As a prominent example, after a considerable amount of handwringing that lasted for many years, HTML finally abolished presentational markup completely. For similar reasons, we somewhat discourage using .Em and .Sy in mdoc(7), except for purposes similar to <em>, <i>, <strong>, and <b> in HTML 5. If you choose a new name, compatibility concerns vanish instantly. But of course, once you publicly announce "this is now ready for production", stability becomes important again, so the design should be well thought out from the start. Finding a major design issue or design gap after it has been in production for, say, two years would be very unfortunate. On the other hand, if you want to continue calling it man(7), then the logical consequence is that compatibility concerns become paramount. Yours, Ingo ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Linux man-pages Makefile portability 2022-07-24 11:09 ` Alejandro Colomar (man-pages) 2022-07-24 15:57 ` Ingo Schwarze @ 2022-07-25 9:28 ` Colin Watson 2022-07-25 12:46 ` bash-completion doesn't support man subsections (was: Linux man-pages Makefile portability) Alejandro Colomar 1 sibling, 1 reply; 15+ messages in thread From: Colin Watson @ 2022-07-25 9:28 UTC (permalink / raw) To: Alejandro Colomar (man-pages) Cc: Ingo Schwarze, linux-man, g.branden.robinson, man-db-devel On Sun, Jul 24, 2022 at 01:09:23PM +0200, Alejandro Colomar (man-pages) wrote: > [TO += Colin; CC += man-dv-devel@; > there's a bug in man(1)'s autocomplete] man-db doesn't ship its own autocomplete code at the moment. This is probably shipped by some other package in your distribution, although I don't know exactly what that would be (maybe bash-completion?). -- Colin Watson (he/him) [cjwatson@debian.org] ^ permalink raw reply [flat|nested] 15+ messages in thread
* bash-completion doesn't support man subsections (was: Linux man-pages Makefile portability) 2022-07-25 9:28 ` Linux man-pages Makefile portability Colin Watson @ 2022-07-25 12:46 ` Alejandro Colomar 0 siblings, 0 replies; 15+ messages in thread From: Alejandro Colomar @ 2022-07-25 12:46 UTC (permalink / raw) To: Colin Watson; +Cc: Ingo Schwarze, linux-man, g.branden.robinson, man-db-devel [-- Attachment #1.1: Type: text/plain, Size: 793 bytes --] Hi Colin, On 7/25/22 11:28, Colin Watson wrote: > On Sun, Jul 24, 2022 at 01:09:23PM +0200, Alejandro Colomar (man-pages) wrote: >> [TO += Colin; CC += man-dv-devel@; >> there's a bug in man(1)'s autocomplete] > > man-db doesn't ship its own autocomplete code at the moment. > This is probably shipped by some other package in your distribution, Heh, I bet it's the same as yours, as you show up as the maintainer ;) $ apt-cache show man-db | grep Maintainer Maintainer: Colin Watson <cjwatson@debian.org> Yeah, I use Debian (Sid). > although I don't know exactly what that would be (maybe bash-completion?). Reported to upstream: <https://github.com/scop/bash-completion/issues/778> Cheers, Alex -- Alejandro Colomar <http://www.alejandro-colomar.es/> [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2022-07-25 12:46 UTC | newest] Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-06-19 17:52 Linux man-pages Makefile portability Alejandro Colomar 2022-06-19 21:06 ` Ingo Schwarze 2022-06-19 22:23 ` Alejandro Colomar 2022-06-20 13:49 ` Ingo Schwarze 2022-07-03 21:44 ` Alejandro Colomar 2022-07-21 14:17 ` Alejandro Colomar 2022-07-22 16:59 ` Ingo Schwarze 2022-07-22 17:37 ` Alejandro Colomar 2022-07-23 18:16 ` Ingo Schwarze 2022-07-24 11:09 ` Alejandro Colomar (man-pages) 2022-07-24 15:57 ` Ingo Schwarze 2022-07-24 17:29 ` Semantic man(7) markup (was: Linux man-pages Makefile portability) G. Branden Robinson 2022-07-24 21:26 ` Ingo Schwarze 2022-07-25 9:28 ` Linux man-pages Makefile portability Colin Watson 2022-07-25 12:46 ` bash-completion doesn't support man subsections (was: Linux man-pages Makefile portability) Alejandro Colomar
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).