[PATCH libmnl 0/1] build: doc: "make" builds & installs a full set of man pages

[PATCH libmnl 0/1] build: doc: "make" builds & installs a full set of man pages

[Duncan Roe]

This patch gets us out of the unfortunate situation where there are man pages
for all functions in (old, libnfnetlink-based) sample program
libnetfilter_queue/utils/nfqnl_test.c but not for the libmnl-based
libnetfilter_queue/examples/nf-queue.c.

From libnetfilter_queue we know how to generate a full set of man pages during
`make` so let's do it for libmnl.

This patch sets up the framework: in future patches I plan to update the
doxygen comments to be more man-page-friendly, e.g. by using \returns and \sa
where appropriate.

Duncan Roe (1):
  build: doc: "make" builds & installs a full set of man pages

 Makefile.am         |   4 +-
 configure.ac        |  24 +++++-
 doxygen.cfg.in      | 176 ++------------------------------------------
 doxygen/Makefile.am |  75 +++++++++++++++++++
 4 files changed, 107 insertions(+), 172 deletions(-)
 create mode 100644 doxygen/Makefile.am

-- 
2.17.5

[PATCH libmnl 1/1] build: doc: "make" builds & installs a full set of man pages

[Duncan Roe]

Repeat what we did for libnetfilter_queue:
 - New makefile in doxygen directory. Rebuilds documentation if any sources
   change that contain doxygen comments:
   - Renames each group man page to the first function listed therein
   - Creates symlinks for subsequently listed functions
   - Deletes _* temp files and moves sctruct-describing man pages to man7
 - Update top-level makefile to visit new subdir doxygen
 - Update top-level configure to only build documentation if doxygen installed
 - Add --with/without-doxygen switch
 - Check whether dot is available when configuring doxygen
 - Reduce size of doxygen.cfg and doxygen build o/p
 - `make distcheck` passes with doxygen enabled
Aditionally, exclude opaque structs mnl_nlmsg_batch & mnl_socket

Signed-off-by: Duncan Roe <duncan_roe@optusnet.com.au>
---
 Makefile.am         |   4 +-
 configure.ac        |  24 +++++-
 doxygen.cfg.in      | 176 ++------------------------------------------
 doxygen/Makefile.am |  75 +++++++++++++++++++
 4 files changed, 107 insertions(+), 172 deletions(-)
 create mode 100644 doxygen/Makefile.am

diff --git a/Makefile.am b/Makefile.am
index dec85d0..94e6935 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -2,8 +2,8 @@ include $(top_srcdir)/Make_global.am
 
 ACLOCAL_AMFLAGS = -I m4
 
-SUBDIRS = src include examples
-DIST_SUBDIRS = src include examples
+SUBDIRS = src include examples doxygen
+DIST_SUBDIRS = src include examples doxygen
 
 pkgconfigdir = $(libdir)/pkgconfig
 pkgconfig_DATA = libmnl.pc
diff --git a/configure.ac b/configure.ac
index 35b2531..a06ae6a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -27,5 +27,27 @@ regular_CFLAGS="-Wall -Waggregate-return -Wmissing-declarations \
 	-Wformat=2 -pipe"
 AC_SUBST([regular_CPPFLAGS])
 AC_SUBST([regular_CFLAGS])
-AC_CONFIG_FILES([Makefile src/Makefile include/Makefile include/libmnl/Makefile include/linux/Makefile include/linux/netfilter/Makefile examples/Makefile examples/genl/Makefile examples/kobject/Makefile examples/netfilter/Makefile examples/rtnl/Makefile libmnl.pc doxygen.cfg])
+AC_CONFIG_FILES([Makefile src/Makefile include/Makefile include/libmnl/Makefile include/linux/Makefile include/linux/netfilter/Makefile examples/Makefile examples/genl/Makefile examples/kobject/Makefile examples/netfilter/Makefile examples/rtnl/Makefile libmnl.pc doxygen.cfg doxygen/Makefile])
+
+AC_ARG_WITH([doxygen], [AS_HELP_STRING([--with-doxygen],
+	    [create doxygen documentation])],
+	    [with_doxygen="$withval"], [with_doxygen=yes])
+
+AS_IF([test "x$with_doxygen" != xno], [
+	AC_CHECK_PROGS([DOXYGEN], [doxygen])
+	AC_CHECK_PROGS([DOT], [dot], [""])
+	AS_IF([test "x$DOT" != "x"],
+	      [AC_SUBST(HAVE_DOT, YES)],
+	      [AC_SUBST(HAVE_DOT, NO)])
+])
+
+AM_CONDITIONAL([HAVE_DOXYGEN], [test -n "$DOXYGEN"])
+AS_IF([test "x$DOXYGEN" = x], [
+	dnl Only run doxygen Makefile if doxygen installed
+	AC_MSG_WARN([Doxygen not found - continuing without Doxygen support])
+])
 AC_OUTPUT
+
+echo "
+libmnl configuration:
+  doxygen:          ${with_doxygen}"
diff --git a/doxygen.cfg.in b/doxygen.cfg.in
index 31f0102..1e42e44 100644
--- a/doxygen.cfg.in
+++ b/doxygen.cfg.in
@@ -1,184 +1,22 @@
-DOXYFILE_ENCODING      = UTF-8
+# Difference with default Doxyfile 1.8.20
 PROJECT_NAME           = @PACKAGE@
 PROJECT_NUMBER         = @VERSION@
 OUTPUT_DIRECTORY       = doxygen
-CREATE_SUBDIRS         = NO
-OUTPUT_LANGUAGE        = English
-BRIEF_MEMBER_DESC      = YES
-REPEAT_BRIEF           = YES
-ABBREVIATE_BRIEF       = 
-ALWAYS_DETAILED_SEC    = NO
-INLINE_INHERITED_MEMB  = NO
+ABBREVIATE_BRIEF       =
 FULL_PATH_NAMES        = NO
-STRIP_FROM_PATH        = 
-STRIP_FROM_INC_PATH    = 
-SHORT_NAMES            = NO
-JAVADOC_AUTOBRIEF      = NO
-QT_AUTOBRIEF           = NO
-MULTILINE_CPP_IS_BRIEF = NO
-INHERIT_DOCS           = YES
-SEPARATE_MEMBER_PAGES  = NO
 TAB_SIZE               = 8
-ALIASES                = 
 OPTIMIZE_OUTPUT_FOR_C  = YES
-OPTIMIZE_OUTPUT_JAVA   = NO
-OPTIMIZE_FOR_FORTRAN   = NO
-OPTIMIZE_OUTPUT_VHDL   = NO
-BUILTIN_STL_SUPPORT    = NO
-CPP_CLI_SUPPORT        = NO
-SIP_SUPPORT            = NO
-DISTRIBUTE_GROUP_DOC   = NO
-SUBGROUPING            = YES
-TYPEDEF_HIDES_STRUCT   = NO
-EXTRACT_ALL            = NO
-EXTRACT_PRIVATE        = NO
-EXTRACT_STATIC         = NO
-EXTRACT_LOCAL_CLASSES  = YES
-EXTRACT_LOCAL_METHODS  = NO
-EXTRACT_ANON_NSPACES   = NO
-HIDE_UNDOC_MEMBERS     = NO
-HIDE_UNDOC_CLASSES     = NO
-HIDE_FRIEND_COMPOUNDS  = NO
-HIDE_IN_BODY_DOCS      = NO
-INTERNAL_DOCS          = NO
-CASE_SENSE_NAMES       = YES
-HIDE_SCOPE_NAMES       = NO
-SHOW_INCLUDE_FILES     = YES
-INLINE_INFO            = YES
-SORT_MEMBER_DOCS       = YES
-SORT_BRIEF_DOCS        = NO
-SORT_GROUP_NAMES       = NO
-SORT_BY_SCOPE_NAME     = NO
-GENERATE_TODOLIST      = YES
-GENERATE_TESTLIST      = YES
-GENERATE_BUGLIST       = YES
-GENERATE_DEPRECATEDLIST= YES
-ENABLED_SECTIONS       = 
-MAX_INITIALIZER_LINES  = 30
-SHOW_USED_FILES        = YES
-SHOW_DIRECTORIES       = NO
-FILE_VERSION_FILTER    = 
-QUIET                  = NO
-WARNINGS               = YES
-WARN_IF_UNDOCUMENTED   = YES
-WARN_IF_DOC_ERROR      = YES
-WARN_NO_PARAMDOC       = NO
-WARN_FORMAT            = "$file:$line: $text"
-WARN_LOGFILE           = 
 INPUT                  = .
-INPUT_ENCODING         = UTF-8
-FILE_PATTERNS          = *.c libmnl.h
+FILE_PATTERNS          = */src/*.c
 RECURSIVE              = YES
-EXCLUDE                = 
-EXCLUDE_SYMLINKS       = NO
-EXCLUDE_PATTERNS       = */.git/* .*.d
-EXCLUDE_SYMBOLS        = EXPORT_SYMBOL
-EXAMPLE_PATH           = 
-EXAMPLE_PATTERNS       = 
-EXAMPLE_RECURSIVE      = NO
-IMAGE_PATH             = 
+EXCLUDE_SYMBOLS        = EXPORT_SYMBOL mnl_nlmsg_batch mnl_socket
+EXAMPLE_PATTERNS       =
 INPUT_FILTER           = "sed 's/EXPORT_SYMBOL//g'"
-FILTER_PATTERNS        = 
-FILTER_SOURCE_FILES    = NO
 SOURCE_BROWSER         = YES
-INLINE_SOURCES         = NO
-STRIP_CODE_COMMENTS    = YES
-REFERENCED_BY_RELATION = NO
-REFERENCES_RELATION    = NO
-REFERENCES_LINK_SOURCE = YES
-USE_HTAGS              = NO
-VERBATIM_HEADERS       = YES
 ALPHABETICAL_INDEX     = NO
-COLS_IN_ALPHA_INDEX    = 5
-IGNORE_PREFIX          = 
-GENERATE_HTML          = YES
-HTML_OUTPUT            = html
-HTML_FILE_EXTENSION    = .html
-HTML_HEADER            = 
-HTML_STYLESHEET        = 
-HTML_ALIGN_MEMBERS     = YES
-GENERATE_HTMLHELP      = NO
-GENERATE_DOCSET        = NO
-DOCSET_FEEDNAME        = "Doxygen generated docs"
-DOCSET_BUNDLE_ID       = org.doxygen.Project
-HTML_DYNAMIC_SECTIONS  = NO
-CHM_FILE               = 
-HHC_LOCATION           = 
-GENERATE_CHI           = NO
-BINARY_TOC             = NO
-TOC_EXPAND             = NO
-DISABLE_INDEX          = NO
-ENUM_VALUES_PER_LINE   = 4
-GENERATE_TREEVIEW      = NO
-TREEVIEW_WIDTH         = 250
+SEARCHENGINE           = NO
 GENERATE_LATEX         = NO
-LATEX_OUTPUT           = latex
 LATEX_CMD_NAME         = latex
-MAKEINDEX_CMD_NAME     = makeindex
-COMPACT_LATEX          = NO
-PAPER_TYPE             = a4wide
-EXTRA_PACKAGES         = 
-LATEX_HEADER           = 
-PDF_HYPERLINKS         = YES
-USE_PDFLATEX           = YES
-LATEX_BATCHMODE        = NO
-LATEX_HIDE_INDICES     = NO
-GENERATE_RTF           = NO
-RTF_OUTPUT             = rtf
-COMPACT_RTF            = NO
-RTF_HYPERLINKS         = NO
-RTF_STYLESHEET_FILE    = 
-RTF_EXTENSIONS_FILE    = 
 GENERATE_MAN           = YES
-MAN_OUTPUT             = man
-MAN_EXTENSION          = .3
-MAN_LINKS              = NO
-GENERATE_XML           = NO
-XML_OUTPUT             = xml
-XML_SCHEMA             = 
-XML_DTD                = 
-XML_PROGRAMLISTING     = YES
-GENERATE_AUTOGEN_DEF   = NO
-GENERATE_PERLMOD       = NO
-PERLMOD_LATEX          = NO
-PERLMOD_PRETTY         = YES
-PERLMOD_MAKEVAR_PREFIX = 
-ENABLE_PREPROCESSING   = YES
-MACRO_EXPANSION        = NO
-EXPAND_ONLY_PREDEF     = NO
-SEARCH_INCLUDES        = YES
-INCLUDE_PATH           = 
-INCLUDE_FILE_PATTERNS  = 
-PREDEFINED             = 
-EXPAND_AS_DEFINED      = 
-SKIP_FUNCTION_MACROS   = YES
-TAGFILES               = 
-GENERATE_TAGFILE       = 
-ALLEXTERNALS           = NO
-EXTERNAL_GROUPS        = YES
-PERL_PATH              = /usr/bin/perl
-CLASS_DIAGRAMS         = YES
-MSCGEN_PATH            = 
-HIDE_UNDOC_RELATIONS   = YES
-HAVE_DOT               = YES
-CLASS_GRAPH            = YES
-COLLABORATION_GRAPH    = YES
-GROUP_GRAPHS           = YES
-UML_LOOK               = NO
-TEMPLATE_RELATIONS     = NO
-INCLUDE_GRAPH          = YES
-INCLUDED_BY_GRAPH      = YES
-CALL_GRAPH             = NO
-CALLER_GRAPH           = NO
-GRAPHICAL_HIERARCHY    = YES
-DIRECTORY_GRAPH        = YES
-DOT_IMAGE_FORMAT       = png
-DOT_PATH               = 
-DOTFILE_DIRS           = 
-DOT_GRAPH_MAX_NODES    = 50
-MAX_DOT_GRAPH_DEPTH    = 0
+HAVE_DOT               = @HAVE_DOT@
 DOT_TRANSPARENT        = YES
-DOT_MULTI_TARGETS      = NO
-GENERATE_LEGEND        = YES
-DOT_CLEANUP            = YES
-SEARCHENGINE           = NO
diff --git a/doxygen/Makefile.am b/doxygen/Makefile.am
new file mode 100644
index 0000000..e0598ab
--- /dev/null
+++ b/doxygen/Makefile.am
@@ -0,0 +1,75 @@
+if HAVE_DOXYGEN
+
+# Be sure to add new source files to this table
+doc_srcs = $(top_srcdir)/src/attr.c     \
+           $(top_srcdir)/src/callback.c \
+           $(top_srcdir)/src/nlmsg.c    \
+           $(top_srcdir)/src/socket.c
+
+doxyfile.stamp: $(doc_srcs) Makefile.am
+	rm -rf html man
+
+# Test for running under make distcheck.
+# If so, sibling src directory will be empty:
+# move it out of the way and symlink the real one while we run doxygen.
+	[ -f ../src/Makefile.in ] || \
+{ set -x; cd ..; mv src src.distcheck; ln -s $(top_srcdir)/src; }
+
+	cd ..; doxygen doxygen.cfg >/dev/null
+
+	[ ! -d ../src.distcheck ] || \
+{ set -x; cd ..; rm src; mv src.distcheck src; }
+
+# Keep this command up to date after adding new functions and source files.
+# The command has to be a single line so the functions work
+# (hence ";\" at the end of every line but the last).
+	main() { set -e; cd man/man3; rm -f _*;\
+setgroup attr mnl_attr_get_type;\
+  add2group mnl_attr_get_len mnl_attr_get_payload_len mnl_attr_get_payload;\
+  add2group mnl_attr_ok mnl_attr_next mnl_attr_type_valid mnl_attr_validate;\
+  add2group mnl_attr_validate2 mnl_attr_parse mnl_attr_parse_nested;\
+  add2group mnl_attr_parse_payload mnl_attr_get_u8 mnl_attr_get_u16;\
+  add2group mnl_attr_get_u32 mnl_attr_get_u64 mnl_attr_get_str mnl_attr_put;\
+  add2group mnl_attr_put_u8 mnl_attr_put_u16 mnl_attr_put_u32 mnl_attr_put_u64;\
+  add2group mnl_attr_put_str mnl_attr_put_strz mnl_attr_nest_start;\
+  add2group mnl_attr_put_check mnl_attr_put_u8_check mnl_attr_put_u16_check;\
+  add2group mnl_attr_put_u32_check mnl_attr_put_u64_check;\
+  add2group mnl_attr_put_str_check mnl_attr_put_strz_check;\
+  add2group mnl_attr_nest_start_check mnl_attr_nest_end mnl_attr_nest_cancel;\
+setgroup batch mnl_nlmsg_batch_start;\
+  add2group mnl_nlmsg_batch_stop mnl_nlmsg_batch_next mnl_nlmsg_batch_reset;\
+  add2group mnl_nlmsg_batch_size mnl_nlmsg_batch_head mnl_nlmsg_batch_current;\
+  add2group mnl_nlmsg_batch_is_empty;\
+setgroup callback mnl_cb_run;\
+  add2group mnl_cb_run2;\
+setgroup nlmsg mnl_nlmsg_size;\
+  add2group mnl_nlmsg_get_payload_len mnl_nlmsg_put_header;\
+  add2group mnl_nlmsg_put_extra_header mnl_nlmsg_get_payload;\
+  add2group mnl_nlmsg_get_payload_offset mnl_nlmsg_ok mnl_nlmsg_next;\
+  add2group mnl_nlmsg_get_payload_tail mnl_nlmsg_seq_ok mnl_nlmsg_portid_ok;\
+  add2group mnl_nlmsg_fprintf;\
+setgroup socket mnl_socket_get_fd;\
+  add2group mnl_socket_get_portid mnl_socket_open mnl_socket_open2;\
+  add2group mnl_socket_fdopen mnl_socket_bind mnl_socket_sendto;\
+  add2group mnl_socket_recvfrom mnl_socket_close mnl_socket_setsockopt;\
+  add2group mnl_socket_getsockopt;\
+};\
+setgroup() { mv $$1.3 $$2.3; BASE=$$2; };\
+add2group() { for i in $$@; do ln -sf $$BASE.3 $$i.3; done; };\
+main
+
+	touch doxyfile.stamp
+
+CLEANFILES = doxyfile.stamp
+
+all-local: doxyfile.stamp
+clean-local:
+	rm -rf $(top_srcdir)/doxygen/man $(top_srcdir)/doxygen/html
+install-data-local:
+	mkdir -p $(DESTDIR)$(mandir)/man3
+	cp --no-dereference --preserve=links,mode,timestamps man/man3/*.3 $(DESTDIR)$(mandir)/man3/
+
+# make distcheck needs uninstall-local
+uninstall-local:
+	rm -r $(DESTDIR)$(mandir) man html doxyfile.stamp
+endif
-- 
2.17.5

Re: [PATCH libmnl 1/1] build: doc: "make" builds & installs a full set of man pages

[Pablo Neira Ayuso]

On Tue, Jun 22, 2021 at 02:19:33PM +1000, Duncan Roe wrote:
> Repeat what we did for libnetfilter_queue:
>  - New makefile in doxygen directory. Rebuilds documentation if any sources
>    change that contain doxygen comments:
>    - Renames each group man page to the first function listed therein
>    - Creates symlinks for subsequently listed functions
>    - Deletes _* temp files and moves sctruct-describing man pages to man7
>  - Update top-level makefile to visit new subdir doxygen
>  - Update top-level configure to only build documentation if doxygen installed
>  - Add --with/without-doxygen switch
>  - Check whether dot is available when configuring doxygen
>  - Reduce size of doxygen.cfg and doxygen build o/p
>  - `make distcheck` passes with doxygen enabled
> Aditionally, exclude opaque structs mnl_nlmsg_batch & mnl_socket

Applied, thanks.

One thing that needs a fix (both libnetfilter_queue and libmnl).

If doxygen is not installed...

configure: WARNING: Doxygen not found - continuing without Doxygen support

it warns that it is missing...

checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating src/Makefile
config.status: creating include/Makefile
config.status: creating include/libmnl/Makefile
config.status: creating include/linux/Makefile
config.status: creating include/linux/netfilter/Makefile
config.status: creating examples/Makefile
config.status: creating examples/genl/Makefile
config.status: creating examples/kobject/Makefile
config.status: creating examples/netfilter/Makefile
config.status: creating examples/rtnl/Makefile
config.status: creating libmnl.pc
config.status: creating doxygen.cfg
config.status: creating doxygen/Makefile
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing depfiles commands
config.status: executing libtool commands

libmnl configuration:
  doxygen:          yes

but it says yes here.

I'd prefer if documentation is not enabled by default, ie. users have
to explicitly specify --with-doxygen=yes to build documentation, so
users explicitly picks what they needs.

Please, follow up with a few patches, thanks.

Re: [PATCH libmnl 1/1] build: doc: "make" builds & installs a full set of man pages

[Duncan Roe]

On Wed, Jun 23, 2021 at 07:26:21PM +0200, Pablo Neira Ayuso wrote:

[...]
>
> Applied, thanks.
>
> One thing that needs a fix (both libnetfilter_queue and libmnl).
>
> If doxygen is not installed...
>
> configure: WARNING: Doxygen not found - continuing without Doxygen support
>
> it warns that it is missing...
>
> checking that generated files are newer than configure... done
> configure: creating ./config.status
> config.status: creating Makefile
> config.status: creating src/Makefile
> config.status: creating include/Makefile
> config.status: creating include/libmnl/Makefile
> config.status: creating include/linux/Makefile
> config.status: creating include/linux/netfilter/Makefile
> config.status: creating examples/Makefile
> config.status: creating examples/genl/Makefile
> config.status: creating examples/kobject/Makefile
> config.status: creating examples/netfilter/Makefile
> config.status: creating examples/rtnl/Makefile
> config.status: creating libmnl.pc
> config.status: creating doxygen.cfg
> config.status: creating doxygen/Makefile
> config.status: creating config.h
> config.status: config.h is unchanged
> config.status: executing depfiles commands
> config.status: executing libtool commands
>
> libmnl configuration:
>   doxygen:          yes
>
> but it says yes here.
>
>
> I'd prefer if documentation is not enabled by default, ie. users have
> to explicitly specify --with-doxygen=yes to build documentation, so
> users explicitly picks what they needs.

I'm fine with *html* being optional:

  --enable-html      build HTML documentation [default=no]

ATM `make install` doesn't do anything with the html dir. With --enable-html, I
guess it should install html/ where --htmldir points [DOCDIR].

But I think not having man pages in the past was a serious deficiency which we
can now address.

Think of it from a (Linux) Distributor's point of view. Man pages take up very
little space in the distribution medium: symlinks are removed and the remaining
pages compressed. Man pages stay compressed on installation and the symlinks are
re-created by the postinstall script (and now as .gz or whatever files).

Typical end-users of the distribution won't have source, so the *need*
documentation.

Personally I'm happy if the build depends on doxygen and fails if it's not
installed.

If you inmsist on only printing a warning when doxygen is not installed then in
that event .configure could output:

> libmnl configuration:
>   doxygen:          no
> man pages:          no

with no "man pages" line when doxygen is installed.

What do you think?

Cheers ... Duncan.

Re: [PATCH libmnl 1/1] build: doc: "make" builds & installs a full set of man pages

[Duncan Roe]

On Sun, Jun 27, 2021 at 02:30:55PM +1000, Duncan Roe wrote:

[...]
>
> > libmnl configuration:
> >   doxygen:          no
> > man pages:          no
>
[...]
I meant:

> libmnl configuration:
>        html:          no
>   man pages:          no

Cheers ... Duncan.

Re: [PATCH libmnl 1/1] build: doc: "make" builds & installs a full set of man pages

[Pablo Neira Ayuso]

On Sun, Jun 27, 2021 at 02:30:55PM +1000, Duncan Roe wrote:
> On Wed, Jun 23, 2021 at 07:26:21PM +0200, Pablo Neira Ayuso wrote:
> 
> [...]
> >
> > Applied, thanks.
> >
> > One thing that needs a fix (both libnetfilter_queue and libmnl).
> >
> > If doxygen is not installed...
> >
> > configure: WARNING: Doxygen not found - continuing without Doxygen support
> >
> > it warns that it is missing...
> >
> > checking that generated files are newer than configure... done
> > configure: creating ./config.status
> > config.status: creating Makefile
> > config.status: creating src/Makefile
> > config.status: creating include/Makefile
> > config.status: creating include/libmnl/Makefile
> > config.status: creating include/linux/Makefile
> > config.status: creating include/linux/netfilter/Makefile
> > config.status: creating examples/Makefile
> > config.status: creating examples/genl/Makefile
> > config.status: creating examples/kobject/Makefile
> > config.status: creating examples/netfilter/Makefile
> > config.status: creating examples/rtnl/Makefile
> > config.status: creating libmnl.pc
> > config.status: creating doxygen.cfg
> > config.status: creating doxygen/Makefile
> > config.status: creating config.h
> > config.status: config.h is unchanged
> > config.status: executing depfiles commands
> > config.status: executing libtool commands
> >
> > libmnl configuration:
> >   doxygen:          yes
> >
> > but it says yes here.
> >
> >
> > I'd prefer if documentation is not enabled by default, ie. users have
> > to explicitly specify --with-doxygen=yes to build documentation, so
> > users explicitly picks what they needs.
> 
> I'm fine with *html* being optional:
> 
>   --enable-html      build HTML documentation [default=no]
> 
> ATM `make install` doesn't do anything with the html dir. With --enable-html, I
> guess it should install html/ where --htmldir points [DOCDIR].
> 
> But I think not having man pages in the past was a serious deficiency which we
> can now address.
> 
> Think of it from a (Linux) Distributor's point of view. Man pages take up very
> little space in the distribution medium: symlinks are removed and the remaining
> pages compressed. Man pages stay compressed on installation and the symlinks are
> re-created by the postinstall script (and now as .gz or whatever files).

We are not Linux distributors, it's up to them to decide what they are
shipping in their packages, this debate is out of our scope. Assuming
that enabling this by default will not make them include this.

Most users rely on libmnl because they use some utility that pulls in
this dependency, most of them are not developers.

> Typical end-users of the distribution won't have source, so the *need*
> documentation.
> 
> Personally I'm happy if the build depends on doxygen and fails if it's not
> installed.

That's probably fine for you, but embedded developer will not be happy
with this dependency.

> If you inmsist on only printing a warning when doxygen is not installed then in
> that event .configure could output:
> 
> > libmnl configuration:
> >   doxygen:          no
> > man pages:          no
> 
> with no "man pages" line when doxygen is installed.

I'd really prefer to retain the existing default that has been in
place for many years.

> What do you think?

It's also possible to provide up to date snapshots in the Netfilter
website to give it more visibility.

BTW, the autogenerated manpage differs quite a bit from standard
manpages in other existing packages? Do you know of any other software
following this approach of converting doxygen to manpage?

Thanks.

Re: [PATCH libmnl 1/1] build: doc: "make" builds & installs a full set of man pages

[Duncan Roe]

On Tue, Jun 29, 2021 at 11:38:37AM +0200, Pablo Neira Ayuso wrote:
> On Sun, Jun 27, 2021 at 02:30:55PM +1000, Duncan Roe wrote:
> > On Wed, Jun 23, 2021 at 07:26:21PM +0200, Pablo Neira Ayuso wrote:
> >
> > [...]
> > >
> > > Applied, thanks.
> > >
> > > One thing that needs a fix (both libnetfilter_queue and libmnl).
> > >
> > > If doxygen is not installed...
> > >
> > > configure: WARNING: Doxygen not found - continuing without Doxygen support
> > >
> > > it warns that it is missing...
> > >
> > > checking that generated files are newer than configure... done
> > > configure: creating ./config.status
> > > config.status: creating Makefile
> > > config.status: creating src/Makefile
> > > config.status: creating include/Makefile
> > > config.status: creating include/libmnl/Makefile
> > > config.status: creating include/linux/Makefile
> > > config.status: creating include/linux/netfilter/Makefile
> > > config.status: creating examples/Makefile
> > > config.status: creating examples/genl/Makefile
> > > config.status: creating examples/kobject/Makefile
> > > config.status: creating examples/netfilter/Makefile
> > > config.status: creating examples/rtnl/Makefile
> > > config.status: creating libmnl.pc
> > > config.status: creating doxygen.cfg
> > > config.status: creating doxygen/Makefile
> > > config.status: creating config.h
> > > config.status: config.h is unchanged
> > > config.status: executing depfiles commands
> > > config.status: executing libtool commands
> > >
> > > libmnl configuration:
> > >   doxygen:          yes
> > >
> > > but it says yes here.
> > >
> > >
> > > I'd prefer if documentation is not enabled by default, ie. users have
> > > to explicitly specify --with-doxygen=yes to build documentation, so
> > > users explicitly picks what they needs.
> >
> > I'm fine with *html* being optional:
> >
> >   --enable-html      build HTML documentation [default=no]
> >
> > ATM `make install` doesn't do anything with the html dir. With --enable-html, I
> > guess it should install html/ where --htmldir points [DOCDIR].
> >
> > But I think not having man pages in the past was a serious deficiency which we
> > can now address.
> >
> > Think of it from a (Linux) Distributor's point of view. Man pages take up very
> > little space in the distribution medium: symlinks are removed and the remaining
> > pages compressed. Man pages stay compressed on installation and the symlinks are
> > re-created by the postinstall script (and now as .gz or whatever files).
>
> We are not Linux distributors, it's up to them to decide what they are
> shipping in their packages, this debate is out of our scope. Assuming
> that enabling this by default will not make them include this.

Not sure about that. It seems to me, most distros build the default
configuration and remove stuff. F.I. if there is a "-devel" package variant then
the base package will have neither man pages nor header files. Yet we always
distribute header files. Man pages are a little bigger, but less than twice the
size of headers for libmnl & libnfq (i.e. compressed pages with symlinks).
>
> Most users rely on libmnl because they use some utility that pulls in
> this dependency, most of them are not developers.

So they won't install the -devel package.

[...]

Will address your other points in a future email,

Cheers ... Duncan.

Re: [PATCH libmnl 1/1] build: doc: "make" builds & installs a full set of man pages

[Duncan Roe]

On Tue, Jun 29, 2021 at 11:38:37AM +0200, Pablo Neira Ayuso wrote: [...]
> Do you know of any other software
> following this approach of converting doxygen to manpage?

No I do not. Otherwise I would have based my patches on them.

Doxygen does not play nicely with GNU autools. The best post I found on the
subject is
http://chris-miceli.blogspot.com/2011/01/integrating-doxygen-with-autotools.html
Please be sure to read the comments at the end if you go there ;)

Cheers ... Duncan.

Re: [PATCH libmnl 1/1] build: doc: "make" builds & installs a full set of man pages

[Duncan Roe]

Hi Pablo,

On Tue, Jun 29, 2021 at 11:38:37AM +0200, Pablo Neira Ayuso wrote:
> BTW, the autogenerated manpage differs quite a bit from standard
> manpages in other existing packages?

I noticed one substantial difference - missing the SYNOPSIS listing the required
#include lines.

The patch below adresses that, for the libnfq pktbuff modules.

After applying the patch, do you still see any substantial shortcomings in e.g.
`man pktb_alloc`?

Using doxygen does impose some stylistic differences, but compared to, say, `man
XShape` I think pktb_alloc(3) looks pretty normal? (You might need to install
libXext-devel for XShape(3)).

If you would apply the libnfq patch, I can get on with fixing the rest of
libnfq and libmnl.

Cheers ... Duncan.

Duncan Roe (1):
  src: doc: supply missing SYNOPSIS in pktbuff man pages

 src/extra/pktbuff.c | 45 +++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 41 insertions(+), 4 deletions(-)

-- 
2.17.5

[PATCH RFC libnetfilter_queue 1/1] src: doc: supply missing SYNOPSIS in pktbuff man pages

[Duncan Roe]

Synopsis only has required headers, since doxygen already output the function
summary. HTML only has 2 small changes for better man page readability:
the Synopsis lines are manonly (had to do that to get SYNOPSIS over to cc 1).

Signed-off-by: Duncan Roe <duncan_roe@optusnet.com.au>
---
 src/extra/pktbuff.c | 45 +++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 41 insertions(+), 4 deletions(-)

diff --git a/src/extra/pktbuff.c b/src/extra/pktbuff.c
index 9bdc6bd..8e9a8c7 100644
--- a/src/extra/pktbuff.c
+++ b/src/extra/pktbuff.c
@@ -23,8 +23,21 @@
 /**
  * \defgroup pktbuff User-space network packet buffer
  *
- * This library provides the user-space network packet buffer. This abstraction
- * is strongly inspired by Linux kernel network buffer, the so-called sk_buff.
+ * These functions provide the user-space network packet buffer.
+ * This abstraction is strongly inspired by Linux kernel network buffer,
+ * the so-called sk_buff.
+ *
+ * \manonly
+.RE
+.RS -7
+.nf
+\fBSYNOPSIS
+.RE
+#include <stdint.h>
+#include <stdbool.h>
+#include <sys/socket.h>
+#include <libnetfilter_queue/pktbuff.h>
+\endmanonly
  *
  * @{
  */
@@ -150,8 +163,20 @@ void pktb_free(struct pkt_buff *pktb)
  * \n
  * 1. Functions to get values of members of opaque __struct pktbuff__, described
  * below
- * \n
+ *
  * 2. Internal functions, described in Module __Internal functions__
+ *
+ * \manonly
+.RE
+.RS -7
+.nf
+\fBSYNOPSIS
+.RE
+#include <stdint.h>
+#include <stdbool.h>
+#include <sys/socket.h>
+#include <libnetfilter_queue/pktbuff.h>
+\endmanonly
  *
  * @{
  */
@@ -159,11 +184,23 @@ void pktb_free(struct pkt_buff *pktb)
 /**
  * \defgroup uselessfns Internal functions
  *
- * \warning Do not use these functions. Instead, always use the mangle
+ * Do not use these functions. Instead, always use the mangle
  * function appropriate to the level at which you are working.
  * \n
  * pktb_mangle() uses all the below functions except _pktb_pull_, which is not
  * used by anything.
+ *
+ * \manonly
+.RE
+.RS -7
+.nf
+\fBSYNOPSIS
+.RE
+#include <stdint.h>
+#include <stdbool.h>
+#include <sys/socket.h>
+#include <libnetfilter_queue/pktbuff.h>
+\endmanonly
  *
  * @{
  */
-- 
2.17.5

Re: [PATCH libmnl 1/1] build: doc: "make" builds & installs a full set of man pages

[Duncan Roe]

Hi Pablo,

On Tue, Jun 29, 2021 at 11:38:37AM +0200, Pablo Neira Ayuso wrote:
> > On Wed, Jun 23, 2021 at 07:26:21PM +0200, Pablo Neira Ayuso wrote:
> > >
[...]
> > > I'd prefer if documentation is not enabled by default, ie. users have
> > > to explicitly specify --with-doxygen=yes to build documentation, so
> > > users explicitly picks what they needs.
[...]
> I'd really prefer to retain the existing default that has been in
> place for many years.
>
Agreed that was a sensible default for many years.

The man pages had obscure names and were prone to clash with other man pages,
e.g. `man tcp` got you the libnfq page instead of tcp(7) and IIRC there was even
a clash between libnfq and libmnl.

At libnfq 1.0.5 there were no such clashes, and `man {any nfq function}` got you
the documentation for that function. However, having man pages on by default
broke `make distcheck`, so it was still sensible to have them off.

Now `make distcheck` passes with man pages on by default.

So I strongly suggest it is no longer sensible to have man page creation off by
default.

For the embedded guys, let's definitely keep --with-doxygen[=yes]. They can
always say no. Or not have it installed in the first place. More of that in an
email to come,

Cheers ... Duncan.

Re: [PATCH libmnl 1/1] build: doc: "make" builds & installs a full set of man pages

[Duncan Roe]

On Tue, Jun 29, 2021 at 11:38:37AM +0200, Pablo Neira Ayuso wrote:
> On Sun, Jun 27, 2021 at 02:30:55PM +1000, Duncan Roe wrote:
> > On Wed, Jun 23, 2021 at 07:26:21PM +0200, Pablo Neira Ayuso wrote:
> >
> > [...]
> > >
> > > Applied, thanks.
> > >
> > > One thing that needs a fix (both libnetfilter_queue and libmnl).
> > >
> > > If doxygen is not installed...
> > >
> > > configure: WARNING: Doxygen not found - continuing without Doxygen support

Can I suggest we change this to:

> configure: WARNING: Doxygen not found - assuming --with-doxygen = no

and set the appropriate variable (not sure how easy that is: I didn't do this
stuff originally)

How about

> libmnl configuration:
> html docs:          no
> man pages:          no

if doxygen is disabled for any reason, otherwise

> libmnl configuration:
> html docs:          yes if --enable-html else no
> man pages:          yes

i.e. never mentioning doxygen?

Cheers ... Duncan.

Re: [PATCH RFC libnetfilter_queue 1/1] src: doc: supply missing SYNOPSIS in pktbuff man pages

[Pablo Neira Ayuso]

On Sat, Jul 17, 2021 at 12:53:50PM +1000, Duncan Roe wrote:
> Synopsis only has required headers, since doxygen already output the function
> summary. HTML only has 2 small changes for better man page readability:
> the Synopsis lines are manonly (had to do that to get SYNOPSIS over to cc 1).

It's fine to add more specific "manonly" information.

The existing autogenerated manpages are still a bit distant to usual
Linux Programmer's Manual manpages at quick glance.

Go ahead.

Re: [PATCH RFC libnetfilter_queue 1/1] src: doc: supply missing SYNOPSIS in pktbuff man pages

[Duncan Roe]

On Thu, Jul 22, 2021 at 07:10:15PM +0200, Pablo Neira Ayuso wrote:
> The existing autogenerated manpages are still a bit distant to usual
> Linux Programmer's Manual manpages at quick glance.

I guess doxygen-generated man pages will always be a little bit "distinctive".

Looking carefully, I see a number of doxygen artefacts that can be
post-processed away.

And I do see one substantive difference: the NAME section should have a list of
functions instead of the (internal use only) module name. So:
> pktbuff - User-space network packet buffer
should be
> pktb_alloc, pktb_data, pktb_len, pktb_free, pktb_mangle, pktb_mangled - User-space network packet buffer

The rest is window-dressing. Here's a list of changes (from top down (mostly)):

 - Fix NAME entry as above
 - If there is a "Modules" section, delete it
 - If "Detailed Description" is empty, delete "Detailed Description" line
   - otherwise rename it "Overview" maybe, since Function Documentation
     contains detailed descriptions
 - Reposition SYNOPSIS line through 1st blank line to before "Functions" line
 - Delete "Functions" line
 - Delete all "Definition at line nnn" lines
 - Delete "Author" section (or should I leave that?)

My sed-fu is not up to doing this. I'll be using the rather more programmable
but little-known Q editor. Q is available at
https://sourceforge.net/projects/q-editor/ or from
https://github.com/duncan-roe/q

I'll put together a Q-macro suite to implement the above. Then I'll post
before/after samples of a short and a long page, say icmp & pktbuff.

If you think the changes are worth having, can then update Makefile.am with a
post-processing section that only runs if Q is available.

Cheeers ... Duncan.

Re: [PATCH RFC libnetfilter_queue 1/1] src: doc: supply missing SYNOPSIS in pktbuff man pages

[Pablo Neira Ayuso]

On Sat, Jul 24, 2021 at 01:49:41PM +1000, Duncan Roe wrote:
> On Thu, Jul 22, 2021 at 07:10:15PM +0200, Pablo Neira Ayuso wrote:
> > The existing autogenerated manpages are still a bit distant to usual
> > Linux Programmer's Manual manpages at quick glance.
> 
> I guess doxygen-generated man pages will always be a little bit "distinctive".
> 
> Looking carefully, I see a number of doxygen artefacts that can be
> post-processed away.

I mean: If you can achieve this via standard features available in
doxygen to keep this maintainable along time, such as \manonly or
such, then that's good.

Otherwise, extending doxygen should be the way to keep it simple.
I understand doxygen might generate a little but "distintive" output,
that's OK. I'd rather not to add more dependencies on this.

Thanks.

Re: [PATCH RFC libnetfilter_queue 1/1] src: doc: supply missing SYNOPSIS in pktbuff man pages

[Duncan Roe]

[-- Attachment #1: Type: text/plain, Size: 1459 bytes --]

On Sat, Jul 24, 2021 at 10:56:33AM +0200, Pablo Neira Ayuso wrote:
> On Sat, Jul 24, 2021 at 01:49:41PM +1000, Duncan Roe wrote:
> > On Thu, Jul 22, 2021 at 07:10:15PM +0200, Pablo Neira Ayuso wrote:
> > > The existing autogenerated manpages are still a bit distant to usual
> > > Linux Programmer's Manual manpages at quick glance.
> >
> > I guess doxygen-generated man pages will always be a little bit "distinctive".
> >
> > Looking carefully, I see a number of doxygen artefacts that can be
> > post-processed away.
>
> I mean: If you can achieve this via standard features available in
> doxygen to keep this maintainable along time, such as \manonly or
> such, then that's good.
>
> Otherwise, extending doxygen should be the way to keep it simple.
> I understand doxygen might generate a little but "distintive" output,
> that's OK. I'd rather not to add more dependencies on this.
>
> Thanks.

Fine with not adding more dependencies.

Using only standard shell tools, I went ahead with post-processing man pages
because there are no standard features available in doxygen to do what I wanted.

I've attached a sample of what I hope you'll see on applying my 2 new patches
"Insert SYNOPSIS sections for man pages" and "Fix NAME entry in man pages".
The sample was made by doxygen-1.9.1, earlier versions e.g. doxygen-1.8.9.1
generate a few extra blank lines.

To view, save attachment and in another window `man ./pktb_alloc.3.gz`.

Cheers ... Duncan.

[-- Attachment #2: pktb_alloc.3.gz --]
[-- Type: application/x-gunzip, Size: 1667 bytes --]

Re: [PATCH RFC libnetfilter_queue 1/1] src: doc: supply missing SYNOPSIS in pktbuff man pages

[Duncan Roe]

On Mon, Aug 09, 2021 at 06:22:17PM +1000, Duncan Roe wrote:
[...]Sat, Jul 24, 2021 at 10:56:33AM +0200, Pablo Neira Ayuso wrote:
> The sample was made by doxygen-1.9.1, earlier versions e.g. doxygen-1.8.9.1
> generate a few extra blank lines.
>
It's the older man command inserting the blank lines, not doxygen.

Cheers ... Duncan.