install-step fails for pandoc-prebuilt man-pages in infiniband-diags/man

* install-step fails for pandoc-prebuilt man-pages in infiniband-diags/man
@ 2019-12-04 14:34 Thorben Römer
  2019-12-05  8:12 ` Leon Romanovsky
  2019-12-09 19:39 ` Jason Gunthorpe
  0 siblings, 2 replies; 5+ messages in thread
From: Thorben Römer @ 2019-12-04 14:34 UTC (permalink / raw)
  To: linux-rdma

I tested the following on release 0.26.0 and 0.26.1, with the *.tar.gz
provided on github. It is probably relevant for a lot more released
archives and future releases. My build environment is restricted and
does not have access to pandoc or rst2man.

The command 'ninja install' fails in environments without pandoc and/or
rst2man. There is an automatic fallback to use man-pages in
buildlib/pandoc-prebuilt that are added for every release-archive.
However, the install step fails with error such as:

CMake Error at infiniband-diags/man/cmake_install.cmake:40 (file):
file INSTALL cannot find
"rdma-core-26.0/buildlib/pandoc-prebuilt/64a3de4dd91635b29f4f8d11a987670751827c60".
Call Stack (most recent call first):
cmake_install.cmake:166 (include)

To reproduce the issue, you can try building (and installing) one of the
*.tar.gz releases from github, while preventing the checks in
buildlib/Findpandoc.cmake and /Findrst2man.cmake from finding anything
(e.g. by appending random letters to the searched binary name).

While investigating, I came up with the following explanation: The
hashes (generated by buildlib/pandoc-prebuilt.py) differ from machine to
machine, as the contents of the *.rst-files are hashed. Most of these
files are processed via cmake's configure_file from *.in.rst-files and
contain custom-per-build-data such as absolute paths. This means that
hashing *.rst-files will produce differing hashes based on the
build-directory-path (among other data points, possibly). I tested this
by looking at the hashes produced by two of my machines, they were
different for all but 3 files (ibcacheedit.8.rst, ibstatus.8.rst and
check_lft_balance.8.rst). These files (and their includes, as their
content is also hashed!) are the only files that to not contain any
differing data when being transformed from *.in.rst to *.rst via
configure_file, which supports my hypothesis.

From my viewpoint, this tells me that

a) many/all released archives do not contain the correct pandoc-prebuilt
files for infiniband-diags/man
b) there is no automated test that builds and installs releases without
pandoc/rst2man present

With my limited time and expertise in the rdma-core project, I was only
able to come up with a solution that I don't find very practical. I will
append a diff of pandoc-prebuilt.py nonetheless, which replaces
hashing-calls for *.rst to *.in.rst if applicable. With these changes,
the hashes for all files are stable between my two machines. I tested
this by building rdma-core with my pandoc-prebuilt.py on a machine with
pandoc+rst2man, which produces the pandoc-prebuilt files. I can copy
those files to buildlib/pandoc-prebuilt on my other machine in the
restricted build environment (that also has my pandoc-prebuilt.py) and
the 'ninja install' step succeeds.

This is probably not the most suitable solution for this problem and I
do NOT consider it fit for production. But maybe it helps someone
understand and/or fix the root issue. Another fix could be to just hash
the filenames instead of their contents.

Thorben

--- rdma-core-26.0/buildlib/pandoc-prebuilt.py  2019-10-02
12:57:38.000000000 +0200
+++ pandoc-prebuilt.py  2019-12-04 15:21:14.755359320 +0100
@@ -6,10 +6,35 @@
 import hashlib
 import re

+# we shouldn't hash *.rst-files, as they can contain absolute paths
after being
+# transformed from an *.in.rst-file via cmake's configure_file. this
leads to
+# differing pandoc-prebuilt-ids from the build process and other build
+# environments that use pandoc-prebuilt (no rst2man and pandoc).
+# this functions instead corrects the filename from *.rst to *.in.rst
+# if applicable (file exists, correct file ext, ...)
+def infile_if_possible(SRC):
+    # only check *.rst files
+    if not SRC.endswith(".rst"):
+        return SRC;
+
+    # already the desired file
+    if SRC.endswith(".in.rst"):
+        return SRC;
+
+    infile = SRC.rsplit(".", 1)[0] + ".in.rst";
+
+    # use *.in.rst file if possible
+    if os.path.exists(infile):
+        return infile;
+
+    # *fallback*, this still leads to the same error
+    # maybe throw / fail here?
+    return SRC;
+
 def hash_rst_includes(incdir,txt):
     h = ""
     for fn in re.findall(br"^..\s+include::\s+(.*)$", txt,
flags=re.MULTILINE):
-        with open(os.path.join(incdir,fn.decode()),"rb") as F:
+        with
open(infile_if_possible(os.path.join(incdir,fn.decode())),"rb") as F:
             h = h +  hashlib.sha1(F.read()).hexdigest();
     return h.encode();

@@ -17,7 +42,7 @@
     """Return a unique ID for the SRC file. For simplicity and
robustness we just
     content hash it"""
     incdir = os.path.dirname(SRC)
-    with open(SRC,"rb") as F:
+    with open(infile_if_possible(SRC),"rb") as F:
         txt = F.read();
         if SRC.endswith(".rst"):
             txt = txt + hash_rst_includes(incdir,txt);

^ permalink raw reply	[flat|nested] 5+ messages in thread