Linux-RDMA Archive on lore.kernel.org
 help / color / Atom feed
From: "Thorben Römer" <thorben.roemer@secunet.com>
To: <linux-rdma@vger.kernel.org>
Subject: install-step fails for pandoc-prebuilt man-pages in infiniband-diags/man
Date: Wed, 4 Dec 2019 15:34:14 +0100
Message-ID: <5d754108-7020-6041-1b7d-bbb3fb2f089b@secunet.com> (raw)

I tested the following on release 0.26.0 and 0.26.1, with the *.tar.gz
provided on github. It is probably relevant for a lot more released
archives and future releases. My build environment is restricted and
does not have access to pandoc or rst2man.

The command 'ninja install' fails in environments without pandoc and/or
rst2man. There is an automatic fallback to use man-pages in
buildlib/pandoc-prebuilt that are added for every release-archive.
However, the install step fails with error such as:

CMake Error at infiniband-diags/man/cmake_install.cmake:40 (file):
file INSTALL cannot find
"rdma-core-26.0/buildlib/pandoc-prebuilt/64a3de4dd91635b29f4f8d11a987670751827c60".
Call Stack (most recent call first):
cmake_install.cmake:166 (include)

To reproduce the issue, you can try building (and installing) one of the
*.tar.gz releases from github, while preventing the checks in
buildlib/Findpandoc.cmake and /Findrst2man.cmake from finding anything
(e.g. by appending random letters to the searched binary name).

While investigating, I came up with the following explanation: The
hashes (generated by buildlib/pandoc-prebuilt.py) differ from machine to
machine, as the contents of the *.rst-files are hashed. Most of these
files are processed via cmake's configure_file from *.in.rst-files and
contain custom-per-build-data such as absolute paths. This means that
hashing *.rst-files will produce differing hashes based on the
build-directory-path (among other data points, possibly). I tested this
by looking at the hashes produced by two of my machines, they were
different for all but 3 files (ibcacheedit.8.rst, ibstatus.8.rst and
check_lft_balance.8.rst). These files (and their includes, as their
content is also hashed!) are the only files that to not contain any
differing data when being transformed from *.in.rst to *.rst via
configure_file, which supports my hypothesis.

From my viewpoint, this tells me that

a) many/all released archives do not contain the correct pandoc-prebuilt
files for infiniband-diags/man
b) there is no automated test that builds and installs releases without
pandoc/rst2man present

With my limited time and expertise in the rdma-core project, I was only
able to come up with a solution that I don't find very practical. I will
append a diff of pandoc-prebuilt.py nonetheless, which replaces
hashing-calls for *.rst to *.in.rst if applicable. With these changes,
the hashes for all files are stable between my two machines. I tested
this by building rdma-core with my pandoc-prebuilt.py on a machine with
pandoc+rst2man, which produces the pandoc-prebuilt files. I can copy
those files to buildlib/pandoc-prebuilt on my other machine in the
restricted build environment (that also has my pandoc-prebuilt.py) and
the 'ninja install' step succeeds.

This is probably not the most suitable solution for this problem and I
do NOT consider it fit for production. But maybe it helps someone
understand and/or fix the root issue. Another fix could be to just hash
the filenames instead of their contents.

Thorben

--- rdma-core-26.0/buildlib/pandoc-prebuilt.py  2019-10-02
12:57:38.000000000 +0200
+++ pandoc-prebuilt.py  2019-12-04 15:21:14.755359320 +0100
@@ -6,10 +6,35 @@
 import hashlib
 import re

+# we shouldn't hash *.rst-files, as they can contain absolute paths
after being
+# transformed from an *.in.rst-file via cmake's configure_file. this
leads to
+# differing pandoc-prebuilt-ids from the build process and other build
+# environments that use pandoc-prebuilt (no rst2man and pandoc).
+# this functions instead corrects the filename from *.rst to *.in.rst
+# if applicable (file exists, correct file ext, ...)
+def infile_if_possible(SRC):
+    # only check *.rst files
+    if not SRC.endswith(".rst"):
+        return SRC;
+
+    # already the desired file
+    if SRC.endswith(".in.rst"):
+        return SRC;
+
+    infile = SRC.rsplit(".", 1)[0] + ".in.rst";
+
+    # use *.in.rst file if possible
+    if os.path.exists(infile):
+        return infile;
+
+    # *fallback*, this still leads to the same error
+    # maybe throw / fail here?
+    return SRC;
+
 def hash_rst_includes(incdir,txt):
     h = ""
     for fn in re.findall(br"^..\s+include::\s+(.*)$", txt,
flags=re.MULTILINE):
-        with open(os.path.join(incdir,fn.decode()),"rb") as F:
+        with
open(infile_if_possible(os.path.join(incdir,fn.decode())),"rb") as F:
             h = h +  hashlib.sha1(F.read()).hexdigest();
     return h.encode();

@@ -17,7 +42,7 @@
     """Return a unique ID for the SRC file. For simplicity and
robustness we just
     content hash it"""
     incdir = os.path.dirname(SRC)
-    with open(SRC,"rb") as F:
+    with open(infile_if_possible(SRC),"rb") as F:
         txt = F.read();
         if SRC.endswith(".rst"):
             txt = txt + hash_rst_includes(incdir,txt);

             reply index

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-04 14:34 Thorben Römer [this message]
2019-12-05  8:12 ` Leon Romanovsky

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=5d754108-7020-6041-1b7d-bbb3fb2f089b@secunet.com \
    --to=thorben.roemer@secunet.com \
    --cc=linux-rdma@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-RDMA Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-rdma/0 linux-rdma/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-rdma linux-rdma/ https://lore.kernel.org/linux-rdma \
		linux-rdma@vger.kernel.org
	public-inbox-index linux-rdma

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-rdma


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git