[PATCH v4 0/7] scalar: implement the subcommand "diagnose"

["Johannes Schindelin via GitGitGadget" <gitgitgadget@gmail.com>]

Over the course of the years, we developed a sub-command that gathers
diagnostic data into a .zip file that can then be attached to bug reports.
This sub-command turned out to be very useful in helping Scalar developers
identify and fix issues.

Changes since v3:

 * We're now using unquote_c_style() instead of rolling our own unquoter.
 * Fixed the added regression test.
 * As pointed out by Scalar's Functional Tests, the
   add_directory_to_archiver() function should not fail when scalar diagnose
   encounters FSMonitor's Unix socket, but only warn instead.
 * Related: add_directory_to_archiver() needs to propagate errors from
   processing subdirectories so that the top-level call returns an error,
   too.

Changes since v2:

 * Clarified in the commit message what the biggest benefit of
   --add-file-with-content is.
 * The <path> part of the -add-file-with-content argument can now contain
   colons. To do this, the path needs to start and end in double-quote
   characters (which are stripped), and the backslash serves as escape
   character in that case (to allow the path to contain both colons and
   double-quotes).
 * Fixed incorrect grammar.
 * Instead of strcmp(<what-we-don't-want>), we now say
   !strcmp(<what-we-want>).
 * The help text for --add-file-with-content was improved a tiny bit.
 * Adjusted the commit message that still talked about spawning plenty of
   processes and about a throw-away repository for the sake of generating a
   .zip file.
 * Simplified the code that shows the diagnostics and adds them to the .zip
   file.
 * The final message that reports that the archive is complete is now
   printed to stderr instead of stdout.

Changes since v1:

 * Instead of creating a throw-away repository, staging the contents of the
   .zip file and then using git write-tree and git archive to write the .zip
   file, the patch series now introduces a new option to git archive and
   uses write_archive() directly (avoiding any separate process).
 * Since the command avoids separate processes, it is now blazing fast on
   Windows, and I dropped the spinner() function because it's no longer
   needed.
 * While reworking the test case, I noticed that scalar [...] <enlistment>
   failed to verify that the specified directory exists, and would happily
   "traverse to its parent directory" on its quest to find a Scalar
   enlistment. That is of course incorrect, and has been fixed as a "while
   at it" sort of preparatory commit.
 * I had forgotten to sign off on all the commits, which has been fixed.
 * Instead of some "home-grown" readdir()-based function, the code now uses
   for_each_file_in_pack_dir() to look through the pack directories.
 * If any alternates are configured, their pack directories are now included
   in the output.
 * The commit message that might be interpreted to promise information about
   large loose files has been corrected to no longer promise that.
 * The test cases have been adjusted to test a little bit more (e.g.
   verifying that specific paths are mentioned in the output, instead of
   merely verifying that the output is non-empty).

Johannes Schindelin (5):
  archive: optionally add "virtual" files
  archive --add-file-with-contents: allow paths containing colons
  scalar: validate the optional enlistment argument
  Implement `scalar diagnose`
  scalar diagnose: include disk space information

Matthew John Cheetham (2):
  scalar: teach `diagnose` to gather packfile info
  scalar: teach `diagnose` to gather loose objects information

 Documentation/git-archive.txt    |  17 ++
 archive.c                        |  61 ++++++-
 contrib/scalar/scalar.c          | 292 ++++++++++++++++++++++++++++++-
 contrib/scalar/scalar.txt        |  12 ++
 contrib/scalar/t/t9099-scalar.sh |  27 +++
 t/t5003-archive-zip.sh           |  20 +++
 6 files changed, 419 insertions(+), 10 deletions(-)


base-commit: ddc35d833dd6f9e8946b09cecd3311b8aa18d295
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1128%2Fdscho%2Fscalar-diagnose-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1128/dscho/scalar-diagnose-v4
Pull-Request: https://github.com/gitgitgadget/git/pull/1128

Range-diff vs v3:

 1:  45662cf582a = 1:  45662cf582a archive: optionally add "virtual" files
 2:  ce4b1b680c9 ! 2:  fdba4ed6f4d archive --add-file-with-contents: allow paths containing colons
     @@ Documentation/git-archive.txt: OPTIONS
      -command-line limits. For non-trivial cases, write an untracked file
      -and use `--add-file` instead.
      +The `<path>` argument can start and end with a literal double-quote
     -+character. In this case, the backslash is interpreted as escape
     -+character. The path must be quoted if it contains a colon, to avoid
     -+the colon from being misinterpreted as the separator between the
     -+path and the contents.
     ++character; The contained file name is interpreted as a C-style string,
     ++i.e. the backslash is interpreted as escape character. The path must
     ++be quoted if it contains a colon, to avoid the colon from being
     ++misinterpreted as the separator between the path and the contents, or
     ++if the path begins or ends with a double-quote character.
      ++
      +The file mode is limited to a regular file, and the option may be
      +subject to platform-dependent command-line limits. For non-trivial
     @@ Documentation/git-archive.txt: OPTIONS
       	Look for attributes in .gitattributes files in the working tree
      
       ## archive.c ##
     +@@
     + #include "parse-options.h"
     + #include "unpack-trees.h"
     + #include "dir.h"
     ++#include "quote.h"
     + 
     + static char const * const archive_usage[] = {
     + 	N_("git archive [<options>] <tree-ish> [<path>...]"),
      @@ archive.c: static int add_file_cb(const struct option *opt, const char *arg, int unset)
       			die(_("Not a regular file: %s"), path);
       		info->content = NULL; /* read the file later */
       	} else {
      -		const char *colon = strchr(arg, ':');
     - 		char *p;
     +-		char *p;
     ++		struct strbuf buf = STRBUF_INIT;
     ++		const char *p = arg;
     ++
     ++		if (*p != '"')
     ++			p = strchr(p, ':');
     ++		else if (unquote_c_style(&buf, p, &p) < 0)
     ++			die(_("unclosed quote: '%s'"), arg);
       
      -		if (!colon)
     --			die(_("missing colon: '%s'"), arg);
     -+		if (*arg != '"') {
     -+			const char *colon = strchr(arg, ':');
     -+
     -+			if (!colon)
     -+				die(_("missing colon: '%s'"), arg);
     -+			p = xstrndup(arg, colon - arg);
     -+			arg = colon + 1;
     -+		} else {
     -+			struct strbuf buf = STRBUF_INIT;
     -+			const char *orig = arg;
     -+
     -+			for (;;) {
     -+				if (!*(++arg))
     -+					die(_("unclosed quote: '%s'"), orig);
     -+				if (*arg == '"')
     -+					break;
     -+				if (*arg == '\\' && *(++arg) == '\0')
     -+					die(_("trailing backslash: '%s"), orig);
     -+				else
     -+					strbuf_addch(&buf, *arg);
     -+			}
     -+
     -+			if (*(++arg) != ':')
     -+				die(_("missing colon: '%s'"), orig);
     -+
     -+			p = strbuf_detach(&buf, NULL);
     -+			arg++;
     -+		}
     ++		if (!p || *p != ':')
     + 			die(_("missing colon: '%s'"), arg);
       
      -		p = xstrndup(arg, colon - arg);
     - 		if (!args->prefix)
     - 			path = p;
     - 		else {
     -@@ archive.c: static int add_file_cb(const struct option *opt, const char *arg, int unset)
     +-		if (!args->prefix)
     +-			path = p;
     +-		else {
     +-			path = prefix_filename(args->prefix, p);
     +-			free(p);
     ++		if (p == arg)
     ++			die(_("empty file name: '%s'"), arg);
     ++
     ++		path = buf.len ?
     ++			strbuf_detach(&buf, NULL) : xstrndup(arg, p - arg);
     ++
     ++		if (args->prefix) {
     ++			char *save = path;
     ++			path = prefix_filename(args->prefix, path);
     ++			free(save);
       		}
       		memset(&info->stat, 0, sizeof(info->stat));
       		info->stat.st_mode = S_IFREG | 0644;
      -		info->content = xstrdup(colon + 1);
     -+		info->content = xstrdup(arg);
     ++		info->content = xstrdup(p + 1);
       		info->stat.st_size = strlen(info->content);
       	}
       	item = string_list_append_nodup(&args->extra_files, path);
 3:  5a3eeb55409 = 3:  da9f52a8240 scalar: validate the optional enlistment argument
 4:  dfe821d10fe ! 4:  87bdc22322b Implement `scalar diagnose`
     @@ contrib/scalar/scalar.c: static int unregister_dir(void)
      +		if (e->d_type == DT_REG)
      +			strvec_pushf(archiver_args, "--add-file=%s", buf.buf);
      +		else if (e->d_type != DT_DIR)
     ++			warning(_("skipping '%s', which is neither file nor "
     ++				  "directory"), buf.buf);
     ++		else if (recurse &&
     ++			 add_directory_to_archiver(archiver_args,
     ++						   buf.buf, recurse) < 0)
      +			res = -1;
     -+		else if (recurse)
     -+		     add_directory_to_archiver(archiver_args, buf.buf, recurse);
      +	}
      +
      +	closedir(dir);
     @@ contrib/scalar/t/t9099-scalar.sh: test_expect_success '`scalar [...] <dir>` erro
      +SQ="'"
      +test_expect_success UNZIP 'scalar diagnose' '
      +	scalar clone "file://$(pwd)" cloned --single-branch &&
     -+	scalar diagnose cloned >out &&
     -+	sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" <out >zip_path &&
     ++	scalar diagnose cloned >out 2>err &&
     ++	sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" <err >zip_path &&
      +	zip_path=$(cat zip_path) &&
      +	test -n "$zip_path" &&
      +	unzip -v "$zip_path" &&
 5:  bb162abd383 ! 5:  3f63b197d42 scalar diagnose: include disk space information
     @@ contrib/scalar/t/t9099-scalar.sh
      @@ contrib/scalar/t/t9099-scalar.sh: SQ="'"
       test_expect_success UNZIP 'scalar diagnose' '
       	scalar clone "file://$(pwd)" cloned --single-branch &&
     - 	scalar diagnose cloned >out &&
     + 	scalar diagnose cloned >out 2>err &&
      +	grep "Available space" out &&
     - 	sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" <out >zip_path &&
     + 	sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" <err >zip_path &&
       	zip_path=$(cat zip_path) &&
       	test -n "$zip_path" &&
 6:  32aaad7cce1 ! 6:  fc1319338fc scalar: teach `diagnose` to gather packfile info
     @@ contrib/scalar/t/t9099-scalar.sh: test_expect_success '`scalar [...] <dir>` erro
       	scalar clone "file://$(pwd)" cloned --single-branch &&
      +	git repack &&
      +	echo "$(pwd)/.git/objects/" >>cloned/src/.git/objects/info/alternates &&
     - 	scalar diagnose cloned >out &&
     + 	scalar diagnose cloned >out 2>err &&
       	grep "Available space" out &&
     - 	sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" <out >zip_path &&
     + 	sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" <err >zip_path &&
      @@ contrib/scalar/t/t9099-scalar.sh: test_expect_success UNZIP 'scalar diagnose' '
       	folder=${zip_path%.zip} &&
       	test_path_is_missing "$folder" &&
 7:  322932f0bb8 ! 7:  e8f5b42f7b7 scalar: teach `diagnose` to gather loose objects information
     @@ contrib/scalar/t/t9099-scalar.sh: test_expect_success UNZIP 'scalar diagnose' '
       	git repack &&
       	echo "$(pwd)/.git/objects/" >>cloned/src/.git/objects/info/alternates &&
      +	test_commit -C cloned/src loose &&
     - 	scalar diagnose cloned >out &&
     + 	scalar diagnose cloned >out 2>err &&
       	grep "Available space" out &&
     - 	sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" <out >zip_path &&
     + 	sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" <err >zip_path &&
      @@ contrib/scalar/t/t9099-scalar.sh: test_expect_success UNZIP 'scalar diagnose' '
       	unzip -p "$zip_path" diagnostics.log >out &&
       	test_file_not_empty out &&

-- 
gitgitgadget