From: Jakub Narebski <jnareb@gmail.com>
To: Derrick Stolee <dstolee@microsoft.com>
Cc: "git\@vger.kernel.org" <git@vger.kernel.org>,
"gitster\@pobox.com" <gitster@pobox.com>,
"stolee\@gmail.com" <stolee@gmail.com>,
"avarab\@gmail.com" <avarab@gmail.com>,
"marten.agren\@gmail.com" <marten.agren@gmail.com>,
"peff\@peff.net" <peff@peff.net>
Subject: Re: [PATCH v3 17/20] fsck: verify commit-graph
Date: Sat, 02 Jun 2018 18:17:36 +0200 [thread overview]
Message-ID: <867enhtb6n.fsf@gmail.com> (raw)
In-Reply-To: <20180524162504.158394-18-dstolee@microsoft.com> (Derrick Stolee's message of "Thu, 24 May 2018 16:26:03 +0000")
Derrick Stolee <dstolee@microsoft.com> writes:
> If core.commitGraph is true, verify the contents of the commit-graph
> during 'git fsck' using the 'git commit-graph verify' subcommand. Run
> this check on all alternates, as well.
All right, so we have one config variable to control the use of
serialized commit-graph feaature. Nice.
>
> We use a new process for two reasons:
>
> 1. The subcommand decouples the details of loading and verifying a
> commit-graph file from the other fsck details.
All right, I can agree with that.
On the other hand using subcommand makes debugging harder, though not in
this case (well separated functionality that can be easily called with a
standalone command to be debugged).
>
> 2. The commit-graph verification requires the commits to be loaded
> in a specific order to guarantee we parse from the commit-graph
> file for some objects and from the object database for others.
I don't quite understand this. Could you explain it in more detail?
>
> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> ---
> Documentation/git-fsck.txt | 3 +++
> builtin/fsck.c | 21 +++++++++++++++++++++
> t/t5318-commit-graph.sh | 8 ++++++++
> 3 files changed, 32 insertions(+)
>
> diff --git a/Documentation/git-fsck.txt b/Documentation/git-fsck.txt
> index b9f060e3b2..ab9a93fb9b 100644
> --- a/Documentation/git-fsck.txt
> +++ b/Documentation/git-fsck.txt
> @@ -110,6 +110,9 @@ Any corrupt objects you will have to find in backups or other archives
> (i.e., you can just remove them and do an 'rsync' with some other site in
> the hopes that somebody else has the object you have corrupted).
>
> +If core.commitGraph is true, the commit-graph file will also be inspected
Shouldn't we use `core.commitGraph` here?
> +using 'git commit-graph verify'. See linkgit:git-commit-graph[1].
> +
> Extracted Diagnostics
> ---------------------
>
> diff --git a/builtin/fsck.c b/builtin/fsck.c
> index ef78c6c00c..a6d5045b77 100644
> --- a/builtin/fsck.c
> +++ b/builtin/fsck.c
> @@ -16,6 +16,7 @@
> #include "streaming.h"
> #include "decorate.h"
> #include "packfile.h"
> +#include "run-command.h"
>
> #define REACHABLE 0x0001
> #define SEEN 0x0002
> @@ -45,6 +46,7 @@ static int name_objects;
> #define ERROR_REACHABLE 02
> #define ERROR_PACK 04
> #define ERROR_REFS 010
> +#define ERROR_COMMIT_GRAPH 020
Minor nitpick and a sidenote: I wonder if it wouldn't be better to
either use hexadecimal constants, or use (1 << n) for all ERROR_*
preprocesor constants.
>
> static const char *describe_object(struct object *obj)
> {
> @@ -815,5 +817,24 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
> }
>
> check_connectivity();
> +
> + if (core_commit_graph) {
> + struct child_process commit_graph_verify = CHILD_PROCESS_INIT;
> + const char *verify_argv[] = { "commit-graph", "verify", NULL, NULL, NULL, NULL };
I see that NULL at index 2 and 3 (at 3rd and 4th place) are here for
"--object-dir" and <alternates-object-dir-path>, the last one is
terminator for that case, but what is next to last NULL (at 5th place)
for?
> + commit_graph_verify.argv = verify_argv;
> + commit_graph_verify.git_cmd = 1;
> +
> + if (run_command(&commit_graph_verify))
> + errors_found |= ERROR_COMMIT_GRAPH;
> +
> + prepare_alt_odb();
> + for (alt = alt_odb_list; alt; alt = alt->next) {
> + verify_argv[2] = "--object-dir";
> + verify_argv[3] = alt->path;
> + if (run_command(&commit_graph_verify))
> + errors_found |= ERROR_COMMIT_GRAPH;
> + }
> + }
For performance reasons it may be better to start those 'git
commit-graph verify' commands asynchronously earlier, so that they can
run in parallel / concurrently wth other checks, and wait for them and
get their error code at the end of git-fsck run.
But that is probably better left for a separate commit.
> +
> return errors_found;
> }
> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh
> index 2680a2ebff..4941937163 100755
> --- a/t/t5318-commit-graph.sh
> +++ b/t/t5318-commit-graph.sh
> @@ -394,4 +394,12 @@ test_expect_success 'detect invalid checksum hash' '
> "incorrect checksum"
> '
>
> +test_expect_success 'git fsck (checks commit-graph)' '
> + cd "$TRASH_DIRECTORY/full" &&
> + git fsck &&
> + corrupt_graph_and_verify $GRAPH_BYTE_FOOTER "\00" \
> + "incorrect checksum" &&
> + test_must_fail git fsck
> +'
All right; though the same caveats apply as with previous commit in
series. Perhaps it would be better to truncate commit-graph file, or
corrupt it in some 'random' place.
> +
> test_done
Best,
--
Jakub Narębski
next prev parent reply other threads:[~2018-06-02 16:17 UTC|newest]
Thread overview: 149+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-04-17 18:10 [RFC PATCH 00/12] Integrate commit-graph into 'fsck' and 'gc' Derrick Stolee
2018-04-17 18:10 ` [RFC PATCH 01/12] fixup! commit-graph: always load commit-graph information Derrick Stolee
2018-04-17 18:10 ` [RFC PATCH 02/12] commit-graph: add 'check' subcommand Derrick Stolee
2018-04-19 13:24 ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 03/12] commit-graph: check file header information Derrick Stolee
2018-04-19 15:58 ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 04/12] commit-graph: parse commit from chosen graph Derrick Stolee
2018-04-19 17:21 ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 05/12] commit-graph: check fanout and lookup table Derrick Stolee
2018-04-20 7:27 ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 06/12] commit: force commit to parse from object database Derrick Stolee
2018-04-20 12:13 ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 07/12] commit-graph: load a root tree from specific graph Derrick Stolee
2018-04-20 12:18 ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 08/12] commit-graph: verify commit contents against odb Derrick Stolee
2018-04-20 16:47 ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 10/12] commit-graph: add '--reachable' option Derrick Stolee
2018-04-20 17:17 ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 09/12] fsck: check commit-graph Derrick Stolee
2018-04-20 16:59 ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 11/12] gc: automatically write commit-graph files Derrick Stolee
2018-04-20 17:34 ` Jakub Narebski
2018-04-20 18:33 ` Ævar Arnfjörð Bjarmason
2018-04-17 18:10 ` [RFC PATCH 12/12] commit-graph: update design document Derrick Stolee
2018-04-20 19:10 ` Jakub Narebski
2018-04-17 18:50 ` [RFC PATCH 00/12] Integrate commit-graph into 'fsck' and 'gc' Derrick Stolee
2018-05-10 17:34 ` [PATCH 00/12] Integrate commit-graph into fsck, gc, and fetch Derrick Stolee
2018-05-10 17:34 ` [PATCH 01/12] commit-graph: add 'verify' subcommand Derrick Stolee
2018-05-10 18:15 ` Martin Ågren
2018-05-10 17:34 ` [PATCH 02/12] commit-graph: verify file header information Derrick Stolee
2018-05-10 18:21 ` Martin Ågren
2018-05-10 17:34 ` [PATCH 03/12] commit-graph: parse commit from chosen graph Derrick Stolee
2018-05-10 17:34 ` [PATCH 04/12] commit-graph: verify fanout and lookup table Derrick Stolee
2018-05-10 18:29 ` Martin Ågren
2018-05-11 15:17 ` Derrick Stolee
2018-05-10 17:34 ` [PATCH 05/12] commit: force commit to parse from object database Derrick Stolee
2018-05-10 17:34 ` [PATCH 06/12] commit-graph: load a root tree from specific graph Derrick Stolee
2018-05-10 17:34 ` [PATCH 07/12] commit-graph: verify commit contents against odb Derrick Stolee
2018-05-10 17:34 ` [PATCH 08/12] fsck: verify commit-graph Derrick Stolee
2018-05-10 17:34 ` [PATCH 09/12] commit-graph: add '--reachable' option Derrick Stolee
2018-05-10 17:34 ` [PATCH 10/12] gc: automatically write commit-graph files Derrick Stolee
2018-05-10 17:34 ` [PATCH 11/12] fetch: compute commit-graph by default Derrick Stolee
2018-05-10 17:34 ` [PATCH 12/12] commit-graph: update design document Derrick Stolee
2018-05-10 19:05 ` [PATCH 00/12] Integrate commit-graph into fsck, gc, and fetch Martin Ågren
2018-05-10 19:22 ` Stefan Beller
2018-05-11 17:23 ` Derrick Stolee
2018-05-11 17:30 ` Martin Ågren
2018-05-10 19:17 ` Ævar Arnfjörð Bjarmason
2018-05-11 17:23 ` Derrick Stolee
2018-05-11 21:15 ` [PATCH v2 00/12] Integrate commit-graph into fsck and gc Derrick Stolee
2018-05-11 21:15 ` [PATCH v2 01/12] commit-graph: add 'verify' subcommand Derrick Stolee
2018-05-12 13:31 ` Martin Ågren
2018-05-14 13:27 ` Derrick Stolee
2018-05-20 12:10 ` Jakub Narebski
2018-05-11 21:15 ` [PATCH v2 02/12] commit-graph: verify file header information Derrick Stolee
2018-05-12 13:35 ` Martin Ågren
2018-05-14 13:31 ` Derrick Stolee
2018-05-20 20:00 ` Jakub Narebski
2018-05-11 21:15 ` [PATCH v2 03/12] commit-graph: test that 'verify' finds corruption Derrick Stolee
2018-05-12 13:43 ` Martin Ågren
2018-05-21 18:53 ` Jakub Narebski
2018-05-24 16:28 ` Derrick Stolee
2018-05-11 21:15 ` [PATCH v2 04/12] commit-graph: parse commit from chosen graph Derrick Stolee
2018-05-12 20:50 ` Martin Ågren
2018-05-11 21:15 ` [PATCH v2 05/12] commit-graph: verify fanout and lookup table Derrick Stolee
2018-05-11 21:15 ` [PATCH v2 06/12] commit: force commit to parse from object database Derrick Stolee
2018-05-12 20:54 ` Martin Ågren
2018-05-11 21:15 ` [PATCH v2 07/12] commit-graph: load a root tree from specific graph Derrick Stolee
2018-05-12 20:55 ` Martin Ågren
2018-05-11 21:15 ` [PATCH v2 08/12] commit-graph: verify commit contents against odb Derrick Stolee
2018-05-12 21:17 ` Martin Ågren
2018-05-14 13:44 ` Derrick Stolee
2018-05-15 21:12 ` Martin Ågren
2018-05-11 21:15 ` [PATCH v2 09/12] fsck: verify commit-graph Derrick Stolee
2018-05-17 18:13 ` Martin Ågren
2018-05-11 21:15 ` [PATCH v2 10/12] commit-graph: add '--reachable' option Derrick Stolee
2018-05-17 18:16 ` Martin Ågren
2018-05-11 21:15 ` [PATCH v2 11/12] gc: automatically write commit-graph files Derrick Stolee
2018-05-17 18:20 ` Martin Ågren
2018-05-11 21:15 ` [PATCH v2 12/12] commit-graph: update design document Derrick Stolee
2018-05-24 16:25 ` [PATCH v3 00/20] Integrate commit-graph into 'fsck' and 'gc' Derrick Stolee
2018-05-24 16:25 ` [PATCH v3 01/20] commit-graph: UNLEAK before die() Derrick Stolee
2018-05-24 22:47 ` Stefan Beller
2018-05-25 0:08 ` Derrick Stolee
2018-05-24 16:25 ` [PATCH v3 02/20] commit-graph: fix GRAPH_MIN_SIZE Derrick Stolee
2018-05-26 18:46 ` Jakub Narebski
2018-05-26 20:30 ` brian m. carlson
2018-06-02 19:43 ` Jakub Narebski
2018-05-24 16:25 ` [PATCH v3 03/20] commit-graph: parse commit from chosen graph Derrick Stolee
2018-05-27 10:23 ` Jakub Narebski
2018-05-29 12:31 ` Derrick Stolee
2018-05-24 16:25 ` [PATCH v3 04/20] commit: force commit to parse from object database Derrick Stolee
2018-05-27 18:04 ` Jakub Narebski
2018-05-24 16:25 ` [PATCH v3 05/20] commit-graph: load a root tree from specific graph Derrick Stolee
2018-05-27 19:12 ` Jakub Narebski
2018-05-24 16:25 ` [PATCH v3 06/20] commit-graph: add 'verify' subcommand Derrick Stolee
2018-05-27 22:55 ` Jakub Narebski
2018-05-30 16:07 ` Derrick Stolee
2018-06-02 21:19 ` Jakub Narebski
2018-06-04 11:30 ` Derrick Stolee
2018-05-24 16:25 ` [PATCH v3 07/20] commit-graph: verify catches corrupt signature Derrick Stolee
2018-05-28 14:05 ` Jakub Narebski
2018-05-29 12:43 ` Derrick Stolee
2018-06-02 22:30 ` Jakub Narebski
2018-05-24 16:25 ` [PATCH v3 08/20] commit-graph: verify required chunks are present Derrick Stolee
2018-05-28 17:11 ` Jakub Narebski
2018-05-24 16:25 ` [PATCH v3 09/20] commit-graph: verify corrupt OID fanout and lookup Derrick Stolee
2018-05-30 13:34 ` Jakub Narebski
2018-05-30 16:18 ` Derrick Stolee
2018-06-02 4:38 ` Duy Nguyen
2018-06-04 11:32 ` Derrick Stolee
2018-06-04 14:42 ` Duy Nguyen
2018-05-24 16:25 ` [PATCH v3 10/20] commit-graph: verify objects exist Derrick Stolee
2018-05-30 19:22 ` Jakub Narebski
2018-05-31 12:53 ` Derrick Stolee
2018-05-24 16:25 ` [PATCH v3 11/20] commit-graph: verify root tree OIDs Derrick Stolee
2018-05-30 22:24 ` Jakub Narebski
2018-05-31 13:16 ` Derrick Stolee
2018-06-02 22:50 ` Jakub Narebski
2018-05-24 16:25 ` [PATCH v3 12/20] commit-graph: verify parent list Derrick Stolee
2018-06-01 23:21 ` Jakub Narebski
2018-05-24 16:25 ` [PATCH v3 13/20] commit-graph: verify generation number Derrick Stolee
2018-06-02 12:23 ` Jakub Narebski
2018-06-04 11:47 ` Derrick Stolee
2018-05-24 16:25 ` [PATCH v3 14/20] commit-graph: verify commit date Derrick Stolee
2018-06-02 12:29 ` Jakub Narebski
2018-05-24 16:25 ` [PATCH v3 15/20] commit-graph: test for corrupted octopus edge Derrick Stolee
2018-06-02 12:39 ` Jakub Narebski
2018-06-04 13:08 ` Derrick Stolee
2018-05-24 16:26 ` [PATCH v3 16/20] commit-graph: verify contents match checksum Derrick Stolee
2018-05-30 12:35 ` SZEDER Gábor
2018-06-02 15:52 ` Jakub Narebski
2018-06-04 11:55 ` Derrick Stolee
2018-05-24 16:26 ` [PATCH v3 17/20] fsck: verify commit-graph Derrick Stolee
2018-06-02 16:17 ` Jakub Narebski [this message]
2018-06-04 11:59 ` Derrick Stolee
2018-05-24 16:26 ` [PATCH v3 18/20] commit-graph: add '--reachable' option Derrick Stolee
2018-06-02 17:34 ` Jakub Narebski
2018-06-04 12:44 ` Derrick Stolee
2018-05-24 16:26 ` [PATCH v3 19/20] gc: automatically write commit-graph files Derrick Stolee
2018-06-02 18:03 ` Jakub Narebski
2018-06-04 12:51 ` Derrick Stolee
2018-05-24 16:26 ` [PATCH v3 20/20] commit-graph: update design document Derrick Stolee
2018-06-02 18:27 ` Jakub Narebski
2018-05-24 21:15 ` [PATCH v3 00/20] Integrate commit-graph into 'fsck' and 'gc' Ævar Arnfjörð Bjarmason
2018-05-25 4:11 ` Junio C Hamano
2018-05-29 4:27 ` Junio C Hamano
2018-05-29 12:37 ` Derrick Stolee
2018-05-29 13:41 ` Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=867enhtb6n.fsf@gmail.com \
--to=jnareb@gmail.com \
--cc=avarab@gmail.com \
--cc=dstolee@microsoft.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=marten.agren@gmail.com \
--cc=peff@peff.net \
--cc=stolee@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).