From: "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> To: git@vger.kernel.org Cc: Jonathan Tan <jonathantanmy@google.com>, Derrick Stolee <derrickstolee@github.com>, Derrick Stolee <dstolee@microsoft.com> Subject: [PATCH v5 10/11] maintenance: add auto condition for commit-graph task Date: Thu, 17 Sep 2020 18:11:51 +0000 [thread overview] Message-ID: <7a2a4e1e5278c7c94e0a6cfe1a1917724f826b0e.1600366313.git.gitgitgadget@gmail.com> (raw) In-Reply-To: <pull.695.v5.git.1600366313.gitgitgadget@gmail.com> From: Derrick Stolee <dstolee@microsoft.com> Instead of writing a new commit-graph in every 'git maintenance run --auto' process (when maintenance.commit-graph.enalbed is configured to be true), only write when there are "enough" commits not in a commit-graph file. This count is controlled by the maintenance.commit-graph.auto config option. To compute the count, use a depth-first search starting at each ref, and leaving markers using the SEEN flag. If this count reaches the limit, then terminate early and start the task. Otherwise, this operation will peel every ref and parse the commit it points to. If these are all in the commit-graph, then this is typically a very fast operation. Users with many refs might feel a slow-down, and hence could consider updating their limit to be very small. A negative value will force the step to run every time. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> --- Documentation/config/maintenance.txt | 10 ++++ builtin/gc.c | 82 ++++++++++++++++++++++++++++ object.h | 1 + 3 files changed, 93 insertions(+) diff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt index 4402b8b49f..7cc6700d57 100644 --- a/Documentation/config/maintenance.txt +++ b/Documentation/config/maintenance.txt @@ -4,3 +4,13 @@ maintenance.<task>.enabled:: `git maintenance run`. These config values are ignored if a `--task` option exists. By default, only `maintenance.gc.enabled` is true. + +maintenance.commit-graph.auto:: + This integer config option controls how often the `commit-graph` task + should be run as part of `git maintenance run --auto`. If zero, then + the `commit-graph` task will not run with the `--auto` option. A + negative value will force the task to run every time. Otherwise, a + positive value implies the command should run when the number of + reachable commits that are not in the commit-graph file is at least + the value of `maintenance.commit-graph.auto`. The default value is + 100. diff --git a/builtin/gc.c b/builtin/gc.c index 13c24bca7d..4db853c561 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -28,6 +28,7 @@ #include "blob.h" #include "tree.h" #include "promisor-remote.h" +#include "refs.h" #define FAILED_RUN "failed to run %s" @@ -710,6 +711,86 @@ struct maintenance_run_opts { int quiet; }; +/* Remember to update object flag allocation in object.h */ +#define SEEN (1u<<0) + +struct cg_auto_data { + int num_not_in_graph; + int limit; +}; + +static int dfs_on_ref(const char *refname, + const struct object_id *oid, int flags, + void *cb_data) +{ + struct cg_auto_data *data = (struct cg_auto_data *)cb_data; + int result = 0; + struct object_id peeled; + struct commit_list *stack = NULL; + struct commit *commit; + + if (!peel_ref(refname, &peeled)) + oid = &peeled; + if (oid_object_info(the_repository, oid, NULL) != OBJ_COMMIT) + return 0; + + commit = lookup_commit(the_repository, oid); + if (!commit) + return 0; + if (parse_commit(commit)) + return 0; + + commit_list_append(commit, &stack); + + while (!result && stack) { + struct commit_list *parent; + + commit = pop_commit(&stack); + + for (parent = commit->parents; parent; parent = parent->next) { + if (parse_commit(parent->item) || + commit_graph_position(parent->item) != COMMIT_NOT_FROM_GRAPH || + parent->item->object.flags & SEEN) + continue; + + parent->item->object.flags |= SEEN; + data->num_not_in_graph++; + + if (data->num_not_in_graph >= data->limit) { + result = 1; + break; + } + + commit_list_append(parent->item, &stack); + } + } + + free_commit_list(stack); + return result; +} + +static int should_write_commit_graph(void) +{ + int result; + struct cg_auto_data data; + + data.num_not_in_graph = 0; + data.limit = 100; + git_config_get_int("maintenance.commit-graph.auto", + &data.limit); + + if (!data.limit) + return 0; + if (data.limit < 0) + return 1; + + result = for_each_ref(dfs_on_ref, &data); + + clear_commit_marks_all(SEEN); + + return result; +} + static int run_write_commit_graph(struct maintenance_run_opts *opts) { struct child_process child = CHILD_PROCESS_INIT; @@ -790,6 +871,7 @@ static struct maintenance_task tasks[] = { [TASK_COMMIT_GRAPH] = { "commit-graph", maintenance_task_commit_graph, + should_write_commit_graph, }, }; diff --git a/object.h b/object.h index 96a2105859..20b18805f0 100644 --- a/object.h +++ b/object.h @@ -73,6 +73,7 @@ struct object_array { * sha1-name.c: 20 * list-objects-filter.c: 21 * builtin/fsck.c: 0--3 + * builtin/gc.c: 0 * builtin/index-pack.c: 2021 * builtin/pack-objects.c: 20 * builtin/reflog.c: 10--12 -- gitgitgadget
next prev parent reply other threads:[~2020-09-17 18:15 UTC|newest] Thread overview: 91+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-08-06 15:48 [PATCH 00/11] Maintenance I: Command, gc and commit-graph tasks Derrick Stolee via GitGitGadget 2020-08-06 15:48 ` [PATCH 01/11] maintenance: create basic maintenance runner Derrick Stolee via GitGitGadget 2020-08-07 22:16 ` Martin Ågren 2020-08-12 21:03 ` Jonathan Nieder 2020-08-12 22:07 ` Junio C Hamano 2020-08-12 22:50 ` Jonathan Nieder 2020-08-14 1:05 ` Derrick Stolee 2020-08-06 15:48 ` [PATCH 02/11] maintenance: add --quiet option Derrick Stolee via GitGitGadget 2020-08-06 15:48 ` [PATCH 03/11] maintenance: replace run_auto_gc() Derrick Stolee via GitGitGadget 2020-08-06 15:48 ` [PATCH 04/11] maintenance: initialize task array Derrick Stolee via GitGitGadget 2020-08-06 15:48 ` [PATCH 05/11] maintenance: add commit-graph task Derrick Stolee via GitGitGadget 2020-08-07 22:29 ` Martin Ågren 2020-08-12 13:30 ` Derrick Stolee 2020-08-14 12:23 ` Martin Ågren 2020-08-06 15:48 ` [PATCH 06/11] maintenance: add --task option Derrick Stolee via GitGitGadget 2020-08-06 15:48 ` [PATCH 07/11] maintenance: take a lock on the objects directory Derrick Stolee via GitGitGadget 2020-08-06 15:48 ` [PATCH 08/11] maintenance: create maintenance.<task>.enabled config Derrick Stolee via GitGitGadget 2020-08-06 15:48 ` [PATCH 09/11] maintenance: use pointers to check --auto Derrick Stolee via GitGitGadget 2020-08-06 15:48 ` [PATCH 10/11] maintenance: add auto condition for commit-graph task Derrick Stolee via GitGitGadget 2020-08-06 15:48 ` [PATCH 11/11] maintenance: add trace2 regions for task execution Derrick Stolee via GitGitGadget 2020-08-18 14:22 ` [PATCH v2 00/11] Maintenance I: Command, gc and commit-graph tasks Derrick Stolee via GitGitGadget 2020-08-18 14:22 ` [PATCH v2 01/11] maintenance: create basic maintenance runner Derrick Stolee via GitGitGadget 2020-08-18 14:22 ` [PATCH v2 02/11] maintenance: add --quiet option Derrick Stolee via GitGitGadget 2020-08-18 14:23 ` [PATCH v2 03/11] maintenance: replace run_auto_gc() Derrick Stolee via GitGitGadget 2020-08-18 14:23 ` [PATCH v2 04/11] maintenance: initialize task array Derrick Stolee via GitGitGadget 2020-08-18 23:46 ` Jonathan Tan 2020-08-18 14:23 ` [PATCH v2 05/11] maintenance: add commit-graph task Derrick Stolee via GitGitGadget 2020-08-18 23:51 ` Jonathan Tan 2020-08-19 15:04 ` Derrick Stolee 2020-08-19 17:43 ` Jonathan Tan 2020-08-18 14:23 ` [PATCH v2 06/11] maintenance: add --task option Derrick Stolee via GitGitGadget 2020-08-19 0:00 ` Jonathan Tan 2020-08-19 0:36 ` Junio C Hamano 2020-08-19 15:09 ` Derrick Stolee 2020-08-19 17:35 ` Jonathan Tan 2020-08-18 14:23 ` [PATCH v2 07/11] maintenance: take a lock on the objects directory Derrick Stolee via GitGitGadget 2020-08-19 0:04 ` Jonathan Tan 2020-08-19 15:10 ` Derrick Stolee 2020-08-18 14:23 ` [PATCH v2 08/11] maintenance: create maintenance.<task>.enabled config Derrick Stolee via GitGitGadget 2020-08-18 14:23 ` [PATCH v2 09/11] maintenance: use pointers to check --auto Derrick Stolee via GitGitGadget 2020-08-18 14:23 ` [PATCH v2 10/11] maintenance: add auto condition for commit-graph task Derrick Stolee via GitGitGadget 2020-08-19 0:09 ` Jonathan Tan 2020-08-19 15:15 ` Derrick Stolee 2020-08-18 14:23 ` [PATCH v2 11/11] maintenance: add trace2 regions for task execution Derrick Stolee via GitGitGadget 2020-08-19 0:11 ` Jonathan Tan 2020-08-18 20:18 ` [PATCH v2 00/11] Maintenance I: Command, gc and commit-graph tasks Junio C Hamano 2020-08-19 14:51 ` Derrick Stolee 2020-08-25 18:33 ` [PATCH v3 " Derrick Stolee via GitGitGadget 2020-08-25 18:33 ` [PATCH v3 01/11] maintenance: create basic maintenance runner Derrick Stolee via GitGitGadget 2020-08-25 18:33 ` [PATCH v3 02/11] maintenance: add --quiet option Derrick Stolee via GitGitGadget 2020-08-25 18:33 ` [PATCH v3 03/11] maintenance: replace run_auto_gc() Derrick Stolee via GitGitGadget 2020-08-25 18:33 ` [PATCH v3 04/11] maintenance: initialize task array Derrick Stolee via GitGitGadget 2020-08-25 18:33 ` [PATCH v3 05/11] maintenance: add commit-graph task Derrick Stolee via GitGitGadget 2020-08-25 18:33 ` [PATCH v3 06/11] maintenance: add --task option Derrick Stolee via GitGitGadget 2020-08-25 18:33 ` [PATCH v3 07/11] maintenance: take a lock on the objects directory Derrick Stolee via GitGitGadget 2020-08-26 23:02 ` Jonathan Tan 2020-08-25 18:33 ` [PATCH v3 08/11] maintenance: create maintenance.<task>.enabled config Derrick Stolee via GitGitGadget 2020-08-25 18:33 ` [PATCH v3 09/11] maintenance: use pointers to check --auto Derrick Stolee via GitGitGadget 2020-08-25 18:33 ` [PATCH v3 10/11] maintenance: add auto condition for commit-graph task Derrick Stolee via GitGitGadget 2020-08-26 23:02 ` Jonathan Tan 2020-08-26 23:56 ` Junio C Hamano 2020-08-25 18:33 ` [PATCH v3 11/11] maintenance: add trace2 regions for task execution Derrick Stolee via GitGitGadget 2020-09-04 13:09 ` [PATCH v4 00/11] Maintenance I: Command, gc and commit-graph tasks Derrick Stolee via GitGitGadget 2020-09-04 13:09 ` [PATCH v4 01/11] maintenance: create basic maintenance runner Derrick Stolee via GitGitGadget 2020-09-04 13:09 ` [PATCH v4 02/11] maintenance: add --quiet option Derrick Stolee via GitGitGadget 2020-09-04 13:09 ` [PATCH v4 03/11] maintenance: replace run_auto_gc() Derrick Stolee via GitGitGadget 2020-09-04 13:09 ` [PATCH v4 04/11] maintenance: initialize task array Derrick Stolee via GitGitGadget 2020-09-04 13:09 ` [PATCH v4 05/11] maintenance: add commit-graph task Derrick Stolee via GitGitGadget 2020-09-04 13:09 ` [PATCH v4 06/11] maintenance: add --task option Derrick Stolee via GitGitGadget 2020-09-04 13:09 ` [PATCH v4 07/11] maintenance: take a lock on the objects directory Derrick Stolee via GitGitGadget 2020-09-04 13:09 ` [PATCH v4 08/11] maintenance: create maintenance.<task>.enabled config Derrick Stolee via GitGitGadget 2020-09-04 13:09 ` [PATCH v4 09/11] maintenance: use pointers to check --auto Derrick Stolee via GitGitGadget 2020-09-04 13:09 ` [PATCH v4 10/11] maintenance: add auto condition for commit-graph task Derrick Stolee via GitGitGadget 2020-09-04 13:09 ` [PATCH v4 11/11] maintenance: add trace2 regions for task execution Derrick Stolee via GitGitGadget 2020-09-17 18:11 ` [PATCH v5 00/11] Maintenance I: Command, gc and commit-graph tasks Derrick Stolee via GitGitGadget 2020-09-17 18:11 ` [PATCH v5 01/11] maintenance: create basic maintenance runner Derrick Stolee via GitGitGadget 2020-09-17 18:11 ` [PATCH v5 02/11] maintenance: add --quiet option Derrick Stolee via GitGitGadget 2020-09-17 18:11 ` [PATCH v5 03/11] maintenance: replace run_auto_gc() Derrick Stolee via GitGitGadget 2020-09-17 18:11 ` [PATCH v5 04/11] maintenance: initialize task array Derrick Stolee via GitGitGadget 2020-09-17 18:11 ` [PATCH v5 05/11] maintenance: add commit-graph task Derrick Stolee via GitGitGadget 2020-09-17 18:11 ` [PATCH v5 06/11] maintenance: add --task option Derrick Stolee via GitGitGadget 2020-09-17 18:11 ` [PATCH v5 07/11] maintenance: take a lock on the objects directory Derrick Stolee via GitGitGadget 2020-09-21 13:36 ` Ævar Arnfjörð Bjarmason 2020-09-21 13:43 ` Derrick Stolee 2020-09-21 19:29 ` Junio C Hamano 2020-09-17 18:11 ` [PATCH v5 08/11] maintenance: create maintenance.<task>.enabled config Derrick Stolee via GitGitGadget 2020-09-17 18:11 ` [PATCH v5 09/11] maintenance: use pointers to check --auto Derrick Stolee via GitGitGadget 2020-09-17 18:11 ` Derrick Stolee via GitGitGadget [this message] 2020-09-17 18:11 ` [PATCH v5 11/11] maintenance: add trace2 regions for task execution Derrick Stolee via GitGitGadget 2020-09-17 18:35 ` [PATCH v5 00/11] Maintenance I: Command, gc and commit-graph tasks Junio C Hamano 2020-09-18 13:14 ` Johannes Schindelin
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=7a2a4e1e5278c7c94e0a6cfe1a1917724f826b0e.1600366313.git.gitgitgadget@gmail.com \ --to=gitgitgadget@gmail.com \ --cc=derrickstolee@github.com \ --cc=dstolee@microsoft.com \ --cc=git@vger.kernel.org \ --cc=jonathantanmy@google.com \ --subject='Re: [PATCH v5 10/11] maintenance: add auto condition for commit-graph task' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).