From: Matheus Tavares <matheus.bernardino@usp.br>
To: git@vger.kernel.org
Cc: gitster@pobox.com, git@jeffhostetler.com,
chriscool@tuxfamily.org, peff@peff.net, newren@gmail.com,
jrnieder@gmail.com, martin.agren@gmail.com
Subject: [PATCH v4 00/19] Parallel Checkout (part I)
Date: Wed, 4 Nov 2020 17:32:59 -0300 [thread overview]
Message-ID: <cover.1604521275.git.matheus.bernardino@usp.br> (raw)
In-Reply-To: <cover.1603937110.git.matheus.bernardino@usp.br>
Changes since v3:
Patch 1:
- Renamed 'enum crlf_action' to 'enum convert_crlf_action', since it's
now public and the latter suits better the global namespace.
Patch 2:
- Implemented the regular [async_]convert_to_working_tree() functions
as thin wrappers around the new _ca() variants.
Patches 5 and 6:
- Properly added blank lines to separate declaration blocks in entry.h.
Patch 8:
- Used a `struct conv_attrs ca_buf` (together with the `ca` pointer)
to avoid the early return in checkout_entry() when
S_ISREG(ce->ce_mode). I think this makes the patch a little easier
to parse and also simplifies the next patch.
Patch 10:
- Removed explicit zero initialization of the static struct parallel_checkout.
- Removed the check_leading_dirs() function, which had a quite generic
name, and integrated the code into write_pc_item().
Note: for this change, I used the find_last_dir_sep() helper, which
is slightly slower since it doesn't take the path's length, and
thus, it has to iterate the whole string. Alternatively, We could
add an strbuf_find_last_dir_sep() variant which takes an strbuf and
starts the search from the end, to save some iterations per path.
But this snippet will be removed in part 2, so I thought it wouldn't
be worth adding a new helper now.
- Rephrased comment about the has_dirs_only_path() call in workers, for
better clarity.
- Changed all unnecessary uses of ++pre_increment in the series to
post_increment++ (patches 10 and 11).
Jeff Hostetler (4):
convert: make convert_attrs() and convert structs public
convert: add [async_]convert_to_working_tree_ca() variants
convert: add get_stream_filter_ca() variant
convert: add conv_attrs classification
Matheus Tavares (15):
entry: extract a header file for entry.c functions
entry: make fstat_output() and read_blob_entry() public
entry: extract cache_entry update from write_entry()
entry: move conv_attrs lookup up to checkout_entry()
entry: add checkout_entry_ca() which takes preloaded conv_attrs
unpack-trees: add basic support for parallel checkout
parallel-checkout: make it truly parallel
parallel-checkout: support progress displaying
make_transient_cache_entry(): optionally alloc from mem_pool
builtin/checkout.c: complete parallel checkout support
checkout-index: add parallel checkout support
parallel-checkout: add tests for basic operations
parallel-checkout: add tests related to clone collisions
parallel-checkout: add tests related to .gitattributes
ci: run test round with parallel-checkout enabled
.gitignore | 1 +
Documentation/config/checkout.txt | 21 +
Makefile | 2 +
apply.c | 1 +
builtin.h | 1 +
builtin/checkout--helper.c | 142 ++++++
builtin/checkout-index.c | 22 +-
builtin/checkout.c | 21 +-
builtin/difftool.c | 3 +-
cache.h | 34 +-
ci/run-build-and-tests.sh | 1 +
convert.c | 130 ++---
convert.h | 96 +++-
entry.c | 102 ++--
entry.h | 55 +++
git.c | 2 +
parallel-checkout.c | 632 ++++++++++++++++++++++++
parallel-checkout.h | 103 ++++
read-cache.c | 12 +-
t/README | 4 +
t/lib-encoding.sh | 25 +
t/lib-parallel-checkout.sh | 46 ++
t/t0028-working-tree-encoding.sh | 25 +-
t/t2080-parallel-checkout-basics.sh | 170 +++++++
t/t2081-parallel-checkout-collisions.sh | 98 ++++
t/t2082-parallel-checkout-attributes.sh | 174 +++++++
unpack-trees.c | 22 +-
27 files changed, 1766 insertions(+), 179 deletions(-)
create mode 100644 builtin/checkout--helper.c
create mode 100644 entry.h
create mode 100644 parallel-checkout.c
create mode 100644 parallel-checkout.h
create mode 100644 t/lib-encoding.sh
create mode 100644 t/lib-parallel-checkout.sh
create mode 100755 t/t2080-parallel-checkout-basics.sh
create mode 100755 t/t2081-parallel-checkout-collisions.sh
create mode 100755 t/t2082-parallel-checkout-attributes.sh
Range-diff against v3:
1: dfc3e0fd62 ! 1: 2726f6dc05 convert: make convert_attrs() and convert structs public
@@ Commit message
Move convert_attrs() declaration from convert.c to convert.h, together
with the conv_attrs struct and the crlf_action enum. This function and
the data structures will be used outside convert.c in the upcoming
- parallel checkout implementation.
+ parallel checkout implementation. Note that crlf_action is renamed to
+ convert_crlf_action, which is more appropriate for the global namespace.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
[matheus.bernardino: squash and reword msg]
@@ convert.c
struct text_stat {
/* NUL, CR, LF and CRLF counts */
unsigned nul, lonecr, lonelf, crlf;
+@@ convert.c: static int text_eol_is_crlf(void)
+ return 0;
+ }
+
+-static enum eol output_eol(enum crlf_action crlf_action)
++static enum eol output_eol(enum convert_crlf_action crlf_action)
+ {
+ switch (crlf_action) {
+ case CRLF_BINARY:
+@@ convert.c: static int has_crlf_in_index(const struct index_state *istate, const char *path)
+ }
+
+ static int will_convert_lf_to_crlf(struct text_stat *stats,
+- enum crlf_action crlf_action)
++ enum convert_crlf_action crlf_action)
+ {
+ if (output_eol(crlf_action) != EOL_CRLF)
+ return 0;
+@@ convert.c: static int encode_to_worktree(const char *path, const char *src, size_t src_len,
+ static int crlf_to_git(const struct index_state *istate,
+ const char *path, const char *src, size_t len,
+ struct strbuf *buf,
+- enum crlf_action crlf_action, int conv_flags)
++ enum convert_crlf_action crlf_action, int conv_flags)
+ {
+ struct text_stat stats;
+ char *dst;
+@@ convert.c: static int crlf_to_git(const struct index_state *istate,
+ return 1;
+ }
+
+-static int crlf_to_worktree(const char *src, size_t len,
+- struct strbuf *buf, enum crlf_action crlf_action)
++static int crlf_to_worktree(const char *src, size_t len, struct strbuf *buf,
++ enum convert_crlf_action crlf_action)
+ {
+ char *to_free = NULL;
+ struct text_stat stats;
+@@ convert.c: static const char *git_path_check_encoding(struct attr_check_item *check)
+ return value;
+ }
+
+-static enum crlf_action git_path_check_crlf(struct attr_check_item *check)
++static enum convert_crlf_action git_path_check_crlf(struct attr_check_item *check)
+ {
+ const char *value = check->value;
+
@@ convert.c: static int git_path_check_ident(struct attr_check_item *check)
return !!ATTR_TRUE(value);
}
@@ convert.c: static int git_path_check_ident(struct attr_check_item *check)
## convert.h ##
-@@ convert.h: enum eol {
- #endif
+@@ convert.h: struct checkout_metadata {
+ struct object_id blob;
};
-+enum crlf_action {
++enum convert_crlf_action {
+ CRLF_UNDEFINED,
+ CRLF_BINARY,
+ CRLF_TEXT,
@@ convert.h: enum eol {
+
+struct conv_attrs {
+ struct convert_driver *drv;
-+ enum crlf_action attr_action; /* What attr says */
-+ enum crlf_action crlf_action; /* When no attr is set, use core.autocrlf */
++ enum convert_crlf_action attr_action; /* What attr says */
++ enum convert_crlf_action crlf_action; /* When no attr is set, use core.autocrlf */
+ int ident;
+ const char *working_tree_encoding; /* Supported encoding or default encoding if NULL */
+};
+
- enum ce_delay_state {
- CE_NO_DELAY = 0,
- CE_CAN_DELAY = 1,
-@@ convert.h: void convert_to_git_filter_fd(const struct index_state *istate,
- int would_convert_to_git_filter_fd(const struct index_state *istate,
- const char *path);
-
+void convert_attrs(const struct index_state *istate,
+ struct conv_attrs *ca, const char *path);
+
- /*
- * Initialize the checkout metadata with the given values. Any argument may be
- * NULL if it is not applicable. The treeish should be a commit if that is
+ extern enum eol core_eol;
+ extern char *check_roundtrip_encoding;
+ const char *get_cached_convert_stats_ascii(const struct index_state *istate,
2: c5fbd1e16d ! 2: fc03417592 convert: add [async_]convert_to_working_tree_ca() variants
@@ convert.c: static int convert_to_working_tree_internal(const struct index_state
return ret | ret_filter;
}
-@@ convert.c: int async_convert_to_working_tree(const struct index_state *istate,
- const struct checkout_metadata *meta,
- void *dco)
- {
-- return convert_to_working_tree_internal(istate, path, src, len, dst, 0, meta, dco);
-+ struct conv_attrs ca;
-+ convert_attrs(istate, &ca, path);
-+ return convert_to_working_tree_internal(&ca, path, src, len, dst, 0, meta, dco);
- }
- int convert_to_working_tree(const struct index_state *istate,
-@@ convert.c: int convert_to_working_tree(const struct index_state *istate,
- size_t len, struct strbuf *dst,
- const struct checkout_metadata *meta)
- {
-- return convert_to_working_tree_internal(istate, path, src, len, dst, 0, meta, NULL);
-+ struct conv_attrs ca;
-+ convert_attrs(istate, &ca, path);
-+ return convert_to_working_tree_internal(&ca, path, src, len, dst, 0, meta, NULL);
-+}
-+
+-int async_convert_to_working_tree(const struct index_state *istate,
+- const char *path, const char *src,
+- size_t len, struct strbuf *dst,
+- const struct checkout_metadata *meta,
+- void *dco)
+int async_convert_to_working_tree_ca(const struct conv_attrs *ca,
+ const char *path, const char *src,
+ size_t len, struct strbuf *dst,
+ const struct checkout_metadata *meta,
+ void *dco)
-+{
+ {
+- return convert_to_working_tree_internal(istate, path, src, len, dst, 0, meta, dco);
+ return convert_to_working_tree_internal(ca, path, src, len, dst, 0, meta, dco);
-+}
-+
+ }
+
+-int convert_to_working_tree(const struct index_state *istate,
+- const char *path, const char *src,
+- size_t len, struct strbuf *dst,
+- const struct checkout_metadata *meta)
+int convert_to_working_tree_ca(const struct conv_attrs *ca,
+ const char *path, const char *src,
+ size_t len, struct strbuf *dst,
+ const struct checkout_metadata *meta)
-+{
+ {
+- return convert_to_working_tree_internal(istate, path, src, len, dst, 0, meta, NULL);
+ return convert_to_working_tree_internal(ca, path, src, len, dst, 0, meta, NULL);
}
@@ convert.c: int convert_to_working_tree(const struct index_state *istate,
len = dst->len;
## convert.h ##
-@@ convert.h: int convert_to_working_tree(const struct index_state *istate,
- const char *path, const char *src,
- size_t len, struct strbuf *dst,
- const struct checkout_metadata *meta);
+@@ convert.h: const char *get_convert_attr_ascii(const struct index_state *istate,
+ int convert_to_git(const struct index_state *istate,
+ const char *path, const char *src, size_t len,
+ struct strbuf *dst, int conv_flags);
+-int convert_to_working_tree(const struct index_state *istate,
+- const char *path, const char *src,
+- size_t len, struct strbuf *dst,
+- const struct checkout_metadata *meta);
+-int async_convert_to_working_tree(const struct index_state *istate,
+- const char *path, const char *src,
+- size_t len, struct strbuf *dst,
+- const struct checkout_metadata *meta,
+- void *dco);
+int convert_to_working_tree_ca(const struct conv_attrs *ca,
+ const char *path, const char *src,
+ size_t len, struct strbuf *dst,
+ const struct checkout_metadata *meta);
- int async_convert_to_working_tree(const struct index_state *istate,
- const char *path, const char *src,
- size_t len, struct strbuf *dst,
- const struct checkout_metadata *meta,
- void *dco);
+int async_convert_to_working_tree_ca(const struct conv_attrs *ca,
+ const char *path, const char *src,
+ size_t len, struct strbuf *dst,
+ const struct checkout_metadata *meta,
+ void *dco);
++static inline int convert_to_working_tree(const struct index_state *istate,
++ const char *path, const char *src,
++ size_t len, struct strbuf *dst,
++ const struct checkout_metadata *meta)
++{
++ struct conv_attrs ca;
++ convert_attrs(istate, &ca, path);
++ return convert_to_working_tree_ca(&ca, path, src, len, dst, meta);
++}
++static inline int async_convert_to_working_tree(const struct index_state *istate,
++ const char *path, const char *src,
++ size_t len, struct strbuf *dst,
++ const struct checkout_metadata *meta,
++ void *dco)
++{
++ struct conv_attrs ca;
++ convert_attrs(istate, &ca, path);
++ return async_convert_to_working_tree_ca(&ca, path, src, len, dst, meta, dco);
++}
int async_query_available_blobs(const char *cmd,
struct string_list *available_paths);
int renormalize_buffer(const struct index_state *istate,
3: c77b16f694 = 3: 8ce20f1031 convert: add get_stream_filter_ca() variant
4: 18c3f4247e = 4: aa1eb461f4 convert: add conv_attrs classification
5: 2caa2c4345 ! 5: cb3dea224b entry: extract a header file for entry.c functions
@@ entry.h (new)
+#define CHECKOUT_INIT { NULL, "" }
+
+#define TEMPORARY_FILENAME_LENGTH 25
-+
+/*
+ * Write the contents from ce out to the working tree.
+ *
@@ entry.h (new)
+ */
+int checkout_entry(struct cache_entry *ce, const struct checkout *state,
+ char *topath, int *nr_checkouts);
++
+void enable_delayed_checkout(struct checkout *state);
+int finish_delayed_checkout(struct checkout *state, int *nr_checkouts);
++
+/*
+ * Unlink the last component and schedule the leading directories for
+ * removal, such that empty directories get removed.
6: bfa52df9e2 ! 6: 46ed6274d7 entry: make fstat_output() and read_blob_entry() public
@@ entry.c: static int write_entry(struct cache_entry *ce,
## entry.h ##
@@ entry.h: int finish_delayed_checkout(struct checkout *state, int *nr_checkouts);
- * removal, such that empty directories get removed.
*/
void unlink_entry(const struct cache_entry *ce);
+
+void *read_blob_entry(const struct cache_entry *ce, unsigned long *size);
+int fstat_checkout_output(int fd, const struct checkout *state, struct stat *st);
-
++
#endif /* ENTRY_H */
7: 91ef17f533 ! 7: a0479d02ff entry: extract cache_entry update from write_entry()
@@ entry.c: static int write_entry(struct cache_entry *ce,
return 0;
## entry.h ##
-@@ entry.h: int finish_delayed_checkout(struct checkout *state, int *nr_checkouts);
- void unlink_entry(const struct cache_entry *ce);
+@@ entry.h: void unlink_entry(const struct cache_entry *ce);
+
void *read_blob_entry(const struct cache_entry *ce, unsigned long *size);
int fstat_checkout_output(int fd, const struct checkout *state, struct stat *st);
+void update_ce_after_write(const struct checkout *state, struct cache_entry *ce,
8: 81e03baab1 ! 8: 5c993cc27f entry: move conv_attrs lookup up to checkout_entry()
@@ entry.c: int checkout_entry(struct cache_entry *ce, const struct checkout *state
{
static struct strbuf path = STRBUF_INIT;
struct stat st;
-+ struct conv_attrs ca;
++ struct conv_attrs ca_buf, *ca = NULL;
if (ce->ce_flags & CE_WT_REMOVE) {
if (topath)
@@ entry.c: int checkout_entry(struct cache_entry *ce, const struct checkout *state
- return write_entry(ce, topath, state, 1);
+ if (topath) {
+ if (S_ISREG(ce->ce_mode)) {
-+ convert_attrs(state->istate, &ca, ce->name);
-+ return write_entry(ce, topath, &ca, state, 1);
++ convert_attrs(state->istate, &ca_buf, ce->name);
++ ca = &ca_buf;
+ }
-+ return write_entry(ce, topath, NULL, state, 1);
++ return write_entry(ce, topath, ca, state, 1);
+ }
strbuf_reset(&path);
@@ entry.c: int checkout_entry(struct cache_entry *ce, const struct checkout *state
- return write_entry(ce, path.buf, state, 0);
+
+ if (S_ISREG(ce->ce_mode)) {
-+ convert_attrs(state->istate, &ca, ce->name);
-+ return write_entry(ce, path.buf, &ca, state, 0);
++ convert_attrs(state->istate, &ca_buf, ce->name);
++ ca = &ca_buf;
+ }
+
+ return write_entry(ce, path.buf, NULL, state, 0);
9: e1b886f823 ! 9: aa635bda21 entry: add checkout_entry_ca() which takes preloaded conv_attrs
@@ entry.c: static void mark_colliding_entries(const struct checkout *state,
{
static struct strbuf path = STRBUF_INIT;
struct stat st;
-- struct conv_attrs ca;
+- struct conv_attrs ca_buf, *ca = NULL;
+ struct conv_attrs ca_buf;
if (ce->ce_flags & CE_WT_REMOVE) {
@@ entry.c: int checkout_entry(struct cache_entry *ce, const struct checkout *state
if (topath) {
- if (S_ISREG(ce->ce_mode)) {
-- convert_attrs(state->istate, &ca, ce->name);
-- return write_entry(ce, topath, &ca, state, 1);
+ if (S_ISREG(ce->ce_mode) && !ca) {
-+ convert_attrs(state->istate, &ca_buf, ce->name);
-+ ca = &ca_buf;
+ convert_attrs(state->istate, &ca_buf, ce->name);
+ ca = &ca_buf;
}
-- return write_entry(ce, topath, NULL, state, 1);
-+ return write_entry(ce, topath, ca, state, 1);
- }
-
- strbuf_reset(&path);
@@ entry.c: int checkout_entry(struct cache_entry *ce, const struct checkout *state,
if (nr_checkouts)
(*nr_checkouts)++;
- if (S_ISREG(ce->ce_mode)) {
-- convert_attrs(state->istate, &ca, ce->name);
-- return write_entry(ce, path.buf, &ca, state, 0);
+ if (S_ISREG(ce->ce_mode) && !ca) {
-+ convert_attrs(state->istate, &ca_buf, ce->name);
-+ ca = &ca_buf;
+ convert_attrs(state->istate, &ca_buf, ce->name);
+ ca = &ca_buf;
}
- return write_entry(ce, path.buf, NULL, state, 0);
@@ entry.h: struct checkout {
+int checkout_entry_ca(struct cache_entry *ce, struct conv_attrs *ca,
+ const struct checkout *state, char *topath,
+ int *nr_checkouts);
-+
+
void enable_delayed_checkout(struct checkout *state);
int finish_delayed_checkout(struct checkout *state, int *nr_checkouts);
- /*
10: 2bdc13664e ! 10: bc8447cd9c unpack-trees: add basic support for parallel checkout
@@ parallel-checkout.c (new)
+ size_t nr, alloc;
+};
+
-+static struct parallel_checkout parallel_checkout = { 0 };
++static struct parallel_checkout parallel_checkout;
+
+enum pc_status parallel_checkout_status(void)
+{
@@ parallel-checkout.c (new)
+ * stat() data, so that they can be found by mark_colliding_entries(),
+ * in the next loop, when necessary.
+ */
-+ for (i = 0; i < parallel_checkout.nr; ++i) {
++ for (i = 0; i < parallel_checkout.nr; i++) {
+ struct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];
+ if (pc_item->status == PC_ITEM_WRITTEN)
+ update_ce_after_write(state, pc_item->ce, &pc_item->st);
+ }
+
-+ for (i = 0; i < parallel_checkout.nr; ++i) {
++ for (i = 0; i < parallel_checkout.nr; i++) {
+ struct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];
+
+ switch(pc_item->status) {
@@ parallel-checkout.c (new)
+ return ret;
+}
+
-+static int check_leading_dirs(const char *path, int len, int prefix_len)
-+{
-+ const char *slash = path + len;
-+
-+ while (slash > path && *slash != '/')
-+ slash--;
-+
-+ return has_dirs_only_path(path, slash - path, prefix_len);
-+}
-+
+static void write_pc_item(struct parallel_checkout_item *pc_item,
+ struct checkout *state)
+{
+ unsigned int mode = (pc_item->ce->ce_mode & 0100) ? 0777 : 0666;
+ int fd = -1, fstat_done = 0;
+ struct strbuf path = STRBUF_INIT;
++ const char *dir_sep;
+
+ strbuf_add(&path, state->base_dir, state->base_dir_len);
+ strbuf_add(&path, pc_item->ce->name, pc_item->ce->ce_namelen);
+
++ dir_sep = find_last_dir_sep(path.buf);
++
+ /*
-+ * At this point, leading dirs should have already been created. But if
-+ * a symlink being checked out has collided with one of the dirs, due to
-+ * file system folding rules, it's possible that the dirs are no longer
-+ * present. So we have to check again, and report any path collisions.
++ * The leading dirs should have been already created by now. But, in
++ * case of path collisions, one of the dirs could have been replaced by
++ * a symlink (checked out after we enqueued this entry for parallel
++ * checkout). Thus, we must check the leading dirs again.
+ */
-+ if (!check_leading_dirs(path.buf, path.len, state->base_dir_len)) {
++ if (dir_sep && !has_dirs_only_path(path.buf, dir_sep - path.buf,
++ state->base_dir_len)) {
+ pc_item->status = PC_ITEM_COLLIDED;
+ goto out;
+ }
@@ parallel-checkout.c (new)
+ * Errors which probably represent a path collision.
+ * Suppress the error message and mark the item to be
+ * retried later, sequentially. ENOTDIR and ENOENT are
-+ * also interesting, but check_leading_dirs() should
-+ * have already caught these cases.
++ * also interesting, but the above has_dirs_only_path()
++ * call should have already caught these cases.
+ */
+ pc_item->status = PC_ITEM_COLLIDED;
+ } else {
@@ parallel-checkout.c (new)
+{
+ size_t i;
+
-+ for (i = 0; i < parallel_checkout.nr; ++i)
++ for (i = 0; i < parallel_checkout.nr; i++)
+ write_pc_item(¶llel_checkout.items[i], state);
+}
+
11: 096e543fd2 ! 11: 815137685a parallel-checkout: make it truly parallel
@@ builtin/checkout--helper.c (new)
+ packet_to_pc_item(line, len, &items[nr++]);
+ }
+
-+ for (i = 0; i < nr; ++i) {
++ for (i = 0; i < nr; i++) {
+ struct parallel_checkout_item *pc_item = &items[i];
+ write_pc_item(pc_item, state);
+ report_result(pc_item);
@@ parallel-checkout.c: static int write_pc_item_to_fd(struct parallel_checkout_ite
*/
ret = convert_to_working_tree_ca(&pc_item->ca, pc_item->ce->name,
new_blob, size, &buf, NULL);
-@@ parallel-checkout.c: static int check_leading_dirs(const char *path, int len, int prefix_len)
- return has_dirs_only_path(path, slash - path, prefix_len);
+@@ parallel-checkout.c: static int close_and_clear(int *fd)
+ return ret;
}
-static void write_pc_item(struct parallel_checkout_item *pc_item,
@@ parallel-checkout.c: static void write_pc_item(struct parallel_checkout_item *pc
+static void send_batch(int fd, size_t start, size_t nr)
+{
+ size_t i;
-+ for (i = 0; i < nr; ++i)
++ for (i = 0; i < nr; i++)
+ send_one_item(fd, ¶llel_checkout.items[start + i]);
+ packet_flush(fd);
+}
@@ parallel-checkout.c: static void write_pc_item(struct parallel_checkout_item *pc
+
+ ALLOC_ARRAY(workers, num_workers);
+
-+ for (i = 0; i < num_workers; ++i) {
++ for (i = 0; i < num_workers; i++) {
+ struct child_process *cp = &workers[i].cp;
+
+ child_process_init(cp);
@@ parallel-checkout.c: static void write_pc_item(struct parallel_checkout_item *pc
+ base_batch_size = parallel_checkout.nr / num_workers;
+ workers_with_one_extra_item = parallel_checkout.nr % num_workers;
+
-+ for (i = 0; i < num_workers; ++i) {
++ for (i = 0; i < num_workers; i++) {
+ struct pc_worker *worker = &workers[i];
+ size_t batch_size = base_batch_size;
+
@@ parallel-checkout.c: static void write_pc_item(struct parallel_checkout_item *pc
+ * Close pipes before calling finish_command() to let the workers
+ * exit asynchronously and avoid spending extra time on wait().
+ */
-+ for (i = 0; i < num_workers; ++i) {
++ for (i = 0; i < num_workers; i++) {
+ struct child_process *cp = &workers[i].cp;
+ if (cp->in >= 0)
+ close(cp->in);
@@ parallel-checkout.c: static void write_pc_item(struct parallel_checkout_item *pc
+ close(cp->out);
+ }
+
-+ for (i = 0; i < num_workers; ++i) {
++ for (i = 0; i < num_workers; i++) {
+ if (finish_command(&workers[i].cp))
+ error(_("checkout worker %d finished with error"), i);
+ }
@@ parallel-checkout.c: static void write_pc_item(struct parallel_checkout_item *pc
+ struct pollfd *pfds;
+
+ CALLOC_ARRAY(pfds, num_workers);
-+ for (i = 0; i < num_workers; ++i) {
++ for (i = 0; i < num_workers; i++) {
+ pfds[i].fd = workers[i].cp.out;
+ pfds[i].events = POLLIN;
+ }
@@ parallel-checkout.c: static void write_pc_item(struct parallel_checkout_item *pc
+ die_errno("failed to poll checkout workers");
+ }
+
-+ for (i = 0; i < num_workers && nr > 0; ++i) {
++ for (i = 0; i < num_workers && nr > 0; i++) {
+ struct pc_worker *worker = &workers[i];
+ struct pollfd *pfd = &pfds[i];
+
@@ parallel-checkout.h: void init_parallel_checkout(void);
+ size_t id;
+ struct object_id oid;
+ unsigned int ce_mode;
-+ enum crlf_action crlf_action;
++ enum convert_crlf_action crlf_action;
+ int ident;
+ size_t working_tree_encoding_len;
+ size_t name_len;
12: 9cfeb4821c ! 12: 2b42621582 parallel-checkout: support progress displaying
@@ parallel-checkout.c: struct parallel_checkout {
+ unsigned int *progress_cnt;
};
- static struct parallel_checkout parallel_checkout = { 0 };
+ static struct parallel_checkout parallel_checkout;
@@ parallel-checkout.c: int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)
return 0;
}
@@ parallel-checkout.c: static void write_items_sequentially(struct checkout *state
{
size_t i;
-- for (i = 0; i < parallel_checkout.nr; ++i)
+- for (i = 0; i < parallel_checkout.nr; i++)
- write_pc_item(¶llel_checkout.items[i], state);
-+ for (i = 0; i < parallel_checkout.nr; ++i) {
++ for (i = 0; i < parallel_checkout.nr; i++) {
+ struct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];
+ write_pc_item(pc_item, state);
+ if (pc_item->status != PC_ITEM_COLLIDED)
13: da99b671e6 = 13: 960116579a make_transient_cache_entry(): optionally alloc from mem_pool
14: d3d561754a = 14: fb9f2f580c builtin/checkout.c: complete parallel checkout support
15: ee34c6e149 = 15: a844451e58 checkout-index: add parallel checkout support
16: 05299a3cc0 = 16: 3733857ffa parallel-checkout: add tests for basic operations
17: 3d140dcacb = 17: c8a2974f81 parallel-checkout: add tests related to clone collisions
18: b26f676cae = 18: 86fccd57d5 parallel-checkout: add tests related to .gitattributes
19: 641c61f9b6 = 19: 7f3e23cc38 ci: run test round with parallel-checkout enabled
--
2.28.0
next prev parent reply other threads:[~2020-11-04 20:33 UTC|newest]
Thread overview: 154+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-08-10 21:33 [RFC PATCH 00/21] [RFC] Parallel checkout Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 01/21] convert: make convert_attrs() and convert structs public Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 02/21] convert: add [async_]convert_to_working_tree_ca() variants Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 03/21] convert: add get_stream_filter_ca() variant Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 04/21] convert: add conv_attrs classification Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 05/21] entry: extract a header file for entry.c functions Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 06/21] entry: make fstat_output() and read_blob_entry() public Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 07/21] entry: extract cache_entry update from write_entry() Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 08/21] entry: move conv_attrs lookup up to checkout_entry() Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 09/21] entry: add checkout_entry_ca() which takes preloaded conv_attrs Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 10/21] unpack-trees: add basic support for parallel checkout Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 11/21] parallel-checkout: make it truly parallel Matheus Tavares
2020-08-19 21:34 ` Jeff Hostetler
2020-08-20 1:33 ` Matheus Tavares Bernardino
2020-08-20 14:39 ` Jeff Hostetler
2020-08-10 21:33 ` [RFC PATCH 12/21] parallel-checkout: add configuration options Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 13/21] parallel-checkout: support progress displaying Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 14/21] make_transient_cache_entry(): optionally alloc from mem_pool Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 15/21] builtin/checkout.c: complete parallel checkout support Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 16/21] checkout-index: add " Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 17/21] parallel-checkout: avoid stat() calls in workers Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 18/21] entry: use is_dir_sep() when checking leading dirs Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 19/21] symlinks: make has_dirs_only_path() track FL_NOENT Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 20/21] parallel-checkout: create leading dirs in workers Matheus Tavares
2020-08-10 21:33 ` [RFC PATCH 21/21] parallel-checkout: skip checking the working tree on clone Matheus Tavares
2020-08-12 16:57 ` [RFC PATCH 00/21] [RFC] Parallel checkout Jeff Hostetler
2020-09-22 22:49 ` [PATCH v2 00/19] Parallel Checkout (part I) Matheus Tavares
2020-09-22 22:49 ` [PATCH v2 01/19] convert: make convert_attrs() and convert structs public Matheus Tavares
2020-09-22 22:49 ` [PATCH v2 02/19] convert: add [async_]convert_to_working_tree_ca() variants Matheus Tavares
2020-09-22 22:49 ` [PATCH v2 03/19] convert: add get_stream_filter_ca() variant Matheus Tavares
2020-09-22 22:49 ` [PATCH v2 04/19] convert: add conv_attrs classification Matheus Tavares
2020-09-22 22:49 ` [PATCH v2 05/19] entry: extract a header file for entry.c functions Matheus Tavares
2020-09-22 22:49 ` [PATCH v2 06/19] entry: make fstat_output() and read_blob_entry() public Matheus Tavares
2020-09-22 22:49 ` [PATCH v2 07/19] entry: extract cache_entry update from write_entry() Matheus Tavares
2020-09-22 22:49 ` [PATCH v2 08/19] entry: move conv_attrs lookup up to checkout_entry() Matheus Tavares
2020-10-01 15:53 ` Jeff Hostetler
2020-10-01 15:59 ` Jeff Hostetler
2020-09-22 22:49 ` [PATCH v2 09/19] entry: add checkout_entry_ca() which takes preloaded conv_attrs Matheus Tavares
2020-09-22 22:49 ` [PATCH v2 10/19] unpack-trees: add basic support for parallel checkout Matheus Tavares
2020-10-05 6:17 ` [PATCH] parallel-checkout: drop unused checkout state parameter Jeff King
2020-10-05 13:13 ` Matheus Tavares Bernardino
2020-10-05 13:45 ` Jeff King
2020-09-22 22:49 ` [PATCH v2 11/19] parallel-checkout: make it truly parallel Matheus Tavares
2020-09-29 19:52 ` Martin Ågren
2020-09-30 14:02 ` Matheus Tavares Bernardino
2020-09-22 22:49 ` [PATCH v2 12/19] parallel-checkout: support progress displaying Matheus Tavares
2020-09-22 22:49 ` [PATCH v2 13/19] make_transient_cache_entry(): optionally alloc from mem_pool Matheus Tavares
2020-09-22 22:49 ` [PATCH v2 14/19] builtin/checkout.c: complete parallel checkout support Matheus Tavares
2020-09-22 22:49 ` [PATCH v2 15/19] checkout-index: add " Matheus Tavares
2020-09-22 22:49 ` [PATCH v2 16/19] parallel-checkout: add tests for basic operations Matheus Tavares
2020-10-20 1:35 ` Jonathan Nieder
2020-10-20 2:55 ` Taylor Blau
2020-10-20 13:18 ` Matheus Tavares Bernardino
2020-10-20 19:09 ` Junio C Hamano
2020-10-20 3:18 ` Matheus Tavares Bernardino
2020-10-20 4:16 ` Jonathan Nieder
2020-10-20 19:14 ` Junio C Hamano
2020-09-22 22:49 ` [PATCH v2 17/19] parallel-checkout: add tests related to clone collisions Matheus Tavares
2020-09-22 22:49 ` [PATCH v2 18/19] parallel-checkout: add tests related to .gitattributes Matheus Tavares
2020-09-22 22:49 ` [PATCH v2 19/19] ci: run test round with parallel-checkout enabled Matheus Tavares
2020-10-29 2:14 ` [PATCH v3 00/19] Parallel Checkout (part I) Matheus Tavares
2020-10-29 2:14 ` [PATCH v3 01/19] convert: make convert_attrs() and convert structs public Matheus Tavares
2020-10-29 23:40 ` Junio C Hamano
2020-10-30 17:01 ` Matheus Tavares Bernardino
2020-10-30 17:38 ` Junio C Hamano
2020-10-29 2:14 ` [PATCH v3 02/19] convert: add [async_]convert_to_working_tree_ca() variants Matheus Tavares
2020-10-29 23:48 ` Junio C Hamano
2020-10-29 2:14 ` [PATCH v3 03/19] convert: add get_stream_filter_ca() variant Matheus Tavares
2020-10-29 23:51 ` Junio C Hamano
2020-10-29 2:14 ` [PATCH v3 04/19] convert: add conv_attrs classification Matheus Tavares
2020-10-29 23:53 ` Junio C Hamano
2020-10-29 2:14 ` [PATCH v3 05/19] entry: extract a header file for entry.c functions Matheus Tavares
2020-10-30 21:36 ` Junio C Hamano
2020-10-29 2:14 ` [PATCH v3 06/19] entry: make fstat_output() and read_blob_entry() public Matheus Tavares
2020-10-29 2:14 ` [PATCH v3 07/19] entry: extract cache_entry update from write_entry() Matheus Tavares
2020-10-29 2:14 ` [PATCH v3 08/19] entry: move conv_attrs lookup up to checkout_entry() Matheus Tavares
2020-10-30 21:58 ` Junio C Hamano
2020-10-29 2:14 ` [PATCH v3 09/19] entry: add checkout_entry_ca() which takes preloaded conv_attrs Matheus Tavares
2020-10-30 22:02 ` Junio C Hamano
2020-10-29 2:14 ` [PATCH v3 10/19] unpack-trees: add basic support for parallel checkout Matheus Tavares
2020-11-02 19:35 ` Junio C Hamano
2020-11-03 3:48 ` Matheus Tavares Bernardino
2020-10-29 2:14 ` [PATCH v3 11/19] parallel-checkout: make it truly parallel Matheus Tavares
2020-10-29 2:14 ` [PATCH v3 12/19] parallel-checkout: support progress displaying Matheus Tavares
2020-10-29 2:14 ` [PATCH v3 13/19] make_transient_cache_entry(): optionally alloc from mem_pool Matheus Tavares
2020-10-29 2:14 ` [PATCH v3 14/19] builtin/checkout.c: complete parallel checkout support Matheus Tavares
2020-10-29 2:14 ` [PATCH v3 15/19] checkout-index: add " Matheus Tavares
2020-10-29 2:14 ` [PATCH v3 16/19] parallel-checkout: add tests for basic operations Matheus Tavares
2020-10-29 2:14 ` [PATCH v3 17/19] parallel-checkout: add tests related to clone collisions Matheus Tavares
2020-10-29 2:14 ` [PATCH v3 18/19] parallel-checkout: add tests related to .gitattributes Matheus Tavares
2020-10-29 2:14 ` [PATCH v3 19/19] ci: run test round with parallel-checkout enabled Matheus Tavares
2020-10-29 19:48 ` [PATCH v3 00/19] Parallel Checkout (part I) Junio C Hamano
2020-10-30 15:58 ` Jeff Hostetler
2020-11-04 20:32 ` Matheus Tavares [this message]
2020-11-04 20:33 ` [PATCH v4 01/19] convert: make convert_attrs() and convert structs public Matheus Tavares
2020-12-05 10:40 ` Christian Couder
2020-12-05 21:53 ` Matheus Tavares Bernardino
2020-11-04 20:33 ` [PATCH v4 02/19] convert: add [async_]convert_to_working_tree_ca() variants Matheus Tavares
2020-12-05 11:10 ` Christian Couder
2020-12-05 22:20 ` Matheus Tavares Bernardino
2020-11-04 20:33 ` [PATCH v4 03/19] convert: add get_stream_filter_ca() variant Matheus Tavares
2020-12-05 11:45 ` Christian Couder
2020-11-04 20:33 ` [PATCH v4 04/19] convert: add conv_attrs classification Matheus Tavares
2020-12-05 12:07 ` Christian Couder
2020-12-05 22:08 ` Matheus Tavares Bernardino
2020-11-04 20:33 ` [PATCH v4 05/19] entry: extract a header file for entry.c functions Matheus Tavares
2020-12-06 8:31 ` Christian Couder
2020-11-04 20:33 ` [PATCH v4 06/19] entry: make fstat_output() and read_blob_entry() public Matheus Tavares
2020-11-04 20:33 ` [PATCH v4 07/19] entry: extract cache_entry update from write_entry() Matheus Tavares
2020-12-06 8:53 ` Christian Couder
2020-11-04 20:33 ` [PATCH v4 08/19] entry: move conv_attrs lookup up to checkout_entry() Matheus Tavares
2020-12-06 9:35 ` Christian Couder
2020-12-07 13:52 ` Matheus Tavares Bernardino
2020-11-04 20:33 ` [PATCH v4 09/19] entry: add checkout_entry_ca() which takes preloaded conv_attrs Matheus Tavares
2020-12-06 10:02 ` Christian Couder
2020-12-07 16:47 ` Matheus Tavares Bernardino
2020-11-04 20:33 ` [PATCH v4 10/19] unpack-trees: add basic support for parallel checkout Matheus Tavares
2020-12-06 11:36 ` Christian Couder
2020-12-07 19:06 ` Matheus Tavares Bernardino
2020-11-04 20:33 ` [PATCH v4 11/19] parallel-checkout: make it truly parallel Matheus Tavares
2020-12-16 22:31 ` Emily Shaffer
2020-12-17 15:00 ` Matheus Tavares Bernardino
2020-11-04 20:33 ` [PATCH v4 12/19] parallel-checkout: support progress displaying Matheus Tavares
2020-11-04 20:33 ` [PATCH v4 13/19] make_transient_cache_entry(): optionally alloc from mem_pool Matheus Tavares
2020-11-04 20:33 ` [PATCH v4 14/19] builtin/checkout.c: complete parallel checkout support Matheus Tavares
2020-11-04 20:33 ` [PATCH v4 15/19] checkout-index: add " Matheus Tavares
2020-11-04 20:33 ` [PATCH v4 16/19] parallel-checkout: add tests for basic operations Matheus Tavares
2020-11-04 20:33 ` [PATCH v4 17/19] parallel-checkout: add tests related to clone collisions Matheus Tavares
2020-11-04 20:33 ` [PATCH v4 18/19] parallel-checkout: add tests related to .gitattributes Matheus Tavares
2020-11-04 20:33 ` [PATCH v4 19/19] ci: run test round with parallel-checkout enabled Matheus Tavares
2020-12-16 14:50 ` [PATCH v5 0/9] Parallel Checkout (part I) Matheus Tavares
2020-12-16 14:50 ` [PATCH v5 1/9] convert: make convert_attrs() and convert structs public Matheus Tavares
2020-12-16 14:50 ` [PATCH v5 2/9] convert: add [async_]convert_to_working_tree_ca() variants Matheus Tavares
2020-12-16 14:50 ` [PATCH v5 3/9] convert: add get_stream_filter_ca() variant Matheus Tavares
2020-12-16 14:50 ` [PATCH v5 4/9] convert: add classification for conv_attrs struct Matheus Tavares
2020-12-16 14:50 ` [PATCH v5 5/9] entry: extract a header file for entry.c functions Matheus Tavares
2020-12-16 14:50 ` [PATCH v5 6/9] entry: make fstat_output() and read_blob_entry() public Matheus Tavares
2020-12-16 14:50 ` [PATCH v5 7/9] entry: extract update_ce_after_write() from write_entry() Matheus Tavares
2020-12-16 14:50 ` [PATCH v5 8/9] entry: move conv_attrs lookup up to checkout_entry() Matheus Tavares
2020-12-16 14:50 ` [PATCH v5 9/9] entry: add checkout_entry_ca() taking preloaded conv_attrs Matheus Tavares
2020-12-16 15:27 ` [PATCH v5 0/9] Parallel Checkout (part I) Christian Couder
2020-12-17 1:11 ` Junio C Hamano
2021-03-23 14:19 ` [PATCH v6 0/9] Parallel Checkout (part 1) Matheus Tavares
2021-03-23 14:19 ` [PATCH v6 1/9] convert: make convert_attrs() and convert structs public Matheus Tavares
2021-03-23 14:19 ` [PATCH v6 2/9] convert: add [async_]convert_to_working_tree_ca() variants Matheus Tavares
2021-03-23 14:19 ` [PATCH v6 3/9] convert: add get_stream_filter_ca() variant Matheus Tavares
2021-03-23 14:19 ` [PATCH v6 4/9] convert: add classification for conv_attrs struct Matheus Tavares
2021-03-23 14:19 ` [PATCH v6 5/9] entry: extract a header file for entry.c functions Matheus Tavares
2021-03-23 14:19 ` [PATCH v6 6/9] entry: make fstat_output() and read_blob_entry() public Matheus Tavares
2021-03-23 14:19 ` [PATCH v6 7/9] entry: extract update_ce_after_write() from write_entry() Matheus Tavares
2021-03-23 14:19 ` [PATCH v6 8/9] entry: move conv_attrs lookup up to checkout_entry() Matheus Tavares
2021-03-23 14:19 ` [PATCH v6 9/9] entry: add checkout_entry_ca() taking preloaded conv_attrs Matheus Tavares
2021-03-23 17:34 ` [PATCH v6 0/9] Parallel Checkout (part 1) Junio C Hamano
2020-10-01 16:42 ` [RFC PATCH 00/21] [RFC] Parallel checkout Jeff Hostetler
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1604521275.git.matheus.bernardino@usp.br \
--to=matheus.bernardino@usp.br \
--cc=chriscool@tuxfamily.org \
--cc=git@jeffhostetler.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=jrnieder@gmail.com \
--cc=martin.agren@gmail.com \
--cc=newren@gmail.com \
--cc=peff@peff.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).