* [PATCH v2 1/4] pack-bitmap.c: check preferred pack validity when opening MIDX bitmap
2022-05-24 18:54 ` [PATCH v2 0/4] pack-objects: fix a pair of MIDX bitmap-related races Taylor Blau
@ 2022-05-24 18:54 ` Taylor Blau
2022-05-24 19:36 ` Ævar Arnfjörð Bjarmason
2022-05-24 18:54 ` [PATCH v2 2/4] builtin/pack-objects.c: avoid redundant NULL check Taylor Blau
` (3 subsequent siblings)
4 siblings, 1 reply; 29+ messages in thread
From: Taylor Blau @ 2022-05-24 18:54 UTC (permalink / raw)
To: git; +Cc: vdye, jonathantanmy, gitster
When pack-objects adds an entry to its packing list, it marks the
packfile and offset containing the object, which we may later use during
verbatim reuse (c.f., `write_reused_pack_verbatim()`).
If the packfile in question is deleted in the background (e.g., due to a
concurrent `git repack`), we'll die() as a result of calling use_pack(),
unless we have an open file descriptor on the pack itself. 4c08018204
(pack-objects: protect against disappearing packs, 2011-10-14) worked
around this by opening the pack ahead of time before recording it as a
valid source for reuse.
4c08018204's treatment meant that we could tolerate disappearing packs,
since it ensures we always have an open file descriptor on any pack that
we mark as a valid source for reuse. This tightens the race to only
happen when we need to close an open pack's file descriptor (c.f., the
caller of `packfile.c::get_max_fd_limit()`) _and_ that pack was deleted,
in which case we'll complain that a pack could not be accessed and
die().
The pack bitmap code does this, too, since prior to dc1daacdcc
(pack-bitmap: check pack validity when opening bitmap, 2021-07-23) it
was vulnerable to the same race.
The MIDX bitmap code does not do this, and is vulnerable to the same
race. Apply the same treatment as dc1daacdcc to the routine responsible
for opening the multi-pack bitmap's preferred pack to close this race.
This patch handles the "preferred" pack (c.f., the section
"multi-pack-index reverse indexes" in
Documentation/technical/pack-format.txt) specially, since pack-objects
depends on reusing exact chunks of that pack verbatim in
reuse_partial_packfile_from_bitmap(). So if that pack cannot be loaded,
the utility of a bitmap is significantly diminished.
Similar to dc1daacdcc, we could technically just add this check in
reuse_partial_packfile_from_bitmap(), since it's possible to use a MIDX
.bitmap without needing to open any of its packs. But it's simpler to do
the check as early as possible, covering all direct uses of the
preferred pack. Note that doing this check early requires us to call
prepare_midx_pack() early, too, so move the relevant part of that loop
from load_reverse_index() into open_midx_bitmap_1().
Subsequent patches handle the non-preferred packs in a slightly
different fashion.
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
pack-bitmap.c | 18 ++++++++++++++++--
1 file changed, 16 insertions(+), 2 deletions(-)
diff --git a/pack-bitmap.c b/pack-bitmap.c
index 97909d48da..d607918407 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -315,6 +315,8 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,
struct stat st;
char *idx_name = midx_bitmap_filename(midx);
int fd = git_open(idx_name);
+ uint32_t i;
+ struct packed_git *preferred;
free(idx_name);
@@ -353,6 +355,20 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,
warning(_("multi-pack bitmap is missing required reverse index"));
goto cleanup;
}
+
+ for (i = 0; i < bitmap_git->midx->num_packs; i++) {
+ if (prepare_midx_pack(the_repository, bitmap_git->midx, i))
+ die(_("could not open pack %s"),
+ bitmap_git->midx->pack_names[i]);
+ }
+
+ preferred = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];
+ if (!is_pack_valid(preferred)) {
+ warning(_("preferred pack (%s) is invalid"),
+ preferred->pack_name);
+ goto cleanup;
+ }
+
return 0;
cleanup:
@@ -429,8 +445,6 @@ static int load_reverse_index(struct bitmap_index *bitmap_git)
* since we will need to make use of them in pack-objects.
*/
for (i = 0; i < bitmap_git->midx->num_packs; i++) {
- if (prepare_midx_pack(the_repository, bitmap_git->midx, i))
- die(_("load_reverse_index: could not open pack"));
ret = load_pack_revindex(bitmap_git->midx->packs[i]);
if (ret)
return ret;
--
2.36.1.94.gb0d54bedca
^ permalink raw reply related [flat|nested] 29+ messages in thread
* Re: [PATCH v2 1/4] pack-bitmap.c: check preferred pack validity when opening MIDX bitmap
2022-05-24 18:54 ` [PATCH v2 1/4] pack-bitmap.c: check preferred pack validity when opening MIDX bitmap Taylor Blau
@ 2022-05-24 19:36 ` Ævar Arnfjörð Bjarmason
2022-05-24 21:38 ` Taylor Blau
0 siblings, 1 reply; 29+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-05-24 19:36 UTC (permalink / raw)
To: Taylor Blau; +Cc: git, vdye, jonathantanmy, gitster
On Tue, May 24 2022, Taylor Blau wrote:
Just nits on the error reporting:
> @@ -353,6 +355,20 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,
> warning(_("multi-pack bitmap is missing required reverse index"));
> goto cleanup;
> }
> +
> + for (i = 0; i < bitmap_git->midx->num_packs; i++) {
> + if (prepare_midx_pack(the_repository, bitmap_git->midx, i))
> + die(_("could not open pack %s"),
> + bitmap_git->midx->pack_names[i]);
Some existing API users of this & their error handling suggest that this
message is wrong. I.e. it's not that we couldn't open it, but that we
could open it and there's something wrong with it. Or perhaps their
messages are misleading?
> + }
> +
> + preferred = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];
> + if (!is_pack_valid(preferred)) {
> + warning(_("preferred pack (%s) is invalid"),
> + preferred->pack_name);
Likewise this? E.g. perhaps the permissions are just wrong or whatever,
per open_packed_git_1().
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 1/4] pack-bitmap.c: check preferred pack validity when opening MIDX bitmap
2022-05-24 19:36 ` Ævar Arnfjörð Bjarmason
@ 2022-05-24 21:38 ` Taylor Blau
2022-05-24 21:51 ` Ævar Arnfjörð Bjarmason
0 siblings, 1 reply; 29+ messages in thread
From: Taylor Blau @ 2022-05-24 21:38 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason; +Cc: git, vdye, jonathantanmy, gitster
On Tue, May 24, 2022 at 09:36:45PM +0200, Ævar Arnfjörð Bjarmason wrote:
>
> On Tue, May 24 2022, Taylor Blau wrote:
>
> Just nits on the error reporting:
>
> > @@ -353,6 +355,20 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,
> > warning(_("multi-pack bitmap is missing required reverse index"));
> > goto cleanup;
> > }
> > +
> > + for (i = 0; i < bitmap_git->midx->num_packs; i++) {
> > + if (prepare_midx_pack(the_repository, bitmap_git->midx, i))
> > + die(_("could not open pack %s"),
> > + bitmap_git->midx->pack_names[i]);
>
> Some existing API users of this & their error handling suggest that this
> message is wrong. I.e. it's not that we couldn't open it, but that we
> could open it and there's something wrong with it. Or perhaps their
> messages are misleading?
I tried to reuse some similar message based on "git grep 'if
(.*prepare_midx_pack'", so this was inspired by:
- the caller in midx.c::write_midx_internal(), whose error is "could
not load pack", and
- the caller in midx.c::verify_midx_file(), whose error is "failed to
load pack"
Are you suggesting we should s/open/load here and use the above error
message? My feeling at the time was that "load" was basically synonymous
with "open" given the context, but if you think they're different
enough, or have a different suggestion LMK.
Thanks,
Taylor
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 1/4] pack-bitmap.c: check preferred pack validity when opening MIDX bitmap
2022-05-24 21:38 ` Taylor Blau
@ 2022-05-24 21:51 ` Ævar Arnfjörð Bjarmason
0 siblings, 0 replies; 29+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-05-24 21:51 UTC (permalink / raw)
To: Taylor Blau; +Cc: git, vdye, jonathantanmy, gitster
On Tue, May 24 2022, Taylor Blau wrote:
> On Tue, May 24, 2022 at 09:36:45PM +0200, Ævar Arnfjörð Bjarmason wrote:
>>
>> On Tue, May 24 2022, Taylor Blau wrote:
>>
>> Just nits on the error reporting:
>>
>> > @@ -353,6 +355,20 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,
>> > warning(_("multi-pack bitmap is missing required reverse index"));
>> > goto cleanup;
>> > }
>> > +
>> > + for (i = 0; i < bitmap_git->midx->num_packs; i++) {
>> > + if (prepare_midx_pack(the_repository, bitmap_git->midx, i))
>> > + die(_("could not open pack %s"),
>> > + bitmap_git->midx->pack_names[i]);
>>
>> Some existing API users of this & their error handling suggest that this
>> message is wrong. I.e. it's not that we couldn't open it, but that we
>> could open it and there's something wrong with it. Or perhaps their
>> messages are misleading?
>
> I tried to reuse some similar message based on "git grep 'if
> (.*prepare_midx_pack'", so this was inspired by:
>
> - the caller in midx.c::write_midx_internal(), whose error is "could
> not load pack", and
> - the caller in midx.c::verify_midx_file(), whose error is "failed to
> load pack"
>
> Are you suggesting we should s/open/load here and use the above error
> message? My feeling at the time was that "load" was basically synonymous
> with "open" given the context, but if you think they're different
> enough, or have a different suggestion LMK.
Perhaps "parse" or something? Anyway with "could not open" I'd assume
open() failed, but in this case it looks like we could open it, but
(mostly?) failed later.
Maybe "could not load midx"? I don't know...
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v2 2/4] builtin/pack-objects.c: avoid redundant NULL check
2022-05-24 18:54 ` [PATCH v2 0/4] pack-objects: fix a pair of MIDX bitmap-related races Taylor Blau
2022-05-24 18:54 ` [PATCH v2 1/4] pack-bitmap.c: check preferred pack validity when opening MIDX bitmap Taylor Blau
@ 2022-05-24 18:54 ` Taylor Blau
2022-05-24 21:44 ` Junio C Hamano
2022-05-24 18:54 ` [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist Taylor Blau
` (2 subsequent siblings)
4 siblings, 1 reply; 29+ messages in thread
From: Taylor Blau @ 2022-05-24 18:54 UTC (permalink / raw)
To: git; +Cc: vdye, jonathantanmy, gitster
Before calling `for_each_object_in_pack()`, the caller
`read_packs_list_from_stdin()` loops through each of the `include_packs`
and checks that its `->util` pointer (which is used to store the `struct
packed_git *` itself) is non-NULL.
This check is redundant, because `read_packs_list_from_stdin()` already
checks that the included packs are non-NULL earlier on in the same
function (and it does not add any new entries in between).
Remove this check, since it is not doing anything in the meantime.
Co-authored-by: Victoria Dye <vdye@github.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
builtin/pack-objects.c | 2 --
1 file changed, 2 deletions(-)
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 014dcd4bc9..ec3193fd95 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -3369,8 +3369,6 @@ static void read_packs_list_from_stdin(void)
for_each_string_list_item(item, &include_packs) {
struct packed_git *p = item->util;
- if (!p)
- die(_("could not find pack '%s'"), item->string);
for_each_object_in_pack(p,
add_object_entry_from_pack,
&revs,
--
2.36.1.94.gb0d54bedca
^ permalink raw reply related [flat|nested] 29+ messages in thread
* Re: [PATCH v2 2/4] builtin/pack-objects.c: avoid redundant NULL check
2022-05-24 18:54 ` [PATCH v2 2/4] builtin/pack-objects.c: avoid redundant NULL check Taylor Blau
@ 2022-05-24 21:44 ` Junio C Hamano
2022-05-25 0:11 ` Taylor Blau
0 siblings, 1 reply; 29+ messages in thread
From: Junio C Hamano @ 2022-05-24 21:44 UTC (permalink / raw)
To: Taylor Blau; +Cc: git, vdye, jonathantanmy
Taylor Blau <me@ttaylorr.com> writes:
> Before calling `for_each_object_in_pack()`, the caller
> `read_packs_list_from_stdin()` loops through each of the `include_packs`
> and checks that its `->util` pointer (which is used to store the `struct
> packed_git *` itself) is non-NULL.
>
> This check is redundant, because `read_packs_list_from_stdin()` already
> checks that the included packs are non-NULL earlier on in the same
> function (and it does not add any new entries in between).
>
> Remove this check, since it is not doing anything in the meantime.
Will it start doing something soon in a later step?
Oh, did you mean that after the earlier for_each_string_list_item()
iteration over include_packs that died on an item with a NULL .util
member, the code did not do anything to cause this second iteration
over the same list to suddenly start seeing an item with NULL .util?
I am puzzled by the mention of "in the meantime".
The patch itself looks correct, of course.
Thanks.
> Co-authored-by: Victoria Dye <vdye@github.com>
> Signed-off-by: Taylor Blau <me@ttaylorr.com>
> ---
> builtin/pack-objects.c | 2 --
> 1 file changed, 2 deletions(-)
>
> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> index 014dcd4bc9..ec3193fd95 100644
> --- a/builtin/pack-objects.c
> +++ b/builtin/pack-objects.c
> @@ -3369,8 +3369,6 @@ static void read_packs_list_from_stdin(void)
>
> for_each_string_list_item(item, &include_packs) {
> struct packed_git *p = item->util;
> - if (!p)
> - die(_("could not find pack '%s'"), item->string);
> for_each_object_in_pack(p,
> add_object_entry_from_pack,
> &revs,
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 2/4] builtin/pack-objects.c: avoid redundant NULL check
2022-05-24 21:44 ` Junio C Hamano
@ 2022-05-25 0:11 ` Taylor Blau
0 siblings, 0 replies; 29+ messages in thread
From: Taylor Blau @ 2022-05-25 0:11 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Taylor Blau, git, vdye, jonathantanmy
On Tue, May 24, 2022 at 02:44:37PM -0700, Junio C Hamano wrote:
> Taylor Blau <me@ttaylorr.com> writes:
>
> > Before calling `for_each_object_in_pack()`, the caller
> > `read_packs_list_from_stdin()` loops through each of the `include_packs`
> > and checks that its `->util` pointer (which is used to store the `struct
> > packed_git *` itself) is non-NULL.
> >
> > This check is redundant, because `read_packs_list_from_stdin()` already
> > checks that the included packs are non-NULL earlier on in the same
> > function (and it does not add any new entries in between).
> >
> > Remove this check, since it is not doing anything in the meantime.
>
> Will it start doing something soon in a later step?
>
> Oh, did you mean that after the earlier for_each_string_list_item()
> iteration over include_packs that died on an item with a NULL .util
> member, the code did not do anything to cause this second iteration
> over the same list to suddenly start seeing an item with NULL .util?
>
> I am puzzled by the mention of "in the meantime".
The latter, sorry for the confusion.
> The patch itself looks correct, of course.
Thanks.
Taylor
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist
2022-05-24 18:54 ` [PATCH v2 0/4] pack-objects: fix a pair of MIDX bitmap-related races Taylor Blau
2022-05-24 18:54 ` [PATCH v2 1/4] pack-bitmap.c: check preferred pack validity when opening MIDX bitmap Taylor Blau
2022-05-24 18:54 ` [PATCH v2 2/4] builtin/pack-objects.c: avoid redundant NULL check Taylor Blau
@ 2022-05-24 18:54 ` Taylor Blau
2022-05-24 19:46 ` Ævar Arnfjörð Bjarmason
` (2 more replies)
2022-05-24 18:54 ` [PATCH v2 4/4] builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects Taylor Blau
2022-05-24 21:38 ` [PATCH v2 0/4] pack-objects: fix a pair of MIDX bitmap-related races Junio C Hamano
4 siblings, 3 replies; 29+ messages in thread
From: Taylor Blau @ 2022-05-24 18:54 UTC (permalink / raw)
To: git; +Cc: vdye, jonathantanmy, gitster
A subsequent patch will teach `want_object_in_pack()` to set its
`*found_pack` and `*found_offset` poitners to NULL when the provided
pack does not pass the `is_pack_valid()` check.
The `--stdin-packs` mode of `pack-objects` is not quite prepared to
handle this. To prepare it for this change, do the following two things:
- Ensure provided packs pass the `is_pack_valid()` check when
collecting the caller-provided packs into the "included" and
"excluded" lists.
- Gracefully handle any _invalid_ packs being passed to
`want_object_in_pack()`.
Calling `is_pack_valid()` early on makes it substantially less likely
that we will have to deal with a pack going away, since we'll have an
open file descriptor on its contents much earlier.
But even packs with open descriptors can become invalid in the future if
we (a) hit our open descriptor limit, forcing us to close some open
packs, and (b) one of those just-closed packs has gone away in the
meantime.
`add_object_entry_from_pack()` depends on having a non-NULL
`*found_pack`, since it passes that pointer to `packed_object_info()`,
meaning that we would SEGV if the pointer became NULL (like we propose
to do in `want_object_in_pack()` in the following patch).
But avoiding calling `packed_object_info()` entirely is OK, too, since
its only purpose is to identify which objects in the included packs are
commits, so that they can form the tips of the advisory traversal used
to discover the object namehashes.
Failing to do this means that at worst we will produce lower-quality
deltas, but it does not prevent us from generating the pack as long as
we can find a copy of each object from the disappearing pack in some
other part of the repository.
Co-authored-by: Victoria Dye <vdye@github.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
builtin/pack-objects.c | 35 ++++++++++++++++++++---------------
1 file changed, 20 insertions(+), 15 deletions(-)
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index ec3193fd95..ffeaecd1d8 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -3201,10 +3201,8 @@ static int add_object_entry_from_pack(const struct object_id *oid,
uint32_t pos,
void *_data)
{
- struct rev_info *revs = _data;
- struct object_info oi = OBJECT_INFO_INIT;
off_t ofs;
- enum object_type type;
+ enum object_type type = OBJ_NONE;
display_progress(progress_state, ++nr_seen);
@@ -3215,20 +3213,25 @@ static int add_object_entry_from_pack(const struct object_id *oid,
if (!want_object_in_pack(oid, 0, &p, &ofs))
return 0;
- oi.typep = &type;
- if (packed_object_info(the_repository, p, ofs, &oi) < 0)
- die(_("could not get type of object %s in pack %s"),
- oid_to_hex(oid), p->pack_name);
- else if (type == OBJ_COMMIT) {
- /*
- * commits in included packs are used as starting points for the
- * subsequent revision walk
- */
- add_pending_oid(revs, NULL, oid, 0);
+ if (p) {
+ struct rev_info *revs = _data;
+ struct object_info oi = OBJECT_INFO_INIT;
+
+ oi.typep = &type;
+ if (packed_object_info(the_repository, p, ofs, &oi) < 0) {
+ die(_("could not get type of object %s in pack %s"),
+ oid_to_hex(oid), p->pack_name);
+ } else if (type == OBJ_COMMIT) {
+ /*
+ * commits in included packs are used as starting points for the
+ * subsequent revision walk
+ */
+ add_pending_oid(revs, NULL, oid, 0);
+ }
+
+ stdin_packs_found_nr++;
}
- stdin_packs_found_nr++;
-
create_object_entry(oid, type, 0, 0, 0, p, ofs);
return 0;
@@ -3346,6 +3349,8 @@ static void read_packs_list_from_stdin(void)
struct packed_git *p = item->util;
if (!p)
die(_("could not find pack '%s'"), item->string);
+ if (!is_pack_valid(p))
+ die(_("packfile %s cannot be accessed"), p->pack_name);
}
/*
--
2.36.1.94.gb0d54bedca
^ permalink raw reply related [flat|nested] 29+ messages in thread
* Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist
2022-05-24 18:54 ` [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist Taylor Blau
@ 2022-05-24 19:46 ` Ævar Arnfjörð Bjarmason
2022-05-24 21:33 ` Taylor Blau
2022-05-24 22:03 ` Junio C Hamano
2022-05-26 19:21 ` Victoria Dye
2 siblings, 1 reply; 29+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-05-24 19:46 UTC (permalink / raw)
To: Taylor Blau; +Cc: git, vdye, jonathantanmy, gitster
On Tue, May 24 2022, Taylor Blau wrote:
> - struct rev_info *revs = _data;
> - struct object_info oi = OBJECT_INFO_INIT;
> off_t ofs;
> - enum object_type type;
> + enum object_type type = OBJ_NONE;
>
> display_progress(progress_state, ++nr_seen);
>
> @@ -3215,20 +3213,25 @@ static int add_object_entry_from_pack(const struct object_id *oid,
> if (!want_object_in_pack(oid, 0, &p, &ofs))
> return 0;
>
> - oi.typep = &type;
> - if (packed_object_info(the_repository, p, ofs, &oi) < 0)
> - die(_("could not get type of object %s in pack %s"),
> - oid_to_hex(oid), p->pack_name);
> - else if (type == OBJ_COMMIT) {
> - /*
> - * commits in included packs are used as starting points for the
> - * subsequent revision walk
> - */
> - add_pending_oid(revs, NULL, oid, 0);
> + if (p) {
> + struct rev_info *revs = _data;
> + struct object_info oi = OBJECT_INFO_INIT;
> +
> + oi.typep = &type;
> + if (packed_object_info(the_repository, p, ofs, &oi) < 0) {
> + die(_("could not get type of object %s in pack %s"),
> + oid_to_hex(oid), p->pack_name);
> + } else if (type == OBJ_COMMIT) {
> + /*
> + * commits in included packs are used as starting points for the
> + * subsequent revision walk
> + */
> + add_pending_oid(revs, NULL, oid, 0);
> + }
> +
> + stdin_packs_found_nr++;
> }
>
> - stdin_packs_found_nr++;
> -
> create_object_entry(oid, type, 0, 0, 0, p, ofs);
Not rhetorical, since I have no idea: Is the behavior change here to
make create_object_entry with type=OBJ_NONE desired? I.e. do we actually
want to create object entries for OBJ_NONE?
If that is the case I for one would find this a bit easier to follow
like this, even if it has some minor duplication, i.e. the intent is
clearer:
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index ffeaecd1d84..a447f6d5164 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -3202,7 +3202,6 @@ static int add_object_entry_from_pack(const struct object_id *oid,
void *_data)
{
off_t ofs;
- enum object_type type = OBJ_NONE;
display_progress(progress_state, ++nr_seen);
@@ -3216,6 +3215,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,
if (p) {
struct rev_info *revs = _data;
struct object_info oi = OBJECT_INFO_INIT;
+ enum object_type type;
oi.typep = &type;
if (packed_object_info(the_repository, p, ofs, &oi) < 0) {
@@ -3230,9 +3230,11 @@ static int add_object_entry_from_pack(const struct object_id *oid,
}
stdin_packs_found_nr++;
- }
- create_object_entry(oid, type, 0, 0, 0, p, ofs);
+ create_object_entry(oid, type, 0, 0, 0, p, ofs);
+ } else {
+ create_object_entry(oid, OBJ_NONE, 0, 0, 0, p, ofs);
+ }
return 0;
}
Or the same with adding "type = OBJ_NONE" to the "else" branch, leaving
the initial "type" uninitialized"?
Or perhaps this is a bug? I see some OBJ_NONE mentions in the code, but
do packfiles really have "none" objects in some fashion as far as
add_object_entry_from_pack() is concerned? (I'm not familiar enough with
this part of the codebase to know).
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist
2022-05-24 19:46 ` Ævar Arnfjörð Bjarmason
@ 2022-05-24 21:33 ` Taylor Blau
2022-05-24 21:49 ` Ævar Arnfjörð Bjarmason
0 siblings, 1 reply; 29+ messages in thread
From: Taylor Blau @ 2022-05-24 21:33 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason; +Cc: git, vdye, jonathantanmy, gitster
On Tue, May 24, 2022 at 09:46:09PM +0200, Ævar Arnfjörð Bjarmason wrote:
>
> On Tue, May 24 2022, Taylor Blau wrote:
>
> > - struct rev_info *revs = _data;
> > - struct object_info oi = OBJECT_INFO_INIT;
> > off_t ofs;
> > - enum object_type type;
> > + enum object_type type = OBJ_NONE;
> >
> > display_progress(progress_state, ++nr_seen);
> >
> > @@ -3215,20 +3213,25 @@ static int add_object_entry_from_pack(const struct object_id *oid,
> > if (!want_object_in_pack(oid, 0, &p, &ofs))
> > return 0;
> >
> > - oi.typep = &type;
> > - if (packed_object_info(the_repository, p, ofs, &oi) < 0)
> > - die(_("could not get type of object %s in pack %s"),
> > - oid_to_hex(oid), p->pack_name);
> > - else if (type == OBJ_COMMIT) {
> > - /*
> > - * commits in included packs are used as starting points for the
> > - * subsequent revision walk
> > - */
> > - add_pending_oid(revs, NULL, oid, 0);
> > + if (p) {
> > + struct rev_info *revs = _data;
> > + struct object_info oi = OBJECT_INFO_INIT;
> > +
> > + oi.typep = &type;
> > + if (packed_object_info(the_repository, p, ofs, &oi) < 0) {
> > + die(_("could not get type of object %s in pack %s"),
> > + oid_to_hex(oid), p->pack_name);
> > + } else if (type == OBJ_COMMIT) {
> > + /*
> > + * commits in included packs are used as starting points for the
> > + * subsequent revision walk
> > + */
> > + add_pending_oid(revs, NULL, oid, 0);
> > + }
> > +
> > + stdin_packs_found_nr++;
> > }
> >
> > - stdin_packs_found_nr++;
> > -
> > create_object_entry(oid, type, 0, 0, 0, p, ofs);
>
> Not rhetorical, since I have no idea: Is the behavior change here to
> make create_object_entry with type=OBJ_NONE desired? I.e. do we actually
> want to create object entries for OBJ_NONE?
This is intentional. OBJ_NONE tells create_object_entry() "we don't know
the type of this object yet", and then `check_object()` (which does the
bulk of the work in the "Counting objects" phase) goes through and fills
in any missing type information.
The caller in `builtin/pack-objects.c::read_object_list_from_stdin()` is
a good example of this (all of the objects created this way start out
with OBJ_NONE).
> If that is the case I for one would find this a bit easier to follow
> like this, even if it has some minor duplication, i.e. the intent is
> clearer:
>
> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> index ffeaecd1d84..a447f6d5164 100644
> --- a/builtin/pack-objects.c
> +++ b/builtin/pack-objects.c
> @@ -3202,7 +3202,6 @@ static int add_object_entry_from_pack(const struct object_id *oid,
> void *_data)
> {
> off_t ofs;
> - enum object_type type = OBJ_NONE;
>
> display_progress(progress_state, ++nr_seen);
>
> @@ -3216,6 +3215,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,
> if (p) {
> struct rev_info *revs = _data;
> struct object_info oi = OBJECT_INFO_INIT;
> + enum object_type type;
>
> oi.typep = &type;
> if (packed_object_info(the_repository, p, ofs, &oi) < 0) {
> @@ -3230,9 +3230,11 @@ static int add_object_entry_from_pack(const struct object_id *oid,
> }
>
> stdin_packs_found_nr++;
> - }
>
> - create_object_entry(oid, type, 0, 0, 0, p, ofs);
> + create_object_entry(oid, type, 0, 0, 0, p, ofs);
> + } else {
> + create_object_entry(oid, OBJ_NONE, 0, 0, 0, p, ofs);
> + }
>
> return 0;
> }
>
> Or the same with adding "type = OBJ_NONE" to the "else" branch, leaving
> the initial "type" uninitialized"?
I'd be fine with that (and don't really have a very strong opinion
either way). Let's see if anybody else has thoughts about it, and then
I'm happy to change it in a subsequent version.
Thanks,
Taylor
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist
2022-05-24 21:33 ` Taylor Blau
@ 2022-05-24 21:49 ` Ævar Arnfjörð Bjarmason
0 siblings, 0 replies; 29+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-05-24 21:49 UTC (permalink / raw)
To: Taylor Blau; +Cc: git, vdye, jonathantanmy, gitster
On Tue, May 24 2022, Taylor Blau wrote:
> On Tue, May 24, 2022 at 09:46:09PM +0200, Ævar Arnfjörð Bjarmason wrote:
>>
>> On Tue, May 24 2022, Taylor Blau wrote:
>>
>> > - struct rev_info *revs = _data;
>> > - struct object_info oi = OBJECT_INFO_INIT;
>> > off_t ofs;
>> > - enum object_type type;
>> > + enum object_type type = OBJ_NONE;
>> >
>> > display_progress(progress_state, ++nr_seen);
>> >
>> > @@ -3215,20 +3213,25 @@ static int add_object_entry_from_pack(const struct object_id *oid,
>> > if (!want_object_in_pack(oid, 0, &p, &ofs))
>> > return 0;
>> >
>> > - oi.typep = &type;
>> > - if (packed_object_info(the_repository, p, ofs, &oi) < 0)
>> > - die(_("could not get type of object %s in pack %s"),
>> > - oid_to_hex(oid), p->pack_name);
>> > - else if (type == OBJ_COMMIT) {
>> > - /*
>> > - * commits in included packs are used as starting points for the
>> > - * subsequent revision walk
>> > - */
>> > - add_pending_oid(revs, NULL, oid, 0);
>> > + if (p) {
>> > + struct rev_info *revs = _data;
>> > + struct object_info oi = OBJECT_INFO_INIT;
>> > +
>> > + oi.typep = &type;
>> > + if (packed_object_info(the_repository, p, ofs, &oi) < 0) {
>> > + die(_("could not get type of object %s in pack %s"),
>> > + oid_to_hex(oid), p->pack_name);
>> > + } else if (type == OBJ_COMMIT) {
>> > + /*
>> > + * commits in included packs are used as starting points for the
>> > + * subsequent revision walk
>> > + */
>> > + add_pending_oid(revs, NULL, oid, 0);
>> > + }
>> > +
>> > + stdin_packs_found_nr++;
>> > }
>> >
>> > - stdin_packs_found_nr++;
>> > -
>> > create_object_entry(oid, type, 0, 0, 0, p, ofs);
>>
>> Not rhetorical, since I have no idea: Is the behavior change here to
>> make create_object_entry with type=OBJ_NONE desired? I.e. do we actually
>> want to create object entries for OBJ_NONE?
>
> This is intentional. OBJ_NONE tells create_object_entry() "we don't know
> the type of this object yet", and then `check_object()` (which does the
> bulk of the work in the "Counting objects" phase) goes through and fills
> in any missing type information.
Ah, I didn't know that.
> The caller in `builtin/pack-objects.c::read_object_list_from_stdin()` is
> a good example of this (all of the objects created this way start out
> with OBJ_NONE).
>
>> If that is the case I for one would find this a bit easier to follow
>> like this, even if it has some minor duplication, i.e. the intent is
>> clearer:
>>
>> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
>> index ffeaecd1d84..a447f6d5164 100644
>> --- a/builtin/pack-objects.c
>> +++ b/builtin/pack-objects.c
>> @@ -3202,7 +3202,6 @@ static int add_object_entry_from_pack(const struct object_id *oid,
>> void *_data)
>> {
>> off_t ofs;
>> - enum object_type type = OBJ_NONE;
>>
>> display_progress(progress_state, ++nr_seen);
>>
>> @@ -3216,6 +3215,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,
>> if (p) {
>> struct rev_info *revs = _data;
>> struct object_info oi = OBJECT_INFO_INIT;
>> + enum object_type type;
>>
>> oi.typep = &type;
>> if (packed_object_info(the_repository, p, ofs, &oi) < 0) {
>> @@ -3230,9 +3230,11 @@ static int add_object_entry_from_pack(const struct object_id *oid,
>> }
>>
>> stdin_packs_found_nr++;
>> - }
>>
>> - create_object_entry(oid, type, 0, 0, 0, p, ofs);
>> + create_object_entry(oid, type, 0, 0, 0, p, ofs);
>> + } else {
>> + create_object_entry(oid, OBJ_NONE, 0, 0, 0, p, ofs);
>> + }
>>
>> return 0;
>> }
>>
>> Or the same with adding "type = OBJ_NONE" to the "else" branch, leaving
>> the initial "type" uninitialized"?
>
> I'd be fine with that (and don't really have a very strong opinion
> either way). Let's see if anybody else has thoughts about it, and then
> I'm happy to change it in a subsequent version.
FWIW I think you should place a particularly low value on my suggestion
of this.
I.e. the last thing we should do is probably to optimize the code to be
read by someone who hadn't spent even 10 minutes finding out such
obvious major code-flow details, i.e. me not knowing about how OBJ_NONE
was used in this case....
So it's probably all fine as-is, but p erhaps others will think it's
good or whatever...
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist
2022-05-24 18:54 ` [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist Taylor Blau
2022-05-24 19:46 ` Ævar Arnfjörð Bjarmason
@ 2022-05-24 22:03 ` Junio C Hamano
2022-05-25 0:14 ` Taylor Blau
2022-05-26 19:21 ` Victoria Dye
2 siblings, 1 reply; 29+ messages in thread
From: Junio C Hamano @ 2022-05-24 22:03 UTC (permalink / raw)
To: Taylor Blau; +Cc: git, vdye, jonathantanmy
Taylor Blau <me@ttaylorr.com> writes:
> Calling `is_pack_valid()` early on makes it substantially less likely
> that we will have to deal with a pack going away, since we'll have an
> open file descriptor on its contents much earlier.
Sorry for asking a stupid question (or two), but I am confused.
This does make sure that we can read and use the contents of the
packfile even when somebody else removes it from the disk by
ensuring that
(1) we have an open file descriptor to it, so that we could open
mmap window into it at will; or
(2) we have a mmap window that covers all of it (this should be the
norm on platforms with vast address space); or
(3) we are in the same state as (1) by opening the packfile to
validate the pack right now.
and during the pack-object we are running (aka "repack"), we can
continue to read from that pack that may have already disappeared
from the disk.
But is that sufficient? Are we writing the resulting new pack(s)
out in such a way that the repository is healthy without the pack
we noticed is disappearing? How do we ensure that?
Thanks.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist
2022-05-24 22:03 ` Junio C Hamano
@ 2022-05-25 0:14 ` Taylor Blau
0 siblings, 0 replies; 29+ messages in thread
From: Taylor Blau @ 2022-05-25 0:14 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, vdye, jonathantanmy
On Tue, May 24, 2022 at 03:03:11PM -0700, Junio C Hamano wrote:
> Taylor Blau <me@ttaylorr.com> writes:
>
> > Calling `is_pack_valid()` early on makes it substantially less likely
> > that we will have to deal with a pack going away, since we'll have an
> > open file descriptor on its contents much earlier.
>
> Sorry for asking a stupid question (or two), but I am confused.
No such thing as a stupid question, so your apology is not necessary in
the slightest :).
> This does make sure that we can read and use the contents of the
> packfile even when somebody else removes it from the disk by
> ensuring that
>
> (1) we have an open file descriptor to it, so that we could open
> mmap window into it at will; or
>
> (2) we have a mmap window that covers all of it (this should be the
> norm on platforms with vast address space); or
>
> (3) we are in the same state as (1) by opening the packfile to
> validate the pack right now.
>
> and during the pack-object we are running (aka "repack"), we can
> continue to read from that pack that may have already disappeared
> from the disk.
>
> But is that sufficient? Are we writing the resulting new pack(s)
> out in such a way that the repository is healthy without the pack
> we noticed is disappearing? How do we ensure that?
It's sufficient in the sense that we're writing out all of the objects
we were asked to (from pack-objects's perspective). Of course, if the
"simultaneous writer" is just removing packs right after they are
opened, that will produce an obviously-broken state. But assuming that
repack isn't removing objects it shouldn't (which I think is a safe
assumption from pack-objects' perspective, since all it cares about is
writing packs that contain the desired set of objects), then we are OK.
Thanks,
Taylor
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist
2022-05-24 18:54 ` [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist Taylor Blau
2022-05-24 19:46 ` Ævar Arnfjörð Bjarmason
2022-05-24 22:03 ` Junio C Hamano
@ 2022-05-26 19:21 ` Victoria Dye
2022-05-26 20:05 ` Taylor Blau
2 siblings, 1 reply; 29+ messages in thread
From: Victoria Dye @ 2022-05-26 19:21 UTC (permalink / raw)
To: Taylor Blau, git; +Cc: jonathantanmy, gitster
Taylor Blau wrote:
> A subsequent patch will teach `want_object_in_pack()` to set its
> `*found_pack` and `*found_offset` poitners to NULL when the provided
s/poitners/pointers
> pack does not pass the `is_pack_valid()` check.
>
> The `--stdin-packs` mode of `pack-objects` is not quite prepared to
> handle this. To prepare it for this change, do the following two things:
>
> - Ensure provided packs pass the `is_pack_valid()` check when
> collecting the caller-provided packs into the "included" and
> "excluded" lists.
>
Is the 'is_pack_valid()' check happening for the "excluded" packs? It looks
like you only added it for the packs in the "included" list in this patch.
> - Gracefully handle any _invalid_ packs being passed to
> `want_object_in_pack()`.
>
> Calling `is_pack_valid()` early on makes it substantially less likely
> that we will have to deal with a pack going away, since we'll have an
> open file descriptor on its contents much earlier.
>
> But even packs with open descriptors can become invalid in the future if
> we (a) hit our open descriptor limit, forcing us to close some open
> packs, and (b) one of those just-closed packs has gone away in the
> meantime.
>
> `add_object_entry_from_pack()` depends on having a non-NULL
> `*found_pack`, since it passes that pointer to `packed_object_info()`,
> meaning that we would SEGV if the pointer became NULL (like we propose
> to do in `want_object_in_pack()` in the following patch).
>
> But avoiding calling `packed_object_info()` entirely is OK, too, since
> its only purpose is to identify which objects in the included packs are
> commits, so that they can form the tips of the advisory traversal used
> to discover the object namehashes.
>
> Failing to do this means that at worst we will produce lower-quality
> deltas, but it does not prevent us from generating the pack as long as
> we can find a copy of each object from the disappearing pack in some
> other part of the repository.
>
The rest of this makes sense and (as far as I can tell) lines up with the
implementation below.
> Co-authored-by: Victoria Dye <vdye@github.com>
> Signed-off-by: Taylor Blau <me@ttaylorr.com>
> ---
> builtin/pack-objects.c | 35 ++++++++++++++++++++---------------
> 1 file changed, 20 insertions(+), 15 deletions(-)
>
> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> index ec3193fd95..ffeaecd1d8 100644
> --- a/builtin/pack-objects.c
> +++ b/builtin/pack-objects.c
> @@ -3201,10 +3201,8 @@ static int add_object_entry_from_pack(const struct object_id *oid,
> uint32_t pos,
> void *_data)
> {
> - struct rev_info *revs = _data;
> - struct object_info oi = OBJECT_INFO_INIT;
> off_t ofs;
> - enum object_type type;
> + enum object_type type = OBJ_NONE;
>
> display_progress(progress_state, ++nr_seen);
>
> @@ -3215,20 +3213,25 @@ static int add_object_entry_from_pack(const struct object_id *oid,
> if (!want_object_in_pack(oid, 0, &p, &ofs))
> return 0;
>
> - oi.typep = &type;
> - if (packed_object_info(the_repository, p, ofs, &oi) < 0)
> - die(_("could not get type of object %s in pack %s"),
> - oid_to_hex(oid), p->pack_name);
> - else if (type == OBJ_COMMIT) {
> - /*
> - * commits in included packs are used as starting points for the
> - * subsequent revision walk
> - */
> - add_pending_oid(revs, NULL, oid, 0);
> + if (p) {
> + struct rev_info *revs = _data;
> + struct object_info oi = OBJECT_INFO_INIT;
> +
> + oi.typep = &type;
> + if (packed_object_info(the_repository, p, ofs, &oi) < 0) {
> + die(_("could not get type of object %s in pack %s"),
> + oid_to_hex(oid), p->pack_name);
> + } else if (type == OBJ_COMMIT) {
> + /*
> + * commits in included packs are used as starting points for the
> + * subsequent revision walk
> + */
> + add_pending_oid(revs, NULL, oid, 0);
> + }
> +
> + stdin_packs_found_nr++;
> }
>
> - stdin_packs_found_nr++;
> -
> create_object_entry(oid, type, 0, 0, 0, p, ofs);
>
> return 0;
> @@ -3346,6 +3349,8 @@ static void read_packs_list_from_stdin(void)
> struct packed_git *p = item->util;
> if (!p)
> die(_("could not find pack '%s'"), item->string);
> + if (!is_pack_valid(p))
> + die(_("packfile %s cannot be accessed"), p->pack_name);
> }
>
> /*
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist
2022-05-26 19:21 ` Victoria Dye
@ 2022-05-26 20:05 ` Taylor Blau
0 siblings, 0 replies; 29+ messages in thread
From: Taylor Blau @ 2022-05-26 20:05 UTC (permalink / raw)
To: Victoria Dye; +Cc: Taylor Blau, git, jonathantanmy, gitster
On Thu, May 26, 2022 at 12:21:48PM -0700, Victoria Dye wrote:
> > pack does not pass the `is_pack_valid()` check.
> >
> > The `--stdin-packs` mode of `pack-objects` is not quite prepared to
> > handle this. To prepare it for this change, do the following two things:
> >
> > - Ensure provided packs pass the `is_pack_valid()` check when
> > collecting the caller-provided packs into the "included" and
> > "excluded" lists.
> >
>
> Is the 'is_pack_valid()' check happening for the "excluded" packs? It looks
> like you only added it for the packs in the "included" list in this patch.
You're right that we don't do it explicitly. That's OK, since we won't
use any objects in excluded packs, and thus don't need to eagerly grab
an descriptor on it to prevent against the race we're handling here.
(In practice, we do end up calling is_pack_valid() on excluded packs
later on, via
- want_found_object() (or one of its many callers), which itself calls
- has_object_kept_pack(), which calls
- find_kept_pack_entry(), which calls
- fill_pack_entry(), which calls
- is_pack_valid(), which calls
but that's a side-effect that doesn't help or hurt us.)
We _do_ need to be able to open the .idx (which happens in
`fill_pack_entry() -> find_pack_entry_one() -> open_pack_index()` , but
we'll fail appropriately when the index cannot be located.
> The rest of this makes sense and (as far as I can tell) lines up with the
> implementation below.
Thanks for taking a look!
Thanks,
Taylor
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v2 4/4] builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects
2022-05-24 18:54 ` [PATCH v2 0/4] pack-objects: fix a pair of MIDX bitmap-related races Taylor Blau
` (2 preceding siblings ...)
2022-05-24 18:54 ` [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist Taylor Blau
@ 2022-05-24 18:54 ` Taylor Blau
2022-05-24 21:38 ` [PATCH v2 0/4] pack-objects: fix a pair of MIDX bitmap-related races Junio C Hamano
4 siblings, 0 replies; 29+ messages in thread
From: Taylor Blau @ 2022-05-24 18:54 UTC (permalink / raw)
To: git; +Cc: vdye, jonathantanmy, gitster
When using a multi-pack bitmap, pack-objects will try to perform its
traversal using a call to `traverse_bitmap_commit_list()`, which calls
`add_object_entry_from_bitmap()` to add each object it finds to its
packing list.
This path can cause pack-objects to add objects from packs that don't
have open pack_fds on them, by avoiding a call to `is_pack_valid()`.
This is because we only call `is_pack_valid()` on the preferred pack (in
order to do verbatim reuse via `reuse_partial_packfile_from_bitmap()`)
and not others when loading a MIDX bitmap.
In this case, `add_object_entry_from_bitmap()` will check whether it
wants each object entry by calling `want_object_in_pack()`, which will
call `want_found_object` (since its caller already supplied a
`found_pack`). In most cases (particularly without `--local`, and when
`ignored_packed_keep_on_disk` and `ignored_packed_keep_in_core` are
both "0"), we'll take the entry from the pack contained in the MIDX
bitmap, all without an open pack_fd.
When we then try to use that entry later to assemble the actual pack,
we'll be susceptible to any simultaneous writers moving that pack out of
the way (e.g., due to a concurrent repack) without having an open file
descriptor, causing races that result in errors like:
remote: Enumerating objects: 1498802, done.
remote: fatal: packfile ./objects/pack/pack-e57d433b5a588daa37fbe946e2b28dfaec03a93e.pack cannot be accessed
remote: aborting due to possible repository corruption on the remote side.
This race can happen even with multi-pack bitmaps, since we may open a
MIDX bitmap that is being rewritten long before its packs are actually
unlinked.
Work around this by calling `is_pack_valid()` from within
`want_found_object()`, matching the behavior in
`want_object_in_pack_one()` (which has an analogous call). Most calls to
`is_pack_valid()` should be basically no-ops, since only the first call
requires us to open a file (subsequent calls realize the file is already
open, and return immediately).
Importantly, when `want_object_in_pack()` is given a non-NULL
`*found_pack`, but `want_found_object()` rejects the copy of the object
in that pack, we must reset `*found_pack` and `*found_offset` to NULL
and 0, respectively. Failing to do so could lead to other checks in
`want_object_in_pack()` (such as `want_object_in_pack_one()`) using the
same (invalid) pack as `*found_pack`, meaning that we don't call
`is_pack_valid()` because `p == *found_pack`. This can lead the caller
to believe it can use a copy of an object from an invalid pack.
An alternative approach to closing this race would have been to call
`is_pack_valid()` on _all_ packs in a multi-pack bitmap on load. This
has a couple of problems:
- it is unnecessarily expensive in the cases where we don't actually
need to open any packs (e.g., in `git rev-list --use-bitmap-index
--count`)
- more importantly, it means any time we would have hit this race,
we'll avoid using bitmaps altogether, leading to significant
slowdowns by forcing a full object traversal
Co-authored-by: Victoria Dye <vdye@github.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
builtin/pack-objects.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index ffeaecd1d8..0a26de166d 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1357,6 +1357,9 @@ static int want_found_object(const struct object_id *oid, int exclude,
if (incremental)
return 0;
+ if (!is_pack_valid(p))
+ return -1;
+
/*
* When asked to do --local (do not include an object that appears in a
* pack we borrow from elsewhere) or --honor-pack-keep (do not include
@@ -1472,6 +1475,9 @@ static int want_object_in_pack(const struct object_id *oid,
want = want_found_object(oid, exclude, *found_pack);
if (want != -1)
return want;
+
+ *found_pack = NULL;
+ *found_offset = 0;
}
for (m = get_multi_pack_index(the_repository); m; m = m->next) {
--
2.36.1.94.gb0d54bedca
^ permalink raw reply related [flat|nested] 29+ messages in thread
* Re: [PATCH v2 0/4] pack-objects: fix a pair of MIDX bitmap-related races
2022-05-24 18:54 ` [PATCH v2 0/4] pack-objects: fix a pair of MIDX bitmap-related races Taylor Blau
` (3 preceding siblings ...)
2022-05-24 18:54 ` [PATCH v2 4/4] builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects Taylor Blau
@ 2022-05-24 21:38 ` Junio C Hamano
2022-05-25 0:16 ` Taylor Blau
4 siblings, 1 reply; 29+ messages in thread
From: Junio C Hamano @ 2022-05-24 21:38 UTC (permalink / raw)
To: Taylor Blau; +Cc: git, vdye, jonathantanmy
Taylor Blau <me@ttaylorr.com> writes:
> verbatim reuse (c.f., `write_reused_pack_verbatim()`).
Unlike "e.g." and "i.e.", I think these should all be "cf." (there
are many others).
> + This patch handles the "preferred" pack (c.f., the section
> + "multi-pack-index reverse indexes" in
> + Documentation/technical/pack-format.txt) specially, since pack-objects
> + depends on reusing exact chunks of that pack verbatim in
> + reuse_partial_packfile_from_bitmap(). So if that pack cannot be loaded,
> + the utility of a bitmap is significantly diminished.
It explains why we care about the "preferred" pack, which is a nice
clarification. It hints that the other packs do not matter as much,
and it is clearly stated that how they are handled is ...
> + Similar to dc1daacdcc, we could technically just add this check in
> + reuse_partial_packfile_from_bitmap(), since it's possible to use a MIDX
> + .bitmap without needing to open any of its packs. But it's simpler to do
> + the check as early as possible, covering all direct uses of the
> + preferred pack. Note that doing this check early requires us to call
> + prepare_midx_pack() early, too, so move the relevant part of that loop
> + from load_reverse_index() into open_midx_bitmap_1().
> +
> + Subsequent patches handle the non-preferred packs in a slightly
> + different fashion.
... left for later steps.
Excellent write-up.
> Signed-off-by: Taylor Blau <me@ttaylorr.com>
>
> @@ pack-bitmap.c: static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,
> +
> + preferred = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];
> + if (!is_pack_valid(preferred)) {
> -+ close(fd);
> + warning(_("preferred pack (%s) is invalid"),
> + preferred->pack_name);
> + goto cleanup;
> 2: 9adf6e1341 < -: ---------- builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects
> -: ---------- > 2: 2719d33f32 builtin/pack-objects.c: avoid redundant NULL check
> -: ---------- > 3: cdc3265ec2 builtin/pack-objects.c: ensure included `--stdin-packs` exist
> -: ---------- > 4: 3fc3a95517 builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 0/4] pack-objects: fix a pair of MIDX bitmap-related races
2022-05-24 21:38 ` [PATCH v2 0/4] pack-objects: fix a pair of MIDX bitmap-related races Junio C Hamano
@ 2022-05-25 0:16 ` Taylor Blau
0 siblings, 0 replies; 29+ messages in thread
From: Taylor Blau @ 2022-05-25 0:16 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, vdye, jonathantanmy
On Tue, May 24, 2022 at 02:38:39PM -0700, Junio C Hamano wrote:
> Taylor Blau <me@ttaylorr.com> writes:
>
> > verbatim reuse (c.f., `write_reused_pack_verbatim()`).
>
> Unlike "e.g." and "i.e.", I think these should all be "cf." (there
> are many others).
Oops. You learn something new everyday! ;-)
> > + This patch handles the "preferred" pack (c.f., the section
> > + "multi-pack-index reverse indexes" in
> > + Documentation/technical/pack-format.txt) specially, since pack-objects
> > + depends on reusing exact chunks of that pack verbatim in
> > + reuse_partial_packfile_from_bitmap(). So if that pack cannot be loaded,
> > + the utility of a bitmap is significantly diminished.
>
> It explains why we care about the "preferred" pack, which is a nice
> clarification. It hints that the other packs do not matter as much,
> and it is clearly stated that how they are handled is ...
>
> > + Similar to dc1daacdcc, we could technically just add this check in
> > + reuse_partial_packfile_from_bitmap(), since it's possible to use a MIDX
> > + .bitmap without needing to open any of its packs. But it's simpler to do
> > + the check as early as possible, covering all direct uses of the
> > + preferred pack. Note that doing this check early requires us to call
> > + prepare_midx_pack() early, too, so move the relevant part of that loop
> > + from load_reverse_index() into open_midx_bitmap_1().
> > +
> > + Subsequent patches handle the non-preferred packs in a slightly
> > + different fashion.
>
> ... left for later steps.
>
> Excellent write-up.
Thanks very much!
Taylor
^ permalink raw reply [flat|nested] 29+ messages in thread