From: Junio C Hamano <gitster@pobox.com>
To: Jeff King <peff@peff.net>
Cc: Johannes Schindelin <johannes.schindelin@gmx.de>,
Johannes Schindelin via GitGitGadget <gitgitgadget@gmail.com>,
git@vger.kernel.org
Subject: Re: [PATCH 0/4] t5552: fix flakiness by introducing proper locking for GIT_TRACE
Date: Fri, 10 Aug 2018 08:58:25 -0700 [thread overview]
Message-ID: <xmqq36vm9psu.fsf@gitster-ct.c.googlers.com> (raw)
In-Reply-To: <20180810140908.GA23507@sigill.intra.peff.net> (Jeff King's message of "Fri, 10 Aug 2018 10:09:08 -0400")
Jeff King <peff@peff.net> writes:
> Here it is as a patch on top of jt/fetch-negotiator-skipping, which lets
> us pursue any fix for interleaved trace output on Windows without the
> pressure of an impending flaky test.
>
> My gut says that looking into making O_APPEND work there is going to be
> the nicest solution, but my gut is not very well versed in Windows
> subtleties. ;)
As we know these tests are only interested in "fetch" lines and not
record of communication in the other direction, I think this is a
nice clean-up that is worth doing.
For only two grep's it may not look worth doing, but it would be
also a good clean-up to give an got_acked helper that sits next to
have_sent and have_not_sent helpers for readability. That is of
course outside the scope of this change.
Will queue. Thansk.
> -- >8 --
> Subject: [PATCH] t5552: suppress upload-pack trace output
>
> The t5552 test script uses GIT_TRACE_PACKET to monitor what
> git-fetch sends and receives. However, because we're
> accessing a local repository, the child upload-pack also
> sends trace output to the same file.
>
> On Linux, this works out OK. We open the trace file with
> O_APPEND, so all writes are atomically positioned at the end
> of the file. No data can be overwritten or omitted. And
> since we prepare our small writes in a strbuf and write them
> with a single write(), we should see each line as an atomic
> unit. The order of lines between the two processes is
> undefined, but the test script greps only for "fetch>" or
> "fetch<" lines. So under Linux, the test results are
> deterministic.
>
> The test fails intermittently on Windows, however,
> reportedly even overwriting bits of the output file (i.e.,
> O_APPEND does not seem to give us an atomic position+write).
>
> Since the test only cares about the trace output from fetch,
> we can just disable the output from upload-pack. That
> doesn't solve the greater question of O_APPEND/trace issues
> under Windows, but it easily fixes the flakiness from this
> test.
>
> Reported-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> Signed-off-by: Jeff King <peff@peff.net>
> ---
> I'm assuming that this really isn't triggerable on Linux. I tried and
> couldn't manage to get it to fail, and the reasoning above explains why.
> But I wasn't 100% clear that Dscho hadn't seen it fail on non-Windows.
>
> t/t5552-skipping-fetch-negotiator.sh | 23 ++++++++++++++++++-----
> 1 file changed, 18 insertions(+), 5 deletions(-)
>
> diff --git a/t/t5552-skipping-fetch-negotiator.sh b/t/t5552-skipping-fetch-negotiator.sh
> index 0a8e0e42ed..0ad50dd839 100755
> --- a/t/t5552-skipping-fetch-negotiator.sh
> +++ b/t/t5552-skipping-fetch-negotiator.sh
> @@ -28,6 +28,19 @@ have_not_sent () {
> done
> }
>
> +# trace_fetch <client_dir> <server_dir> [args]
> +#
> +# Trace the packet output of fetch, but make sure we disable the variable
> +# in the child upload-pack, so we don't combine the results in the same file.
> +trace_fetch () {
> + client=$1; shift
> + server=$1; shift
> + GIT_TRACE_PACKET="$(pwd)/trace" \
> + git -C "$client" fetch \
> + --upload-pack 'unset GIT_TRACE_PACKET; git-upload-pack' \
> + "$server" "$@"
> +}
> +
> test_expect_success 'commits with no parents are sent regardless of skip distance' '
> git init server &&
> test_commit -C server to_fetch &&
> @@ -42,7 +55,7 @@ test_expect_success 'commits with no parents are sent regardless of skip distanc
> # "c1" has no parent, it is still sent as "have" even though it would
> # normally be skipped.
> test_config -C client fetch.negotiationalgorithm skipping &&
> - GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch "$(pwd)/server" &&
> + trace_fetch client "$(pwd)/server" &&
> have_sent c7 c5 c2 c1 &&
> have_not_sent c6 c4 c3
> '
> @@ -65,7 +78,7 @@ test_expect_success 'when two skips collide, favor the larger one' '
> # the next "have" sent will be "c1" (from "c6" skip 4) and not "c4"
> # (from "c5side" skip 1).
> test_config -C client fetch.negotiationalgorithm skipping &&
> - GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch "$(pwd)/server" &&
> + trace_fetch client "$(pwd)/server" &&
> have_sent c5side c11 c9 c6 c1 &&
> have_not_sent c10 c8 c7 c5 c4 c3 c2
> '
> @@ -91,7 +104,7 @@ test_expect_success 'use ref advertisement to filter out commits' '
> # not need to send any ancestors of "c3", but we still need to send "c3"
> # itself.
> test_config -C client fetch.negotiationalgorithm skipping &&
> - GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch origin to_fetch &&
> + trace_fetch client origin to_fetch &&
> have_sent c5 c4^ c2side &&
> have_not_sent c4 c4^^ c4^^^
> '
> @@ -121,7 +134,7 @@ test_expect_success 'handle clock skew' '
> # and sent, because (due to clock skew) its only parent has already been
> # popped off the priority queue.
> test_config -C client fetch.negotiationalgorithm skipping &&
> - GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch "$(pwd)/server" &&
> + trace_fetch client "$(pwd)/server" &&
> have_sent c2 c1 old4 old2 old1 &&
> have_not_sent old3
> '
> @@ -153,7 +166,7 @@ test_expect_success 'do not send "have" with ancestors of commits that server AC
> test_commit -C server commit-on-b1 &&
>
> test_config -C client fetch.negotiationalgorithm skipping &&
> - GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch "$(pwd)/server" to_fetch &&
> + trace_fetch client "$(pwd)/server" to_fetch &&
> grep " fetch" trace &&
>
> # fetch-pack sends 2 requests each containing 16 "have" lines before
next prev parent reply other threads:[~2018-08-10 15:58 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-08-09 17:35 [PATCH 0/4] t5552: fix flakiness by introducing proper locking for GIT_TRACE Johannes Schindelin via GitGitGadget
2018-08-09 17:35 ` [PATCH 1/4] Introduce a function to lock/unlock file descriptors when appending Johannes Schindelin via GitGitGadget
2018-08-09 19:01 ` Junio C Hamano
2018-08-10 18:32 ` Johannes Schindelin
2018-08-09 17:35 ` [PATCH 2/4] mingw: implement lock_or_unlock_fd_for_appending() Johannes Schindelin via GitGitGadget
2018-08-09 17:35 ` [PATCH 3/4] trace: lock the trace file to avoid racy trace_write() calls Johannes Schindelin via GitGitGadget
2018-08-09 17:35 ` [PATCH 4/4] trace: verify that locking works Johannes Schindelin via GitGitGadget
2018-08-09 19:47 ` [PATCH 0/4] t5552: fix flakiness by introducing proper locking for GIT_TRACE Jeff King
2018-08-09 20:49 ` Junio C Hamano
2018-08-09 21:08 ` Junio C Hamano
2018-08-09 21:32 ` Jeff King
2018-08-10 14:09 ` Jeff King
2018-08-10 15:58 ` Junio C Hamano [this message]
2018-08-10 15:58 ` Junio C Hamano
2018-08-10 16:43 ` Johannes Schindelin
2018-08-10 17:15 ` Jeff King
2018-08-10 18:40 ` Junio C Hamano
2018-08-10 19:34 ` Johannes Schindelin
2018-08-10 16:15 ` Johannes Sixt
2018-08-10 16:51 ` Jeff Hostetler
2018-08-10 16:57 ` Jeff Hostetler
2018-08-10 17:08 ` Johannes Sixt
2018-08-10 18:34 ` Junio C Hamano
2018-08-10 18:56 ` Jeff King
2018-08-13 19:02 ` [PATCH] mingw: enable atomic O_APPEND Johannes Sixt
2018-08-13 20:20 ` Junio C Hamano
2018-08-13 21:05 ` Johannes Sixt
2018-08-13 21:22 ` Ævar Arnfjörð Bjarmason
2018-08-13 21:55 ` Junio C Hamano
2018-08-13 22:37 ` Jeff King
2018-08-14 13:47 ` Ævar Arnfjörð Bjarmason
2018-08-14 14:53 ` Jeff King
2018-08-14 18:29 ` Johannes Sixt
2018-08-14 19:17 ` Jeff King
2018-08-14 13:01 ` Jeff Hostetler
2018-08-14 14:38 ` Junio C Hamano
2018-08-10 19:47 ` [PATCH v2 0/4] t5552: fix flakiness by introducing proper locking for GIT_TRACE Johannes Schindelin via GitGitGadget
2018-08-10 19:47 ` [PATCH v2 1/4] Introduce a function to lock/unlock file descriptors when appending Johannes Schindelin via GitGitGadget
2018-08-10 20:05 ` Junio C Hamano
2018-08-10 21:31 ` Johannes Schindelin
2018-08-10 19:47 ` [PATCH v2 2/4] mingw: implement lock_or_unlock_fd_for_appending() Johannes Schindelin via GitGitGadget
2018-08-10 19:47 ` [PATCH v2 3/4] trace: lock the trace file to avoid racy trace_write() calls Johannes Schindelin via GitGitGadget
2018-08-10 19:47 ` [PATCH v2 4/4] trace: verify that locking works Johannes Schindelin via GitGitGadget
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=xmqq36vm9psu.fsf@gitster-ct.c.googlers.com \
--to=gitster@pobox.com \
--cc=git@vger.kernel.org \
--cc=gitgitgadget@gmail.com \
--cc=johannes.schindelin@gmx.de \
--cc=peff@peff.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).