From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
To: Ivan Frade <ifrade@google.com>
Cc: Jonathan Tan <jonathantanmy@google.com>,
gitgitgadget@gmail.com, git@vger.kernel.org,
sunshine@sunshineco.com
Subject: Re: [PATCH v6 1/2] fetch-pack: redact packfile urls in traces
Date: Thu, 11 Nov 2021 01:01:32 +0100 [thread overview]
Message-ID: <211111.86tugjpn1x.gmgdl@evledraar.gmail.com> (raw)
In-Reply-To: <CANQMx9U2sRB9Qm3zxvpOwn8cqRYyA0S0jJ2=JsspJ5hcRd_XOA@mail.gmail.com>
On Wed, Nov 10 2021, Ivan Frade wrote:
> On Mon, Nov 8, 2021 at 5:53 PM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>>
> ...
>>... Let's just:
>>
>> 1. Start reading the section
>> 2. Turn off tracing
>> 3. Parse the URIs as we go
>> 3. When done (or on the fly), scrub URIs, log any backlog suppressed trace, and turn on tracing again
>
> This is a more generic redacting mechanism, but I understood that
> there is no need for it. Previous comments went in the direction of
> removing generality (e.g. not looking for a URI anywhere in the
> packet, but specifically for the packfile line format) and now this
> patch is very specific to redact packfile-uri lines in the protocol.
It's less generic, because it would live in the loop that consumes the
lines.
>> Instead of:
>>
>> 1. Set a flag to scrub stuff
>> 2. Because of the disconnect between fetch-pack.c and pkt-line.c,
>> effectively implement a new parser for data we're already going to be
>> parsing some microseconds later during the course of the request.
>
> pkt-line is only looking for the "<n-hex-chars>SP" shape. True that it
> encodes some protocol knowledge, but it is hardly a new parser.
Yeah, but why have find_packfile_uri_path() at all instead of just
moving the parsing code around?
We've already got the code that parses these lines, it's just a few
lines removed from the code you're adding...
>> That "turn off the trace" could be passing down a string_list/strbuf, or
>> even doing the same via a nev member in "struct packet_reader", both
>> would be simpler than needing to re-do the parse.
>
> Saving the lines and delaying the tracing could also produce weird
> outputs, no? e.g. 3 lines received, the second doesn't validate, the
> program aborts and the trace doesn't show any of the lines that caused
> the problem. Or we would need to iterate in parallel through lines and
> saved-log-lines assuming they match 1:1. Nothing unsolvable, but I am
> not sure it is worthy the effort now.
It would only be weird if you do :
download_later =
while (consume lines)
download_later += buffer_lines;
log lines;
I'm suggesting:
download_later =
while (consume lines)
raw, to_log = parse line
log line(to_log)
download_later += raw
Sure, you'll need to do something in the case where the line doesn't
validate, should you redact it still, or log it as is? Anyway, that's
also a caveat you've got now.
That's not iterating in parallel, having one for-loop instead of two.
I see now that that approach would also solve at least one
bug/misfeature in the packfile-uri handling, i.e.:
for (i = 0; i < packfile_uris.nr; i++) {
[...]
start_command(...) [... to download the URI ...]
[...]
die("fetch-pack: pack downloaded from %s does not match expected hash %.*s",
}
I.e. we've already received all the URIs, but then do validation on them
one at a time, so we might only notice that the server has sent us bad
data for the Nth URI after first downloading the first N-1 URIs.
next prev parent reply other threads:[~2021-11-11 0:13 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-08 16:03 [PATCH 0/2] fetch-pack: redact packfile urls in traces Ivan Frade via GitGitGadget
2021-10-08 16:03 ` [PATCH 1/2] " Ivan Frade via GitGitGadget
2021-10-08 19:36 ` Ævar Arnfjörð Bjarmason
2021-10-08 23:15 ` Ivan Frade
2021-10-08 16:03 ` [PATCH 2/2] Documentation: packfile-uri hash can be longer than 40 hex chars Ivan Frade via GitGitGadget
2021-10-08 19:43 ` Ævar Arnfjörð Bjarmason
2021-10-09 2:20 ` [PATCH v2] fetch-pack: redact packfile urls in traces Ivan Frade via GitGitGadget
2021-10-11 20:39 ` Junio C Hamano
2021-10-26 19:32 ` Ivan Frade
2021-10-19 22:57 ` [PATCH v3] " Ivan Frade via GitGitGadget
2021-10-20 11:41 ` Ævar Arnfjörð Bjarmason
2021-10-26 22:49 ` [PATCH v4 0/2] " Ivan Frade via GitGitGadget
2021-10-26 22:49 ` [PATCH v4 1/2] " Ivan Frade via GitGitGadget
2021-10-28 1:01 ` Junio C Hamano
2021-10-28 22:15 ` Ivan Frade
2021-10-28 22:46 ` Junio C Hamano
2021-10-26 22:49 ` [PATCH v4 2/2] http-fetch: redact url on die() message Ivan Frade via GitGitGadget
2021-10-28 16:39 ` Ævar Arnfjörð Bjarmason
2021-10-28 17:25 ` Eric Sunshine
2021-10-28 22:44 ` Ivan Frade
2021-10-28 22:41 ` Ivan Frade
2021-10-29 23:18 ` Junio C Hamano
2021-11-09 1:54 ` Ævar Arnfjörð Bjarmason
2021-10-28 22:51 ` [PATCH v5 0/2] fetch-pack: redact packfile urls in traces Ivan Frade via GitGitGadget
2021-10-28 22:51 ` [PATCH v5 1/2] " Ivan Frade via GitGitGadget
2021-10-28 23:21 ` Junio C Hamano
2021-10-29 18:42 ` Ivan Frade
2021-10-29 19:59 ` Junio C Hamano
2021-11-08 22:43 ` Jonathan Tan
2021-10-28 22:51 ` [PATCH v5 2/2] http-fetch: redact url on die() message Ivan Frade via GitGitGadget
2021-10-29 18:42 ` [PATCH v6 0/2] fetch-pack: redact packfile urls in traces Ivan Frade via GitGitGadget
2021-10-29 18:42 ` [PATCH v6 1/2] " Ivan Frade via GitGitGadget
2021-11-08 23:01 ` Jonathan Tan
2021-11-09 1:36 ` Ævar Arnfjörð Bjarmason
2021-11-10 23:44 ` Ivan Frade
2021-11-11 0:01 ` Ævar Arnfjörð Bjarmason [this message]
2021-11-10 21:18 ` Ivan Frade
2021-10-29 18:42 ` [PATCH v6 2/2] http-fetch: redact url on die() message Ivan Frade via GitGitGadget
2021-11-08 23:06 ` Jonathan Tan
2021-11-10 23:51 ` [PATCH v7 0/2] fetch-pack: redact packfile urls in traces Ivan Frade via GitGitGadget
2021-11-10 23:51 ` [PATCH v7 1/2] " Ivan Frade via GitGitGadget
2021-11-10 23:51 ` [PATCH v7 2/2] http-fetch: redact url on die() message Ivan Frade via GitGitGadget
2021-11-12 4:43 ` [PATCH v7 0/2] fetch-pack: redact packfile urls in traces Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=211111.86tugjpn1x.gmgdl@evledraar.gmail.com \
--to=avarab@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitgitgadget@gmail.com \
--cc=ifrade@google.com \
--cc=jonathantanmy@google.com \
--cc=sunshine@sunshineco.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).