Re: [GSOC] [QUESTION] ref-filter: can %(raw) implement reuse oi.content?

[Christian Couder <christian.couder@gmail.com>]

Hi ZheNing,

On Thu, Aug 19, 2021 at 3:39 AM ZheNing Hu <adlternative@gmail.com> wrote:
>
> Hi, Christian and Hariom,
>
> I want to use this patch series as the temporary final version of GSOC project:
>
> https://github.com/adlternative/git/commits/cat-file-reuse-ref-filter-logic

I am still not very happy with the last patch in the series,but it can
be improved later.

> Due to the branch ref-filter-opt-code-logic or branch
> ref-filter-opt-perf patch series
> temporarily unable to reflect its optimization to git cat-file
> --batch. Therefore, using
> branch cat-file-reuse-ref-filter-logic is the most effective now.
>
> This is the final performance regression test result:
> Test                                        upstream/master   this
> tree
> ------------------------------------------------------------------------------------
> 1006.2: cat-file --batch-check              0.06(0.06+0.00)
> 0.08(0.07+0.00) +33.3%
> 1006.3: cat-file --batch-check with atoms   0.06(0.04+0.01)
> 0.06(0.06+0.00) +0.0%
> 1006.4: cat-file --batch                    0.49(0.47+0.02)
> 0.48(0.47+0.01) -2.0%
> 1006.5: cat-file --batch with atoms         0.48(0.44+0.03)
> 0.47(0.46+0.01) -2.1%
>
> git cat-file --batch has a performance improvement of about 2%.
> git cat-file --batch-check still has a performance gap of 33.3%.
>
> The performance degradation of git cat-file --batch-check is actually
> not very big.
>
> upstream/master (225bc32a98):
>
> $ hyperfine --warmup=10  "~/git/bin-wrappers/git cat-file
> --batch-check --batch-all-objects"
> Benchmark #1: ~/git/bin-wrappers/git cat-file --batch-check --batch-all-objects
>  Time (mean ± σ):     596.2 ms ±   5.7 ms    [User: 563.0 ms, System: 32.5 ms]
>  Range (min … max):   586.9 ms … 607.9 ms    10 runs
>
> cat-file-reuse-ref-filter-logic (709a0c5c12):
>
> $ hyperfine --warmup=10  "~/git/bin-wrappers/git cat-file
> --batch-check --batch-all-objects"
> Benchmark #1: ~/git/bin-wrappers/git cat-file --batch-check --batch-all-objects
>  Time (mean ± σ):     601.3 ms ±   5.8 ms    [User: 566.9 ms, System: 33.9 ms]
>  Range (min … max):   596.7 ms … 613.3 ms    10 runs
>
> The execution time of git cat-file --batch-check is only a few
> milliseconds away.

Yeah, it looks like less than 1% overhead.

Great work!

> But look at the execution time changes of git cat-file --batch:
>
> upstream/master (225bc32a98):
>
> $ time ~/git/bin-wrappers/git cat-file --batch --batch-all-objects
> >/dev/null
> /home/adl/git/bin-wrappers/git cat-file --batch --batch-all-objects >
>  24.61s user 0.30s system 99% cpu 24.908 total
>
> cat-file-reuse-ref-filter-logic (709a0c5c12):
>
> $ time ~/git/bin-wrappers/git cat-file --batch --batch-all-objects >/dev/null
> cat-file --batch --batch-all-objects > /dev/null  25.10s user 0.30s
> system 99% cpu 25.417 total
>
> The execution time has been reduced by nearly 0.5 seconds.

It looks like it has increased by 0.5s, not been reduced.

> Intuition
> tells me that the performance improvement of git cat-file --batch will be
> more important.
>
> In fact, git cat-file origin code directly adds the obtained object data
> to the output buffer; But after using ref-filter logic, it needs to copy
> the object data to the intermediate data (atom_value), and finally
> to the output buffer. At present, we cannot easily eliminate intermediate
> data, because git for-each-ref --sort has a lot of dependence on it,
> but we can reduce the overhead of copying or allocating memory as
> much as possible.

Ok.

> I had an idea that I didn't implement before: partial data delayed evaluation.
> Or to be more specific, waiting until the data is about to be added to
> the output
> buffer, form specific output content, this may be a way to bypass the
> intermediate
> data.

Yeah, that might be a good idea.