Merge ORT performance in the wild

Merge ORT performance in the wild

[Derrick Stolee]

Hello all, especially Elijah.

We finally shipped Git 2.33.0 (with VFS for Git) to the Windows OS
engineers, and the microsoft/git fork enables the ORT merge strategy
by default in this version.

I know that we are queued to include the ORT merge strategy as on
by default in Git 2.34.0, and to further support that change (and
thank Elijah for working hard on the feature), I wanted to share
some early data on our users interacting with it.

We have about 250 users who have upgraded, relative to ~2,300 users
who were active on the previous version. However, we saw sufficiently
high use of 'git cherry-pick' and 'git merge' in these early users to
share these results for how they are experiencing the new world:

| Builtin     | Percentile | Recursive | ORT   |
|-------------|------------|-----------|-------|
| cherry-pick | 50         |  18.4s    | 14.1s |
| cherry-pick | 80         |  34.9s    | 15.4s |
| cherry-pick | 95         | 117.9s    | 17.7s |
| merge       | 50         |   7.7s    |  1.2s |
| merge       | 80         |  17.9s    | 12.7s |
| merge       | 95         |  58.9s    | 22.3s |

This matches the results from a synthetic performance test I ran
in our monorepos: ORT is always faster, but its outlier performance
is far faster than the outlier performance of the 'recursive'
strategy.

Thanks!
-Stolee

Re: Merge ORT performance in the wild

[Elijah Newren]

On Tue, Oct 5, 2021 at 12:42 PM Derrick Stolee <stolee@gmail.com> wrote:
>
> Hello all, especially Elijah.
>
> We finally shipped Git 2.33.0 (with VFS for Git) to the Windows OS
> engineers, and the microsoft/git fork enables the ORT merge strategy
> by default in this version.
>
> I know that we are queued to include the ORT merge strategy as on
> by default in Git 2.34.0, and to further support that change (and
> thank Elijah for working hard on the feature), I wanted to share
> some early data on our users interacting with it.
>
> We have about 250 users who have upgraded, relative to ~2,300 users
> who were active on the previous version. However, we saw sufficiently
> high use of 'git cherry-pick' and 'git merge' in these early users to
> share these results for how they are experiencing the new world:
>
> | Builtin     | Percentile | Recursive | ORT   |
> |-------------|------------|-----------|-------|
> | cherry-pick | 50         |  18.4s    | 14.1s |
> | cherry-pick | 80         |  34.9s    | 15.4s |
> | cherry-pick | 95         | 117.9s    | 17.7s |
> | merge       | 50         |   7.7s    |  1.2s |
> | merge       | 80         |  17.9s    | 12.7s |
> | merge       | 95         |  58.9s    | 22.3s |
>
> This matches the results from a synthetic performance test I ran
> in our monorepos: ORT is always faster, but its outlier performance
> is far faster than the outlier performance of the 'recursive'
> strategy.

Great news, thanks for sharing.

This is basically just testing one of the optimizations, too.  I'm
still curious what the performance difference would be on your large
repositories, comparing git <= v2.30's recursive (i.e. before my
optimizations, some of which benefitted recursive) with rename
detection _not_ turned off, and current ort.  I suspect you'd have a
hard time convincing users to use the former, though.  IIRC, Ben
reported one example merge took on the order of an hour, but I don't
know which repository that was with or if it even had many renames.
Anyway, it's nice to know that even without the consideration of
rename detection in the old comparison numbers, you're still seeing
nice speedups.