[PATCH 0/2] [RFC] scalar: prepare documentation for future work

[PATCH 0/2] [RFC] scalar: prepare documentation for future work

[Victoria Dye via GitGitGadget]

What's this series?
===================

The questions of "what is Scalar?" and "why is Scalar part of Git?" have
been discussed at length already [1], but in preparation for the next few
series I plan to submit "finishing" the initial Scalar implementation
effort, I'd like to start by reframing its purpose.

Beginning with the "Philosophy of Scalar" [2], much of the existing
documentation refers to Scalar as "opinionated" - not unlike git itself.
However, with scalar now part of Git, continuing to describe it as
"opinionated" could unintentionally portray it as a divergence from "the
rest of Git" and its development community.

Rather than think of Scalar as an opinionated take on Git, I prefer to think
of it as a manifestation of Git's own "opinions" for a specific use case,
like many of the built-in commands. Over the years, developers have
contributed tons of features - often gated behind config settings - that
address performance issues they've encountered working in large
repositories: untracked cache, sparse index, merge-ort, etc. Some of these
features (like merge-ort) provide near-universal benefit to users and become
part of Git's default configuration. More often, though, these features are
large and difficult to prove are stable, or require some advanced manual
configuration to use properly, or simply make the "average" Git user's
experience worse (e.g., improving performance in in large repos, but
degrading it in smaller repos).

We've accumulated so many of these features that it's become difficult for
users to even find all of them, let alone use them. Git "knows" these
features exist, but also "knows" not to enable them by default because
they're either too experimental or not helpful to most users. Scalar is a
way for Git to take all of these features, bundle them together, and tell a
user, "Hey, do you need to make your large repo faster? Use this!" without
touching the default user experience.

All that to say, Scalar is essentially a collection of porcelain commands
[3] that fill a substantial UX niche in Git. Now, let's get into Scalar's
"future state".


A plan for Scalar
=================

Given the slightly tweaked "philosophy" above, my ultimate goal for Scalar
is to have it contain only what is too experimental or too large
repo-focused to be a default option or behavior in Git. Over time, some
features may be moved out of Scalar and into Git defaults as they are proven
stable and beneficial to the vast majority of users [4].

So what do we need to get there?

At a high level, the remaining work to "finalize" Scalar (past this RFC) can
be broken into three parts:

 1. Complete a few more features and subcommands of Scalar (integrate with
    the built-in FSMonitor & implement scalar help).
 2. Move stable, general-purpose parts of 'scalar.c' into other Git
    builtins/libraries (mainly scalar diagnose, either as part of git
    bug-report [5] or a new git built-in).
 3. Move Scalar out of contrib/ and into the "top-level" of Git. Includes
    expanded testing, especially performance testing.

The first makes scalar "feature complete" enough to be valuable to large
repo users (per my entirely subjective assessment, at least). The second
brings it in line with the goal of making Scalar only contain what can't
exist as a default feature of Git. Once those are finished, I think Scalar
will be out of its "work-in-progress" phase and ready to use as a built-in
component of Git (accompanied by sufficient testing, of course).


What's Scalar's future?
=======================

With the completion of the tasks listed in the previous section, there's
nothing left from the original Scalar CLI (in the Microsoft fork of Git) to
upstream. I don't consider that the "end" of supporting Scalar because, for
it to remain an effective tool, it'll need to stay up-to-date with the
latest performance features introduced to Git.

For example, one possible future extension might be to have Scalar enable
the sparse index by default, especially when more built-ins are updated with
sparse index compatibility. I'm interested in hearing what other features
might fit well there!


What I'm looking for
====================

The two patches in this series revise existing documentation to match the
description above. Please let me know if 1) those revisions match your
understanding of the above, 2) they convey the intent clearly, and (most
importantly) 3) they reflect a reasonable direction to take Scalar. And, of
course, if there's something I missed, I welcome any and all feedback &
ideas!

Thanks,

 * Victoria

[1]
https://lore.kernel.org/git/b67bbef4-e4c3-b6a7-1c7f-7d405902ef8b@gmail.com/

[2]
https://github.com/microsoft/git/blob/173a9b783a414/contrib/scalar/docs/philosophy.md

[3] If you're wondering "why not call it git-scalar then?", my short answer
is 1) backward compatibility with the original "scalar" CLI, and 2) its
scope is larger and more persistent than a single command; Scalar works
across multiple repos, and each Scalar subcommand has scope and
functionality on par with a built-in command. I'm happy to talk through this
more if people are still unsure, though!

[4] That being said, Scalar's existence should not make Git more risk averse
in accepting new general-purpose features.

[5] https://lore.kernel.org/git/220517.86y1z05gja.gmgdl@evledraar.gmail.com/

Victoria Dye (2):
  scalar: reword command documentation to clarify purpose
  scalar: convert README.md into a technical design doc

 contrib/scalar/README.md            |  82 ------------------
 contrib/scalar/scalar.txt           |   9 +-
 contrib/scalar/technical/scalar.txt | 127 ++++++++++++++++++++++++++++
 3 files changed, 131 insertions(+), 87 deletions(-)
 delete mode 100644 contrib/scalar/README.md
 create mode 100644 contrib/scalar/technical/scalar.txt


base-commit: e4a4b31577c7419497ac30cebe30d755b97752c5
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1275%2Fvdye%2Fscalar%2Fdoc-update-rfc-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1275/vdye/scalar/doc-update-rfc-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1275
-- 
gitgitgadget

[PATCH 1/2] scalar: reword command documentation to clarify purpose

[Victoria Dye via GitGitGadget]

From: Victoria Dye <vdye@github.com>

Rephrase documentation to describe scalar as a "large repo management tool"
rather than an "opinionated management tool". The new description is
intended to more directly reflect the utility of scalar to better guide
users in preparation for scalar being built and installed as part of Git.

Signed-off-by: Victoria Dye <vdye@github.com>
---
 contrib/scalar/scalar.txt | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index c0425e06533..1a12dc45077 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -3,7 +3,7 @@ scalar(1)
 
 NAME
 ----
-scalar - an opinionated repository management tool
+scalar - A tool for managing large Git repositories
 
 SYNOPSIS
 --------
@@ -20,10 +20,9 @@ scalar delete <enlistment>
 DESCRIPTION
 -----------
 
-Scalar is an opinionated repository management tool. By creating new
-repositories or registering existing repositories with Scalar, your Git
-experience will speed up. Scalar sets advanced Git config settings,
-maintains your repositories in the background, and helps reduce data sent
+Scalar is a repository management tool that optimizes Git for use in large
+repositories. Scalar improves performance by configuring advanced Git settings,
+maintaining repositories in the background, and helping to reduce data sent
 across the network.
 
 An important Scalar concept is the enlistment: this is the top-level directory
-- 
gitgitgadget

[PATCH 2/2] scalar: convert README.md into a technical design doc

[Victoria Dye via GitGitGadget]

From: Victoria Dye <vdye@github.com>

Replace 'README.md' with 'technical/scalar.txt' (still in 'contrib/'). In
addition to reformatting for asciidoc, elaborate on the background, purpose,
and design choices that went into Scalar.

This document is intended to persist in the 'Documentation/technical/'
directory after Scalar has been moved into the root of Git (out of
'contrib/'). Before then, it will also contain a "Roadmap" section detailing
the remaining series needed to finish the initial version of Scalar. The
section will be removed once Scalar is moved to the repo root, but in the
meantime serves as a guide for readers to keep up with progress on the
feature.

Signed-off-by: Victoria Dye <vdye@github.com>
---
 contrib/scalar/README.md            |  82 ------------------
 contrib/scalar/technical/scalar.txt | 127 ++++++++++++++++++++++++++++
 2 files changed, 127 insertions(+), 82 deletions(-)
 delete mode 100644 contrib/scalar/README.md
 create mode 100644 contrib/scalar/technical/scalar.txt

diff --git a/contrib/scalar/README.md b/contrib/scalar/README.md
deleted file mode 100644
index 634b5771ed3..00000000000
--- a/contrib/scalar/README.md
+++ /dev/null
@@ -1,82 +0,0 @@
-# Scalar - an opinionated repository management tool
-
-Scalar is an add-on to Git that helps users take advantage of advanced
-performance features in Git. Originally implemented in C# using .NET Core,
-based on the learnings from the VFS for Git project, most of the techniques
-developed by the Scalar project have been integrated into core Git already:
-
-* partial clone,
-* commit graphs,
-* multi-pack index,
-* sparse checkout (cone mode),
-* scheduled background maintenance,
-* etc
-
-This directory contains the remaining parts of Scalar that are not (yet) in
-core Git.
-
-## Roadmap
-
-The idea is to populate this directory via incremental patch series and
-eventually move to a top-level directory next to `gitk-git/` and to `git-gui/`. The
-current plan involves the following patch series:
-
-- `scalar-the-beginning`: The initial patch series which sets up
-  `contrib/scalar/` and populates it with a minimal `scalar` command that
-  demonstrates the fundamental ideas.
-
-- `scalar-c-and-C`: The `scalar` command learns about two options that can be
-  specified before the command, `-c <key>=<value>` and `-C <directory>`.
-
-- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.
-
-- `scalar-and-builtin-fsmonitor`: The built-in FSMonitor is enabled in `scalar
-  register` and in `scalar clone`, for an enormous performance boost when
-  working in large worktrees. This patch series necessarily depends on Jeff
-  Hostetler's FSMonitor patch series to be integrated into Git.
-
-- `scalar-gentler-config-locking`: Scalar enlistments are registered in the
-  user's Git config. This usually does not represent any problem because it is
-  rare for a user to register an enlistment. However, in Scalar's functional
-  tests, Scalar enlistments are created galore, and in parallel, which can lead
-  to lock contention. This patch series works around that problem by re-trying
-  to lock the config file in a gentle fashion.
-
-- `scalar-extra-docs`: Add some extensive documentation that has been written
-  in the original Scalar project (all subject to discussion, of course).
-
-- `optionally-install-scalar`: Now that Scalar is feature (and documentation)
-  complete and is verified in CI builds, let's offer to install it.
-
-- `move-scalar-to-toplevel`: Now that Scalar is complete, let's move it next to
-  `gitk-git/` and to `git-gui/`, making it a top-level command.
-
-The following two patch series exist in Microsoft's fork of Git and are
-publicly available. There is no current plan to upstream them, not because I
-want to withhold these patches, but because I don't think the Git community is
-interested in these patches.
-
-There are some interesting ideas there, but the implementation is too specific
-to Azure Repos and/or VFS for Git to be of much help in general (and also: my
-colleagues tried to upstream some patches already and the enthusiasm for
-integrating things related to Azure Repos and VFS for Git can be summarized in
-very, very few words).
-
-These still exist mainly because the GVFS protocol is what Azure Repos has
-instead of partial clone, while Git is focused on improving partial clone:
-
-- `scalar-with-gvfs`: The primary purpose of this patch series is to support
-  existing Scalar users whose repositories are hosted in Azure Repos (which
-  does not support Git's partial clones, but supports its predecessor, the GVFS
-  protocol, which is used by Scalar to emulate the partial clone).
-
-  Since the GVFS protocol will never be supported by core Git, this patch
-  series will remain in Microsoft's fork of Git.
-
-- `run-scalar-functional-tests`: The Scalar project developed a quite
-  comprehensive set of integration tests (or, "Functional Tests"). They are the
-  sole remaining part of the original C#-based Scalar project, and this patch
-  adds a GitHub workflow that runs them all.
-
-  Since the tests partially depend on features that are only provided in the
-  `scalar-with-gvfs` patch series, this patch cannot be upstreamed.
diff --git a/contrib/scalar/technical/scalar.txt b/contrib/scalar/technical/scalar.txt
new file mode 100644
index 00000000000..d785a5c036a
--- /dev/null
+++ b/contrib/scalar/technical/scalar.txt
@@ -0,0 +1,127 @@
+Scalar
+======
+
+Scalar is a built-in repository management tool that optimizes Git for use in
+large repositories. It accomplishes this by helping users to take advantage of
+advanced performance features in Git. Unlike most other Git built-in commands,
+Scalar is not executed as a subcommand of 'git'; rather, it is built as a
+separate executable containing its own series of subcommands.
+
+Background
+----------
+
+Scalar was originally designed as an add-on to Git and implemented as a .NET
+Core application. It was created based on the learnings from the VFS for Git
+project (another application aimed at improving the experience of working with
+large repositories). As part of its initial implementation, Scalar relied on
+custom features in the Microsoft fork of Git that have since been integrated
+into core Git:
+
+* partial clone,
+* commit graphs,
+* multi-pack index,
+* sparse checkout (cone mode),
+* scheduled background maintenance,
+* etc
+
+With the requisite Git functionality in place and a desire to bring the benefits
+of Scalar to the larger Git community, the Scalar application itself was ported
+from C# to C and integrated upstream.
+
+Features
+--------
+
+Scalar is comprised of two major pieces of functionality: automatically
+configuring built-in Git performance features and managing repository
+enlistments.
+
+The Git performance features configured by Scalar (see "Background" for
+examples) confer substantial performance benefits to large repositories, but are
+either too experimental to enable for all of Git yet, or only benefit large
+repositories. As new features are introduced, Scalar should be updated
+accordingly to incorporate them. This will prevent the tool from becoming stale
+while also providing a path for more easily bringing features to the appropriate
+users.
+
+Enlistments are how Scalar knows which repositories on a user's system should
+utilize Scalar-configured features. This allows it to update performance
+settings when new ones are added to the tool, as well as centrally manage
+repository maintenance. The enlistment structure - a root directory with a
+`src/` subdirectory containing the cloned repository itself - is designed to
+encourage users to route build outputs outside of the repository to avoid the
+performance-limiting overhead of ignoring those files in Git.
+
+Design
+------
+
+Scalar is implemented in C and interacts with Git via a mix of child process
+invocations of Git and direct usage of `libgit.a`. Internally, it is structured
+much like other built-ins with subcommands (e.g., `git stash`), containing a
+`cmd_<subcommand>()` function for each subcommand, routed through a `cmd_main()`
+function. Most options are unique to each subcommand, with `scalar` respecting
+some "global" `git` options (e.g., `-c` and `-C`).
+
+Because `scalar` is not invoked as a Git subcommand (like `git scalar`), it is
+built and installed as its own executable in the `bin/` directory, alongside
+`git`, `git-gui`, etc.
+
+Roadmap
+-------
+
+NOTE: this section will be removed once the remaining tasks outlined in this
+roadmap are complete.
+
+Scalar is a large enough project that it is being upstreamed incrementally,
+living in `contrib/` until it is feature-complete. So far, the following patch
+series have been accepted:
+
+- `scalar-the-beginning`: The initial patch series which sets up
+  `contrib/scalar/` and populates it with a minimal `scalar` command that
+  demonstrates the fundamental ideas.
+
+- `scalar-c-and-C`: The `scalar` command learns about two options that can be
+  specified before the command, `-c <key>=<value>` and `-C <directory>`.
+
+- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.
+
+Roughly speaking (and subject to change), the following series are needed to
+"finish" this initial version of Scalar:
+
+- Finish Scalar features: Enable the built-in FSMonitor in Scalar
+  enlistments and implement `scalar help`. At the end of this series, Scalar
+  should be feature-complete from the perspective of a user.
+
+- Generalize features not specific to Scalar: In the spirit of
+  making Scalar configure only what is needed for large repo performance, move
+  common utilities into other parts of Git. Some of this will be internal-only,
+  but one major change will be generalizing `scalar diagnose` for use with any
+  Git repository.
+
+- Move Scalar to toplevel: Make `scalar` a built-in component of Git by
+  moving it out of `contrib/` and into the root of `git`. The actual change will
+  be relatively small, but this series will also contain expanded testing to
+  ensure Scalar is stable and performant.
+
+Finally, there are two additional patch series that exist in Microsoft's fork of
+Git, but there is no current plan to upstream them. There are some interesting
+ideas there, but the implementation is too specific to Azure Repos and/or VFS
+for Git to be of much help in general.
+
+These still exist mainly because the GVFS protocol is what Azure Repos has
+instead of partial clone, while Git is focused on improving partial clone:
+
+- `scalar-with-gvfs`: The primary purpose of this patch series is to support
+  existing Scalar users whose repositories are hosted in Azure Repos (which does
+  not support Git's partial clones, but supports its predecessor, the GVFS
+  protocol, which is used by Scalar to emulate the partial clone).
+
+  Since the GVFS protocol will never be supported by core Git, this patch series
+  will remain in Microsoft's fork of Git.
+
+- `run-scalar-functional-tests`: The Scalar project developed a quite
+  comprehensive set of integration tests (or, "Functional Tests"). They are the
+  sole remaining part of the original C#-based Scalar project, and this patch
+  adds a GitHub workflow that runs them all.
+
+  Since the tests partially depend on features that are only provided in the
+  `scalar-with-gvfs` patch series, this patch cannot be upstreamed.
-- 
gitgitgadget

Re: [PATCH 0/2] [RFC] scalar: prepare documentation for future work

[Derrick Stolee]

On 6/29/2022 12:58 PM, Victoria Dye via GitGitGadget wrote:

> Beginning with the "Philosophy of Scalar" [2], much of the existing
> documentation refers to Scalar as "opinionated" - not unlike git itself.
> However, with scalar now part of Git, continuing to describe it as
> "opinionated" could unintentionally portray it as a divergence from "the
> rest of Git" and its development community.

I appreciate this perspective.
 
> Rather than think of Scalar as an opinionated take on Git, I prefer to think
> of it as a manifestation of Git's own "opinions" for a specific use case,

This is a good reframing.

> All that to say, Scalar is essentially a collection of porcelain commands
> [3] that fill a substantial UX niche in Git. Now, let's get into Scalar's
> "future state".

(I was going to comment here about Scalar as proving ground, but...)
  
> A plan for Scalar
> =================
> 
> Given the slightly tweaked "philosophy" above, my ultimate goal for Scalar
> is to have it contain only what is too experimental or too large
> repo-focused to be a default option or behavior in Git. Over time, some
> features may be moved out of Scalar and into Git defaults as they are proven
> stable and beneficial to the vast majority of users [4].

Here you point out that the new goal is for Scalar to be a proving ground
for new features. We can more readily update the recommended Scalar config
than we can update Git defaults.

For example, Scalar's first version included setting up cone-mode sparse-
checkout when that was a brand-new feature. This new default was adopted
in core Git two years later.

> So what do we need to get there?
> 
> At a high level, the remaining work to "finalize" Scalar (past this RFC) can
> be broken into three parts:
> 
>  1. Complete a few more features and subcommands of Scalar (integrate with
>     the built-in FSMonitor & implement scalar help).

The builtin FS Monitor setting is one that has also been part of the Scalar
implementation for a while, but I also doubt that we will make it a default
setting for _all_ Git repositories. Here, we are a good proving ground for
"recommended config for large repos" until we create an alternative under
the 'git' executable.

>  2. Move stable, general-purpose parts of 'scalar.c' into other Git
>     builtins/libraries (mainly scalar diagnose, either as part of git
>     bug-report [5] or a new git built-in).
> 
>  3. Move Scalar out of contrib/ and into the "top-level" of Git. Includes
>     expanded testing, especially performance testing.

This is an interesting order change from the initial plan. I like that
we are minimizing the code footprint of Scalar before doing the move out
of contrib/. But also, this changes the complex nature of it being in
contrib/ but also wanting to change APIs in libgit.a for the sake of
Scalar.

Your step 2 is focused on changing APIs in libgit.a _for the sake of
Git commands_ and in the meantime removing code from contrib/ and calling
those APIs in libgit.a instead. I like the idea.

This does delay the conversion of Scalar out of contrib, which is worth
thinking about.

> What's Scalar's future?
> =======================
> 
> With the completion of the tasks listed in the previous section, there's
> nothing left from the original Scalar CLI (in the Microsoft fork of Git) to
> upstream. I don't consider that the "end" of supporting Scalar because, for
> it to remain an effective tool, it'll need to stay up-to-date with the
> latest performance features introduced to Git.
> 
> For example, one possible future extension might be to have Scalar enable
> the sparse index by default, especially when more built-ins are updated with
> sparse index compatibility. I'm interested in hearing what other features
> might fit well there!

I did a double-take, "We're not doing that already?" Of course, we enabled
the sparse-index by default across all of microsoft/git. We did that when
microsoft/git had early versions of the sparse index integrations that are
now in shipped versions of Git. It might be worth considering updating the
default to enable sparse index automatically for repos using cone-mode
sparse-checkout. (This is an alternative to using Scalar as that vector,
only because the sparse index might be stable enough to recommend it for
all of Git. It's also possible that we don't want to enable the sparse index
by default in Git proper until more integrations are complete.)
 
> What I'm looking for
> ====================
> 
> The two patches in this series revise existing documentation to match the
> description above. Please let me know if 1) those revisions match your
> understanding of the above, 2) they convey the intent clearly, and (most
> importantly) 3) they reflect a reasonable direction to take Scalar. And, of
> course, if there's something I missed, I welcome any and all feedback &
> ideas!

Thanks for the clear cover letter. I'll review the patches with this in
mind.

Thanks,
-Stolee

Re: [PATCH 2/2] scalar: convert README.md into a technical design doc

[Derrick Stolee]

On 6/29/2022 12:58 PM, Victoria Dye via GitGitGadget wrote:
> From: Victoria Dye <vdye@github.com>
> 
> Replace 'README.md' with 'technical/scalar.txt' (still in 'contrib/'). In
> addition to reformatting for asciidoc, elaborate on the background, purpose,
> and design choices that went into Scalar.
> 
> This document is intended to persist in the 'Documentation/technical/'
> directory after Scalar has been moved into the root of Git (out of

I wonder if it is a good idea to move this into Documentation/technical/ _now_,
so we can have this document as part of "core Git's roadmap" as much as we can
say that. Part of the roadmap is moving the Scalar code out of contrib/ and
finalizing how users (or distributors) can install the 'scalar' binary.

It can be helpful to include the details of what steps to take to compile and
test the 'scalar' executable. That documentation will then be updated when
Scalar moves out of contrib/.

> +Scalar
> +======
> +
> +Scalar is a built-in repository management tool that optimizes Git for use in

I think "built-in" is unnecessary here. It makes me think of Git builtins and
otherwise does not add much to the description.

> +large repositories. It accomplishes this by helping users to take advantage of
> +advanced performance features in Git. Unlike most other Git built-in commands,
> +Scalar is not executed as a subcommand of 'git'; rather, it is built as a
> +separate executable containing its own series of subcommands.

Good to have this distinction.

> +Scalar is a large enough project that it is being upstreamed incrementally,
> +living in `contrib/` until it is feature-complete. So far, the following patch
> +series have been accepted:
> +
> +- `scalar-the-beginning`: The initial patch series which sets up
> +  `contrib/scalar/` and populates it with a minimal `scalar` command that
> +  demonstrates the fundamental ideas.
> +
> +- `scalar-c-and-C`: The `scalar` command learns about two options that can be
> +  specified before the command, `-c <key>=<value>` and `-C <directory>`.
> +
> +- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.
> +
> +Roughly speaking (and subject to change), the following series are needed to
> +"finish" this initial version of Scalar:

I like how you have these two sections. As you move forward in this roadmap,
the diff to the document should look like you are _moving_ the sentence above.

> +- Finish Scalar features: Enable the built-in FSMonitor in Scalar
> +  enlistments and implement `scalar help`. At the end of this series, Scalar
> +  should be feature-complete from the perspective of a user.

Wow, we're so close!

> +- Generalize features not specific to Scalar: In the spirit of
> +  making Scalar configure only what is needed for large repo performance, move

(nit: Strange wrapping on the first line)

> +  common utilities into other parts of Git. Some of this will be internal-only,
> +  but one major change will be generalizing `scalar diagnose` for use with any
> +  Git repository.
> +
> +- Move Scalar to toplevel: Make `scalar` a built-in component of Git by

Here, I would say "included" instead of built-in. Or is there another term we
should use to avoid confusion with "git <cmd>" builtins.

> +  moving it out of `contrib/` and into the root of `git`. The actual change will
> +  be relatively small, but this series will also contain expanded testing to
> +  ensure Scalar is stable and performant.

Perhaps "This change will also integrate Scalar into Git's test suite and CI
checks."

You mention "performant" which makes me think that performance tests are intended
to be part of this change. It makes me think it would be interesting to have our
existing performance tests create a mode where they compare a "vanilla" Git repo
to one registered with Scalar, but otherwise runs the same tests already in the
t/perf/ test scripts. This is a wide aside so feel free to ignore me.

> +Finally, there are two additional patch series that exist in Microsoft's fork of
> +Git, but there is no current plan to upstream them. There are some interesting
> +ideas there, but the implementation is too specific to Azure Repos and/or VFS
> +for Git to be of much help in general.
> +
> +These still exist mainly because the GVFS protocol is what Azure Repos has
> +instead of partial clone, while Git is focused on improving partial clone:
> +
> +- `scalar-with-gvfs`: The primary purpose of this patch series is to support
> +  existing Scalar users whose repositories are hosted in Azure Repos (which does
> +  not support Git's partial clones, but supports its predecessor, the GVFS
> +  protocol, which is used by Scalar to emulate the partial clone).
> +
> +  Since the GVFS protocol will never be supported by core Git, this patch series
> +  will remain in Microsoft's fork of Git.
> +
> +- `run-scalar-functional-tests`: The Scalar project developed a quite
> +  comprehensive set of integration tests (or, "Functional Tests"). They are the
> +  sole remaining part of the original C#-based Scalar project, and this patch
> +  adds a GitHub workflow that runs them all.
> +
> +  Since the tests partially depend on features that are only provided in the
> +  `scalar-with-gvfs` patch series, this patch cannot be upstreamed.

We've had a good track record of identifying when an upstream change causes a
failure in the Scalar Functional Tests and then porting that test into the Git test
suite. I expect that to continue, but it has also not happened in a while. This is
probably because the test coverage now has a significant amount of overlap. Many
of the tests were created specifically for VFS for Git and strangeness around the
virtual filesystem layer. Several of those have found interesting corner cases in
Git's sparse-checkout feature, but these have probably been resolved by now.

Thanks for doing this work, Victoria. I'm excited to see how the project progresses
with your new plan.

Thanks,
-Stolee

Re: [PATCH 0/2] [RFC] scalar: prepare documentation for future work

[Johannes Schindelin]

Hi Victoria,

thank you for an excellent example of a well-crafted cover letter.

Also, thank you for your fresh perspective, I believe you raised a valid
point when you pointed out that there are better ways to describe Scalar's
intention than the term "opinionated".

On Wed, 29 Jun 2022, Victoria Dye via GitGitGadget wrote:

> A plan for Scalar
> =================
>
> Given the slightly tweaked "philosophy" above, my ultimate goal for Scalar
> is to have it contain only what is too experimental or too large
> repo-focused to be a default option or behavior in Git. Over time, some
> features may be moved out of Scalar and into Git defaults as they are proven
> stable and beneficial to the vast majority of users [4].
>
> So what do we need to get there?
>
> At a high level, the remaining work to "finalize" Scalar (past this RFC) can
> be broken into three parts:
>
>  1. Complete a few more features and subcommands of Scalar (integrate with
>     the built-in FSMonitor & implement scalar help).
>  2. Move stable, general-purpose parts of 'scalar.c' into other Git
>     builtins/libraries (mainly scalar diagnose, either as part of git
>     bug-report [5] or a new git built-in).
>  3. Move Scalar out of contrib/ and into the "top-level" of Git. Includes
>     expanded testing, especially performance testing.
>
> The first makes scalar "feature complete" enough to be valuable to large
> repo users (per my entirely subjective assessment, at least). The second
> brings it in line with the goal of making Scalar only contain what can't
> exist as a default feature of Git. Once those are finished, I think Scalar
> will be out of its "work-in-progress" phase and ready to use as a built-in
> component of Git (accompanied by sufficient testing, of course).

This plan makes a ton of sense to me.

Earlier, I tried to skip step 2, but you are absolutely correct that it
would benefit the Git project much more if it wasn't skipped.

Thank you for working on this!
Dscho

Re: [PATCH 2/2] scalar: convert README.md into a technical design doc

[Victoria Dye]

Derrick Stolee wrote:
> On 6/29/2022 12:58 PM, Victoria Dye via GitGitGadget wrote:
>> From: Victoria Dye <vdye@github.com>
>>
>> Replace 'README.md' with 'technical/scalar.txt' (still in 'contrib/'). In
>> addition to reformatting for asciidoc, elaborate on the background, purpose,
>> and design choices that went into Scalar.
>>
>> This document is intended to persist in the 'Documentation/technical/'
>> directory after Scalar has been moved into the root of Git (out of
> 
> I wonder if it is a good idea to move this into Documentation/technical/ _now_,
> so we can have this document as part of "core Git's roadmap" as much as we can
> say that. Part of the roadmap is moving the Scalar code out of contrib/ and
> finalizing how users (or distributors) can install the 'scalar' binary.
> 

I initially hesitated in doing this because I felt it was a bit premature,
but your comment prompted me to look more closely at other docs in
'Documentation/technical'. Precedent seems to indicate that the doc should
be added *before* implementation (either in the same series as the
implementation [1] or separately [2]), so I'll do the same here. Thanks for
the suggestion!

[1] https://lore.kernel.org/git/daa9a6bcefbce977606484d502f5cfd7fca63ecc.1614111270.git.gitgitgadget@gmail.com/
[2] https://lore.kernel.org/git/20171208192636.13678-1-git@jeffhostetler.com/

> It can be helpful to include the details of what steps to take to compile and
> test the 'scalar' executable. That documentation will then be updated when
> Scalar moves out of contrib/.
> 

As part of the move out of 'contrib/', I was planning on having Scalar built
and installed the same as any built-in (albeit in 'bin/' - like 'gitk',
'git-cvsserver', etc. - rather than 'libexec/git-core'). In that case, there
won't be any special steps needed to build/install 'scalar', so any
instructions here would be temporary. I could include those instructions in
the meantime, but with Scalar incomplete, I'm not sure whether that would be
valuable.

All that said, if later reviews/iterations lead to Scalar needing special
steps to build/install, I'll definitely include them here.

>> +Scalar
>> +======
>> +
>> +Scalar is a built-in repository management tool that optimizes Git for use in
> 
> I think "built-in" is unnecessary here. It makes me think of Git builtins and
> otherwise does not add much to the description.
> 
>> +large repositories. It accomplishes this by helping users to take advantage of
>> +advanced performance features in Git. Unlike most other Git built-in commands,
>> +Scalar is not executed as a subcommand of 'git'; rather, it is built as a
>> +separate executable containing its own series of subcommands.
> 
> Good to have this distinction.
> 
>> +Scalar is a large enough project that it is being upstreamed incrementally,
>> +living in `contrib/` until it is feature-complete. So far, the following patch
>> +series have been accepted:
>> +
>> +- `scalar-the-beginning`: The initial patch series which sets up
>> +  `contrib/scalar/` and populates it with a minimal `scalar` command that
>> +  demonstrates the fundamental ideas.
>> +
>> +- `scalar-c-and-C`: The `scalar` command learns about two options that can be
>> +  specified before the command, `-c <key>=<value>` and `-C <directory>`.
>> +
>> +- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.
>> +
>> +Roughly speaking (and subject to change), the following series are needed to
>> +"finish" this initial version of Scalar:
> 
> I like how you have these two sections. As you move forward in this roadmap,
> the diff to the document should look like you are _moving_ the sentence above.
> 
>> +- Finish Scalar features: Enable the built-in FSMonitor in Scalar
>> +  enlistments and implement `scalar help`. At the end of this series, Scalar
>> +  should be feature-complete from the perspective of a user.
> 
> Wow, we're so close!
> 
>> +- Generalize features not specific to Scalar: In the spirit of
>> +  making Scalar configure only what is needed for large repo performance, move
> 
> (nit: Strange wrapping on the first line)
> 
>> +  common utilities into other parts of Git. Some of this will be internal-only,
>> +  but one major change will be generalizing `scalar diagnose` for use with any
>> +  Git repository.
>> +
>> +- Move Scalar to toplevel: Make `scalar` a built-in component of Git by
> 
> Here, I would say "included" instead of built-in. Or is there another term we
> should use to avoid confusion with "git <cmd>" builtins.
> 
>> +  moving it out of `contrib/` and into the root of `git`. The actual change will
>> +  be relatively small, but this series will also contain expanded testing to
>> +  ensure Scalar is stable and performant.
> 
> Perhaps "This change will also integrate Scalar into Git's test suite and CI
> checks."
> 
> You mention "performant" which makes me think that performance tests are intended
> to be part of this change. It makes me think it would be interesting to have our
> existing performance tests create a mode where they compare a "vanilla" Git repo
> to one registered with Scalar, but otherwise runs the same tests already in the
> t/perf/ test scripts. This is a wide aside so feel free to ignore me.
> 

This is a really interesting idea! My original plan was to add some basic
tests around the operations 'scalar' should (directly or indirectly) speed
up. I think I'll still need to do that anyway (e.g., for things like 'scalar
clone' vs 'git clone'), but I'll also try to find a (repeatable) way to
compare standard repo vs. Scalar enlistment performance in the existing perf
tests.

>> +Finally, there are two additional patch series that exist in Microsoft's fork of
>> +Git, but there is no current plan to upstream them. There are some interesting
>> +ideas there, but the implementation is too specific to Azure Repos and/or VFS
>> +for Git to be of much help in general.
>> +
>> +These still exist mainly because the GVFS protocol is what Azure Repos has
>> +instead of partial clone, while Git is focused on improving partial clone:
>> +
>> +- `scalar-with-gvfs`: The primary purpose of this patch series is to support
>> +  existing Scalar users whose repositories are hosted in Azure Repos (which does
>> +  not support Git's partial clones, but supports its predecessor, the GVFS
>> +  protocol, which is used by Scalar to emulate the partial clone).
>> +
>> +  Since the GVFS protocol will never be supported by core Git, this patch series
>> +  will remain in Microsoft's fork of Git.
>> +
>> +- `run-scalar-functional-tests`: The Scalar project developed a quite
>> +  comprehensive set of integration tests (or, "Functional Tests"). They are the
>> +  sole remaining part of the original C#-based Scalar project, and this patch
>> +  adds a GitHub workflow that runs them all.
>> +
>> +  Since the tests partially depend on features that are only provided in the
>> +  `scalar-with-gvfs` patch series, this patch cannot be upstreamed.
> 
> We've had a good track record of identifying when an upstream change causes a
> failure in the Scalar Functional Tests and then porting that test into the Git test
> suite. I expect that to continue, but it has also not happened in a while. This is
> probably because the test coverage now has a significant amount of overlap. Many
> of the tests were created specifically for VFS for Git and strangeness around the
> virtual filesystem layer. Several of those have found interesting corner cases in
> Git's sparse-checkout feature, but these have probably been resolved by now.
> 
> Thanks for doing this work, Victoria. I'm excited to see how the project progresses
> with your new plan.
> 
> Thanks,
> -Stolee

[PATCH v2 0/2] scalar: prepare documentation for future work

[Victoria Dye via GitGitGadget]

Changes since V1
================

 * Moved 'contrib/scalar/technical/scalar.txt' to
   'Documentation/technical/scalar.txt'.
 * Rephrased/removed references to Scalar being "built-in" to avoid
   confusion with the 'git ' built-ins.
 * Fixed line wrapping of bullet points in "Roadmap".
 * Removed the "[RFC]" tag.


What's this series?
===================

The questions of "what is Scalar?" and "why is Scalar part of Git?" have
been discussed at length already [1], but in preparation for the next few
series I plan to submit "finishing" the initial Scalar implementation
effort, I'd like to start by reframing its purpose.

Beginning with the "Philosophy of Scalar" [2], much of the existing
documentation refers to Scalar as "opinionated" - not unlike git itself.
However, with scalar now part of Git, continuing to describe it as
"opinionated" could unintentionally portray it as a divergence from "the
rest of Git" and its development community.

Rather than think of Scalar as an opinionated take on Git, I prefer to think
of it as a manifestation of Git's own "opinions" for a specific use case,
like many of the built-in commands. Over the years, developers have
contributed tons of features - often gated behind config settings - that
address performance issues they've encountered working in large
repositories: untracked cache, sparse index, merge-ort, etc. Some of these
features (like merge-ort) provide near-universal benefit to users and become
part of Git's default configuration. More often, though, these features are
large and difficult to prove are stable, or require some advanced manual
configuration to use properly, or simply make the "average" Git user's
experience worse (e.g., improving performance in in large repos, but
degrading it in smaller repos).

We've accumulated so many of these features that it's become difficult for
users to even find all of them, let alone use them. Git "knows" these
features exist, but also "knows" not to enable them by default because
they're either too experimental or not helpful to most users. Scalar is a
way for Git to take all of these features, bundle them together, and tell a
user, "Hey, do you need to make your large repo faster? Use this!" without
touching the default user experience.

All that to say, Scalar is essentially a collection of porcelain commands
[3] that fill a substantial UX niche in Git. Now, let's get into Scalar's
"future state".


A plan for Scalar
=================

Given the slightly tweaked "philosophy" above, my ultimate goal for Scalar
is to have it contain only what is too experimental or too large
repo-focused to be a default option or behavior in Git. Over time, some
features may be moved out of Scalar and into Git defaults as they are proven
stable and beneficial to the vast majority of users [4].

So what do we need to get there?

At a high level, the remaining work to "finalize" Scalar (past this RFC) can
be broken into three parts:

 1. Complete a few more features and subcommands of Scalar (integrate with
    the built-in FSMonitor & implement scalar help).
 2. Move stable, general-purpose parts of 'scalar.c' into other Git
    builtins/libraries (mainly scalar diagnose, either as part of git
    bug-report [5] or a new git built-in).
 3. Move Scalar out of contrib/ and into the "top-level" of Git. Includes
    expanded testing, especially performance testing.

The first makes scalar "feature complete" enough to be valuable to large
repo users (per my entirely subjective assessment, at least). The second
brings it in line with the goal of making Scalar only contain what can't
exist as a default feature of Git. Once those are finished, I think Scalar
will be out of its "work-in-progress" phase and ready to use as a built-in
component of Git (accompanied by sufficient testing, of course).


What's Scalar's future?
=======================

With the completion of the tasks listed in the previous section, there's
nothing left from the original Scalar CLI (in the Microsoft fork of Git) to
upstream. I don't consider that the "end" of supporting Scalar because, for
it to remain an effective tool, it'll need to stay up-to-date with the
latest performance features introduced to Git.

For example, one possible future extension might be to have Scalar enable
the sparse index by default, especially when more built-ins are updated with
sparse index compatibility. I'm interested in hearing what other features
might fit well there!


What I'm looking for
====================

The two patches in this series revise existing documentation to match the
description above. Please let me know if 1) those revisions match your
understanding of the above, 2) they convey the intent clearly, and (most
importantly) 3) they reflect a reasonable direction to take Scalar. And, of
course, if there's something I missed, I welcome any and all feedback &
ideas!

Thanks,

 * Victoria

[1]
https://lore.kernel.org/git/b67bbef4-e4c3-b6a7-1c7f-7d405902ef8b@gmail.com/

[2]
https://github.com/microsoft/git/blob/173a9b783a414/contrib/scalar/docs/philosophy.md

[3] If you're wondering "why not call it git-scalar then?", my short answer
is 1) backward compatibility with the original "scalar" CLI, and 2) its
scope is larger and more persistent than a single command; Scalar works
across multiple repos, and each Scalar subcommand has scope and
functionality on par with a built-in command. I'm happy to talk through this
more if people are still unsure, though!

[4] That being said, Scalar's existence should not make Git more risk averse
in accepting new general-purpose features.

[5] https://lore.kernel.org/git/220517.86y1z05gja.gmgdl@evledraar.gmail.com/

Victoria Dye (2):
  scalar: reword command documentation to clarify purpose
  scalar: convert README.md into a technical design doc

 Documentation/technical/scalar.txt | 127 +++++++++++++++++++++++++++++
 contrib/scalar/README.md           |  82 -------------------
 contrib/scalar/scalar.txt          |   9 +-
 3 files changed, 131 insertions(+), 87 deletions(-)
 create mode 100644 Documentation/technical/scalar.txt
 delete mode 100644 contrib/scalar/README.md


base-commit: e4a4b31577c7419497ac30cebe30d755b97752c5
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1275%2Fvdye%2Fscalar%2Fdoc-update-rfc-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1275/vdye/scalar/doc-update-rfc-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/1275

Range-diff vs v1:

 1:  38c0b4dff90 = 1:  38c0b4dff90 scalar: reword command documentation to clarify purpose
 2:  870bd90e47e ! 2:  65eda9755d2 scalar: convert README.md into a technical design doc
     @@ Metadata
       ## Commit message ##
          scalar: convert README.md into a technical design doc
      
     -    Replace 'README.md' with 'technical/scalar.txt' (still in 'contrib/'). In
     -    addition to reformatting for asciidoc, elaborate on the background, purpose,
     -    and design choices that went into Scalar.
     +    Adapt the content from 'contrib/scalar/README.md' into a design document in
     +    'Documentation/technical/'. In addition to reformatting for asciidoc,
     +    elaborate on the background, purpose, and design choices that went into
     +    Scalar.
      
     -    This document is intended to persist in the 'Documentation/technical/'
     -    directory after Scalar has been moved into the root of Git (out of
     -    'contrib/'). Before then, it will also contain a "Roadmap" section detailing
     -    the remaining series needed to finish the initial version of Scalar. The
     -    section will be removed once Scalar is moved to the repo root, but in the
     -    meantime serves as a guide for readers to keep up with progress on the
     -    feature.
     +    Most of this document will persist in the 'Documentation/technical/' after
     +    Scalar has been moved out of 'contrib/' and into the root of Git. Until that
     +    time, it will also contain a temporary "Roadmap" section detailing the
     +    remaining series needed to finish the initial version of Scalar. The section
     +    will be removed once Scalar is moved to the repo root, but in the meantime
     +    serves as a guide for readers to keep up with progress on the feature.
      
          Signed-off-by: Victoria Dye <vdye@github.com>
      
     - ## contrib/scalar/README.md (deleted) ##
     -@@
     --# Scalar - an opinionated repository management tool
     --
     --Scalar is an add-on to Git that helps users take advantage of advanced
     --performance features in Git. Originally implemented in C# using .NET Core,
     --based on the learnings from the VFS for Git project, most of the techniques
     --developed by the Scalar project have been integrated into core Git already:
     --
     --* partial clone,
     --* commit graphs,
     --* multi-pack index,
     --* sparse checkout (cone mode),
     --* scheduled background maintenance,
     --* etc
     --
     --This directory contains the remaining parts of Scalar that are not (yet) in
     --core Git.
     --
     --## Roadmap
     --
     --The idea is to populate this directory via incremental patch series and
     --eventually move to a top-level directory next to `gitk-git/` and to `git-gui/`. The
     --current plan involves the following patch series:
     --
     --- `scalar-the-beginning`: The initial patch series which sets up
     --  `contrib/scalar/` and populates it with a minimal `scalar` command that
     --  demonstrates the fundamental ideas.
     --
     --- `scalar-c-and-C`: The `scalar` command learns about two options that can be
     --  specified before the command, `-c <key>=<value>` and `-C <directory>`.
     --
     --- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.
     --
     --- `scalar-and-builtin-fsmonitor`: The built-in FSMonitor is enabled in `scalar
     --  register` and in `scalar clone`, for an enormous performance boost when
     --  working in large worktrees. This patch series necessarily depends on Jeff
     --  Hostetler's FSMonitor patch series to be integrated into Git.
     --
     --- `scalar-gentler-config-locking`: Scalar enlistments are registered in the
     --  user's Git config. This usually does not represent any problem because it is
     --  rare for a user to register an enlistment. However, in Scalar's functional
     --  tests, Scalar enlistments are created galore, and in parallel, which can lead
     --  to lock contention. This patch series works around that problem by re-trying
     --  to lock the config file in a gentle fashion.
     --
     --- `scalar-extra-docs`: Add some extensive documentation that has been written
     --  in the original Scalar project (all subject to discussion, of course).
     --
     --- `optionally-install-scalar`: Now that Scalar is feature (and documentation)
     --  complete and is verified in CI builds, let's offer to install it.
     --
     --- `move-scalar-to-toplevel`: Now that Scalar is complete, let's move it next to
     --  `gitk-git/` and to `git-gui/`, making it a top-level command.
     --
     --The following two patch series exist in Microsoft's fork of Git and are
     --publicly available. There is no current plan to upstream them, not because I
     --want to withhold these patches, but because I don't think the Git community is
     --interested in these patches.
     --
     --There are some interesting ideas there, but the implementation is too specific
     --to Azure Repos and/or VFS for Git to be of much help in general (and also: my
     --colleagues tried to upstream some patches already and the enthusiasm for
     --integrating things related to Azure Repos and VFS for Git can be summarized in
     --very, very few words).
     --
     --These still exist mainly because the GVFS protocol is what Azure Repos has
     --instead of partial clone, while Git is focused on improving partial clone:
     --
     --- `scalar-with-gvfs`: The primary purpose of this patch series is to support
     --  existing Scalar users whose repositories are hosted in Azure Repos (which
     --  does not support Git's partial clones, but supports its predecessor, the GVFS
     --  protocol, which is used by Scalar to emulate the partial clone).
     --
     --  Since the GVFS protocol will never be supported by core Git, this patch
     --  series will remain in Microsoft's fork of Git.
     --
     --- `run-scalar-functional-tests`: The Scalar project developed a quite
     --  comprehensive set of integration tests (or, "Functional Tests"). They are the
     --  sole remaining part of the original C#-based Scalar project, and this patch
     --  adds a GitHub workflow that runs them all.
     --
     --  Since the tests partially depend on features that are only provided in the
     --  `scalar-with-gvfs` patch series, this patch cannot be upstreamed.
     -
     - ## contrib/scalar/technical/scalar.txt (new) ##
     + ## Documentation/technical/scalar.txt (new) ##
      @@
      +Scalar
      +======
      +
     -+Scalar is a built-in repository management tool that optimizes Git for use in
     -+large repositories. It accomplishes this by helping users to take advantage of
     ++Scalar is a repository management tool that optimizes Git for use in large
     ++repositories. It accomplishes this by helping users to take advantage of
      +advanced performance features in Git. Unlike most other Git built-in commands,
      +Scalar is not executed as a subcommand of 'git'; rather, it is built as a
      +separate executable containing its own series of subcommands.
     @@ contrib/scalar/technical/scalar.txt (new)
      +Roughly speaking (and subject to change), the following series are needed to
      +"finish" this initial version of Scalar:
      +
     -+- Finish Scalar features: Enable the built-in FSMonitor in Scalar
     -+  enlistments and implement `scalar help`. At the end of this series, Scalar
     -+  should be feature-complete from the perspective of a user.
     ++- Finish Scalar features: Enable the built-in FSMonitor in Scalar enlistments
     ++  and implement `scalar help`. At the end of this series, Scalar should be
     ++  feature-complete from the perspective of a user.
      +
     -+- Generalize features not specific to Scalar: In the spirit of
     -+  making Scalar configure only what is needed for large repo performance, move
     -+  common utilities into other parts of Git. Some of this will be internal-only,
     -+  but one major change will be generalizing `scalar diagnose` for use with any
     -+  Git repository.
     ++- Generalize features not specific to Scalar: In the spirit of making Scalar
     ++  configure only what is needed for large repo performance, move common
     ++  utilities into other parts of Git. Some of this will be internal-only, but one
     ++  major change will be generalizing `scalar diagnose` for use with any Git
     ++  repository.
      +
     -+- Move Scalar to toplevel: Make `scalar` a built-in component of Git by
     -+  moving it out of `contrib/` and into the root of `git`. The actual change will
     -+  be relatively small, but this series will also contain expanded testing to
     -+  ensure Scalar is stable and performant.
     ++- Move Scalar to toplevel: Move Scalar out of `contrib/` and into the root of
     ++  `git`, including updates to build and install it with the rest of Git. This
     ++  change will incorporate Scalar into the Git CI and test framework, as well as
     ++  expand regression and performance testing to ensure the tool is stable.
      +
      +Finally, there are two additional patch series that exist in Microsoft's fork of
      +Git, but there is no current plan to upstream them. There are some interesting
     @@ contrib/scalar/technical/scalar.txt (new)
      +
      +  Since the tests partially depend on features that are only provided in the
      +  `scalar-with-gvfs` patch series, this patch cannot be upstreamed.
     +
     + ## contrib/scalar/README.md (deleted) ##
     +@@
     +-# Scalar - an opinionated repository management tool
     +-
     +-Scalar is an add-on to Git that helps users take advantage of advanced
     +-performance features in Git. Originally implemented in C# using .NET Core,
     +-based on the learnings from the VFS for Git project, most of the techniques
     +-developed by the Scalar project have been integrated into core Git already:
     +-
     +-* partial clone,
     +-* commit graphs,
     +-* multi-pack index,
     +-* sparse checkout (cone mode),
     +-* scheduled background maintenance,
     +-* etc
     +-
     +-This directory contains the remaining parts of Scalar that are not (yet) in
     +-core Git.
     +-
     +-## Roadmap
     +-
     +-The idea is to populate this directory via incremental patch series and
     +-eventually move to a top-level directory next to `gitk-git/` and to `git-gui/`. The
     +-current plan involves the following patch series:
     +-
     +-- `scalar-the-beginning`: The initial patch series which sets up
     +-  `contrib/scalar/` and populates it with a minimal `scalar` command that
     +-  demonstrates the fundamental ideas.
     +-
     +-- `scalar-c-and-C`: The `scalar` command learns about two options that can be
     +-  specified before the command, `-c <key>=<value>` and `-C <directory>`.
     +-
     +-- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.
     +-
     +-- `scalar-and-builtin-fsmonitor`: The built-in FSMonitor is enabled in `scalar
     +-  register` and in `scalar clone`, for an enormous performance boost when
     +-  working in large worktrees. This patch series necessarily depends on Jeff
     +-  Hostetler's FSMonitor patch series to be integrated into Git.
     +-
     +-- `scalar-gentler-config-locking`: Scalar enlistments are registered in the
     +-  user's Git config. This usually does not represent any problem because it is
     +-  rare for a user to register an enlistment. However, in Scalar's functional
     +-  tests, Scalar enlistments are created galore, and in parallel, which can lead
     +-  to lock contention. This patch series works around that problem by re-trying
     +-  to lock the config file in a gentle fashion.
     +-
     +-- `scalar-extra-docs`: Add some extensive documentation that has been written
     +-  in the original Scalar project (all subject to discussion, of course).
     +-
     +-- `optionally-install-scalar`: Now that Scalar is feature (and documentation)
     +-  complete and is verified in CI builds, let's offer to install it.
     +-
     +-- `move-scalar-to-toplevel`: Now that Scalar is complete, let's move it next to
     +-  `gitk-git/` and to `git-gui/`, making it a top-level command.
     +-
     +-The following two patch series exist in Microsoft's fork of Git and are
     +-publicly available. There is no current plan to upstream them, not because I
     +-want to withhold these patches, but because I don't think the Git community is
     +-interested in these patches.
     +-
     +-There are some interesting ideas there, but the implementation is too specific
     +-to Azure Repos and/or VFS for Git to be of much help in general (and also: my
     +-colleagues tried to upstream some patches already and the enthusiasm for
     +-integrating things related to Azure Repos and VFS for Git can be summarized in
     +-very, very few words).
     +-
     +-These still exist mainly because the GVFS protocol is what Azure Repos has
     +-instead of partial clone, while Git is focused on improving partial clone:
     +-
     +-- `scalar-with-gvfs`: The primary purpose of this patch series is to support
     +-  existing Scalar users whose repositories are hosted in Azure Repos (which
     +-  does not support Git's partial clones, but supports its predecessor, the GVFS
     +-  protocol, which is used by Scalar to emulate the partial clone).
     +-
     +-  Since the GVFS protocol will never be supported by core Git, this patch
     +-  series will remain in Microsoft's fork of Git.
     +-
     +-- `run-scalar-functional-tests`: The Scalar project developed a quite
     +-  comprehensive set of integration tests (or, "Functional Tests"). They are the
     +-  sole remaining part of the original C#-based Scalar project, and this patch
     +-  adds a GitHub workflow that runs them all.
     +-
     +-  Since the tests partially depend on features that are only provided in the
     +-  `scalar-with-gvfs` patch series, this patch cannot be upstreamed.

-- 
gitgitgadget

[PATCH v2 1/2] scalar: reword command documentation to clarify purpose

[Victoria Dye via GitGitGadget]

From: Victoria Dye <vdye@github.com>

Rephrase documentation to describe scalar as a "large repo management tool"
rather than an "opinionated management tool". The new description is
intended to more directly reflect the utility of scalar to better guide
users in preparation for scalar being built and installed as part of Git.

Signed-off-by: Victoria Dye <vdye@github.com>
---
 contrib/scalar/scalar.txt | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index c0425e06533..1a12dc45077 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -3,7 +3,7 @@ scalar(1)
 
 NAME
 ----
-scalar - an opinionated repository management tool
+scalar - A tool for managing large Git repositories
 
 SYNOPSIS
 --------
@@ -20,10 +20,9 @@ scalar delete <enlistment>
 DESCRIPTION
 -----------
 
-Scalar is an opinionated repository management tool. By creating new
-repositories or registering existing repositories with Scalar, your Git
-experience will speed up. Scalar sets advanced Git config settings,
-maintains your repositories in the background, and helps reduce data sent
+Scalar is a repository management tool that optimizes Git for use in large
+repositories. Scalar improves performance by configuring advanced Git settings,
+maintaining repositories in the background, and helping to reduce data sent
 across the network.
 
 An important Scalar concept is the enlistment: this is the top-level directory
-- 
gitgitgadget

[PATCH v2 2/2] scalar: convert README.md into a technical design doc

[Victoria Dye via GitGitGadget]

From: Victoria Dye <vdye@github.com>

Adapt the content from 'contrib/scalar/README.md' into a design document in
'Documentation/technical/'. In addition to reformatting for asciidoc,
elaborate on the background, purpose, and design choices that went into
Scalar.

Most of this document will persist in the 'Documentation/technical/' after
Scalar has been moved out of 'contrib/' and into the root of Git. Until that
time, it will also contain a temporary "Roadmap" section detailing the
remaining series needed to finish the initial version of Scalar. The section
will be removed once Scalar is moved to the repo root, but in the meantime
serves as a guide for readers to keep up with progress on the feature.

Signed-off-by: Victoria Dye <vdye@github.com>
---
 Documentation/technical/scalar.txt | 127 +++++++++++++++++++++++++++++
 contrib/scalar/README.md           |  82 -------------------
 2 files changed, 127 insertions(+), 82 deletions(-)
 create mode 100644 Documentation/technical/scalar.txt
 delete mode 100644 contrib/scalar/README.md

diff --git a/Documentation/technical/scalar.txt b/Documentation/technical/scalar.txt
new file mode 100644
index 00000000000..08bc09c225a
--- /dev/null
+++ b/Documentation/technical/scalar.txt
@@ -0,0 +1,127 @@
+Scalar
+======
+
+Scalar is a repository management tool that optimizes Git for use in large
+repositories. It accomplishes this by helping users to take advantage of
+advanced performance features in Git. Unlike most other Git built-in commands,
+Scalar is not executed as a subcommand of 'git'; rather, it is built as a
+separate executable containing its own series of subcommands.
+
+Background
+----------
+
+Scalar was originally designed as an add-on to Git and implemented as a .NET
+Core application. It was created based on the learnings from the VFS for Git
+project (another application aimed at improving the experience of working with
+large repositories). As part of its initial implementation, Scalar relied on
+custom features in the Microsoft fork of Git that have since been integrated
+into core Git:
+
+* partial clone,
+* commit graphs,
+* multi-pack index,
+* sparse checkout (cone mode),
+* scheduled background maintenance,
+* etc
+
+With the requisite Git functionality in place and a desire to bring the benefits
+of Scalar to the larger Git community, the Scalar application itself was ported
+from C# to C and integrated upstream.
+
+Features
+--------
+
+Scalar is comprised of two major pieces of functionality: automatically
+configuring built-in Git performance features and managing repository
+enlistments.
+
+The Git performance features configured by Scalar (see "Background" for
+examples) confer substantial performance benefits to large repositories, but are
+either too experimental to enable for all of Git yet, or only benefit large
+repositories. As new features are introduced, Scalar should be updated
+accordingly to incorporate them. This will prevent the tool from becoming stale
+while also providing a path for more easily bringing features to the appropriate
+users.
+
+Enlistments are how Scalar knows which repositories on a user's system should
+utilize Scalar-configured features. This allows it to update performance
+settings when new ones are added to the tool, as well as centrally manage
+repository maintenance. The enlistment structure - a root directory with a
+`src/` subdirectory containing the cloned repository itself - is designed to
+encourage users to route build outputs outside of the repository to avoid the
+performance-limiting overhead of ignoring those files in Git.
+
+Design
+------
+
+Scalar is implemented in C and interacts with Git via a mix of child process
+invocations of Git and direct usage of `libgit.a`. Internally, it is structured
+much like other built-ins with subcommands (e.g., `git stash`), containing a
+`cmd_<subcommand>()` function for each subcommand, routed through a `cmd_main()`
+function. Most options are unique to each subcommand, with `scalar` respecting
+some "global" `git` options (e.g., `-c` and `-C`).
+
+Because `scalar` is not invoked as a Git subcommand (like `git scalar`), it is
+built and installed as its own executable in the `bin/` directory, alongside
+`git`, `git-gui`, etc.
+
+Roadmap
+-------
+
+NOTE: this section will be removed once the remaining tasks outlined in this
+roadmap are complete.
+
+Scalar is a large enough project that it is being upstreamed incrementally,
+living in `contrib/` until it is feature-complete. So far, the following patch
+series have been accepted:
+
+- `scalar-the-beginning`: The initial patch series which sets up
+  `contrib/scalar/` and populates it with a minimal `scalar` command that
+  demonstrates the fundamental ideas.
+
+- `scalar-c-and-C`: The `scalar` command learns about two options that can be
+  specified before the command, `-c <key>=<value>` and `-C <directory>`.
+
+- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.
+
+Roughly speaking (and subject to change), the following series are needed to
+"finish" this initial version of Scalar:
+
+- Finish Scalar features: Enable the built-in FSMonitor in Scalar enlistments
+  and implement `scalar help`. At the end of this series, Scalar should be
+  feature-complete from the perspective of a user.
+
+- Generalize features not specific to Scalar: In the spirit of making Scalar
+  configure only what is needed for large repo performance, move common
+  utilities into other parts of Git. Some of this will be internal-only, but one
+  major change will be generalizing `scalar diagnose` for use with any Git
+  repository.
+
+- Move Scalar to toplevel: Move Scalar out of `contrib/` and into the root of
+  `git`, including updates to build and install it with the rest of Git. This
+  change will incorporate Scalar into the Git CI and test framework, as well as
+  expand regression and performance testing to ensure the tool is stable.
+
+Finally, there are two additional patch series that exist in Microsoft's fork of
+Git, but there is no current plan to upstream them. There are some interesting
+ideas there, but the implementation is too specific to Azure Repos and/or VFS
+for Git to be of much help in general.
+
+These still exist mainly because the GVFS protocol is what Azure Repos has
+instead of partial clone, while Git is focused on improving partial clone:
+
+- `scalar-with-gvfs`: The primary purpose of this patch series is to support
+  existing Scalar users whose repositories are hosted in Azure Repos (which does
+  not support Git's partial clones, but supports its predecessor, the GVFS
+  protocol, which is used by Scalar to emulate the partial clone).
+
+  Since the GVFS protocol will never be supported by core Git, this patch series
+  will remain in Microsoft's fork of Git.
+
+- `run-scalar-functional-tests`: The Scalar project developed a quite
+  comprehensive set of integration tests (or, "Functional Tests"). They are the
+  sole remaining part of the original C#-based Scalar project, and this patch
+  adds a GitHub workflow that runs them all.
+
+  Since the tests partially depend on features that are only provided in the
+  `scalar-with-gvfs` patch series, this patch cannot be upstreamed.
diff --git a/contrib/scalar/README.md b/contrib/scalar/README.md
deleted file mode 100644
index 634b5771ed3..00000000000
--- a/contrib/scalar/README.md
+++ /dev/null
@@ -1,82 +0,0 @@
-# Scalar - an opinionated repository management tool
-
-Scalar is an add-on to Git that helps users take advantage of advanced
-performance features in Git. Originally implemented in C# using .NET Core,
-based on the learnings from the VFS for Git project, most of the techniques
-developed by the Scalar project have been integrated into core Git already:
-
-* partial clone,
-* commit graphs,
-* multi-pack index,
-* sparse checkout (cone mode),
-* scheduled background maintenance,
-* etc
-
-This directory contains the remaining parts of Scalar that are not (yet) in
-core Git.
-
-## Roadmap
-
-The idea is to populate this directory via incremental patch series and
-eventually move to a top-level directory next to `gitk-git/` and to `git-gui/`. The
-current plan involves the following patch series:
-
-- `scalar-the-beginning`: The initial patch series which sets up
-  `contrib/scalar/` and populates it with a minimal `scalar` command that
-  demonstrates the fundamental ideas.
-
-- `scalar-c-and-C`: The `scalar` command learns about two options that can be
-  specified before the command, `-c <key>=<value>` and `-C <directory>`.
-
-- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.
-
-- `scalar-and-builtin-fsmonitor`: The built-in FSMonitor is enabled in `scalar
-  register` and in `scalar clone`, for an enormous performance boost when
-  working in large worktrees. This patch series necessarily depends on Jeff
-  Hostetler's FSMonitor patch series to be integrated into Git.
-
-- `scalar-gentler-config-locking`: Scalar enlistments are registered in the
-  user's Git config. This usually does not represent any problem because it is
-  rare for a user to register an enlistment. However, in Scalar's functional
-  tests, Scalar enlistments are created galore, and in parallel, which can lead
-  to lock contention. This patch series works around that problem by re-trying
-  to lock the config file in a gentle fashion.
-
-- `scalar-extra-docs`: Add some extensive documentation that has been written
-  in the original Scalar project (all subject to discussion, of course).
-
-- `optionally-install-scalar`: Now that Scalar is feature (and documentation)
-  complete and is verified in CI builds, let's offer to install it.
-
-- `move-scalar-to-toplevel`: Now that Scalar is complete, let's move it next to
-  `gitk-git/` and to `git-gui/`, making it a top-level command.
-
-The following two patch series exist in Microsoft's fork of Git and are
-publicly available. There is no current plan to upstream them, not because I
-want to withhold these patches, but because I don't think the Git community is
-interested in these patches.
-
-There are some interesting ideas there, but the implementation is too specific
-to Azure Repos and/or VFS for Git to be of much help in general (and also: my
-colleagues tried to upstream some patches already and the enthusiasm for
-integrating things related to Azure Repos and VFS for Git can be summarized in
-very, very few words).
-
-These still exist mainly because the GVFS protocol is what Azure Repos has
-instead of partial clone, while Git is focused on improving partial clone:
-
-- `scalar-with-gvfs`: The primary purpose of this patch series is to support
-  existing Scalar users whose repositories are hosted in Azure Repos (which
-  does not support Git's partial clones, but supports its predecessor, the GVFS
-  protocol, which is used by Scalar to emulate the partial clone).
-
-  Since the GVFS protocol will never be supported by core Git, this patch
-  series will remain in Microsoft's fork of Git.
-
-- `run-scalar-functional-tests`: The Scalar project developed a quite
-  comprehensive set of integration tests (or, "Functional Tests"). They are the
-  sole remaining part of the original C#-based Scalar project, and this patch
-  adds a GitHub workflow that runs them all.
-
-  Since the tests partially depend on features that are only provided in the
-  `scalar-with-gvfs` patch series, this patch cannot be upstreamed.
-- 
gitgitgadget

Re: [PATCH 2/2] scalar: convert README.md into a technical design doc

[Derrick Stolee]

On 7/11/2022 7:05 PM, Victoria Dye wrote:
> Derrick Stolee wrote:
>> On 6/29/2022 12:58 PM, Victoria Dye via GitGitGadget wrote:
>>> From: Victoria Dye <vdye@github.com>

>> It can be helpful to include the details of what steps to take to compile and
>> test the 'scalar' executable. That documentation will then be updated when
>> Scalar moves out of contrib/.
>>
> 
> As part of the move out of 'contrib/', I was planning on having Scalar built
> and installed the same as any built-in (albeit in 'bin/' - like 'gitk',
> 'git-cvsserver', etc. - rather than 'libexec/git-core'). In that case, there
> won't be any special steps needed to build/install 'scalar', so any
> instructions here would be temporary. I could include those instructions in
> the meantime, but with Scalar incomplete, I'm not sure whether that would be
> valuable.

Ok, I think you don't need those extra steps if the plan is to compile and
test by default. I think we might want to consider the installation steps
and whether or not distributors will want to have an opt-in option for the
scalar binary at that point. Fine to leave that until later.

>> You mention "performant" which makes me think that performance tests are intended
>> to be part of this change. It makes me think it would be interesting to have our
>> existing performance tests create a mode where they compare a "vanilla" Git repo
>> to one registered with Scalar, but otherwise runs the same tests already in the
>> t/perf/ test scripts. This is a wide aside so feel free to ignore me.
>>
> 
> This is a really interesting idea! My original plan was to add some basic
> tests around the operations 'scalar' should (directly or indirectly) speed
> up. I think I'll still need to do that anyway (e.g., for things like 'scalar
> clone' vs 'git clone'), but I'll also try to find a (repeatable) way to
> compare standard repo vs. Scalar enlistment performance in the existing perf
> tests.

It's tricky since our performance tests don't clone across a network boundary,
but maybe we could create a new class of tests to operate against a Git server
specified by the tester. Definitely out of scope for this series.

Thanks,
-Stolee

Re: [PATCH v2 0/2] scalar: prepare documentation for future work

[Derrick Stolee]

On 7/11/2022 8:06 PM, Victoria Dye via GitGitGadget wrote:
> 
> Changes since V1
> ================
> 
>  * Moved 'contrib/scalar/technical/scalar.txt' to
>    'Documentation/technical/scalar.txt'.
>  * Rephrased/removed references to Scalar being "built-in" to avoid
>    confusion with the 'git ' built-ins.
>  * Fixed line wrapping of bullet points in "Roadmap".
>  * Removed the "[RFC]" tag.

Sorry I'm late to respond. I added a reply to the v1 discussion,
but this v2 looks good to me. I'm looking forward to seeing your
next steps.

Thanks!
-Stolee