Re: [PATCH v7 01/17] scalar: add a README with a roadmap

[Johannes Schindelin <Johannes.Schindelin@gmx.de>]

Hi Stolee,

On Wed, 17 Nov 2021, Derrick Stolee wrote:

> On 11/17/2021 9:19 AM, Johannes Schindelin via GitGitGadget wrote:
> > From: Johannes Schindelin <johannes.schindelin@gmx.de>
> >
> > The Scalar command will be contributed incrementally, over a bunch of
> > patch series. Let's document what Scalar is about, and then describe the
> > patch series that are planned.
> >
> > Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> > ---
> >  contrib/scalar/README.md | 71 ++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 71 insertions(+)
> >  create mode 100644 contrib/scalar/README.md
> >
> > diff --git a/contrib/scalar/README.md b/contrib/scalar/README.md
> > new file mode 100644
> > index 00000000000..7898a683ba5
> > --- /dev/null
> > +++ b/contrib/scalar/README.md
> > @@ -0,0 +1,71 @@
> > +# Scalar - an opinionated repository management tool
> > +
> > +Scalar is an add-on to Git, helping Git scale to very large repositories and
> > +worktrees.
>
> I would rephrase this as "Scalar is an add-on to Git that helps users take
> advantage of advanced performance features in Git."

Good idea!

> Git scales just fine, only it helps to enable some features that are off
> by default.
>
> > Originally implemented in C# using .NET Core, based on the learnings
> > +from the VFS for Git project, most of the techniques developed by the Scalar
> > +project have been integrated into core Git already:
> > +
> > +* partial clone,
> > +* commit graphs,
> > +* multi-pack index,
> > +* sparse checkout (cone mode),
> > +* scheduled background maintenance,
> > +* etc
> > +
> > +This directory contains the remaining parts of Scalar that are not (yet) in
> > +core Git.
> > +
> > +## Roadmap
> > +
> > +The idea is to populate this directory via incremental patch series and
> > +eventually move to a top-level directory next to `gitk-git/` and to `git-gui/`. The
> > +current plan involves the following patch series:
> > +
> > +- `scalar-the-beginning`: The initial patch series which sets up
> > +  `contrib/scalar/` and populates it with a minimal `scalar` command that
> > +  demonstrates the fundamental ideas.
> > +
> > +- `scalar-c-and-C`: The `scalar` command learns about two options that can be
> > +  specified before the command, `-c <key>=<value>` and `-C <directory>`.
> > +
> > +- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.
> > +
> > +- `scalar-and-builtin-fsmonitor`: The built-in FSMonitor is enabled in `scalar
> > +  init` and in `scalar clone`, for an enormous performance boost when working
> > +  in large worktrees. This patch series necessarily depends on Jeff Hostetler's
> > +  FSMonitor patch series to be integrated into Git.
>
> You say 'scalar init' but do you mean 'scalar register'?

D'oh. Yes, I meant `scalar register`. I was thinking of `git init` too
much when I wrote that.

> > +- `scalar-gentler-config-locking`: Scalar enlistments are registered in the
> > +  user's Git config. This usually does not represent any problem because it is
> > +  rare for a user to register an enlistment. However, in Scalar's functional
> > +  tests, Scalar enlistments are created galore, and in parallel, which can lead
> > +  to lock contention. This patch series works around that problem by re-trying
> > +  to lock the config file in a gentle fashion.
> > +
> > +- `scalar-extra-docs`: Add some extensive documentation that has been written
> > +  in the original Scalar project (all subject to discussion, of course).
> > +
> > +- `optionally-install-scalar`: Now that Scalar is feature (and documentation)
> > +  complete and is verified in CI builds, let's offer to install it.
> > +
> > +- `move-scalar-to-toplevel`: Now that Scalar is complete, let's move it next to
> > +  `gitk-git/` and to `git-gui/`, making it a top-level command.
>
> This final one is where we can make the final call about where Scalar should
> exist in the tree and how optional it should be. This would also move the
> Scalar man pages into Documentation/, along with possibly the docs from
> 'scalar-extra-docs', and the tests into t/. The benefit of leaving this until
> the end is that we can see the entirety of Scalar before making a final call.

Yes.

It allows the current patch series to focus on the core functionality of
Scalar. The `move-scalar-to-toplevel` patch series will present an
excellent opportunity to discuss the merits and complications of accepting
Scalar as a top-level command into Git, without having to delay the
current patch series any further.

> > +The following two patch series exist, but there is no plan to integrate them
> > +into Git's source tree:
> > +
> > +- `scalar-with-gvfs`: The primary purpose of this patch series is to support
> > +  existing Scalar users whose repositories are hosted in Azure Repos (which
> > +  does not support Git's partial clones, but supports its predecessor, the GVFS
> > +  protocol, which is used by Scalar to emulate the partial clone).
> > +
> > +  Since the GVFS protocol will never be supported by core Git, this patch
> > +  series will remain in Microsoft's fork of Git.
> > +
> > +- `run-scalar-functional-tests`: The Scalar project developed a quite
> > +  comprehensive set of integration tests (or, "Functional Tests"). They are the
> > +  sole remaining part of the original C#-based Scalar project, and this patch
> > +  adds a GitHub workflow that runs them all.
> > +
> > +  Since the tests partially depend on features that are only provided in the
> > +  `scalar-with-gvfs` patch series, this patch cannot be upstreamed.
>
> These topics (in some form or another) exist on microsoft/git and are available
> via GPL, so we don't intend to say "we are withholding these patches" but instead
> are saying "We don't think the Git community is interested in these patches."
> There are some interesting ideas there, but the implementation is too specific to
> Azure Repos to be of much help in general. These still exist mainly because the
> GVFS protocol is what Azure Repos has instead of partial clone. We are focused
> instead on improving partial clone.

Good point. What I wrote does not fully reflect that. I have changed it.

Do you see anything in the remainder of the patch series that still needs
to be improved?

Thanks,
Dscho