fixing workdirs

fixing workdirs

[Pierre Habouzit]

[-- Attachment #1: Type: text/plain, Size: 2028 bytes --]

At work we (ab-)use workdirs a lot, though, workdirs aren't for
everybody, and as our company grows, not everybody uses them sanely.

The two problems (that are well known to this list, and is the reason
why git new-workdir is in contrib afaict) with workdirs are:

  - the HEAD reflogs aren't shared, which means that pruning a working
    directory may trash accessible stuff from the reflog of another one.

  - if two workdirs are on the same branch at the same time, really,
    really, *REALLY* bad things may happen if one isn't aware of that
    fact.

I'm intending to adress those issues, though I would like to know how it
would be received. My current plan is this one. Have a git workdir
command, with a few subcommands (create, move, rename, ...), that
addresses both of the previous issues.

for the first one, the fix is simple: workdirs have now a name, and
their HEAD reflog lives in the "master" git repository reflog namespace
under logs/workdir/$workdir_name/HEAD. The workdir HEAD reflog is then a
symlink to the masters.

In this way, all workdirs see all the reflogs of every single workdir,
and pruning is safe again.


For the second one, when a workdir is created, a [workdir "foo"] section
is added to the master directory, with a path configuration variable
pointing to the ... path of the working directory. My plan would be to
teach git checkout to lean about that, and when there are workdirs
set up, git checkout would check that no other workdir is currently "on
the same branch", and would refuse to checkout to a branch that is
already checkouted elsewhere.


The current state of my git-workdir.sh is attached, though before I
start diving into the checkout builtin, I wanted to be sure that's the
way to go, and if there isn't any other issue I could have missed, plus
if this work has any chance to enter git.git :)
-- 
·O·  Pierre Habouzit
··O                                                madcoder@debian.org
OOO                                                http://www.madism.org

[-- Attachment #2: git-workdir.sh --]
[-- Type: application/x-sh, Size: 3640 bytes --]

Re: fixing workdirs

[Tait]

Pierre Habouzit <madcoder_madism.org> said (on 2010/07/08):
> ...                                    The workdir HEAD reflog is then a
> symlink to the masters.

#include <std-symlink-rant>

On programs (like git) pretending to be cross-platform, symlinks should
be avoided. They are to varying degrees, painful on non-*nix operating
systems.

Windows is an especially compatibility-breaking example, not only on the
programming side, but also in relation to user interface, and compatibility
with other programs. Programming-wise, documentation is sparse and
would require lots of platform-specific work-arounds. The user-interface
support is worse than terrible. And even if git does everything right,
there's no guarantee a copy, backup/restore, antivirus program, etc. won't
come along and corrupt the environment git so carefully created. Many of
those other programs don't properly handle Windows reparse points. For
those interested, http://shell-shocked.org/article.php?id=284 gives a
reasonable-looking overview of the details on Windows.

Re: fixing workdirs

[Joshua Jensen]

  ----- Original Message -----
From: Tait
Date: 7/8/2010 12:37 PM
> Pierre Habouzit<madcoder_madism.org>  said (on 2010/07/08):
>> ...                                    The workdir HEAD reflog is then a
>> symlink to the masters.
> #include<std-symlink-rant>
>
> Windows is an especially compatibility-breaking example, not only on the
> programming side, but also in relation to user interface, and compatibility
> with other programs. Programming-wise, documentation is sparse and
> would require lots of platform-specific work-arounds. The user-interface
> support is worse than terrible. And even if git does everything right,
> there's no guarantee a copy, backup/restore, antivirus program, etc. won't
> come along and corrupt the environment git so carefully created. Many of
> those other programs don't properly handle Windows reparse points. For
> those interested, http://shell-shocked.org/article.php?id=284 gives a
> reasonable-looking overview of the details on Windows.
I'm going to go out on a limb here and say that I suspect symbolic links 
work fine in Windows Vista and 7, because they are used all over the 
file system Microsoft installs.  Symbolic links for files, in 
particular, didn't exist until Vista.

Josh

Re: fixing workdirs

[Pierre Habouzit]

On Thu, Jul 08, 2010 at 11:37:14AM -0700, Tait wrote:
> 
> Pierre Habouzit <madcoder_madism.org> said (on 2010/07/08):
> > ...                                    The workdir HEAD reflog is then a
> > symlink to the masters.
> 
> #include <std-symlink-rant>
> 
> On programs (like git) pretending to be cross-platform, symlinks should
> be avoided. They are to varying degrees, painful on non-*nix operating
> systems.
> 
> Windows is an especially compatibility-breaking example, not only on the
> programming side, but also in relation to user interface, and compatibility
> with other programs. Programming-wise, documentation is sparse and
> would require lots of platform-specific work-arounds. The user-interface
> support is worse than terrible. And even if git does everything right,
> there's no guarantee a copy, backup/restore, antivirus program, etc. won't
> come along and corrupt the environment git so carefully created. Many of
> those other programs don't properly handle Windows reparse points. For
> those interested, http://shell-shocked.org/article.php?id=284 gives a
> reasonable-looking overview of the details on Windows.

Well that's how git-new-workdir works, and you don't /need/ git
new-workdir to do actual git work. Note that git is cross compatible to
all POSIX conformant filesystems, it's just windows that I can think of
that won't work with it... They don't provide any sane replacement for
the feature, so I don't see what we can do about it anyways.

-- 
·O·  Pierre Habouzit
··O                                                madcoder@debian.org
OOO                                                http://www.madism.org

Re: fixing workdirs

[Junio C Hamano]

Pierre Habouzit <madcoder@madism.org> writes:

> for the first one, the fix is simple: workdirs have now a name, and
> their HEAD reflog lives in the "master" git repository reflog namespace
> under logs/workdir/$workdir_name/HEAD. The workdir HEAD reflog is then a
> symlink to the masters.

I think this is a sane thing to do, except for the "symlink" part but that
would be just a minor implementation detail.

> For the second one, when a workdir is created, a [workdir "foo"] section
> is added to the master directory, with a path configuration variable
> pointing to the ... path of the working directory.

Ok.

> ... git checkout would check that no other workdir is currently "on
> the same branch", and would refuse to checkout to a branch that is
> already checkouted elsewhere.

I am personally fine with this, but if there is no way to override this
refusal it may break some people's existing workflow.  I dunno.

Re: fixing workdirs

[Avery Pennarun]

On Thu, Jul 8, 2010 at 3:40 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Pierre Habouzit <madcoder@madism.org> writes:
>> ... git checkout would check that no other workdir is currently "on
>> the same branch", and would refuse to checkout to a branch that is
>> already checkouted elsewhere.
>
> I am personally fine with this, but if there is no way to override this
> refusal it may break some people's existing workflow.  I dunno.

Perhaps it would be better to refuse to *update* a particular ref if
some other worktree has it checked out?  I don't think actually having
multiple trees checked out to the same branch is a problem, so much as
what happens when they start committing.

Have fun,

Avery

Re: fixing workdirs

[Pierre Habouzit]

On Thu, Jul 08, 2010 at 12:40:11PM -0700, Junio C Hamano wrote:
> Pierre Habouzit <madcoder@madism.org> writes:
> 
> > for the first one, the fix is simple: workdirs have now a name, and
> > their HEAD reflog lives in the "master" git repository reflog namespace
> > under logs/workdir/$workdir_name/HEAD. The workdir HEAD reflog is then a
> > symlink to the masters.
> 
> I think this is a sane thing to do, except for the "symlink" part but that
> would be just a minor implementation detail.

What would you suggest instead of the symlink then ? (knowing that all
the workdir is just a full symlink farm at them moment).

> > For the second one, when a workdir is created, a [workdir "foo"] section
> > is added to the master directory, with a path configuration variable
> > pointing to the ... path of the working directory.
> 
> Ok.

> > ... git checkout would check that no other workdir is currently "on
> > the same branch", and would refuse to checkout to a branch that is
> > already checkouted elsewhere.
> 
> I am personally fine with this, but if there is no way to override this
> refusal it may break some people's existing workflow.  I dunno.

Well it's probably fine to have a switch to override it of course, or
like it was suggested to prevent updating the reference instead. But
that sounds harder, and if you want to override it, it means that a lot
of commands will have to take a new argument (update-ref, commit, reset,
rebase, ...).

I'm more in favor of having checkout refusing to checkout if the
reference is already checkout-ed elsewhere, with a --ignore-workdirs or
similar switch to override this. git-checkout would still print out a
warning about the fact that /maybe/ the user is doing something crazy,
and then he'll be on his own. Plus it doesn't slow references updates
for the workdir case, only the branch switch which is way nicer.

-- 
·O·  Pierre Habouzit
··O                                                madcoder@debian.org
OOO                                                http://www.madism.org

Re: fixing workdirs

[Junio C Hamano]

Pierre Habouzit <madcoder@madism.org> writes:

> On Thu, Jul 08, 2010 at 12:40:11PM -0700, Junio C Hamano wrote:
>> Pierre Habouzit <madcoder@madism.org> writes:
>> 
>> > for the first one, the fix is simple: workdirs have now a name, and
>> > their HEAD reflog lives in the "master" git repository reflog namespace
>> > under logs/workdir/$workdir_name/HEAD. The workdir HEAD reflog is then a
>> > symlink to the masters.
>> 
>> I think this is a sane thing to do, except for the "symlink" part but that
>> would be just a minor implementation detail.
>
> What would you suggest instead of the symlink then ? (knowing that all
> the workdir is just a full symlink farm at them moment).

I can imagine that we may want to have a general mechanism to help an
object store that belongs to one "primary" repository be aware of ref-like
things that live outside of the repoistory itself, and not just a special
purpose hack suitable only to handle the workdirs.  E.g., we have talked
about a "fork" created by "clone -s" wanting the forkee repository to be
aware of its refs, so that rewinding the refs in the forkee repository and
then running gc there won't remove the objects now unnecessary in the
forkee but still needed by the forker repository.

It shouldn't be hard to do something similar to "gitdir: " support for
this without using a symlink, no?

Re: fixing workdirs

[Pierre Habouzit]

On Fri, Jul 09, 2010 at 03:25:15PM -0700, Junio C Hamano wrote:
> Pierre Habouzit <madcoder@madism.org> writes:
> 
> > On Thu, Jul 08, 2010 at 12:40:11PM -0700, Junio C Hamano wrote:
> >> Pierre Habouzit <madcoder@madism.org> writes:
> >> 
> >> > for the first one, the fix is simple: workdirs have now a name, and
> >> > their HEAD reflog lives in the "master" git repository reflog namespace
> >> > under logs/workdir/$workdir_name/HEAD. The workdir HEAD reflog is then a
> >> > symlink to the masters.
> >> 
> >> I think this is a sane thing to do, except for the "symlink" part but that
> >> would be just a minor implementation detail.
> >
> > What would you suggest instead of the symlink then ? (knowing that all
> > the workdir is just a full symlink farm at them moment).
> 
> I can imagine that we may want to have a general mechanism to help an
> object store that belongs to one "primary" repository be aware of ref-like
> things that live outside of the repoistory itself, and not just a special
> purpose hack suitable only to handle the workdirs.  E.g., we have talked
> about a "fork" created by "clone -s" wanting the forkee repository to be
> aware of its refs, so that rewinding the refs in the forkee repository and
> then running gc there won't remove the objects now unnecessary in the
> forkee but still needed by the forker repository.
> 
> It shouldn't be hard to do something similar to "gitdir: " support for
> this without using a symlink, no?

Sorry for the delay, I was on vacation.

Okay, I see, this makes sense. I'll see what I can do on this path,
though it's probably harder than simply extending gitfiles. We still
want a .git/ directory as we want most of the top level stuff to be
local to each repository (HEAD, ORIG_HEAD, ...) but not:

  - subdirectories (most of them being references, logs, ...)
  - the lock (which is kind of the weak point in my proposal atm, yours
    is nicer)
  - config (or do we want a cascading semantics: local workdir config,
    master repository config, user .gitconfig, /etc/.... ? I think not
    but ...)
  - packed-refs

Plus, to make everything work, the reflogs of a given "workdir" (or
shared clone) must be put in a different namespace to avoid clashes.
Though this is probably the simplest bit.

All in all, I'm afraid to have to look at every single git script that
for now writes without thinking twice under .git/ :/
-- 
·O·  Pierre Habouzit
··O                                                madcoder@debian.org
OOO                                                http://www.madism.org

Re: fixing workdirs

[Jan Hudec]

On Thu, Jul 08, 2010 at 13:08:42 +0200, Pierre Habouzit wrote:
> At work we (ab-)use workdirs a lot, though, workdirs aren't for
> everybody, and as our company grows, not everybody uses them sanely.
> 
> The two problems (that are well known to this list, and is the reason
> why git new-workdir is in contrib afaict) with workdirs are:
> 
>   - the HEAD reflogs aren't shared, which means that pruning a working
>     directory may trash accessible stuff from the reflog of another one.
>
>   - if two workdirs are on the same branch at the same time, really,
>     really, *REALLY* bad things may happen if one isn't aware of that
>     fact.

It should be possible to guard against those bad things. Basically the bad
things happen when a symbolic HEAD's target is changed in a way the HEAD is
not aware. But in those, and only those, cases the HEAD's reflog is not
updated. So some operations could check whether HEAD@{0} resolves to the same
commit as HEAD and if not they would either fail or automatically detach
HEAD.

That would serve as a safeguard not just against one worktree commiting to
a branch checked out in another worktree. It would also safeguard against
pushing to the branch provided the code in update that updates HEAD's reflog
was disabled for non-bare repositories.

> I'm intending to adress those issues, though I would like to know how it
> would be received. My current plan is this one. Have a git workdir
> command, with a few subcommands (create, move, rename, ...), that
> addresses both of the previous issues.
> 
> for the first one, the fix is simple: workdirs have now a name, and
> their HEAD reflog lives in the "master" git repository reflog namespace
> under logs/workdir/$workdir_name/HEAD. The workdir HEAD reflog is then a
> symlink to the masters.
> 
> In this way, all workdirs see all the reflogs of every single workdir,
> and pruning is safe again.

The problem with it is that it falls apart when somebody forgets to use the
'git worktree remove' command and deletes (or renames) the work-tree
manually.

Alternative idea would be to add support for new layout with one repository
and multiple worktrees. The worktrees would be restricted to siblings of the
.git directory where the repository lives with the repository itself being
marked as bare. So the layout would be:

  somedir/
  somedir/.git/           <- the repository, makred as bare
  somedir/worktree1/      <- a worktree
  somedir/worktree1/.git/ <- worktree's .git in current format
  somedir/worktree2/      <- another worktree
  somedir/worktree2/.git/ <- worktree's .git in current format
  ...

Such scheme would exchange some flexibility for more safety. The worktrees
could be created with current new-workdir. They would be found as simply
$GIT_DIR/../*/.git for purpose of solving the two problems above. On the
other hand a method for preparing this layout (git init --multiple-trees,
git clone --multiple-trees or something like that) would be needed.

IIRC bazaar uses scheme like this for it's multiple-worktree support.

> For the second one, when a workdir is created, a [workdir "foo"] section
> is added to the master directory, with a path configuration variable
> pointing to the ... path of the working directory. My plan would be to
> teach git checkout to lean about that, and when there are workdirs
> set up, git checkout would check that no other workdir is currently "on
> the same branch", and would refuse to checkout to a branch that is
> already checkouted elsewhere.

Wait a second. Don't you have the HEADs and their reflogs for workdirs stored
in the master? So you can just compare with all the other HEADs there, no?

The bookkeeping would break if you just deleted the workdir without telling
git. The configuration will break even more readily as renaming also breaks
it. 

Kind regards,
Jan

-- 
						 Jan 'Bulb' Hudec <bulb@ucw.cz>