git-testadd: Execute a command with only the staged changes in Git applied

git-testadd: Execute a command with only the staged changes in Git applied

[Øyvind A. Holm]

This is a script I created some weeks ago, and I've found it to be
immensely useful. Here is a snippet from git-testadd --help:

  If you have lots of unrelated uncommitted changes in the current
  repository and want to split up the commit, how can you easily check
  if the changes passes the test suite? With all the other unrelated
  changes it can be hard to make sure that only relevant changes becomes
  part of the commit, and that they don't result in regressions. This
  script clones the repository to the directory ".testadd.tmp" in the
  current directory and applies the staged chenges there (unless
  -u/--unmodified or -p/--pristine is specified), chdirs to the same
  relative directory in the clone and executes the command specified on
  the command line there.

The script is well-tested, and also have a test suite you can run to
make sure it works on your *nix system. Place git-testadd.t in a
subdirectory one level under the script location, chdir to that
directory and execute "./git-testadd.t". It also works with binary
files.

Available from

  https://gitlab.com/sunny256/utils/raw/master/git-testadd
  https://gitlab.com/sunny256/utils/raw/master/tests/git-testadd.t

It's also on GitHub, just replace "gitlab" with "github" in the URLs.
And of course, ideas and patches for new functionality/fixes are always
welcome.

        Øyvind

Re: git-testadd: Execute a command with only the staged changes in Git applied

[Øyvind A. Holm]

On 28 July 2016 at 18:37, Junio C Hamano <gitster@pobox.com> wrote:
> Øyvind A. Holm <sunny@sunbase.org> writes:
> > This is a script I created some weeks ago, and I've found it to be
> > immensely useful. Here is a snippet from git-testadd --help:
> >
> >   If you have lots of unrelated uncommitted changes in the current
> >   repository and want to split up the commit, how can you easily
> >   check if the changes passes the test suite? With all the other
> >   unrelated changes it can be hard to make sure that only relevant
> >   changes becomes part of the commit, and that they don't result in
> >   regressions. This script clones the repository to the directory
> >   ".testadd.tmp" in the current directory and applies the staged
> >   chenges there (unless -u/--unmodified or -p/--pristine is
> >   specified), chdirs to the same relative directory in the clone and
> >   executes the command specified on the command line there.
>
> So in short, this solves the same problem as "git stash --keep" but in
> a more scalable way, in the sense that "git stash --keep" allows you
> to instantiate what you have in the index so that your working tree
> can be used for such a test, but you cannot do anything else while you
> are waiting for the test to finish, and "testadd" allows you to keep
> hacking in the working tree while a test runs in its own temporary
> checkout (and presumably you can have more than one running, which
> would allow you to scale more)?

That's correct, the test clone is entirely separated from the working
copy, and you can keep working while the tests are running in the clone.
Combined with git-gui and/or "git add -p/git reset -p", it's easy to
tweak the staged changes until things are ok.

Also, there is a -l/--label option that creates a clone directory with
the name ".testadd-[LABEL].tmp", so you can have several test clones at
the same time, all with different staged changes. There is also a
-r/--ref option that tries to apply the staged changes onto another
commit, and the command will only run if the apply succeeds. Also, this
won't create dangling heads like "git stash --keep" does.

        Øyvind

Re: git-testadd: Execute a command with only the staged changes in Git applied

[Jakub Narębski]

W dniu 2016-07-28 o 18:56, Øyvind A. Holm pisze:
> On 28 July 2016 at 18:37, Junio C Hamano <gitster@pobox.com> wrote:
>> Øyvind A. Holm <sunny@sunbase.org> writes:

>>> This is a script I created some weeks ago, and I've found it to be
>>> immensely useful. Here is a snippet from git-testadd --help:
>>>
>>>   If you have lots of unrelated uncommitted changes in the current
>>>   repository and want to split up the commit, how can you easily
>>>   check if the changes passes the test suite? With all the other
>>>   unrelated changes it can be hard to make sure that only relevant
>>>   changes becomes part of the commit, and that they don't result in
>>>   regressions. This script clones the repository to the directory
>>>   ".testadd.tmp" in the current directory and applies the staged
>>>   chenges there (unless -u/--unmodified or -p/--pristine is
>>>   specified), chdirs to the same relative directory in the clone and
>>>   executes the command specified on the command line there.
>>
>> So in short, this solves the same problem as "git stash --keep" but in
>> a more scalable way, in the sense that "git stash --keep" allows you
>> to instantiate what you have in the index so that your working tree
>> can be used for such a test, but you cannot do anything else while you
>> are waiting for the test to finish, and "testadd" allows you to keep
>> hacking in the working tree while a test runs in its own temporary
>> checkout (and presumably you can have more than one running, which
>> would allow you to scale more)?
> 
> That's correct, the test clone is entirely separated from the working
> copy, and you can keep working while the tests are running in the clone.
> Combined with git-gui and/or "git add -p/git reset -p", it's easy to
> tweak the staged changes until things are ok.

I wonder if using `git worktree` instead of `git clone` (well, local
clone uses hardlinks, so it is not that costly as it looks like) would
be a better solution.
 
> Also, there is a -l/--label option that creates a clone directory with
> the name ".testadd-[LABEL].tmp", so you can have several test clones at
> the same time, all with different staged changes. There is also a
> -r/--ref option that tries to apply the staged changes onto another
> commit, and the command will only run if the apply succeeds. Also, this
> won't create dangling heads like "git stash --keep" does.

Nice.

Re: git-testadd: Execute a command with only the staged changes in Git applied

[Øyvind A. Holm]

On 28 July 2016 at 21:31, Jakub Narębski <jnareb@gmail.com> wrote:
> W dniu 2016-07-28 o 18:56, Øyvind A. Holm pisze:
> > Øyvind A. Holm <sunny@sunbase.org> writes:
> > > This is a script I created some weeks ago, and I've found it to be
> > > immensely useful. Here is a snippet from git-testadd --help:
> > > [...]
> > >   This script clones the repository to the directory
> > >   ".testadd.tmp" in the current directory and applies the staged
> > >   chenges there (unless -u/--unmodified or -p/--pristine is
> > >   specified), chdirs to the same relative directory in the clone
> > >   and executes the command specified on the command line there.
> >
> > That's correct, the test clone is entirely separated from the
> > working copy, and you can keep working while the tests are running
> > in the clone. Combined with git-gui and/or "git add -p/git reset
> > -p", it's easy to tweak the staged changes until things are ok.
>
> I wonder if using `git worktree` instead of `git clone` (well, local
> clone uses hardlinks, so it is not that costly as it looks like) would
> be a better solution.

That's an interesting idea. Have to test it out. This is the result from
the current master in linux.git:

With clone:

  $ time git testadd pwd
  git-testadd: Using ".testadd.tmp" as destination directory
  Cloning into '.testadd.tmp'...
  done.
  Checking out files: 100% (55256/55256), done.
  git-testadd: Applying staged changes

  git-testadd: Executing "pwd" in /home/sunny/src/test-wt/.testadd.tmp
  /home/sunny/src/test-wt/.testadd.tmp

  real    0m10.464s
  user    0m5.983s
  sys     0m2.790s
  $

With worktree:

  $ time git worktree add testaddtmp
  Preparing testaddtmp (identifier testaddtmp)
  Checking out files: 100% (55256/55256), done.
  HEAD is now at 194dc87 Add braces to avoid "ambiguous ‘else’" compiler
  warnings

  real    0m10.343s
  user    0m6.010s
  sys     0m2.523s
  $

Both tests were run with cold cache ("echo 3 >/proc/sys/vm/drop_caches"
as root). It seems as there's no difference, and that git clone is as
fast as it can get without breaking physical laws. And we probably
shouldn't do that. :)

Z poważaniem,
Øyvind

Re: git-testadd: Execute a command with only the staged changes in Git applied

[Øyvind A. Holm]

On 29 July 2016 at 00:38, Junio C Hamano <gitster@pobox.com> wrote:
> Øyvind A. Holm <sunny@sunbase.org> writes:
> > Jakub Narębski <jnareb@gmail.com> wrote:
> > > I wonder if using `git worktree` instead of `git clone` (well,
> > > local clone uses hardlinks, so it is not that costly as it looks
> > > like) would be a better solution.
> >
> > That's an interesting idea. Have to test it out. This is the result
> > from the current master in linux.git:
> >
> > With clone:
> > ...
> > With worktree:
> > ...
> >
> > Both tests were run with cold cache ("echo 3
> > >/proc/sys/vm/drop_caches" as root). It seems as there's no
> > >difference, and that git clone is as fast as it can get without
> > >breaking physical laws. And we probably shouldn't do that. :)
>
> I expect that writing the 55k+ files in the working tree would
> dominate the cost.  Local clone would make some hardlinks in .git
> (including ones in .git/object/*) but the cost of that would not be
> too high as long as the repository is well packed; "git worktree"
> would reduce that part of the cost from "git clone", but both incur
> the cost of "checkout", i.e. actually populating the new working tree.
>
> Does the test directory even need to look anything like Git?  In other
> words, would it suffice if it resembled the result of running "git
> archive | tar xf -"?  I suspect that it would not make much
> difference, either, for the same reason, though ;-).

Using git archive saved 1.6 seconds:

  $ mkdir testdir; git archive HEAD | (cd testdir && time tar x)

  real    0m8.881s
  user    0m0.440s
  sys     0m2.740s
  $

But when .git is missing in the subdir, git apply doesn't work. Can't
think of any way to get that to work without involving patch(1) and
friends, and then the binary diffs goes out of the window.

But I'm quite satisfied with 10.4 seconds with cold diskcache and >55K
files. Very impressive, actually.

- Øyvind