Input welcome for "Git Bisect and Testing" presentation at GTAC 2010

Input welcome for "Git Bisect and Testing" presentation at GTAC 2010

[Christian Couder]

Hi,

The Google Test Automation Conference (http://www.gtac.biz) committee accepted 
my proposal for a "Git Bisect and Testing" presentation at the GTAC 2010 in 
India on October 28 and 29. So I am working on it.

It will be roughly based on my previous "Fighting regressions with git bisect" 
article (http://www.kernel.org/pub/software/scm/git/docs/git-bisect-lk2009.html)
and presentation (http://www.linux-kongress.org/2009/slides/fighting_regressions_with_git_bisect_christian_couder.pdf). 
But I would be happy to add more concrete examples of how "git bisect" is used 
in the real world. And more generally I am very open to any kind of feedback 
about "git bisect".

For example it can be how "git bisect run" is used automatically in build 
systems, or how "git bisect" is used manually on GUI applications in your 
company, or how you use Avery Pennarun's "git-builder" (http://github.com/apenwarr/gitbuilder, it uses "git bisect" 
underneath), or what features you would want to see added, or what have been 
your experience with it, and so on.

Thanks in advance,
Christian.

PS: Because of this presentation I won't come to the GitTogether this year.
Anyway I wish people going there an happy GitTogether'10!   

Re: Input welcome for "Git Bisect and Testing" presentation at GTAC 2010

[Avery Pennarun]

On Sun, Sep 12, 2010 at 1:25 PM, Christian Couder
<chriscool@tuxfamily.org> wrote:
> For example it can be how "git bisect run" is used automatically in build
> systems, or how "git bisect" is used manually on GUI applications in your
> company, or how you use Avery Pennarun's "git-builder" (http://github.com/apenwarr/gitbuilder, it uses "git bisect"
> underneath), or what features you would want to see added, or what have been
> your experience with it, and so on.

Ha, when I saw my name in the (very short) cc: list, I was wondering
why I was being included explicitly here.  Thanks for reminding me
that I wrote gitbuilder and it uses git-bisect :)  It's interesting in
that it has quite a few followers on github, but I don't get any
feedback about it.  (I assume either it Just Works or it Just Doesn't
Work :))

Since you asked, here are some passing comments about git-bisect
and/or gitbuilder:

- for most smallish projects, doing 'git bisect' by hand (especially
'git bisect run') gives you the most instant gratification and is the
easiest way to find where you introduced an error.  gitbuilder takes
longer and is thus less interesting.

- for bisect to be useful, you really need to be careful to commit
things in logical chunks and to make sure each commit *works*.  I've
noticed that people do their commits totally differently when they
are/aren't thinking about future bisections.  That means bisect is a
bit of an "artificial" construct: you can't just bisect your way
through just *any* source tree and get useful results, because in many
commercial projects (at least), most commits don't even run.  Sad, but
true.  So you really need to train developers to think seriously about
making their commits small, atomic, and non-broken before pushing.

- gitbuilder, conversely, is more useful for very large projects with
extensive test suites.  For example, at work we have a project that
takes about an hour to build from scratch and run through all the
tests. Bisecting this sort of project by hand is a huge pain.  In
fact, even running the automated tests before committing is a lot to
ask - and people often don't do it.  Thus, having a system that
*automatically* grinds away at releases and pinpoints the sources of
error can be very valuable.

- Because gitbuilder builds all your branches from multiple
repositories automatically, failing the gitbuilder test doesn't have
to be an embarrassment because it doesn't have to hurt anybody but
yourself; if it catches a failure, just rebase your branch and get it
turning green before committing to mainline.  gitbuilder has been
instrumental in a) helping get our internal development projects to
zero unit test failures in the mainline at all times, and b) making it
easy to decide which branches are ready to merge and which aren't, and
c) knowing who to complain to when a problem is introduced even so.

- gitbuilder uses git-bisect (actually git rev-list --bisect) in a
peculiar way: with --first-parent.  This is on purpose, although you
have to think carefully about your workflow in order to benefit from
it.  There are two advantages to doing it this way: 1) the gitbuilder
status display makes way more sense to normal people because it can
just show a linear sequence of commits; 2) to a maintainer, merges
into the mainline are an all-or-nothing sort of thing.  If I merged in
a branch and it started failing, then I don't want to bisect it, I
want to complain to the person whose branch I merged in.  And that
person's branch should *already* be built by gitbuilder and bisected
away, so he should have no trouble tracing it by looking at the
gitbuilder stats for his own branch.  So far, users here have found
this way of thinking about the bisection to be pretty easy to
understand, much more so than git's rather mind-melting DAG graphs
that you might get from things like 'git bisect visualize'.  It's also
pretty resilient against people who make frequent non-working commits
in their side branches; the mainline maintainer never has to worry
about "accidentally" bisecting into them, which removes the need for
things like 'git bisect skip'.  I've never much trusted 'git bisect
skip', though now that it's pseudorandom it supposedly works better
than it once did, so maybe I should adjust my opinion.

I'm sorry I'll miss meeting you and/or seeing your presentation at
GitTogether.  Have fun in India!

Have fun,

Avery

Re: Input welcome for "Git Bisect and Testing" presentation at GTAC 2010

[Christian Couder]

On Monday 13 September 2010 10:04:50 Avery Pennarun wrote:
> On Sun, Sep 12, 2010 at 1:25 PM, Christian Couder
> 
> Ha, when I saw my name in the (very short) cc: list, I was wondering
> why I was being included explicitly here.  Thanks for reminding me
> that I wrote gitbuilder and it uses git-bisect :)  It's interesting in
> that it has quite a few followers on github, but I don't get any
> feedback about it.  (I assume either it Just Works or it Just Doesn't
> Work :))

[...]

> I'm sorry I'll miss meeting you and/or seeing your presentation at
> GitTogether.  Have fun in India!

Thanks and thanks for your feedback. I will miss meeting you and many other 
guys at GitTogether too.

Best regards,
Christian.