Re: http://www.kernel.org/doc/local/git-quick.html#bisect

Re: http://www.kernel.org/doc/local/git-quick.html#bisect

[Rob Landley]

On Friday 15 May 2009 05:18:29 Florian Mickler wrote:
> Hi Rob!
>
> I'm writing to this emailadress, because I went down from $subject to
> doc/local to doc/  and yours was the first email-adress I found.
>
> Who is maintaining $subject? I found it via google searching for git
> bisect.

Um, me?

I wrote it way back when, because I had to learn how to use git enough to do a 
bisect and it wasn't properly documented then.  (At least three other people 
have written up their own bisect howtos on it since then.)

People poke me about it from time to time, but mostly I personally use 
mercurial.

> [ rest of mail is only relevant for git-quick.html maintainer: ]
>
> The git-bisect description is really _wrong_. ( There should be a 
> stronger word for that... like ''badwrong'' or smth).

The sky is falling, the sky is falling!

> You _never_ _never_ guess "git bisect
> good" or "git bisect bad".

Yes I do, and it works just fine.

> Instead you should use
> git bisect skip or
> git bisect visualize and choose another commit to test.

I believe that both of those options were implemented _after_ I wrote the 
above document.  (People keep telling me that the user interface of git can't 
be said to "suck" because it keeps changing.)

> If you guess wrong, your bisection result is not valid anymore.

Which is why it tells you how to back up to where you guessed so you can try 
it the other way when you guess wrong.  (Did you stop reading when you 
encountered the first thing you disagreed with?)  Lemme cut and paste the 
appropriate chunk:

  If you guess wrong (hint: every revision bisect wants
  to test after that comes out the opposite of your guess, all the way to the
  end) do a <b>git bisect replay ../bisect.log</b> to restart from your
  saved position, and guess the other way.

The ability to do that sort of thing would presumably be why they implemented 
the replay option in the first place.

> (and that really sucks, because all your compiliations after that one
> are _worthless_ ).

No, they demonstrate that the bug isn't in that fork, at which point you back 
up and check the other fork.

These days they've added stuff to do it in a slightly more automated fashion, 
but I note that I was bisecting in mercurial before it had any bisect 
support, just using the sequence numbers and a basic understanding of how 
branches and merges work on a nonlinear repository.  I wrote a doc for people 
who might conceivably want to understand how the process actually _works_...

> But if you defer the decision for that commit (via git-skip) you at
> least get a range of commits including the bad
> commit, or if the commit immidiatly before the bad commit is
> buildable, you even get the right bad commit.
>
> i'm quoting the text that neeeds to be _removed_:

I'm going to give you the benefit of the doubt and assume that "neeeds" isn't 
a typo rather than a "badwrong" _word_ indicating how _falling_ 
the -=>*[(SKY)]*<=- is today.

Last time somebody poked me about this I added this bit:

  Note: you can also tell git itself to skip a commit you can't test with
  git bisect skip.  Unfortunately, the git-bisect man page gives this
  warning about the "skip" command:

    But computing the commit to test may be slower afterwards and git may
    eventually not be able to tell the first bad among a bad and one or more
    "skip"ped commits.

Which is what the Ubuntu 8.04 (still the LTS release, until october anyway) 
man page for git-bisect says.  I.E. the automation can't actually guarantee 
it will find the problem, and this describes a manual way so you can 
understand what it's actually doing.

Let me know when they sky _finishes_ falling for you, and then if you'd like 
to send a slightly less hysterical message I might take a closer look at what 
git's doing this week and see if it's actually changed since the last time I 
was paying attention.  (Keep in mind that since my laptop's still running 
Ubuntu LTS, because 8.10's wireless support spontaneously ate itself so badly 
I had to reinstall, and 9.04 doesn't like Intel 3D chipsets.  So I'll have to 
build git from source to see how the "current" versions differ from that...)

But you have yet to convince me anything's actually wrong...

Rob
-- 
Latency is more important than throughput. It's that simple. - Linus Torvalds

Re: http://www.kernel.org/doc/local/git-quick.html#bisect

[Florian Mickler]

[-- Attachment #1: Type: text/plain, Size: 6430 bytes --]

Hi!

Sorry. It seems we started off on the wrong foot.

I know that one never ends learning and I am glad you took the
time to write this stuff down because it helps. 
So let this be clear between us, i'm not trying to offend you and the
criticism is only about git-quick.html as it is found by google.  

I choose the 'the sky is falling' attitude to 'shout into the
woods' and provoke at least some response. (not knowing where or who
the maintainer was... )


On Fri, 15 May 2009 18:20:04 -0500
Rob Landley <rob@landley.net> wrote:

> On Friday 15 May 2009 05:18:29 Florian Mickler wrote:
> > Hi Rob!
> >
> > I'm writing to this emailadress, because I went down from $subject
> > to doc/local to doc/  and yours was the first email-adress I found.
> >
> > Who is maintaining $subject? I found it via google searching for git
> > bisect.
> 
> Um, me?

perhaps you should add that bit of information to it then? 

> 
> I wrote it way back when, because I had to learn how to use git
> enough to do a bisect and it wasn't properly documented then.  (At
> least three other people have written up their own bisect howtos on
> it since then.)
> 
> People poke me about it from time to time, but mostly I personally
> use mercurial.

which is fine and i appreciate your effort.

> 
> > Instead you should use
> > git bisect skip or
> > git bisect visualize and choose another commit to test.
> 
> I believe that both of those options were implemented _after_ I wrote
> the above document.  (People keep telling me that the user interface
> of git can't be said to "suck" because it keeps changing.)

yes. so my email is all the more relevant, or not?

> 
> > If you guess wrong, your bisection result is not valid anymore.
> 
> Which is why it tells you how to back up to where you guessed so you
> can try it the other way when you guess wrong.  (Did you stop reading
> when you encountered the first thing you disagreed with?)  Lemme cut
> and paste the appropriate chunk:

of course i stopped _not_. Who do you think i am! :)

> 
>   If you guess wrong (hint: every revision bisect wants
>   to test after that comes out the opposite of your guess, all the
> way to the end) do a <b>git bisect replay ../bisect.log</b> to
> restart from your saved position, and guess the other way.

which is exactly my point: you need to replay and wasted 10
 reboot+compile cycles.
AND it may be possible that every test after your guess comes out the
opposite of your guess but you guessed right nonetheless. this does
not tell you that your guess was wrong! it is not even a pre-requisit
for your guess being wrong. 

suppose the following: 

Good--+---B--1+--------BAD
       \     /  
        ----2   

("B" being the bug-introducing changeset, "1" the first tested commit,
"2" the 2nd tested commit, "-" some other commit and "+" some merge or
branchpoint. )

you guess wrong at 1 (this means you classify 1 good.)
second commit is 2 which is good, as there the bug B was never
introduced. so you get 2 as good. which is not the opposite of your
guess. 

> The ability to do that sort of thing would presumably be why they
> implemented the replay option in the first place.

well,  i would say they introduced it so one can doublecheck the tests
if one realizes that maybe unplugging the speakers is overshadowing the
bug "speakers don't work in rc5, but in rc1 they worked". or smth like
that. but this is irrelevant.

> 
> > (and that really sucks, because all your compiliations after that
> > one are _worthless_ ).
> 
> No, they demonstrate that the bug isn't in that fork, at which point
> you back up and check the other fork.
no... you never know because you
have no way of knowing if the bisection succeeded or not. 

> > But if you defer the decision for that commit (via git-skip) you at
> > least get a range of commits including the bad
> > commit, or if the commit immidiatly before the bad commit is
> > buildable, you even get the right bad commit.
> >
> > i'm quoting the text that neeeds to be _removed_:
> 
> I'm going to give you the benefit of the doubt and assume that
> "neeeds" isn't a typo rather than a "badwrong" _word_ indicating how
> _falling_ the -=>*[(SKY)]*<=- is today.

if you use git-bisect skip there is the (large) chance that your
skipped commit is based ontop of a test-able commit which already has
the bug. so you don't  need to guess, but have the chance to get a
correct bisection result.  so i think the 'needs' is warranted.
(relative to the impact git-bisect has on one's life...)

> 
> Last time somebody poked me about this I added this bit:
> 
>   Note: you can also tell git itself to skip a commit you can't test
> with git bisect skip.  Unfortunately, the git-bisect man page gives
> this warning about the "skip" command:
> 
>     But computing the commit to test may be slower afterwards and git
> may eventually not be able to tell the first bad among a bad and one
> or more "skip"ped commits.

this is nonsense (the ''being slower part'').  the other part:
the first bad among a bad and one or more skipped commits is of course
true. but what do you expect? at least you can defer the guessing to
the end of the bisection. maybe the skipped commits are ''cut off'' by
test-able commits. so you don't need to guess about them . at the very
least you get a range of commits which contain the bad one. 


> Let me know when they sky _finishes_ falling for you, and then if
> you'd like to send a slightly less hysterical message I might take a
> closer look at what git's doing this week and see if it's actually
> changed since the last time I was paying attention.  (Keep in mind
> that since my laptop's still running Ubuntu LTS, because 8.10's
> wireless support spontaneously ate itself so badly I had to
> reinstall, and 9.04 doesn't like Intel 3D chipsets.  So I'll have to
> build git from source to see how the "current" versions differ from
> that...)

i guess this is not necessary, as it is not implementation specific but
fundamental to the princip of binary-search. 

> 
> But you have yet to convince me anything's actually wrong...

of course! i never expect someone to do smth without being convinced.
some do, but thats a sign of not caring, being intimidated or being
stupid. 

sincerely,

Florian

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: http://www.kernel.org/doc/local/git-quick.html#bisect

[Rob Landley]

On Sunday 17 May 2009 16:11:35 Florian Mickler wrote:
> Hi!
>
> Sorry. It seems we started off on the wrong foot.
>
> I know that one never ends learning and I am glad you took the
> time to write this stuff down because it helps.
> So let this be clear between us, i'm not trying to offend you and the
> criticism is only about git-quick.html as it is found by google.
>
> I choose the 'the sky is falling' attitude to 'shout into the
> woods' and provoke at least some response. (not knowing where or who
> the maintainer was... )

You've managed that, yes.

> > > Instead you should use
> > > git bisect skip or
> > > git bisect visualize and choose another commit to test.
> >
> > I believe that both of those options were implemented _after_ I wrote
> > the above document.  (People keep telling me that the user interface
> > of git can't be said to "suck" because it keeps changing.)
>
> yes. so my email is all the more relevant, or not?

hg log -v git-quick.html says:

  changeset:   101:ae115ea731f2
  user:        Rob Landley <rob@landley.net>
  date:        Tue Aug 19 20:59:33 2008 -0500
  files:       local/git-quick.html
  description:
  Rafal Milecki brought the "git bisect skip" command to my attention.

I.E. I added a note about git bisect skip just under a year ago.  (Yes, I 
maintained revisions of that document in a mercurial archive.  No, it wasn't 
the only thing in the archive.)

> ("B" being the bug-introducing changeset, "1" the first tested commit,
> "2" the 2nd tested commit, "-" some other commit and "+" some merge or
> branchpoint. )
>
> you guess wrong at 1 (this means you classify 1 good.)
> second commit is 2 which is good, as there the bug B was never
> introduced. so you get 2 as good. which is not the opposite of your
> guess.

If you converge all the way to a single changeset and it doesn't reproduce the 
bug, you guessed wrong and have to back up.  As described.

No mechanism can automatically find a bug that was introduced in a group of 
revisions that doesn't build for you, or cannot otherwise get far enough to 
reproduce the bug.  (For that you have to patch the source to get it to work 
will enough you can test for that specific bug.)

> > The ability to do that sort of thing would presumably be why they
> > implemented the replay option in the first place.
>
> well,  i would say they introduced it so one can doublecheck the tests
> if one realizes that maybe unplugging the speakers is overshadowing the
> bug "speakers don't work in rc5, but in rc1 they worked". or smth like
> that. but this is irrelevant.

So the fact they introduced that feature before "skip" was implemented doesn't 
enter into this somehow?

> > > (and that really sucks, because all your compiliations after that
> > > one are _worthless_ ).
> >
> > No, they demonstrate that the bug isn't in that fork, at which point
> > you back up and check the other fork.
>
> no... you never know because you
> have no way of knowing if the bisection succeeded or not.

You can always either find the first commit that reproduced the bug, or 
confirm that the bug was introduced in a versions that doesn't build for you.  
The automation doesn't have any more information to work with than the manual 
method does; it can't do more than that either.

Apparently, you don't understand the method I described.  Obviously, my 
description of this method was crap.

> > > But if you defer the decision for that commit (via git-skip) you at
> > > least get a range of commits including the bad
> > > commit, or if the commit immidiatly before the bad commit is
> > > buildable, you even get the right bad commit.
> > >
> > > i'm quoting the text that neeeds to be _removed_:
> >
> > I'm going to give you the benefit of the doubt and assume that
> > "neeeds" isn't a typo rather than a "badwrong" _word_ indicating how
> > _falling_ the -=>*[(SKY)]*<=- is today.
>
> if you use git-bisect skip there is the (large) chance that your
> skipped commit is based ontop of a test-able commit which already
> has the bug 

If it's testable, why are you skipping it?

Why would the chance be larger for it to have the bug than to not have the 
bug?  (And if you think it has the bug, guess that it's bad then.)

> > Last time somebody poked me about this I added this bit:
> >
> >   Note: you can also tell git itself to skip a commit you can't test
> > with git bisect skip.  Unfortunately, the git-bisect man page gives
> > this warning about the "skip" command:
> >
> >     But computing the commit to test may be slower afterwards and git
> > may eventually not be able to tell the first bad among a bad and one
> > or more "skip"ped commits.
>
> this is nonsense (the ''being slower part'').

Why are you telling _me_ about your dissatisfaction with a quote from the 
git-bisect man page?

> > But you have yet to convince me anything's actually wrong...
>
> of course! i never expect someone to do smth without being convinced.
> some do, but thats a sign of not caring, being intimidated or being
> stupid.

In this case it would be "not caring", at least not enough to argue with you 
about this.

You're insisting that the mention of "git-bisect skip" I added a year ago 
wasn't sufficient, and what's there is potentially misleading and harmful to 
minors and so on, and thus it is vitally important that several screens worth 
of material be removed.  (Despite the technique working for me just fine.)

*shrug*  I never claimed to be a git expert.  I leave that to you.

I've removed the potentially misleading document from kernel.org.  There are 
plenty of other git tutorials, written by people who know/care far more about 
git than I do, and I have no intention of distracting you from them.

Rob
-- 
Latency is more important than throughput. It's that simple. - Linus Torvalds

Re: http://www.kernel.org/doc/local/git-quick.html#bisect

[Florian Mickler]

[-- Attachment #1: Type: text/plain, Size: 5923 bytes --]

On Mon, 18 May 2009 02:54:20 -0500
Rob Landley <rob@landley.net> wrote:


> You've managed that, yes.
good. please, i'm a friendly guy.. at least i think so. i'm
terribly sorry if i upset you. 


> 
> > ("B" being the bug-introducing changeset, "1" the first tested
> > commit, "2" the 2nd tested commit, "-" some other commit and "+"
> > some merge or branchpoint. )
> >
> > you guess wrong at 1 (this means you classify 1 good.)
> > second commit is 2 which is good, as there the bug B was never
> > introduced. so you get 2 as good. which is not the opposite of your
> > guess.
> 
> If you converge all the way to a single changeset and it doesn't
> reproduce the bug, you guessed wrong and have to back up.  As
> described.

but in this case the changeset (mergecommit) has the bug, as the
mergecommit is on-top of 1 which is on top of B. 

reverting it is probably non-trivial. replaying with all the
guesses reverted needs 2^[nr-of-guesses] compiles. which is not
feasible if there are many buildbreakages. 

> 
> No mechanism can automatically find a bug that was introduced in a
> group of revisions that doesn't build for you, or cannot otherwise
> get far enough to reproduce the bug.  (For that you have to patch the
> source to get it to work will enough you can test for that specific
> bug.)

yes.

> So the fact they introduced that feature before "skip" was
> implemented doesn't enter into this somehow?

well, maybe if one is interested at the _history_ of
git-quick.html and git-bisect...  but i'm more interested in the
present and future of things. :) (if you know what i mean)


> > > > (and that really sucks, because all your compiliations after
> > > > that one are _worthless_ ).
> > >
> > > No, they demonstrate that the bug isn't in that fork, at which
> > > point you back up and check the other fork.
> >
> > no... you never know because you
> > have no way of knowing if the bisection succeeded or not.
> 
> You can always either find the first commit that reproduced the bug,
> or confirm that the bug was introduced in a versions that doesn't
> build for you. The automation doesn't have any more information to
> work with than the manual method does; it can't do more than that
> either.
> 
> Apparently, you don't understand the method I described.  Obviously,
> my description of this method was crap.

well, maybe i didn't tried hard enough to understand it? but i think
guessing is best to be deferred to the _end_ of the bisection. 


> > if you use git-bisect skip there is the (large) chance that your
> > skipped commit is based ontop of a test-able commit which already
> > has the bug 
> 
> If it's testable, why are you skipping it?
no. i try again. 

how i imagine patching: you apply one patch on top of the other. so
that you get a 'pile' :) there is a 'base' and a 'tip'. 

and here's what i meant to say:

given this 'pile of commits':
base---xxxxxxxx--B-xxxxxxxx---tip

x are commits that don't build, B is the bug-introducing changeset and
- are other commits. 

if you skip all the (x) commits, then you never guess but get a
completely right and sane bisection result pointing to B. because you
can test the commit right after B (which is BAD), and right before B
which is good. 

so you never guessed and you never replayed, you just skipped all the
way through those not building commits and have pinpointed the B-Commit
nonetheless... see?

with 'git bisect visualize' (which fires gitk up, only showing the
commits framed by your nearest good and bad decision)  you can even look
in the log for the corresponding build-fix and test that commit instead.


on the other hand:

base---xxxBxxx---tip

in this situation, skipping dosn't do any good. but it dosn't hurt
also. you just get this none-building range of commits as containing
the bug. 


but if:

base-xx--xxBxx--xx--tip

you win, because those 4 outer nonbuilding commits are ''cut off'' and
you just get to choose between 5 commits instead.

> 
> Why would the chance be larger for it to have the bug than to not
> have the bug?  (And if you think it has the bug, guess that it's bad
> then.)

what i meant: if you bisect through the kernel (2^17 commits from 2.6.27
to 2.6.30-rc5) your chance is large that any nonbuilding-patch-range is
small and cut-off  from your bug-introducing commit. (that is there are
building commits that are either good or bad between B and the
build-breakage)



> 
> Why are you telling _me_ about your dissatisfaction with a quote from
> the git-bisect man page?
It was just because, well,  you quoted it. I'm  sorry. 


> You're insisting that the mention of "git-bisect skip" I added a year
> ago wasn't sufficient, and what's there is potentially misleading and
> harmful to minors and so on, and thus it is vitally important that
> several screens worth of material be removed.  (Despite the technique
> working for me just fine.)
> 
> *shrug*  I never claimed to be a git expert.  I leave that to you.
> 
> I've removed the potentially misleading document from kernel.org.
> There are plenty of other git tutorials, written by people who
> know/care far more about git than I do, and I have no intention of
> distracting you from them.
> 
> Rob

but you are working overtime with this method! perhaps it's ok, if you
don't have to do too many guesses and i never tried your method. but if
i tried to explain git bisect the way i _think_ it is right and
understandable , and you are satisfied with it, would you consider
putting it back up?

as it is already pretty high in google, it would be a shame to just
take it down. especially as this was the only thing that tripped me up. 
(and not finding the adress to send a corrected paragraph freaked me
out, then. again, i'm sorry. )


Sincerely,
Florian

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: http://www.kernel.org/doc/local/git-quick.html#bisect

[Rob Landley]

On Monday 18 May 2009 14:40:22 Florian Mickler wrote:
> On Mon, 18 May 2009 02:54:20 -0500
>
> Rob Landley <rob@landley.net> wrote:
> > You've managed that, yes.
>
> good. please, i'm a friendly guy.. at least i think so. i'm
> terribly sorry if i upset you.
>
> > > ("B" being the bug-introducing changeset, "1" the first tested
> > > commit, "2" the 2nd tested commit, "-" some other commit and "+"
> > > some merge or branchpoint. )
> > >
> > > you guess wrong at 1 (this means you classify 1 good.)
> > > second commit is 2 which is good, as there the bug B was never
> > > introduced. so you get 2 as good. which is not the opposite of your
> > > guess.
> >
> > If you converge all the way to a single changeset and it doesn't
> > reproduce the bug, you guessed wrong and have to back up.  As
> > described.
>
> but in this case the changeset (mergecommit) has the bug, as the
> mergecommit is on-top of 1 which is on top of B.
>
> reverting it is probably non-trivial. replaying with all the
> guesses reverted needs 2^[nr-of-guesses] compiles. which is not
> feasible if there are many buildbreakages.

Reverting in what sense?  "git replay" is an O(1) operation that returns you 
to a previous state, which you've presumably already tested.  Why would you 
recompile a previously tested state?

If you mean "coming up with a patch to fix the intermediate step enough you 
can test it", that has nothing to do with git.  Reverting cherry-picked 
patches is sometimes a way to do that, but usually other patches apply on top 
of that so the revert won't apply to the middle of the broken range.  Another 
way to fix it is finding a later patch that fixed whatever it was (sometimes 
the first version that compiles after the "broken" range does that), and 
sometimes you have to fix it up by hand.

Mostly, you have to read the code, figure out what broke, and fix it.  Or you 
hope very much that it's _not_ in the broken range and continue to bisect 
until you've proved it is.  (And yes, confirming that means changing what 
you're testing for halfway through a bisect, and thus changing what 
you're "good" and "bad" answers mean, and _that_ means understanding what 
you're _doing_, no matter which tools you're using to do it.)

> > Apparently, you don't understand the method I described.  Obviously,
> > my description of this method was crap.
>
> well, maybe i didn't tried hard enough to understand it?

If my description required more effort to understand than you put into this 
conversation telling me how I should throw it out and replace it, then the 
description was crap.

> what i meant: if you bisect through the kernel (2^17 commits from 2.6.27
> to 2.6.30-rc5) your chance is large that any nonbuilding-patch-range is
> small and cut-off  from your bug-introducing commit.

The first bug I tried to bisect in the kernel was introduced somewhere in a 
range of about 3000 commits that didn't compile.  (I tend to test non-x86 
architectures.  Some non-x86 boards only get regression tested once or twice 
a year.)

> but you are working overtime with this method! perhaps it's ok, if you
> don't have to do too many guesses and i never tried your method

Are you saying you never tried it, but felt the need to complain about it?

> . but if 
> i tried to explain git bisect the way i _think_ it is right and
> understandable , and you are satisfied with it, would you consider
> putting it back up?

No, but you're welcome to write your own document explaining git bisect the 
way you think is "right and understandable", and post it anywhere you like.

I'm done arguing with you.

Rob
-- 
Latency is more important than throughput. It's that simple. - Linus Torvalds