rebase: strange failures to apply patc 3-way

rebase: strange failures to apply patc 3-way

[Max Horn]

Recently I have observed very strange failures in "git rebase" that cause it to fail to work automatically in situations where it should trivially be able to do so.

In case it matter, here's my setup: git 1.8.2.rc2.4.g7799588 (i.e. git.git master branch) on Mac OS X. The repos clone is on a HFS+ partition, not on a network volume. No gitattributes are being used.  Regarding the origin of the repos (I hope it doesn't matter, but just in case): The repository in question used to be a CVS repository; it was decided to switch to Mercurial (not my decision ;-) and I performed the conversion; I first converted it to git using cvs2git (from the cvs2svn suite), then performed some history cleanup, and finally used "hg convert" to convert it to git. Recently, I have been accessing the repository from git via the gitifyhg tool <https://github.com/buchuki/gitifyhg>. 

Anyway, several strange things just happened (and I had similar experiences in the past days and weeks, but that was using an older git, and I thought I was just doing something wrong).

Specifically, I wanted to rebase a branch with some experimental commits. The strange things started with this:

$ git rebase MY-NEW-BASE
First, rewinding head to replay your work on top of it...
Applying: SOME COMMIT
Applying: SOME OTHER COMMIT
...
Applying: COMMIT A
Applying: COMMIT B
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
error: Your local changes to the following files would be overwritten by merge:
	some/source.file
Please, commit your changes or stash them before you can merge.
Aborting
Failed to merge in the changes.
Patch failed at 0014 COMMIT B
The copy of the patch that failed is found in:
   /path/to/my/repo/.git/rebase-apply/patch

When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".


Now, what is strange about this is that the failed patch actually applies cleanly:

$ patch -p1 < /path/to/my/repo/.git/rebase-apply/patch
patching file some/source.file
$

And there is no subtle merge issue here, either: That patch is the only one to have touched the surrounding code since 1999! There is no source of conflict there!

Anyway. The tale gets stranger, as I was trying to do this again (no changes were made to the repos in between, this is a straight continuation from above):

$ git rebase --abort
$ git rebase MY-NEW-BASE
First, rewinding head to replay your work on top of it...
Applying: SOME COMMIT
Applying: SOME OTHER COMMIT
...
Applying: COMMIT A
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
error: Your local changes to the following files would be overwritten by merge:
	some/othersource.file
	some/yetanother.file
Please, commit your changes or stash them before you can merge.
Aborting
Failed to merge in the changes.
Patch failed at 0013 COMMIT A
The copy of the patch that failed is found in:
   /path/to/my/repo/.git/rebase-apply/patch

When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".



So suddenly it fails to apply the commit A, the one before the previously failing commit. Huh? But again, the failing patch applies cleanly (and after all, rebase was able to apply it in my previous attempt).  And again, the patch actually applies cleanly. So one more try:


$ git rebase --abort
$ git rebase MY-NEW-BASE
First, rewinding head to replay your work on top of it...
Applying: SOME COMMIT
Applying: SOME OTHER COMMIT
...
Applying: COMMIT A
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
error: Your local changes to the following files would be overwritten by merge:
	some/othersource.file
Please, commit your changes or stash them before you can merge.
Aborting
Failed to merge in the changes.
Patch failed at 0013 COMMIT A
The copy of the patch that failed is found in:
   /path/to/my/repo/.git/rebase-apply/patch

When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".


Again it fails in commit A -- but this time, it only thinks one file is problematic. HUH? Again, the patch actually applies cleanly:

$ patch -p1 < /path/to/my/repo/.git/rebase-apply/patch
patching file some/othersource.file
patching file some/yetanother.file


At this point, things stabilized, and when I now abort and reattempt the merge, it always fails in the same way. This time trying to apply a change to a code comment that was last changed in 1997 (though for one hunk, a few lines before the changed lines, there is a line that was changed in 2008... but I assure you, that line is there in the ancestors of both the branch I want to rebase, and also in the MY-NEW-BASE branch I rebase onto).


Something seems to be really fishy here and I wonder if anybody has an idea what's going wrong here. Is this a bug in git? Is my repos broken in some way? Note that "git fsck" reported nothing except some dangling objects. Any other ideas? 


Cheers,
Max

Re: rebase: strange failures to apply patc 3-way

[Max Horn]

A follow up:

I was able to reproduce this behavior on my Mac with multiple git versions down to 1.6.0.6. On the other hand, I copied the whole work tree to a virtual machine running Ubuntu, and there the issue did not occur.

All in all, I suspect that Mac OS X and/or the filesystem (HFS+ with journaling, not case sensitive (the default)) might be at fault. Still, this is quite puzzling and annoying, and so I still wonder if anybody has any insights on this.

And: is this a known issue, I wonder?


Cheers,
Max


On 07.03.2013, at 11:16, Max Horn wrote:

> Recently I have observed very strange failures in "git rebase" that cause it to fail to work automatically in situations where it should trivially be able to do so.
> 
> In case it matter, here's my setup: git 1.8.2.rc2.4.g7799588 (i.e. git.git master branch) on Mac OS X. The repos clone is on a HFS+ partition, not on a network volume. No gitattributes are being used.  Regarding the origin of the repos (I hope it doesn't matter, but just in case): The repository in question used to be a CVS repository; it was decided to switch to Mercurial (not my decision ;-) and I performed the conversion; I first converted it to git using cvs2git (from the cvs2svn suite), then performed some history cleanup, and finally used "hg convert" to convert it to git. Recently, I have been accessing the repository from git via the gitifyhg tool <https://github.com/buchuki/gitifyhg>. 
> 
> Anyway, several strange things just happened (and I had similar experiences in the past days and weeks, but that was using an older git, and I thought I was just doing something wrong).
> 
> Specifically, I wanted to rebase a branch with some experimental commits. The strange things started with this:
> 
> $ git rebase MY-NEW-BASE
> First, rewinding head to replay your work on top of it...
> Applying: SOME COMMIT
> Applying: SOME OTHER COMMIT
> ...
> Applying: COMMIT A
> Applying: COMMIT B
> Using index info to reconstruct a base tree...
> Falling back to patching base and 3-way merge...
> error: Your local changes to the following files would be overwritten by merge:
> 	some/source.file
> Please, commit your changes or stash them before you can merge.
> Aborting
> Failed to merge in the changes.
> Patch failed at 0014 COMMIT B
> The copy of the patch that failed is found in:
>   /path/to/my/repo/.git/rebase-apply/patch
> 
> When you have resolved this problem, run "git rebase --continue".
> If you prefer to skip this patch, run "git rebase --skip" instead.
> To check out the original branch and stop rebasing, run "git rebase --abort".
> 
> 
> Now, what is strange about this is that the failed patch actually applies cleanly:
> 
> $ patch -p1 < /path/to/my/repo/.git/rebase-apply/patch
> patching file some/source.file
> $
> 
> And there is no subtle merge issue here, either: That patch is the only one to have touched the surrounding code since 1999! There is no source of conflict there!
> 
> Anyway. The tale gets stranger, as I was trying to do this again (no changes were made to the repos in between, this is a straight continuation from above):
> 
> $ git rebase --abort
> $ git rebase MY-NEW-BASE
> First, rewinding head to replay your work on top of it...
> Applying: SOME COMMIT
> Applying: SOME OTHER COMMIT
> ...
> Applying: COMMIT A
> Using index info to reconstruct a base tree...
> Falling back to patching base and 3-way merge...
> error: Your local changes to the following files would be overwritten by merge:
> 	some/othersource.file
> 	some/yetanother.file
> Please, commit your changes or stash them before you can merge.
> Aborting
> Failed to merge in the changes.
> Patch failed at 0013 COMMIT A
> The copy of the patch that failed is found in:
>   /path/to/my/repo/.git/rebase-apply/patch
> 
> When you have resolved this problem, run "git rebase --continue".
> If you prefer to skip this patch, run "git rebase --skip" instead.
> To check out the original branch and stop rebasing, run "git rebase --abort".
> 
> 
> 
> So suddenly it fails to apply the commit A, the one before the previously failing commit. Huh? But again, the failing patch applies cleanly (and after all, rebase was able to apply it in my previous attempt).  And again, the patch actually applies cleanly. So one more try:
> 
> 
> $ git rebase --abort
> $ git rebase MY-NEW-BASE
> First, rewinding head to replay your work on top of it...
> Applying: SOME COMMIT
> Applying: SOME OTHER COMMIT
> ...
> Applying: COMMIT A
> Using index info to reconstruct a base tree...
> Falling back to patching base and 3-way merge...
> error: Your local changes to the following files would be overwritten by merge:
> 	some/othersource.file
> Please, commit your changes or stash them before you can merge.
> Aborting
> Failed to merge in the changes.
> Patch failed at 0013 COMMIT A
> The copy of the patch that failed is found in:
>   /path/to/my/repo/.git/rebase-apply/patch
> 
> When you have resolved this problem, run "git rebase --continue".
> If you prefer to skip this patch, run "git rebase --skip" instead.
> To check out the original branch and stop rebasing, run "git rebase --abort".
> 
> 
> Again it fails in commit A -- but this time, it only thinks one file is problematic. HUH? Again, the patch actually applies cleanly:
> 
> $ patch -p1 < /path/to/my/repo/.git/rebase-apply/patch
> patching file some/othersource.file
> patching file some/yetanother.file
> 
> 
> At this point, things stabilized, and when I now abort and reattempt the merge, it always fails in the same way. This time trying to apply a change to a code comment that was last changed in 1997 (though for one hunk, a few lines before the changed lines, there is a line that was changed in 2008... but I assure you, that line is there in the ancestors of both the branch I want to rebase, and also in the MY-NEW-BASE branch I rebase onto).
> 
> 
> Something seems to be really fishy here and I wonder if anybody has an idea what's going wrong here. Is this a bug in git? Is my repos broken in some way? Note that "git fsck" reported nothing except some dangling objects. Any other ideas? 
> 
> 
> Cheers,
> Max
> 
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: rebase: strange failures to apply patc 3-way

[Andrew Wong]

On 3/8/13, Max Horn <max@quendi.de> wrote:
> All in all, I suspect that Mac OS X and/or the filesystem (HFS+ with
> journaling, not case sensitive (the default)) might be at fault. Still, this
> is quite puzzling and annoying, and so I still wonder if anybody has any
> insights on this.

When "rebase" errors out at COMMIT A, try manually running "git apply"
on the patch file (rebase-apply/patch) a couple times, and see if the
error occurs randomly. You'd have to do a "reset --hard" to revert the
changes done by "git apply" every time before you run it again. The
error from "git apply" might shed more light on the issue.

Re: rebase: strange failures to apply patc 3-way

[Max Horn]

Am 08.03.2013 um 16:32 schrieb Andrew Wong <andrew.kw.w@gmail.com>:

> On 3/8/13, Max Horn <max@quendi.de> wrote:
>> All in all, I suspect that Mac OS X and/or the filesystem (HFS+ with
>> journaling, not case sensitive (the default)) might be at fault. Still, this
>> is quite puzzling and annoying, and so I still wonder if anybody has any
>> insights on this.
> 
> When "rebase" errors out at COMMIT A, try manually running "git apply"
> on the patch file (rebase-apply/patch) a couple times, and see if the
> error occurs randomly. You'd have to do a "reset --hard" to revert the
> changes done by "git apply" every time before you run it again. The
> error from "git apply" might shed more light on the issue.

I tried this a dozen times, but 'git apply' failed to fail even once. No surprise there, given that the patch that throws off rebase every time is clean and simple. I am flabbergasted :-(

Re: rebase: strange failures to apply patc 3-way

[Andrew Wong]

On 3/8/13, Max Horn <max@quendi.de> wrote:
> I tried this a dozen times, but 'git apply' failed to fail even once. No
> surprise there, given that the patch that throws off rebase every time is
> clean and simple. I am flabbergasted :-(

Hm, what if you add in the "--index" flag? i.e.
    git apply --index .git/rebase-apply/patch

Wonder if that makes any difference...

Re: rebase: strange failures to apply patc 3-way

[Max Horn]

Am 08.03.2013 um 19:02 schrieb Andrew Wong <andrew.kw.w@gmail.com>:

> On 3/8/13, Max Horn <max@quendi.de> wrote:
>> I tried this a dozen times, but 'git apply' failed to fail even once. No
>> surprise there, given that the patch that throws off rebase every time is
>> clean and simple. I am flabbergasted :-(
> 
> Hm, what if you add in the "--index" flag? i.e.
>    git apply --index .git/rebase-apply/patch
> 
> Wonder if that makes any difference...

Same result, it works fine.

Re: rebase: strange failures to apply patc 3-way

[Andrew Wong]

On 3/8/13, Max Horn <max@quendi.de> wrote:
> Same result, it works fine.

Just shooting in the dark here... I wonder if there's some background
process running in OS X that's messing with the files/directories
while rebase is working... backup, virus scan, etc? Or maybe some
programs that you're using at the same time? Maybe also make sure you
don't have any programs (shells, editors, etc.) opened that's
accessing those files/directories?

Does the error always happen at COMMIT A and COMMIT B? Or is it all
over the place?

In cases where COMMIT A succeeded, did it say it did a 3-way merge? Or
was it exactly as the output in your original message? i.e. no message
at all

Re: rebase: strange failures to apply patc 3-way

[Max Horn]

On 08.03.2013, at 20:20, Andrew Wong wrote:

> On 3/8/13, Max Horn <max@quendi.de> wrote:
>> Same result, it works fine.
> 
> Just shooting in the dark here... I wonder if there's some background
> process running in OS X that's messing with the files/directories
> while rebase is working... backup, virus scan, etc? Or maybe some
> programs that you're using at the same time? Maybe also make sure you
> don't have any programs (shells, editors, etc.) opened that's
> accessing those files/directories?

I am pretty sure no other programs are accessing those files at the same time. But just to make sure I quite most programs. No virus scanner running. No backup running -- although, OS X automatically runs hourly backups as part of Time Machine... So just to be triple certain, I black listed the repos dir in both the "Time Machine" backup service and the "Spotlight" indexing service.

No diference. In the end I even did a reboot, but that made no differences either (which I am quite relieved about, I must say ;-).


> 
> Does the error always happen at COMMIT A and COMMIT B? Or is it all
> over the place?

It tends to fail in separate places, but eventually "stabilizes". E.g. I just did a couple test rebases, and it failed twice in commit 14, then the third time in commit 15 (which underlines once more that the failures are inappropriate).

The fourth time, something new and weird happened:

$ git rebase --abort
$ git rebase NEW-PARENT 
Cannot rebase: You have unstaged changes.
Please commit or stash them.
$

This is quite suspicious. It appears that git for some reason things a file is dirty when it isn't. That could explain the other rebase failures too, couldn't it? But what might cause such a thing?


I checked with "git st" and it reported no changed files. Executing the "rebase" once again then "worked" as before... I.e. it got stuck in commit 15. The next time it got till commit 16. Then back to commit 15. Etc. Now it is getting stuck on commit 17 (but it doesn't always go up as it did right now).


> 
> In cases where COMMIT A succeeded, did it say it did a 3-way merge? Or
> was it exactly as the output in your original message? i.e. no message
> at all

It's always a variation of the same message as shown in my original email. I.e.:

Applying: ...
...
Applying: commit XYZ
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
error: Your local changes to the following files would be overwritten by merge:
	some/file
Please, commit your changes or stash them before you can merge.
Aborting
Failed to merge in the changes.
Patch failed at 0015 commit XYZ
The copy of the patch that failed is found in:
   /path/to/repos/.git/rebase-apply/patch





Thanks,
Max

Re: rebase: strange failures to apply patc 3-way

[Andrew Wong]

On 03/09/13 06:26, Max Horn wrote:
> It tends to fail in separate places, but eventually "stabilizes". E.g. I just did a couple test rebases, and it failed twice in commit 14, then the third time in commit 15 (which underlines once more that the failures are inappropriate).
>
> The fourth time, something new and weird happened:
>
> $ git rebase --abort
> $ git rebase NEW-PARENT 
> Cannot rebase: You have unstaged changes.
> Please commit or stash them.
> $
>
> This is quite suspicious. It appears that git for some reason things a file is dirty when it isn't. That could explain the other rebase failures too, couldn't it? But what might cause such a thing?
Yea, that's really suspicious. This could mean there's an issue with
when git is checking the index. Try running these a couple times in a
clean work tree:
    $ git update-index --refresh
    $ git diff-files

In a clean work tree, these commands should print nothing. But in your
case, these might print random files that git thinks have been modified...

If the commands do print out some files, check the timestamp from the
git index and the filesystem:
    $ git ls-files --debug file1 file2
    $ stat -f "%N %m %c" file1 file2

Is this repo on a network drive? Or is it local drive in your Mac?

Re: rebase: strange failures to apply patc 3-way

[Andrew Wong]

On 03/09/13 13:32, Andrew Wong wrote:
> Yea, that's really suspicious. This could mean there's an issue with
> when git is checking the index. Try running these a couple times in a
> clean work tree:
>     $ git update-index --refresh
>     $ git diff-files
>
> In a clean work tree, these commands should print nothing. But in your
> case, these might print random files that git thinks have been modified...
Before you run those commands each time, you probably have to "touch"
couple files to trigger the issue:
    $ touch file1 file2

Maybe use touch on the files that git rebase has been reporting error?

Re: rebase: strange failures to apply patc 3-way

[Max Horn]

Sorry for taking so long to reply... :-/

On 09.03.2013, at 19:32, Andrew Wong wrote:

> On 03/09/13 06:26, Max Horn wrote:
>> It tends to fail in separate places, but eventually "stabilizes". E.g. I just did a couple test rebases, and it failed twice in commit 14, then the third time in commit 15 (which underlines once more that the failures are inappropriate).
>> 
>> The fourth time, something new and weird happened:
>> 
>> $ git rebase --abort
>> $ git rebase NEW-PARENT 
>> Cannot rebase: You have unstaged changes.
>> Please commit or stash them.
>> $
>> 
>> This is quite suspicious. It appears that git for some reason things a file is dirty when it isn't. That could explain the other rebase failures too, couldn't it? But what might cause such a thing?
> Yea, that's really suspicious. This could mean there's an issue with
> when git is checking the index. Try running these a couple times in a
> clean work tree:
>    $ git update-index --refresh
>    $ git diff-files
> 
> In a clean work tree, these commands should print nothing. But in your
> case, these might print random files that git thinks have been modified...

I did run

  touch lib/*.* src/*.* && git update-index --refresh && git diff-files

a couple dozen times (the "problematic" files where in src/ and lib), but nothing happened. I just re-checked, and the rebase still fails in the same why...

Perhaps I should add some printfs into the git source to figure out what exactly it thinks is not right about those files... i.e. how does it come to the conclusion that I have local changes, exactly. I don't know how Git does that -- does it take the mtime from (l)stat into account? Perhaps problems with my machine's clock could be responsible?


> 
> If the commands do print out some files, check the timestamp from the
> git index and the filesystem:
>    $ git ls-files --debug file1 file2
>    $ stat -f "%N %m %c" file1 file2
> 
> Is this repo on a network drive? Or is it local drive in your Mac?

Local (some more details also described in my first email on this thread, but I'll happily provide more data if I can).

Thanks again,
Max

Re: rebase: strange failures to apply patc 3-way

[Andrew Wong]

On 3/10/13, Max Horn <max@quendi.de> wrote:
> I did run
>
>   touch lib/*.* src/*.* && git update-index --refresh && git diff-files
>
> a couple dozen times (the "problematic" files where in src/ and lib), but
> nothing happened. I just re-checked, and the rebase still fails in the same
> why...
>
> Perhaps I should add some printfs into the git source to figure out what
> exactly it thinks is not right about those files... i.e. how does it come to
> the conclusion that I have local changes, exactly. I don't know how Git does
> that -- does it take the mtime from (l)stat into account? Perhaps problems
> with my machine's clock could be responsible?

Instead of using "touch", maybe it'd be better if you run "ls-files"
and "stat" at the point where rebase failed. You should run the
command as soon as rebase failed. Don't try to run any git commands,
as they might change the index state.

And yes, git does make use of mtime and ctime from lstat to some
degree when detecting file changes. Inserting printf's to print the
timestamp might help, but the output might be too overwhelming to make
out useful information, especially during a rebase.

BTW, it looks like "stat" command on OS X only prints out timestamps
in seconds, and doesn't show you the nanoseconds part, which may be
significant in your situation. Instead of using the "stat" command,
try using this python command to print out the nanoseconds parts:
python -c "import sys;import os;s=os.stat(sys.argv[1]);print('%d, %f,
%f' % (s.st_size, s.st_ctime, s.st_mtime))" file1

Perhaps you could hack git-am.sh a bit to get more debugging info too.
Hm, maybe a good place to start is un-silencing the output of "git
apply". Inside "git-am.sh", you should see a line like:
    git apply $squelch
Remove the $squelch, and see what output it generates.

Also, since you're getting the 3-way merge, you could also insert the
"ls-files" and "stat" right after "git-merge-recursive", but before
"die".

Re: rebase: strange failures to apply patc 3-way

[Max Horn]

On 11.03.2013, at 20:15, Andrew Wong wrote:

> On 3/10/13, Max Horn <max@quendi.de> wrote:
>> I did run
>> 
>>  touch lib/*.* src/*.* && git update-index --refresh && git diff-files
>> 
>> a couple dozen times (the "problematic" files where in src/ and lib), but
>> nothing happened. I just re-checked, and the rebase still fails in the same
>> why...
>> 
>> Perhaps I should add some printfs into the git source to figure out what
>> exactly it thinks is not right about those files... i.e. how does it come to
>> the conclusion that I have local changes, exactly. I don't know how Git does
>> that -- does it take the mtime from (l)stat into account? Perhaps problems
>> with my machine's clock could be responsible?
> 
> Instead of using "touch", maybe it'd be better if you run "ls-files"
> and "stat" at the point where rebase failed. You should run the
> command as soon as rebase failed. Don't try to run any git commands,
> as they might change the index state.

So I tried this:

  git rebase branches/stable-4.6  # ... which leads to the error
  git ls-files -m

but got nothing. Perhaps you had something else in mind, though, but I am not quite sure what... sorry for being dense, but if you let me know what exactly you meant, I'll be happy to try that. As for the stat command:

> 
> And yes, git does make use of mtime and ctime from lstat to some
> degree when detecting file changes. Inserting printf's to print the
> timestamp might help, but the output might be too overwhelming to make
> out useful information, especially during a rebase.
> 
> BTW, it looks like "stat" command on OS X only prints out timestamps
> in seconds, and doesn't show you the nanoseconds part, which may be
> significant in your situation. Instead of using the "stat" command,
> try using this python command to print out the nanoseconds parts:
> python -c "import sys;import os;s=os.stat(sys.argv[1]);print('%d, %f,
> %f' % (s.st_size, s.st_ctime, s.st_mtime))" file1

I can do that, but I am not quite sure what the output should tell me? BTW, I think OS X just does not provide the stat information on a nanosecond level, at least that python command doesn't seem to yield useful extra data... Here is what I got after I just triggered the failing rebase (today it's again a different file it fails on that yesterday...). For comparision, I run stat first on the file with "conflicts", and then on a file that is not being touched by the rebase:

  File: ‘BAD-FILE’
  Size: 21058     	Blocks: 48         IO Block: 4096   regular file
Device: e000004h/234881028d	Inode: 19108072    Links: 1
Access: (0644/-rw-r--r--)  Uid: (  502/   mhorn)   Gid: (   20/   staff)
Access: 2013-03-11 21:40:14.000000000 +0100
Modify: 2013-03-11 21:40:12.000000000 +0100
Change: 2013-03-11 21:40:14.000000000 +0100
 Birth: 2013-03-11 21:40:12.000000000 +0100
  File: ‘NEUTRAL-FILE’
  Size: 1425      	Blocks: 8          IO Block: 4096   regular file
Device: e000004h/234881028d	Inode: 18180450    Links: 1
Access: (0644/-rw-r--r--)  Uid: (  502/   mhorn)   Gid: (   20/   staff)
Access: 2013-03-11 17:58:30.000000000 +0100
Modify: 2013-03-10 14:20:39.000000000 +0100
Change: 2013-03-10 14:20:39.000000000 +0100
 Birth: 2013-02-27 16:18:57.000000000 +0100

Here is the output of the python script for the two files:
21058, 1363034414.000000, 1363034412.000000
1425, 1362921639.000000, 1362921639.000000


One thing I notice is that ctime equals atime and both are larger than mtime. But I have no clue if that has any significance... hmm



> Perhaps you could hack git-am.sh a bit to get more debugging info too.
> Hm, maybe a good place to start is un-silencing the output of "git
> apply". Inside "git-am.sh", you should see a line like:
>    git apply $squelch
> Remove the $squelch, and see what output it generates.


error: BAD-FILE: does not match index

I inserted "git ls-files -m" before that but that didn't print anything nor had it any other effect.

> 
> Also, since you're getting the 3-way merge, you could also insert the
> "ls-files" and "stat" right after "git-merge-recursive", but before
> "die".
> 

Aha, so that was interesting: I inserted both a "stat" invocation, and also invoked the "md5" tool to be able to tell whether the file content matched what I thought it should match. Doing that caused the breaking commit to shift. So, I added a second stat / md5 pair on the new breaking file. After doing that, the rebase suddenly completed! I reset -- hard to the previous state, tried again, and again it worked. And this after consistently failing (albeit in differing places) in maybe a hundred or more tries before. Removing those two calls, it failed again. I tried inserting a "sleep 1", just in case it's a race, but with that it still failed.

I then re-added the stat / md5 calls, and -- it failed again, but this much earlier (in commit 8, not around commits 14-18). 

So I added the new conflicting fail to the stat/md5 calls (for a total of three files), and it failed in commit 4. Despite adding the new file (number 4),  it got stuck in this very commit in several attempts. I then for a while did some other stuff (like reading through the "git ls-files" man page ;-). Trying the rebase once more, with the stat / md5 calls on four files, it went back to failing in the place it started out failing today (commit 16). Adding / remove the stat&md5 invocations didn't seem to have an effect at this point.



Thanks,
Max

Re: rebase: strange failures to apply patc 3-way

[Max Horn]

PS: Just as a side note, I should mention that I have done tons of rebases on various repositories on this very machine: same hard drive, same file system; the git version of course has changed over time, but as I already described, I can reproduce the same issue with older git versions.

All I want to say here is: While this may of course be a bug of OS X or my machine may be faulty or whatever, it is not as if this issue is occurring all over the place. It is so far somehow specific to this repository, or even the particular branch. So it is probably not something as simple as a faulty clock, because then I should be seeing rebases fail all over the place, right? (Hm, OK, admittedly I didn't try any other big rebases in the last couple days. And right now I have no good example at hand, i.e. a non-trivial branch that cleanly rebases to some other branch)


Cheers,
Max

Re: rebase: strange failures to apply patc 3-way

[Andrew Wong]

On 3/11/13, Max Horn <max@quendi.de> wrote:
> So I tried this:
>
>   git rebase branches/stable-4.6  # ... which leads to the error
>   git ls-files -m
>
> but got nothing. Perhaps you had something else in mind, though, but I am
> not quite sure what... sorry for being dense, but if you let me know what
> exactly you meant, I'll be happy to try that. As for the stat command:

I'm still digesting the information form the email. Just want to
quickly mention that the "ls-files" command should be:
    $ git ls-files --debug file1 file2

You should always get output. It prints out stat-like information for
files in the index.

Re: rebase: strange failures to apply patc 3-way

[Andrew Wong]

On 3/11/13, Max Horn <max@quendi.de> wrote:
> PS: Just as a side note, I should mention that I have done tons of rebases
> on various repositories on this very machine: same hard drive, same file
> system; the git version of course has changed over time, but as I already
> described, I can reproduce the same issue with older git versions.

What if you do a "git clone" from this repo to an entirely new repo? I
wonder if the rebase issue still happens in the new repo...

Could you also post the .git/config file from the repo?

If supported, git could actually make use of threading when doing
"stat"... it should be disabled by default though, but you could try
disabling it with this config:
    git config core.preloadindex false

But I don't know why that'd only affect this one repo and not the
others though...

Re: rebase: strange failures to apply patc 3-way

[Max Horn]

On 11.03.2013, at 22:34, Andrew Wong wrote:

> On 3/11/13, Max Horn <max@quendi.de> wrote:
>> So I tried this:
>> 
>>  git rebase branches/stable-4.6  # ... which leads to the error
>>  git ls-files -m
>> 
>> but got nothing. Perhaps you had something else in mind, though, but I am
>> not quite sure what... sorry for being dense, but if you let me know what
>> exactly you meant, I'll be happy to try that. As for the stat command:
> 
> I'm still digesting the information form the email. Just want to
> quickly mention that the "ls-files" command should be:
>    $ git ls-files --debug file1 file2
> 
> You should always get output. It prints out stat-like information for
> files in the index.

Aha, I see. I inserted that into git-am, once right before git-merge-recursive and once right after it (just before a call to rerere).


This is what I got;


Applying: COMMIT X
error: BAD-FILE: does not match index
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
GOOD-FILE
  ctime: 1362921639:0
  mtime: 1362921639:0
  dev: 234881028	ino: 18665879
  uid: 502	gid: 20
  size: 25429	flags: 0
BAD-FILE
  ctime: 1363040264:0
  mtime: 1363040264:0
  dev: 234881028	ino: 19170773
  uid: 502	gid: 20
  size: 52906	flags: 0
error: Your local changes to the following files would be overwritten by merge:
	BAD-FILE
Please, commit your changes or stash them before you can merge.
Aborting
GOOD-FILE
  ctime: 1362921639:0
  mtime: 1362921639:0
  dev: 234881028	ino: 18665879
  uid: 502	gid: 20
  size: 25429	flags: 0
BAD-FILE
  ctime: 1363040264:0
  mtime: 1363040264:0
  dev: 234881028	ino: 19170773
  uid: 502	gid: 20
  size: 52906	flags: 0
Failed to merge in the changes.
Patch failed at 0015 COMMIT X
...

So the ctime/mtine for the BAD-FILE do not differ.
After this, I also run "ls-files" on the command line, and got:

GOOD-FILE gd
  ctime: 1362921639:0
  mtime: 1362921639:0
  dev: 234881028	ino: 18665879
  uid: 502	gid: 20
  size: 25429	flags: 0
BAD-FILE
  ctime: 1363040266:0
  mtime: 1363040264:0
  dev: 234881028	ino: 19170773
  uid: 502	gid: 20
  size: 52906	flags: 0

So now the ctime of the bad file is more recent than the mtime.


Max

Re: rebase: strange failures to apply patc 3-way

[Max Horn]

On 11.03.2013, at 23:10, Andrew Wong wrote:

> On 3/11/13, Max Horn <max@quendi.de> wrote:
>> PS: Just as a side note, I should mention that I have done tons of rebases
>> on various repositories on this very machine: same hard drive, same file
>> system; the git version of course has changed over time, but as I already
>> described, I can reproduce the same issue with older git versions.
> 
> What if you do a "git clone" from this repo to an entirely new repo? I
> wonder if the rebase issue still happens in the new repo...

The problem seems to be non-existent in a clone. 

> 
> Could you also post the .git/config file from the repo?

[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
	ignorecase = true
	precomposeunicode = false

Other than that, it just contains some a [remote] section and several [branch] sections. None of these contains any fancy (i.e. the branch sections just say "remote = origin" and give the name of the remote branch).

Looking at the git config man page to check what each of my config settings does, I discovered "trustctime". And adding
	trustctime = false
to .git/config made the rebase work, every single time. Huh. 


Adding this to the fact that a clone works fine, I wonder if  something *is* touching my files, but just in that directory. But what could it be? One nagging suspicion is the "file versioning" feature Apple introduced as part of Time Machine in OS X 10.7; it's kind of a "version control system for n00bs" for arbitrary documents. It has caused me some pain in the past.

But I just re-checked, and problematic repos is explicitly on the Time Machine exclusion list. I also used the "tmutil isexcluded FILES" to verify that the problematic files are really on the TM exclusion list. Finally, I moved the one of the repos subdirectory containing most of the problematic files, and then run "git checkout". In other instances, this sufficed to "disassociate" a file from an unwanted TM version history. But doing that had no effect here, i.e. also with the freshly regenerated files, the problems appear.

> 
> If supported, git could actually make use of threading when doing
> "stat"... it should be disabled by default though, but you could try
> disabling it with this config:
>    git config core.preloadindex false
> 
> But I don't know why that'd only affect this one repo and not the
> others though...
> 
This setting doesn't seem to have any effect on the issue at hand.


Cheers,
Max

Re: rebase: strange failures to apply patc 3-way

[Andrew Wong]

On 3/11/13, Max Horn <max@quendi.de> wrote:
> Looking at the git config man page to check what each of my config settings
> does, I discovered "trustctime". And adding
> 	trustctime = false
> to .git/config made the rebase work, every single time. Huh.
>
>
> Adding this to the fact that a clone works fine, I wonder if  something *is*
> touching my files, but just in that directory. But what could it be? One
> nagging suspicion is the "file versioning" feature Apple introduced as part
> of Time Machine in OS X 10.7; it's kind of a "version control system for
> n00bs" for arbitrary documents. It has caused me some pain in the past.
>
> But I just re-checked, and problematic repos is explicitly on the Time
> Machine exclusion list. I also used the "tmutil isexcluded FILES" to verify
> that the problematic files are really on the TM exclusion list. Finally, I
> moved the one of the repos subdirectory containing most of the problematic
> files, and then run "git checkout". In other instances, this sufficed to
> "disassociate" a file from an unwanted TM version history. But doing that
> had no effect here, i.e. also with the freshly regenerated files, the
> problems appear.

Would you be able to turn off Time Machine completely and do a few
tests? If it does works, then it becomes a matter of fixing Time
Machine...

Re: rebase: strange failures to apply patc 3-way

[Max Horn]

[-- Attachment #1: Type: text/plain, Size: 2648 bytes --]


On 11.03.2013, at 23:54, Andrew Wong wrote:

> On 3/11/13, Max Horn <max@quendi.de> wrote:
>> Looking at the git config man page to check what each of my config settings
>> does, I discovered "trustctime". And adding
>> 	trustctime = false
>> to .git/config made the rebase work, every single time. Huh.
>> 
>> 
>> Adding this to the fact that a clone works fine, I wonder if  something *is*
>> touching my files, but just in that directory. But what could it be? One
>> nagging suspicion is the "file versioning" feature Apple introduced as part
>> of Time Machine in OS X 10.7; it's kind of a "version control system for
>> n00bs" for arbitrary documents. It has caused me some pain in the past.
>> 
>> But I just re-checked, and problematic repos is explicitly on the Time
>> Machine exclusion list. I also used the "tmutil isexcluded FILES" to verify
>> that the problematic files are really on the TM exclusion list. Finally, I
>> moved the one of the repos subdirectory containing most of the problematic
>> files, and then run "git checkout". In other instances, this sufficed to
>> "disassociate" a file from an unwanted TM version history. But doing that
>> had no effect here, i.e. also with the freshly regenerated files, the
>> problems appear.
> 
> Would you be able to turn off Time Machine completely and do a few
> tests? If it does works, then it becomes a matter of fixing Time
> Machine...

I did turn it off just now, but no effect. Then again, daemons like revisiond were still running...


One more thing: I used the "fs_usage" to monitor what process accessed what file during one of those failing "git rebase" runs. I then searched through this to determine which processes accessed the "bad" file in this time. The result where these processes (aggregated):

  git
  revisiond
  fseventsd
  git-merge-recursive

Note that Time Machine was not running, but revisiond is... from its manpage:

     revisiond is the daemon that manages document revisions created by applications and system services.

     There are no configurations to revisiond, and users should not run revisiond manually.

This is the process I am suspecting. Specifically, a fchflags call it makes (see that attached excerpt of the fs_usage output). I am not sure, but my guess would be that it sets SF_ARCHIVED on the file. Causing its ctime to wander... Yuck.

But I don't know how to control it... and I am not sure if I can just kill it. Hrm. I'll try to dig some more into it, perhaps I can somehow confirm that this is *really* the cause of the problems, and somehow prevent it.


Cheers,
Max


[-- Attachment #2: git-rebase-debug.txt --]
[-- Type: text/plain, Size: 6515 bytes --]

00:59:54.349156  lstat64                                src/BAD_FILE                                             0.000050   git.623953
00:59:54.349160  open              F=5        (R_____)  src/BAD_FILE                                             0.000004   git.623953
00:59:54.350659  close             F=5                                                                           0.000007   git.623953
00:59:54.371716  lstat64                                src/BAD_FILE                                             0.000002   git.623955
00:59:54.429674  lstat64                                src/BAD_FILE                                             0.000002   git.623959
00:59:54.600060  lstat64                                src/BAD_FILE                                             0.000007   git.623959
00:59:54.600151  stat64                                 /Users/mhorn/the-path-to-my-repository/src/BAD_FILE      0.000006   revisiond.623963
00:59:54.600154  stat64                                 /Users/mhorn/the-path-to-my-repository/src               0.000003   revisiond.623963
00:59:54.600160  open              F=7        (R_____)  /Users/mhorn/the-path-to-my-repository/src/BAD_FILE      0.000006   revisiond.623963
00:59:54.600161  fstatfs64         F=7                                                                           0.000002   revisiond.623963
00:59:54.600163  close             F=7                                                                           0.000002   revisiond.623963
00:59:54.600387  unlink                                 src/BAD_FILE                                             0.000328 W git.623959
00:59:54.600429  open              F=5        (_WC__E)  src/BAD_FILE                                             0.000039   git.623959
00:59:54.602910  write             F=5    B=0x4000                                                               0.000040   git.623959
00:59:54.602932  write             F=5    B=0x4000                                                               0.000017   git.623959
00:59:54.602947  write             F=5    B=0x4000                                                               0.000011   git.623959
00:59:54.602958  write             F=5    B=0x4000                                                               0.000009   git.623959
00:59:54.602969  write             F=5    B=0x4000                                                               0.000009   git.623959
00:59:54.602977  write             F=5    B=0x12f5                                                               0.000007   git.623959
00:59:54.602983  fstat64           F=5                                                                           0.000004   git.623959
00:59:54.603032  close             F=5                                                                           0.000049   git.623959
00:59:54.621731  lstat64                                /Users/mhorn/the-path-to-my-repository/src/BAD_FILE      0.000004   fseventsd.1664
00:59:54.882870  lstat64                                src/BAD_FILE                                             0.000002   git.623993
00:59:54.882872  open              F=5        (R_____)  src/BAD_FILE                                             0.000003   git.623993
00:59:54.883042  close             F=5                                                                           0.000002   git.623993
00:59:55.021956  lstat64                                src/BAD_FILE                                             0.000003   git.624027
00:59:55.021959  open              F=4        (R_____)  src/BAD_FILE                                             0.000003   git.624027
00:59:55.022138  close             F=4                                                                           0.000003   git.624027
00:59:56.600440  open              F=7        (R_____)  /Users/mhorn/the-path-to-my-repository/src/BAD_FILE      0.000014   revisiond.623963
00:59:56.600442  fcntl             F=7   <GETPATH>                                                               0.000002   revisiond.623963
00:59:56.600445  close             F=7                                                                           0.000004   revisiond.623963
00:59:56.600449  stat64                                 /Users/mhorn/the-path-to-my-repository/src/BAD_FILE      0.000004   revisiond.623963
00:59:56.600452  stat64                                 /Users/mhorn/the-path-to-my-repository/src               0.000003   revisiond.623963
00:59:56.600455  open              F=7        (R_____)  /Users/mhorn/the-path-to-my-repository/src/BAD_FILE      0.000004   revisiond.623963
00:59:56.600457  fstatfs64         F=7                                                                           0.000002   revisiond.623963
00:59:56.600459  close             F=7                                                                           0.000002   revisiond.623963
00:59:56.600687  open              F=7        (R_____)  /Users/mhorn/the-path-to-my-repository/src/BAD_FILE      0.000006   revisiond.623963
00:59:56.600688  fstat64           F=7                                                                           0.000002   revisiond.623963
00:59:56.600698  fchflags          F=7   <UNKNOWN>       [garbled output removed]                                0.000010   revisiond.623963
00:59:56.600701  close             F=7                                                                           0.000003   revisiond.623963
00:59:56.624161  lstat64                                /Users/mhorn/the-path-to-my-repository/src/BAD_FILE      0.000004   fseventsd.1664
00:59:56.981172  lstat64                                src/BAD_FILE                                             0.000004   git.624517
00:59:57.015622  lstat64                                src/BAD_FILE                                             0.000005   git.624527
00:59:57.118844  lstat64                                src/BAD_FILE                                             0.000005   git-merge-recurs.624544
01:00:00.194634  lstat64                                src/BAD_FILE                                             0.000002   git.624580
01:00:00.194637  open              F=5        (R_____)  src/BAD_FILE                                             0.000003   git.624580
01:00:00.194815  close             F=5                                                                           0.000003   git.624580

Re: rebase: strange failures to apply patc 3-way

[Max Horn]

On 11.03.2013, at 23:54, Andrew Wong wrote:

> On 3/11/13, Max Horn <max@quendi.de> wrote:
>> Looking at the git config man page to check what each of my config settings
>> does, I discovered "trustctime". And adding
>> 	trustctime = false
>> to .git/config made the rebase work, every single time. Huh.
>> 
>> 
>> Adding this to the fact that a clone works fine, I wonder if  something *is*
>> touching my files, but just in that directory. But what could it be? One
>> nagging suspicion is the "file versioning" feature Apple introduced as part
>> of Time Machine in OS X 10.7; it's kind of a "version control system for
>> n00bs" for arbitrary documents. It has caused me some pain in the past.
>> 
>> But I just re-checked, and problematic repos is explicitly on the Time
>> Machine exclusion list. I also used the "tmutil isexcluded FILES" to verify
>> that the problematic files are really on the TM exclusion list. Finally, I
>> moved the one of the repos subdirectory containing most of the problematic
>> files, and then run "git checkout". In other instances, this sufficed to
>> "disassociate" a file from an unwanted TM version history. But doing that
>> had no effect here, i.e. also with the freshly regenerated files, the
>> problems appear.
> 
> Would you be able to turn off Time Machine completely and do a few
> tests? If it does works, then it becomes a matter of fixing Time
> Machine...
> 

sudo launchctl unload /System/Library/LaunchDaemons/com.apple.revisiond.plist

-> this exits revisiond, and prevents launchd from respawning it. After that, the problem is gone, on several attempts. And once I reactivate revisions, the problem reoccurs.

So it seems pretty clear what the cause of this is. Now I still need to figure out why it affects that particular repository and not others. But at this point I guess it is safe to say that Apple's auto-crap-magic revisiond is the root of the problem, and git is purely a victim. So I'll stop spamming this list with this issue for now, and see if I can figure out a fix that is less invasive than turning off revisiond completely (which some application rely on, so disabling it may break them badly).

Andrew, thank you very much for your persistent guidance and help. I definitely owe you a beverage-of-your-choice ;-).

Cheers,
Max

Re: rebase: strange failures to apply patc 3-way

[John Szakmeister]

On Mon, Mar 11, 2013 at 8:29 PM, Max Horn <max@quendi.de> wrote:
[snip]
>
> sudo launchctl unload /System/Library/LaunchDaemons/com.apple.revisiond.plist
>
> -> this exits revisiond, and prevents launchd from respawning it. After that, the problem is gone, on several attempts. And once I reactivate revisions, the problem reoccurs.
>
> So it seems pretty clear what the cause of this is. Now I still need to figure out why it affects that particular repository and not others. But at this point I guess it is safe to say that Apple's auto-crap-magic revisiond is the root of the problem, and git is purely a victim. So I'll stop spamming this list with this issue for now, and see if I can figure out a fix that is less invasive than turning off revisiond completely (which some application rely on, so disabling it may break them badly).

I just wanted to say thank you for spamming the list with this.  I've
not upgraded to a new Mac OS X yet, but now I know I need to watch out
for this issue.  I'm glad you were able to get to the bottom of it,
and I'd love to hear if you find a reasonable solution.

-John

Re: rebase: strange failures to apply patc 3-way

[Andrew Wong]

On 03/11/13 20:29, Max Horn wrote:
> sudo launchctl unload /System/Library/LaunchDaemons/com.apple.revisiond.plist
>
> -> this exits revisiond, and prevents launchd from respawning it. After that, the problem is gone, on several attempts. And once I reactivate revisions, the problem reoccurs.
>
> So it seems pretty clear what the cause of this is. Now I still need to figure out why it affects that particular repository and not others. But at this point I guess it is safe to say that Apple's auto-crap-magic revisiond is the root of the problem, and git is purely a victim. So I'll stop spamming this list with this issue for now, and see if I can figure out a fix that is less invasive than turning off revisiond completely (which some application rely on, so disabling it may break them badly).
>
> Andrew, thank you very much for your persistent guidance and help. I definitely owe you a beverage-of-your-choice ;-).
Awesome! I was starting to worry that we'd never figure out what the
issue is. Good thing you know your way around OS X and its debugging
tools. :)

Maybe drop an email if you figure out what's causing revisiond to be so
persistent about those specific files. Might help any who run into
similar issues in the future.

Cheers!

Re: rebase: strange failures to apply patc 3-way

[Jeff King]

On Tue, Mar 12, 2013 at 01:29:45AM +0100, Max Horn wrote:

> So it seems pretty clear what the cause of this is. Now I still need
> to figure out why it affects that particular repository and not
> others. But at this point I guess it is safe to say that Apple's
> auto-crap-magic revisiond is the root of the problem, and git is
> purely a victim. So I'll stop spamming this list with this issue for
> now, and see if I can figure out a fix that is less invasive than
> turning off revisiond completely (which some application rely on, so
> disabling it may break them badly).

>From "git help config":

  core.trustctime
    If false, the ctime differences between the index and the working
    tree are ignored; useful when the inode change time is regularly
    modified by something outside Git (file system crawlers and some
    backup systems). See git-update- index(1). True by default.

-Peff

Re: rebase: strange failures to apply patc 3-way

[Andrew Wong]

On 03/11/13 21:01, Jeff King wrote:
> From "git help config":
>
>   core.trustctime
>     If false, the ctime differences between the index and the working
>     tree are ignored; useful when the inode change time is regularly
>     modified by something outside Git (file system crawlers and some
>     backup systems). See git-update- index(1). True by default.
In an earlier email, Max did mention setting the config does workaround
the issue. But it's still strange that it only happens to a few specific
files in this particular repo. The issue never popped up in other repos
that he has, which I assume has all been excluded from Time Machine...

Re: rebase: strange failures to apply patc 3-way

[Jeff King]

On Mon, Mar 11, 2013 at 09:03:42PM -0400, Andrew Wong wrote:

> On 03/11/13 21:01, Jeff King wrote:
> > From "git help config":
> >
> >   core.trustctime
> >     If false, the ctime differences between the index and the working
> >     tree are ignored; useful when the inode change time is regularly
> >     modified by something outside Git (file system crawlers and some
> >     backup systems). See git-update- index(1). True by default.
>
> In an earlier email, Max did mention setting the config does workaround
> the issue. But it's still strange that it only happens to a few specific
> files in this particular repo. The issue never popped up in other repos
> that he has, which I assume has all been excluded from Time Machine...

Ah, sorry, I missed the earlier reference to it. trustctime is the only
sensible thing to do from the git side. I think figuring out the rest is
deep Apple magic that I'm not qualified to comment on.

-Peff

Re: rebase: strange failures to apply patc 3-way

[Heiko Voigt]

On Mon, Mar 11, 2013 at 09:10:04PM -0400, Jeff King wrote:
> On Mon, Mar 11, 2013 at 09:03:42PM -0400, Andrew Wong wrote:
> 
> > On 03/11/13 21:01, Jeff King wrote:
> > > From "git help config":
> > >
> > >   core.trustctime
> > >     If false, the ctime differences between the index and the working
> > >     tree are ignored; useful when the inode change time is regularly
> > >     modified by something outside Git (file system crawlers and some
> > >     backup systems). See git-update- index(1). True by default.
> >
> > In an earlier email, Max did mention setting the config does workaround
> > the issue. But it's still strange that it only happens to a few specific
> > files in this particular repo. The issue never popped up in other repos
> > that he has, which I assume has all been excluded from Time Machine...
> 
> Ah, sorry, I missed the earlier reference to it. trustctime is the only
> sensible thing to do from the git side. I think figuring out the rest is
> deep Apple magic that I'm not qualified to comment on.

>From what I read isn't the ctime the only thing that changes? I read
that you tried this option and the issues disappeared? Maybe the
resolution (or part of it) is to change the default of core.trustctime
to false on osx?

Cheers Heiko